add new deploy label to manage third-party gpu client pods#2607
Open
tariq1890 wants to merge 1 commit into
Open
add new deploy label to manage third-party gpu client pods#2607tariq1890 wants to merge 1 commit into
tariq1890 wants to merge 1 commit into
Conversation
|
coverage: 31.9%. remained the same — new-gpu-deploy-client-labels into main |
tariq1890
force-pushed
the
new-gpu-deploy-client-labels
branch
3 times, most recently
from
3b41429 to
420e19f
Compare
tariq1890
requested review from
cdesiniotis,
karthikvetrivel,
rahulait,
rajathagasthya and
shivamerla
as code owners
cdesiniotis
reviewed
Jul 7, 2026
This was referenced Jul 7, 2026
tariq1890
force-pushed
the
new-gpu-deploy-client-labels
branch
3 times, most recently
from
d9b29eb to
c3c5712
Compare
Contributor
Author
|
Thanks for the review @rahulait . Looks like I missed a change. Seeing the errors below when testing mig-manager |
tariq1890
force-pushed
the
new-gpu-deploy-client-labels
branch
from
c3c5712 to
9b77c0f
Compare
Signed-off-by: Tariq Ibrahim <tibrahim@nvidia.com>
tariq1890
force-pushed
the
new-gpu-deploy-client-labels
branch
from
9b77c0f to
45fb0db
Compare
Contributor
Author
|
This PR has been updated based on review comments provided in: |
rajathagasthya
approved these changes
Jul 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces a new node label
"nvidia.com/gpu.deploy.client": "true"to designate pods that are gpu clients (like our operandsdcgm,mig-manager,device-plugin,gpu-feature-discovery) which are not first-class components of the gpu-operator.Motivation
Advanced users bring in their own gpu client pods that require management/admin access to the GPUs so they too need to bypass the device plugin for gpu allocation much like the gpu-operator's operands. Today, users who bring their own gpu client pods will face issues during driver upgrades as the gpu-operator is unaware of the existence of these pods which actively hold gpu device handles thus interrupting a driver upgrade. Users today will have to manually evict these pods during these gpu client pods so that the upgrade can proceed. This scales poorly and also entails heavy operational overhead as it involves human intervention.
Solution
Users who bring their own gpu client pods can now take advantage of this feature by adding the
"nvidia.com/gpu.deploy.client": "true"label to the nodeSelector of their gpu client's resource manifest (whether Daemonset/Deployment/StatefulSet/Job). With this new label added to the nodeSelector, the gpu-operator will then be able to bounce gpu-client pods automatically during a driver upgrade without any human intervention.Related PRs: