Add GPUCluster CRD and controller for DRA-based stack by karthikvetrivel · Pull Request #2571 · NVIDIA/gpu-operator

karthikvetrivel · 2026-06-23T15:47:15Z

1. Overview

We introduce a new CRD named GPUCluster and a new controller for reconciling it. Like ClusterPolicy today, it is a singleton, cluster-scoped CRD that configures the operands needed to enable GPUs in Kubernetes. GPUCluster represents the new DRA-based software-enablement stack; it is an evolution of ClusterPolicy. Unlike ClusterPolicy, it does not manage the driver or a device plugin: the driver is either preinstalled on the host or managed by NVIDIADriver CRs, and GPUs are surfaced to workloads through DRA.

A GPUCluster may coexist with a ClusterPolicy in the same cluster, with every GPU node served by exactly one of the two stacks.

2. Testing

Unit tests cover the new API, operand rendering, controller status handling, and the node-labeling logic. I also tested this end-to-end on a 2-node heterogeneous cluster (A100 + T4, Kubernetes v1.34):

GPUCluster-only install via Helm: all operands reconcile to ready. DRA allocation was validated for full GPUs, MIG, and adminAccess. The dra-driver-validator was verified against both host-installed and NVIDIADriver-managed drivers.
Coexistence: ClusterPolicy and GPUCluster run in one cluster, with one node per plane. Each node runs only its plane's operands, with extended resources on the device-plugin node and ResourceSlices on the DRA node.
Mode switching: flipping nvidia.com/gpu-operator.mode in either direction swaps the operand set. A shared NVIDIADriver-managed driver survives the switch without a reload.

Quick author design note: the driver daemonset deployed by the NVIDIADriver CR is not gated on the deploy mode so that it is not evicted and re-scheduled when the user switches modes.

Label lifecycle: driver-manager pause values survive reconciles. A node's previous-plane labels are cleaned up, and unrecognized gpu.deploy.* labels are left untouched.

I have not tested ComputeDomain functionality yet.

3. Next Steps

PRs to land:

We also need minor changes to the upgrade controller, which I'm working on as a follow-up PR. Context: DRA nodes are auto-enrolled in driver upgrades (the shared driver DaemonSet has no mode gate) but can never pass the hard-coded nvidia-operator-validator validation gate, leaving them cordoned in validation-required indefinitely — until fixed, autoUpgrade should be disabled for NVIDIADriver CRs serving DRA nodes. The upgrade controller is also currently ClusterPolicy-gated and must change.

After we have a new validator daemonset, this PR can be marked as ready:

Drain the DRA kubelet-plugin and validator across driver reloads k8s-driver-manager#203

Moved from #2513 (re-created with the head branch on NVIDIA/gpu-operator instead of a fork, to enable stacked PRs). The earlier review discussion — including the GPUClusterConfig → GPUCluster naming suggestion, since adopted — lives in #2513.

cdesiniotis · 2026-06-24T15:35:40Z

+# NVIDIADriver CR). GPUClusterConfig does not manage the driver or device plugin
+# itself; it waits for driver readiness before deploying the DRA driver.
+gpuClusterConfig:
+  enabled: false


We should think a bit more on the right interface for this. A few questions come to mind:

Is enabled the right name for this field? As currently implemented, setting gpuClusterConfig.enabled=true will create a default GPUClusterConfig CR and will NOT create a default ClusterPolicy CR when the helm chart gets rendered. This may not be clear to the user.

Should the draDriver struct be embedded under the top-level gpuClusterConfig struct?

I was thinking about this. The logic has changed since your comment and now a ClusterPolicy CR is always created.

If we remove this enabled flag, are we okay with a GPUCluster CR always being created? This will cause a issue on clusters where the API server doesn't serve DRA APIs.

and now a ClusterPolicy CR is always created.

IMO we should provide the ability to NOT create a ClusterPolicy CR (in favor of a GPUCluster CR) on helm chart install. I am not quite sure what the interface should be here.

Thinking out loud here, what about:

defaultCRs: clusterPolicy: enabled: true gpuCluster: enabled: false

E.g. to only deploy a default GPUCluster CR, one would set defaultCRs.clusterPolicy.enabled=false defaultCRs.gpuCluster.enabeld=true.

@tariq1890 @rahulait @rajathagasthya any thoughts here?

What about making the booleans on the same level?

defaultCRsEnabled: clusterPolicy: true gpuCluster: false

coveralls · 2026-07-01T15:31:44Z

coverage: 32.636% (+0.7%) from 31.9% — kv-gpuclusterconfig-crd into main

cdesiniotis · 2026-07-07T16:40:05Z

+	// The driver auto-upgrade annotation is derived from ClusterPolicy spec.
+	if nlc.clusterPolicy != nil {
+		if err := nlc.applyDriverAutoUpgradeAnnotation(ctx); err != nil {
+			return reconcile.Result{}, err
+		}


The auto-upgrade annotation is used by the k8s-driver-manager init container in the driver daemonset. So we need to set it regardless if ClusterPolicy or GPUCluster is used. We need to decouple the applyDriverAutoUpgradeAnnotation method from clusterpolicy -- it should account for when clusterpolicy is nil.

cdesiniotis · 2026-07-07T16:44:33Z

+	if _, ok := labels[consts.GPUOperatorModeLabelKey]; ok {
+		return false
+	}
+	if nlc.defaultMode == "" {


Question -- would it be better to always default to one stack? E.g. if the user misconfigured the PREFERRED_MODE env or left it empty, shouldn't we fallback to a default set in the code?

If the user misconfigured the env or left it empty, defaultMode would still resolve to device-plugin (see resolveDefaultMode).

I can see how this guard is a little confusing. This check exists in case neither GPUCluster CR or ClusterPolicy CR exists (in which case, the mode would be an empty string on every node). Right now, this shouldn't be a concern but is defensive against future changes in the node labeling controller.

I can add a concise comment to clarify, unless you prefer removing it. I have no strong preference.

No strong opinion from me either. I think I slightly prefer removing since A) this method should never be called if neither a ClusterPolicy or GPUCluster CR exist (as Reconcile will return early in this case), and B) a reasonable assumption to make is that defaultMode is resolved / set prior to reconcileModeLabel() being called.

cdesiniotis · 2026-07-07T16:46:53Z

+			modified = true
+		}
+		return modified
+	case consts.GPUOperatorModeDevicePlugin:


Question -- shouldn't we be calling sweepOtherPlaneStateLabels() in this case as well?

Under the GPUOperatorModeDevicePlugin case, we use the legacy ClusterPolicy engine.

https://github.com/NVIDIA/gpu-operator/blob/main/controllers/state_manager.go#L396-L405

At some point, I think it makes sense to unify these sweeps into a single shared sync engine (i.e. pull removeGPUStateLabels/addGPUStateLabels out of gpuWorkloadConfiguration) but I think that is out of scope for this PR.

cdesiniotis · 2026-07-07T17:03:20Z

+  name: nvidia-dra-validator
+  namespace: {{ .Namespace }}
+  labels:
+    app.kubernetes.io/name: nvidia-dra-validator


Question -- what's the reason for naming this nvidia-dra-validator instead of nvidia-operator-validator? If we kept the same name (and node selector) as the validator deployed by ClusterPolicy, wouldn't that solve our issue with the driver upgrade-controller (and how it depends on the readiness of the nvidia-operator-validator pod)?

Good point. One note: the upgrade controller keys on the pod label, not the name

WithValidationEnabled("app=nvidia-operator-validator")

So I changed the daemonset pod template to have the same app app: nvidia-operator-validator. I didn't make the names the same for collisions reasons (even if the pods land on different clusters, I believe this is still an issue).

cdesiniotis · 2026-07-07T17:18:08Z

Note, there are a few other changes needed in this file. E.g. add a sample GPUCluster CR in .metadata.annotations.alm-examples, update the operator's ClusterRole and Role definitions, etc.

Good catch, updated.

tariq1890 · 2026-07-07T17:42:41Z

          effect: NoSchedule
      nodeSelector:
        nvidia.com/gpu.deploy.vfio-manager: "true"
+        nvidia.com/gpu-operator.mode: "device-plugin"


I am thinking if we should rename this to something like

nvidia.com/gpu-operator.resource-allocation.mode: "device-plugin"

I like this name more.

@cdesiniotis @rahulait @rajathagasthya thoughts?

I also prefer the proposed name over what we currently have.

tariq1890 · 2026-07-07T17:57:04Z

+
+// getDriverLibraryPath returns path to `libnvidia-ml.so.1` in the driver root.
+// The folder for this file is also expected to be the location of other driver files.
+func (r root) getDriverLibraryPath() (string, error) {


Look's like there is a lot of duplicated code here (duplicate of cmd/validator/find.go). Can we look into re-using the code in cmd/validator instead?

Updated to pull out some common pieces. It should look a lot better now.

Pause the DRA validator with other claim-holding clients and keep the kubelet plugin available through targeted and full node drains. Stop the plugin only after those claims are released and before unloading the driver, then restore both deployment labels. This consumes labels from NVIDIA/gpu-operator#2571. NVIDIA DRA user-pod discovery and eviction is handled separately by NVIDIA#204. Signed-off-by: Karthik Vetrivel <kvetrivel@nvidia.com>

Signed-off-by: Karthik Vetrivel <kvetrivel@nvidia.com>

cdesiniotis

Made another pass. @karthikvetrivel thanks for the patience on this one.

cdesiniotis · 2026-07-10T21:38:13Z

+	// GFD defines the spec for the standalone GPU Feature Discovery operand. Enabled by default,
+	// but the reused enabled field carries no server-side default; the controller defaults nil enabled.
+	GFD *nvidiav1.GPUFeatureDiscoverySpec `json:"gfd,omitempty"`


As discussed with @shivamerla, let's omit GFD for now. We can add it to the GPUCluster CR in a follow-up PR if we ever change our mind. But for now, my understanding is that kubernetes-sigs/dra-driver-nvidia-gpu#1247 will remove the hard dependency on GFD.

cdesiniotis · 2026-07-10T21:47:39Z

+type DRADriverGPUsSpec struct {
+	// Enabled indicates if the gpus capability of the DRA driver is enabled.
+	// +kubebuilder:default=true
+	Enabled *bool `json:"enabled,omitempty"`


This was pointed out by @rajathagasthya in the design doc, but I believe we can remove this field. The GPUCluster CR assumes that GPUs are managed by the DRA Driver, right? I don't think I see a use case where one is using the GPUCluster CR but wants to disable the GPU portion of the DRA driver (e.g. only use compute domains).

cdesiniotis · 2026-07-10T21:48:19Z

+// DaemonSet; the controller is a separate Deployment.
+type DRADriverComputeDomainsSpec struct {
+	// Enabled indicates if the computeDomains capability of the DRA driver is enabled.
+	// +kubebuilder:default=true


@shivamerla are you okay if we enable compute domains by default?

cdesiniotis · 2026-07-10T21:52:25Z

          effect: NoSchedule
      nodeSelector:
        nvidia.com/gpu.deploy.vfio-manager: "true"
+        nvidia.com/gpu-operator.mode: "device-plugin"


I also prefer the proposed name over what we currently have.

cdesiniotis · 2026-07-10T21:53:05Z

    - name: gpu-operator-image
      image: ghcr.io/nvidia/gpu-operator:main-latest
+    - name: nvidia-dra-driver-image
+      image: registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu:v0.4.0


Let's append the image digest as well.

cdesiniotis · 2026-07-10T21:54:55Z

                  - name: "VALIDATOR_IMAGE"
                    value: "ghcr.io/nvidia/gpu-operator:main-latest"
+                  - name: "DRA_DRIVER_IMAGE"
+                    value: "registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu:v0.4.0"


Let's append the image digest as well.

cdesiniotis · 2026-07-10T22:06:04Z

+
+// runValidation mirrors nvidia-validator's host-then-container probe, but validates
+// with `nvidia-smi --version` only. It returns the env-contract driverInfo on success.
+func runValidation() (driverInfo, error) {


Question -- how does this differ from the driver-validation code we have in cmd/nvidia-validator? IIUC the driver validation code is not specific to DRA, so why introduce a new executable, dra-driver-validator, in the first place?

cdesiniotis · 2026-07-10T22:16:22Z

+// writeDriverReady writes the driver-ready env contract sourced by the
+// kubelet-plugin containers. Only NVIDIA_DRIVER_ROOT and DRIVER_ROOT_CTR_PATH are
+// emitted: those are the only values the DRA kubelet plugins read (see
+// cmd/gpu-kubelet-plugin/main.go in k8s-dra-driver-gpu). The plugin derives its
+// dev root from DRIVER_ROOT_CTR_PATH itself.


Is this the primary motivation for creating a new dra-driver-validator executable instead of reusing nvidia-validator? If yes, I would argue we should just re-use the driver-validation code from the nvidia-validator binary as-is instead. (feel free to push back if there is another difference I am not seeing)

Note, exporting the DEV_ROOT specific envvars should have no negative side effects since they don't mean anything to the dra-driver-kubelet-plugin.

Just for reference, the DRA driver infers the dev root here: https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/blob/9001f17e513115a9366987bf5fd9f7850ac52368/cmd/gpu-kubelet-plugin/device_state.go#L88 (it will either resolve to DRIVER_ROOT_CTR_PATH or /, which aligns with what our nvidia-validator code does).

cdesiniotis · 2026-07-10T22:32:30Z

+	if _, ok := labels[consts.GPUOperatorModeLabelKey]; ok {
+		return false
+	}
+	if nlc.defaultMode == "" {


No strong opinion from me either. I think I slightly prefer removing since A) this method should never be called if neither a ClusterPolicy or GPUCluster CR exist (as Reconcile will return early in this case), and B) a reasonable assumption to make is that defaultMode is resolved / set prior to reconcileModeLabel() being called.

cdesiniotis · 2026-07-10T22:46:02Z

Not a blocker -- should we consider adding validations in the helm chart to prevent users from deploying the GPUCluster CR on a k8s cluster that does not support DRA? See what I had previously proposed: https://github.com/NVIDIA/gpu-operator/pull/1541/changes#diff-b056279a4ae42adaeb1878776dd3d6e426565027c47ee47f7f8cc607708d5b16

tariq1890 · 2026-07-07T18:03:16Z

+		mgr.GetClient(),
+		mgr.GetScheme())
+	if err != nil {
+		return fmt.Errorf("error creating state manager: %v", err)


Suggested change

return fmt.Errorf("error creating state manager: %v", err)

return fmt.Errorf("error creating state manager: %w", err)

tariq1890 · 2026-07-07T18:08:43Z

+
+	preferredMode := os.Getenv(consts.PreferredModeEnvName)
+	if clusterPolicy != nil && gpuCluster != nil &&
+		preferredMode != consts.GPUOperatorModeDevicePlugin && preferredMode != consts.GPUOperatorModeDRA {


For the mode, let's use a string-based enum so that we add a constraint on the acceptable values. If an invalid value is provided, we should just error out clearly explaining that the supplied value for the preferred mode is invalid.

tariq1890 · 2026-07-07T18:09:27Z

+	return modified
+}
+
+// sweepOtherPlaneStateLabels deletes the given other-plane deploy-label keys, value-blind.


What is "other-plane"?

tariq1890 · 2026-07-07T18:10:33Z

 		name          string
 		useCRD        *bool
 		driverEnabled *bool
+		gccExists     bool


Suggested change

gccExists bool

gpuClusterExists bool

tariq1890 · 2026-07-10T23:00:32Z

+	// when disabled, dcgm-exporter uses its embedded nv-hostengine. NOTE: the reused enabled
+	// field carries no server-side default and its IsEnabled() treats nil as enabled, so the
+	// controller must default nil enabled to disabled here.
+	DCGM *nvidiav1.DCGMSpec `json:"dcgm,omitempty"`


@cdesiniotis @shivamerla Do we want to enable DCGM by default? Something to think about given that we may need to integrate NVSentinel into our operand stack. Changing the default later on may not be as easy

karthikvetrivel requested review from cdesiniotis and rahulait as code owners June 23, 2026 15:47

karthikvetrivel requested a review from cdesiniotis June 23, 2026 15:47

karthikvetrivel requested review from rajathagasthya and shivamerla as code owners June 23, 2026 15:47

karthikvetrivel requested review from rahulait and shivamerla June 23, 2026 15:47

karthikvetrivel requested a review from tariq1890 as a code owner June 23, 2026 15:47

karthikvetrivel requested a review from tariq1890 June 23, 2026 15:47

This was referenced Jun 23, 2026

Add GPUClusterConfig CRD and controller for DRA-based stack #2513

Closed

Support NVIDIADriver reconciliation without a ClusterPolicy #2572

Draft

cdesiniotis reviewed Jun 24, 2026

View reviewed changes

Comment thread manifests/state-gfd/0400_resourceclaimtemplate.yaml

karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch from 5691bbc to e5dcecd Compare July 1, 2026 15:24

karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch 4 times, most recently from a4c09b7 to 6b5ac48 Compare July 2, 2026 19:50

karthikvetrivel changed the title ~~Add GPUClusterConfig CRD and controller for DRA-based stack~~ Add GPUCluster CRD and controller for DRA-based stack Jul 2, 2026

karthikvetrivel marked this pull request as draft July 2, 2026 20:19

karthikvetrivel marked this pull request as ready for review July 6, 2026 16:48

cdesiniotis reviewed Jul 7, 2026

View reviewed changes

tariq1890 reviewed Jul 7, 2026

View reviewed changes

karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch 2 times, most recently from 3b7cc55 to 4ad2c9d Compare July 8, 2026 18:48

karthikvetrivel requested review from cdesiniotis and tariq1890 July 8, 2026 18:50

karthikvetrivel added 6 commits July 10, 2026 16:05

Add GPUCluster v1alpha1 API, CRD, and generated client

64b69c9

Signed-off-by: Karthik Vetrivel <kvetrivel@nvidia.com>

Add DRA driver operand and dra-driver-validator init container

dbb9947

Signed-off-by: Karthik Vetrivel <kvetrivel@nvidia.com>

Add GFD, DCGM, DCGM Exporter, and DRA validation operands to GPUCluster

63ab1fa

Signed-off-by: Karthik Vetrivel <kvetrivel@nvidia.com>

Add GPUCluster controller with singleton status and GPU node labeling

16f0d53

Signed-off-by: Karthik Vetrivel <kvetrivel@nvidia.com>

Add per-node stack selection between device-plugin and DRA planes

da52abc

Signed-off-by: Karthik Vetrivel <kvetrivel@nvidia.com>

Add GPUCluster Helm install with ClusterPolicy/NVIDIADriver coexistence

b0803b4

Signed-off-by: Karthik Vetrivel <kvetrivel@nvidia.com>

karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch from 4ad2c9d to b0803b4 Compare July 10, 2026 20:18

cdesiniotis reviewed Jul 10, 2026

View reviewed changes

tariq1890 reviewed Jul 10, 2026

View reviewed changes

	return fmt.Errorf("error creating state manager: %v", err)
	return fmt.Errorf("error creating state manager: %w", err)

Uh oh!

Conversation

karthikvetrivel commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. Overview

2. Testing

3. Next Steps

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coveralls commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tariq1890 Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cdesiniotis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

karthikvetrivel commented Jun 23, 2026 •

edited

Loading

coveralls commented Jul 1, 2026 •

edited

Loading

tariq1890 Jul 7, 2026 •

edited

Loading