Skip to content

[Bug]: NVIDIADriver spec.labels can override the driver DaemonSet's app and component labels #3059

Open

Description

@CloneOfAlex

Describe the bug
manifests/state-driver/0500_daemonset.yaml (lines 23–24 and 64–65) appends .Driver.Spec.Labels verbatim after the operator's own app and app.kubernetes.io/component labels, on both the DaemonSet metadata and the pod template.

internal/render decodes the rendered manifest with k8s.io/apimachinery/pkg/util/yaml.NewYAMLOrJSONDecoder, which uses non-strict yaml.YAMLToJSON (vendor/k8s.io/apimachinery/pkg/util/yaml/decoder.go:100). Duplicate mapping keys are resolved last-wins without an error. I confirmed this with the vendored sigs.k8s.io/yaml: a mapping with app twice returns the second value from YAMLToJSON, while YAMLToJSONStrict returns key "app" already set in map.

Consequences, depending on the key a user sets in spec.labels:

  • app: the pod template label no longer matches the immutable selector app: {{ .Driver.AppName }}, so the DaemonSet create/update is rejected and the driver is not deployed on that node pool.
  • app.kubernetes.io/component: the DaemonSet and pods lose component=nvidia-driver. That label is used by the upgrade controller (BuildState with app.kubernetes.io/component=nvidia-driver), orphaned-pod detection (controllers/nodelabeling_controller.go:593) and the validator's isDriverManagedByOperator (cmd/nvidia-validator/main.go:713). Auto-upgrade stops seeing the driver and the validator treats the node as host-driver.

The ClusterPolicy path already protects these keys in applyCommonDaemonsetLabel (controllers/object_controls.go:809). Draft PR #2973 adds equivalent filtering for GPUCluster only.

To Reproduce
Found by code review; not yet reproduced on a cluster. Expected repro:

  1. Create an NVIDIADriver with spec.labels: {app.kubernetes.io/component: custom} and upgradePolicy.autoUpgrade: true.
  2. Change the driver version.
  3. Observe that the driver DaemonSet has app.kubernetes.io/component: custom and no node receives an upgrade state label.

Same with spec.labels: {app: custom} → DaemonSet create fails with a selector/template mismatch.

Expected behavior
Reserved keys in spec.labels are ignored (with a warning/condition) or rejected at admission, as on the ClusterPolicy path.

Suggested fix
Filter reserved keys before rendering (shared helper with the ClusterPolicy path), and/or add CEL validation on NVIDIADriverSpec.Labels rejecting app and app.kubernetes.io/component. Optionally switch the renderer to strict YAML so duplicate keys fail loudly.

Environment (please provide the following information):

  • GPU Operator Version: main @ f806d6b
  • Found by code inspection; affects any platform using the NVIDIADriver CRD.

Activity

  1. added
    bugIssue/PR to expose/discuss/fix a bug
    needs-triageissue or PR has not been assigned a priority-px label
    on Oct 13, 2026
  2. vjsairam commented on Oct 17, 2026

    @vjsairam

    @CloneOfAlex are you planning to send a fix for this one? If not, I'd like to take it. @ajavanma, your draft #2973 leaves NVIDIADriver rendering unchanged, so I'd keep this to the NVIDIADriver path and reuse your selector-key helper once it lands, unless you'd rather extend #2973 to cover it.

  3. ajavanma commented on Oct 17, 2026

    @ajavanma
    Contributor

    Thanks @vjsairam. I'd prefer to keep #2973 focused on GPUCluster, so a separate NVIDIADriver fix sounds good from my side. Reusing the helper once it lands makes sense, provided @CloneOfAlex isn't already working on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugIssue/PR to expose/discuss/fix a bugneeds-triageissue or PR has not been assigned a priority-px label

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions