You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: NVIDIADriver spec.labels can override the driver DaemonSet's app and component labels #3059
Describe the bug manifests/state-driver/0500_daemonset.yaml (lines 23–24 and 64–65) appends .Driver.Spec.Labels verbatim after the operator's own app and app.kubernetes.io/component labels, on both the DaemonSet metadata and the pod template.
internal/render decodes the rendered manifest with k8s.io/apimachinery/pkg/util/yaml.NewYAMLOrJSONDecoder, which uses non-strict yaml.YAMLToJSON (vendor/k8s.io/apimachinery/pkg/util/yaml/decoder.go:100). Duplicate mapping keys are resolved last-wins without an error. I confirmed this with the vendored sigs.k8s.io/yaml: a mapping with app twice returns the second value from YAMLToJSON, while YAMLToJSONStrict returns key "app" already set in map.
Consequences, depending on the key a user sets in spec.labels:
app: the pod template label no longer matches the immutable selector app: {{ .Driver.AppName }}, so the DaemonSet create/update is rejected and the driver is not deployed on that node pool.
app.kubernetes.io/component: the DaemonSet and pods lose component=nvidia-driver. That label is used by the upgrade controller (BuildState with app.kubernetes.io/component=nvidia-driver), orphaned-pod detection (controllers/nodelabeling_controller.go:593) and the validator's isDriverManagedByOperator (cmd/nvidia-validator/main.go:713). Auto-upgrade stops seeing the driver and the validator treats the node as host-driver.
The ClusterPolicy path already protects these keys in applyCommonDaemonsetLabel (controllers/object_controls.go:809). Draft PR #2973 adds equivalent filtering for GPUCluster only.
To Reproduce
Found by code review; not yet reproduced on a cluster. Expected repro:
Create an NVIDIADriver with spec.labels: {app.kubernetes.io/component: custom} and upgradePolicy.autoUpgrade: true.
Change the driver version.
Observe that the driver DaemonSet has app.kubernetes.io/component: custom and no node receives an upgrade state label.
Same with spec.labels: {app: custom} → DaemonSet create fails with a selector/template mismatch.
Expected behavior
Reserved keys in spec.labels are ignored (with a warning/condition) or rejected at admission, as on the ClusterPolicy path.
Suggested fix
Filter reserved keys before rendering (shared helper with the ClusterPolicy path), and/or add CEL validation on NVIDIADriverSpec.Labels rejecting app and app.kubernetes.io/component. Optionally switch the renderer to strict YAML so duplicate keys fail loudly.
Environment (please provide the following information):
@CloneOfAlex are you planning to send a fix for this one? If not, I'd like to take it. @ajavanma, your draft #2973 leaves NVIDIADriver rendering unchanged, so I'd keep this to the NVIDIADriver path and reuse your selector-key helper once it lands, unless you'd rather extend #2973 to cover it.
Thanks @vjsairam. I'd prefer to keep #2973 focused on GPUCluster, so a separate NVIDIADriver fix sounds good from my side. Reusing the helper once it lands makes sense, provided @CloneOfAlex isn't already working on this.
Activity
vjsairam commentedon Oct 17, 2026
@CloneOfAlex are you planning to send a fix for this one? If not, I'd like to take it. @ajavanma, your draft #2973 leaves NVIDIADriver rendering unchanged, so I'd keep this to the NVIDIADriver path and reuse your selector-key helper once it lands, unless you'd rather extend #2973 to cover it.
ajavanma commentedon Oct 17, 2026
Thanks @vjsairam. I'd prefer to keep #2973 focused on GPUCluster, so a separate NVIDIADriver fix sounds good from my side. Reusing the helper once it lands makes sense, provided @CloneOfAlex isn't already working on this.