Repository navigation
[Bug]: NRI doesn't work trying to check nvidia-smi from the validation pod instead of host #2382
Description
Describe the bug
When enabling NRI via --set cdi.nriPluginEnabled=true the toolkit validation pod fails with nvidia-smi not found in path.
I've looked at the code and it seems it's trying to execute nvidia-smi from inside the validation container as opposed to how driver validation validates from the host system.
See:
gpu-operator/cmd/nvidia-validator/main.go
Line 1132 in e6cd031
This executes nvidia-smi from inside the validation container and it would fail, but for driver validation
see:
gpu-operator/cmd/nvidia-validator/main.go
Line 745 in e6cd031
Fix would be to check nvidia-smi similarly to how driver validation works
nvidia-smi is executed from the host
To Reproduce
Install helm chart with --set cdi.nriPluginEnabled=true
Expected behavior
NRI plugin works
Environment (please provide the following information):
- GPU Operator Version: v26.3.1
- OS: Talos 1.13.0-rc.0
- Kernel Version: 6.18.22-talos
- Container Runtime Version: containerd 2.2.3
- Kubernetes Distro and Version: Talos v1.35.2
Information to attach (optional if deemed irrelevant)
- kubernetes pods status:
kubectl get pods -n OPERATOR_NAMESPACE - kubernetes daemonset status:
kubectl get ds -n OPERATOR_NAMESPACE - If a pod/ds is in an error state or pending state
kubectl describe pod -n OPERATOR_NAMESPACE POD_NAME - If a pod/ds is in an error state or pending state
kubectl logs -n OPERATOR_NAMESPACE POD_NAME --all-containers - Output from running
nvidia-smifrom the driver container:kubectl exec DRIVER_POD_NAME -n OPERATOR_NAMESPACE -c nvidia-driver-ctr -- nvidia-smi - containerd logs
journalctl -u containerd > containerd.log
Collecting full debug bundle (optional):
curl -o must-gather.sh -L https://raw.githubusercontent.com/NVIDIA/gpu-operator/main/hack/must-gather.sh
chmod +x must-gather.sh
./must-gather.sh
NOTE: please refer to the must-gather script for debug data collected.
This bundle can be submitted to us via email: operator_feedback@nvidia.com
Activity
tariq1890 commentedon Apr 20, 2026
Can you share more details? Which daemonset is failing here?
Please share the yaml manifest of the failing daemonset if you can.
frezbo commentedon Apr 20, 2026
Yes,
logs:
Others pods in init status are basically waiting for setup
frezbo commentedon Apr 20, 2026
Yaml dump of validator:
happy to provide further information is needed
tariq1890 commentedon Apr 20, 2026
It looks like you don't deploy the
nvidia-container-toolkit. Is that expected?Please note that the toolkit daemonset pod is what runs the NRI Plugin server and performs the CDI device injection. Without the toolkit pod, this will not work
frezbo commentedon Apr 20, 2026
yes, that's correct, the host os already ships the toolkit components
helm command:
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia helm repo update helm upgrade --wait --install -n gpu-operator gpu-operator nvidia/gpu-operator \ --set driver.enabled=false \ --set toolkit.enabled=false \ --set hostPaths.driverInstallDir=/usr/local \ --set cdi.nriPluginEnabled=true❯ talosctl -n 172.16.33.136 ls /usr/local/bin | grep nvidia 172.16.33.136 nvidia-bug-report.sh 172.16.33.136 nvidia-cdi-hook 172.16.33.136 nvidia-container-runtime 172.16.33.136 nvidia-container-runtime-hook 172.16.33.136 nvidia-container-runtime.cdi 172.16.33.136 nvidia-container-runtime.legacy 172.16.33.136 nvidia-ctk 172.16.33.136 nvidia-ctk-installer 172.16.33.136 nvidia-cuda-mps-control 172.16.33.136 nvidia-cuda-mps-server 172.16.33.136 nvidia-debugdump 172.16.33.136 nvidia-installer 172.16.33.136 nvidia-modprobe 172.16.33.136 nvidia-ngx-updater 172.16.33.136 nvidia-pcc 172.16.33.136 nvidia-persistenced 172.16.33.136 nvidia-persistenced-wrapper 172.16.33.136 nvidia-powerd 172.16.33.136 nvidia-settings 172.16.33.136 nvidia-sleep.sh 172.16.33.136 nvidia-smi 172.16.33.136 nvidia-uninstall 172.16.33.136 nvidia-xconfigfrezbo commentedon Apr 20, 2026
Does this mean the host doesn't need to ship the toolkit ? Just trying to wrap my head around here
is my understanding correct that the toolkit validation with error:
failed to find
nvidia-smisince cdi injection did not happen?tariq1890 commentedon Apr 20, 2026
Well, the host's pre-installed toolkit will not help as it doesn't bring up an NRI Plugin server. To leverage the NRI Plugin feature here, you will need to enable
nvidia-container-toolkitin the ClusterPolicy CR. So, the NRI Plugin is only useable if the toolkit is deployed a daemonset.Yes, If the toolkit-validation fails with this error, It most likely points to the toolkit not being able to inject the gpu driver binaries, shared libraries and gpu device nodes into the container successfully.
frezbo commentedon Apr 21, 2026
Thanks this clears up stuff, I would like to move this issue to a github discussion if possible
Since Talos is an immtuable host OS, the toolkit binaries are brought in as part of OS, so I would like to have a general discussion on how to then enable NRI if that comes as part of toolkit
Since when i enable toolkit, it fails on Talos due to host OS being immutable
I'm not sure why it tries to create containerd config when using NRI since there's no need to inject runtimeclass anymore
frezbo commentedon Apr 21, 2026
From docs https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/cdi.html#about-the-node-resource-interface-nri-plugin