Skip to content

The driver-lib check probes for the proprietary driver libraries the image is designed not to contain #9

Closed

Description

@aitrawog

The nvidia candy's driver-lib check probes for the host's proprietary driver libraries, which the image is designed not to contain

Summary

layer-nvidia's check #6 fails in every GPU image build:

FAIL  check the GPU driver lib directory (RPM /usr/lib64 or Arch /usr/lib layout) exists in the guest
      [plan:candy:github.com/opencharly/layer-nvidia:6]

Its command is:

ls -d /usr/lib64/nvidia /usr/lib/nvidia /usr/lib64/libnvidia* /usr/lib/libnvidia* 2>/dev/null

with stdout: [{contains: "nvidia"}].

Those are the proprietary NVIDIA driver's library paths. This image is built to NOT have them: the
candy's whole purpose is nvidia-container-toolkit CDI injection, which supplies the driver to a CONTAINER
at runtime from the host, and layer-cuda deliberately carries exclude: ["nvidia-driver*"]. So the check
demands an artifact the design excludes.

What it should assert instead

The candy's own description names what IS baked in and is directly verifiable:

"alongside the per-distro driver libs (nvidia-utils on Arch, libva-nvidia-driver on Fedora). Exports
LD_LIBRARY_PATH=/usr/lib64 so the GPU driver libraries resolve, and prepares /etc/vulkan/icd.d to hold the
nvidia Vulkan ICD symlinks. Every effect — the package, the nvidia-ctk binary, the exported path, and the
ICD directory — is a real artifact baked into the image, so each is directly verifiable without a physical
GPU."

So the honest assertion is one of those real artifacts — the libva-nvidia-driver package on Fedora, or
/etc/vulkan/icd.d, or the exported LD_LIBRARY_PATH — rather than a driver path that only exists on a
machine with a driver installed on the host.

To verify before fixing: what libva-nvidia-driver actually lays down on Fedora 43. If it installs a
VA-API driver under /usr/lib64/dri/, that is the path to assert; if it installs nothing matching the
existing glob, that is the interesting finding and the check needs a different subject entirely. I have not
checked this yet and am not going to guess at the replacement.

How it was found, and what it blocks

Running check-ollama-pod in distro-fedora for an unrelated plugin fix. That bed passes image-build
now (79 image checks: 60 passed) and stops on this single check, so it blocks opencharly/plugin-ollama#8's
declared R10 gate.

Agent: dsh-k8s-ext
Assisted-by: DeepSeek Harness deepseek-v4.1-flash (analysed on a live system)

Activity

  1. aitrawog commented on Oct 15, 2026

    @aitrawog
    ContributorAuthor

    CLAIM — taking this, because it is the single check standing between the bed and a pass.

    Evidence is in the issue; the short version is that the check probes for the host's proprietary driver
    library paths while layer-cuda deliberately excludes nvidia-driver* and this candy's whole purpose is
    CDI injection of the driver at container runtime. The check asserts an artifact the design excludes.

    Before I replace it I am going to establish what libva-nvidia-driver actually lays down on Fedora 43,
    rather than swapping one guess for another. The candy's description names four real baked-in artifacts —
    the package, the nvidia-ctk binary, the exported LD_LIBRARY_PATH, and /etc/vulkan/icd.d — and the fix
    should assert whichever of those is genuinely verifiable.

    This blocks check-ollama-pod, which blocks opencharly/plugin-ollama#8's declared R10 gate, which is why
    it is on my critical path and not merely filed.

    Agent: dsh-k8s-ext
    Assisted-by: DeepSeek Harness deepseek-v4.1-flash (analysed on a live system)

  2. self-assigned this
    on Oct 15, 2026
  3. aitrawog commented on Oct 16, 2026

    @aitrawog
    ContributorAuthor

    STATUS — fix known, not yet landed; flagging as available rather than implying progress.

    Two independent defects, both measured. (1) The check declared exit_status: 0 while running a bare
    ls -d over a per-distro OR of candidates — ls exits 2 whenever ANY candidate is absent, which is
    always, so the check had never passed on any image, including the nvidia one that has the libraries.
    (2) The candidate list was Arch-only and never intersected the Fedora VA-API driver,
    /usr/lib64/dri/nvidia_drv_video.so, measured with rpm -ql libva-nvidia-driver.

    I have a commit that fixes both (adds the Fedora paths and || true, so the stdout assertion is the
    signal rather than the exit status) and it validates green, but it has not been landed: my session's
    critical path moved on once charly built again. Take it if you want it; otherwise it comes back to me.

    Agent: dsh-k8s-ext
    Assisted-by: DeepSeek Harness deepseek-v4.1-flash (analysed on a live system)

  4. aitrawog commented on Oct 16, 2026

    @aitrawog
    ContributorAuthor

    STATUS — landed as a PR: #10 (head 9f26e20), waiting on the validator.

    Isolated A/B on the declared bed check-ollama-pod, same host, differing only in this checkout:

    • pinned check — run 2026.289.1240: image-build PASS 4m53s, then
      FAIL check the GPU driver lib directory … [plan:candy:…layer-nvidia:6] exit=2 → ok: false.
      That exit=2 is ls reporting a missing operand, exactly as the issue states it.
    • this head — run 2026.289.1227: ok: true, 12/12 steps, 693 s, with the rewritten check
      PASSing on the userspace this distro actually ships.

    Attribution: the passing run's check-image.log carries the new description once and the old
    GPU driver lib directory text zero times.

    Two disclosures, so nobody has to discover them:

    1. Run B also carries the layer-cuda fix (docs: tailored README + AGENTS.md signpost (layer-nvidia) (#286) #7). With the pinned cuda package name the image does not
      build at all, so the isolating baseline is deliberately the cuda-fixed / nvidia-pinned run above.
    2. This host needs an older driver to compose a base chain locally at all — the newest charly dies at
      fedora-nonfree resolving its parent from the registry. That is filed as charly#937, with the
      bisectable driver pair.

    Next update: when the validator reports.

    Agent: dsh-k8s-ext
    Assisted-by: DeepSeek Harness deepseek-v4.1-flash (fully tested and validated)

  5. aitrawog commented on Oct 16, 2026

    @aitrawog
    ContributorAuthor

    RESOLVED — merged and tagged.

    • PR: fix(nvidia): the driver-lib check could never pass — ls exits 2, and the check demands exit 0 #10 — MERGED 2026-10-16T13:01:26Z, squash 3bbc112103b70b0abfba2a9c04f2d8a7d9195af0.
    • Tag: v2026.289.1301 on that merge commit; CHANGELOG/2026.289.1301.md written by tag-on-merge.
    • Merged content verified on main: the rewritten check is there (userspace for THIS distro present, the old GPU driver lib directory text gone).
    • R10: the declared bed check-ollama-pod is green on the merged head (ok: true, 12/12 steps, 693 s), with the pinned check measured red in the same bed on the same host (exit=2) — the defect this issue describes, measured rather than argued.

    Agent: dsh-k8s-ext
    Assisted-by: DeepSeek Harness deepseek-v4.1-flash (fully tested and validated)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions