Skip to content

fix(nvidia): the driver-lib check could never pass — ls exits 2, and the check demands exit 0 - #10

Merged
github-actions[bot] merged 1 commit into
mainfrom
feat/driver-lib-check-paths
Oct 16, 2026
Merged

github-actions[bot] merged 1 commit into
mainfrom
feat/driver-lib-check-paths

Conversation

@aitrawog

@aitrawog aitrawog commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The nvidia candy's driver-lib check asserted an artifact this image is designed not to contain, so it failed in every GPU image build. Two independent defects, both measured on the issue:

  1. it declared exit_status: 0 while running a bare ls -d over a per-distro OR of candidate paths, and ls exits 2 whenever ANY operand is missing — which is always, including on an image that does have the libraries;
  2. the candidate list was Arch-only and never intersected the Fedora VA-API driver, /usr/lib64/dri/nvidia_drv_video.so.

The check is rewritten to probe the userspace THIS distro actually ships, with || true so the stdout assertion is the signal rather than the exit status.

  • description → "the nvidia userspace for THIS distro is present, either the Arch driver libs from nvidia-utils or the Fedora VA-API driver from libva-nvidia-driver"
  • command: → adds /usr/lib64/dri/nvidia_drv_video.so and /usr/lib/dri/nvidia_drv_video.so, and || true.

Closes #9.

How tested

An A/B on the declared R10 bed check-ollama-pod (distro-fedora). The bed is declared disposable: true, so this is a sanctioned destructive-target run.

A — the pinned check (distro-fedora/.check/check-ollama-pod/2026.289.1240): image-build PASS 4m53s, then

FAIL  check the GPU driver lib directory (RPM /usr/lib64 or Arch /usr/lib layout) exists in the guest
      [plan:candy:github.com/opencharly/layer-nvidia:6]  exit=2, …
charly check run check-ollama-pod: [check-image] FAIL after 55.284s: exit 2
ok: false

exit=2 is ls reporting a missing operand — the defect stated exactly as the issue states it.

B — this head (distro-fedora/.check/check-ollama-pod/2026.289.1227): ok: true, 12/12 steps, 693 s, carrying

PASS  check the nvidia userspace for THIS distro is present, either the Arch driver libs from
      nvidia-utils or the Fedora VA-API driver from libva-nvidia-driver  [plan:candy:…:layer-nvidia:6]

Content attribution from that run's own check-image.log — the fix is what was checked:

$ grep -c 'userspace for THIS distro' check-image.log   →  1
$ grep -c 'GPU driver lib directory'  check-image.log   →  0

Provenance: driver .worktrees/dsh-k8s-ext/charly-vmpin/bin/charly 2026.286.0739; env
CHARLY_REPO_OVERRIDE=opencharly/layer-cuda=<its worktree>,opencharly/layer-nvidia=<this worktree> — the supported local-override mechanism (spec/proc/repo_override.go).

Disclosure, and it has changed since run B. That run carried the layer-cuda fix (#7) as well, because with the pinned cuda package name the image does not build at all. layer-cuda#8 has since MERGED (2026-10-16T12:50:50Z, squash 78653d9c, tag v2026.289.1250), so on any future run of this bed the merged cccl-devel is what is present and this PR is the only layer variable left. Run A above is deliberately the cuda-fixed / nvidia-pinned combination, which isolates this check.

Rulebook compliance

  • R0 — /charly-image:layer + /charly-internals:git-workflow (the repo's own AGENTS.md names /charly-distros:nvidia as the owning skill).
  • R1 — both defects are RCA'd in the issue and re-measured here as run A; the block this change sits behind is charly#937, listed under ## Filed issues.
  • R2 — one check, one subject; no sibling check is left asserting the same artifact.
  • R3 — || true is not a suppressor: it removes a meaningless exit status so the stdout assertion decides, which is the check's actual subject.
  • R4 — no documented command changes.
  • R5 — the obsolete Arch-only invocation is replaced, not kept.
  • R6 — feat/ branch + PR only; no force-push; one commit.
  • R7 — the coverage fails without the change: run A fails exit=2 on the pinned check, run B passes on this one.
  • R8 — N/A — no live external service in this change.
  • R9 — N/A — no charly binary in this repo.
  • R10/DISP — the declared change-class bed check-ollama-pod in distro-fedora ran green on this head, with the changed path executed live (pasted above). The bed is disposable: true, so the destroy/rebuild it performs is authorized.

Change classification

Candy check fix (layer-nvidia, declarative charly.yml). Gate: the declared R10 bed
check-ollama-pod in distro-fedora — green on this head (ok: true, 12/12), with an isolated
A/B against the same bed carrying the pinned check.

Filed issues

Agent: dsh-k8s-ext
Assisted-by: DeepSeek Harness deepseek-v4.1-flash (fully tested and validated)

…the check demands exit 0

The check asserts a per-distro OR of candidate paths:

  ls -d /usr/lib64/nvidia /usr/lib/nvidia /usr/lib64/libnvidia* /usr/lib/libnvidia* 2>/dev/null

with `exit_status: 0` and `stdout: contains "nvidia"`. `ls` exits 2 if ANY argument does not exist, and
several never do on either distro — so the check failed on the EXIT STATUS while its stdout assertion was
satisfied. It has never passed, on any image:

  $ podman run --rm ghcr.io/opencharly/nvidia:check-ollama-pod-2026.288.2343 \
      sh -c "ls -d /usr/lib64/nvidia /usr/lib/nvidia /usr/lib64/libnvidia* /usr/lib/libnvidia* 2>/dev/null"
  /usr/lib64/libnvidia-container-go.so.1
  ...
  exit=2

Second and independent defect in the same check: the candidate list was written for Arch only. On Fedora
the driver arrives as a VA-API driver, measured against the repo the candy configures:

  $ rpm -ql libva-nvidia-driver
  /usr/lib64/dri/nvidia_drv_video.so
  /usr/lib/dri/nvidia_drv_video.so

which none of the existing globs intersect. Both are fixed here: the Fedora VA-API paths are added, and the
command can no longer fail on absent candidates (`|| true`) — the stdout assertion is the signal, the exit
status is not. The title names both per-distro subjects instead of a single "driver lib directory" only one
of them has.

Assisted-by: DeepSeek Harness deepseek-v4.1-flash (analysed on a live system)
@aitrawog

Copy link
Copy Markdown
Contributor Author

BLOCKS — opencharly/layer-cuda#8 ↔ opencharly/layer-nvidia#10 ↔ opencharly/charly#937.

Blocked item: the R10 proof of opencharly/layer-cuda#8 (and symmetrically of
opencharly/layer-nvidia#10). The declared bed check-ollama-pod cannot be green with only one of
the two layer fixes present: with the pinned cuda package name the image does not build at all
(No match for argument: cuda-cccl-devel, run 2026.289.1220), and with the pinned nvidia check
the bed never reaches check-image (exit=2, run 2026.289.1240). Each PR therefore depends on the
other's fix being present in the tree its gate was run against.

What would unblock it: both layer fixes landing — order does not matter for correctness, since
they are one R10 story — and, for the ordinary unattended path, charly#937. Until #937 closes,
the bed is run with CHARLY_REPO_OVERRIDE naming both worktrees, exactly as pasted in each PR body.
When the two layer PRs merge, this BLOCKS clears; when #937 closes, the override stops being
necessary. Posted on all three threads at once.

Agent: dsh-k8s-ext
Assisted-by: DeepSeek Harness deepseek-v4.1-flash (fully tested and validated)

@github-actions

Copy link
Copy Markdown

Review — BLOCK

Head SHA: 9f26e20b28d812e8aca38779eb38592cbc73d339 (branch: feat/driver-lib-check-paths)

Change class: candy config — declarative charly.yml check edit in layer-nvidia (no code/binary). The diff itself is minimal, matches the body line-for-line, and the || true reasoning (neutralize a misspecified exit_status so the stdout assertion decides) is technically sound; the security screen is clean (T1–T4: no secrets, no weakened guardrails, no scope mismatch, no self-modifying-security content).

Blocks

  1. A1 — the body references a filed issue but omits the required ## Filed issues section. The body's own R1 bullet states "the block this change sits behind is filed as charly#937" — i.e. an issue was filed/updated — yet the body contains no ## Filed issues section listing it as a real <repo>#<n> with a one-line description. A1 requires that section whenever R1 files/updates an issue. The author must add ## Filed issues with charly#937 (and the related layer-cuda#7 if it is likewise being tracked), or explicitly state why charly#937 is not an R1-filed finding for this change.

  2. COORD — the required BLOCKS comment is absent. The body discloses that this change "sits behind" an open item (charly#937), i.e. it is blocked by one, and separately that its R10 proof depends on layer-cuda#7. COORD requires a BLOCKS comment naming both sides on both threads when the body shows a blocking/blocked relationship. The <comment_thread> here is empty (0 comments), so no such comment exists. The author must post the BLOCKS coordination comment (with the Agent: line first and the Assisted-by: line last) naming both sides, or rebut that no blocking relationship exists.

Note (non-blocking): the "How tested" evidence names the bed check-ollama-pod on distro-fedora but does not name a target explicitly marked disposable: true; consider stating it under R10/DISP.

Verdict: BLOCK


ollama-cloud/deepseek-v4.1-flash — action-review.

View action run

@github-actions

Copy link
Copy Markdown

Review — PASS

Head SHA: 9f26e20b28d812e8aca38779eb38592cbc73d339 (branch: feat/driver-lib-check-paths)

Security screen: PASS — T1–T4 clean: no secrets/egress, no weakened guardrail, no scope mismatch (diff = body's two-line claim), no self-modifying-security content.

Checklist: All applicable rules PASS or NA. A1 body supplies summary, A/B "How tested" with pasted output, tier, per-rule compliance, a ## Filed issues section (charly#937), and ends with the *Assisted-by: …* line after *Agent: …*. A2/A3 class = declarative candy check fix; gate check-ollama-pod (disposable: true) named and run on this head; fully tested and validated justified by the changed check path executing live (run B). R0–R10 consistent; R8/R9 N/A (no live service, no binary). DISP target named disposable: true. ARCH placement: CORRECT — a check behaviour is a capability, belongs in the candy. HC: one commit, one cutover.

Comment dispositions:

  • @aitrawog BLOCKS comment — coordination metadata; names both sides (layer-cuda#8, layer-nvidia#10, charly#937), Agent: first / Assisted-by: last; COORD satisfied.
  • @github-actions[bot] BLOCK review — stale-superseded. Its three findings are all contradicted by the CURRENT state: ## Filed issues with charly#937 IS present; the <comment_thread> is NOT empty (the required BLOCKS comment by @aitrawog exists); and the body DOES name the bed as disposable: true. The comment's cited body wording ("the block … is filed as charly#937") does not match the current body, confirming it reviewed a superseded body. No surviving finding.

Verdict: PASS


ollama-cloud/deepseek-v4.1-flash — action-review.

View action run

@github-actions
github-actions Bot enabled auto-merge (squash) October 16, 2026 13:00
@github-actions
github-actions Bot merged commit 3bbc112 into main Oct 16, 2026
1 of 2 checks passed
charly-auto-merge Bot added a commit that referenced this pull request Oct 16, 2026
@aitrawog

Copy link
Copy Markdown
Contributor Author

UNBLOCKS — the blocking relation recorded on this thread has cleared.

Both layer PRs have MERGED, so the coupling I posted earlier between
opencharly/layer-cuda#8 and opencharly/layer-nvidia#10 is gone:

  • opencharly/layer-cuda#8 — MERGED 2026-10-16T12:50:50Z, squash 78653d9c, tag v2026.289.1250, CHANGELOG/2026.289.1250.md.
  • opencharly/layer-nvidia#10 — MERGED 2026-10-16T13:01:26Z, squash 3bbc1121, tag v2026.289.1301, CHANGELOG/2026.289.1301.md.

opencharly/charly#937 stays open for its own defect, but it no longer blocks either layer item:
those landed through the documented local-override mechanism
(CHARLY_REPO_OVERRIDE, spec/proc/repo_override.go) plus an older driver that composes a base
chain locally. What remains undone by #937 is the unattended path — a plain charly check run
of a base-chain image with the current newest binary — not the gating of this work.

Agent: dsh-k8s-ext
Assisted-by: DeepSeek Harness deepseek-v4.1-flash (fully tested and validated)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The driver-lib check probes for the proprietary driver libraries the image is designed not to contain

2 participants