[aliencaocao]

Follow-up to #109. #109 publishes snap4Container (NVIDIA bulletin 5744) with one fix per release train, but nothing in the audit reads a SNAP-4 version yet. This PR adds the read and the grade.

Collection

SNAP-4 runs as a container on the BlueField-3 Arm cores, so the virtio-net check reads it on the same DPU path it already uses for the controller:

• If the check is running on the DPU, it reads locally. Otherwise it sends one RShim SSH command, which costs at most one SSH timeout. The SSH settings are the same as for virtnet: BatchMode=yes, so it never prompts, and nothing is written to known_hosts.
• It lists the running containers with crictl ps and crictl images, retrying under sudo -n, and reads the version from the doca_snap:<x.y.z>-doca… image tag. If no container is running, the version comes from the kubelet manifest in /etc/kubelet.d, since kubelet starts that image on restart.
• The read only lists. It never pulls, starts, stops or execs into a container.
• It is skipped for hosts with no BlueField-3 and for cards in NIC mode, because the Arm cores are off.
• Each host gets one of these states: version, not_installed (the DPU answered with no SNAP-4), not_running (NIC mode), not_applicable, incomplete or unknown.
• The fleet rollup keeps every reading, and a coverage gap withdraws a clean rollup the same way it does for the controller.
• The SRUN budget comment is updated: 185 s worst case, 245 s with dispatch, against 300 s.

Grade (securityVersions.snap4Container)

Each reading is graded against its own train:

• A version on a listed train needs that train's fix: 4.9.0 fails against 4.9.1, and 4.5.5 passes.
• A version between or below the trains has no fix of its own, so it must move to the next fix above it: 4.7.x needs 4.9.1, and 4.3.x needs 4.5.5.
• A version newer than every fix passes, e.g. 4.12.0.

The worst reading decides the cluster result:

• A below-minimum reading fails even when coverage is incomplete.
• Passing readings clear the cluster only when every DPU was read or cannot run SNAP-4 right now.
• With no readings, the result is not_applicable for no BlueField-3, no SNAP-4 deployed, or NIC mode. In DPU mode the result is unknown, which asks the provider to attest.
• Until the refreshed table includes snap4Container, a reading grades unknown, while absence still grades not_applicable.

Surfaces

• A new snap4-container check in cmax audit security. The bulletin link and CVEs come from the minimum table.
• A securityVersions.snap4Container.status criterion in the full audit, reproduction commands, the website anchor and the README lists.
• Slurm and k8s findings rules, CLI-only (no criteria id).
• The k8s driver-pod path does not reach the DPU, so a DPU-mode card is unknown there.

Tests

tests/audit/test_snap4_container.py (28 tests) covers:

• Listing parsing: image-ID to tag mapping, manifest fallback, an untagged container, and missing crictl.
• Host reads over RShim and locally, including JSON output with no trailing newline.
• NIC mode and no-BlueField hosts never dialling the DPU.
• An unreachable DPU, and the rollup and coverage gap.
• The per-train grading cases.

Existing contract tests now also cover --snap4-json. The full suite passes on 3.14 and the touched files pass on 3.10.

[aliencaocao]

Real-cluster testing (wheel built from 6e4673d)

Hydra

• cmax audit security -y and cmax audit -y on gpu-001 both ran with no tracebacks.
• The new row shows as SKIPPED Security Versions / SNAP-4 Container, with the verdict not_applicable: "SNAP-4 does not apply: scale-out checks do not apply to standalone".
• virtio-net-check.py --collect-host on gpu-002, gpu-004 and gpu-007 gives snap4.state = not_applicable, since there is no BlueField-3. The SSH call was skipped.
• I ran the DPU-side script through a real shell. It printed @@SNAP4-NO-CRICTL@@, which parses to listed: false.

VP

• All 18 compute nodes are taken by job 3841 for up to 2 days, so I could not get a Slurm allocation. There is no fleet run.
• login-0 does have BlueField-3 hardware (15b3:a2dc). There, --collect-host gives snap4.state = incomplete ("BlueField-3 is present but its mode could not be read"), because mlxconfig is not installed. The evaluator turns that into snap4Container.status = unknown.

Not tested on hardware
Neither cluster has a BlueField-3 in DPU mode that is reachable over RShim with SNAP-4 deployed. So the actual crictl/manifest read is covered only by the stubbed tests, which is why this PR stays a draft.

[aliencaocao]

CoreWeave SUNK test (crwv-1, crwv-3)

Both clusters run Slurm-on-Kubernetes. slurmd pods have no lspci, mlxconfig, mstconfig, crictl, or RShim (/dev/rshim* and tmfifo_net0 are absent), and sudo -n works. sysfs shows 8× 15b3:101e, 8× 15b3:1021, and 1× 15b3:a2dc (BlueField-3).

Cluster	Nodes	Before fixes	After (5dfa5f5 wheel)
crwv-1 (Ubuntu 24.04)	h100-206-[165,137]	snap4Container unknown, "lspci is not installed"	BF3 found via sysfs; incomplete/unknown, "mode could not be read (mlxconfig is not installed)"
crwv-3 (Ubuntu 22.04)	slurm-h100-206-077	same	same

crwv-2 stayed queued behind other jobs, so I cancelled it.

The run found two problems, now fixed on this branch:

• bb4f7a6: when lspci is missing, the virtio-net check now reads /sys/bus/pci/devices. Hosts without a BlueField-3 now grade not_applicable instead of unknown. The source is recorded as pciSource. An unreadable sysfs entry keeps the scan incomplete.
• e3913ac and 5dfa5f5: a missing mlxconfig was reported as "the sudo -n retry did not answer". It now says the tool is missing and that NVIDIA MFT provides it.

Both cmax audit security -y and cmax audit -y exited with no tracebacks. The live and shipped minimum tables both carry the 4.5.5 and 4.9.1 floors. Full suite: 2542 passed. check_public_data passed.

The actual SNAP-4 read path (mode read, then the container listing on the DPU) still has not run on real hardware: no cluster tested so far has mlxconfig plus a BlueField-3 in DPU mode. This stays a draft until that runs, on VP.

[aliencaocao]

VP test (16× GB300 worker-[1-16], Slurm 26.05.4), wheel = this branch + #114:

• cmax audit security -y ran on all 16 hosts; virtio/SNAP-4 workers returned 16/16 host records (first run without Audit fan-out: label srun output and reassemble chunked records #114 got 14/16 due to srun 1024-byte output interleaving).
• snap4Container: incomplete / unknown — "BlueField-3 is present but its mode could not be read (mlxconfig is not installed)". VP workers have 2× BF3 but no MFT/mlxconfig, rshim or tmfifo, so the DPU-mode read path is still unexercised here.
• Other failures are unrelated (guest-kernel Ubuntu CVE status, NVIDIA driver floor).

Staying draft.

[aliencaocao]

VP re-test with NVIDIA MFT 4.33.0-169 (arm64, userspace mft deb only, no kernel module) installed on worker-[1-16]:

• First run with mlxconfig present still read the mode on only 2/16 hosts: the full mlxconfig -e -d <dev> q dump takes ~22 s idle (~60 s with 3 in flight) on these BF3 cards, past the 15 s tool timeout. Fixed in b7b3f80: query only INTERNAL_CPU_MODEL INTERNAL_CPU_OFFLOAD_ENGINE INTERNAL_CPU_RSHIM (~1.7 s); fall back to the full dump (60 s budget) only if firmware answers Unknown Parameter.
• Re-run (this branch + Audit fan-out: label srun output and reassemble chunked records #114), all 16 hosts read via sudo mlxconfig:
  • snap4Container: not_applicable / not_running — BF3 in NIC mode (INTERNAL_CPU_OFFLOAD_ENGINE=1).
  • virtioNetBluefield: pass, exposure none, coverageComplete true.
  • dpuHostIsolation: fail on all hosts — INTERNAL_CPU_RSHIM=0 (RShim unrestricted), no /dev/rshim0 or tmfifo_net0.
• The DPU-mode SNAP-4 version path is still unexercised (no DPU-mode BF3 available). Staying draft.

[aliencaocao]

Updated the NIC-mode RShim remediation to name all three provider-side options:

1. If tenants keep host root on the BlueField, re-image the card from a known-good BFB on every re-provision, before a new tenant gets it.
2. Block tenant sudo over the BlueField while it is in NIC mode (so only the provider can reach it); mlxconfig -d <device> s INTERNAL_CPU_RSHIM=1 plus a power cycle is real isolation only here.
3. Run it in DPU mode with mlxprivhost restricted, which NVIDIA supports there and which removes the host's reach.

NVIDIA does not support mlxprivhost zero-trust in NIC mode, so option 3 requires a mode change.