Repository navigation
Audit fan-out: label srun output and reassemble chunked records - #114
Merged
Merged
Conversation
srun forwards task output in 1024-byte chunks and interleaves chunks from different tasks. On VP (16 GB300 nodes, Slurm 26.05) a 1120-byte virtio-net record was split around another node's record and both were lost: 14 of 16 host records. Run the fan-out with --label and rebuild each task's lines before parsing, in parse_json_lines and in the HBM and VF checks that parse srun output themselves.
aliencaocao
requested review from
JordanNanos,
Prathmesh234 and
samharshe
as code owners
November 5, 2026 17:59
Member
Author
|
Verified on VP (16× GB300): with this fix the virtio-net/SNAP-4 fan-out returns 16/16 host records; before it, 14/16 (records ~1120 bytes split by srun's 1024-byte chunking). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug: every audit check that collects per-node results through
srunruns one worker per node, and each worker prints one JSON record. srun forwards each task's output in 1024-byte chunks and interleaves chunks from different tasks. A record longer than 1024 bytes can arrive split around another node's record, and then neither record parses.Seen on VP (16 GB300 workers, Slurm 26.05.4): the virtio-net check returned 14 of 16 host records. With the SNAP-4 fields from #111 each record is about 1120 bytes. Raw stdout showed a 2144-byte line (task A's first 1024 bytes followed by task B's whole record) and then A's last 96 bytes on a line of their own. A synthetic 16-task test reproduces it for any line over 1024 bytes and never for lines under it.
Fix:
slurm_python_commandadds--label, so srun delivers each chunk as its own<task>:line.join_srun_labeled_outputstitches each task's lines back together. A full 1024-byte chunk continues on that task's next chunk; a shorter one ends the line. A full chunk that already parses as JSON also ends the line, so a line of exactly 1024 bytes isn't merged into the next one. Output that isn't entirely labeled (k8s exec, test stubs) is returned unchanged.parse_json_linesruns the join first. That covers platform_config, vboost, storage_config, nic-topology and virtio-net. hbm_memory_exposure and vf-interface-exposure parse srun output with their own loops, so each now calls the join directly.Testing:
check_public_datapassed.