Skip to content

Audit fan-out: label srun output and reassemble chunked records - #114

Merged
aliencaocao merged 1 commit into
masterfrom
fix/srun-fanout-labeled-output
Nov 5, 2026
Merged

aliencaocao merged 1 commit into
masterfrom
fix/srun-fanout-labeled-output

Conversation

@aliencaocao

Copy link
Copy Markdown
Member

Bug: every audit check that collects per-node results through srun runs one worker per node, and each worker prints one JSON record. srun forwards each task's output in 1024-byte chunks and interleaves chunks from different tasks. A record longer than 1024 bytes can arrive split around another node's record, and then neither record parses.

Seen on VP (16 GB300 workers, Slurm 26.05.4): the virtio-net check returned 14 of 16 host records. With the SNAP-4 fields from #111 each record is about 1120 bytes. Raw stdout showed a 2144-byte line (task A's first 1024 bytes followed by task B's whole record) and then A's last 96 bytes on a line of their own. A synthetic 16-task test reproduces it for any line over 1024 bytes and never for lines under it.

Fix:

  • slurm_python_command adds --label, so srun delivers each chunk as its own <task>: line.
  • New join_srun_labeled_output stitches each task's lines back together. A full 1024-byte chunk continues on that task's next chunk; a shorter one ends the line. A full chunk that already parses as JSON also ends the line, so a line of exactly 1024 bytes isn't merged into the next one. Output that isn't entirely labeled (k8s exec, test stubs) is returned unchanged.
  • parse_json_lines runs the join first. That covers platform_config, vboost, storage_config, nic-topology and virtio-net. hbm_memory_exposure and vf-interface-exposure parse srun output with their own loops, so each now calls the join directly.

Testing:

  • New tests reproduce the VP split, a 16-task interleave of 3000-byte records, the exactly-1024-byte boundary, unlabeled passthrough, and labeled noise lines.
  • On VP, live: 3 runs each of the real virtio-net worker and a 3000-byte synthetic worker across 16 nodes. Every run produced 16 of 16 hosts after reassembly.
  • Full suite: 2515 passed. check_public_data passed.

srun forwards task output in 1024-byte chunks and interleaves chunks from
different tasks. On VP (16 GB300 nodes, Slurm 26.05) a 1120-byte
virtio-net record was split around another node's record and both were
lost: 14 of 16 host records. Run the fan-out with --label and rebuild each
task's lines before parsing, in parse_json_lines and in the HBM and VF
checks that parse srun output themselves.
@aliencaocao

Copy link
Copy Markdown
Member Author

Verified on VP (16× GB300): with this fix the virtio-net/SNAP-4 fan-out returns 16/16 host records; before it, 14/16 (records ~1120 bytes split by srun's 1024-byte chunking).

@aliencaocao
aliencaocao merged commit f26c5ce into master Nov 5, 2026
5 checks passed
@aliencaocao
aliencaocao deleted the fix/srun-fanout-labeled-output branch November 5, 2026 20:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant