Add DeepSeek-V4.1-Flash four-Spark cycle profile (Engram tables on NVMe, stock vLLM dsv41-feat image, DSpark k=5)

[sethforprivacy]

Summary

Adds a four-Spark cycle profile for deepseek-ai/DeepSeek-V4.1-Flash (stock checkpoint, revision dba1be0a…): TP4 over SparkRing's patched NCCL on a switchless 0-1-2-3-0 cycle, with the model's two Engram lookup tables left on each rank's NVMe so the 475 GiB checkpoint fits four GB10s (78.8 GiB per rank text-only, 81.6 GiB with the DSpark draft layers and vision encoder). Tracks #260.

The serving engine is a self-built stock upstream vLLM image (dsv41-feat @ e47aa780 on the 8a728663 nightly, FlashInfer 0.7.0rc1, _C_stable_libtorch rebuilt for GB10) with seven bind-mounted patches from tonyd2wild/Kai (MIT, md5-pinned): Engram-on-disk with the rank-offset fix and graph-safe prestaging, SM12x page sizes, and the SM12x decode top-k. The gb10-vllm-serving lineage cannot serve this model (no deepseek_v41, no CSA2/Engram/mHC/block-5 DSpark kernels in B12X). Status is implemented, not qualified: there is no public image digest to replay.

Tracking issue: #260 (WIP notes and the interim results comment live there).

What is added

• recipes/deepseek-v41-flash-cycle.json — machine-readable contract (sparkring-recipe/v1).
• scripts/config/deepseek-v41-flash-cycle.env.example + scripts/deepseek_v41_cycle_serve.sh — env-file-driven per-rank launcher; --check is fully offline and validates the model directory, patch md5s, NCCL library and cycle transport values, then prints the exact docker run.
• scripts/test_deepseek_v41_cycle_launcher.py — offline contracts (placeholders rejected, rendered command matches the recipe, worker/eager/text-only variants, fail-closed on contract violations, recipe ↔ receipt ↔ patch md5 agreement).
• runtime/deepseek-v41-gb10/ — build-image.sh (pinned sources; cgroup-bounded so the FlashInfer layers can be built on a node that is still serving), image-receipt.json, the seven patches with mounts.txt and MD5SUMS, and the two FlashInfer prewarm/verify tools.
• docs/DEEPSEEK_V41_FLASH_QUICKSTART.md, docs/profiles/DEEPSEEK_V41_FLASH.md.
• performance/records/deepseek-v41-flash/cycle-tp4-dspark5-graphs-20260910.{md,json} — live benchmark, needle, vision/tools, soak and fabric-probe results.
• Index rows in README.md, docs/profiles/README.md, recipes/README.md; a THIRD_PARTY_NOTICES section; .ruff.toml excludes the vendored, md5-pinned files.

Measured (one private cycle, 2026-09-10/11, all ranks rebooted before every boot)

Benchmark tables from the first serving boot — 300K context, 8 sequences, 8,192 batched tokens, gmu 0.80, block 128, DSpark k=5 (probabilistic draft), FULL_AND_PIECEWISE graphs, tools + vision on, temperature 0, thinking off, fixed eight-category prompt set (counting excluded from means):

C	per-stream decode mean	aggregate	mean TTFT
1	56.2 tok/s	49.8	0.32 s
4	33.8	115.8	0.39
6	31.4	159.9	0.48

C1 per stream: code 77.3, math 76.9, tables 76.4, reasoning 60.0, JSON 58.9, summary 36.8, prose 33.9, narrative 29.9. Cold prefill 1,376 → 1,574 tok/s from 3K to 93K prompt tokens. Needle at 131K and 262K pass. Vision + tool calling 7/7. KV pool 1,171,588 tokens (3.91× 300K); 15–16 GiB MemAvailable per rank while serving. 20-minute c=8 soak: 648 requests, 0 failures, 0 hangs, 95 tok/s median, no drift. Fabric probe: 88 decode-shaped all-reduces per step cost ~5 ms on the cycle (60 KB p50 64 µs; 1 MB p50 185 µs over the two rails).

Lever campaign (one variable per rebooted boot, compact probe set, ≈ ±5 % run-to-run band): NCCL 8 channels neutral; Engram 64 reader threads slightly positive and free; max-num-seqs 16 neutral at C≤8 and 285 tok/s aggregate at C16 (admission option); max-num-batched-tokens 16384 did not boot at 0.80; greedy draft neutral to +5 %; a 430,080-token limit at gmu 0.85 with VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 gave KV 3.09M tokens and a 400K needle pass (397,753 tokens, TTFT 289 s, 1,378 tok/s prefill) at unchanged speed but only 7–9 GiB MemAvailable. The recipe records the combination at gmu 0.83: KV 2,182,642 tokens (5.07× 430K), 13–15 GiB MemAvailable per rank. No lever moves decode beyond noise; the step is set by the model's ~88 all-reduces, which the fabric probes show already cost ~5 ms of a ~57 ms step.

Six-hour c=8 soak on the recorded profile (temperature 1.0, top-p 0.95, 256-token budgets, eight categories rotating, memory sampled every 60 s): 1,417 waves, 11,336 requests, 1.99M completion tokens, 0 failures, 0 hangs; aggregate median 92.1 tok/s (78.0–104.1), no drift (first/middle/last 100 waves 90.4 / 92.9 / 91.1), TTFT p50 0.71 s throughout, MemAvailable per rank flat at 14 / 15–16 / 13–14 / 13–14 GiB across 344 samples.

Validation

• ruff check --select E,F,W --ignore E501 spark_transport runtime scripts performance integrations/lil: passes.
• pytest scripts/test_deepseek_v41_cycle_launcher.py: 10 passed (the rest of scripts/ was not run locally: one existing test imports torch, which this machine lacks).
• scripts/check_markdown_links.py . and scripts/check_release_safety.py .: clean.
• Live: text-only eager boot, the first serving boot (benchmark, needle 131K/262K, vision/tools, 20-min soak), nine lever boots (each after a fleet reboot), the 400K needle, and the six-hour soak on the recorded profile — all in the performance record. Hardware evidence is from our own cluster; nobody else's hosts were touched.

Limitations and questions

• No public image; every operator builds their own and pins IMAGE_ID. The build takes ~35 min on an idle GB10 (FlashInfer from source dominates).
• The seven patches are upstream-pending (vLLM #56214 open; #56228/#56208 merged). Each can be dropped as it lands.
• 1M context not run on this profile (430,080 recorded, 400K needle pass); adaptive verification off (flashinfer #5015); output quality beyond needle recall and the end-to-end checks not evaluated; single benchmark batch per cell.
• vLLM logs a Duplicate NCCL runtime WARNING (path assert, both 2.30.7); the patched library is the one mapped and loaded by PyNccl.
• Open to maintainers: the profile row is placed next to the Vision-Exp entry as another non-SparkRing-image profile; happy to move it under a research-only heading if preferred, and to swap bind-mounted patches for a published image if one is wanted.

Adapted work: tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark (MIT) — patches, prewarm/verify tools, benchmark method and prompt set; credited in THIRD_PARTY_NOTICES.md.

[sethforprivacy]

Pushed a follow-up on the same branch (2bbbe4a, 34ec5fd): the Engram disk loader's head split turned out to be the prefill bottleneck, and the fix is now part of the recipe.

What the traces showed. A rank-0 torch trace of one 16K prefill put 48.6 % of GPU time in AllReduce, but an idle-fleet PyNccl sweep ran the same 84 MB all-reduce at 84–109 Gb/s (the PCIe Gen5 ×4 ceiling of the ConnectX-7) under every protocol/channel/buffer variant, so half of that was ranks waiting for each other. Per-rank traces located it: the checkpoint lays its 24 hash columns out order-major, the loader split them contiguously, and rank 3 (six four-gram columns, nearly every row unique) issued 320,066 row reads to rank 0's 60,664 (six bigram columns, heavily deduped), stalling its GPU 1.6 s per chunk while the others sat in NCCL.

Change. Two env-gated additions to patches/engram.py (stock behaviour when unset): DSV41_ENGRAM_BALANCED=1 assigns hash columns strided over TP ranks (two heads of each n-gram order per rank) and permutes the all-gather back; DSV41_ENGRAM_PACKED_DIR reads from per-rank packed sparse shards (weight + scale adjacent, 264 B/row, one preadv instead of two) built once with tools/pack_engram_rows.py (~9 min per rank, 48 GB allocated). tools/test_engram_packed.py checks packed == two-read byte-for-byte. Launcher gains ENGRAM_BALANCED / ENGRAM_PACKED_DIR; quickstart has the pack step; the record has the traces and the lever table.

Measured on the recorded profile (rebooted boot): prefill 1,590 → 1,873 tok/s at 16K, 1,745 → 2,058 at 64K; burst 8×4K TTFT p50 11.1 → 9.8 s; 131K/262K needle pass at +17 % prefill; decode, draft acceptance and prompt-set output unchanged; 20-minute c=8 soak 632 requests, 0 failures, memory flat.

Also recorded as neutral on this profile (one rebooted boot each): FlashInfer b12x MXFP8 dense backend, NCCL_PROTO=Simple, 128 Engram reader threads, rejection_sample_method=standard. CI checks re-run locally: ruff, launcher tests (11), links, release safety.

[FujitsuPolycom]

AI-assisted pre-merge review

This review was produced with AI assistance (triage of the diff, records, and receipts); a maintainer verified the findings before posting.

Verified clean:

• Branch merges conflict-free onto current main (1b70930, post-Add R33 ARM64 image and qualified TP2/TP4 SparkCache profiles #262): test-merge simulated locally; the only overlapping files (README.md, docs/profiles/README.md) are additive table rows, and the R33 publication.json / profile surface from Add R33 ARM64 image and qualified TP2/TP4 SparkCache profiles #262 is untouched after merge.
• CI gates green on this PR (docs links, lint, pinned LIL bridge, release safety, tests).
• scripts/check_release_safety.py and scripts/check_markdown_links.py re-run locally on the PR tree: 0 findings, 761 links clean.
• recipes/deepseek-v41-flash-cycle.json ↔ runtime/deepseek-v41-gb10/image-receipt.json ↔ patches/MD5SUMS chain is internally consistent (vLLM dsv41-feat @ e47aa780, FlashInfer 0.7.0rc1, base image digest-pinned, seven patch md5s keyed per file).
• Status labels are honest per the repository's evidence vocabulary: quickstart and profile doc both state "implemented, live-benchmarked on one private cycle; not qualified."

Fixed in fa49d7f (pushed to this branch):

1. README.md profiles table cited the boot-2 probe profile (300K context / ~1.17M KV); the recipe and quickstart record 430,080 tokens and 2,182,642 KV tokens (5.07×). Row updated to 430,080 / ~2.18M.
2. README.md benchmark table cited the stock-loader cold-prefill figure (1,574 at 93K); the record's balanced+packed loader section measures 1,873 tok/s at 16K and 2,058 tok/s at 64K. Row updated to those figures, which the linked record documents.

Both are table-text corrections only; no behavior change. The 93K stock-loader number remains correctly recorded in the record itself.

[sethforprivacy]

Note that I have a likely major improvement to pre-fill coming shortly, will open a new PR when ready.

[FujitsuPolycom]

Excellent news!! I'm also working on a complete refactor of the repo layout in a new branch, so keep it coming and I'll get things cleaned and reorganized on the refactor. Current layout is all over the place.