Add DeepSeek-V4.1-Flash four-Spark cycle profile (Engram tables on NVMe, stock vLLM dsv41-feat image, DSpark k=5)

.ruff.toml

@@ -1,5 +1,9 @@
 # The recovered reference-runtime snapshot is preserved source evidence. Keep
 # its two known upstream style findings visible here instead of rewriting it.
+# runtime/deepseek-v41-gb10/patches and tools are vendored byte-for-byte from an MIT
+# upstream and pinned by md5 (THIRD_PARTY_NOTICES.md); they are not rewritten here.
+extend-exclude = ["runtime/deepseek-v41-gb10/patches", "runtime/deepseek-v41-gb10/tools"]
+
 [lint.per-file-ignores]
 "runtime/patches/00-reference-vllm/added/v1/worker/gpu/spec_decode/dspark/speculator.py" = ["E731"]
 "runtime/patches/00-reference-vllm/added/v1/worker/gpu/spec_decode/dspark/utils.py" = ["F401"]

README.md

@@ -39,6 +39,7 @@ per-request Context limit.
 | GLM-5.3 Flash NVFP4-Spark · native MTP3, switched | [Shared SparkRing source image](runtime/sparkring/source_image/README.md) | Operator-selected NCCL links | TP4/DCP1 | 1M | 16 | — | [Switched quickstart](docs/GLM53_SWITCHED_TP4_QUICKSTART.md) |
 | GLM-5.2 EXL3 3.5-bpw | [SparkRing vLLM/ExLlamaV3 build](runtime/exl3-r7/README.md) | [SIRCL + NCCL](docs/SIRCL.md) | TP4/DCP4 | 1M | 16 | ~1.2M | [Quickstart](docs/GLM52_35BPW_QUICKSTART.md) |
 | DeepSeek-V4-Flash-0731 | [SparkRing vLLM/B12X image](runtime/deepseek0731-gb10/README.md) | [Patched NCCL](spark_transport/nccl/README.md) | TP4/DCP1 | 1M | 32 | ~1M | [Quickstart](docs/DEEPSEEK_V4_FLASH_QUICKSTART.md) |
+| DeepSeek-V4.1-Flash · Engram on NVMe · DSpark k=5 | [Self-built stock vLLM `dsv41-feat` image](runtime/deepseek-v41-gb10/README.md) | [Patched NCCL](spark_transport/nccl/README.md) | TP4/DCP1 | 430,080 | 8 | ~2.18M | [Quickstart](docs/DEEPSEEK_V41_FLASH_QUICKSTART.md) |
 | Qwen3.8-27B EXL3 K5/K6 | [SparkRing vLLM/ExLlamaV3 build](runtime/qwen38/README.md) | [Patched NCCL](spark_transport/nccl/README.md) | TP4/DCP1 | 1M | 64 | — | [Quickstart](docs/QWEN38_27B_EXL3_K5K6_QUICKSTART.md) |
 | DeepSeek-V4-Flash-Vision-Exp with DSpark (research-only) | [Anemll image / MiaAI-Lab recipe](runtime/deepseek-vision-exp/profile.json) | [SparkRing patched NCCL](spark_transport/nccl/README.md) | TP4 | 1M | 48 | — | [Quickstart](docs/DEEPSEEK_V4_FLASH_VISION_EXP_TP4_QUICKSTART.md) |
 
@@ -122,6 +123,7 @@ Results attempt to reflect real world use-case numbers in all instances unless o
 | [GLM-5.2 EXL3 3.5-bpw · 4 Sparks](performance/records/glm-3.5bpw/normalized-base-20260822.md) | 16K | 671 (16K) | 20.15 | 64.13 | C8: 64.13 | 25.39 |
 | [DeepSeek-V4-Flash DSpark · 2 Sparks](performance/records/deepseek-v4-flash/normalized-tp2-base-temp1-n5-20260823.md) | 16K | 1,926 (16K) | 58.36 | 162.69 | C32: 307.13 | 59.31 |
 | [DeepSeek-V4-Flash-0731 · 4 Sparks](performance/records/deepseek-v4-flash/normalized-tp4-base-temp1-n5-20260823.md) | 16K | 2,488 (16K) | 68.84 | 265.16 | C32: 508.11 | 95.77 |
+| [DeepSeek-V4.1-Flash · Engram on NVMe · DSpark k=5 · 4 Sparks](performance/records/deepseek-v41-flash/cycle-tp4-dspark5-graphs-20260910.md) | short prompts, temp 0 | 1,873 (16K) / 2,058 (64K) | 56.2 | — | C6: 159.9 | 77.3 |
 | [Qwen3.8-27B EXL3 K5/K6 · 2 Sparks](performance/records/qwen38-27b/normalized-tp2-1m-probmtp-temp1-20260823.md) | 16K | 1,367 (16K) | 29.50 | 142.20 | C16: 184.39 | 39.95 |
 | [Qwen3.8-27B EXL3 K5/K6 · 4 Sparks](performance/records/qwen38-27b/normalized-tp4-1m-probmtp-temp1-20260823.md) | 16K | 1,964 (16K) | 35.07 | 191.02 | C8: 191.02 | 48.46 |
 

THIRD_PARTY_NOTICES.md

@@ -387,3 +387,34 @@ remain applicable to the built library. Source preparation does not distribute
 model weights. The parent image also contains CUDA, framework and kernel
 dependencies governed by their respective licenses; rebuilding this recipe
 does not replace or remove those obligations.
+
+## 15. DeepSeek-V4.1-Flash GB10 profile artifacts
+
+### 15a. tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark (patches included)
+
+`runtime/deepseek-v41-gb10/patches/` contains seven Python files copied byte-for-byte from
+https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark (`patch/` directory, boot 9,
+MIT License, Copyright (c) 2026 tonyd2wild; SM12x page-size and top-k fixes authored by Kai as
+credited in that repository). They are bind-mounted over the vLLM `dsv41-feat` tree at launch and
+are modified files of vLLM (Apache-2.0). The Engram-on-disk method is theirs. `tools/prewarm5.py`
+and `tools/verify5.py` in the same directory are from the same repository, as are the benchmark
+method, prompt set and harnesses referenced by `performance/records/deepseek-v41-flash/`.
+These files are pinned by md5 and excluded from linting in `.ruff.toml`.
+
+### 15b. vLLM `dsv41-feat` branch (referenced; built into the operator's image)
+
+`runtime/deepseek-v41-gb10/build-image.sh` copies the Python tree of `vllm-project/vllm`
+commit `e47aa780bccf59f59dfa2cbb18e17a10b4fe69ba` (Apache-2.0) over the `vllm/vllm-openai`
+nightly image and rebuilds `_C_stable_libtorch` from that tree with NVIDIA CUTLASS v4.7.1
+(BSD-3-Clause). No vLLM source is included in this repository.
+
+### 15c. FlashInfer (referenced; built into the operator's image)
+
+The image builds FlashInfer commit `07869c61ba581e6d6b8ad8d142f4a6c89b707cc1` (Apache-2.0)
+with its pinned CUTLASS, CCCL and spdlog submodules. No FlashInfer source is included here.
+
+### 15d. Model weights (not included)
+
+`deepseek-ai/DeepSeek-V4.1-Flash` is distributed by DeepSeek under the MIT License. No weights or
+model files are included; operators fetch the checkpoint themselves.
+

docs/DEEPSEEK_V41_FLASH_QUICKSTART.md

@@ -0,0 +1,194 @@
+# DeepSeek-V4.1-Flash four-Spark cycle quickstart
+
+Serve `deepseek-ai/DeepSeek-V4.1-Flash` (the stock checkpoint) as four tensor-parallel
+ranks on a directly cabled four-Spark cycle, with the model's two Engram lookup tables
+left on each rank's NVMe.
+
+**Status: implemented; live-benchmarked on one private cycle; not qualified.** The
+profile runs a stock upstream vLLM image that you build yourself from pinned sources
+([`runtime/deepseek-v41-gb10`](../runtime/deepseek-v41-gb10/README.md)); no public
+image digest exists to replay. The machine-readable contract is
+[`recipes/deepseek-v41-flash-cycle.json`](../recipes/deepseek-v41-flash-cycle.json); the
+evidence is in the [profile record](profiles/DEEPSEEK_V41_FLASH.md) and the
+[benchmark record](../performance/records/deepseek-v41-flash/cycle-tp4-dspark5-graphs-20260910.md).
+
+## Why this profile is shaped the way it is
+
+DeepSeek-V4.1-Flash is 475 GiB on disk: a 552B MoE backbone whose routed experts are already
+MXFP4, plus two FP8 Engram n-gram tables of 94.6 GiB each. Four GB10s hold 121.7 GiB each.
+With the tables row-sharded in memory a rank needs about 118.8 GiB before KV, activations and
+the CUDA context, so it does not fit; vLLM's `cpu_offload` does not help because pinned host
+memory on GB10 is the same pool the GPU allocates from. Keeping the tables in the safetensors
+shards and reading the 48 rows a token needs on demand brings a rank to **78.8 GiB**
+(text-only) or **81.6 GiB** (with the DSpark draft layers and the vision encoder). That
+method, and the SM12x fixes the vLLM `dsv41-feat` branch still needs on GB10, come from
+[tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark](https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark)
+(MIT; SM12x page fixes by Kai) and are bind-mounted over the image unchanged. What this
+profile adds is the switchless-cycle transport: SparkRing's patched NCCL and the four-rank
+cycle environment, so no Ethernet switch is needed.
+
+| | value |
+|---|---|
+| Ranks / cabling | 0–3, four DACs as `0-1-2-3-0`, two RoCE devices per rank |
+| Weights resident per rank | 78.79 GiB text-only; 81.6 GiB with DSpark draft + vision (measured) |
+| Engram tables | on each rank's NVMe, read on demand (23.6 GiB per rank per table not allocated); balanced hash-column split + packed single-read shards (`ENGRAM_BALANCED=1`, `ENGRAM_PACKED_DIR`) |
+| Request limit / sequences / scheduler tokens | 430,080 / 8 / 8,192 |
+| `--gpu-memory-utilization` | 0.83 with `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` → KV 10.94 GiB = 2,182,642 tokens (5.07× 430K) measured; 13–15 GiB MemAvailable per rank |
+| Speculation | DSpark k=5, greedy draft, block rejection, adaptive verification off |
+| CUDA graphs | `FULL_AND_PIECEWISE`, capture sizes = every multiple of 5 and 6 up to 48 |
+| Block size | 128 (required; an auto-picked 64 fails KV initialization) |
+| Vision / tools | up to 4 images per request; `deepseek_v41` tool and reasoning parsers |
+| Environment template / launcher | `scripts/config/deepseek-v41-flash-cycle.env.example` / `scripts/deepseek_v41_cycle_serve.sh` |
+
+## 1. Prepare the ranks
+
+Complete [the prerequisites](PREREQUISITES.md) and the [bootstrap guide](BOOTSTRAP.md) for a
+four-Spark cycle, including forwarding and the relay routes: a torch rendezvous reaches ranks
+that are not directly cabled only if every node relays for its neighbours. Run
+`scripts/ring_doctor.py --site <site> --verify` and require a clean reachability matrix.
+
+Check the GPUs before trusting any number: GB10s can latch below 1 GHz with nothing visible
+in `nvidia-smi` except speed (clear it by unplugging the adapter for 30–60 s), and can sit in a
+slower hidden state under load. A 15 s fp16 matmul burn on a healthy unit reads about
+2.2–2.4 GHz at 80 W or more.
+
+### Weights
+
+Download the stock checkpoint once and place a complete, byte-identical copy on **every rank's
+local NVMe** (about 510 GB each). Do not serve the checkpoint over NFS: the Engram rows are
+read from shards 47 and 48 at serve time, and the read path is the first thing that bounds
+throughput under concurrency. Pin the revision; this profile records
+`dba1be0a40aa45a94ad051997016db3960a90277` (revision `df42c109…` has identical weights and
+`config.json`; only the reference encoder scripts differ). Between two Sparks a direct link
+copies at ~600 MB/s, so copy to one rank and replicate over the fabric.
+
+### Image
+
+Build the image on an idle Spark with
+[`runtime/deepseek-v41-gb10/build-image.sh`](../runtime/deepseek-v41-gb10/build-image.sh),
+then `docker save | ssh <rank> docker load` to the other three and confirm identical image
+IDs. Record the ID in `IMAGE_ID` in every env file; the launcher refuses a mismatch. Read the
+[builder README](../runtime/deepseek-v41-gb10/README.md) first: the stable-extension compile
+needs an idle node (one CUTLASS translation unit takes a compiler process past 7 GiB), while
+the FlashInfer layers can be built inside a 7 GiB cgroup on a node that is still serving.
+
+### Patches and NCCL
+
+Copy `runtime/deepseek-v41-gb10/patches/` (seven files, `mounts.txt`, `MD5SUMS`) to the same
+absolute path on every rank; the launcher verifies the md5s. Extract SparkRing's patched
+NCCL from any published SparkRing image and put it at the same path on every rank:
+
+```bash
+docker create --name nccl-tmp <sparkring image> true
+docker cp -L nccl-tmp:/opt/sparkring/nccl/libnccl.so.2 /path/to/libnccl.so.2
+docker rm nccl-tmp
+```
+
+The image's own pip NCCL is also 2.30.7; vLLM logs a `Duplicate NCCL runtime` warning
+because the two paths differ. The preloaded library is the one mapped in every process and
+the one PyNccl loads by path.
+
+### Engram packed shards (once per rank)
+
+The stock loader reads two 4 KiB pages per Engram row (weight and scale sit ~24 GB apart in the
+shard) and splits the 24 hash columns contiguously, which hands rank 3 the six four-gram columns
+(nearly every row unique) and rank 0 the six bigram columns (heavily repeated): per-rank traces of
+one 16K prefill showed rank 3 issuing 320K row reads to rank 0's 60K and the other ranks waiting
+for it at the next all-reduce. The recipe therefore sets `ENGRAM_BALANCED=1` (strided columns,
+two heads of each order per rank) and reads from packed shards built once per rank:
+
+```bash
+docker run --rm --entrypoint python3 --memory 6g \
+  -v "$MODEL_HOST_PATH:/models/DeepSeek-V4.1-Flash:ro" -v "$CACHE_HOST_PATH:/cache" \
+  -v "$PWD/runtime/deepseek-v41-gb10/tools:/tools:ro" "$IMAGE" \
+  /tools/pack_engram_rows.py --model-dir /models/DeepSeek-V4.1-Flash --out-dir /cache/engram-packed \
+  --tp 4 --rank "$NODE_RANK" --balanced
+```
+
+About nine minutes per rank; the two sparse files show 101 GB logical / ~48 GB allocated. The
+manifest records the covered ranges and the loader refuses a shard that does not cover the rank's
+columns (it then logs a warning and reads the checkpoint shards directly). Measured on the cycle:
+prefill 1,590 → 1,873 tok/s at 16K and 1,745 → 2,058 at 64K, burst TTFT p50 11.1 → 9.8 s, decode
+and acceptance unchanged, 131K/262K needle pass.
+
+### Environment
+
+Copy the template once per rank and resolve every placeholder. `NODE_RANK` and
+`VLLM_HOST_IP` differ between ranks; everything else must be byte-identical on all four —
+a configuration mismatch hangs the rendezvous with no error.
+
+```bash
+cp scripts/config/deepseek-v41-flash-cycle.env.example /path/to/rank-0.env   # and 1, 2, 3
+scripts/deepseek_v41_cycle_serve.sh --check /path/to/rank-0.env
+```
+
+`--check` is offline: it validates the file, the model directory (`config.json` must name
+`DeepseekV41ForCausalLM`, shard 48 must be present), the patch md5s, the NCCL library and
+the cycle transport values, then prints the exact `docker run` command. Compare the printed
+serving values across the four ranks before launching.
+
+## 2. Launch one rank per host
+
+Reboot the ranks before a first launch or a measurement (memory fragmentation on GB10 costs
+real throughput), then start workers 3, 2 and 1 before rank 0:
+
+```bash
+scripts/deepseek_v41_cycle_serve.sh --run /path/to/rank-3.env   # on rank 3
+scripts/deepseek_v41_cycle_serve.sh --run /path/to/rank-2.env   # on rank 2
+scripts/deepseek_v41_cycle_serve.sh --run /path/to/rank-1.env   # on rank 1
+scripts/deepseek_v41_cycle_serve.sh --run /path/to/rank-0.env   # on rank 0, the API host
+docker logs -f deepseek-v41-flash-r0
+```
+
+`--run` refuses to start if the container exists, if `MemAvailable` is under 100 GiB, or if
+the image identity differs from `IMAGE_ID`. Stop every rank (rank 0 first) before relaunching:
+a worker that starts while an old head still listens on the rendezvous port joins the old
+head and hangs.
+
+Expect about eight minutes to readiness from local NVMe: ~4 min of weights, ~1 min for the
+DSpark draft layers, then graph capture and FlashInfer autotune. Lines to look for:
+
+```text
+Engram DISK mode: layer 1 rows [<start>, <end>) read from model-00047-of-00048.safetensors
+Model loading took 78.79 GiB memory            (text-only) / consumed 85.71 GiB (serving shape)
+GPU KV cache size: 2,182,642 tokens, Maximum concurrency for 430,080 tokens per request: 5.07x
+Application startup complete.
+```
+
+With `ENGRAM_BALANCED=1` each rank logs `BALANCED column assignment, rank r owns hash columns [...]`
+(six distinct columns per rank) and, when the packed shard is used, `PACKED single-read shard`.
+Without it the Engram row ranges must differ per rank and together cover the table; identical `off=`
+values on every rank mean the rank-offset fix is not mounted.
+
+## 3. Verify rank 0
+
+```bash
+curl --fail http://localhost:8000/health
+curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
+  "model":"deepseek-v4.1-flash",
+  "messages":[{"role":"user","content":"Count from 1 to 30, comma separated, then say done."}],
+  "max_tokens":120,"temperature":0}'
+```
+
+Thinking is off by default; a request enables it with `"chat_template_kwargs": {"thinking": true}`.
+With `API_KEY_FILE` set, add `-H 'Authorization: Bearer <key>'` to the chat request; `/health` stays keyless
+so router health probes keep working. Rank 0 without `API_KEY_FILE` is an open server — set it before
+the port sits behind any route that does not authenticate on its own.
+Check `SpecDecoding metrics` in the log for a mean acceptance length above one, and
+`/metrics` for zero preemptions under load.
+
+## 4. Sizing
+
+`--kv-cache-memory-bytes` is not used here; `--gpu-memory-utilization 0.83` sizes the pool
+against ~113 GiB free at startup, and `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` stops the
+profiler reserving an estimated 1.5 GiB for graphs that measure 0.54 GiB on this profile. On the
+recorded boot the pool came out at 10.94 GiB (2,182,642 tokens, 5.07× the 430,080-token limit)
+with 13–15 GiB MemAvailable per rank while serving. 0.85 was measured too (3,085,606 tokens,
+400K needle pass, same speed) but left only 7–9 GiB MemAvailable, so the recorded profile keeps
+the headroom; 0.80 at a 300,000-token limit gave 1,171,588 tokens. Raising `--max-num-batched-tokens`
+to 16,384 did not boot at 0.80 (the profiler run needs 1.9 GiB of KV for one full-length request
+and 1.84 GiB was left). `--max-num-seqs 8` is the soaked value; 16 booted at 0.80 with 13–15 GiB
+free, was neutral up to eight streams and reached 285 tok/s aggregate on the prompt set at 16
+streams, so it is a valid admission option when per-stream speed matters less than throughput.
+With DSpark k=5 every decode batch is a multiple of 5 or 6 tokens and the graph capture list
+follows from the sequence cap. 1M context has not been run on this profile.