Add R33 TP4/DCP4 SparkCache profile and qualify it on the four-Spark ring

README.md

@@ -25,9 +25,9 @@ reproducible benchmarks, and [test results](performance/).
    [validation checks](docs/PROFILE_VALIDATION.md).
 
 ## Profiles
-
-KV is approximate total token capacity. The shared TP4 figure is for DCP1;
-the TP2 figure is a reference estimate. `—` means no capacity
+KV is approximate total token capacity. The shared TP4 figures are per
+layout: DCP1 ~2.28M, DCP4 ~8.36M (both 24 GiB FP8 KV per rank). The TP2
+figure is a reference estimate. `—` means no capacity
 is recorded. Startup reports the actual capacity, which is separate from the
 per-request Context limit.
 
@@ -36,6 +36,7 @@ per-request Context limit.
 | Model / predictor | Serving stack | Transport | Layout | Context | Sequences | KV (tokens) | Guide |
 |---|---|---|---|---:|---:|---:|---|
 | **GLM-5.3 Flash NVFP4-Spark · native MTP3 + SparkCache** | [Generic R33 SparkRing image](runtime/sparkring/jovian-r33/image/README.md) | [Mesh + dual-domain NCCL](runtime/glm53-spark-mtp3-mesh/README.md) | TP4/DCP1 | 1M | 16 | ~2.28M | [Ring quickstart](docs/GLM53_TP4_PREFILL_QUICKSTART.md) |
+| **GLM-5.3 Flash NVFP4-Spark · native MTP3 + SparkCache** | [Generic R33 SparkRing image](runtime/sparkring/jovian-r33/image/README.md) | [Mesh + dual-domain NCCL](runtime/glm53-spark-mtp3-mesh/README.md) | TP4/DCP4 | 1M | 16 | ~8.36M | [DCP4 record](performance/records/glm53-flash/r33-image020-tp4-dcp4-sparkcache-20260911.md) |
 | GLM-5.3 Flash NVFP4-Spark · native MTP3, switched | [Shared SparkRing source image](runtime/sparkring/source_image/README.md) | Operator-selected NCCL links | TP4/DCP1 | 1M | 16 | — | [Switched quickstart](docs/GLM53_SWITCHED_TP4_QUICKSTART.md) |
 | GLM-5.2 EXL3 3.5-bpw | [SparkRing vLLM/ExLlamaV3 build](runtime/exl3-r7/README.md) | [SIRCL + NCCL](docs/SIRCL.md) | TP4/DCP4 | 1M | 16 | ~1.2M | [Quickstart](docs/GLM52_35BPW_QUICKSTART.md) |
 | DeepSeek-V4-Flash-0731 | [SparkRing vLLM/B12X image](runtime/deepseek0731-gb10/README.md) | [Patched NCCL](spark_transport/nccl/README.md) | TP4/DCP1 | 1M | 32 | ~1M | [Quickstart](docs/DEEPSEEK_V4_FLASH_QUICKSTART.md) |
@@ -81,7 +82,7 @@ For deployment with the shared image, use the matching two- or four-Spark entry
 
 | Profile | Layout | Retained guide | Replacement |
 |---|---|---|---|
-| NVFP4-Spark MTP3 cache/checkpoint mesh | TP4/DCP4 | [Pinned cache/checkpoint setup](docs/GLM53_MTP3_CACHE_CHECKPOINTS_QUICKSTART.md) | [Shared-image MTP3 profiles](docs/GLM53_TP4_PREFILL_QUICKSTART.md) |
+| NVFP4-Spark MTP3 cache/checkpoint mesh | TP4/DCP4 | [Pinned cache/checkpoint setup](docs/GLM53_MTP3_CACHE_CHECKPOINTS_QUICKSTART.md) | [R33 DCP4 SparkCache](performance/records/glm53-flash/r33-image020-tp4-dcp4-sparkcache-20260911.md) |
 | NVFP4 with BF16 DFlash2 | TP4/DCP1, DCP2 or DCP4 | [Pinned DFlash2 setup](docs/GLM53_JJ_R8_GB10_SPARKCACHE_TP4_QUICKSTART.md) | [Shared-image native MTP3](docs/GLM53_TP4_PREFILL_QUICKSTART.md) |
 | NVFP4-Spark MTP3 with 5 GiB KV per rank | TP2/DCP1 | [Pinned TP2 setup](https://github.com/FujitsuPolycom/sparkring/blob/2f01b6ee8f6173745c4b6b165498bbef82fc03f1/docs/GLM53_FLASH_SPARK_TP2_EXPERIMENTAL_QUICKSTART.md) | [Shared-image NVFP4-Spark TP2](runtime/profiles/glm53-flash-spark-tp2/README.md) |
 | Original NVFP4 MTP3 with 6.75 GiB KV per rank | TP2/DCP1 | [Pinned original-NVFP4 setup](https://github.com/FujitsuPolycom/sparkring/blob/2f01b6ee8f6173745c4b6b165498bbef82fc03f1/runtime/profiles/glm53-flash-nvfp4-tp2/README.md) | [Shared-image NVFP4-Spark TP2](runtime/profiles/glm53-flash-spark-tp2/README.md) |

docs/GLM53_TP4_PREFILL_QUICKSTART.md

@@ -7,8 +7,11 @@ The exact evidence and limits are in the
 This guide downloads or builds a source-pinned image and selects it through SparkRing's
 existing managed mesh deployment. It uses four NVIDIA Sparks, native MTP depth
 three, continuation-prefill coalescing, token-sharded mHC, and NCCL across both
-host PCIe domains. DCP1 is the qualified profile. This R33 profile contract does
-not expose a DCP4 launch profile.
+host PCIe domains. DCP1 is the baked-in qualified profile. A TP4/DCP4
+profile (SparkCache and cache-disabled variants) is available through the
+profile-contract overlay (`R33_PROFILE_CONTRACT_HOST_ROOT`; see the
+Reproduction section of the record) and is bounded-qualified in
+[r33-image020-tp4-dcp4-sparkcache-20260911](../performance/records/glm53-flash/r33-image020-tp4-dcp4-sparkcache-20260911.md).
 
 | Setting | DCP1 profile |
 |---|---|
@@ -21,7 +24,7 @@ not expose a DCP4 launch profile.
 | Coalescing and mHC prefill sharding | Enabled |
 | TP4 mesh and dual-domain NCCL | Enabled |
 | SparkCache connector / compact index cache | Enabled / disabled |
-| DCP top-k owner exchange | Disabled; its implementation requires DCP4 |
+| DCP top-k owner exchange | DCP1: disabled (implementation requires DCP4). DCP4 overlay: active (`full-CKV gather` prefill path) |
 
 The maximum context is a request-length limit. It is distinct from the total
 KV capacity reported at startup and does not establish a tested concurrency

docs/profiles/README.md

@@ -33,6 +33,7 @@ published configuration hash.
 |---|---|---|---|
 | GLM-5.3 Flash NVFP4-Spark with native MTP3 | four-Spark mesh, TP4/DCP1 or DCP4 | **research-only** shared-source profiles without SparkCache; no R33 cache qualification inherited | [Source profiles](../../runtime/sparkring/source_image/README.md), [image source lock](../../runtime/sparkring/source_image/glm53-tp4-lock.json) |
 | GLM-5.3 Flash NVFP4-Spark with MTP3 and SparkCache | four-Spark ring, TP4/DCP1 | R33: **qualified** for bounded startup, 112K exact-answer prefill and 8K restart restore; 24 GiB KV/rank; configured 1M request limit is not a tested 1M workload | [Quickstart](../GLM53_TP4_PREFILL_QUICKSTART.md), [qualification](../../performance/records/glm53-flash/r33-image020-tp4-sparkcache-20260911.md) |
+| GLM-5.3 Flash NVFP4-Spark with MTP3 and SparkCache | four-Spark ring, TP4/DCP4 | R33: **qualified** for bounded startup, 37K exact-answer prefill, prefix reuse, mHC 8192/2048, and 8K restore after planned restart and after fault injection; 24 GiB KV/rank (~8.36M tokens); configured 1M request limit is not a tested 1M workload; managed fabric installation required before launch | [Record](../../performance/records/glm53-flash/r33-image020-tp4-dcp4-sparkcache-20260911.md), [activation receipt](../../runtime/sparkring/jovian-r33/profiles/evidence/tp4-dcp4-sparkcache-activation-20260911.json) |
 | GLM-5.3 Flash NVFP4-Spark with MTP3 and SparkCache | two Sparks, TP2/DCP1, one DAC and both host PCIe domains | R33: **qualified** for bounded startup, 8K restart restore and C8 correctness; managed B12X, 7.5 GiB KV/rank; configured 1M request limit is not a tested 1M workload | [Quickstart](../../runtime/profiles/glm53-flash-spark-tp2/README.md), [qualification](../../performance/records/glm53-flash/r33-image020-tp2-sparkcache-20260911.md) |
 | GLM-5.3 Flash NVFP4-Spark with native MTP3 | switched four-Spark TP4/DCP1 | Provided as-is; switched hardware not validated | [Quickstart](../GLM53_SWITCHED_TP4_QUICKSTART.md) |
 | DeepSeek-V4-Flash-Vision-Exp with DSpark | four-Spark cycle, TP4 | Anemll image and MiaAI-Lab recipe with SparkRing patched NCCL; contributor-reported observations; independent reproduction is not claimed | [Artifact contract](../../runtime/deepseek-vision-exp/profile.json), [recipe](../../recipes/deepseek-v4-flash-vision-exp-tp4.json), [quickstart](../DEEPSEEK_V4_FLASH_VISION_EXP_TP4_QUICKSTART.md) |

performance/records/glm53-flash/r33-image020-tp4-dcp4-sparkcache-20260911.md

@@ -0,0 +1,214 @@
+# Generic R33 image TP4/DCP4 SparkCache qualification
+
+Status: **qualified** for the bounded functional checks below; served from
+the unchanged published R33 image. Throughput values are **research-only**
+observations and were not recorded in this session's harness.
+
+## Conditions
+
+Four NVIDIA GB10 systems ran the published generic R33 image
+`ghcr.io/fujitsupolycom/sparkring@sha256:1328a4f6f483014021a66a757012793629bd054d28d0fe4d5e581fa4aed776ef`
+(config ID `sha256:3c7779ad71dd0d5d6fae4c98e04b94c377429306158c2259fc44635892b8b8e4`).
+The deployment used the managed mesh renderer with the DCP4 profile
+templates added to the R33 profile contract in the same change, rendered
+against the existing `tp4-dcp1-sparkcache` site
+(`runtime/glm53-spark-mtp3-mesh/profile.py`), with the updated
+profile-contract directory and entrypoint supplied through
+`R33_PROFILE_CONTRACT_HOST_ROOT` bind mounts. The published image bytes,
+its artifact lock, and its NCCL library identity are unchanged.
+
+The model was `local-inference-lab/GLM-5.3-Flash-NVFP4-Spark` revision
+`df116c4fb16b1d37ae43d2cfd624de26ffbc832e` (config SHA-256
+`e1c0246a44ebefb5fd6383fb57aebbf7ac69ff6e7b23e989c0571b279a0eca23`). The
+serving profile used TP4/DCP4, PP1, `dcp_comm_backend=ag_rs`, InstantTensor
+target loading, static MTP3, 24 GiB FP8 KV per rank, 16 sequences, an
+8,192-token scheduler budget, a 1,048,576-token request limit,
+hardware-forwarded ring links with the fabric plan installed on every rank
+(routes, ingress qdiscs with tc flower rules, and two RDMA-TX rewrite
+markers per rank), dual-domain NCCL, token-sharded mHC,
+continuation-prefill coalescing, and SparkCache with async page capture.
+
+The RoCEnante overlay pins `decode_context_parallel_size: 4`; DCP4
+activations therefore require the managed fabric installation before
+launch. DCP1 activations disarm that overlay and do not require it.
+
+## Evidence
+
+- Engine initialization on rank 0 reported TP4/DCP4, `dcp_comm_backend=ag_rs`,
+  MTP3, InstantTensor, and the profile's exact
+  `cudagraph_capture_sizes=[4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64]`.
+- Graph capture completed on every rank (`Graph capturing finished in
+  61–78 secs`, 2.95–3.11 GiB).
+- The mapped NCCL library hashed to
+  `84a4b8d83fb5fa1f0d640d311ad38b45140672dae9889775fe1e4a3990479e47`
+  (`libnccl.so.2.31.2`, NCCL 2.31.2) on all four ranks, and the HCAs span
+  PCI domains `0000` (`rocep1s0f0/f1`) and `0002` (`roceP2p1s0f0/f1`).
+- Four exact-answer semantic requests returned exact responses.
+- Two completed cold prompts of 37,032 tokens each with zero cached tokens,
+  no `sample_tokens` timeout, and no fatal engine error.
+- DCP prefill execution used the cross-rank path: worker logs on every rank
+  report `Using full-CKV gather for GLM5Next B12X DCP prefill`
+  (`b12x_mla_sparse.py`), the DCP top-k owner-exchange machinery being
+  active rather than a DCP1 fallback.
+- A repeated prompt with an identical 37,019-token prefix hit the cache on
+  its second run: 36,352 cached tokens, 15.8 s → 5.0 s wall time.
+- mHC token sharding executed on every rank:
+  `GLM_MHC_PREFILL rank=<r> rows=8192 owner_rows=2048 rs=90 ag=90 aux=0`,
+  two executions per rank.
+- SparkCache reported `sparkcache_ranks=4 … healthy=1` throughout; page
+  snapshots committed (`kind=page_snapshot outcome=committed payload up to
+  100,200,614 B`).
+- A deterministic writer/restart/reader sequence: the writer created a
+  7,831-token entry; the whole ring restarted (planned container recreation,
+  cold engine start); the reader restored 6,144 tokens from SparkCache on
+  every rank in 50.6–54.8 ms (112–121 K tok/s, 46.3 MiB per rank).
+- A fault-injection recovery: all four containers were killed (`docker
+  kill`, SIGKILL), relaunched cold, returned to healthy serving, and the
+  reader restored 6,144 tokens on every rank in 51.9–57.7 ms (106–118 K
+  tok/s).
+
+## Measurement
+
+(2026-09-11 evening pass, cache-enabled ring. An additional diagnostic
+RouteFinal startup and a cache-disabled observation follow this section.)
+
+Prefill values are medians of three cold requests per prompt size; all nine
+requests reported zero cached tokens (unique prompt text per sample; prompt
+lengths within ~1% of the target sizes are recorded in the sample table).
+Decode used 10-second windows with zero request errors; each window was
+seeded by a warmup request over the full context so decode ran against a
+resident prefix. MTP-normalized steps/s counts target-model forward passes
+only (`spec_decode_num_drafts_total` delta per window); the effective
+acceptance length divides `spec_decode_num_accepted_tokens_total` deltas.
+Throughput values remain **research-only** observations: the harness does
+not pin clocks, warm-up policy, or a timing revision, so the tables are not
+a reproducible benchmark or a speedup claim.
+
+| Prompt tokens (actual) | Cold prefill tok/s, median of 3 |
+|---:|---:|
+| 8,192 (8,185–8,186) | 2,398.4 |
+| 16,384 (16,341–16,342) | 2,766.2 |
+| 32,768 (32,652–32,653) | 2,998.4 |
+
+| Context | Concurrency | C1/C4 aggregate decode tok/s | MTP-normalized steps/s | Effective acceptance length |
+|---:|---:|---:|---:|---:|
+| 8,192 | 1 | 50.2 | 18.2 | 2.76 |
+| 32,768 | 1 | 48.8 | 18.3 | 2.67 |
+| 8,192 | 4 | 100.7 | 35.7 | 2.82 |
+| 32,768 | 4 | 104.4 | 37.2 | 2.80 |
+
+## Result
+
+C1 decode matches the TP4/DCP1 record within 2–3%; C4 aggregate decode
+reaches 0.76–0.82× of DCP1, consistent with the cross-rank full-CKV gather
+(`Using full-CKV gather for GLM5Next B12X DCP prefill`) whose cost grows
+with scheduling pressure. Effective acceptance length is at or slightly
+above the DCP1 record (2.21–2.76). The DCP4 exchange buys ~3.7× KV
+capacity (8.36M vs 2.28M tokens on the same 24 GiB per rank).
+
+## RouteFinal dual-domain diagnostic startup
+
+A diagnostic start rendered the same site with `nccl_debug: INFO`
+(`NCCL_DEBUG=INFO`) through the unchanged published image. The ring reached
+HEALTH-OK and served an exact semantic answer. Every rank emitted 32
+`NET/IB RouteFinal` lines covering all four HCAs across both PCI domains:
+`rocep1s0f0` and `rocep1s0f1` (domain 0000) and `roceP2p1s0f0` and
+`roceP2p1s0f1` (domain 0002), 8 RouteFinal records per HCA, with
+`crossNic 1`, 32 channels, and `Connected all rings, use ring PXN 0 GDR 0`.
+This is the dual-domain route attribution the DCP1 record uses, now
+confirmed active for DCP4.
+
+## Six-case prefix-hit regression (1,027-token suffix included)
+
+After a 32,256-token cache hit established the base entry, six suffix
+geometries completed with exact expected answers: 515, 513, 531, **1,027**,
+512, and 2,048 tokens, with 28,160–28,672 cached tokens per case. As in the
+DCP1 record the harness checked completion liveness and exact answers, not
+raw output equivalence across shapes. No #220-style zero-hit behavior was
+observed: every case hit the cache.
+
+## Cache-disabled tp4-dcp4 observation
+
+A separate ring start rendered the cache-disabled `tp4-dcp4` profile
+(`SPARKCACHE_ENABLED=0`). Prefill medians of three cold samples per size
+(all zero-cached): 8,192 → 2,451.4 tok/s; 16,384 → 2,885.7; 32,768 →
+3,141.7 (one 8,192 sample absorbed a fresh-start JIT cost, 524 tok/s, and
+the median remains the middle of three). Decode windows: 8,192 c1 → 50.2
+(norm 18.4, acc 2.73); 32,768 c1 → 52.8 (norm 18.5, acc 2.85); 8,192 c4 →
+61.1 (norm 21.9, acc 2.80); 32,768 c4 → 104.6 (norm 35.2, acc 2.97). No
+snapshot writes or restores occurred in any worker log; the SIRCL
+capability vote remained (transport handshake) and vLLM's in-engine prefix
+caching stayed enabled (28,160-token hits in the suffix cases), so
+`SPARKCACHE_ENABLED=0` disables the SparkCache snapshot/capture layer only.
+
+## Conclusion
+
+The TP4/DCP4 SparkCache profile is bounded-qualified on the unchanged
+published R33 image through the profile-contract overlay: managed four-rank
+startup with graph capture, exact-answer serving, DCP prefill gather
+evidence, dual-domain RouteFinal attribution, the six-case prefix-hit
+regression including the 1,027-token suffix, planned-restart and SIGKILL
+fault-injection restore, and the 8.36M-token KV pool with 1M request
+admission. #220's zero-hit behavior was not observed. The cache-disabled
+`tp4-dcp4` profile starts and serves as an observation row: functional
+checks (four-rank start, semantic answer, 1M admission) passed on that
+start as well.
+
+## Reproduction (overlay and quickstart)
+
+The published image contains only the DCP1 profile contract. A DCP4 start
+requires the contract overlay from this repository:
+
+1. Clone this repo on every host and export
+   `R33_PROFILE_CONTRACT_HOST_ROOT=<repo>/runtime/sparkring/jovian-r33/profiles`
+   (the launcher bind-mounts it read-only at the contract host root and the
+   entrypoint prefers the overlaid contract; DCP1 starts do not need it).
+2. Render the site once on the head host:
+   `python3 runtime/glm53-spark-mtp3-mesh/profile.py render --site <site>.json
+   --bundle <bundle_root> --output <rendered> --image-receipt
+   <image-receipt.json>` with the site's `runtime_profile` set to
+   `tp4-dcp4-sparkcache` (or `tp4-dcp4`). The renderer emits one
+   `rank<N>.env` per rank; copy them to each host (the rank-0 env stays on
+   the head).
+3. The DCP4 overlays arm `decode_context_parallel_size: 4`; the managed
+   fabric plan (routes, tc flower qdiscs, RDMA-TX markers) must be
+   installed before launch — DCP1 silently disarms this and DCP4 does not.
+   The `R33_PROFILE_CONTRACT_HOST_ROOT` bind mount and the fabric plan are
+   the only two requirements that differ from a DCP1 start.
+4. Launch rank by rank:
+   `R33_PROFILE_CONTRACT_HOST_ROOT=... bash <launch-site>/launch-rank.sh <rank>
+   <rendered>/rank<rank>.env` (or the site's own rank env paths). Rank 0
+   reaches readiness ~4–5 minutes after container start; `/health` on the
+   rank-0 API returns 200 and a semantic request returns an exact answer.
+5. Cache-disabled starts use the same steps with `runtime_profile`:
+   `tp4-dcp4`; the renderer sets `SPARKCACHE_ENABLED=0` and the rank-local
+   cache root directory must still exist (the launcher requires it even
+   when the cache is disabled).
+
+The rendered env files are rank-specific (HOST_IP differs per rank); do not
+reuse a rank-0 env on a peer host.
+
+The engine admitted the one-million-token request limit and reported
+`GPU KV cache size: 8,364,901 tokens` (the DCP4 sharded pool; the DCP1
+profile reports ~2.28M on the same 24 GiB/rank).
+
+The activation receipt is
+[`evidence/tp4-dcp4-sparkcache-activation-20260911.json`](../../../runtime/sparkring/jovian-r33/profiles/evidence/tp4-dcp4-sparkcache-activation-20260911.json),
+bound to the profile contract SHA-256 recorded inside it and passing
+`verify_profile.py validate-activation`.
+
+## Limitations
+
+This record does not include three-coordinated-cold-start latency evidence.
+No completed one-million-token request was run; multimodal correctness,
+switched hardware, other models, and sustained-memory behavior do not
+inherit this qualification. The configured limit is not a tested workload.
+The mounted profile-contract overlay is verified for internal consistency
+and by the profile contract SHA; its contract bytes are newer than the
+baked image contract and the published image remains byte-unchanged.
+Prefill and decode throughput windows are research-only observations (see
+Measurement); the INFO RouteFinal startup was a separate diagnostic start,
+not the qualification serving configuration. The cache-disabled `tp4-dcp4`
+row is an observation, not a standalone qualification, and the in-engine
+vLLM prefix cache remained enabled on that start (SparkCache snapshot layer
+disabled only).