Add and live-qualify a TP4/DCP4 launch profile on the R33 image

[FujitsuPolycom]

Parent

Follow-up to the merged R33 image release (#262). Related: #220 (native MTP3 sparse prefix retention, suspected defect affecting TP4 DCP4 capture), #254 (rejecting dynamic GLM DFlash depth on TP4 DCP4 plans).

What to build

The generic R33 ARM64 image (ghcr.io/fujitsupolycom/sparkring@sha256:1328a4f6f483014021a66a757012793629bd054d28d0fe4d5e581fa4aed776ef, source commit 86ae2153a9177fea487d1b208305624a33d66616) ships only tp4-dcp1 and tp4-dcp1-sparkcache launch profiles. The TP4/DCP1 quickstart states: "DCP1 is the qualified profile. This R33 profile contract does not expose a DCP4 launch profile."

A TP4/DCP4 profile means four GB10 ranks with tensor-parallel degree four and data/context-parallel degree four, so each rank owns one quarter of requests and a quarter of the context window; KV spans 24 GiB per rank with the DCP top-k owner exchange moving keys between rank groups. The mesh owner-exchange and fused-endpoint source exists and passed CPU/ARM64 binding tests, but the R33 profile contract and renderer cannot select it. No live serving run has exercised DCP4 on the R33 image.

This issue adds the missing profile and qualifies it live.

Steps

1. Render a tp4-dcp4-sparkcache launch profile in runtime/sparkring/jovian-r33/profiles/ that binds the existing receipt contract (sparkring-r33-image-receipt/v1), validates via verify_profile.py, and selects an empty activation_blockers list when the image is pulled from the published digest. Include the DCP top-k owner exchange and fused mesh endpoints. Cache stays disabled for the first profile (matching the TP4 DCP1 pattern: cache-disabled first, cache-enabled second).
2. Add profile-contract tests in runtime/sparkring/jovian-r33/profiles/test_profile_contract.py mirroring the DCP1 cases: schema field coverage, receipt binding, activation-blocker enumeration, mesh renderer output for four ranks (verify the four-HCA NCCL ring and six-QP virtual mesh the DCP1 profile documents).
3. Live qualification on DGX4 — the existing four-rank mesh must be briefly stopped; this requires operator presence. Qualification set, mirroring the TP4/DCP1 record performance/records/glm53-flash/r33-image020-tp4-sparkcache-20260911.md:
   • Four-rank startup through managed gates; verify --tensor-parallel-size 4 and DCP degree 4 in the effective command
   • Cold prefill medians of three samples at 8,192 / 16,384 / 32,768 tokens (exact-token harness)
   • Decode cells at 8K/32K for C1 and C4 with MTP-normalized steps/s
   • Exact-answer retrieval and the six-case prefix-hit regression (the 1,027-token suffix case matters for the packed checkpoint adapter)
   • Dual-domain NCCL route attribution across both host PCIe domains (the RouteFinal evidence pattern the DCP1 record uses)
   • DCP top-k owner-exchange activation evidence from worker logs, not environment-variable presence
4. Optional cache profile (tp4-dcp4-sparkcache): capture, restore across process restart, mixed prefill, and failed-restore recovery, gated by the same steps the TP2/TP4 DCP1 cache records document. Note issue [bug] Native MTP3: pinned sparse prefix retention may discard reachable cache states (SparkCache / TP4 DCP4) #220 (sparse prefix retention discarding reachable states) is unresolved and affects DCP4 capture; if the zero-hit reproduction fires during cache qualification, stop and record it rather than qualifying past it.
5. Documentation: expose the DCP4 profile in docs/GLM53_TP4_PREFILL_QUICKSTART.md and README.md; write a qualification record under performance/records/glm53-flash/ with explicit status (qualified or research-only per evidence, not assumed); add the qualification entry to runtime/sparkring/jovian-r33/publication.json when the record is finalized.

Current evidence for DCP4 on this model (all pre-R33, all research-only)

• performance/records/glm53-flash/b12x-kda-dcp4-20260903.md — TP4/DCP4 perf observation on the earlier sparkring-glm53-sparkcache image sha256:0d4029b3…, TP4/DCP4, 24 GiB KV/rank, BF16 DFlash2 depth seven. Performance only; no functional qualification.
• outputs/issue220-dcp4-live-20260909/ (Codex worktree, not in repo) — live DCP4 run on fingerprint vllm-0.1.dev1+gd377796e8: healthy four-rank startup, one arithmetic restore smoke passed. Pre-dates the R33 composition entirely.

Acceptance criteria

• tp4-dcp4 (cache-disabled) renders from a verified R33 receipt with empty activation_blockers
• Profile-contract tests cover the four-rank DCP4 renderer output and receipt binding
• Live receipt on DGX4: cold 8K/16K/32K prefill medians (3 samples each), C1/C4 decode with MTP normalization, exact-answer retrieval, six-case prefix regression, dual-domain NCCL route attribution
• DCP top-k owner exchange confirmed active in worker logs
• Qualification record written with honest scope: what is qualified vs research-only (e.g. multimodal, sustained soak, 1M-token requests stay unclaimed)
• publication.json gains a tp4-dcp4 qualification entry only when the record is finalized; docs updated in the same change
• If [bug] Native MTP3: pinned sparse prefix retention may discard reachable cache states (SparkCache / TP4 DCP4) #220's zero-hit behavior reproduces on DCP4, the cache profile is not published past it; the failure is recorded as a tracked limitation

Blocked by

• None — can start immediately. Live qualification (step 3) requires the four-rank DGX4 mesh to be taken down and restored, so it needs operator scheduling; the profile and test work (steps 1-2) can proceed AFK against the published image.

[FujitsuPolycom]

Completed by #265 (merged): the TP4/DCP4 SparkCache profile renders from a verified R33 receipt with empty activation_blockers, is covered by profile-contract and image-contract tests, and is live-qualified on the DGX4 ring.

Acceptance evidence added in the final commits of #265 and recorded in performance/records/glm53-flash/r33-image020-tp4-dcp4-sparkcache-20260911.md:

• Cold prefill medians of 3 at 8K/16K/32K (2,398 / 2,766 / 2,998 tok/s; all zero-cached) and C1/C4 decode windows at 8K/32K with MTP-normalized steps/s (18.2–18.3 C1; 35.7–37.2 C4) and effective acceptance lengths (2.67–2.82). Research-only throughput, per the record.
• RouteFinal dual-domain diagnostic startup with NCCL_DEBUG=INFO: 32 RouteFinal records per rank across all four HCAs on both PCI domains, crossNic=1, HEALTH-OK + exact answer.
• Six-case prefix-hit regression after a 32,256-token cache hit, including the 1,027-token suffix: all exact, 28,160–28,672 cached tokens, no [bug] Native MTP3: pinned sparse prefix retention may discard reachable cache states (SparkCache / TP4 DCP4) #220 zero-hit behavior.
• DCP top-k owner-exchange evidence from worker logs (Using full-CKV gather for GLM5Next B12X DCP prefill on every rank).
• Cache-disabled tp4-dcp4: separate start, SPARKCACHE_ENABLED=0 rendered, prefill/decode observation recorded (snapshot layer off; in-engine prefix cache remains — scope noted honestly).
• publication.json gains the tp4-dcp4-sparkcache bounded-qualified entry with the record hash and activation receipt.
• Reproduction guide (contract overlay via R33_PROFILE_CONTRACT_HOST_ROOT, profile.py render, rank env distribution, fabric requirement) is in the record; the TP4 prefill quickstart no longer claims DCP4 is absent.

Unclaimed as before: no completed 1M-token request, no sustained soak, multimodal/switched hardware out of scope. Closing.

[FujitsuPolycom]

Correction (via #269, merged): the earlier note said worker logs proved the DCP top-k owner-exchange machinery active. Verified in the published image source: this build has no separate topk-to-owner exchange step — top-k runs over the all-gathered global CKV cache and ownership is resolved arithmetically in _map_global_topk_to_gathered_ckv_kernel (owner = (tok // DCP_INTERLEAVE) % DCP_SIZE). The record now cites the verified chain (gate condition + DCP-group all-gather + owner-mapping kernel) rather than implying a distinct exchange primitive.

[FujitsuPolycom]

Record corrections merged (#271): (1) the first measurement pass is relabeled potentially affected — the DeepSeek distribution jobs ran 13:09–13:42 Central on ranks 0/1/3 and overlapped the 37K probes, prefill medians, and decode windows (host-level journal evidence, not container census); the earlier 'no rerun required' claim is withdrawn, and a rerun is pending a ring restart fix (15:22:14 multi-rank stop event documented in the record's Conditions). (2) The C4-decode vs DCP1 gap attribution to the prefill-only full-CKV gather is withdrawn — no attribution is claimed without profiling evidence. (3) The owner-exchange acceptance item is recorded as amended, not passed: this image resolves top-k ownership arithmetically over the all-gathered cache; there is no separate exchange primitive.