Add exact GLM-5.3 adaptive MTP and DFlash7 SparkCache runtimes

README.md

@@ -65,8 +65,9 @@ See the [profile registry](docs/profiles/README.md) for recipe identities and ev
 ### GLM-5.3 Flash research observation
 
 **Research-only — 16K context, single observation.** The SparkCache-enabled
-profile recorded 2,371 tok/s prefill and 36.06 tok/s sustained
-C1 decode on random tokens. No A/B baseline has been completed.
+profile recorded 2,371 tok/s prefill and 36.06 tok/s sustained C1 decode on
+random tokens. No A/B baseline has been completed. C4 and C8 were capacity-limited
+and are omitted rather than reported as throughput results.
 
 | Profile | Prefill | C1 decode | C8 decode | Highest valid decode | Coding peak |
 |---|---:|---:|---:|---:|---:|

docs/GLM53_B12X_KDA_ADAPTIVE_MTP_SPARKCACHE_TP4_QUICKSTART.md

@@ -2,15 +2,16 @@
 
 Status: **implemented, not qualified**. This guide builds vLLM commit
 `0b67266a0f37d6146a8403fb8482403c62f412d5` and the SparkCache overlay from
-commit `20838ace3ebda570ca039cb7f1976c29da554b39` for four DGX Spark systems at
+commit `5d571018de5b63a9a90e5c11e6d6e86bbff4a957`, Git tree
+`e864ed9ad64f771188fdb59aa9738e348134d636`, for four DGX Spark systems at
 TP4/DCP1.
 
 The serving profile uses embedded MTP with maximum depth five, initial depth
 three, and a 32-step acceptance window. Fastsafetensors uses queue size one.
 TP4 makes the pinned vLLM loader select `nogds=True`, so model loading uses
 pipelined host I/O without GPU Direct Storage.
 
-The profile reserves 20 GiB of FP8 KV per rank and enables SparkCache native
+The profile reserves 20 GiB of FP8 KV per rank and enables SparkCache CUDA
 restore, tail-only publication, shared restore trunks, and bounded shared GPU
 prefix leases. Image construction and distribution do not require stopping an
 existing service. Do not run the launch command until all four ranks have the
@@ -25,7 +26,7 @@ storage. Clone both repositories beside each other:
 git clone https://github.com/FujitsuPolycom/sparkring.git sparkring
 git -C sparkring checkout --detach <revision-containing-this-guide>
 git clone https://github.com/FujitsuPolycom/sparkcache.git sparkcache
-git -C sparkcache checkout --detach 20838ace3ebda570ca039cb7f1976c29da554b39
+git -C sparkcache checkout --detach 5d571018de5b63a9a90e5c11e6d6e86bbff4a957
 
 IMAGE='sparkring-glm53-runtime:b12x-kda-adaptive-mtp-0b67266a-arm64' \
 BUILD_RECEIPT="$PWD/glm53-b12x-kda-adaptive-mtp-runtime-receipt.json" \
@@ -38,8 +39,8 @@ python sparkcache/deploy/glm53_flash/build_image.py \
   --containerfile deploy/glm53_flash/Containerfile.b12x-kda-adaptive-mtp \
   --base-image "${runtime_image}" \
   --base-image-id "${runtime_image_id}" \
-  --source-sha256 4998b24f4f504aeeb9bf92769ec720e282f546e6726d89fdfd06c4efa8d17c10 \
-  --sparkcache-revision 20838ace3ebda570ca039cb7f1976c29da554b39 \
+  --source-sha256 f7c0565521fddeff7085e4cc08043cb8d1e2bde33abc67f83b8608a162d05b88 \
+  --sparkcache-revision 5d571018de5b63a9a90e5c11e6d6e86bbff4a957 \
   --output-image sparkring-glm53-sparkcache:b12x-kda-adaptive-mtp-0b67266a-arm64
 ```
 
@@ -48,10 +49,10 @@ Record immutable local identities:
 ```bash
 sparkcache_image='sparkring-glm53-sparkcache:b12x-kda-adaptive-mtp-0b67266a-arm64'
 sparkcache_image_id="$(docker image inspect --format '{{.Id}}' "${sparkcache_image}")"
-native_sha256="$(docker run --rm --entrypoint sha256sum "${sparkcache_image}" \
+cuda_placement_sha256="$(docker run --rm --entrypoint sha256sum "${sparkcache_image}" \
   /opt/sparkcache-src/sparkcache/native/build-cuda/libspark_cache_placement.so \
   | cut -d ' ' -f1)"
-test "${#native_sha256}" -eq 64
+test "${#cuda_placement_sha256}" -eq 64
 ```
 
 The runtime builder verifies the complete first-parent vLLM history from
@@ -71,7 +72,7 @@ python sparkring/scripts/prepare_glm53_b12x_kda_adaptive_mtp_profile.py \
   --image-id "${sparkcache_image_id}" \
   --parent-image "${runtime_image}" \
   --parent-image-id "${runtime_image_id}" \
-  --native-library-sha256 "${native_sha256}" \
+  --cuda-placement-library-sha256 "${cuda_placement_sha256}" \
   --profile-output profile.json \
   --site-output site.yaml
 ```

docs/GLM53_DFLASH7_PYTHON_OVERLAY_SPARKCACHE_TP4_QUICKSTART.md

@@ -0,0 +1,124 @@
+# Serve GLM-5.3 with external DFlash7 and the exact Python-overlay runtime
+
+Status: **implemented**, not qualified. The image builder, profile resolver,
+and four-rank dry-run contract pass without GPUs. No image digest from this
+path has completed TP4/DCP1 model loading, semantic generation, SparkCache
+store/restart/restore, or concurrency qualification.
+
+## Runtime contract
+
+| Role | Exact identity |
+|---|---|
+| vLLM native extensions and wheel metadata | `da4d7be6c97434f6942292ed8abbf4b32dc44355` |
+| vLLM Python source | `0b67266a0f37d6146a8403fb8482403c62f412d5`, tree `ba9484ccb33aa56e90ff2f447f15ca9b9da97639` |
+| B12X | `b1d541f9e71a35f030d45fae437630fff7507c2a`, tree `c69cdec1c59a08e8e0e549f930fa8abcfb5134ae` |
+| SparkCache reconstructed-page placement, canonical CUDA configuration, and bounded page-delta reads | `5ec6a9953ad5d39120298bbfc26e95a6fa4b1dc3`, tree `94c236b9dfbf5f70075eb47877fd9caaa5d8c249`, clean source SHA-256 `bc238f96e550c7ec27d4081dd1f2e741d404aaf5c8572d89ccc5e76812be4d63` |
+| DFlash draft-loader separation | Patch SHA-256 `39b567013ee7aed79f63200ed460129587933dc77fb430decdf19f78178de279`, postimage SHA-256 `98acbae2b3bb4482d83f9637c163ce7c92707ccdf6561b7e431f23337f151cf4` |
+| Unused DeepEP removal | Distribution `deep_ep==2.0.0+local`, removal receipt SHA-256 `65514f44829e7d176b0b2cacc9559ed22724e525b7041a8bcd4d2e02d1f372e3` |
+| Target | `local-inference-lab/GLM-5.3-Flash-NVFP4@520de24eabf507659eaef7c70f14fd584527facc` |
+| External draft | `incoai/GLM-5.3-Flash-DFlash2@dc77ff1c99eeb2df044ee3d4f0094eb033fee410`, BF16 weights SHA-256 `b33c03475ba7322cf398828f2d8d1be376df30dc05c6b40c28c8ea8da23e410b` |
+
+The serving contract uses seven speculative tokens, draft TP4, target FP8 KV,
+32 sequences, and 256-token vLLM blocks. SparkCache selects
+`tail-cow-v1`, which maps opaque GLM pages to the `page-tail-cow-v1`
+namespace, and uses the canonical CUDA restore keys.
+
+## Build the image
+
+Run on Linux ARM64 from a clean checkout containing this guide:
+
+```bash
+IMAGE='sparkring-glm53-sparkcache:dflash7-vllm-python-0b67266-native-da4d7be-b12x-b1d541f-arm64' \
+BUILD_RECEIPT="$PWD/glm53-dflash7-python-overlay-image-receipt.json" \
+bash runtime/glm53-flash-dflash7-python-overlay/build-image.sh
+```
+
+The builder verifies the public da4 image, the 31-file Python overlay, retained
+native ELFs and dispatch operators, B12X, SparkCache clean source, the CUDA
+placement library, four exact vLLM patches, and the eleven-file lease contract.
+The base-image inspection must identify exactly one installed distribution,
+`deep_ep==2.0.0+local`, as the owner of the `deep_ep` module. The derived image
+uninstalls that exact distribution and verifies that `deep_ep` is absent. The
+profiles use B12X kernels and PYNCCL collectives, so DeepEP is not a serving
+dependency. The builder does not push the image.
+
+Both profiles leave Torch thread selection to vLLM, select language-model-only
+serving, and disable the unsupported symmetric-memory and FlashInfer all-reduce
+candidates. PYNCCL remains bound to `/opt/sparkring/nccl/libnccl.so.2`, B12X
+remains the attention, MoE, and linear backend, and the all-reduce RMS fusion is
+disabled. ModelOpt experimental-quantization and FP8 KV accuracy warnings remain
+visible because they describe real format and accuracy limitations; the profiles
+do not suppress warnings.
+
+## Choose the target loader
+
+Two profiles share the same image and DFlash7 cache identity:
+
+| Profile | Status | Loader behavior |
+|---|---|---|
+| `glm53-flash-dflash7-python-overlay-safetensors-sparkcache-tp4-dcp1.example.json` | **implemented**, not qualified | Uses global safetensors for target and draft. This follows the qualified-compatible loader shape but still requires live qualification on the composed 0b image. |
+| `glm53-flash-dflash7-python-overlay-fastsafetensors-sparkcache-tp4-dcp1.example.json` | **implemented**, not qualified | Uses global fastsafetensors with queue size one for the target and `draft_load_config={"load_format":"safetensors"}` for DFlash. |
+
+The image applies an exact-input vLLM patch that passes
+`SpeculativeConfig.draft_load_config` to the DFlash model loader. The image
+receipt verifies patch SHA-256
+`39b567013ee7aed79f63200ed460129587933dc77fb430decdf19f78178de279` and
+postimage SHA-256
+`98acbae2b3bb4482d83f9637c163ce7c92707ccdf6561b7e431f23337f151cf4`.
+Both profiles remain unqualified until live four-rank gates pass.
+
+SparkCache PR #26 accepts the canonical CUDA keys used by both profiles. No
+local PR25 compatibility profile or legacy-key rewrite is part of this path.
+
+## Resolve the profile and inspect the plan
+
+Copy `scripts/config/glm53-flash-tp4-site.example.yaml`
+outside version control and replace every address, interface, SSH target,
+device, host path, and image identity. Select one profile template:
+
+```bash
+receipt="$PWD/glm53-dflash7-python-overlay-image-receipt.json"
+image='sparkring-glm53-sparkcache:dflash7-vllm-python-0b67266-native-da4d7be-b12x-b1d541f-arm64'
+profile_template='scripts/config/glm53-flash-dflash7-python-overlay-safetensors-sparkcache-tp4-dcp1.example.json'
+
+python scripts/prepare_glm53_dflash7_python_overlay_profile.py \
+  --profile-template "$profile_template" \
+  --site-template /path/to/resolved-glm53-site.yaml \
+  --image "$image" \
+  --image-id "$(jq -r .image_id "$receipt")" \
+  --cuda-placement-library-sha256 "$(jq -r .artifacts.sparkcache_cuda_placement_sha256 "$receipt")" \
+  --native-elf-manifest-sha256 "$(jq -r .runtime_contract.native_elf_manifest_sha256 "$receipt")" \
+  --native-dispatch-manifest-sha256 "$(jq -r .runtime_contract.native_dispatch_manifest_sha256 "$receipt")" \
+  --source-receipt-sha256 "$(jq -r .artifacts.source_receipt_sha256 "$receipt")" \
+  --profile-output /path/to/glm53-dflash7-profile.json \
+  --site-output /path/to/glm53-dflash7-site.yaml
+
+python scripts/sparkring_generic_launcher.py \
+  --site /path/to/glm53-dflash7-site.yaml \
+  --profile /path/to/glm53-dflash7-profile.json \
+  plan
+```
+
+`plan` is offline. Inspect every rank action before a lifecycle command.
+
+## Cache namespace impact
+
+The external DFlash weights SHA-256 is stored as
+`spark_cache_draft_checkpoint_sha256`. It cannot share entries with embedded
+MTP profiles. `tail-cow-v1` also separates these entries from snapshot-v1
+manifests. The two target-loader profiles share a namespace because loader
+choice does not change target or draft model state; each profile uses a
+different cache root and one-shot clear token while qualification is pending.
+
+[SparkCache pull request #30](https://github.com/FujitsuPolycom/sparkcache/pull/30)
+combines canonical CUDA configuration names, replacement of a partial terminal
+HMA page when an authenticated cache boundary falls inside that page, and an
+eight-worker reader for authenticated page-delta chunks. The reader preserves
+manifest descriptor order after concurrent reads.
+Moving from SparkCache commit
+`5d571018de5b63a9a90e5c11e6d6e86bbff4a957` to the pinned commit
+`5ec6a9953ad5d39120298bbfc26e95a6fa4b1dc3` does not change the namespace.
+Checkpoint identities, page-delta wire schemas, record vocabulary, digest
+salts, parallel geometry, vLLM patches, the lease contract, and the CUDA
+placement ABI are unchanged. Compatible `page-tail-cow-v1` entries remain
+eligible. Null-block publication failures remain unsupported.

docs/GLM53_E10536A_SPARKCACHE_TP4_QUICKSTART.md

@@ -19,8 +19,8 @@ pipelined shard loading without GPU Direct Storage. The queue can retain one
 additional shard-sized device buffer during model loading.
 
 All profiles use TP4/DCP1, 20 GiB FP8 KV per rank, a 524,288-token request
-limit, 32 sequences, native direct restore, two restore lanes, eight native
-I/O workers, two 256 MiB arenas, shared GPU-prefix leases, and one-shot cache
+limit, 32 sequences, SparkCache CUDA restore, two restore lanes, eight CUDA
+restore I/O workers, two 256 MiB placement arenas, shared GPU-prefix leases, and one-shot cache
 clearing.
 
 ## Build immutable inputs
@@ -57,10 +57,10 @@ Record the exact output identities:
 ```bash
 sparkcache_image='sparkring-glm53-sparkcache:e10536a-source-arm64'
 sparkcache_image_id="$(docker image inspect --format '{{.Id}}' "${sparkcache_image}")"
-native_sha256="$(docker run --rm --entrypoint sha256sum "${sparkcache_image}" \
+cuda_placement_sha256="$(docker run --rm --entrypoint sha256sum "${sparkcache_image}" \
   /opt/sparkcache-src/sparkcache/native/build-cuda/libspark_cache_placement.so \
   | cut -d ' ' -f1)"
-test "${#native_sha256}" -eq 64
+test "${#cuda_placement_sha256}" -eq 64
 ```
 
 ## Resolve a serving profile
@@ -88,7 +88,7 @@ python sparkring/scripts/prepare_glm53_e105_profile.py \
   --image-id "${sparkcache_image_id}" \
   --parent-image "${runtime_image}" \
   --parent-image-id "${runtime_image_id}" \
-  --native-library-sha256 "${native_sha256}" \
+  --cuda-placement-library-sha256 "${cuda_placement_sha256}" \
   --profile-output profile.json \
   --site-output site.yaml
 ```

docs/PREREQUISITES.md

@@ -6,6 +6,7 @@ hardware and operator conditions required by the
 [GLM-5.3 Flash SparkCache quickstart](GLM53_FLASH_DFLASH2_BF16_SPARKCACHE_TP4_QUICKSTART.md),
 [source-built GLM-5.3 e10536a quickstart](GLM53_E10536A_SPARKCACHE_TP4_QUICKSTART.md),
 [GLM-5.3 adaptive-MTP and live-tensor KDA quickstart](GLM53_B12X_KDA_ADAPTIVE_MTP_SPARKCACHE_TP4_QUICKSTART.md),
+[GLM-5.3 DFlash7 Python-overlay quickstart](GLM53_DFLASH7_PYTHON_OVERLAY_SPARKCACHE_TP4_QUICKSTART.md),
 [GLM-5.3 Flash cache-disabled quickstart](GLM53_FLASH_DFLASH2_BF16_TP4_QUICKSTART.md),
 [DeepSeek quickstart](DEEPSEEK_V4_FLASH_QUICKSTART.md),
 [Qwen3.8-27B pair quickstart](QWEN38_27B_EXL3_K5K6_PAIR_QUICKSTART.md), and

recipes/sparkcache/README.md

@@ -90,7 +90,7 @@ Limitations: The published receipts use a 4,096-token scheduler budget; other
 budgets are operator choices whose performance and capacity behavior is not
 recorded here, except that the GLM-5.3 Flash receipt uses 8,192. The DeepSeek
 receipts cover DCP1, not DCP2 or DCP4. These
-recipes disable streaming snapshots and native restore, so the receipts do not
+recipes disable streaming snapshots and SparkCache CUDA restore, so the receipts do not
 cover either mode. Other images, checkpoints, and cache geometries are also
 outside the recorded evidence. The GLM-5.3 Flash receipt covers an 8,192-token
 restored span and does not establish throughput neutrality or larger-span

recipes/sparkcache/deepseek-v4-flash-0731-tp2-dcp1.json

@@ -64,7 +64,7 @@
     "min_span_tokens": 256,
     "max_span_tokens": 1048576,
     "streaming_snapshots": false,
-    "native_restore": false,
+    "cuda_restore": false,
     "root_requirement": "Use a dedicated rank-local host directory mounted at the same container path on both ranks."
   },
   "evidence": {
@@ -78,7 +78,7 @@
       "The 17179869184-byte reservation, block size 256, and 4096-token scheduler budget carry forward unchanged, but their capacity with the larger context and sequence target must be measured.",
       "The complete checkpoint manifest is generated from the pinned revision and must reproduce the recorded digest. A different digest is a different composition and does not reproduce this qualification.",
       "The published receipt covers DeepSeek DCP1, not DCP2 or DCP4.",
-      "This recipe disables streaming snapshots and native restore; the published receipt does not cover either mode."
+      "This recipe disables streaming snapshots and SparkCache CUDA restore; the published receipt does not cover either mode."
     ],
     "record": "recipes/sparkcache/README.md",
     "artifact_scope": "The exact SparkCache wheel, runtime image, checkpoint identity, topology, and historical max_model_len=131072, max_num_seqs=6 serving values stated in evidence.conditions.",

recipes/sparkcache/deepseek-v4-flash-0731-tp4-dcp1.json

@@ -54,7 +54,7 @@
     "min_span_tokens": 256,
     "max_span_tokens": 1048576,
     "streaming_snapshots": false,
-    "native_restore": false,
+    "cuda_restore": false,
     "root_requirement": "Use one dedicated rank-local host directory per physical rank, mounted at the same container path on all four ranks."
   },
   "evidence": {
@@ -68,7 +68,7 @@
       "The 34359738368-byte reservation, 32-sequence limit, block size 256, and 4096-token scheduler budget carry forward unchanged, but capacity at the larger context must be measured.",
       "The complete checkpoint manifest is generated from the pinned revision and must reproduce the recorded digest. A different digest is a different composition and does not reproduce this qualification.",
       "The published receipt covers DeepSeek DCP1, not DCP2 or DCP4.",
-      "This recipe disables streaming snapshots and native restore; the published receipt does not cover either mode."
+      "This recipe disables streaming snapshots and SparkCache CUDA restore; the published receipt does not cover either mode."
     ],
     "record": "recipes/sparkcache/README.md",
     "artifact_scope": "The exact SparkCache wheel, runtime image, checkpoint identity, topology, and historical max_model_len=524288 serving values stated in evidence.conditions.",

recipes/sparkcache/glm52-exl3-r7-3.5bpw-tp4-dcp4.json

@@ -57,7 +57,7 @@
     "min_span_tokens": 256,
     "max_span_tokens": 1048576,
     "streaming_snapshots": false,
-    "native_restore": false,
+    "cuda_restore": false,
     "root_requirement": "Use one dedicated rank-local host directory per physical rank, mounted at the same container path on all four ranks."
   },
   "restart_contract": {
@@ -75,7 +75,7 @@
       "The qualified receipt covers max_model_len=262144; the current max_model_len=1048576 target requires a larger CKV-gather workspace and live memory validation.",
       "The qualified operator image is identified by local image ID but has no published immutable image reference. A clean-checkout rebuild has implemented status until it passes the promotion checklist.",
       "The qualified lane requires preserving and regenerating the 40-query-row exact-state receipt, whose path contains q40-exact-state-serving-v1, before every coordinated restart.",
-      "This recipe disables streaming snapshots and native restore; the published receipt does not cover either mode.",
+      "This recipe disables streaming snapshots and SparkCache CUDA restore; the published receipt does not cover either mode.",
       "Full GLM reasoning-trace equality is an inconclusive diagnostic under this runtime: repeated fixed-seed requests produced different non-empty reasoning bodies with identical final content and finish reason stop."
     ],
     "record": "https://github.com/FujitsuPolycom/sparkcache/blob/999bc13d6c0b52a6cee3c90487ac9b8643ba3f99/GLM52_A2_LIVE_VALIDATION.md",

recipes/sparkcache/glm53-flash-nvfp4-dflash2-bf16-tp4-dcp1.json

@@ -83,7 +83,7 @@
     "min_span_tokens": 4096,
     "max_span_tokens": 524288,
     "streaming_snapshots": false,
-    "native_restore": false,
+    "cuda_restore": false,
     "root_requirement": "Use one dedicated rank-local host directory per physical rank, mounted at /cache/jit on every rank."
   },
   "evidence": {
@@ -97,7 +97,7 @@
       "A rebuilt image has implemented status until its immutable digest passes the same live qualification.",
       "The first request after API startup recomputed because the scheduler had not received a complete four-rank inventory checkpoint; the following identical request formed quorum and restored.",
       "The receipt does not establish throughput neutrality or restore performance for spans larger than 8192 tokens.",
-      "Streaming snapshots, native direct restore, MTP drafting, and other DFlash checkpoints are unsupported by this evidence.",
+      "Streaming snapshots, SparkCache CUDA restore, MTP drafting, and other DFlash checkpoints are unsupported by this evidence.",
       "The source-built runtime uses the Triton KDA prefill backend and stock safetensors loading; FlashKDA prefill and InstantTensor checkpoint loading are unsupported by this image.",
       "The optional deep_ep import emits a duplicate-NCCL warning. vLLM selects the source-built NCCL library through VLLM_NCCL_SO_PATH and serving proceeds.",
       "The target repository does not record its base-checkpoint revision."