Bind GLM runtimes to recurrent tail publication

.github/workflows/ci.yml

@@ -53,7 +53,7 @@ jobs:
             --index-url https://download.pytorch.org/whl/cpu \
             "torch==${TORCH_VERSION}"
       - name: Test maintained Python trees
-        run: python -m pytest spark_transport runtime/exl3-r7 runtime/glm53-flash runtime/deepseek0731-gb10 runtime/qwen38 runtime/test_public_overlay.py performance/harnesses scripts -q -rs
+        run: python -m pytest spark_transport runtime/exl3-r7 runtime/glm53-flash runtime/glm53-flash-adaptive-mtp-python-overlay runtime/glm53-flash-dflash7-python-overlay runtime/deepseek0731-gb10 runtime/qwen38 runtime/test_public_overlay.py performance/harnesses scripts -q -rs
 
   docs-links:
     name: docs links

AGENTS.md

@@ -104,7 +104,7 @@ test imports torch:
 python -m pip install -r requirements-dev.txt
 python -m pip install --index-url https://download.pytorch.org/whl/cpu "torch==2.11.0"
 ruff check --select E,F,W --ignore E501 spark_transport runtime scripts performance
-python -m pytest spark_transport runtime/exl3-r7 runtime/glm53-flash runtime/deepseek0731-gb10 runtime/qwen38 runtime/test_public_overlay.py performance/harnesses scripts -q -rs
+python -m pytest spark_transport runtime/exl3-r7 runtime/glm53-flash runtime/glm53-flash-adaptive-mtp-python-overlay runtime/glm53-flash-dflash7-python-overlay runtime/deepseek0731-gb10 runtime/qwen38 runtime/test_public_overlay.py performance/harnesses scripts -q -rs
 ```
 
 The test suite is CPU-only contract coverage. It does not validate CUDA,

docs/GLM53_B12X_KDA_ADAPTIVE_MTP_SPARKCACHE_TP4_QUICKSTART.md

@@ -1,9 +1,10 @@
 # Serve GLM-5.3 with adaptive MTP, live-tensor B12X KDA, and SparkCache
 
-Status: **implemented, not qualified**. This guide builds vLLM commit
-`0b67266a0f37d6146a8403fb8482403c62f412d5` and the SparkCache overlay from
-commit `5ec6a9953ad5d39120298bbfc26e95a6fa4b1dc3`, Git tree
-`94c236b9dfbf5f70075eb47877fd9caaa5d8c249`, for four DGX Spark systems at
+Status: **implemented, not qualified**. This guide retains vLLM native
+extensions from `da4d7be6c97434f6942292ed8abbf4b32dc44355`, overlays Python
+source `0b67266a0f37d6146a8403fb8482403c62f412d5`, and installs SparkCache
+commit `c56f77f97b3da907d32e888d82046359a62f0f88`, Git tree
+`deac36758f86695cd13f07b2870c2e49842aed9c`, for four DGX Spark systems at
 TP4/DCP1. The adaptive-MTP composition has GPU-free contract coverage but no
 four-rank persistent-restore or performance qualification.
 
@@ -13,69 +14,59 @@ TP4 makes the pinned vLLM loader select `nogds=True`, so model loading uses
 pipelined host I/O without GPU Direct Storage.
 
 The profile reserves 20 GiB of FP8 KV per rank and enables SparkCache CUDA
-restore, tail-only publication, shared restore trunks, and bounded shared GPU
-prefix leases. Image construction and distribution do not require stopping an
-existing service. Do not run the launch command until all four ranks have the
-same verified image ID.
+restore, tail-only copy-on-write publication, shared-segment restore, and
+bounded shared GPU prefix leases. The vLLM overlay emits only hash-proven
+recurrent replay boundaries; SparkCache keeps publication pending while a
+request has no hand-off and rejects incomplete, contradictory, or changed
+evidence. Image construction and distribution
+do not require stopping an existing service. Do not run the launch command
+until all four ranks have the same verified image ID.
 
-## Build the runtime and SparkCache overlay
+## Build the exact Python-overlay image
 
-Use Linux ARM64 with Docker BuildKit and at least 250 GiB of free local
-storage. Clone both repositories beside each other:
+Use Linux ARM64 with Docker BuildKit and enough space for the exact source
+trees, B12X wheel, SparkCache CUDA placement library, and derived image:
 
 ```bash
 git clone https://github.com/FujitsuPolycom/sparkring.git sparkring
 git -C sparkring checkout --detach <revision-containing-this-guide>
-git clone https://github.com/FujitsuPolycom/sparkcache.git sparkcache
-git -C sparkcache checkout --detach 5ec6a9953ad5d39120298bbfc26e95a6fa4b1dc3
-
-IMAGE='sparkring-glm53-runtime:b12x-kda-adaptive-mtp-0b67266a-arm64' \
-BUILD_RECEIPT="$PWD/glm53-b12x-kda-adaptive-mtp-runtime-receipt.json" \
-bash sparkring/runtime/glm53-flash-b12x-kda-adaptive-mtp/build-image.sh
-
-runtime_image='sparkring-glm53-runtime:b12x-kda-adaptive-mtp-0b67266a-arm64'
-runtime_image_id="$(docker image inspect --format '{{.Id}}' "${runtime_image}")"
-python sparkcache/deploy/glm53_flash/build_image.py \
-  --repository "$PWD/sparkcache" \
-  --containerfile deploy/glm53_flash/Containerfile.b12x-kda-adaptive-mtp \
-  --base-image "${runtime_image}" \
-  --base-image-id "${runtime_image_id}" \
-  --source-sha256 bc238f96e550c7ec27d4081dd1f2e741d404aaf5c8572d89ccc5e76812be4d63 \
-  --sparkcache-revision 5ec6a9953ad5d39120298bbfc26e95a6fa4b1dc3 \
-  --output-image sparkring-glm53-sparkcache:b12x-kda-adaptive-mtp-0b67266a-arm64
+cd sparkring
+IMAGE='sparkring-glm53-sparkcache:vllm-python-0b67266-native-da4d7be-b12x-b1d541f-arm64' \
+BUILD_RECEIPT="$PWD/glm53-public-python-overlay-image-receipt.json" \
+bash runtime/glm53-flash-adaptive-mtp-python-overlay/build-image.sh
 ```
 
-Record immutable local identities:
-
-```bash
-sparkcache_image='sparkring-glm53-sparkcache:b12x-kda-adaptive-mtp-0b67266a-arm64'
-sparkcache_image_id="$(docker image inspect --format '{{.Id}}' "${sparkcache_image}")"
-cuda_placement_sha256="$(docker run --rm --entrypoint sha256sum "${sparkcache_image}" \
-  /opt/sparkcache-src/sparkcache/native/build-cuda/libspark_cache_placement.so \
-  | cut -d ' ' -f1)"
-test "${#cuda_placement_sha256}" -eq 64
-```
-
-The runtime builder verifies the complete first-parent vLLM history from
-`da4d7be` through adaptive MTP and the three live-tensor B12X KDA commits. The
-SparkCache build verifies LF Linux preimages, four exact patches, and eleven
-postimage source files. The pinned SparkCache source accepts the canonical
-`spark_cache_cuda_*` keys in the profile directly; no legacy-key translation
-is part of this composition.
+The builder verifies the 31-file Python overlay, retained native ELF and
+dispatch manifests, B12X `b1d541f`, SparkCache clean source SHA-256
+`788686e858ba4af01f535e95122c7650f412fddc40cd221a0924f4ce2b32ff98`,
+four SparkCache patches, recurrent-boundary producer patch SHA-256
+`5a6561a5bbab990dcd03bfd6a485ea26c3b5a578c2fd61b76305767b16dbfba0`,
+and lease contract SHA-256
+`f36ed14eaf1f97a5dffa94bda8151b1e0fa182afc0d121b757b70bebc6a43811`.
+The receipt binds these sources to the local image ID. The builder does not
+push the image or contact serving hosts.
+
+`runtime/glm53-flash-b12x-kda-adaptive-mtp/` remains an exact full-source
+builder for its older SparkCache contract. It does not apply the recurrent
+producer after SparkCache's vLLM patches, so it is unsupported with SparkCache
+`c56f77f` and must not be substituted in this guide.
 
 ## Resolve the TP4 profile
 
 ```bash
-profile_template='sparkring/scripts/config/glm53-flash-b12x-kda-mtp5-adaptive-fastsafetensors-sparkcache-tp4-dcp1.example.json'
-site_template='sparkring/scripts/config/glm53-flash-b12x-kda-adaptive-mtp-tp4-site.example.yaml'
-python sparkring/scripts/prepare_glm53_b12x_kda_adaptive_mtp_profile.py \
+receipt="$PWD/glm53-public-python-overlay-image-receipt.json"
+image='sparkring-glm53-sparkcache:vllm-python-0b67266-native-da4d7be-b12x-b1d541f-arm64'
+profile_template='scripts/config/glm53-flash-public-python-overlay-mtp5-adaptive-fastsafetensors-sparkcache-tp4-dcp1.example.json'
+site_template='scripts/config/glm53-flash-b12x-kda-adaptive-mtp-tp4-site.example.yaml'
+python scripts/prepare_glm53_public_python_overlay_profile.py \
   --profile-template "${profile_template}" \
   --site-template "${site_template}" \
-  --image "${sparkcache_image}" \
-  --image-id "${sparkcache_image_id}" \
-  --parent-image "${runtime_image}" \
-  --parent-image-id "${runtime_image_id}" \
-  --cuda-placement-library-sha256 "${cuda_placement_sha256}" \
+  --image "${image}" \
+  --image-id "$(jq -r .image_id "${receipt}")" \
+  --cuda-placement-library-sha256 "$(jq -r .artifacts.sparkcache_cuda_placement_sha256 "${receipt}")" \
+  --native-elf-manifest-sha256 "$(jq -r .runtime_contract.native_elf_manifest_sha256 "${receipt}")" \
+  --native-dispatch-manifest-sha256 "$(jq -r .runtime_contract.native_dispatch_manifest_sha256 "${receipt}")" \
+  --source-receipt-sha256 "$(jq -r .artifacts.source_receipt_sha256 "${receipt}")" \
   --profile-output profile.json \
   --site-output site.yaml
 ```
@@ -86,7 +77,9 @@ SparkCache directory. Do not change MTP depths, the observation window,
 loader queue, source identities, or attestation command.
 
 The one-shot clear token is recorded only after a successful SparkCache-owned
-cache removal. Restarting this unchanged profile does not clear again.
+cache removal. Restarting this unchanged profile does not clear again. The
+resolved profile requires the recurrent-boundary patch label before any
+container starts.
 
 `--prefill-schedule-interval` is not part of this implemented profile. Test
 interval `8` as a separate research-only profile so its mixed prefill/decode
@@ -95,33 +88,34 @@ tradeoff cannot be confused with adaptive-MTP or SparkCache results.
 ## Verify and launch
 
 ```bash
-python sparkring/scripts/preflight.py \
+python scripts/preflight.py \
   --site site.yaml --strict-placeholders --json preflight.json
-python sparkring/scripts/sparkring_generic_launcher.py \
+python scripts/sparkring_generic_launcher.py \
   --site site.yaml --profile profile.json plan > start-plan.json
-python sparkring/scripts/sparkring_generic_launcher.py \
+python scripts/sparkring_generic_launcher.py \
   --site site.yaml --profile profile.json \
   --execute \
-  --confirmation START_GLM53_FLASH_MTP5_ADAPTIVE_FASTSAFETENSORS_TP4 \
+  --confirmation START_GLM53_FLASH_PUBLIC_PYTHON_OVERLAY_MTP5_ADAPTIVE_FASTSAFETENSORS_TP4 \
   start
 ```
 
 The final command changes the four-rank serving deployment. Tail rank zero:
 
 ```bash
 ssh operator@rank0.example.net \
-  'docker logs --follow --tail 120 glm53-flash-b12x-kda-mtp5-adaptive-fastsafetensors-sparkcache-tp4-r0 2>&1'
+  'docker logs --follow --tail 120 glm53-flash-public-python-overlay-mtp5-adaptive-fastsafetensors-sparkcache-tp4-r0 2>&1'
 ```
 
 Wait for health, then run the exact semantic request:
 
 ```bash
 api_endpoint='http://rank0.example.net:8015'
-served_model='glm-5.3-flash-nvfp4-b12x-kda-mtp5-adaptive-fastsafetensors-0b67266a-tp4'
+served_model='glm-5.3-flash-nvfp4-python-overlay-0b67266-on-da4d7be-b12x-b1d541f-mtp5-adaptive-tp4'
 until curl --fail --silent "${api_endpoint}/health" >/dev/null; do sleep 5; done
-python sparkcache/deploy/glm53_flash/qualification_request.py \
-  --endpoint "${api_endpoint}" --model "${served_model}" \
-  --kind semantic --output semantic.json
+curl --fail --silent --show-error "${api_endpoint}/v1/completions" \
+  -H 'Content-Type: application/json' \
+  -d "{\"model\":\"${served_model}\",\"prompt\":\"The capital of France is\",\"max_tokens\":16,\"temperature\":0}" \
+  > semantic.json
 ```
 
 Construction support does not prove four-rank serving. Qualification requires
@@ -133,10 +127,13 @@ NCCL error, or traceback.
 
 The overlay does not change SparkCache wire fields, digest salts, 256-token
 geometry, or stored object schemas. Its embedded-MTP digest is SHA-256 over
-`glm53-embedded-mtp-runtime-v1`, the target identity, the full vLLM commit,
-maximum depth five, and `adaptive:3:32`, separated by zero bytes.
-
-Including the vLLM revision gives this runtime a distinct draft-state cache
-identity from the e105 adaptive-MTP profile. Stored entries therefore
-recompute instead of crossing the KDA source boundary without byte-equivalence
-evidence.
+`glm53-embedded-mtp-composed-runtime-v1`, the target identity, the overlaid
+vLLM Python commit, retained native vLLM commit, B12X commit, maximum depth
+five, and `adaptive:3:32`, separated by zero bytes.
+
+Including both retained-native and overlaid-Python revisions gives this runtime
+a distinct draft-state cache identity from the e105 adaptive-MTP and older
+full-source profiles. Stored entries therefore recompute instead of crossing a
+runtime boundary without byte-equivalence evidence. SparkCache `49c517e` does
+not change wire fields, digest salts, or 256-token geometry; the new lease
+contract accepts the producer postimages needed to prove recurrent publication.

docs/GLM53_DFLASH7_PYTHON_OVERLAY_SPARKCACHE_TP4_QUICKSTART.md

@@ -1,14 +1,17 @@
 # Serve GLM-5.3 with external DFlash7 and the exact Python-overlay runtime
 
-Status: **implemented**. The image builder, profile resolver, and four-rank
-dry-run contract pass without GPUs. Local image ID
+Status: **implemented, not qualified** for the current build contract. The
+image builder, profile resolver, and four-rank dry-run contract pass without
+GPUs. Historical local image ID
 `sha256:eef863d8bc578815a80b0e2d9f0d745102b6363415225101fd92171a2e5a55cb`
 is **qualified** only for the TP4/DCP1 startup, health, semantic generation,
 arbitrary page-boundary replay, and 131,072- and 262,144-token restore cases
 in the
 [bounded validation record](../performance/records/glm53-flash/dflash7-python-overlay-pr30-live-validation.md).
-The exact image has no retained C2/C8/C16 or DFlash response-quality evidence.
-A rebuilt image has a different identity and requires its own live checks.
+That historical image used SparkCache `5ec6a9953ad5d39120298bbfc26e95a6fa4b1dc3`;
+it is not an artifact of the current contract below. It has no retained
+C2/C8/C16 or DFlash response-quality evidence. A current build has a different
+identity and requires its own live checks.
 
 ## Runtime contract
 
@@ -17,7 +20,8 @@ A rebuilt image has a different identity and requires its own live checks.
 | vLLM native extensions and wheel metadata | `da4d7be6c97434f6942292ed8abbf4b32dc44355` |
 | vLLM Python source | `0b67266a0f37d6146a8403fb8482403c62f412d5`, tree `ba9484ccb33aa56e90ff2f447f15ca9b9da97639` |
 | B12X | `b1d541f9e71a35f030d45fae437630fff7507c2a`, tree `c69cdec1c59a08e8e0e549f930fa8abcfb5134ae` |
-| SparkCache reconstructed-page placement, canonical CUDA configuration, and bounded page-delta reads | `5ec6a9953ad5d39120298bbfc26e95a6fa4b1dc3`, tree `94c236b9dfbf5f70075eb47877fd9caaa5d8c249`, clean source SHA-256 `bc238f96e550c7ec27d4081dd1f2e741d404aaf5c8572d89ccc5e76812be4d63` |
+| SparkCache shared-segment restore, tail-only copy-on-write publication, canonical CUDA configuration, and bounded page-delta reads | `c56f77f97b3da907d32e888d82046359a62f0f88`, tree `deac36758f86695cd13f07b2870c2e49842aed9c`, clean source SHA-256 `788686e858ba4af01f535e95122c7650f412fddc40cd221a0924f4ce2b32ff98` |
+| Recurrent replay-boundary producer | Patch SHA-256 `5a6561a5bbab990dcd03bfd6a485ea26c3b5a578c2fd61b76305767b16dbfba0`; produces the four postimages accepted by SparkCache lease contract SHA-256 `f36ed14eaf1f97a5dffa94bda8151b1e0fa182afc0d121b757b70bebc6a43811` |
 | DFlash draft-loader separation | Patch SHA-256 `39b567013ee7aed79f63200ed460129587933dc77fb430decdf19f78178de279`, postimage SHA-256 `98acbae2b3bb4482d83f9637c163ce7c92707ccdf6561b7e431f23337f151cf4` |
 | Unused DeepEP removal | Distribution `deep_ep==2.0.0+local`, removal receipt SHA-256 `65514f44829e7d176b0b2cacc9559ed22724e525b7041a8bcd4d2e02d1f372e3` |
 | Target | `local-inference-lab/GLM-5.3-Flash-NVFP4@520de24eabf507659eaef7c70f14fd584527facc` |
@@ -40,7 +44,9 @@ bash runtime/glm53-flash-dflash7-python-overlay/build-image.sh
 
 The builder verifies the public da4 image, the 31-file Python overlay, retained
 native ELFs and dispatch operators, B12X, SparkCache clean source, the CUDA
-placement library, four exact vLLM patches, and the eleven-file lease contract.
+placement library, four SparkCache vLLM patches, the recurrent-boundary
+producer patch, and the eleven-file lease contract. The resulting profile also
+requires the producer patch label before starting any container.
 The base-image inspection must identify exactly one installed distribution,
 `deep_ep==2.0.0+local`, as the owner of the `deep_ep` module. The derived image
 uninstalls that exact distribution and verifies that `deep_ep` is absent. The
@@ -62,7 +68,7 @@ Two profiles share the same image and DFlash7 cache identity:
 | Profile | Status | Loader behavior |
 |---|---|---|
 | `glm53-flash-dflash7-python-overlay-safetensors-sparkcache-tp4-dcp1.example.json` | **implemented**, not qualified | Uses global safetensors for target and draft. This follows the qualified-compatible loader shape but still requires live qualification on the composed 0b image. |
-| `glm53-flash-dflash7-python-overlay-fastsafetensors-sparkcache-tp4-dcp1.example.json` | **qualified** only for the recorded image and bounded cases | Uses global fastsafetensors with queue size one for the target and `draft_load_config={"load_format":"safetensors"}` for DFlash. |
+| `glm53-flash-dflash7-python-overlay-fastsafetensors-sparkcache-tp4-dcp1.example.json` | **implemented**, not qualified for the current build | Uses global fastsafetensors with queue size one for the target and `draft_load_config={"load_format":"safetensors"}` for DFlash. The historical receipt remains evidence only for its recorded image. |
 
 The image applies an exact-input vLLM patch that passes
 `SpeculativeConfig.draft_load_config` to the DFlash model loader. The image
@@ -74,9 +80,11 @@ The all-safetensors profile remains unqualified. The fastsafetensors result
 belongs only to the image ID and cases named above; it does not transfer to a
 rebuild.
 
-SparkCache commit `5ec6a9953ad5d39120298bbfc26e95a6fa4b1dc3`
-accepts the canonical CUDA keys used by both profiles. No legacy-key rewrite
-is part of this path.
+SparkCache commit `c56f77f97b3da907d32e888d82046359a62f0f88`
+accepts the canonical CUDA keys used by both profiles. It consumes the
+hash-proven recurrent hand-off emitted by this exact vLLM overlay and cancels
+publication when the metadata is absent, incomplete, or contradictory.
+No legacy-key rewrite is part of this path.
 
 ## Resolve the profile and inspect the plan
 
@@ -156,7 +164,7 @@ content, writes the token's completion marker only after successful removal,
 and treats later starts with the same token as no-ops. Change the token only
 when another intentional cache reset is required.
 
-`--prefill-schedule-interval` is not part of the qualified DFlash7 profile.
+`--prefill-schedule-interval` is not part of the current DFlash7 profiles.
 Test interval `8` in a separate research-only profile so its mixed
 prefill/decode tradeoff is measured independently.
 
@@ -171,13 +179,11 @@ different cache root and one-shot clear token so loader observations remain
 isolated.
 
 The pinned SparkCache source combines canonical CUDA configuration names,
-replacement of a partial terminal HMA page when an authenticated cache
-boundary falls inside that page, and an eight-worker reader for authenticated
-page-delta chunks. The reader preserves manifest descriptor order after
-concurrent reads. Moving from SparkCache commit
-`5d571018de5b63a9a90e5c11e6d6e86bbff4a957` to
-`5ec6a9953ad5d39120298bbfc26e95a6fa4b1dc3` does not change the namespace.
-Checkpoint identities, page-delta wire schemas, record vocabulary, digest
-salts, parallel geometry, vLLM patches, the lease contract, and the CUDA
-placement ABI are unchanged. Compatible `page-tail-cow-v1` entries remain
-eligible. Null-block publication failures remain unsupported.
+authenticated shared-segment restore, an eight-worker ordered page-delta
+reader, and tail-only copy-on-write publication. Cache identities, digest
+salts, 256-token chunk geometry, page-delta wire bytes, and the CUDA placement
+ABI are unchanged. The source change does not change the namespace, so
+compatible `page-tail-cow-v1` entries remain eligible.
+The vLLM lease-contract bytes do change to accept the recurrent-boundary
+postimages. Missing or malformed boundary evidence is a cache miss and
+recomputation, never an unverified publication.