diff --git a/GLM53_NATIVE_RESTORE_PERFORMANCE_VALIDATION.md b/GLM53_NATIVE_RESTORE_PERFORMANCE_VALIDATION.md deleted file mode 100644 index c55cab1..0000000 --- a/GLM53_NATIVE_RESTORE_PERFORMANCE_VALIDATION.md +++ /dev/null @@ -1,225 +0,0 @@ -# GLM-5.3 SparkCache CUDA restore and shared-prefix validation - -The repository path retains its compatibility filename so existing evidence -links remain valid. In this record, SparkCache CUDA restore and SparkCache CUDA -placement are the canonical capability and data-movement terms. - -Date: 2026-08-29 - -## Status - -SparkCache direct CUDA restore, verified multi-group recovery, bounded shared GPU-prefix -reuse, C2/C8/C16 completion, shared-trunk C16 completion, and continued -generation are **qualified** for the exact GLM-5.3 Flash TP4/DCP1 runtime -identified below. - -The qualification does not cover unrelated-cold C16 behavior, interference -with unrelated decode traffic, C24/C32 cohorts, more than sixteen waiting -followers, another checkpoint, another topology, or another vLLM source tree. -Those cases remain outside the qualification evidence. Tail-only publication -and page-semantic GLM prefix aliases are **unsupported**. - -The committed semantic receipts used a suffix-only predicate: content ending -in `SPARKCACHE_GLM53_OK` was recorded as `semantic_match: true`. They prove -continued generation and the expected suffix, but they do not prove that the -visible response contained exactly `SPARKCACHE_GLM53_OK`. Exact-output semantic -qualification requires a receipt produced by the equality validator in -`deploy/glm53_flash/qualification_request.py`. - -## Qualified runtime identity - -| Attribute | Value | -|---|---| -| SparkCache source revision | `2b86fb9d02fa3595cca5caa864b81aedce44b8bb` | -| SparkCache source-tree SHA-256 | `b3e84d220e215bdad99455a7eefb431b9aea248e0edb6ff417319c420433f55a` | -| vLLM source revision | `local-inference-lab/vllm@da4d7be6c97434f6942292ed8abbf4b32dc44355` | -| Serving topology | GLM-5.3 Flash, TP4/DCP1, one rank on each of `spark-r0` through `spark-r3` | -| Scheduler capacity | `--max-num-seqs 32` | -| Restore concurrency | Two host restore workers and two SparkCache CUDA placement lanes per rank | -| Native staging | Two 256 MiB mapped-host arenas per rank | -| Persistent prefix | 131,072 tokens and 813,068,464 encoded bytes per rank | -| Runtime receipt | `evidence/glm53-flash-dflash7-bf16/hotlease-2b86fb9-runtime.json` | - -The runtime receipt records one immutable image ID per rank, native library -SHA-256 `683cb9e0420da9c68e3263093077fdbcaa400913ff0fb1d18639771213220605`, -scheduler SHA-256 -`4f8793c4ac4bf356a89c829b6e75b189e6bc4a74c97135208952d0bad1678f15`, -and KV-cache-manager SHA-256 -`ee03dc9ce2b720c0be6e9f572d23580ba96eff68fe3406250557e83071654af0`. - -The direct page-placement implementation is identified by these commits: - -| Responsibility | Revision | -|---|---| -| Restore phase timing | `175f9401984a03744d7fe1a985d7c2ef6035f949` | -| SparkCache CUDA page placement | `71f367be07788d611698a251fe866d678b0034ae` | -| Multi-slab restore and exact-prefix discovery | `8e7f5fc62fd4fffdd661aca9ea634cf130c45d1a` | -| Direct pipelined slab restore | `94c44930a13df5c668d777e0270e7d8203069d7c` | -| Authenticated span-table bound | `9dbf73c0caab89b24346567e2769752ac746e114` | - -The 131,072-token single-request measurement used source-tree SHA-256 -`368cc18dbccc262a1f2a1f1eef5aced90690031abd1f2fedf3d192e60a67012b` -and parent/runtime image -`sha256:7c007cf673c35f5818da7fea8faa343304baed00f489efdcbd027d6616b8a290`. -The shared-prefix qualification used source revision `2b86fb9d...` and -source-tree SHA-256 `b3e84d...` identified in the table above. - -## Implemented restore path - -SparkCache CUDA restore reads immutable `.spcc` objects directly into alternating -mapped-host arenas with `pread`, hashes every complete file in place, validates -its authenticated extent table, and submits only validated spans to the CUDA -page-placement kernel. Read work and CUDA submission overlap across slabs. - -This path avoids Python `ContextChunk` reconstruction and an 813 MiB -intermediate join/copy. The adapter accepts at most 4,096 authenticated spans, -matching the validated native ABI. These changes do not alter `CacheIdentity`, -digest values, 256-token logical geometry, or the on-disk exact-manifest and -chunk formats. - -## Practical restore gains - -### One 131,072-token prefix - -Each rank restored 813,068,464 bytes through four slabs: - -| Rank | Cache service | Read and hash | Native submit | CUDA finish | -|---:|---:|---:|---:|---:| -| 0 | 141.9 ms | 77.9 ms | 25.4 ms | 3.4 ms | -| 1 | 131.3 ms | 84.6 ms | 13.4 ms | 3.4 ms | -| 2 | 139.0 ms | 92.7 ms | 14.1 ms | 3.4 ms | -| 3 | 250.1 ms | 140.7 ms | 20.1 ms | 3.4 ms | - -The Python reconstruction pipeline measured 1.29--1.46 seconds per rank for -the same stored prefix. Native cache service therefore reduced the slowest-rank -time to 250.1 ms. End-to-end client latency was 0.907 seconds, including -scheduler work, live-token execution, and DFlash generation. A separate -historical canary found the expected marker suffix, and HTTP health remained -200. - -### Eight concurrent 16K prefixes - -The Python/Torch placement path produced 1.54--1.57 second submission spikes; -eight clients completed in 9.45--10.64 seconds. SparkCache CUDA placement submitted in -6--15 ms, and two restore lanes completed eight clients in approximately -1.2--2.1 seconds. This diagnostic isolates page placement as the dominant -serialized cost in that workload; it is not a separate deployment -qualification. - -## Concurrency and shared GPU blocks - -The independent-restore baseline issued a complete external restore for every -request: two per rank for C2, eight for C8, and sixteen for C16. - -| Cohort | Client min | Client p50 | Client max | -|---:|---:|---:|---:| -| C2 | 1.282 s | 1.282 s | 1.377 s | -| C8 | 0.947 s | 2.017 s | 3.129 s | -| C16 | 1.063 s | 3.363 s | 5.335 s | - -Bounded shared-prefix leases changed the rank-local work from sixteen complete -813 MiB restores to one. The standard chat C16 measurement completed every -request at 2.980 seconds p50 and 5.064 seconds maximum. This measurement -includes sixteen large prompt-tokenization operations. - -Pretokenized requests preserve the exact chat-template token sequence while -moving tokenizer work outside the timed interval: - -| Cohort | External restores per rank | Client min | Client p50 | Client max | -|---:|---:|---:|---:|---:| -| C2 | 1 | 0.541 s | 0.541 s | 0.737 s | -| C8, retained lease | 0 | 0.407 s | 1.211 s | 1.218 s | -| C16 | 1 | 0.735 s | 2.698 s | 2.701 s | -| C16, common trunk and distinct tails | 1 | 2.407 s | 3.894 s | 3.896 s | - -The pretokenized C16 restore used 104.2--165.3 ms of cache service per rank -and at most 3.2 ms of restore-queue wait. Every request completed, queues -drained to zero, no engine exited, and the historical post-run canary found the -expected marker suffix. - -Qualified receipts: - -- `evidence/glm53-flash-dflash7-bf16/hotlease-2b86fb9-128k-c2-pretokenized.json`; -- `evidence/glm53-flash-dflash7-bf16/hotlease-2b86fb9-128k-c8-pretokenized.json`; -- `evidence/glm53-flash-dflash7-bf16/hotlease-2b86fb9-128k-c16.json`; -- `evidence/glm53-flash-dflash7-bf16/hotlease-2b86fb9-128k-c16-pretokenized.json`; -- `evidence/glm53-flash-dflash7-bf16/hotlease-2b86fb9-128k-c16-shared-trunk-pretokenized.json`; -- `evidence/glm53-flash-dflash7-bf16/post-hotlease-2b86fb9-semantic.json`. - -Baseline receipts: - -- `evidence/glm53-flash-dflash7-bf16/native-128k-c2-b00c6d4.json`; -- `evidence/glm53-flash-dflash7-bf16/native-128k-c8-hot-b00c6d4.json`; -- `evidence/glm53-flash-dflash7-bf16/native-128k-c16-hot-b00c6d4.json`; -- `evidence/glm53-flash-dflash7-bf16/post-b00c6d4-semantic.json`. - -## Shared-prefix ownership - -One request leads restoration for a persistent digest. Followers wait without -allocating another external restore. A lease becomes visible only after every -worker reports successful restoration and the scheduler normalizes the -leader's multi-group HMA block tables. - -The recurrent-cache manager's authenticated checkpoint slot replaces the -logical partial-boundary slot when required. Every physically partial -2,304-token page is copied into a dedicated immutable block before the lease -becomes attachable. Followers acquire ordinary vLLM block references and use -copy-on-write handling for private tails. - -The implementation permits sixteen waiting followers per leader, retains at -most two reusable leases, and expires a lease after fifteen seconds. Allocation -pressure releases lease references before refusing ordinary serving blocks. -Failure to create or attach a lease skips sharing; it does not make unverified -state eligible. - -## Verified-or-recompute recovery - -The pinned vLLM scheduler patch handles hybrid requests whose restored state is -invalid in any KV-cache group. It discards the complete external prefix across -all groups and computes the request normally. Partially verified hybrid state -is never published. - -A live recovery canary removed rank 3's manifest for one advertised -4,096-token entry. Ranks 0--2 verified their pages, rank 3 reported the missing -entry, and the scheduler discarded all external blocks before recomputation. -The request completed in 1.94 seconds, health remained 200, and no traceback or -engine exit occurred. Evidence: -`evidence/glm53-flash-dflash7-bf16/hotlease-2b86fb9-recovery-canary.json`. - -The ten-file vLLM source contract and GPU-free patch tests cover single-group, -multi-group, disjoint-group, and asynchronous recovery behavior. - -## Diagnostic records - -Several receipts document safe optimization rejection while shared HMA block -ownership was being derived. They are historical diagnostics, not qualified -performance results: - -- `evidence/glm53-flash-dflash7-bf16/singleflight-128k-c16-830a117.json`; -- `evidence/glm53-flash-dflash7-bf16/singleflight-retention18432-128k-c16.json`; -- `evidence/glm53-flash-dflash7-bf16/hotlease-d30cdea-128k-c16.json`; -- `evidence/glm53-flash-dflash7-bf16/hotlease-599b65a-128k-c16.json`. - -The retention-interval experiment did not produce all-group local-prefix hits -and is not part of the deployment contract. The qualified implementation uses -explicit vLLM block references instead of depending on ordinary hash -rediscovery. - -## Repository validation - -Source revision `2b86fb9d02fa3595cca5caa864b81aedce44b8bb` passed: - -- `python -m pytest sparkcache -q`: 678 passed, 4 skipped; -- `python -m pytest deploy -q`: 99 passed, 1 skipped; -- `python -m ruff check sparkcache deploy`: all checks passed. - -## Qualification limits - -- The shared-prefix measurements establish C2, C8, and C16 behavior under a - C32 scheduler ceiling; they do not establish C24 or C32 performance. -- The unrelated-cold C16 matrix and unrelated-decode interference measurement - remain outside the qualification evidence. -- Sparse row-prefix aliases are implemented and GPU-free tested, but this GLM - record does not qualify them because GLM uses opaque page storage. -- Growing conversations can still publish another complete snapshot. -- No cross-TP canonical shard format or network storage backend is implemented. diff --git a/GLM53_SPARKCACHE_CUDA_RESTORE_PERFORMANCE_VALIDATION.md b/GLM53_SPARKCACHE_CUDA_RESTORE_PERFORMANCE_VALIDATION.md new file mode 100644 index 0000000..2d3e4fe --- /dev/null +++ b/GLM53_SPARKCACHE_CUDA_RESTORE_PERFORMANCE_VALIDATION.md @@ -0,0 +1,319 @@ +# GLM-5.3 SparkCache CUDA full-snapshot validation + +Date: 2026-08-30 + +## Status + +Full `snapshot-v1` opaque-page restore is **qualified** at 131,072 tokens and C1 +for the exact GLM-5.3 Flash DFlash7 TP4/DCP1 artifacts identified below. The +qualification covers verified all-rank placement, multi-group recovery, and +continued generation. + +The flat `sparkcache-page-snapshot-manifest/v2` artifact is also qualified for +one exact C1 case: 813,068,464 encoded bytes in 13 authenticated objects, +sequential object read/hash of 1.35–1.50 seconds, an all-rank SparkCache CUDA +restore of 1.55–1.70 seconds, and the exact expected codeword before and after +restart. This result is bound to local image +`sha256:35b58a7bf414059c65b8f74e4e4b17ee6a81b7008e1bffbc9bd298b5e08c739e`, +SparkCache source `a1511d26a1fe2b17b24561bc52e376bf7f54b06a`, and source tree +`4d5b8eb8c5c13793ee7a1e67b2b34bd38fcf4ddb`. SparkCache `229d7d6` is the +implementation origin of the flat-v2 schema and header-inclusive statistics +fix, not the complete qualified runtime identity. No public OCI digest is +qualified by this record. + +Tail-only opaque-page deltas, 16-member host-base read coalescing, and +multi-root concurrent restore are **research-only**. In the exact DFlash7 C2 +case, both restored page-delta responses failed the codeword oracle while the +same requests succeeded through recomputation. Completion, timing, and +structural verification do not override that semantic result. + +Earlier C2/C8/C16 receipts used a suffix-only predicate. They remain useful +performance diagnostics, but they do not qualify semantic concurrency or +shared GPU-prefix attachment. Arbitrary earlier opaque-page aliases remain +**unsupported**. + +## Qualified runtime identity + +| Attribute | Value | +|---|---| +| SparkCache source revision | `2b86fb9d02fa3595cca5caa864b81aedce44b8bb` | +| SparkCache source-tree SHA-256 | `b3e84d220e215bdad99455a7eefb431b9aea248e0edb6ff417319c420433f55a` | +| vLLM source revision | `local-inference-lab/vllm@da4d7be6c97434f6942292ed8abbf4b32dc44355` | +| Serving topology | GLM-5.3 Flash, TP4/DCP1, one rank on each of `spark-r0` through `spark-r3` | +| Scheduler capacity | `--max-num-seqs 32` | +| Restore concurrency | Two host restore workers and two SparkCache CUDA placement lanes per rank | +| SparkCache CUDA staging | Two 256 MiB mapped-host arenas per rank | +| Persistent prefix | 131,072 tokens and 813,068,464 encoded bytes per rank | +| Runtime receipt | `evidence/glm53-flash-dflash7-bf16/hotlease-2b86fb9-runtime.json` | + +The runtime receipt records one immutable image ID per rank, CUDA placement library +SHA-256 `683cb9e0420da9c68e3263093077fdbcaa400913ff0fb1d18639771213220605`, +scheduler SHA-256 +`4f8793c4ac4bf356a89c829b6e75b189e6bc4a74c97135208952d0bad1678f15`, +and KV-cache-manager SHA-256 +`ee03dc9ce2b720c0be6e9f572d23580ba96eff68fe3406250557e83071654af0`. + +The direct page-placement implementation is identified by these commits: + +| Responsibility | Revision | +|---|---| +| Restore phase timing | `175f9401984a03744d7fe1a985d7c2ef6035f949` | +| SparkCache CUDA page placement | `71f367be07788d611698a251fe866d678b0034ae` | +| Multi-slab restore and exact-prefix discovery | `8e7f5fc62fd4fffdd661aca9ea634cf130c45d1a` | +| Direct pipelined slab restore | `94c44930a13df5c668d777e0270e7d8203069d7c` | +| Authenticated span-table bound | `9dbf73c0caab89b24346567e2769752ac746e114` | + +The 131,072-token single-request measurement used source-tree SHA-256 +`368cc18dbccc262a1f2a1f1eef5aced90690031abd1f2fedf3d192e60a67012b` +and parent/runtime image +`sha256:7c007cf673c35f5818da7fea8faa343304baed00f489efdcbd027d6616b8a290`. +The historical shared-prefix diagnostics used source revision `2b86fb9d...` +and source-tree SHA-256 `b3e84d...` identified in the table above. + +### Qualified flat-v2 artifact + +| Attribute | Value | +|---|---| +| SparkCache source revision | `a1511d26a1fe2b17b24561bc52e376bf7f54b06a` | +| SparkCache source tree | `4d5b8eb8c5c13793ee7a1e67b2b34bd38fcf4ddb` | +| SparkCache source-tree SHA-256 | `6651f2823c816fac93779cbca54a8f19c0ed262830953149f3a87d189d1f833b` | +| Flat-v2/header-fix implementation origin | `229d7d6158261e9510ab99d7e82d532abb9ade01` | +| Local image ID | `sha256:35b58a7bf414059c65b8f74e4e4b17ee6a81b7008e1bffbc9bd298b5e08c739e` | +| Parent image ID | `sha256:cc2c0e2f812f4b78d5b91f863aaf46fd8e8e505844245aa50911af1fb8e061c0` | +| Publication identity | `snapshot-v1` | +| Root schema | `sparkcache-page-snapshot-manifest/v2` | +| Persistent prefix | 131,072 tokens and 813,068,464 encoded bytes per rank | +| Physical objects | 13 authenticated objects per rank, each at most 64 MiB | +| Serving topology | GLM-5.3 Flash DFlash7, TP4/DCP1, one rank per DGX Spark | +| Qualified workload | C1 publication, restart, all-rank restore, and exact codeword | + +The measured all-rank SparkCache CUDA restore was 1.55–1.70 seconds, of which +sequential macro-object read/hash consumed 1.35–1.50 seconds. The codeword +oracle matched exactly before restart and after restoration. The result +qualifies this artifact and workload only; it does not qualify a registry +artifact, tail-delta roots, or concurrent restored roots. + +The sequential read/hash phase is a measured performance limitation of the +flat-v2 CUDA path. The identified legacy 512-object path used bounded parallel +reads within each arena slab. Existing legacy roots remain readable and follow +that restore branch, but connector publication writes flat-v2 roots and exposes +no operator setting for legacy publication. + +### Research-only four-reader prefetch + +SparkCache `eabe7fd0c878db7384ef87fe80a1e96b9bedcf67` contains bounded +four-reader prefetch for flat version 2 objects. +Its Git tree is `d88a65ea265a6f212367baa8c4a4970079d6b08a` and its deployable +SparkCache source SHA-256 is +`d9a7800ce201b0671676fc8d71423947c7b24e4797758db727c06e0e684495fe`. +It authenticates the first object before parsing the snapshot header, then +reads and authenticates at most four remaining objects concurrently into +request-private host buffers before waiting for a placement arena. One host +batch retains at most 256 MiB beyond the two placement-owned arenas, allowing +storage reads to overlap placement of the preceding batch. Mapped-arena copy +and CUDA submission remain in manifest order and begin only after every read in +the batch succeeds. The configured I/O worker count may reduce concurrency to +one; values above four remain capped at four. Diagnostics separate read/hash, +arena wait, host copy, submission-call, and final completion time. + +The version 2 root validator authenticates the ordered contiguous object +descriptors and each object's SHA-256 before direct placement. The root's +`snapshot_sha256` remains part of authenticated schema metadata, but the direct +path no longer repeats SHA-256 over the complete byte stream after every object +digest has matched. Persisted object and geometry checks are unchanged. + +Status: **research-only**. ARM64 image +`sha256:df4e09a32cdbf1c0e69cc7c4c9e95d890d6c7a1e3eaac84f969912a16fd27dd3` +structurally verified 813,068,464 bytes in 13 objects on every rank. Read and +authentication took 484.1–528.8 ms, placement took 323.7–330.9 ms including +293.2–297.7 ms of arena wait, final completion took 129.5–131.6 ms, and cache +service took 1,231.7–1,331.2 ms. + +The restored response was `spark`; the deterministic oracle required `red`. +A one-token-changed prompt with the same length recomputed `red` in 55.141 +seconds. Structural validation therefore did not establish semantic +correctness. The image is rejected for deployment and does not replace the +single-reader qualification. The exact evidence is recorded in +[`flat-v2-four-reader-semantic-rejection-eabe7fd.json`](evidence/glm53-flash-dflash7-bf16/flat-v2-four-reader-semantic-rejection-eabe7fd.json). + +## Implemented restore path + +SparkCache CUDA restore reads immutable `.spcc` objects directly into alternating +mapped-host arenas with `pread`, hashes every complete file in place, validates +its authenticated extent table, and submits only validated spans to the CUDA +page-placement kernel. Read work and CUDA submission overlap across slabs. + +This path avoids Python `ContextChunk` reconstruction and an 813 MiB +intermediate join/copy. The adapter accepts at most 4,096 authenticated spans, +matching the validated SparkCache CUDA ABI. These changes do not alter `CacheIdentity`, +digest values, 256-token logical geometry, or the on-disk exact-manifest and +chunk formats. + +## Practical restore gains + +### One 131,072-token prefix + +Each rank restored 813,068,464 bytes through four slabs: + +| Rank | Cache service | Read and hash | CUDA placement submit | CUDA finish | +|---:|---:|---:|---:|---:| +| 0 | 141.9 ms | 77.9 ms | 25.4 ms | 3.4 ms | +| 1 | 131.3 ms | 84.6 ms | 13.4 ms | 3.4 ms | +| 2 | 139.0 ms | 92.7 ms | 14.1 ms | 3.4 ms | +| 3 | 250.1 ms | 140.7 ms | 20.1 ms | 3.4 ms | + +The Python reconstruction pipeline measured 1.29--1.46 seconds per rank for +the same stored prefix. SparkCache CUDA restore therefore reduced the slowest-rank +time to 250.1 ms. End-to-end client latency was 0.907 seconds, including +scheduler work, live-token execution, and DFlash generation. A separate +historical canary found the expected marker suffix, and HTTP health remained +200. + +### Eight concurrent 16K prefixes: diagnostic timing + +The Python/Torch placement path produced 1.54--1.57 second submission spikes; +eight clients completed in 9.45--10.64 seconds. SparkCache CUDA placement submitted in +6--15 ms, and two restore lanes completed eight clients in approximately +1.2--2.1 seconds. This diagnostic isolates page placement as the dominant +serialized cost in that workload. The timing is diagnostic and does not +qualify semantic concurrency. + +## Historical concurrency diagnostics + +The independent-restore baseline issued a complete external restore for every +request: two per rank for C2, eight for C8, and sixteen for C16. + +| Cohort | Client min | Client p50 | Client max | +|---:|---:|---:|---:| +| C2 | 1.282 s | 1.282 s | 1.377 s | +| C8 | 0.947 s | 2.017 s | 3.129 s | +| C16 | 1.063 s | 3.363 s | 5.335 s | + +Bounded shared-prefix leases changed the observed rank-local work from sixteen complete +813 MiB restores to one. The standard chat C16 measurement completed every +request at 2.980 seconds p50 and 5.064 seconds maximum. This measurement +includes sixteen large prompt-tokenization operations. + +Pretokenized requests preserve the exact chat-template token sequence while +moving tokenizer work outside the timed interval: + +| Cohort | External restores per rank | Client min | Client p50 | Client max | +|---:|---:|---:|---:|---:| +| C2 | 1 | 0.541 s | 0.541 s | 0.737 s | +| C8, retained lease | 0 | 0.407 s | 1.211 s | 1.218 s | +| C16 | 1 | 0.735 s | 2.698 s | 2.701 s | +| C16, shared exact prefix and distinct tails | 1 | 2.407 s | 3.894 s | 3.896 s | + +The pretokenized C16 restore used 104.2--165.3 ms of cache service per rank +and at most 3.2 ms of restore-queue wait. Every request completed, queues +drained to zero, no engine exited, and the historical post-run canary found the +expected marker suffix. + +Diagnostic shared-prefix receipts: + +- `evidence/glm53-flash-dflash7-bf16/hotlease-2b86fb9-128k-c2-pretokenized.json`; +- `evidence/glm53-flash-dflash7-bf16/hotlease-2b86fb9-128k-c8-pretokenized.json`; +- `evidence/glm53-flash-dflash7-bf16/hotlease-2b86fb9-128k-c16.json`; +- `evidence/glm53-flash-dflash7-bf16/hotlease-2b86fb9-128k-c16-pretokenized.json`; +- `evidence/glm53-flash-dflash7-bf16/hotlease-2b86fb9-128k-c16-shared-trunk-pretokenized.json`; +- `evidence/glm53-flash-dflash7-bf16/post-hotlease-2b86fb9-semantic.json`. + +Baseline receipts: + +- `evidence/glm53-flash-dflash7-bf16/native-128k-c2-b00c6d4.json`; +- `evidence/glm53-flash-dflash7-bf16/native-128k-c8-hot-b00c6d4.json`; +- `evidence/glm53-flash-dflash7-bf16/native-128k-c16-hot-b00c6d4.json`; +- `evidence/glm53-flash-dflash7-bf16/post-b00c6d4-semantic.json`. + +## Implemented shared-prefix ownership + +One request leads restoration for a persistent digest. Followers wait without +allocating another external restore. A lease becomes visible only after every +worker reports successful restoration and the scheduler normalizes the +leader's multi-group HMA block tables. + +The recurrent-cache manager's authenticated checkpoint slot replaces the +logical partial-boundary slot when required. Every physically partial +2,304-token page is copied into a dedicated immutable block before the lease +becomes attachable. Followers acquire ordinary vLLM block references and use +copy-on-write handling for private tails. + +The implementation permits sixteen waiting followers per leader, retains at +most two reusable leases, and expires a lease after fifteen seconds. Allocation +pressure releases lease references before refusing ordinary serving blocks. +Failure to create or attach a lease skips sharing; it does not make unverified +state eligible. These bounds describe the implementation. They have no +exact-output semantic concurrency qualification in this record. + +## Capacity observations and planned bounds + +The exact DFlash7 runtime used a 20 GiB KV-cache setting. C2×128K is an observed +safe capacity candidate, not a qualified cached workload. C6×128K admitted only +one request at a time, used approximately 39–41% of GPU KV capacity for that +request, and serialized completion over 61–313 seconds. It therefore does not +establish C6 capacity or concurrency support. + +C8×64K and C16×32K are planned qualification points and have no live evidence. +Sixteen independent 131,072-token requests are unsupported at 20 GiB unless +requests share a GPU-resident trunk or the deployment provides more KV +capacity. + +## Verified-or-recompute recovery + +The pinned vLLM scheduler patch handles hybrid requests whose restored state is +invalid in any KV-cache group. It discards the complete external prefix across +all groups and computes the request normally. Partially verified hybrid state +is never published. + +A live recovery canary removed rank 3's manifest for one advertised +4,096-token entry. Ranks 0--2 verified their pages, rank 3 reported the missing +entry, and the scheduler discarded all external blocks before recomputation. +The request completed in 1.94 seconds, health remained 200, and no traceback or +engine exit occurred. Evidence: +`evidence/glm53-flash-dflash7-bf16/hotlease-2b86fb9-recovery-canary.json`. + +The ten-file vLLM source contract and GPU-free patch tests cover single-group, +multi-group, disjoint-group, and asynchronous recovery behavior. + +## Diagnostic records + +Several receipts document safe optimization rejection while shared HMA block +ownership was being derived. They are historical diagnostics, not qualified +performance results: + +- `evidence/glm53-flash-dflash7-bf16/singleflight-128k-c16-830a117.json`; +- `evidence/glm53-flash-dflash7-bf16/singleflight-retention18432-128k-c16.json`; +- `evidence/glm53-flash-dflash7-bf16/hotlease-d30cdea-128k-c16.json`; +- `evidence/glm53-flash-dflash7-bf16/hotlease-599b65a-128k-c16.json`. + +The retention-interval experiment did not produce all-group local-prefix hits +and is not part of the deployment contract. The implementation uses +explicit vLLM block references instead of depending on ordinary hash +rediscovery; this record does not qualify that path semantically. + +## Repository validation + +Source revision `2b86fb9d02fa3595cca5caa864b81aedce44b8bb` passed: + +- `python -m pytest sparkcache -q`: 678 passed, 4 skipped; +- `python -m pytest deploy -q`: 99 passed, 1 skipped; +- `python -m ruff check sparkcache deploy`: all checks passed. + +## Qualification limits + +- The qualified workload is C1 full-snapshot restore at 131,072 tokens. +- Four-reader flat-v2 prefetch at SparkCache `eabe7fd` is research-only because + its structurally verified restore failed the exact semantic oracle. It does + not inherit the `35b58a7…` result. +- Historical C2/C8/C16 timing and completion receipts are diagnostic because + they do not carry exact-output semantic proof. +- Tail-delta publication, shared host-base reads, shared GPU-prefix attachment, + and multi-root cached concurrency are research-only. +- At 20 GiB, C2×128K is an observed capacity candidate; C8×64K and C16×32K are + planned and unqualified; C16×128K is unsupported without GPU trunk sharing + or additional KV capacity. +- Sparse row-prefix aliases are implemented and GPU-free tested, but this GLM + record does not qualify them because GLM uses opaque page storage. +- Neither the `da4d7be6` record nor local images `ed60…`, `cc2c0e2…`, and + `35b58a7…` qualify tail-only publication or restored-delta concurrency. +- No cross-TP canonical shard format or network storage backend is implemented. diff --git a/README.md b/README.md index 183c1c7..c12f3d0 100644 --- a/README.md +++ b/README.md @@ -1,290 +1,313 @@ # SparkCache -> [!WARNING] -> **Alpha research software.** Use SparkCache for evaluation. APIs, cache -> formats, deployment patches, and supported profiles may change. - -SparkCache is a persistent, rank-local NVMe context cache for vLLM's -KV-Connector-V1 interface. Each tensor-parallel worker stores and restores only -the model state owned by its physical rank. - -Sparkcache KV/context Restore reads each rank's local filesystem. SparkCache does not send KV -payload over a network link. vLLM collectives may still use Ethernet, a -switched fabric, or a switchless ring. - -Deployment labels use TP for tensor-parallel degree and DCP for -decode-context-parallel degree. - -PyPI version `0.1.0a3` is **implemented** and GPU-free tested. Its package -artifact has no live serving qualification. Qualification belongs to an exact -artifact, model, topology, and vLLM source contract. - -The public GLM-5.3 OCI artifact is qualified for its recorded 8,192-token -Python-placement restore. The source deployment separately qualifies native -131,072-token restore and bounded shared-prefix reuse. See -[the public image record](deploy/glm53_flash/IMAGE_ANNOUNCEMENT.md), -[the GLM-5.3 validation](GLM53_FLASH_DFLASH7_LIVE_VALIDATION.md), and -[the SparkCache CUDA restore record](GLM53_NATIVE_RESTORE_PERFORMANCE_VALIDATION.md). - -The public image does not contain the model checkpoints. SparkCache CUDA restore and -shared GPU-prefix qualification belong to a later source-bound runtime that has -no published OCI digest. - -## Implemented capabilities +SparkCache is a persistent, rank-local NVMe context cache for vLLM. It lets a +request restore the longest verified prompt prefix available on every required +tensor-parallel rank, then compute only the uncached tail. This turns repeated +long-context prefill into local storage work without introducing a cache +network or moving one rank's model state through another rank. + +Each worker stores only the state owned by its physical rank. A restore becomes +usable only after identity, topology, all-rank availability, object integrity, +and coordinated completion agree. If that proof is incomplete, vLLM computes +the request normally. Optional cache work never becomes a serving dependency. + +The [interactive prefix-reuse explorer](docs/sparkcache-prefix-explainer.html) +shows longest-prefix selection, row descriptor segments, copy-on-write tails, +64 MiB page objects, and the boundary between implemented sharing mechanisms +and qualified serving behavior. + +## Qualified outcomes + +Qualification belongs to the exact artifact, model, topology, vLLM source, and +bounded workload named by its evidence record. + +| Qualified deployment | Bound demonstrated | Recorded outcome | Evidence | +|---|---:|---|---| +| `sparkcache==0.1.0a1`, DeepSeek-V4 TP2/DCP1 and TP4/DCP1 | 73,728 restored tokens | 413.9–517.0 ms cache service per rank | [release-wheel validation](MULTI_MODEL_LIVE_VALIDATION.md) | +| `sparkcache==0.1.0a2`, GLM-5.2 TP4/DCP4 | 225,536 restored tokens | 3.17–4.17 s cache service per rank | [package validation](GLM52_A2_LIVE_VALIDATION.md) | +| GLM-5.3 DFlash7 TP4/DCP1 source runtime at `da4d7be6` | `snapshot-v1`, 131,072 tokens, C1 | 131–250 ms cold SparkCache CUDA restore per rank; continued generation reached the recorded marker | [CUDA restore validation](GLM53_SPARKCACHE_CUDA_RESTORE_PERFORMANCE_VALIDATION.md) | +| GLM-5.3 DFlash7 TP4/DCP1 local image `35b58a7…`, SparkCache `a1511d2` | `snapshot-v1`, `sparkcache-page-snapshot-manifest/v2`, 131,072 tokens, C1 | 13 authenticated objects; sequential object read/hash took 1.35–1.50 s, all-rank SparkCache CUDA restore took 1.55–1.70 s, and the exact codeword matched before and after restart | [CUDA restore validation](GLM53_SPARKCACHE_CUDA_RESTORE_PERFORMANCE_VALIDATION.md) | + +C1, C8, and C16 mean one, eight, and sixteen concurrent requests. At the tested +20 GiB KV-cache setting, C2×128K is an observed capacity candidate, not a +qualified SparkCache workload. C6×128K admitted one request at a time and +serialized completion over 61–313 seconds. C8×64K and C16×32K are planned and +unqualified. Sixteen independent 128K requests are unsupported at that capacity +unless they share a GPU-resident trunk or the deployment provides more KV +capacity. + +## Capability status + +Status words are deliberate: + +- **implemented**: present in this source tree with GPU-free behavioral tests; +- **qualified**: exercised by the exact live artifact and bounded case linked; +- **research-only**: experimental work that does not define serving support; +- **unsupported**: no compatible serving contract or qualification exists. | Capability | Status | Scope | |---|---|---| -| Content-addressed persistent snapshots | **implemented** | Immutable chunks, manifest-last publication, rank-local capacity control, and verified restore | -| Longest stored exact-boundary discovery | **implemented** | One incremental token-digest pass; longest all-rank candidate wins | -| Sparse row-prefix aliases | **implemented** | Authenticated metadata over `per_token_rows`; GPU-free regression coverage | -| Opaque hybrid-memory-allocator page snapshots | **qualified** | Listed DeepSeek-V4 and GLM deployments only | -| Native direct page restore | **qualified** | Exact GLM-5.3 TP4/DCP1 source deployment recorded below | -| Concurrent shared GPU-prefix reuse | **qualified** | Up to 16 waiting followers, two retained prefixes, 15-second retention; GLM-5.3 through 16 concurrent requests | +| Content-addressed persistent snapshots | **implemented** | Immutable objects, manifest-last publication, verified reads, capacity maintenance | +| Longest stored-boundary discovery | **implemented** | One incremental token-digest pass; longest all-rank candidate wins | +| Sparse row-prefix aliases | **implemented** | Authenticated `per_token_rows` descriptor segments; no live serving qualification | +| Tail-only row publication | **implemented** | Opt-in `tail-cow-v1`; GPU-free tested, no live serving qualification | +| Opaque HMA page snapshots | **qualified** | Exact DeepSeek-V4 and GLM deployments linked above | +| Tail-only opaque-page deltas | **research-only** | Implemented and GPU-free tested; C2 restored-delta responses failed exact semantics while recomputation succeeded | +| 64 MiB flat page macro objects | **qualified** | Exact C1 evidence is SparkCache `a1511d2` / image `35b58a7…` with sequential reads | +| Four-reader flat-page prefetch | **research-only** | SparkCache `eabe7fd` passed structural checks but failed the exact 131,072-token semantic oracle | +| SparkCache CUDA restore and placement | **qualified** | Exact GLM-5.3 TP4/DCP1 source artifacts in the linked records | +| Shared exact-prefix GPU blocks | **implemented** | Bounded vLLM lease path; earlier timing receipts are diagnostic, not exact-output semantic qualification | +| Different-root shared row segments | **implemented** | Authenticated `per_token_rows` descriptor-prefix sharing; GPU-free tested, not live qualified | +| Shared opaque-page base reads | **research-only** | A 16-member host-base coalescer is implemented and GPU-free tested at SparkCache `a1511d2`; C2 multi-root restored-delta semantics failed; [review evidence](https://github.com/FujitsuPolycom/sparkcache/pull/42) | | Streaming snapshots | **research-only** | GLM-5.2 DCP4 inventory; disabled for opaque page profiles | | Buddy replication | **research-only** | Protocol and receiver state exist; no network carrier is included | +| Heat and SSD write-control model | **research-only** | Independent offline design in [PR #36](https://github.com/FujitsuPolycom/sparkcache/pull/36); excluded from serving and package imports | -## Verified persistent-restore model +SparkCache is alpha research software. APIs, cache formats, patches, and +supported profiles may change. -SparkCache uses a verified-or-recompute rule. Restored blocks reach inference -only after identity, compatibility, all-rank availability, and payload-integrity -checks succeed. +## How prefix reuse works -If any physical rank cannot prove its shard, vLLM discards the complete external -prefix on every rank and computes the request normally. Partially verified -hybrid state is never published. +The logical cache boundary is 256 tokens. -The persistent transaction has six steps: - -1. The scheduler hashes eligible 256-token boundaries in one incremental pass. -2. Every physical rank advertises structurally compatible reusable entries for - its worker-process generation. -3. The scheduler selects the longest digest advertised by every expected rank. -4. Workers read their local objects, verify encoded bytes, and place state into - private request blocks. -5. Coordinated worker completion accepts the whole external prefix or rejects - it on every rank. -6. A completed prefill publishes immutable chunks before an atomic, fsynced - manifest makes the entry discoverable. +1. The scheduler hashes eligible boundaries in one incremental pass. +2. Each worker advertises compatible exact roots and permitted row aliases from + its active worker-process generation. +3. The scheduler selects the longest digest advertised by every expected + physical rank. +4. Workers authenticate their rank-local root and objects, then place state in + request-private blocks. +5. Coordinated completion accepts the whole external prefix on every rank or + causes the request to recompute it everywhere. +6. Publication writes immutable payload objects before an atomic, fsynced + manifest makes the root discoverable. Cache identity binds checkpoint digests, model layout, TP/DCP degrees, physical -rank, chunk geometry, record schema, draft policy, and page-reuse policy. -Incompatible entries produce a cache miss. - -Persistent files contain no CUDA pointers, allocator block tables, physical -slot coordinates, or transport sequence numbers. - -## Quickstart for qualified deployments - -Install the package artifact that matches the intended qualification evidence: +rank, logical chunk geometry, record schema, draft policy, and page-reuse +policy. Persistent files contain no CUDA pointers, allocator block tables, +physical slot coordinates, or transport sequence numbers. + +## Storage and publication + +### Row-oriented state + +`per_token_rows` stores independently reusable rank-owned tensor rows. +Authenticated descriptor segments contain at most 16 chunk descriptors. +Sparse aliases normally select 4,096-token boundaries plus the source boundary +and retain at most 64 aliases without copying payload bytes. + +The opt-in `tail-cow-v1` identity publishes immutable replacement and extension +chunks after a verified base. A partial final chunk is replaced rather than +mutated. Row aliases and row tails are **implemented** and GPU-free tested; live +model-serving qualification is absent. + +### Opaque page state + +`block_pages_v1` stores complete hybrid-memory-allocator page snapshots. Flat +v2 roots use `sparkcache-page-snapshot-manifest/v2` and content-addressed +objects of at most 64 MiB. An 813,068,464-byte flat snapshot therefore needs 13 +payload objects rather than 512 logical-chunk files. Version 1 flat manifests +remain readable. Physical grouping does not change the 256-token identity or +admission geometry. + +The qualified `35b58a7…` v2 artifact reads and hashes its 13 macro objects +sequentially. That phase measured 1.35–1.50 seconds within the 1.55–1.70-second +all-rank restore. The result is correct but slower than the identified legacy +512-object path, which used bounded parallel reads within each arena slab. + +Four-reader flat-v2 prefetch at SparkCache +`eabe7fd0c878db7384ef87fe80a1e96b9bedcf67` is **research-only**. Its bounded +reader passed GPU-free integrity and ordering tests and structurally verified +all four rank-local 131,072-token snapshots. The restored response was `spark` +instead of the required `red`; an equivalent recomputation returned `red`. +Image `df4e09a…` is therefore rejected as a deployable restore artifact and +does not replace the qualified single-reader image. The exact identity, +timings, and semantic result are recorded in the +[research receipt](evidence/glm53-flash-dflash7-bf16/flat-v2-four-reader-semantic-rejection-eabe7fd.json). + +Conversation extensions use `sparkcache-hybrid-page-delta/v1` page semantics +and `sparkcache-page-delta-manifest/v2` roots. The delta binds the exact base, +layout, block counts, and recurrent or sliding boundary. It reuses only +byte-identical pages, including the correct replacement of a boundary that +falls inside a physical page. Delta and flat payload objects are at most +64 MiB; version 1 delta roots remain readable. At most two deltas form one +graph before compaction publishes a fresh flat root. + +Tail-only opaque-page publication is implemented and GPU-free tested, but its +serving status is **research-only**. In the exact DFlash7 C2 live case, +reconstructed page-delta roots completed restore yet failed the exact codeword +check; the recomputation control returned the expected codewords. Flat +`snapshot-v1` publication remains the qualified path. Arbitrary earlier +opaque-page aliases are **unsupported** because truncating an encoded snapshot +does not create a valid earlier context. + +### Shared GPU prefixes and row segments + +Concurrent requests for one persistent digest coalesce around one restore. +After all workers succeed, patched vLLM may retain the verified multi-group +block table as a bounded shared-prefix lease. Each follower owns ordinary block +references and computes a request-private GPU tail. A partial page is copied to +a dedicated immutable block before attachment. + +The implementation permits at most 16 waiting followers, keeps at most two +prefixes eligible for 15 seconds, and releases lease references under allocation +pressure. Those limits describe the implementation, not a semantic concurrency +qualification. Earlier C16 receipts establish timing and completion only; they +did not use the exact-output codeword oracle. + +For `per_token_rows`, different selected roots can also name one identical +authenticated descriptor prefix. Every rank must prove the same descriptor +sequence; a mismatch causes the affected request to recompute. Different-root +row-segment sharing is **implemented** and GPU-free tested, but no live model +artifact qualifies it. + +### Shared opaque-page base reads + +Concurrent `block_pages_v1` page-delta restores with independently +authenticated result roots may share one rank-local read of an identical +immutable base. Each request still authenticates its private delta, +reconstructs its own result snapshot, and performs request-private placement. +The coordinator admits at most two simultaneous flights, 16 cumulative +participants per flight, 1 GiB declared bytes per flight, and 2 GiB of peak +reservations. Followers do not occupy loader lanes while the base is pending; +an unrelated restore can use another configured lane. + +The base buffer is released after every registered member acquires or abandons +it. This is bounded request-cohort I/O sharing, not a retained host-memory tier, +and it never shares mutable recurrent pages. The 16-member coordinator is +implemented and GPU-free tested at SparkCache `a1511d2`. Its serving status is +**research-only** because the exact C2 multi-root restored-delta case failed +semantic comparison while the same requests succeeded through recomputation. + +## Installation and qualified entry points + +Install the package artifact that matches the intended evidence: ```bash -# GLM-5.2 EXL3 3.5-bpw at TP4/DCP4 -python -m pip install sparkcache==0.1.0a2 - # DeepSeek-V4 at TP2/DCP1 or TP4/DCP1 python -m pip install sparkcache==0.1.0a1 -``` - -Use [the GLM-5.2 deployment guide](deploy/glm52_35bpw/README.md) or -[the DeepSeek-V4 deployment guide](deploy/deepseek_v4/README.md). Each builder -verifies its accepted serving source and emits a source-bound overlay receipt. -The DeepSeek profiles in `0.1.0a2` are **implemented** but **unqualified** for -that package artifact. DeepSeek qualification remains bound to `0.1.0a1`. - -The GLM-5.3 source deployment uses this repository and -[its deployment guide](deploy/glm53_flash/README.md): - -```bash -git clone https://github.com/FujitsuPolycom/sparkcache.git -cd sparkcache -python -m pip install '.[connector]' +# GLM-5.2 EXL3 3.5-bpw at TP4/DCP4 +python -m pip install sparkcache==0.1.0a2 ``` -The qualified 8,192-token public image is available by immutable digest: +Use the [DeepSeek-V4 guide](deploy/deepseek_v4/README.md) or +[GLM-5.2 guide](deploy/glm52_35bpw/README.md). The DeepSeek profiles included +in `0.1.0a2` are **implemented**, but DeepSeek package qualification remains +bound to `0.1.0a1`. + +The public GLM-5.3 image is qualified for its recorded 8,192-token Python/Torch +page restore: ```bash docker pull ghcr.io/fujitsupolycom/sparkring-glm53-sparkcache@sha256:cd4045bba2a0f3dc55361560f8c3a3f171939854db28d48dfdae58eed9c44943 ``` -The matching SparkRing procedure is pinned at -[`FujitsuPolycom/sparkring@6e9e3ace`](https://github.com/FujitsuPolycom/sparkring/blob/6e9e3acef62886a71531310673463972944b2b84/docs/GLM53_FLASH_DFLASH2_BF16_SPARKCACHE_TP4_QUICKSTART.md). +The [GLM-5.3 source guide](deploy/glm53_flash/README.md) describes +SparkCache CUDA restore and adaptive-MTP runtimes. Those source-bound artifacts +do not have a published OCI digest. PyPI `0.1.0a3` is **implemented** and +GPU-free package tested; it has no live serving qualification. -The wheel contains the Python package and exact-hash vLLM source contracts. It -omits `patches/`, `deploy/`, and optional C++/CUDA sources, so a wheel alone -does not construct a supported vLLM serving runtime. - -Configure vLLM with connector module path -`sparkcache.spark_context_cache_connector`. A source deployment adds the +Configure vLLM with connector module +`sparkcache.spark_context_cache_connector`. Source deployments add the directory containing `sparkcache/` to `PYTHONPATH`. -## Qualified models and measured results - -Qualification applies only to the named artifact, model, topology, and source -contract. Detailed commands, runtime identities, and receipts remain in the -linked validation records. - -The `0.1.0a1` serving receipts used `--max-num-batched-tokens 4096`. A value of -8192 is known to run, but performance and capacity at that value are outside -the package qualification. - -| Artifact and deployment | Restored tokens | Cache service per rank | Evidence | -|---|---:|---:|---| -| `0.1.0a1`, DeepSeek-V4-Flash-0731 TP2/DCP1 | 73,728 | 459.8–517.0 ms | [release-wheel validation](MULTI_MODEL_LIVE_VALIDATION.md) | -| `0.1.0a1`, DeepSeek-V4-Flash-0731 TP4/DCP1 | 73,728 | 413.9–494.6 ms | [release-wheel validation](MULTI_MODEL_LIVE_VALIDATION.md) | -| `0.1.0a1`, GLM-5.2 EXL3 3.5-bpw TP4/DCP4 | 225,536 | 3.39–3.95 s | [release-wheel validation](MULTI_MODEL_LIVE_VALIDATION.md) | -| `0.1.0a2`, GLM-5.2 EXL3 3.5-bpw TP4/DCP4 | 225,536 | 3.17–4.17 s | [package validation](GLM52_A2_LIVE_VALIDATION.md) | -| Public GLM-5.3 OCI image, TP4/DCP1 | 8,192 | 156.8–171.8 ms | [public image record](deploy/glm53_flash/IMAGE_ANNOUNCEMENT.md) | -| GLM-5.3 source deployment, TP4/DCP1 | 8,192 | 147.2–194.0 ms | [GLM-5.3 validation](GLM53_FLASH_DFLASH7_LIVE_VALIDATION.md) | - -The native GLM-5.3 work changes practical restore cost across prefix sizes and -concurrency. C1, C8, and C16 mean one, eight, and sixteen concurrent requests. - -Client timing includes scheduler and model work. Cache-service timing isolates -rank-local restore work. The pretokenized result is not compared with chat API -timing. - -| Prefix and concurrency | Comparison | Recorded result | -|---|---|---| -| 8,192 tokens, C1 | qualified Python page restore | 147.2–194.0 ms cache service per rank | -| 16,384 tokens, C8 | Python/Torch placement vs SparkCache CUDA placement | 9.45–10.64 s vs 1.2–2.1 s client latency | -| 131,072 tokens, C1 | reconstruction pipeline vs cold direct mapped-arena restore | 1.29–1.46 s vs 131–250 ms cache service per rank; a host-warm restore reached 104–165 ms | -| 131,072-token shared prefix, C16 | independent restores vs shared verified GPU blocks | rank-local work fell from 16 × 813 MB to 1 × 813 MB; standard-chat client p50 fell from 3.363 s to 2.980 s | -| 131,072-token shared prefix, pretokenized C16 | standalone measurement | 2.698 s client p50 and 2.701 s maximum | - -The 16-request shared-prefix qualification used vLLM `--max-num-seqs 32`. -Every request succeeded and each rank performed one external restore. The -historical post-run canary accepted any content ending in -`SPARKCACHE_GLM53_OK`; its receipt proves continued generation and the marker -suffix, not exact visible output. - -Two- and eight-request identical-prefix cohorts and a 16-request shared-trunk -cohort with distinct tails also passed under the recorded runtime. - -## Prefix reuse - -The logical cache boundary is 256 tokens. The scheduler computes wire-compatible -digests at each eligible boundary and selects the longest reusable digest -advertised by every physical rank. - -An exact manifest represents one complete aligned snapshot. A grown conversation -can reuse an earlier exact snapshot when every rank still advertises that exact -boundary. - -For row-oriented storage, identified by `per_token_rows`, SparkCache also -publishes authenticated sparse aliases over already durable chunk payloads. -Exact manifests take precedence over aliases with the same digest. - -Default alias publication selects 4,096-token descriptor boundaries and the -source boundary, retaining at most 64 aliases. Alias graphs participate in TTL, -LRU, capacity accounting, invalidation, and orphan collection. - -Opaque hybrid page storage, identified by `block_pages_v1`, encodes a complete -boundary snapshot. Schema-capable flat publication uses the authenticated -`sparkcache-page-snapshot-manifest/v2` root and partition that opaque byte -stream into content-addressed objects of at most 64 MiB. The root separately -records the 256-token logical chunk size and count used by identity and -admission; physical extents are not independently usable token ranges. - -Version 1 flat manifests, which store one encoded `.spcc` file per logical -chunk, remain readable. The v2 representation does not change `CacheIdentity`, -digest salts, logical chunk geometry, or either the default flat namespace or -the opt-in `page-tail-cow-v1` namespace. A schema-incompatible reader does not -reinterpret a v2 root as v1: strict manifest validation makes it a cache miss. Consequently, -a mixed-version rollback can lose a reusable cache entry but cannot serve it -under the wrong storage contract. - -Concurrent requests for one persistent digest are coalesced around one restore. -After every worker finishes, patched vLLM retains the verified multi-group block -table as a bounded shared-prefix lease. - -A lease permits at most 16 waiting followers and remains eligible for 15 -seconds. At most two prefixes remain retained; allocation pressure releases -lease references before denying ordinary serving allocations. - -Each partial physical page is copied into a dedicated immutable block before a -lease becomes attachable. Followers use ordinary vLLM block references and -copy-on-write handling for their private tails. - ## Operations `spark_cache_max_bytes` is the high watermark for filesystem-allocated bytes under one rank-local root. Maintenance evicts least-recently-used roots down to -`spark_cache_low_watermark_bytes`. - -`spark_cache_ttl_seconds` expires roots by recency; zero disables TTL. Capacity -accounting counts shared chunks and alias segments once and preserves every -object referenced by a surviving root. - -Each asynchronous restore emits one compact `sparkcache-restore-timing/v1` JSON -record. It separates queue wait, manifest lookup, read and verification, -reconstruction, device submission, and CUDA synchronization. - -Timing collection is diagnostic. Missing timing data cannot make cached state -eligible. A restore either supplies verified state or causes request -recomputation. - -Repeated restores do not rewrite KV payload files. A successful restore may -update manifest recency metadata at most once per minute. - -Publishing a reusable context writes immutable payload objects and a manifest. -A growing conversation can publish another complete snapshot, so SparkCache -must not be described as having negligible SSD wear. - -Operators should monitor the NVMe Data Units Written counter and compare its -daily change with workload publication volume. Device endurance depends on -context size, unique publication rate, retention, and storage amplification. - -SparkCache does not expose hourly write budgets, daily write budgets, or a -physical-write-amplification estimate. Those controls are **unsupported**. - -## vLLM source contracts and native components - -SparkCache verifies whole-file hashes and required symbols before accepting a -patched vLLM source tree. A different hash is unsupported until its ownership -and recovery behavior are derived and tested. - -| vLLM source contract | Status | Scope | +`spark_cache_low_watermark_bytes`. `spark_cache_ttl_seconds` expires roots by +recency; zero disables TTL. Capacity accounting counts shared objects once and +preserves every object referenced by a surviving root. + +Each asynchronous restore emits one `sparkcache-restore-timing/v1` JSON record +covering queue wait, manifest lookup, read and verification, reconstruction, +device submission, and CUDA synchronization. Timing is diagnostic and cannot +make an entry eligible. + +Released opaque-page base-read cohorts emit +`sparkcache-page-base-restore-flight/v1` summaries with authenticated base +identity, participants, physical and avoided reads, bytes, duration, +cancellations, outcome, worker generation, and storage mode. Prompt content is +not logged. + +Repeated restores do not rewrite payload objects. Publication writes immutable +objects for row tails, page deltas, and periodic flat compactions. +Operators should monitor NVMe Data Units Written against publication volume. +Hourly or daily write-budget enforcement and cache-specific physical-write +amplification are **unsupported** in the serving runtime. + +## Exact vLLM source contracts + +SparkCache verifies accepted file hashes and required symbols before using a +patched vLLM tree. A different source is **unsupported** until its ownership, +copying, cleanup, and recovery behavior are derived and tested. + +| vLLM source | Status | Contract scope | |---|---|---| -| `vllm-project/vllm@fcc614141e5e9ab18cb304c476f7feed2a9552e3` with `patches/vllm/` | **implemented** | Exact patch inputs are published; no standalone public runtime builder is provided | -| vLLM build `e2666d9a6` with `patches/vllm-e2666d9a6/` | **qualified** | DeepSeek-V4 and GLM-5.2 builders verify source, patch, and postimage hashes | -| `local-inference-lab/vllm@da4d7be6c97434f6942292ed8abbf4b32dc44355` with `patches/vllm-da4d7be/` | **qualified** | GLM-5.3 HMA recovery, SparkCache CUDA restore, and bounded shared-prefix attachment | - -The GLM-5.3 contract at -[`vllm-kv-block-lease-contract-da4d7be.json`](sparkcache/runtime_patches/vllm-kv-block-lease-contract-da4d7be.json) -pins ten vLLM files and the symbols used for ownership, copying, and recovery. - -`libspark_cache_placement` provides the optional C++/CUDA placement path. Its -page ABI uses mapped host arenas, authenticated copy spans, and a CUDA scatter -kernel for opaque hybrid pages. - -Native loading requires an explicit library path and SHA-256. CUDA 13 builds -run a GPU-free byte-exact reference test and a CUDA hybrid-page probe before -model-serving qualification. - -SparkCache CUDA restore reads `.spcc` objects into alternating mapped -arenas, hashes complete files in place, validates authenticated extents, and -overlaps read work with CUDA submission. For a flat v2 page root it also -re-authenticates the persisted manifest identity before placement, submits one -bounded object at a time, and verifies the complete snapshot digest before the -parked request may resume. - -## Repository map and development validation +| `vllm-project/vllm@fcc614141e5e9ab18cb304c476f7feed2a9552e3` | **implemented** | Exact inputs in `patches/vllm/`; no standalone public runtime builder | +| vLLM build `e2666d9a6` | **qualified** | DeepSeek-V4 and GLM-5.2 builders in `patches/vllm-e2666d9a6/` | +| `local-inference-lab/vllm@da4d7be6c97434f6942292ed8abbf4b32dc44355` | **qualified** | GLM-5.3 full-snapshot HMA recovery and SparkCache CUDA restore; shared-prefix attachment is implemented but lacks exact-output concurrency qualification | +| `local-inference-lab/vllm@e10536aadf02a18fccddda7ec939c33147e8b0b3` | **implemented** | Adaptive-MTP integration and ten-file lease contract; no four-rank qualification | +| `local-inference-lab/vllm@0b67266a0f37d6146a8403fb8482403c62f412d5` | **implemented** | Exact 31-file Python overlay over the `da4d7be6` compiled extensions; full-snapshot C1 is qualified on the named local artifacts, while tail-delta and multi-root concurrency remain research-only; a source-built `0b67266a` wheel is **unsupported** | + +`libspark_cache_placement` is the optional C++/CUDA component. Its page ABI +uses mapped host arenas, authenticated extents, and a CUDA scatter kernel. +Loading it requires an explicit absolute path and SHA-256. SparkCache CUDA +restore authenticates complete objects before placement and verifies a flat +snapshot's complete digest before the parked request may resume. + +## Qualification boundaries + +- Full `snapshot-v1` opaque-page restore is **qualified** at 131,072 tokens and + C1 for the exact GLM-5.3 DFlash7 TP4/DCP1 artifacts named in the validation + record. +- Flat `sparkcache-page-snapshot-manifest/v2` and header-inclusive source-byte + accounting originated at SparkCache `229d7d6`. Qualification belongs to the + complete runtime source `a1511d26a1fe2b17b24561bc52e376bf7f54b06a`, tree + `4d5b8eb8c5c13793ee7a1e67b2b34bd38fcf4ddb`, and local image + `sha256:35b58a7bf414059c65b8f74e4e4b17ee6a81b7008e1bffbc9bd298b5e08c739e`: + 13 objects, sequential read/hash in 1.35–1.50 seconds, all-rank SparkCache + CUDA restore in 1.55–1.70 seconds, and the exact codeword before and after + restart. No published OCI digest is qualified. +- Bounded four-reader flat-v2 prefetch at SparkCache `eabe7fd` is + **research-only**. Its structurally verified 131,072-token restore failed the + exact codeword oracle, so image `df4e09a…` is not deployable and does not + inherit the `35b58a7…` qualification. +- Tail-only page deltas and opaque-page base-read cohorts are **research-only**. + The implementation and GPU-free tests remain, but the exact C2 restored-delta + case failed semantics while recomputation succeeded. +- The heat/write-control work in PR #36 is **research-only** and independent of + this runtime stack. It reports hypothetical admission and write pressure; it + does not enforce serving budgets or affect restore eligibility. +- At 20 GiB of KV cache, C2×128K is only an observed safe capacity candidate. + C6×128K admitted one request at a time and completed serially in 61–313 + seconds. C8×64K and C16×32K are planned and unqualified. Sixteen independent + 128K requests are unsupported without GPU trunk sharing or additional KV + capacity. Multi-root cached concurrency is not qualified at any of those + points. +- Cross-topology and heterogeneous-TP reuse are **unsupported**. Identity is + bound to topology and physical rank; there is no canonical cross-shard + format. +- DeepSeek-V4 opaque pages at DCP2 or DCP4 are **unsupported** because page + ownership and rolling-state sharding are undefined for those layouts. +- Qwen recurrent-state persistence and network cache backends are + **unsupported**. + +## Repository map and validation | Path | Responsibility | |---|---| -| `sparkcache/spark_context_cache_connector.py` | scheduler admission, worker I/O, all-rank availability, restore coalescing, shared-prefix coordination, and vLLM callbacks | -| `sparkcache/persistent_context_cache/cache_manifest.py` | exact manifests, row-prefix aliases, immutable chunks, lookup, invalidation, capacity, and garbage collection | -| `sparkcache/spark_context_cache_cuda_hybrid_restore.py` | authenticated CUDA reads, slab planning, and mapped-arena page placement | -| `sparkcache/spark_context_cache_restore_timing.py` | machine-readable asynchronous restore timing | -| `sparkcache/runtime_patches/` | exact-hash vLLM source contracts and GPU-free patch execution tests | +| `sparkcache/spark_context_cache_connector.py` | scheduler admission, worker I/O, restore coordination, and vLLM callbacks | +| `sparkcache/persistent_context_cache/cache_manifest.py` | manifests, aliases, immutable objects, lookup, invalidation, capacity, and garbage collection | +| `sparkcache/page_base_read_flights.py` | bounded request-cohort sharing for authenticated opaque-page base reads | +| `sparkcache/spark_context_cache_cuda_hybrid_restore.py` | SparkCache CUDA object reads, slab planning, and page placement | +| `sparkcache/runtime_patches/` | exact-hash vLLM source contracts and GPU-free patch tests | | `sparkcache/native/` | C++/CUDA ABI, parser, reference implementation, kernel, and probes | -| `deploy/deepseek_v4/` | DeepSeek-V4 build, launch, capacity, corruption, and semantic procedures | -| `deploy/glm52_35bpw/` | GLM-5.2 TP4/DCP4 inspection, build, launch, and semantic procedures | -| `deploy/glm53_flash/` | GLM-5.3 image, connector, benchmark, publication, and source-verification tools | -| `evidence/glm53-flash-dflash7-bf16/` | immutable GLM-5.3 request, runtime, recovery, and concurrency receipts | +| `deploy/` | exact deployment builders, launchers, and verification procedures | +| `evidence/` | immutable request, runtime, recovery, and concurrency receipts | -Run the GPU-free repository checks with: +Run the GPU-free checks with: ```bash python -m pytest sparkcache -q @@ -295,119 +318,10 @@ python -m ruff check sparkcache deploy CUDA execution requires a CUDA 13 build from [`sparkcache/native/CMakeLists.txt`](sparkcache/native/CMakeLists.txt). -## Limitations and research work - -Tail-only publication for `per_token_rows` is **implemented** with GPU-free -regression coverage and no live model-serving qualification. -The opt-in `tail-cow-v1` publication schema writes only immutable replacement -and extension chunks after an all-rank reusable boundary. It uses a distinct -cache namespace. Default `snapshot-v1` deployments retain their existing wire -identity and full-snapshot publication behavior. - -Tail-only publication for `block_pages_v1` is also **implemented** with -GPU-free regression coverage and no live model-serving qualification. The -page-semantic `sparkcache-hybrid-page-delta/v1` codec binds -the exact base snapshot and recurrent/sliding boundary and reuses only -byte-identical opaque pages. Restore reconstructs and verifies the complete -snapshot before Python or native page placement. Arbitrary earlier-prefix -aliases cannot be derived from opaque page snapshots. - -Page-delta publication writes `sparkcache-page-delta-manifest/v2` metadata over -authenticated byte extents of at most 64 MiB. This physical grouping reduces -the 1,024 delta files implied by a 262,144-token logical boundary to at most 24 -objects for a 1,575,821,491-byte delta. Reads retain at most four extent -payloads in addition to one assembled delta buffer. The logical admission and -digest boundary remains 256 tokens. Version 1 page-delta manifests remain -readable; cache identity, digest salts, and the `page-tail-cow-v1` namespace do -not change. Restore still materializes one authenticated delta buffer and the -verified reconstructed snapshot before placement. Direct placement from base -and delta extents is unsupported by this schema. - -Persistent base-segment read sharing for `block_pages_v1` page-delta roots is -**implemented and GPU-free tested; serving qualification is not established**. -Requests whose -authenticated result roots name the same base root may share one verified, -immutable base-snapshot buffer per rank. Every request independently reads and -authenticates its private delta, reconstructs its result snapshot, and performs -request-private placement. Mutable recurrent pages are never shared. - -One process admits at most two base-read cohorts and 16 cumulative participants -per cohort. A base must declare at most 1 GiB, and peak byte reservations across -cohorts must fit within 2 GiB. Followers remain outside the load lanes until the -base read completes. With two load lanes, a later unrelated restore can use the -other lane while one base read is pending. -The buffer is released when every admitted participant acquires or abandons the -result; there is no retained host-memory cache or time-based reuse. Requests -that exceed a sharing bound use an independent restore. One -`sparkcache-page-base-restore-flight/v1` summary records the physical read, -participants, avoided reads, bytes, duration, outcome, worker generation, and -storage mode without prompt content. - -This read-scheduling behavior does not change `CacheIdentity`, digest salts, -logical chunk geometry, or the `page-tail-cow-v1` namespace. Version 1 and -version 2 page-delta roots, legacy flat bases, and flat macro-object bases remain -separately authenticated by their persisted schemas. - -Flat page publication uses the same 64-MiB extent ceiling. Publication retains -at most two extent payloads per durable batch; Python restore retains at most -four extent payloads in addition to the assembled snapshot. SparkCache CUDA -restore avoids that assembled snapshot and authenticates one extent in a -mapped arena before submitting its copy spans. A flat 813,068,464-byte -snapshot therefore requires 13 payload objects rather than 512 logical-chunk -files; the manifest remains the atomic visibility point. - -Bounded flat-object prefetch is **research-only**. The implementation -authenticates up to four version 2 objects concurrently in request-private host -buffers and then copies them into mapped placement arenas in manifest order. -Its GPU-free integrity, ordering, concurrency, and memory-bound tests pass. - -The exact GLM-5.3 TP4/DCP1 serving evaluation for SparkCache -`eabe7fd0c878db7384ef87fe80a1e96b9bedcf67` structurally verified all four -rank-local 131,072-token snapshots but returned `spark` instead of the expected -`red`. An equivalent recomputation returned `red`. Consequently, the -four-reader implementation is not a deployable restore path and does not -replace the single-reader qualification. See the -[immutable research receipt](evidence/glm53-flash-dflash7-bf16/flat-v2-four-reader-semantic-rejection-eabe7fd.json). - -Flat macro publication and its SparkCache CUDA restore path are -**implemented and GPU-free tested, not live qualified**. The object-count -geometry above follows the format contract; it is not a claim of measured -latency improvement. - -Opaque HMA snapshots cannot be shortened by truncating chunk lists. SparkCache -therefore uses the page-semantic format and distinct namespace described above. -At most two page deltas may form one graph; the following publication compacts -the context into a fresh flat snapshot. Live GLM latency and write-volume -qualification for this path remains outstanding. - -Sparse row-prefix aliases are **implemented** but have no live model-serving -qualification. Their behavior is covered by GPU-free publication, discovery, -restore, capacity, and corruption regressions. - -The GLM shared-prefix runtime is qualified through 16 concurrent requests under -`--max-num-seqs 32`. Cohorts of 24 or 32 requests and more than 16 waiting -followers are **unsupported** by qualification evidence. - -The unrelated-cold 16-request matrix and decode-interference measurement are -**unqualified**. The shared-prefix measurements do not establish those workload -bounds. - -Cross-topology or heterogeneous-TP reuse is **unsupported**. Persistent identity -is bound to topology and physical rank; no canonical cross-shard format exists. - -DeepSeek-V4 opaque HMA pages at DCP2 or DCP4 are **unsupported** because page -ownership and rolling-state sharding are undefined for those layouts. - -Streaming snapshots and buddy replication remain **research-only**. Qwen -recurrent-state persistence and network cache backends are **unsupported**. - ## License and support -SparkCache is licensed under Apache-2.0. See [`LICENSE`](LICENSE) for the full -terms. - -Report defects and compatibility requests through the -[SparkCache issue tracker](https://github.com/FujitsuPolycom/sparkcache/issues). - -Include the package or source revision, vLLM source contract, model profile, -topology, and relevant receipt paths. +SparkCache is licensed under Apache-2.0. See [`LICENSE`](LICENSE). Report +defects and compatibility requests through the +[issue tracker](https://github.com/FujitsuPolycom/sparkcache/issues), including +the package or source revision, vLLM contract, model profile, topology, and +relevant receipt paths. diff --git a/ROADMAP.md b/ROADMAP.md index 58bb2e2..9dd6dea 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -25,10 +25,11 @@ page cache identities while leaving the default snapshot identity unchanged. GPU-free coverage proves copy-on-write extension, bounded page-delta compaction, recurrent/sliding boundary geometry, corruption removal, -reference-aware maintenance, and verified reconstruction. Live GLM-5.3 -qualification must still measure publication bytes, cold restore latency, -SparkCache CUDA-placement latency after page reconstruction, SSD writes, and continued -generation across repeated conversation extensions. +reference-aware maintenance, and verified reconstruction. The exact GLM-5.3 +DFlash7 C2 restored-delta case completed structural restore but failed its +codeword oracle while recomputation succeeded. Qualification therefore still +requires exact-output continued generation, publication bytes, cold restore +latency, SparkCache CUDA-placement latency after reconstruction, and SSD writes. ### Per-entry retention controls @@ -56,10 +57,20 @@ confirm completion. ### SparkCache CUDA restore expansion -**Status: research-only qualification work.** Native multi-group page restore -is implemented and source-runtime-qualified for the recorded GLM-5.3 TP4/DCP1 -profile. Tail page deltas reconstruct a fully verified snapshot before Python -or SparkCache CUDA placement, but that path has no live performance qualification. +**Status: research-only qualification work.** Multi-group full-snapshot page +restore is implemented and source-runtime-qualified for the recorded GLM-5.3 +TP4/DCP1 profile at 131,072 tokens and C1. Flat-v2 macro objects are qualified +for exact 13-object C1 image `35b58a7…` at SparkCache `a1511d2`; their +sequential read/hash phase measured 1.35–1.50 seconds. Tail page deltas +reconstruct a fully verified snapshot before Python or SparkCache CUDA +placement, but the C2 live case failed exact semantics and does not qualify that +path. + +PR43 final head `eabe7fd` implements bounded four-reader flat-v2 prefetch, +removes the redundant complete-stream hash pass after ordered per-object +authentication, and has GPU-free coverage. A live artifact must repeat the exact +C1 restart/codeword workload before the prefetch scheduling can replace the +retained sequential qualification. DeepSeek-V4 opaque HMA pages retain their verified Python restore path. CUDA support for that profile must describe all five page groups, preserve each diff --git a/deploy/deepseek_v4/TP4_RUNBOOK.md b/deploy/deepseek_v4/TP4_RUNBOOK.md index 68f8ba6..06be28b 100644 --- a/deploy/deepseek_v4/TP4_RUNBOOK.md +++ b/deploy/deepseek_v4/TP4_RUNBOOK.md @@ -19,8 +19,8 @@ or transport ports. - SparkCache: Python hybrid-memory-allocator (HMA) block-page codec, restore, and bounded LRU enabled - Capacity: 200 GiB high and 180 GiB low watermark per rank-local root -- Not qualified: DCP2/DCP4, shared writers, streaming snapshots, native direct - restore, expert parallelism, or an unpinned checkpoint/runtime +- Not qualified: DCP2/DCP4, shared writers, streaming snapshots, SparkCache + CUDA restore, expert parallelism, or an unpinned checkpoint/runtime The launcher fails closed when any of these inputs drift. Do not weaken a check to make an unfamiliar image start; derive and test a profile for that diff --git a/deploy/deepseek_v4/tp4_profile.json b/deploy/deepseek_v4/tp4_profile.json index 1eb3268..052410e 100644 --- a/deploy/deepseek_v4/tp4_profile.json +++ b/deploy/deepseek_v4/tp4_profile.json @@ -4,7 +4,7 @@ "cache_model_profile": "deepseek-v4-fp8-hma", "published_runtime_base": "ghcr.io/fujitsupolycom/gb10-vllm-serving@sha256:6fc26fdad81a18f0fff67ce0a05f6d90165625ea2e1cac8a6f39bfb462017028", "sparkcache": { - "source_sha256": "15d62a07088d1212bfea60cecae868844cb582a3ac5273189a24532617ac5590" + "source_sha256": "01643ea44598bf5ce2433b41768884ce975006488744a3d02a4600a7c05d6a3d" }, "model": { "repository": "deepseek-ai/DeepSeek-V4-Flash-0731", diff --git a/deploy/glm52_35bpw/TROUBLESHOOTING.md b/deploy/glm52_35bpw/TROUBLESHOOTING.md index 43d11fa..f7c2e16 100644 --- a/deploy/glm52_35bpw/TROUBLESHOOTING.md +++ b/deploy/glm52_35bpw/TROUBLESHOOTING.md @@ -20,6 +20,6 @@ The Spark inter-rank collective layer is abbreviated `SIRCL` below. | The miss semantic gate succeeds, but no durable hit is available after restart. | The HTTP response waits for background cache publication. | Do not restart until every rank logs a matching snapshot and commit digest and its manifest/chunks pass filesystem checks. The miss command returning is not the store barrier. | | A hit returns the right text without a four-rank external restore. | Semantic equality alone proves SparkCache persistence. | After a coordinated restart, require four-rank manifest discovery, the scheduler quorum-hit log, a restore log on every rank, the exact response, and a post-restore canary. | -For the qualification baseline, keep streaming snapshots and native direct +For the qualification baseline, keep streaming snapshots and SparkCache CUDA restore disabled. The bounded rank-local NVMe policy remains 200 GiB high, 180 GiB low, and TTL zero. diff --git a/deploy/glm52_35bpw/profile.json b/deploy/glm52_35bpw/profile.json index cd4ebca..25ca806 100644 --- a/deploy/glm52_35bpw/profile.json +++ b/deploy/glm52_35bpw/profile.json @@ -5,7 +5,7 @@ "published_runtime_base": "ghcr.io/fujitsupolycom/gb10-vllm-serving@sha256:6fc26fdad81a18f0fff67ce0a05f6d90165625ea2e1cac8a6f39bfb462017028", "base_image_requirement": "exact GLM-5.2 3.5-bpw R7 image recorded by the source container inspection", "sparkcache": { - "source_sha256": "15d62a07088d1212bfea60cecae868844cb582a3ac5273189a24532617ac5590" + "source_sha256": "01643ea44598bf5ce2433b41768884ce975006488744a3d02a4600a7c05d6a3d" }, "model": { "repository": "brandonmusic/GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78", diff --git a/deploy/glm53_flash/README.md b/deploy/glm53_flash/README.md index bb3dede..1574cce 100644 --- a/deploy/glm53_flash/README.md +++ b/deploy/glm53_flash/README.md @@ -9,9 +9,26 @@ records: - [`GLM53_FLASH_DFLASH7_LIVE_VALIDATION.md`](../../GLM53_FLASH_DFLASH7_LIVE_VALIDATION.md) records an 8,192-token persistent restore through the Python page-placement path at 147.2--194.0 ms per rank. -- [`GLM53_NATIVE_RESTORE_PERFORMANCE_VALIDATION.md`](../../GLM53_NATIVE_RESTORE_PERFORMANCE_VALIDATION.md) - records SparkCache direct CUDA restore of a 131,072-token prefix, multi-group - recovery, and bounded shared GPU-prefix reuse through C16. +- [`GLM53_SPARKCACHE_CUDA_RESTORE_PERFORMANCE_VALIDATION.md`](../../GLM53_SPARKCACHE_CUDA_RESTORE_PERFORMANCE_VALIDATION.md) + records SparkCache CUDA restore of a 131,072-token full snapshot, multi-group + recovery, and the exact 13-object flat-v2 C1 restart case. Its concurrency + receipts are diagnostic rather than exact-output semantic qualification. + +The flat-v2 semantic result belongs to local image +`sha256:35b58a7bf414059c65b8f74e4e4b17ee6a81b7008e1bffbc9bd298b5e08c739e`, +SparkCache source `a1511d26a1fe2b17b24561bc52e376bf7f54b06a`, and source tree +`4d5b8eb8c5c13793ee7a1e67b2b34bd38fcf4ddb`. SparkCache `229d7d6` is the +flat-v2 schema and header-accounting implementation origin, not the complete +runtime source. Sequential macro-object read/hash consumed 1.35–1.50 seconds +of the 1.55–1.70-second all-rank restore. + +Four-reader flat-v2 prefetch at SparkCache +`eabe7fd0c878db7384ef87fe80a1e96b9bedcf67` is **research-only**. Image +`sha256:df4e09a32cdbf1c0e69cc7c4c9e95d890d6c7a1e3eaac84f969912a16fd27dd3` +structurally verified all four rank-local 131,072-token snapshots but returned +`spark` instead of the required `red`; recomputation returned `red`. It is not +a deployable restore artifact. The executable setup remains pinned to image +`35b58a7…` and its sequential measurements above. Qualification applies only to the checkpoint revisions, source contracts, topology, settings, and immutable artifacts named in those records. It does @@ -23,10 +40,12 @@ The qualified 8,192-token Python-placement artifact is public at digest Its immutable parent and build provenance are recorded in [`IMAGE_ANNOUNCEMENT.md`](IMAGE_ANNOUNCEMENT.md). -Native 131,072-token restore and shared GPU-prefix reuse are qualified only for -the source-bound runtime named in the native validation record. No public OCI -digest carries that runtime. [`PUBLISHING.md`](PUBLISHING.md) requires every -rebuilt digest to complete its own four-rank qualification. +The 131,072-token full-snapshot restore is qualified only for the source-bound +runtime and local image named in the SparkCache CUDA validation record. No +public OCI digest carries that runtime. [`PUBLISHING.md`](PUBLISHING.md) +requires every rebuilt digest to complete its own four-rank qualification. +Tail deltas, host-base read coalescing, and multi-root concurrent restore are +research-only. ## Stored state and prefix behavior @@ -48,13 +67,16 @@ are **implemented** only for `per_token_rows`. Creating an earlier GLM prefix by truncating an opaque page manifest is **unsupported**. Tail-only GLM publication requires a page-semantic format and a distinct cache namespace. -Concurrent requests for the same persistent digest use one restore leader. +The implementation lets concurrent requests for the same persistent digest use +one restore leader. After all workers report successful restoration, patched vLLM retains the leader's normalized multi-group block table as a bounded shared-prefix lease. The implementation permits sixteen waiting followers, two retained leases, and a fifteen-second lease lifetime. Partial physical pages are copied into dedicated immutable blocks before followers can attach. Lease rejection skips -the optimization and lets requests restore or recompute normally. +the optimization and lets requests restore or recompute normally. These are +implementation bounds; the exact-output evidence in the validation record does not qualify +concurrent shared-prefix serving. ## Checkpoint and runtime identity @@ -203,9 +225,15 @@ SparkCache CUDA restore is disabled unless the launch supplies all of: - a 64, 128, or 256 MiB `SPARK_CONTEXT_CACHE_CUDA_PLACEMENT_ARENA_BYTES` value. -The qualified 128K runtime used two host restore workers, two SparkCache CUDA placement -lanes, and two 256 MiB mapped-host arenas per rank. Streaming snapshots remain -unsupported for opaque page storage. +The qualified 128K C1 runtime used two host restore workers, two SparkCache CUDA +placement lanes, and two 256 MiB mapped-host arenas per rank. Streaming +snapshots remain unsupported for opaque page storage. + +At the tested 20 GiB KV-cache setting, C2×128K is an observed capacity +candidate, not a qualified cached workload. C6×128K admitted only one request +at a time and serialized completion over 61–313 seconds. C8×64K and C16×32K +are planned and unqualified. C16×128K is unsupported without GPU trunk sharing +or additional KV capacity. The matching four-Spark launch and artifact-verification procedure is pinned at @@ -234,8 +262,9 @@ python -m deploy.glm53_flash.concurrency_benchmark \ ``` The default fixture reproduces the recorded 131,072-token persistent prefix. -See the SparkCache CUDA restore validation record for exact runtime identities, results, -and committed receipts. +The benchmark records timing and completion, but a serving qualification also +requires the exact-output oracle used by the validation record. See that +record for exact runtime identities, results, and status boundaries. ## Compatibility diff --git a/docs/sparkcache-prefix-explainer.html b/docs/sparkcache-prefix-explainer.html index 165a18a..fafea1e 100644 --- a/docs/sparkcache-prefix-explainer.html +++ b/docs/sparkcache-prefix-explainer.html @@ -3,7 +3,7 @@
- +