Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 47 additions & 32 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,45 +1,38 @@
# SparkCache

> [!WARNING]
> **Alpha research software.** Use SparkCache for evaluation. APIs, cache
> formats, deployment patches, and supported profiles may change.
> **Evaluation use only.** Serving support is limited to the exact artifacts,
> models, topologies, and source contracts marked **qualified** below. APIs,
> cache formats, deployment patches, and supported profiles may change.

SparkCache is a persistent, rank-local NVMe context cache for vLLM's
KV-Connector-V1 interface. Each tensor-parallel worker stores and restores only
the model state owned by its physical rank.
KV-Connector-V1 interface. It lets subsequent requests reuse the longest
verified stored prefix instead of repeating the corresponding prefill. Each
tensor-parallel worker stores and restores only the model state owned by its
physical rank.

Sparkcache KV/context Restore reads each rank's local filesystem. SparkCache does not send KV
payload over a network link. vLLM collectives may still use Ethernet, a
switched fabric, or a switchless ring.
Normal restore reads each rank's local filesystem; SparkCache does not send KV
payload over a cache network. vLLM collectives may still use Ethernet, a
switched fabric, or a switchless ring. State that cannot pass identity,
compatibility, all-rank availability, and payload-integrity checks is rejected,
and vLLM computes the request normally.

Deployment labels use TP for tensor-parallel degree and DCP for
decode-context-parallel degree.

PyPI version `0.1.0a3` is **implemented** and GPU-free tested. Its package
artifact has no live serving qualification. Qualification belongs to an exact
artifact, model, topology, and vLLM source contract.

The public GLM-5.3 OCI artifact is qualified for its recorded 8,192-token
Python-placement restore. The source deployment separately qualifies native
131,072-token restore and bounded shared-prefix reuse. See
[the public image record](deploy/glm53_flash/IMAGE_ANNOUNCEMENT.md),
[the GLM-5.3 validation](GLM53_FLASH_DFLASH7_LIVE_VALIDATION.md), and
[the native restore record](GLM53_NATIVE_RESTORE_PERFORMANCE_VALIDATION.md).

The public image does not contain the model checkpoints. Native restore and
shared GPU-prefix qualification belong to a later source-bound runtime that has
no published OCI digest.

## Implemented capabilities

| Capability | Status | Scope |
|---|---|---|
| Content-addressed persistent snapshots | **implemented** | Immutable chunks, manifest-last publication, rank-local capacity control, and verified restore |
| Longest stored exact-boundary discovery | **implemented** | One incremental token-digest pass; longest all-rank candidate wins |
| Sparse row-prefix aliases | **implemented** | Authenticated metadata over `per_token_rows`; GPU-free regression coverage |
| Sparse row-prefix aliases | **implemented** | Authenticated metadata over `per_token_rows`; no live serving qualification |
| Tail-only row publication | **implemented** | Opt-in `tail-cow-v1` storage for `per_token_rows`; no live serving qualification |
| Tail-only opaque-page publication | **implemented** | Authenticated `block_pages_v1` deltas; reconstruction and placement have no live serving qualification |
| Different-root row-segment sharing | **implemented** | Coalesces authenticated `per_token_rows` trunks; no live serving qualification |
| Opaque hybrid-memory-allocator page snapshots | **qualified** | Listed DeepSeek-V4 and GLM deployments only |
| Native direct page restore | **qualified** | Exact GLM-5.3 TP4/DCP1 source deployment recorded below |
| Concurrent shared GPU-prefix reuse | **qualified** | Up to 16 waiting followers, two retained prefixes, 15-second retention; GLM-5.3 through 16 concurrent requests |
| SparkCache C++/CUDA page restore | **qualified** | Flat GLM-5.3 TP4/DCP1 snapshots under the recorded source deployment |
| Concurrent exact-prefix GPU reuse | **qualified** | Up to 16 waiting followers, two retained prefixes, 15-second retention; GLM-5.3 through 16 concurrent requests |
| Streaming snapshots | **research-only** | GLM-5.2 DCP4 inventory; disabled for opaque page profiles |
| Buddy replication | **research-only** | Protocol and receiver state exist; no network carrier is included |

Expand Down Expand Up @@ -73,6 +66,24 @@ Incompatible entries produce a cache miss.
Persistent files contain no CUDA pointers, allocator block tables, physical
slot coordinates, or transport sequence numbers.

## Artifact and qualification scope

PyPI version `0.1.0a3` has GPU-free package validation but no live serving
qualification. Live qualification is bound to an exact artifact, model,
topology, and vLLM source contract.

The public GLM-5.3 OCI artifact is qualified for its recorded 8,192-token
Python/Torch page restore. A separate source deployment qualifies SparkCache
C++/CUDA page restore at 131,072 tokens and bounded exact-prefix GPU reuse. See
[the public image record](deploy/glm53_flash/IMAGE_ANNOUNCEMENT.md),
[the GLM-5.3 validation](GLM53_FLASH_DFLASH7_LIVE_VALIDATION.md), and
[the C++/CUDA restore record](GLM53_NATIVE_RESTORE_PERFORMANCE_VALIDATION.md).

The public image omits model checkpoints. The C++/CUDA restore and shared-prefix
qualification belongs to a source-bound runtime without a published OCI
digest. The adaptive-MTP runtime described in the GLM-5.3 deployment guide is
implemented but requires its own four-rank qualification record.

## Quickstart for qualified deployments

Install the package artifact that matches the intended qualification evidence:
Expand Down Expand Up @@ -147,7 +158,7 @@ timing.
| Prefix and concurrency | Comparison | Recorded result |
|---|---|---|
| 8,192 tokens, C1 | qualified Python page restore | 147.2–194.0 ms cache service per rank |
| 16,384 tokens, C8 | Python/Torch placement vs native placement | 9.45–10.64 s vs 1.2–2.1 s client latency |
| 16,384 tokens, C8 | Python/Torch page restore vs SparkCache C++/CUDA page restore | 9.45–10.64 s vs 1.2–2.1 s client latency |
| 131,072 tokens, C1 | reconstruction pipeline vs cold direct mapped-arena restore | 1.29–1.46 s vs 131–250 ms cache service per rank; a host-warm restore reached 104–165 ms |
| 131,072-token shared prefix, C16 | independent restores vs shared verified GPU blocks | rank-local work fell from 16 × 813 MB to 1 × 813 MB; standard-chat client p50 fell from 3.363 s to 2.980 s |
| 131,072-token shared prefix, pretokenized C16 | standalone measurement | 2.698 s client p50 and 2.701 s maximum |
Expand Down Expand Up @@ -217,8 +228,10 @@ Repeated restores do not rewrite KV payload files. A successful restore may
update manifest recency metadata at most once per minute.

Publishing a reusable context writes immutable payload objects and a manifest.
A growing conversation can publish another complete snapshot, so SparkCache
must not be described as having negligible SSD wear.
The default `snapshot-v1` schema writes a complete grown snapshot. The opt-in
`tail-cow-v1` schema writes immutable row tails or authenticated opaque-page
deltas and periodically compacts bounded delta graphs. SparkCache must not be
described as having negligible SSD wear under either schema.

Operators should monitor the NVMe Data Units Written counter and compare its
daily change with workload publication volume. Device endurance depends on
Expand All @@ -237,7 +250,9 @@ and recovery behavior are derived and tested.
|---|---|---|
| `vllm-project/vllm@fcc614141e5e9ab18cb304c476f7feed2a9552e3` with `patches/vllm/` | **implemented** | Exact patch inputs are published; no standalone public runtime builder is provided |
| vLLM build `e2666d9a6` with `patches/vllm-e2666d9a6/` | **qualified** | DeepSeek-V4 and GLM-5.2 builders verify source, patch, and postimage hashes |
| `local-inference-lab/vllm@da4d7be6c97434f6942292ed8abbf4b32dc44355` with `patches/vllm-da4d7be/` | **qualified** | GLM-5.3 HMA recovery, native restore, and bounded shared-prefix attachment |
| `local-inference-lab/vllm@da4d7be6c97434f6942292ed8abbf4b32dc44355` with `patches/vllm-da4d7be/` | **qualified** | GLM-5.3 HMA recovery, SparkCache C++/CUDA restore, and bounded shared-prefix attachment |
| `local-inference-lab/vllm@e10536aadf02a18fccddda7ec939c33147e8b0b3` with `patches/vllm-e10536a/` | **implemented** | Adaptive-MTP integration and ten-file lease contract; no four-rank qualification |
| `local-inference-lab/vllm@0b67266a0f37d6146a8403fb8482403c62f412d5` with `patches/vllm-glm53-b12x-kda-adaptive-mtp/` | **implemented** | Adaptive MTP, live-tensor B12X KDA, and eleven-file runtime contract; no four-rank qualification |

The GLM-5.3 contract at
[`vllm-kv-block-lease-contract-da4d7be.json`](sparkcache/runtime_patches/vllm-kv-block-lease-contract-da4d7be.json)
Expand All @@ -251,9 +266,9 @@ Native loading requires an explicit library path and SHA-256. CUDA 13 builds
run a GPU-free byte-exact reference test and a CUDA hybrid-page probe before
model-serving qualification.

Native direct restore reads `.spcc` objects into alternating mapped arenas,
hashes complete files in place, validates authenticated extents, and overlaps
read work with CUDA submission.
SparkCache's C++/CUDA restore path reads `.spcc` objects into alternating
mapped arenas, hashes complete files in place, validates authenticated extents,
and overlaps read work with CUDA submission into vLLM-owned cache pages.

## Repository map and development validation

Expand Down