Warning
SparkCache is experimental software. Pin package, image, and source revisions when you need a reproducible deployment.
SparkCache gives vLLM persistent memory for previously computed prompts. It saves reusable context to local storage.
SparkCache can restore the longest matching prefix later—even after vLLM evicts that prefix from its GPU KV cache, or after the model process, container, or server restarts.
Repeated conversations, shared system prompts, and growing documents can skip much of the repeated prompt processing.
The cache follows the model's distributed layout: every worker reads and writes only the state owned by its physical rank. Normal cache traffic stays on local storage rather than crossing the network.
Cached model state—the data vLLM needs to continue from a saved context—is restored only after its identity, compatibility, and contents are verified. If those checks do not pass, vLLM computes the prompt normally.
| Capability | In plain language | Status |
|---|---|---|
| Persistent snapshots | Save reusable context as immutable, verified objects. | implemented |
| Independent restore and publication controls | Restore existing entries without capturing new snapshots, or enable either direction separately. | implemented |
| Longest exact-prefix selection | Reuse the longest stored part of a prompt, not only a complete prompt match. | implemented |
| Sparse row-prefix aliases | Point to reusable earlier row boundaries without copying their payloads. | implemented |
| Complete manager-page snapshots | Preserve model-managed pages whose state is not exposed as ordinary rows. | implemented |
| Asynchronous capture with source leases | Keep source pages allocated until every physical worker finishes reading them, including after request completion or cancellation. | implemented; TP4/DCP1 evidence |
| Copy-on-write publication | Store only changed row tails or changed physical pages when extending a stored context. | implemented |
| SparkCache CUDA restore | Move verified page data into request-owned GPU blocks through a C++/CUDA path. | implemented |
| Shared bases and GPU prefixes | Read a common stored base once and let bounded concurrent requests share the restored GPU prefix. | implemented |
| Streaming publication | Gather completed rows while inference continues. | research-only |
| Buddy replication | Copy stored objects to another host for repair. The protocol exists, but no network carrier is included. | research-only |
| Cross-topology reuse | Reuse one stored entry across different physical shard layouts. | unsupported |
Status describes repository behavior. Model, runtime, topology, and live-test details belong to the deployment profiles linked below.
- The scheduler hashes eligible prompt boundaries in one pass.
- Every rank reports which matching entries it can read.
- The scheduler chooses the longest entry available on every expected rank.
- Each worker verifies and restores its local state.
- If any check fails, vLLM computes the prompt normally.
- Completed prefills publish immutable objects before exposing a manifest.
Install the published Python package:
python -m pip install 'sparkcache[connector]==0.1.0a2'PyPI publishes version 0.1.0a2. This repository declares version
0.1.0a3; install from a source checkout when working with that unreleased
repository version.
For repository development:
git clone https://github.com/FujitsuPolycom/sparkcache.git
cd sparkcache
python -m pip install -e '.[test,lint]'The package provides the connector and storage implementation. A working model deployment also needs a compatible vLLM runtime and a deployment profile.
Profiles keep model-specific settings, image identities, launch commands, measurements, and known limits out of the generic cache design.
| Model family | Guide |
|---|---|
| GLM-5.3 Flash, native MTP3 | Four-Spark MTP3 cache/checkpoint quickstart · Implementation and issue status |
| GLM-5.3 Flash, DFlash2 | Four-node DFlash2 quickstart · SparkCache integration notes |
| GLM-5.2 EXL3 3.5-bpw | deploy/glm52_35bpw/README.md |
| DeepSeek-V4 | deploy/deepseek_v4/README.md |
- Cached state reaches inference only after identity, compatibility, all-rank availability, and payload-integrity checks succeed.
- A rejected restore becomes an ordinary cache miss and recomputation.
- Cache work must not delay unrelated serving.
- Immutable objects are written before an atomic manifest exposes an entry.
- Cache identity includes model layout, checkpoint contents, topology, physical rank, storage schema, and page-reuse policy.
- Persistent files never contain CUDA pointers, allocator block tables, or transport sequence numbers.
| Topic | Document |
|---|---|
| Package setup and configuration | sparkcache/README.md |
| CUDA placement and snapshot libraries | sparkcache/native/README.md |
| Generic vLLM capture ownership and source contracts | docs/jj-connector-read-leases.md |
| Whole-prefix restore ownership | docs/PRIVATE_RESTORE_SAFETY.md |
| Interactive prefix explorer | docs/sparkcache-prefix-explainer.html |
| Research ideas and unsupported designs | ROADMAP.md |
| Open correctness defects | DEFECTS.md |
| Security | SECURITY.md |
| Contributing | CONTRIBUTING.md |
python -m pytest sparkcache -q
python -m pytest deploy -q
python -m ruff check .Behavior changes need GPU-free regression tests. Changes to cache identity, digest salts, or stored geometry must make incompatible entries miss cleanly.
SparkCache builds on vLLM and work from the local inference community. Deployment profiles identify the exact upstream runtime, kernels, models, and quantized artifacts they use.
Apache-2.0. See LICENSE.