Retain immutable cache pages through connector-job reads

README.md

@@ -31,6 +31,7 @@ If those checks do not pass, vLLM computes the prompt normally.
 | Longest exact-prefix selection | Reuse the longest stored part of a prompt, not only a complete prompt match. | **implemented** |
 | Sparse row-prefix aliases | Point to reusable earlier row boundaries without copying their payloads. | **implemented** |
 | Complete manager-page snapshots | Preserve model-managed pages whose state is not exposed as ordinary rows. | **implemented** |
+| Asynchronous capture with source leases | Keep source pages allocated until every physical worker finishes reading them, including after request completion or cancellation. | **implemented**; [TP4/DCP1 evidence](docs/evidence/connector-job-gb10-tp4-dcp1.md) |
 | Copy-on-write publication | Store only changed row tails or changed physical pages when extending a stored context. | **implemented** |
 | SparkCache CUDA restore | Move verified page data into request-owned GPU blocks through a C++/CUDA path. | **implemented** |
 | Shared bases and GPU prefixes | Read a common stored base once and let bounded concurrent requests share the restored GPU prefix. | **implemented** |
@@ -104,6 +105,8 @@ measurements, and known limits out of the generic cache design.
 |---|---|
 | Package setup and configuration | [`sparkcache/README.md`](sparkcache/README.md) |
 | CUDA placement and snapshot libraries | [`sparkcache/native/README.md`](sparkcache/native/README.md) |
+| Generic vLLM capture ownership and source contracts | [`docs/jj-connector-read-leases.md`](docs/jj-connector-read-leases.md) |
+| Whole-prefix restore ownership | [`docs/PRIVATE_RESTORE_SAFETY.md`](docs/PRIVATE_RESTORE_SAFETY.md) |
 | Interactive prefix explorer | [`docs/sparkcache-prefix-explainer.html`](docs/sparkcache-prefix-explainer.html) |
 | Research ideas and unsupported designs | [`ROADMAP.md`](ROADMAP.md) |
 | Open correctness defects | [`DEFECTS.md`](DEFECTS.md) |

deploy/deepseek_v4/tp4_profile.json

@@ -4,7 +4,7 @@
   "cache_model_profile": "deepseek-v4-fp8-hma",
   "published_runtime_base": "ghcr.io/fujitsupolycom/gb10-vllm-serving@sha256:6fc26fdad81a18f0fff67ce0a05f6d90165625ea2e1cac8a6f39bfb462017028",
   "sparkcache": {
-    "source_sha256": "7efffbec85cce754b7aad057cead39f4b1e1cb7f35833e8df8e487d3d4a46867"
+    "source_sha256": "ce0cacb42427d6fdebb20f613b67fda883973c5522ef5d4d049707e934ba9fe5"
   },
   "model": {
     "repository": "deepseek-ai/DeepSeek-V4-Flash-0731",

deploy/glm52_35bpw/profile.json

@@ -5,7 +5,7 @@
   "published_runtime_base": "ghcr.io/fujitsupolycom/gb10-vllm-serving@sha256:6fc26fdad81a18f0fff67ce0a05f6d90165625ea2e1cac8a6f39bfb462017028",
   "base_image_requirement": "exact GLM-5.2 3.5-bpw R7 image recorded by the source container inspection",
   "sparkcache": {
-    "source_sha256": "7efffbec85cce754b7aad057cead39f4b1e1cb7f35833e8df8e487d3d4a46867"
+    "source_sha256": "ce0cacb42427d6fdebb20f613b67fda883973c5522ef5d4d049707e934ba9fe5"
   },
   "model": {
     "repository": "brandonmusic/GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78",

docs/PRIVATE_RESTORE_SAFETY.md

@@ -0,0 +1,28 @@
+# Whole-prefix restore ownership
+
+Status: **implemented**, covered by GPU-free regression tests. GPU qualification
+of these admission and null-block guards remains required.
+
+Whole-prefix placement writes the complete restored snapshot; its load plan
+does not carry a suffix-only write mask. A request with already-computed local
+tokens can reference pages shared with another request. The connector therefore
+declines external restoration for that request and lets vLLM compute its suffix.
+This can replace an external-cache offer with recomputation, but preserves the
+local prefix and avoids writing through unproven page ownership.
+
+If a restore already owns allocated blocks, repeated lookup waits for its
+completion rather than offering another writer or allowing recomputation into
+those blocks. An unallocated offer is retired if a local prefix appears.
+
+Failed restores report only their nonzero block IDs. Block `0` is shared null
+padding in vLLM, not evidence that another request's cache is damaged. The same
+rule applies when shutdown rejects a queued load before execution.
+
+The guards apply to whole-prefix placement independently of the capture mode
+or model parallelism. Cache identities, stored formats, and native ABIs are
+unchanged. Refusing an unproven destination is a cache miss, not a cache-format
+migration.
+
+Tests: `sparkcache/test_restore_private_admission.py` and
+`sparkcache/test_restore_failure_isolation.py`. The failure tests execute the
+shipped HMA recovery method against failed and unrelated request block tables.

docs/evidence/connector-job-gb10-tp4-dcp1.json

@@ -0,0 +1,218 @@
+{
+  "schema": "sparkcache-connector-job-live-evidence/v1",
+  "status": "qualified-for-listed-cases",
+  "sparkcache_revision": "2bc05bc9e94a4344758e48db36f69a46dafa6946",
+  "vllm_revision": "df62335d8248587f8d3fd1d9a234d1c162a9b84d",
+  "b12x_revision": "0b6d61c37c87ae49d2f9d20d38b9da023146e243",
+  "image": "sha256:cd92adc4436c61290dbecc35362db447c7ae69e1d03bdb99704af0ed6c517b38",
+  "conditions": {
+    "model": "local-inference-lab/GLM-5.3-Flash-NVFP4-Spark",
+    "model_revision": "df116c4fb16b1d37ae43d2cfd624de26ffbc832e",
+    "tensor_parallel_size": 4,
+    "decode_context_parallel_size": 1,
+    "speculative_tokens": 3,
+    "batch_tokens": 8192,
+    "block_tokens": 512,
+    "coalescing": true,
+    "mhc_prefill_sharding": true,
+    "compact_index_cache": false,
+    "capture_slots": 2,
+    "capture_slot_bytes": 536870912
+  },
+  "capture_retirement_cases": [
+    {
+      "kind": "prefill",
+      "index": 0,
+      "native_capture_verified_physical_ranks": [
+        0,
+        1,
+        2,
+        3
+      ],
+      "common_captured_prefixes": [
+        {
+          "tokens": 7680,
+          "context_digest": "5501adf1edc02e57bc007c37d8db64de62032b93d5081f1659fa0506ed2e3cb3"
+        }
+      ],
+      "post_drain_values": {
+        "sparkcache_ranks": 4.0,
+        "sparkcache_capture_pending_rank_slots": 0.0,
+        "sparkcache_capture_completed_rank_slots": 0.0,
+        "sparkcache_capture_delayed_requests": 0.0,
+        "sparkcache_capture_delayed_rank_slots": 0.0,
+        "sparkcache_capture_retained_pages": 0.0,
+        "sparkcache_capture_oldest_delayed_ms": 0.0,
+        "sparkcache_capture_uncertain_ranks": 0.0,
+        "sparkcache_capture_busy_ranks": 0.0,
+        "sparkcache_publication_pending_rank_slots": 0.0
+      }
+    },
+    {
+      "kind": "prefill",
+      "index": 1,
+      "native_capture_verified_physical_ranks": [
+        0,
+        1,
+        2,
+        3
+      ],
+      "common_captured_prefixes": [
+        {
+          "tokens": 15872,
+          "context_digest": "9211bc84d8579ee52822b3d1ef216ca1bc7cdc47e437e069544d8fbcf2682260"
+        }
+      ],
+      "post_drain_values": {
+        "sparkcache_ranks": 4.0,
+        "sparkcache_capture_pending_rank_slots": 0.0,
+        "sparkcache_capture_completed_rank_slots": 0.0,
+        "sparkcache_capture_delayed_requests": 0.0,
+        "sparkcache_capture_delayed_rank_slots": 0.0,
+        "sparkcache_capture_retained_pages": 0.0,
+        "sparkcache_capture_oldest_delayed_ms": 0.0,
+        "sparkcache_capture_uncertain_ranks": 0.0,
+        "sparkcache_capture_busy_ranks": 0.0,
+        "sparkcache_publication_pending_rank_slots": 0.0
+      }
+    },
+    {
+      "kind": "prefill",
+      "index": 2,
+      "native_capture_verified_physical_ranks": [
+        0,
+        1,
+        2,
+        3
+      ],
+      "common_captured_prefixes": [
+        {
+          "tokens": 7680,
+          "context_digest": "340fb240aaeb79c3db0b0bba4eab29c1588ee717c3ee6636b124bfd8c1febf87"
+        }
+      ],
+      "post_drain_values": {
+        "sparkcache_ranks": 4.0,
+        "sparkcache_capture_pending_rank_slots": 0.0,
+        "sparkcache_capture_completed_rank_slots": 0.0,
+        "sparkcache_capture_delayed_requests": 0.0,
+        "sparkcache_capture_delayed_rank_slots": 0.0,
+        "sparkcache_capture_retained_pages": 0.0,
+        "sparkcache_capture_oldest_delayed_ms": 0.0,
+        "sparkcache_capture_uncertain_ranks": 0.0,
+        "sparkcache_capture_busy_ranks": 0.0,
+        "sparkcache_publication_pending_rank_slots": 0.0
+      }
+    },
+    {
+      "kind": "prefill",
+      "index": 3,
+      "native_capture_verified_physical_ranks": [
+        0,
+        1,
+        2,
+        3
+      ],
+      "common_captured_prefixes": [
+        {
+          "tokens": 15872,
+          "context_digest": "b6428e877496ad42b9305eea7368b209171f2c7a9e991c1875a5fe4b30f1409c"
+        }
+      ],
+      "post_drain_values": {
+        "sparkcache_ranks": 4.0,
+        "sparkcache_capture_pending_rank_slots": 0.0,
+        "sparkcache_capture_completed_rank_slots": 0.0,
+        "sparkcache_capture_delayed_requests": 0.0,
+        "sparkcache_capture_delayed_rank_slots": 0.0,
+        "sparkcache_capture_retained_pages": 0.0,
+        "sparkcache_capture_oldest_delayed_ms": 0.0,
+        "sparkcache_capture_uncertain_ranks": 0.0,
+        "sparkcache_capture_busy_ranks": 0.0,
+        "sparkcache_publication_pending_rank_slots": 0.0
+      }
+    },
+    {
+      "kind": "cancellation",
+      "index": 0,
+      "native_capture_verified_physical_ranks": [
+        0,
+        1,
+        2,
+        3
+      ],
+      "common_captured_prefixes": [
+        {
+          "tokens": 16384,
+          "context_digest": "87bb71884a5409ad5a5cfe04bf7fd1a5acd897968514beea97bc09876de55785"
+        }
+      ],
+      "post_drain_values": {
+        "sparkcache_ranks": 4.0,
+        "sparkcache_capture_pending_rank_slots": 0.0,
+        "sparkcache_capture_completed_rank_slots": 0.0,
+        "sparkcache_capture_delayed_requests": 0.0,
+        "sparkcache_capture_delayed_rank_slots": 0.0,
+        "sparkcache_capture_retained_pages": 0.0,
+        "sparkcache_capture_oldest_delayed_ms": 0.0,
+        "sparkcache_capture_uncertain_ranks": 0.0,
+        "sparkcache_capture_busy_ranks": 0.0,
+        "sparkcache_publication_pending_rank_slots": 0.0
+      }
+    },
+    {
+      "kind": "cancellation",
+      "index": 1,
+      "native_capture_verified_physical_ranks": [
+        0,
+        1,
+        2,
+        3
+      ],
+      "common_captured_prefixes": [
+        {
+          "tokens": 16384,
+          "context_digest": "5573a7d3c5cc6914e1e8efe9066f677ecd60cd8cea6f1e1f944defe0cc28a1ec"
+        }
+      ],
+      "post_drain_values": {
+        "sparkcache_ranks": 4.0,
+        "sparkcache_capture_pending_rank_slots": 0.0,
+        "sparkcache_capture_completed_rank_slots": 0.0,
+        "sparkcache_capture_delayed_requests": 0.0,
+        "sparkcache_capture_delayed_rank_slots": 0.0,
+        "sparkcache_capture_retained_pages": 0.0,
+        "sparkcache_capture_oldest_delayed_ms": 0.0,
+        "sparkcache_capture_uncertain_ranks": 0.0,
+        "sparkcache_capture_busy_ranks": 0.0,
+        "sparkcache_publication_pending_rank_slots": 0.0
+      }
+    }
+  ],
+  "persistent_restore_cases": [
+    {
+      "prompt_tokens": 8192,
+      "restored_tokens": 7680,
+      "context_digest": "07eb980ee37d6144707a10511bf39045e8171390eaa0b993302fa46ad78a47ba",
+      "all_four_workers_verified": true,
+      "exact_semantics_passed": true
+    },
+    {
+      "prompt_tokens": 16384,
+      "restored_tokens": 15872,
+      "context_digest": "7fde152f137997bc5636a7c011bfa5db17cd435bbae22fe3fe9a037b4cb01a45",
+      "all_four_workers_verified": true,
+      "exact_semantics_passed": true
+    }
+  ],
+  "raw_receipt_sha256": {
+    "capture_stress": "dd2637e02ccea4601aa5f61534163ea8636e140cd297bba5e363a2a44cdf00f4",
+    "roundtrip": "981a5bc6955fe295280acfe097a4db84e669cbb2eb99fca2218a660aed51e3e5"
+  },
+  "limits": [
+    "Finite text workloads; no direct GPU-timeline or allocator-reference instrumentation.",
+    "The private-restore admission and null-block safeguards added after this tested revision have CPU coverage only.",
+    "The separate standalone JJ recovery head 9b87df5 has CPU/source-contract qualification; this serving run uses df62335.",
+    "No TP2 SparkCache or multimodal qualification is established by these TP4 tests."
+  ]
+}

docs/evidence/connector-job-gb10-tp4-dcp1.md

@@ -0,0 +1,49 @@
+# Asynchronous capture and persistent restore on four GB10 GPUs
+
+Status: **qualified** for the cases and revisions below. The
+[machine-readable record](connector-job-gb10-tp4-dcp1.json) contains physical-rank
+capture evidence, post-retirement counters, restored-prefix digests, and source
+receipt hashes.
+
+## Conditions
+
+The model is `local-inference-lab/GLM-5.3-Flash-NVFP4-Spark`, revision
+`df116c4fb16b1d37ae43d2cfd624de26ffbc832e`, served on four GB10 GPUs with TP4,
+DCP1, three speculative tokens, 8192-token forward batches, and 512-token cache
+blocks. Continuation coalescing and token-sharded mHC prefill are enabled.
+Compact index-cache gathering is disabled. SparkCache uses asynchronous
+connector-job capture with two 512-MiB native slots per physical rank and CUDA
+restore into verified destination pages.
+
+| Component | Revision |
+| --- | --- |
+| SparkCache | `2bc05bc9e94a4344758e48db36f69a46dafa6946` |
+| vLLM | `df62335d8248587f8d3fd1d9a234d1c162a9b84d` |
+| B12X | `0b6d61c37c87ae49d2f9d20d38b9da023146e243` |
+| Container image | `sha256:cd92adc4436c61290dbecc35362db447c7ae69e1d03bdb99704af0ed6c517b38` |
+
+## Results
+
+| Check | Measurement | Result |
+| --- | --- | --- |
+| Request completion | Four cold 8K/16K requests with one generated token | All four ranks completed native capture; pending work, delayed work, retained pages, and uncertain reads returned to zero after each case. |
+| Request cancellation | Two active 8K/16K requests cancelled during generation | The engine logged cancellation; all four ranks completed capture retirement and the same ownership counters returned to zero. |
+| Persistent 8K restore | Captured prefix loaded after all model processes restarted | 7680 tokens restored; every rank matched the stored prefix digest; the exact-answer check passed. |
+| Persistent 16K restore | Captured prefix loaded after all model processes restarted | 15872 tokens restored; every rank matched the stored prefix digest; the exact-answer check passed. |
+
+The results establish that capture can outlive request completion or cancellation
+without retaining source pages indefinitely, and that these persisted prefixes
+remain usable across a full model-process restart. Tail recomputation accounts
+for the difference between prompt length and restored length.
+
+## Limits
+
+The additional guards described in [whole-prefix restore
+ownership](../PRIVATE_RESTORE_SAFETY.md) have CPU coverage but were not installed
+in this image. Their integrated GPU validation remains required. The standalone
+JJ recovery revision `9b87df5d47b9c7163d1105ac5ea8c0a088baafc9` has CPU and source-contract
+coverage; this record does not qualify that different composition.
+
+These checks do not qualify SparkCache on TP2, cross-composition state reuse,
+multimodal requests, arbitrary concurrent eviction, or every native failure
+condition. They make no throughput or model-quality claim.

docs/evidence/connector-job-integration-cpu.json

@@ -0,0 +1,48 @@
+{
+  "schema": "sparkcache-integration-cpu-validation/v1",
+  "status": "qualified",
+  "conditions": {
+    "platform": "Windows",
+    "python": "3.12",
+    "torch": "2.11.0+cpu",
+    "source_sha256": "e5930ba01d0974896108bbf29bc92913a6a6473ca0bfd913e5e8726fc12f27b3"
+  },
+  "commands": [
+    {
+      "command": "python -m pytest sparkcache research deploy -q -rs",
+      "passed": 1305,
+      "skipped": 9,
+      "seconds": 34.43,
+      "scope": "CPU; platform/compiler/source-fixture skips remain for Linux CI"
+    },
+    {
+      "command": "python -m ruff check .",
+      "result": "passed"
+    },
+    {
+      "command": "python -m build",
+      "result": "wheel and source distribution built"
+    },
+    {
+      "command": "python -m twine check <distributions>",
+      "result": "passed"
+    },
+    {
+      "command": "python tools/verify_distribution.py <wheel> <sdist> --version 0.1.0a3",
+      "result": "passed, including isolated wheel installation and required connector modules/contracts"
+    }
+  ],
+  "standalone_jj_contract": {
+    "vllm_revision": "9b87df5d47b9c7163d1105ac5ea8c0a088baafc9",
+    "matched_committed_source_files": 10,
+    "source_bytes": "Canonical LF Git blobs"
+  },
+  "gpu_qualification": "The additional private-restore admission and null-block guards require integrated GPU qualification; live capture evidence names SparkCache 2bc05bc9e94a4344758e48db36f69a46dafa6946.",
+  "common_glm_contract": {
+    "vllm_revision": "17bd258075f44dda8b405f384732f3c78d03f308",
+    "matched_committed_source_files": 10,
+    "source_bytes": "Canonical LF Git blobs; connector files match df62335 exactly",
+    "vllm_cpu_passed": 162,
+    "gpu_qualified": false
+  }
+}