Reduce maintenance rescans and attribute accepted prompt reuse

deploy/deepseek_v4/tp4_profile.json

@@ -4,7 +4,7 @@
   "cache_model_profile": "deepseek-v4-fp8-hma",
   "published_runtime_base": "ghcr.io/fujitsupolycom/gb10-vllm-serving@sha256:6fc26fdad81a18f0fff67ce0a05f6d90165625ea2e1cac8a6f39bfb462017028",
   "sparkcache": {
-    "source_sha256": "72f33311d1ee5811c2b3e4eefda98c9d17b5afe8d9d015a2e3c26c893f25aabc"
+    "source_sha256": "7efffbec85cce754b7aad057cead39f4b1e1cb7f35833e8df8e487d3d4a46867"
   },
   "model": {
     "repository": "deepseek-ai/DeepSeek-V4-Flash-0731",
@@ -23,7 +23,7 @@
     "kv_cache_bytes_per_rank": 34359738368,
     "max_model_len": 524288,
     "max_num_seqs": 32,
-    "gpu_memory_utilization": 0.70,
+    "gpu_memory_utilization": 0.7,
     "tokenizer_mode": "deepseek_v4",
     "speculation_method": "dspark",
     "speculation_tokens": 5,

deploy/glm52_35bpw/profile.json

@@ -5,7 +5,7 @@
   "published_runtime_base": "ghcr.io/fujitsupolycom/gb10-vllm-serving@sha256:6fc26fdad81a18f0fff67ce0a05f6d90165625ea2e1cac8a6f39bfb462017028",
   "base_image_requirement": "exact GLM-5.2 3.5-bpw R7 image recorded by the source container inspection",
   "sparkcache": {
-    "source_sha256": "72f33311d1ee5811c2b3e4eefda98c9d17b5afe8d9d015a2e3c26c893f25aabc"
+    "source_sha256": "7efffbec85cce754b7aad057cead39f4b1e1cb7f35833e8df8e487d3d4a46867"
   },
   "model": {
     "repository": "brandonmusic/GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78",

deploy/glm53_flash/MTP3_STATUS.md

@@ -29,21 +29,35 @@ identified above. It requires a rebuilt image and startup validation.
 
 ## What remains in the performance issues
 
+Source builds include request-attribution accounting, same-pass maintenance
+inventory reuse, and optional deletion pacing. The immutable image identified
+above does not contain these additions. Attribution also requires the matching
+instrumented SparkRing scheduler; a package update alone cannot supply its events.
+
 | Issue | Implemented | Remaining work |
 |---|---|---|
-| [#60: sustained publication and eviction slowdown](https://github.com/FujitsuPolycom/sparkcache/issues/60) | Fewer repeated restore reads and hashes, bounded restore arenas, tiled native placement, publication-dependency protection, publication-backlog gauges, optional periodic full captures, and capacity guidance. | Maintenance still scans and sorts the inventory and evicts toward the low watermark; it has no per-pass time or entry budget. Reproduce the reported slowdown with a near-full 40 GiB store and matched before/after probes. |
-| [#61: growing conversations lose local prefix reuse](https://github.com/FujitsuPolycom/sparkcache/issues/61) | Runtime GPU-lease accounting, preference for longer local prefixes, recurrent checkpoint retention, and opt-in restore/lease traces. | Exact per-request local-hit, verified-restore, and recompute token counters are not implemented. Validate local retention on the reported long-context, multi-turn workload with occasional images. |
+| [#60: sustained publication and eviction slowdown](https://github.com/FujitsuPolycom/sparkcache/issues/60) | Reduced restore work, publication-dependency protection, backlog gauges, capacity guidance, same-pass inventory reuse, and opt-in deletion-attempt limits and cooldown. | Full inventory scans and individual filesystem operations still have no hard time bound. Validate the combined implementation against near-full 40 GiB traffic and matched before/after probes. |
+| [#61: growing conversations lose local prefix reuse](https://github.com/FujitsuPolycom/sparkcache/issues/61) | Runtime lease accounting, local-prefix preference, checkpoint retention, and opt-in request attribution for accepted target execution. | Deploy the matching scheduler instrumentation and validate attribution and local retention on the reported long-context conversations with occasional images. |
 
 One saver admission per rank bounds concurrent optional work; it does not bound
-the duration of an inventory scan or eviction pass. See
+the duration of an inventory scan. Deletion pacing is disabled by default.
+Smaller deletion budgets can repeat scans and increase total work. See
 [capacity and cleanup](../../sparkcache/README.md#capacity-and-cleanup).
 Backlog reports distinguish pending publications, their oldest age, and active
 maintenance, but reports may not refresh while the connector is idle.
 
-Reuse traces distinguish restore offers, verified worker completion, and GPU
-lease attachment. Their scheduler prefix-token field is block-aligned input,
-not an exact local hash-hit measurement. API `cached_tokens` alone is not enough
-to attribute reuse to GPU retention rather than persistent restoration.
+Matching traces distinguish restore offers, worker completion, and GPU lease
+attachment. Their scheduler prefix-token input and API `cached_tokens` alone
+cannot establish exact attribution.
+
+The [request-attribution record](../../sparkcache/README.md#request-reuse-attribution)
+uses authoritative scheduler events to distinguish consumed local prefixes,
+verified external reuse, and accepted prompt computation. Counts accumulate
+across preemption attempts; incomplete observations are labeled explicitly.
+
+The [maintenance inventory benchmark](../../docs/maintenance-inventory-validation.md)
+measures reduced metadata work on CPU fixtures. It is not a DGX4 serving result
+or confirmation that the original slowdown is resolved.
 
 Both issues should remain open until their remaining implementation questions
 and workload-specific results are recorded explicitly.

docs/maintenance-inventory-validation.json

@@ -0,0 +1,137 @@
+{
+  "schema": "sparkcache-maintenance-inventory-benchmark/v1",
+  "status": "research-only",
+  "conditions": {
+    "platform": "Windows-11-10.0.26200-SP0",
+    "filesystem": "NTFS",
+    "filesystem_verification": "Get-Volume -DriveLetter C",
+    "python_execution": "native Windows",
+    "paired_method": "Separate sequential processes constructed equivalent temporary caches and invoked SparkContextCacheConnector._maintain_capacity(force=True).",
+    "timing": "Median of seven complete maintenance/reconciliation passes after an untimed instrumented pass; garbage collection precedes each timer.",
+    "excluded": [
+      "fixture construction",
+      "publication",
+      "Python imports",
+      "I/O counting instrumentation"
+    ],
+    "included": [
+      "exclusive cache lock",
+      "complete reference inventory",
+      "offer reconciliation"
+    ],
+    "eviction": "disabled by a 1000000000-byte capacity ceiling",
+    "gpu_execution": false,
+    "serving_or_nvme_qualification": false,
+    "tool": "tools/benchmark_maintenance_inventory.py"
+  },
+  "sources": {
+    "baseline": {
+      "revision": "7fc9f85509624a0f9c5bc6bd7c5f70f1a43d389e",
+      "cache_manifest_sha256_lf": "88caa5b104407a212b4344b82ab914d6c5c7e72d021976ba6d1c44d13f7f2f2e",
+      "connector_sha256_lf": "082f2820f45cdf951629d045b5d316461f86372fae70425be9b5903825395a97",
+      "revision_matches_recorded_source_hashes": true
+    },
+    "survivor_inventory": {
+      "mechanism": "MaintenanceReport.surviving_entries supplies metadata-qualified offers from the exclusive inventory pass.",
+      "cache_manifest_sha256_lf": "ddb3e2e00808791ed585122b8ab1171ac7db391c19fc62151f0f39107dee256a",
+      "connector_sha256_lf": "f92b1fcaf3385e5d4487b246f06c1de3fb02e8f05b67c055ab760d91ccd86eb4"
+    }
+  },
+  "flat_page_history": {
+    "layout": "Opaque 64-byte attention pages grouped by 256 tokens, plus a 32-byte recurrent-state page.",
+    "branches": 8,
+    "extensions_per_branch": 32,
+    "shared_initial_tokens": 512,
+    "extension_tokens": 256,
+    "roots": 257,
+    "unique_chunks": 257,
+    "chunk_references": 4481,
+    "baseline": {
+      "root_reads": 514,
+      "chunk_stats": 4995,
+      "elapsed_ms": [
+        185.0543999898946,
+        183.6131000018213,
+        173.09400001249742,
+        178.17310000828002,
+        174.59690000396222,
+        174.0127999946708,
+        175.22859999735374
+      ],
+      "median_ms": 175.22859999735374
+    },
+    "survivor_inventory": {
+      "root_reads": 257,
+      "chunk_stats": 514,
+      "elapsed_ms": [
+        72.49789999332279,
+        70.9155000076862,
+        72.2200000018347,
+        70.33850000880193,
+        76.28729999123607,
+        89.75840000493918,
+        72.21110000682529
+      ],
+      "median_ms": 72.2200000018347
+    },
+    "conclusion": "Complete pass latency decreased 58.8 percent in this CPU fixture. Each root was read once; shared-chunk metadata probes scaled with unique files rather than root references. This does not establish a serving-throughput improvement."
+  },
+  "token_rows": {
+    "roots": 64,
+    "unique_chunks": 32,
+    "chunk_references": 2048,
+    "baseline": {
+      "root_reads": 128,
+      "chunk_stats": 2112,
+      "elapsed_ms": [
+        37.09349999553524,
+        37.26160000951495,
+        37.37550000369083,
+        37.524400002439506,
+        37.23980000359006,
+        41.24779999256134,
+        52.597900008549914
+      ],
+      "median_ms": 37.37550000369083
+    },
+    "survivor_inventory": {
+      "root_reads": 64,
+      "chunk_stats": 64,
+      "elapsed_ms": [
+        10.044100010418333,
+        10.23929999792017,
+        10.25409999419935,
+        10.973099997499958,
+        10.138300000107847,
+        9.93929999822285,
+        9.973099993658252
+      ],
+      "median_ms": 10.138300000107847
+    }
+  },
+  "combined_source_repeat": {
+    "revision": "11658f0a3b9f155af1d95ff78a807c9b374f6093",
+    "kind": "pages",
+    "roots": 257,
+    "unique_chunks": 257,
+    "references": 4481,
+    "platform": "Windows-11-10.0.26200-SP0",
+    "filesystem": "NTFS",
+    "manifest_source_sha256": "f2cc0bc7570b59ed068ad3273b234c0f8c9bd945a6a68dca0014012a118d1400",
+    "connector_source_sha256": "f92b1fcaf3385e5d4487b246f06c1de3fb02e8f05b67c055ab760d91ccd86eb4",
+    "io_counts": {
+      "root_reads": 257,
+      "chunk_stats": 514
+    },
+    "elapsed_ms": [
+      76.01800000702497,
+      72.13889999547973,
+      73.28540000889916,
+      82.84220000496134,
+      89.14029999868944,
+      79.15170000342187,
+      78.41010000265669
+    ],
+    "median_ms": 78.41010000265669
+  }
+}

docs/maintenance-inventory-validation.md

@@ -0,0 +1,54 @@
+# Maintenance inventory and offer reconciliation
+
+Status: **implemented**. Serving-performance qualification is **research-only**.
+
+`MaintenanceReport.surviving_entries` identifies metadata-qualified roots from
+the exclusive inventory pass. Offer reconciliation reuses that result without
+rereading roots or probing shared payload files once per reference.
+
+The [CPU benchmark record](maintenance-inventory-validation.json) measures the
+complete maintenance and reconciliation call on Windows 11 with NTFS. It
+compares equivalent fixtures against source revision `7fc9f85509624a0f9c5bc6bd7c5f70f1a43d389e`.
+
+| Fixture | Baseline median | Inventory median | Root reads | Chunk metadata probes |
+|---|---:|---:|---:|---:|
+| Eight branches, 32 flat page extensions each | 175.23 ms | 72.22 ms | 514 to 257 | 4,995 to 514 |
+| 64 token-row roots sharing 32 chunks | 37.38 ms | 10.14 ms | 128 to 64 | 2,112 to 64 |
+
+A repeat with combined source `11658f0a3b9f155af1d95ff78a807c9b374f6093`,
+including canonical string ordering, measured 78.41 ms for the flat-page fixture
+and retained the same reduced I/O counts. The JSON records that source and its
+seven observations separately.
+
+The page fixture includes attention and recurrent-state bytes. It exercises
+257 roots and 4,481 chunk references with tiny payloads, not a loaded model.
+The measurements do not qualify DGX serving throughput or NVMe behavior.
+
+Each normal pass still reads the complete inventory and traverses its reference
+graph. Root parsing falls from two passes to one, and chunk metadata probes
+scale with unique files instead of the total number of root references.
+
+Run the [benchmark tool](../tools/benchmark_maintenance_inventory.py) from the
+checkout being measured. Use the same tool against each source checkout.
+Install the repository's CPU test dependencies; the connector fixture stubs
+its model-runtime interfaces.
+
+```powershell
+$env:BENCH_FILESYSTEM = 'NTFS'
+$env:BENCH_KIND = 'pages'
+python tools/benchmark_maintenance_inventory.py 8 32
+$env:BENCH_KIND = 'rows'
+python tools/benchmark_maintenance_inventory.py 64 32
+```
+
+Set `BENCH_FILESYSTEM` to the independently verified filesystem. Fixture
+construction and counting instrumentation are outside the seven timed passes.
+The artifact records normalized source hashes for both measured implementations.
+
+Regression tests in `sparkcache/test_maintenance_survivors.py` cover shared
+payload metadata, corrupt exact roots shadowing aliases, storage-mode
+eligibility, concurrent publication, and repeated inventory freshness.
+
+The snapshot does not authorize cache restoration. Payload integrity remains
+mandatory at restore, and same-sized corruption becomes a verified miss.
+No cache identity, namespace, or persisted format changes.

sparkcache/README.md

@@ -311,6 +311,96 @@ A larger gap amortizes maintenance across more writes, but each pass evicts
 more data and can increase future misses. Compare those costs with observed
 publication age and maintenance activity before changing the gap.
 
+### Deletion pacing
+
+Status: **implemented**; serving performance with these controls is
+**research-only**. Both settings default to `0`, preserving unrestricted passes:
+
+- `spark_cache_maintenance_max_deletions` caps filesystem unlink attempts per
+  pass, including failed attempts, across manifests, aliases, debris,
+  descriptor segments, and payload objects. Its environment fallback is
+  `SPARK_CONTEXT_CACHE_MAINTENANCE_MAX_DELETIONS`.
+- `spark_cache_maintenance_interval_ms` sets a minimum cooldown after a pass
+  finishes or fails. Forced post-commit calls also respect it; skipped calls
+  do not extend it. Its environment fallback is
+  `SPARK_CONTEXT_CACHE_MAINTENANCE_INTERVAL_MS`.
+
+Explicit connector settings take precedence over environment values. A test
+configuration can select `128` deletion attempts and `1000` milliseconds;
+these values are not a qualified serving-performance recommendation.
+
+With a deletion budget, existing orphan payloads and debris are reclaimed
+before additional roots are selected.
+
+A pass can stop above the low watermark
+or the capacity maximum; optional store admission remains blocked while
+capacity is unsatisfied.
+
+A process-local pressure target retains the low watermark across passes,
+even after usage falls below the high watermark. Restarting the process
+loses this target; a later high-watermark crossing establishes it again.
+
+Background retries complete deferred cleanup without
+making serving wait. Root-directory durability barriers still precede object
+removal, and protected publication roots retain their complete object graphs.
+
+This is **not a scan-size or wall-clock bound**. Each admitted pass authenticates
+the complete reference inventory and reconciles survivors.
+
+Status: **implemented**. A completed pass also returns metadata-qualified
+surviving roots while the exclusive filesystem guard is held.
+
+The connector
+uses this inventory to reconcile offers without rereading every root or
+restatting shared chunks once per reference.
+
+Qualification checks logical
+file sizes, not allocated disk space. Exact roots shadow aliases even when
+invalid; aliases are eligible only for token-row storage.
+
+Protected roots and
+their referenced objects retain the same deletion rules.
+
+The inventory is not retained between passes. Failed or busy passes use
+independent metadata probes.
+
+If the offered-digest inventory changes during a
+pass or probe, reconciliation defers withdrawals to a stable pass, preserving
+concurrent publication of the same digest.
+
+The connector counter
+`capacity_stale_inventory_snapshots` and worker capacity-report field
+`maintenance_stale_inventory_snapshots` count these deferrals.
+
+Restore still
+authenticates payload bytes; metadata qualification cannot establish their
+integrity or turn an invalid restore into a hit.
+
+One filesystem
+operation or durability barrier can take arbitrarily long. Smaller deletion
+budgets can increase total scan work.
+
+Cooldown trades reclamation throughput
+for gaps between that work. A larger watermark gap does not remove this cost.
+
+`MaintenanceReport.deletion_attempts` counts admitted unlink attempts;
+`work_pending` reports cleanup deferred by the budget or orphan-first policy;
+`skipped_cooldown` reports a pass skipped before inventory work.
+
+Connector
+counters `capacity_deletion_attempts`, `capacity_budget_exhausted`, and
+`capacity_skipped_cooldown` expose the same activity. Capacity log records
+include `deletion_attempts` and `work_pending`.
+
+Prometheus gauges
+`vllm:sparkcache_maintenance_deletion_attempts`,
+`vllm:sparkcache_maintenance_budget_exhausted`, and
+`vllm:sparkcache_maintenance_skipped_cooldown` sum reported cumulative counts
+across ranks.
+
+They reset with workers and reflect the last worker reports,
+not independent scrape-time measurements.
+
 `spark_cache_ttl_seconds` expires manifests by recency; zero disables TTL.
 Maintenance preserves shared objects referenced by surviving manifests.
 
@@ -424,6 +514,39 @@ The counters describe host-side operations. They do not report filesystem
 allocation, NVMe Data Units Written, controller write amplification, or NAND
 writes.
 
+### Request reuse attribution
+
+Status: **implemented** with an optional scheduler callback. A runtime without
+that callback cannot produce exact request attribution from connector offers.
+
+Set `SPARK_CONTEXT_CACHE_TRACE_REUSE=1` before startup. An instrumented scheduler
+emits one `request_cache_attribution` event in `sparkcache-reuse-trace/v1` at
+request cleanup. No request IDs are added to Prometheus labels.
+
+| Field | Meaning |
+|---|---|
+| `local_tokens_reused` | GPU-resident prompt tokens consumed by accepted target execution, including resident shared leases. |
+| `external_tokens_reused` | Prompt tokens consumed after a successful all-rank persistent restore and the scheduler's final-token adjustment. |
+| `prompt_tokens_computed` | Prompt intervals completed by accepted target execution, accumulated across preemption attempts. |
+| `preemptions` | Request preemption generation observed by the scheduler. |
+| `attribution_complete` | True only for normal completion with observed prompt completion and no missing or invalid accounting boundary. |
+
+Counts cover accepted target-prompt work across attempts. They can exceed the
+original prompt length after preemption. They exclude output tokens, draft
+execution, replay inside kernels, and rejected worker output.
+
+An offered restore earns no credit. A verified restore aborted before target
+execution earns no reused-token credit. A follower consuming a resident GPU
+lease records local reuse, even if a different request restored that lease.
+
+The restored state span and external prompt tokens reused are distinct. A
+restore can write an already-local prefix, and a full prompt hit still needs
+the final prompt token recomputed for sampling logits.
+
+Incomplete observations remain labeled incomplete. Their token fields are not
+added to the connector's `attribution_completed_*` aggregate counters. Request
+cleanup releases the ledger even when logging fails.
+
 Telemetry is observational. It cannot change publication, restore, cache
 identity, or serving decisions.