read-write: growing conversations lose local prefix-cache hits on ~41 % of continuation turns and are served by restores (23 s) instead; hot turns 37 s vs 15 s in restore-only

[sethforprivacy]

Summary

In read-write mode with tail-cow-v2 + async page capture, a growing multi-turn conversation loses its local (GPU) prefix-cache hits on a large share of continuation turns; those turns are then served by a SparkCache restore of the conversation's own earlier turns (published minutes before) at 10-30 s per turn instead of a ~1 s local hit. In restore-only mode the same conversation shape keeps 87-97 % local hits. Paired 60-minute soaks with fresh, never-published conversations on the same lane, same day, same launcher (four GB10, GLM-5.3 TP4/DCP4, image 0d4029b3…, sparkcache 6605717, 300 s lease, 4.32M-token KV pool never above 28 % used).

Setup

Replay: 4 concurrent agents, each a conversation starting at 106-340K tokens and growing ~2K tokens per turn (assistant reply + "tool result"), thinking on, 512-token replies, a small image every 10th turn, new conversation after 160K. Fresh seed per arm, so no cross-run restores are possible. usage.prompt_tokens_details.cached_tokens is recorded per turn (it reports local prefix-cache hits in this build).

Result

	read-write	restore-only (SPARKCACHE_ACCESS_MODE=restore-only, async capture off)
turns in 60 min / prompt tokens	263 / 36.9M	195 / 26.6M
continuation turns (same conversation, +<10K tokens)	247	183
… with cached_tokens < 50 % of prompt	102 (41 %), median 23.4 s	30 (16 %), median 166 s
… with ≥ 50 % local cache	145, median 37.0 s	153, median 15.2 s
post-soak idle 5K-prompt decode (3 probes)	34.4 / 26.4 / 20.5 tok/s	35.2 / 30.5 / 29.4 tok/s

One agent's trace in read-write (prompt size / local cached % / latency):

t1 105k/0%/355s  t2 107k/96%/93s  t3 109k/97%/32s … t8 118k/98%/19s
t9 120k/0%/15s  t10 122k/0%/11s  t11 124k/0%/21s … t22 145k/0%/30s
t23 147k/98%/16s  t24 149k/98%/15s  t25 151k/0%/10s … t30 161k/0%/37s

A 120-145K prompt cannot be recomputed in 11-30 s on this lane (restore-only's genuine misses take ~166 s under the same load), so the 0 % turns were restored from the page store: the local prefix cache no longer held the conversation's previous turns although KV usage stayed under 28 % and the 300 s lease had not expired between turns (turns arrive every 15-40 s). The same agent shape in restore-only stays at 87-97 % local hits.

Two effects follow: (1) hot continuation turns are 2.5× slower in read-write (37 s vs 15 s), which on this traffic outweighs the per-publish cost measured on an idle lane (-10 % first token for a new 5K prompt); (2) the post-soak slowdown reported in #60. Read-write still completes more turns overall because it rescues genuinely new/evicted contexts (restore-only's 30 misses at 166 s), which is the value of the store — but the rescued case should be the exception, not 41 % of continuation turns.

Question / ask

How are a finished request's KV blocks returned to vLLM after the async capture completes (delayed free / get_finished)? If they are freed without their block hashes staying in the local prefix cache, or if the lease/COW ownership makes the next turn's lookup miss locally, that would explain the trace. A per-request counter of "local hit vs external restore vs recompute" tokens in the connector's metrics would make this visible without a replay harness.

Data (per-turn logs, CSVs, probe JSON) in our site repo docs/weekend-2026-09-05/evidence/soak-fair-*; happy to rerun with an instrumented build.

[FujitsuPolycom]

Status: implemented for the runtime changes listed below. Resolution of the local-prefix loss reported in this issue remains unverified; the four-Spark native-MTP3 deployment remains research-only.

SparkRing #236 integrates GPU-lease accounting, preference for longer locally available prefixes, and recurrent checkpoint retention from SparkRing #227. The runtime source-transform contract describes those changes. Installing SparkCache's Python package alone does not install them.

The GLM-5.3 native-MTP3 cache/checkpoint image includes SparkCache revision 48bbd2be4a7b972e56632a2d7b934bac5460f272. Its immutable registry reference is:

ghcr.io/fujitsupolycom/sparkring-glm53-sparkcache@sha256:11a556a54041fd823d152a7f051ac4f7c617dc539030df26e93008392fee0746

Use its matching deployment guide. SparkRing #227's code is incorporated through #236, so that component PR is not a separate installation prerequisite.

Qualified evidence: the exact-image serving record contains a 5,628-token conversation start and a 5,650-token continuation. Both returned the requested word correctly; the continuation reported 4,608 cached tokens. That API value does not establish whether reuse came from GPU retention or persistent restoration. Repeated red/blue/red image requests also passed, but their 223-token prompts were below the persistent-cache minimum span. Those results establish bounded response correctness, not resolution of the long-context local-retention regression.

Remaining work: opt-in reuse traces cover external restore offers, verified worker completion, and GPU lease attachment. Their scheduler prefix-token field is block-aligned input, not an exact local hash-hit count. The merged source and published image do not provide exact per-request local-hit, verified-restore, and recompute token counters.

This issue remains open for that diagnostic gap and validation of the reported long-context growing conversations with occasional images. The existing small-cache soak and short smoke checks should be retained as regression evidence without treating them as a matched reproduction of this issue.

Source implementation awaiting review: draft SparkCache #65 and draft SparkRing #239 implement the missing per-request attribution using authoritative scheduler events. They distinguish consumed local prefixes, verified external reuse, and accepted target-prompt computation across preemption attempts, with incomplete observations labeled explicitly. CPU tests cover failed/cancelled restores, shared leases, stale completion, preemption, and allocation retries. The changes are not merged or included in the published image identified above; instrumented serving validation and the original-workload reproduction remain pending.