Near-full page store: publish/evict work after an hour of long-context traffic slows subsequent requests ~35 % (5K-prompt decode 21 vs 32 tok/s); disappears in restore-only

[sethforprivacy]

Summary

On a four-GB10 GLM-5.3 TP4/DCP4 ring (sparkring operator image 0d4029b3…, sparkcache 6605717, tail-cow-v2, two 3 GiB async capture slots, SPARKCACHE_MAX_BYTES=40 GiB / low watermark 32 GiB per rank, 300 s lease), an hour of long-context agent traffic leaves the lane measurably slower for requests that run right after it, and the effect disappears with the store off. The page store was near its cap the whole time and evicted ~660 entries during one such hour. Evidence below is from paired probes around three 60-minute soaks (4 concurrent conversations, 75k–300k tokens, ~2k-token tail prefill per turn, thinking on).

Observation

Idle probes (3 × one ~5.3K-token prompt, 300 output tokens, reasoning_effort=low) before and after each soak, per-stream decode tok/s and first-token latency:

lane	before soak	after soak
read-write, NCCL	30.7 / 33.6 / 34.9	30.4 / 21.2 / 25.4 (one first token 4.31 s vs 2.5 s normal)
read-write, SIRCL dual rail	32.0 / 32.0 / 34.2	20.8 / 21.2 / 28.6 (one first token 4.48 s)
restore-only (SPARKCACHE_ACCESS_MODE=restore-only, async capture off), NCCL	34.3 / 29.5 / 29.8	32.1 / 32.6 / 33.4, first tokens 2.26–2.34 s

An inter-token-latency probe (8 sequential 5.3K requests) on a settled lane shows a uniform 73–75 ms decode step in every mode; the slow samples above are ~110 ms/step for the whole request, i.e. something else is using the GPUs/CPU while they run. Store state during the read-write soaks:

before: sparkcache: capacity ranks=4 entries=1251 used=142.3/160.0GiB healthy=yes
after:  sparkcache: capacity ranks=4 entries=592  used=129.2/160.0GiB healthy=yes

so ~660 entries were evicted during the hour. During the soaks, page_delta commits that normally take 0.2–0.7 s intermittently took 5–6 s on every rank (two such commits also preceded our 2026-09-04 22:23Z DCP hang, sparkring #215, though causality there is unproven). Separately, the price of publishing on a settled lane is small and consistent: -10 % first-token latency for a new 5K prompt and -3..-5 % single-prompt prefill at 16K/64K with the store off; decode unchanged.

Asks

1. Is eviction (and manifest/commit work behind it) bounded per scheduler step, or can a near-full store keep the saver busy continuously? The post-soak slowdown suggests the latter.
2. A gauge for eviction/commit backlog (bytes or entries pending, oldest pending commit age) next to the vllm:sparkcache_capture_* ownership gauges from Retain asynchronous store completion across worker polls #57, so operators can see this instead of inferring it from probe latency.
3. Guidance on sizing SPARKCACHE_MAX_BYTES vs the low watermark for a lane whose working set exceeds the store (should the watermark gap be much larger than one hour of publications?).

Happy to rerun with an instrumented build; the probe scripts and per-soak CSVs are in our site repo.

[FujitsuPolycom]

Status: implemented for the publication diagnostics and sizing guidance below. The sustained slowdown reported here is not established resolved; the four-Spark native-MTP3 deployment remains research-only.

1. Is eviction bounded per scheduler step, and can the saver stay busy?

Publication and capacity maintenance run on background worker threads. Serving does not synchronously execute a full eviction pass, and optional saver admission is limited to one publication per rank. That is a concurrency limit, not a CPU, I/O, or elapsed-time budget. Maintenance scans and sorts the inventory, selects roots toward the low watermark, then durably removes roots before collecting unreferenced objects. Successive publications can keep background storage work busy; the implementation does not guarantee an idle interval between passes. This makes resource interference plausible, but does not prove it caused the reported slowdown. See the maintenance implementation.

2. Which backlog gauges are available?

Merged #63 provides these alongside the capture-ownership gauges:

Gauge	Meaning
vllm:sparkcache_publication_pending_rank_slots	Pending saver admissions summed across physical ranks; not unique user requests or bytes.
vllm:sparkcache_publication_oldest_pending_seconds	Oldest admission age, including capture, queueing, commit, and post-commit reconciliation.
vllm:sparkcache_maintenance_active_ranks	Ranks performing an inventory scan or survivor reconciliation.

These are sampled worker reports. They may stop refreshing while the engine is idle; a repeated Prometheus scrape does not advance a cached age. There is no pending-byte or remaining-eviction-entry gauge. Correlate the reports with capture/commit timings and report freshness. Metric definitions and limitations.

3. How should the high/low watermark gap be sized?

There is no requirement for the gap to exceed one hour of publications. For the reported 40 GiB maximum and 32 GiB low watermark, the gap is 8 GiB per rank. Choose capacity from the reusable working set, and choose the gap from the largest admitted publication, measured retained-byte growth, and reclamation rate. As a planning estimate under steady growth, time between pressure events is approximately gap divided by net retained-byte growth rate; deduplication, shared references, and bursty traffic affect that estimate.

A larger gap can reduce scan frequency but removes more retained context per pass, potentially increasing restores or recomputation. It cannot make an oversized working set fit. Tune against measured maintenance frequency/duration and reuse, rather than total logical bytes published in an arbitrary hour. Capacity guidance.

Deployment and evidence: merged #62 and #63 are included in the GLM-5.3 native-MTP3 cache/checkpoint image published through SparkRing #236:

ghcr.io/fujitsupolycom/sparkring-glm53-sparkcache@sha256:11a556a54041fd823d152a7f051ac4f7c617dc539030df26e93008392fee0746

Its eight-check serving record qualifies short correctness checks with 40 GiB configured per rank and periodic full captures disabled, but the store was not filled. The separate 551-response cache-pressure record used 2 GiB per rank, text-only traffic, and enabled periodic captures. Neither is a matched reproduction of this issue.

This issue remains open. The offered probe scripts and per-soak CSVs should define the reproduction and before/after comparison. A maintenance work budget is an implementation option to test against that workload, not an established explanation of the slowdown.

Source implementation awaiting review: draft SparkCache #65 reuses the guarded inventory for survivor reconciliation and preserves concurrently published offers. Its CPU benchmark records reduced metadata work; it does not qualify near-full DGX4 serving. Optional deletion pacing remains disabled by default because smaller batches can increase total scan work. Draft SparkRing #239 prepares the matching instrumented runtime. These changes are not merged or included in the published image identified above.