Expose cache memory budgets, isolate Docker polling, and validate MTP3 reuse

docs/GLM53_JJ_R8_GB10_SPARKCACHE_TP4_QUICKSTART.md

@@ -305,6 +305,11 @@ SPARKCACHE_ACCESS_MODE=read-write   # restore existing entries and publish new o
 SPARKCACHE_ACCESS_MODE=restore-only # restore existing entries; never capture new prompts
 ```
 
+Keep `SPARKCACHE_ASYNC_PAGE_CAPTURE=auto` from the environment template to
+disable capture automatically in restore-only mode or when SparkCache is off.
+An explicit `1` with either setting is rejected. Explicit `0` in read-write
+mode selects synchronous publication.
+
 The GLM-5.3 profile retains a verified shared GPU prefix for up to five
 minutes so one restore can serve an extended request queue:
 
@@ -360,20 +365,34 @@ and `IMAGE_ID`; keep the page-tail settings unchanged.
 The recommended DCP4 profile enables bounded asynchronous page capture:
 
 ```bash
-SPARKCACHE_ASYNC_PAGE_CAPTURE=1
+SPARKCACHE_ASYNC_PAGE_CAPTURE=auto
 SPARKCACHE_ASYNC_CAPTURE_SLOT_BYTES=auto
 SPARKCACHE_ASYNC_CAPTURE_SLOT_COUNT=2
 ```
 
 The `auto` slot policy selects 8 GiB for DCP1, 5 GiB for DCP2, or 3 GiB for
 DCP4. Two capture slots let the background publisher consume one completed
 capture while a later capture uses the other. Restore separately overlaps
-bounded NVMe reads and CUDA placement through two 256 MiB mapped arenas. More
-restore arenas are not part of this profile because measured arena waits did
-not justify the additional unified-memory pressure. DCP1 and DCP2 page-tail
+bounded NVMe reads and CUDA placement through two 256 MiB mapped arenas per
+load lane. Eight lanes reserve 4 GiB per rank; adding the DCP4 capture slots
+gives 10 GiB per rank (40 GiB across TP4) of configured payload capacity,
+before KV, models, metadata, retained objects and transport. Restore-only
+omits the 6 GiB capture ring. DCP1 and DCP2 page-tail
 capture have no matching live record; use complete snapshots or test those
 layouts separately.
 
+Print the resolved memory plan without host or GPU access before launching:
+
+```bash
+SPARKRING_PRINT_MEMORY_PLAN=1 bash runtime/glm53-flash-jj-r8-gb10/launch-rank.sh \
+  0 /path/to/rank.env
+```
+
+Set `SPARKCACHE_BUFFER_BUDGET_BYTES` in that configuration to enforce a
+per-rank restore-plus-capture payload ceiling; zero disables it. The report
+shows KV separately and names excluded memory. It does not predict total
+resident memory or replace a hardware capacity test.
+
 The environment template enables `DFLASH_WARMUP=1`. Rank 0 waits for the API,
 then exercises every concurrency from C1 through C16 and scheduled prompt
 spans covering DFlash's Triton block-size specializations. DFlash depth seven

docs/GLM53_SPARK_MTP3_MESH_QUICKSTART.md

@@ -19,6 +19,13 @@ first. It reuses the shared blank-cluster bootstrap and adds the secondary
 data interfaces, GID/MTU checks, and driver configuration required below.
 Return here to pull the published image and deploy the model.
 
+**Preparing hosts that have already run GPU workloads?** Consider a reboot
+before installation and the first model start, after stopping active workloads.
+It can reduce memory fragmentation even when plenty of RAM appears free.
+The [startup memory gate](../runtime/glm53-spark-mtp3-mesh/MANAGED_MESH.md#automatic-startup-memory-preparation)
+checks available memory and large contiguous free blocks; rebooting does not
+replace that check. A reboot interrupts every workload on that host.
+
 The profile uses the `GLM-5.3-Flash-NVFP4-Spark` target's built-in multi-token
 predictor with three speculative tokens. Graph-native SIRCL handles most
 captured target verification, fused SIRCL handles large eager prefill, and

docs/PREREQUISITES.md

@@ -86,6 +86,15 @@ cluster installer.
 
 The managed profile adds these requirements:
 
+Before installation and the first model start, consider rebooting hosts that
+have run large models or repeated GPU workloads. Stop active workloads first.
+A reboot can reduce unified-memory fragmentation and restore large contiguous
+free blocks; a high total-free-memory reading alone does not establish loader
+readiness. This is a recommendation, not a requirement for every installation.
+The [managed startup memory check](../runtime/glm53-spark-mtp3-mesh/MANAGED_MESH.md#automatic-startup-memory-preparation)
+still applies after reboot and attempts compaction before requiring further
+recovery. Reboots interrupt all workloads on the affected host.
+
 - Four physical cables in the cycle `0-1-2-3-0`, with each rank's port 0/f0
   connected to the next rank's port 1/f1; management stays on a separate LAN.
 - Four configured RDMA functions per host: primary and Socket Direct

performance/harnesses/validation/README.md

@@ -81,3 +81,116 @@ unique-prefix samples per context. It calibrates prompts using `/tokenize`,
 records actual usage and cache evidence, and writes incremental JSONL receipts.
 It requests one output token; its metric is prompt tokens divided by TTFT.
 See the shared runbook for commands and the complete measurement protocol.
+
+## Growing-conversation soak and idle probes
+
+Status: **implemented**. `conversation_soak.py` reuses the prefill probe's
+HTTP and streaming helpers to measure concurrent conversations that grow by a
+calibrated number of prompt tokens each turn. It runs sequential, unique-prefix
+5,300-token probes before and immediately after the conversation workers finish.
+It never changes the serving stack, speculation method, cache access mode, or
+namespace. Select those externally and record their identities with `--metadata`.
+The `--arm` option labels a receipt; it does not configure the server.
+
+Print the request and output bounds without any network activity first. This
+example targets the GLM-5.3 native-MTP3 profile:
+
+```bash
+python3 performance/harnesses/validation/conversation_soak.py \
+  --plan --model glm-5.3-flash-spark --arm baseline-mtp3 --context-limit 1m \
+  --concurrency 2 --start-tokens 32768 --max-turns-per-agent 4 \
+  --tail-tokens 2048 --max-tokens 64 --probe-output-tokens 64 \
+  --duration-seconds 300 --max-soak-prompt-tokens 1000000 \
+  --reset-tokens 160000 --seed 2026090601
+```
+
+Replace `--plan` with `--endpoint http://192.0.2.10:8015` and
+`--output /existing-results-directory/baseline-mtp3.jsonl` to execute. The
+short shape admits at most eight conversation requests and six idle probes.
+Every request is guarded by successful server tokenization and the declared
+context limit. Calibration preserves prior messages and includes the same
+chat-template options used for generation.
+
+For a longer MTP3 run, use `--concurrency 4 --start-tokens 100000
+--max-turns-per-agent 200 --duration-seconds 3600 --max-soak-prompt-tokens
+50000000 --reset-tokens 160000 --max-tokens 512 --probe-output-tokens 300`.
+Keep the remaining options explicit and run `--plan` first. The time limit
+stops admission of new soak requests; calibration, probes, and requests already
+in flight can extend wall time. The HTTP timeout is an inactivity timeout.
+The global token ceiling counts full prompt tokens admitted across all workers,
+including cached tokens, and excludes the separately bounded idle probes.
+Each worker stops when its next prompt cannot fit the remaining shared token
+budget; other workers may still admit smaller prompts. Token or turn limits
+can end the run before the configured duration. Workers rotate to a fresh
+conversation before its next prompt and output would exceed `--reset-tokens`.
+Request errors stop further soak admissions; other in-flight requests can finish.
+
+Use identical seeds, shapes, template settings, MTP3 settings and model
+identities for paired runs. Provide isolated persistent namespaces externally
+when cold independence matters, or explicitly record retained cache state.
+The same seed produces the same initial synthetic prompts, while actual
+assistant replies grow subsequent histories; receipts preserve those replies
+and prompt hashes. Distinct seeds create fresh contexts but do not establish a
+byte-identical paired comparison. Probe prefixes differ between the before and
+after phases; reported cached-token counts remain available to check whether
+either population was already cached. A run name does not isolate GPU prefix
+caching or disk state.
+
+`--image /path/to/fixture.png --image-every 10` adds the same local PNG or JPEG
+to every tenth continuation. The receipt records its SHA-256. Keep the fixture
+alongside the receipt when reproducing a multimodal run. Image bytes are sent
+as a data URL and never fetched from another server. The image must be at most
+4 MiB. Tokenization must account for it; an endpoint that cannot tokenize this
+shape fails calibration instead of using an estimated count.
+
+Each turn records the client request ID sent in `X-Request-ID`, response and
+server request IDs when returned, start time, actual usage, tokenized prompt
+count, prompt hash, assistant text and reasoning, finish reason, elapsed time,
+TTFT, and timestamp offsets for every content or reasoning delta. TTFT runs
+from client submission to the first nonempty content or reasoning delta.
+The decode estimate is `(completion_tokens - 1) / (last_delta - first_delta)`;
+streaming chunks may contain several tokens, so this is not a token-level
+inter-token latency measurement. The full delta timestamps and usage allow
+recalculation. `finish_reason=length` is accepted for the bounded load and is
+recorded as output-budget exhaustion, not a correctness pass.
+
+The summary compares continuation turns with reported cached tokens below or
+above half the prompt, using the condition of a positive prompt increase below
+10,000 tokens in the same conversation. Missing cached-token usage is excluded
+from that classification. These are server-reported numbers: the harness does
+not infer local reuse, external restore, or recomputation from latency. Correlate
+request IDs with connector and vLLM logs to establish those causes. Tokenization
+uses the serving host's CPU and is excluded from each chat latency; its overhead
+and in-flight work affect the interval before the after probes.
+
+Analyze a saved receipt offline with:
+
+```bash
+python3 performance/harnesses/validation/conversation_soak.py \
+  --model glm-5.3-flash-spark --arm analysis --context-limit 1m \
+  --analyze /existing-results-directory/baseline-mtp3.jsonl
+```
+
+JSONL records are flushed after each completion and existing output files are
+never overwritten. API keys come from the environment and are omitted from
+receipts; error bodies are omitted as well. Preserve model/image/source IDs,
+topology, actual KV allocation, access mode, namespace, concurrent traffic and
+initial store occupancy in the supplied metadata or an accompanying evidence
+record. One short successful run does not qualify a long soak or a performance
+improvement. Offline tests use in-memory HTTP fixtures and contact no hosts.
+
+`analyze_conversation_reuse.py` joins a saved receipt to the compact
+`sparkcache-reuse-trace/v1` records in saved rank logs. Supply the expected
+physical ranks explicitly. A verified-restore classification requires a
+successful completion for the same engine request, digest and token span on
+every expected rank; an offer alone is insufficient. Scheduler lease-attachment
+events are reported separately. API-only evidence is labeled `reported_cached`:
+positive counts do not identify local versus external reuse, and zero counts
+do not exclude a GPU lease.
+
+The join accepts recorded request/response IDs and the runtime's exact
+eight-hex-character engine suffix. Ambiguous joins cannot establish a restore
+quorum. The output preserves per-rank queue, service and phase timings without
+summing token spans across ranks. Log captures can contain requests outside the
+receipt window; inspect unmatched IDs and timestamps before assigning them to a
+missing-rank or request failure.