diff --git a/docs/GLM53_JJ_R8_GB10_SPARKCACHE_TP4_QUICKSTART.md b/docs/GLM53_JJ_R8_GB10_SPARKCACHE_TP4_QUICKSTART.md index 66e93de2..8892c6e3 100644 --- a/docs/GLM53_JJ_R8_GB10_SPARKCACHE_TP4_QUICKSTART.md +++ b/docs/GLM53_JJ_R8_GB10_SPARKCACHE_TP4_QUICKSTART.md @@ -305,6 +305,11 @@ SPARKCACHE_ACCESS_MODE=read-write # restore existing entries and publish new o SPARKCACHE_ACCESS_MODE=restore-only # restore existing entries; never capture new prompts ``` +Keep `SPARKCACHE_ASYNC_PAGE_CAPTURE=auto` from the environment template to +disable capture automatically in restore-only mode or when SparkCache is off. +An explicit `1` with either setting is rejected. Explicit `0` in read-write +mode selects synchronous publication. + The GLM-5.3 profile retains a verified shared GPU prefix for up to five minutes so one restore can serve an extended request queue: @@ -360,7 +365,7 @@ and `IMAGE_ID`; keep the page-tail settings unchanged. The recommended DCP4 profile enables bounded asynchronous page capture: ```bash -SPARKCACHE_ASYNC_PAGE_CAPTURE=1 +SPARKCACHE_ASYNC_PAGE_CAPTURE=auto SPARKCACHE_ASYNC_CAPTURE_SLOT_BYTES=auto SPARKCACHE_ASYNC_CAPTURE_SLOT_COUNT=2 ``` @@ -368,12 +373,26 @@ SPARKCACHE_ASYNC_CAPTURE_SLOT_COUNT=2 The `auto` slot policy selects 8 GiB for DCP1, 5 GiB for DCP2, or 3 GiB for DCP4. Two capture slots let the background publisher consume one completed capture while a later capture uses the other. Restore separately overlaps -bounded NVMe reads and CUDA placement through two 256 MiB mapped arenas. More -restore arenas are not part of this profile because measured arena waits did -not justify the additional unified-memory pressure. DCP1 and DCP2 page-tail +bounded NVMe reads and CUDA placement through two 256 MiB mapped arenas per +load lane. Eight lanes reserve 4 GiB per rank; adding the DCP4 capture slots +gives 10 GiB per rank (40 GiB across TP4) of configured payload capacity, +before KV, models, metadata, retained objects and transport. Restore-only +omits the 6 GiB capture ring. DCP1 and DCP2 page-tail capture have no matching live record; use complete snapshots or test those layouts separately. +Print the resolved memory plan without host or GPU access before launching: + +```bash +SPARKRING_PRINT_MEMORY_PLAN=1 bash runtime/glm53-flash-jj-r8-gb10/launch-rank.sh \ + 0 /path/to/rank.env +``` + +Set `SPARKCACHE_BUFFER_BUDGET_BYTES` in that configuration to enforce a +per-rank restore-plus-capture payload ceiling; zero disables it. The report +shows KV separately and names excluded memory. It does not predict total +resident memory or replace a hardware capacity test. + The environment template enables `DFLASH_WARMUP=1`. Rank 0 waits for the API, then exercises every concurrency from C1 through C16 and scheduled prompt spans covering DFlash's Triton block-size specializations. DFlash depth seven diff --git a/docs/GLM53_SPARK_MTP3_MESH_QUICKSTART.md b/docs/GLM53_SPARK_MTP3_MESH_QUICKSTART.md index fa259e9a..9c0deb93 100644 --- a/docs/GLM53_SPARK_MTP3_MESH_QUICKSTART.md +++ b/docs/GLM53_SPARK_MTP3_MESH_QUICKSTART.md @@ -19,6 +19,13 @@ first. It reuses the shared blank-cluster bootstrap and adds the secondary data interfaces, GID/MTU checks, and driver configuration required below. Return here to pull the published image and deploy the model. +**Preparing hosts that have already run GPU workloads?** Consider a reboot +before installation and the first model start, after stopping active workloads. +It can reduce memory fragmentation even when plenty of RAM appears free. +The [startup memory gate](../runtime/glm53-spark-mtp3-mesh/MANAGED_MESH.md#automatic-startup-memory-preparation) +checks available memory and large contiguous free blocks; rebooting does not +replace that check. A reboot interrupts every workload on that host. + The profile uses the `GLM-5.3-Flash-NVFP4-Spark` target's built-in multi-token predictor with three speculative tokens. Graph-native SIRCL handles most captured target verification, fused SIRCL handles large eager prefill, and diff --git a/docs/PREREQUISITES.md b/docs/PREREQUISITES.md index 11e7148a..fefa2cf7 100644 --- a/docs/PREREQUISITES.md +++ b/docs/PREREQUISITES.md @@ -86,6 +86,15 @@ cluster installer. The managed profile adds these requirements: +Before installation and the first model start, consider rebooting hosts that +have run large models or repeated GPU workloads. Stop active workloads first. +A reboot can reduce unified-memory fragmentation and restore large contiguous +free blocks; a high total-free-memory reading alone does not establish loader +readiness. This is a recommendation, not a requirement for every installation. +The [managed startup memory check](../runtime/glm53-spark-mtp3-mesh/MANAGED_MESH.md#automatic-startup-memory-preparation) +still applies after reboot and attempts compaction before requiring further +recovery. Reboots interrupt all workloads on the affected host. + - Four physical cables in the cycle `0-1-2-3-0`, with each rank's port 0/f0 connected to the next rank's port 1/f1; management stays on a separate LAN. - Four configured RDMA functions per host: primary and Socket Direct diff --git a/performance/harnesses/validation/README.md b/performance/harnesses/validation/README.md index 5bef1c6b..d80b2af6 100644 --- a/performance/harnesses/validation/README.md +++ b/performance/harnesses/validation/README.md @@ -81,3 +81,116 @@ unique-prefix samples per context. It calibrates prompts using `/tokenize`, records actual usage and cache evidence, and writes incremental JSONL receipts. It requests one output token; its metric is prompt tokens divided by TTFT. See the shared runbook for commands and the complete measurement protocol. + +## Growing-conversation soak and idle probes + +Status: **implemented**. `conversation_soak.py` reuses the prefill probe's +HTTP and streaming helpers to measure concurrent conversations that grow by a +calibrated number of prompt tokens each turn. It runs sequential, unique-prefix +5,300-token probes before and immediately after the conversation workers finish. +It never changes the serving stack, speculation method, cache access mode, or +namespace. Select those externally and record their identities with `--metadata`. +The `--arm` option labels a receipt; it does not configure the server. + +Print the request and output bounds without any network activity first. This +example targets the GLM-5.3 native-MTP3 profile: + +```bash +python3 performance/harnesses/validation/conversation_soak.py \ + --plan --model glm-5.3-flash-spark --arm baseline-mtp3 --context-limit 1m \ + --concurrency 2 --start-tokens 32768 --max-turns-per-agent 4 \ + --tail-tokens 2048 --max-tokens 64 --probe-output-tokens 64 \ + --duration-seconds 300 --max-soak-prompt-tokens 1000000 \ + --reset-tokens 160000 --seed 2026090601 +``` + +Replace `--plan` with `--endpoint http://192.0.2.10:8015` and +`--output /existing-results-directory/baseline-mtp3.jsonl` to execute. The +short shape admits at most eight conversation requests and six idle probes. +Every request is guarded by successful server tokenization and the declared +context limit. Calibration preserves prior messages and includes the same +chat-template options used for generation. + +For a longer MTP3 run, use `--concurrency 4 --start-tokens 100000 +--max-turns-per-agent 200 --duration-seconds 3600 --max-soak-prompt-tokens +50000000 --reset-tokens 160000 --max-tokens 512 --probe-output-tokens 300`. +Keep the remaining options explicit and run `--plan` first. The time limit +stops admission of new soak requests; calibration, probes, and requests already +in flight can extend wall time. The HTTP timeout is an inactivity timeout. +The global token ceiling counts full prompt tokens admitted across all workers, +including cached tokens, and excludes the separately bounded idle probes. +Each worker stops when its next prompt cannot fit the remaining shared token +budget; other workers may still admit smaller prompts. Token or turn limits +can end the run before the configured duration. Workers rotate to a fresh +conversation before its next prompt and output would exceed `--reset-tokens`. +Request errors stop further soak admissions; other in-flight requests can finish. + +Use identical seeds, shapes, template settings, MTP3 settings and model +identities for paired runs. Provide isolated persistent namespaces externally +when cold independence matters, or explicitly record retained cache state. +The same seed produces the same initial synthetic prompts, while actual +assistant replies grow subsequent histories; receipts preserve those replies +and prompt hashes. Distinct seeds create fresh contexts but do not establish a +byte-identical paired comparison. Probe prefixes differ between the before and +after phases; reported cached-token counts remain available to check whether +either population was already cached. A run name does not isolate GPU prefix +caching or disk state. + +`--image /path/to/fixture.png --image-every 10` adds the same local PNG or JPEG +to every tenth continuation. The receipt records its SHA-256. Keep the fixture +alongside the receipt when reproducing a multimodal run. Image bytes are sent +as a data URL and never fetched from another server. The image must be at most +4 MiB. Tokenization must account for it; an endpoint that cannot tokenize this +shape fails calibration instead of using an estimated count. + +Each turn records the client request ID sent in `X-Request-ID`, response and +server request IDs when returned, start time, actual usage, tokenized prompt +count, prompt hash, assistant text and reasoning, finish reason, elapsed time, +TTFT, and timestamp offsets for every content or reasoning delta. TTFT runs +from client submission to the first nonempty content or reasoning delta. +The decode estimate is `(completion_tokens - 1) / (last_delta - first_delta)`; +streaming chunks may contain several tokens, so this is not a token-level +inter-token latency measurement. The full delta timestamps and usage allow +recalculation. `finish_reason=length` is accepted for the bounded load and is +recorded as output-budget exhaustion, not a correctness pass. + +The summary compares continuation turns with reported cached tokens below or +above half the prompt, using the condition of a positive prompt increase below +10,000 tokens in the same conversation. Missing cached-token usage is excluded +from that classification. These are server-reported numbers: the harness does +not infer local reuse, external restore, or recomputation from latency. Correlate +request IDs with connector and vLLM logs to establish those causes. Tokenization +uses the serving host's CPU and is excluded from each chat latency; its overhead +and in-flight work affect the interval before the after probes. + +Analyze a saved receipt offline with: + +```bash +python3 performance/harnesses/validation/conversation_soak.py \ + --model glm-5.3-flash-spark --arm analysis --context-limit 1m \ + --analyze /existing-results-directory/baseline-mtp3.jsonl +``` + +JSONL records are flushed after each completion and existing output files are +never overwritten. API keys come from the environment and are omitted from +receipts; error bodies are omitted as well. Preserve model/image/source IDs, +topology, actual KV allocation, access mode, namespace, concurrent traffic and +initial store occupancy in the supplied metadata or an accompanying evidence +record. One short successful run does not qualify a long soak or a performance +improvement. Offline tests use in-memory HTTP fixtures and contact no hosts. + +`analyze_conversation_reuse.py` joins a saved receipt to the compact +`sparkcache-reuse-trace/v1` records in saved rank logs. Supply the expected +physical ranks explicitly. A verified-restore classification requires a +successful completion for the same engine request, digest and token span on +every expected rank; an offer alone is insufficient. Scheduler lease-attachment +events are reported separately. API-only evidence is labeled `reported_cached`: +positive counts do not identify local versus external reuse, and zero counts +do not exclude a GPU lease. + +The join accepts recorded request/response IDs and the runtime's exact +eight-hex-character engine suffix. Ambiguous joins cannot establish a restore +quorum. The output preserves per-rank queue, service and phase timings without +summing token spans across ranks. Log captures can contain requests outside the +receipt window; inspect unmatched IDs and timestamps before assigning them to a +missing-rank or request failure. diff --git a/performance/harnesses/validation/analyze_conversation_reuse.py b/performance/harnesses/validation/analyze_conversation_reuse.py new file mode 100644 index 00000000..ff4dd5de --- /dev/null +++ b/performance/harnesses/validation/analyze_conversation_reuse.py @@ -0,0 +1,233 @@ +"""Join saved conversation receipts to rank-local SparkCache reuse traces; no network access.""" +import argparse +from collections import Counter, defaultdict +import hashlib +import json +import math +from pathlib import Path +import re +import statistics + +TRACE_SCHEMA = "sparkcache-reuse-trace/v1" +MARKER = "spark-context-cache-reuse:" + + +def parse_trace(line): + """Accept Docker log prefixes and JSON log envelopes without interpreting prose.""" + raw = line.split(MARKER, 1)[1].lstrip() if MARKER in line and not line.lstrip().startswith("{") else line.lstrip() + try: + value, _ = json.JSONDecoder().raw_decode(raw) + except (ValueError, TypeError): + return None + if not isinstance(value, dict): + return None + if value.get("schema") == TRACE_SCHEMA: + return value + for field in ("log", "message"): + if isinstance(value.get(field), str) and MARKER in value[field]: + return parse_trace(value[field]) + return None + + +def aliases(record): + values = set() + for name in ("request_id", "response_id", "server_request_id"): + value = record.get(name) + if isinstance(value, str) and value: + values.add(value) + values.add(value.removeprefix("chatcmpl-") if value.startswith("chatcmpl-") else "chatcmpl-" + value) + return values + + +def finite_number(value): + return type(value) in (int, float) and math.isfinite(value) and value >= 0 + + +def median(values): + valid = [value for value in values if finite_number(value)] + return statistics.median(valid) if valid else None + + +def latest_completions(traces, expected): + candidates = defaultdict(list) + for trace in traces: + rank = trace.get("rank") + if (trace.get("event") == "worker_restore_completed" and trace.get("role") == "worker" + and type(rank) is int and rank in expected and type(trace.get("time_ns")) is int): + candidates[rank].append(trace) + latest, conflicts = {}, [] + for rank, rows in candidates.items(): + newest = max(row["time_ns"] for row in rows) + current = [row for row in rows if row["time_ns"] == newest] + identities = {json.dumps(row, sort_keys=True) for row in current} + if len(identities) > 1: + conflicts.append(rank) + else: + latest[rank] = current[0] + return latest, conflicts + + +def classify(turn, traces, expected, *, ambiguous=False): + latest, conflicts = latest_completions(traces, expected) + verified = len(latest) == len(expected) and not conflicts + identities = set() + for row in latest.values(): + span = row.get("requested_span_tokens") + if type(span) is not int or not isinstance(row.get("digest"), str): + verified = False + continue + verified &= (row.get("outcome") == "verified" and span > 0 + and type(row.get("verified_span_tokens")) is int + and row.get("verified_span_tokens") == span + and bool(row["digest"])) + identities.add((row.get("digest"), span, row.get("request_id"))) + verified = bool(verified and len(identities) == 1 and turn.get("valid") and not ambiguous) + attached = [row for row in traces if row.get("event") == "gpu_lease_attached" + and row.get("role") == "scheduler" and type(row.get("lease_span_tokens")) is int + and row["lease_span_tokens"] > 0] + cached = turn.get("cached_tokens_reported") + usage = turn.get("usage") or {} + prompt = usage.get("prompt_tokens") + positive_report = type(cached) is int and type(prompt) is int and 0 < cached <= prompt + if ambiguous or not turn.get("valid"): + source = "unknown" + elif verified: + source = "verified_all_rank_external_restore" + elif attached: + source = "gpu_lease_attached" + elif positive_report: + source = "reported_cached" + else: + source = "unknown" + return {"type": "turn_analysis", "request_id": turn.get("request_id"), + "response_id": turn.get("response_id"), "identity": turn.get("identity"), + "phase": turn.get("phase"), "continuation": turn.get("continuation", False), + "elapsed_seconds": turn.get("elapsed_seconds"), "ttft_seconds": turn.get("ttft_seconds"), + "source": source, "trace_join_ambiguous": ambiguous, + "cached_tokens_reported": cached, "prompt_tokens_reported": prompt, + "all_rank_worker_verification": verified, + "verified_span_tokens": next(iter(identities))[1] if verified else None, + "gpu_lease_attachment_observed": bool(attached), + "lease_span_tokens_observed": max((row["lease_span_tokens"] for row in attached), default=None), + "offer_count": sum(row.get("event") == "external_restore_offer" for row in traces), + "expected_ranks": sorted(expected), "completed_ranks": sorted(latest), + "missing_ranks": sorted(expected - latest.keys()), "conflicting_ranks": sorted(conflicts), + "worker_completions_by_rank": {str(rank): row for rank, row in sorted(latest.items())}, + "trace_events": traces, + "reported_cache_interpretation": "API cached tokens only; positive counts do not distinguish local reuse from external restore, and zero counts do not exclude GPU lease reuse"} + + +def analyze(receipts, traces, expected_ranks): + expected = set(expected_ranks) + if not expected or any(type(rank) is not int or rank < 0 for rank in expected): + raise ValueError("Expected physical ranks must be explicit nonnegative integers") + turns = [row for row in receipts if row.get("type") == "turn"] + owners, response_owners, joined, ambiguous = defaultdict(set), defaultdict(set), defaultdict(list), set() + for index, turn in enumerate(turns): + for alias in aliases(turn): + owners[alias].add(index) + response_id = turn.get("response_id") + if isinstance(response_id, str) and response_id: + response_owners[response_id].add(index) + seen, unmatched, duplicated = set(), 0, 0 + for trace in traces: + encoded = json.dumps(trace, sort_keys=True) + if encoded in seen: + duplicated += 1 + continue + seen.add(encoded) + request_id = trace.get("request_id") + matches = set(owners.get(request_id, set())) if isinstance(request_id, str) else set() + if isinstance(request_id, str): + # vLLM appends an eight-hex engine nonce to the observed API response + # ID. Only that exact suffix grammar is accepted; never prefix-match. + match = re.fullmatch(r"(.+)-[0-9a-fA-F]{8}", request_id) + if match: + matches.update(response_owners.get(match.group(1), set())) + if len(matches) == 1: + joined[next(iter(matches))].append(trace) + elif matches: + ambiguous.update(matches) + else: + unmatched += 1 + rows = [classify(turn, joined[index], expected, ambiguous=index in ambiguous) + for index, turn in enumerate(turns)] + by_source = {} + by_phase_and_source = {} + for source in sorted({row["source"] for row in rows}): + population = [row for row in rows if row["source"] == source] + by_source[source] = {"turns": len(population), + "median_latency_seconds": median(row["elapsed_seconds"] for row in population), + "median_ttft_seconds": median(row["ttft_seconds"] for row in population)} + for phase in sorted({row.get("phase") or "unknown" for row in rows}): + by_phase_and_source[phase] = {} + for source in by_source: + population = [row for row in rows if (row.get("phase") or "unknown") == phase and row["source"] == source] + if population: + by_phase_and_source[phase][source] = { + "turns": len(population), "continuations": sum(row["continuation"] for row in population), + "median_latency_seconds": median(row["elapsed_seconds"] for row in population), + "median_ttft_seconds": median(row["ttft_seconds"] for row in population)} + timings = {} + for rank in sorted(expected): + population = [trace for row in rows for trace in row["trace_events"] + if trace.get("role") == "worker" and trace.get("event") == "worker_restore_completed" + and type(trace.get("rank")) is int and trace["rank"] == rank] + phases = sorted({phase for row in population for phase in (row.get("phase_ms") or {})}) + timings[str(rank)] = {"completions": len(population), + "outcomes": dict(Counter(row.get("outcome", "unknown") for row in population)), + **{f"median_{field}": median(row.get(field) for row in population) + for field in ("queue_wait_ms", "service_ms", "end_to_end_ms")}, + "median_phase_ms": {phase: median((row.get("phase_ms") or {}).get(phase) for row in population) + for phase in phases}} + return {"schema": "sparkring-conversation-reuse-analysis/v1", "trace_schema": TRACE_SCHEMA, + "expected_ranks": sorted(expected), "turns": rows, + "summary": {"turns": len(rows), "by_source": by_source, "by_phase_and_source": by_phase_and_source, + "per_rank_restore_timings": timings, + "unmatched_trace_events": unmatched, "duplicate_trace_events": duplicated, + "ambiguous_turns": len(ambiguous)}, + "limits": ["No recomputation or local-miss inference from zero or missing API cache counts", + "Offers do not prove an executed restore; every expected physical rank must verify the same span and digest", + "Worker verification plus successful client completion does not expose vLLM's internal receive-aggregation decision", + "An observed lease attachment proves attachment, not its full subsequent lifetime", + "Rank token counts are never summed; source classes may coexist and the strongest observed evidence is selected"]} + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--receipt", type=Path, required=True) + parser.add_argument("--log", type=Path, action="append", required=True, + help="Saved rank log; repeat for every physical rank") + parser.add_argument("--expected-ranks", required=True, help="Physical ranks, for example 0,1,2,3") + parser.add_argument("--output", type=Path, required=True) + args = parser.parse_args() + try: + ranks = [int(value) for value in args.expected_ranks.split(",")] + if len(set(ranks)) != len(ranks): + raise ValueError("Expected ranks must be unique") + receipts = [json.loads(line) for line in args.receipt.read_text(encoding="utf-8").splitlines() if line.strip()] + traces, malformed, sources = [], 0, [] + for path in args.log: + sources.append({"path": str(path), "sha256": hashlib.sha256(path.read_bytes()).hexdigest()}) + with path.open(encoding="utf-8", errors="replace") as stream: + for line in stream: + record = parse_trace(line) + if record: + traces.append(record) + elif MARKER in line: + malformed += 1 + result = analyze(receipts, traces, ranks) + result["inputs"] = {"receipt": str(args.receipt), + "receipt_sha256": hashlib.sha256(args.receipt.read_bytes()).hexdigest(), "logs": sources} + result["summary"]["malformed_trace_lines"] = malformed + with args.output.open("x", encoding="utf-8") as stream: + json.dump(result, stream, indent=2) + stream.write("\n") + except (ValueError, OSError) as error: + parser.error(str(error)) + print(json.dumps(result["summary"], sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/performance/harnesses/validation/conversation_soak.py b/performance/harnesses/validation/conversation_soak.py new file mode 100644 index 00000000..266cadc5 --- /dev/null +++ b/performance/harnesses/validation/conversation_soak.py @@ -0,0 +1,323 @@ +"""Measure bounded growing conversations and idle probes; never change server cache state.""" +import argparse +import base64 +from concurrent.futures import ThreadPoolExecutor +import hashlib +import json +import math +import os +from pathlib import Path +import statistics +import sys +import threading +import time +import uuid + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +from needle_hunt import base_url # noqa: E402 +from prefill_probe import build_text, events, has_token, request, tokens # noqa: E402 + +SCHEMA = "sparkring-conversation-soak/v1" + + +def digest(value): + return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(",", ":")).encode()).hexdigest() + + +def fixture(identity, chars): + text = build_text(identity, chars) + return text.replace("Read the notes and reply OK.", + "Analyze these maintenance notes in detail and recommend next actions.", 1) + + +def count_tokens(config, messages): + payload = {"model": config["model"], "messages": messages, "add_generation_prompt": True, + "chat_template_kwargs": config["chat_template_kwargs"]} + with request(config["endpoint"] + "/tokenize", payload, config["api_key"], config["timeout"]) as response: + count = json.load(response).get("count") + if type(count) is not int or count <= 0: + raise ValueError("Tokenizer must return a positive integer count") + return count + + +def calibrated_user(config, history, identity, target_total, image=None): + """Calibrate the added user message while preserving every preceding message byte.""" + previous = count_tokens(config, history) if history else 0 + chars = max(256, (target_total - previous) * 5) + for _ in range(10): + text = fixture(identity, chars) + content = text if image is None else [ + {"type": "text", "text": text}, {"type": "image_url", "image_url": {"url": image}}] + messages = history + [{"role": "user", "content": content}] + count = count_tokens(config, messages) + if abs(count - target_total) <= max(8, (target_total - previous) * 0.01): + return messages, count + chars = max(128, min(64 * 1024 * 1024, round(chars * max(1, target_total - previous) + / max(1, count - previous)))) + raise ValueError("Token calibration did not converge") + + +def stream_turn(config, messages, count, max_tokens, identity, phase, *, request_id=None, clock=time.perf_counter): + if count + max_tokens > config["context_limit"]: + raise ValueError("Tokenized prompt plus output exceeds the context limit") + request_id = request_id or "soak-" + uuid.uuid4().hex + payload = {"model": config["model"], "messages": messages, "max_tokens": max_tokens, + "stream": True, "stream_options": {"include_usage": True}, + "temperature": config["temperature"], "seed": config["seed"], "top_p": 1, + "chat_template_kwargs": config["chat_template_kwargs"]} + effort = config["probe_reasoning_effort"] if phase != "soak" else config["reasoning_effort"] + if effort: + payload["reasoning_effort"] = effort + chunks, content, reasoning, usage, response_id, finish = [], [], [], None, None, None + started_unix, started = time.time(), clock() + with request(config["endpoint"] + "/v1/chat/completions", payload, config["api_key"], + config["timeout"], request_id=request_id) as response: + header_id = response.headers.get("X-Request-ID") if hasattr(response, "headers") else None + for event in events(response): + now = clock() + if event.get("id"): + response_id = event["id"] + if has_token(event): + chunks.append(now - started) + if event.get("usage"): + usage = event["usage"] + for choice in event.get("choices", []): + delta = choice.get("delta", {}) + if delta.get("content"): + content.append(delta["content"]) + if delta.get("reasoning_content") or delta.get("reasoning"): + reasoning.append(delta.get("reasoning_content") or delta["reasoning"]) + finish = choice.get("finish_reason") or finish + elapsed = clock() - started + if not chunks or not isinstance(usage, dict) or type(usage.get("prompt_tokens")) is not int or usage["prompt_tokens"] <= 0: + raise ValueError("Missing first delta or authoritative prompt usage") + completion = usage.get("completion_tokens") + if type(completion) is not int or completion < 1 or finish not in ("stop", "length"): + raise ValueError("Missing output usage or normal finish reason") + cached = (usage.get("prompt_tokens_details") or {}).get("cached_tokens") + if cached is not None and (type(cached) is not int or not 0 <= cached <= usage["prompt_tokens"]): + raise ValueError("Invalid reported cached-token count") + assistant = {"role": "assistant", "content": "".join(content)} + if reasoning: + assistant["reasoning_content"] = "".join(reasoning) + span = chunks[-1] - chunks[0] + record = {"type": "turn", "phase": phase, "identity": identity, "valid": True, + "request_id": request_id, "server_request_id": header_id, "response_id": response_id, + "started_unix": started_unix, "prompt_sha256": digest(messages), + "assistant": assistant, "tokenized_prompt_tokens": count, "usage": usage, + "cached_tokens_reported": cached, "cached_fraction_reported": cached / usage["prompt_tokens"] + if cached is not None and usage["prompt_tokens"] > 0 else None, + "ttft_seconds": chunks[0], "elapsed_seconds": elapsed, + "content_delta_offsets_seconds": chunks, "decode_span_seconds": span, + "decode_tokens_per_second_estimate": (completion - 1) / span if span > 0 else None, + "finish_reason": finish, "output_budget_exhausted": finish == "length"} + return record, assistant + + +def median(values): + available = [value for value in values if value is not None] + return statistics.median(available) if available else None + + +def summarize(records): + good = [r for r in records if r.get("type") == "turn" and r.get("valid")] + continuation = [r for r in good if r.get("continuation")] + known = [r for r in continuation if r["cached_fraction_reported"] is not None] + low = [r for r in known if r["cached_fraction_reported"] < 0.5] + high = [r for r in known if r["cached_fraction_reported"] >= 0.5] + probes = {phase: {"samples": len(rows), + "cached_token_counts_reported": [r["cached_tokens_reported"] for r in rows], + "median_ttft_seconds": median(r["ttft_seconds"] for r in rows), + "median_decode_tokens_per_second_estimate": median(r["decode_tokens_per_second_estimate"] for r in rows)} + for phase in ("before", "after") + for rows in [[r for r in good if r["phase"] == phase]]} + return {"type": "summary", "valid_turns": len(good), + "soak_turns": sum(r["phase"] == "soak" for r in good), + "errors": sum(r.get("type") == "error" for r in records), + "continuations": len(continuation), "continuations_with_cache_usage": len(known), + "continuations_below_half_cached": len(low), + "fraction_below_half_cached": len(low) / len(known) if known else None, + "low_cached_median_latency_seconds": median(r["elapsed_seconds"] for r in low), + "high_cached_median_latency_seconds": median(r["elapsed_seconds"] for r in high), + "probes": probes, + "cache_interpretation": "server-reported cached tokens; no inference of local hit, external restore, or recompute"} + + +def execute(config, output): + lock, stop = threading.Lock(), threading.Event() + records, consumed = [], 0 + image = config.get("image_data") + safe_config = {key: value for key, value in config.items() if key not in ("api_key", "image_data")} + with Path(output).open("x", encoding="utf-8") as stream: + def emit(record): + with lock: + records.append(record) + line = json.dumps(record) + stream.write(line + "\n") + stream.flush() + print(line, flush=True) + + emit({"type": "start", "schema": SCHEMA, "config": safe_config, + "harness_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), + "http_helper_sha256": hashlib.sha256(Path(__file__).with_name("prefill_probe.py").read_bytes()).hexdigest(), + "started_unix": time.time()}) + + def probe(phase, index): + identity = f"{config['seed']}-probe-{phase}-{index}" + request_id = "soak-" + uuid.uuid4().hex + try: + messages, count = calibrated_user(config, [], identity, config["probe_tokens"]) + record, _ = stream_turn(config, messages, count, config["probe_output_tokens"], identity, + phase, request_id=request_id) + emit(record) + except Exception as error: + emit({"type": "error", "phase": phase, "identity": identity, + "request_id": request_id, "error": type(error).__name__}) + stop.set() + + def worker(agent, deadline): + nonlocal consumed + history, prior_count, conversation, turn = [], None, 0, 0 + for _ in range(config["max_turns_per_agent"]): + if stop.is_set() or time.monotonic() >= deadline: + break + if prior_count is not None and prior_count + config["tail_tokens"] + config["max_tokens"] > config["reset_tokens"]: + history, prior_count, turn = [], None, 0 + conversation += 1 + identity = f"{config['seed']}-agent-{agent}-conversation-{conversation}-turn-{turn}" + request_id = "soak-" + uuid.uuid4().hex + try: + initial = config["start_tokens"][agent % len(config["start_tokens"])] + target = initial if prior_count is None else prior_count + config["tail_tokens"] + picture = image if image and turn > 0 and turn % config["image_every"] == 0 else None + messages, count = calibrated_user(config, history, identity, target, picture) + if count + config["max_tokens"] > config["context_limit"]: + raise ValueError("Context limit exceeded") + with lock: + if stop.is_set() or time.monotonic() >= deadline or consumed + count > config["max_soak_prompt_tokens"]: + break + consumed += count + record, assistant = stream_turn(config, messages, count, config["max_tokens"], identity, + "soak", request_id=request_id) + record.update(agent=agent, conversation=conversation, turn=turn, + prompt_token_growth=count - prior_count if prior_count is not None else None, + continuation=prior_count is not None and 0 < count - prior_count < 10000, + image_added=picture is not None) + emit(record) + history, prior_count, turn = messages + [assistant], count, turn + 1 + except Exception as error: + emit({"type": "error", "phase": "soak", "identity": identity, + "request_id": request_id, "error": type(error).__name__}) + stop.set() + break + + try: + for index in range(config["probe_repeats"]): + probe("before", index) + if stop.is_set(): + break + deadline = time.monotonic() + config["duration_seconds"] + with ThreadPoolExecutor(max_workers=config["concurrency"]) as pool: + list(pool.map(lambda agent: worker(agent, deadline), range(config["concurrency"]))) + if not stop.is_set(): + for index in range(config["probe_repeats"]): + probe("after", index) + if stop.is_set(): + break + except Exception as error: + emit({"type": "error", "phase": "probe", "error": type(error).__name__}) + summary = summarize(records) + summary["soak_prompt_tokens_admitted"] = consumed + summary["success"] = not summary["errors"] and summary["soak_turns"] > 0 + summary["completed_unix"] = time.time() + emit(summary) + return 0 if summary["success"] else 2 + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--endpoint") + parser.add_argument("--model", required=True) + parser.add_argument("--arm", required=True, help="Evidence label only; never changes server configuration") + parser.add_argument("--seed", type=int, default=20260906) + parser.add_argument("--concurrency", type=int, default=4) + parser.add_argument("--start-tokens", default="75000,106000,128000,150000") + parser.add_argument("--tail-tokens", type=tokens, default=2048) + parser.add_argument("--reset-tokens", type=tokens, default=300000) + parser.add_argument("--context-limit", type=tokens, required=True) + parser.add_argument("--max-tokens", type=tokens, default=512) + parser.add_argument("--duration-seconds", type=float, default=300) + parser.add_argument("--max-turns-per-agent", type=int, default=20) + parser.add_argument("--max-soak-prompt-tokens", type=int, default=10000000) + parser.add_argument("--temperature", type=float, default=1) + parser.add_argument("--reasoning-effort", default="low") + parser.add_argument("--probe-reasoning-effort", default="low") + parser.add_argument("--probe-tokens", type=tokens, default=5300) + parser.add_argument("--probe-output-tokens", type=tokens, default=300) + parser.add_argument("--probe-repeats", type=int, default=3) + parser.add_argument("--timeout", type=float, default=900, help="HTTP inactivity timeout, not a wall-clock deadline") + parser.add_argument("--api-key-env", default="OPENAI_API_KEY") + parser.add_argument("--chat-template-kwargs", default="{}") + parser.add_argument("--metadata", default="{}", help="JSON evidence identities; do not include credentials") + parser.add_argument("--image", type=Path, help="Optional local PNG or JPEG, at most 4 MiB") + parser.add_argument("--image-every", type=int, default=10) + parser.add_argument("--output", type=Path) + parser.add_argument("--plan", action="store_true", help="Print bounds without tokenization or HTTP requests") + parser.add_argument("--analyze", type=Path, help="Summarize an existing JSONL receipt without HTTP") + args = parser.parse_args() + if args.analyze: + print(json.dumps(summarize([json.loads(line) for line in args.analyze.read_text(encoding="utf-8").splitlines()]))) + return 0 + config = vars(args).copy() + for name in ("output", "plan", "analyze", "image"): + config.pop(name) + try: + config["start_tokens"] = [tokens(value) for value in args.start_tokens.split(",")] + config["chat_template_kwargs"] = json.loads(args.chat_template_kwargs) + config["metadata"] = json.loads(args.metadata) + if not isinstance(config["chat_template_kwargs"], dict) or not isinstance(config["metadata"], dict): + raise ValueError("Template options and metadata must be JSON objects") + if not 1 <= args.concurrency <= 32 or not 1 <= args.max_turns_per_agent <= 10000 or not 0 <= args.probe_repeats <= 20 or args.image_every < 1: + raise ValueError("Invalid concurrency, repetition, or image frequency") + if not math.isfinite(args.duration_seconds) or not 0 < args.duration_seconds <= 14400: + raise ValueError("Duration must be positive and at most 4 hours") + if not 1 <= args.max_soak_prompt_tokens <= 1000000000: + raise ValueError("Soak token budget must be between 1 and 1000000000") + if not math.isfinite(args.timeout) or not 0 < args.timeout <= 3600 or not 0 <= args.temperature <= 2: + raise ValueError("Invalid timeout or temperature") + if max(config["start_tokens"]) + args.max_tokens > args.reset_tokens or args.reset_tokens > args.context_limit: + raise ValueError("Starting prompts plus output must fit reset threshold and context limit") + if args.probe_tokens + args.probe_output_tokens > args.context_limit: + raise ValueError("Probe prompt plus output must fit context limit") + if args.endpoint: + config["endpoint"] = base_url(args.endpoint) + if args.image: + if args.image.stat().st_size > 4 * 1024 * 1024: + raise ValueError("Image exceeds 4 MiB") + raw = args.image.read_bytes() + mime = "image/png" if raw.startswith(b"\x89PNG\r\n\x1a\n") else "image/jpeg" if raw.startswith(b"\xff\xd8\xff") else None + if mime is None: + raise ValueError("Image must be PNG or JPEG") + config["image_sha256"] = hashlib.sha256(raw).hexdigest() + image_data = "data:" + mime + ";base64," + base64.b64encode(raw).decode() + else: + image_data = None + except (ValueError, OSError, argparse.ArgumentTypeError) as error: + parser.error(str(error)) + if args.plan: + print(json.dumps({"schema": SCHEMA, "config": config, + "max_chat_requests": args.concurrency * args.max_turns_per_agent + 2 * args.probe_repeats, + "max_output_tokens": args.concurrency * args.max_turns_per_agent * args.max_tokens + + 2 * args.probe_repeats * args.probe_output_tokens, + "duration_scope": "soak admission window; probes, calibration and in-flight requests may extend elapsed time", + "status": "implemented; no hardware qualification"})) + return 0 + if not args.endpoint or not args.output: + parser.error("Execution requires --endpoint and --output") + config["api_key"] = os.environ.get(args.api_key_env, "") + config["image_data"] = image_data + return execute(config, args.output) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/performance/harnesses/validation/prefill_probe.py b/performance/harnesses/validation/prefill_probe.py index d89134f3..a47b9c62 100644 --- a/performance/harnesses/validation/prefill_probe.py +++ b/performance/harnesses/validation/prefill_probe.py @@ -20,8 +20,10 @@ def tokens(value): return result -def request(url, payload, key, timeout): +def request(url, payload, key, timeout, request_id=None): headers = {'Content-Type': 'application/json'} + if request_id: + headers['X-Request-ID'] = request_id if key: headers['Authorization'] = 'Bearer ' + key class NoRedirect(urllib.request.HTTPRedirectHandler): diff --git a/performance/harnesses/validation/test_analyze_conversation_reuse.py b/performance/harnesses/validation/test_analyze_conversation_reuse.py new file mode 100644 index 00000000..2f688aa5 --- /dev/null +++ b/performance/harnesses/validation/test_analyze_conversation_reuse.py @@ -0,0 +1,149 @@ +"""Reuse evidence tests are entirely offline; incomplete quorum never verifies.""" +import importlib.util +import json +from pathlib import Path + +import pytest + +spec = importlib.util.spec_from_file_location("reuse_analysis_test", Path(__file__).with_name("analyze_conversation_reuse.py")) +analysis = importlib.util.module_from_spec(spec) +spec.loader.exec_module(analysis) + + +def turn(**overrides): + return {"type": "turn", "request_id": "soak-one", "response_id": "chatcmpl-soak-one", "valid": True, + "phase": "soak", "cached_tokens_reported": 0, "usage": {"prompt_tokens": 12000}, + "elapsed_seconds": 5, "ttft_seconds": 1, **overrides} + + +def completion(rank, **overrides): + return {"schema": analysis.TRACE_SCHEMA, "event": "worker_restore_completed", + "request_id": "chatcmpl-soak-one", "role": "worker", "rank": rank, + "dcp_rank": rank % 2, "time_ns": 100, "digest": "0123456789ab", + "requested_span_tokens": 10000, "verified_span_tokens": 10000, "outcome": "verified", + "queue_wait_ms": 1 + rank, "service_ms": 20, "end_to_end_ms": 21 + rank, + "phase_ms": {"read": 12, "place": 8}, **overrides} + + +def test_log_prefix_and_docker_envelope(): + record = completion(0) + line = "2026-09-06 (Worker_TP0) INFO " + analysis.MARKER + json.dumps(record) + assert analysis.parse_trace(line) == record + assert analysis.parse_trace(json.dumps({"log": line, "stream": "stderr"})) == record + assert analysis.parse_trace("not a trace") is None + assert analysis.parse_trace(analysis.MARKER + "{partial") is None + + +def test_every_physical_rank_required_and_tokens_not_summed(): + report = analysis.analyze([turn()], [completion(rank) for rank in range(4)], [0, 1, 2, 3]) + row = report["turns"][0] + assert row["source"] == "verified_all_rank_external_restore" + assert row["verified_span_tokens"] == 10000 + assert row["missing_ranks"] == [] + assert report["summary"]["per_rank_restore_timings"]["3"]["median_queue_wait_ms"] == 4 + assert report["summary"]["per_rank_restore_timings"]["2"]["median_phase_ms"] == {"read": 12, "place": 8} + + +@pytest.mark.parametrize("traces", [ + [completion(0), completion(1)], + [completion(rank, outcome="recompute", verified_span_tokens=0) for rank in range(4)], + [completion(rank, digest="different" if rank == 3 else "0123456789ab") for rank in range(4)], + [completion(rank, requested_span_tokens=11000 if rank == 3 else 10000) for rank in range(4)], + [completion(rank, role="scheduler") for rank in range(4)], +]) +def test_partial_failed_or_mismatched_completions_do_not_verify(traces): + row = analysis.analyze([turn()], traces, [0, 1, 2, 3])["turns"][0] + assert row["source"] == "unknown" + assert row["verified_span_tokens"] is None + + +def test_offer_is_not_restore_and_zero_api_not_local_miss(): + offer = {"schema": analysis.TRACE_SCHEMA, "request_id": "chatcmpl-soak-one", "role": "scheduler", + "event": "external_restore_offer", "selected_span_tokens": 10000} + row = analysis.analyze([turn()], [offer], [0, 1, 2, 3])["turns"][0] + assert row["source"] == "unknown" + assert row["offer_count"] == 1 + assert not row["all_rank_worker_verification"] + missing = analysis.analyze([turn(cached_tokens_reported=None)], [], [0])["turns"][0] + assert missing["source"] == "unknown" + + +def test_gpu_attachment_and_positive_report_are_separate_evidence(): + attached = {"schema": analysis.TRACE_SCHEMA, "request_id": "soak-one", "event": "gpu_lease_attached", + "role": "scheduler", "lease_span_tokens": 9000} + row = analysis.analyze([turn(cached_tokens_reported=8000)], [attached], [0, 1, 2, 3])["turns"][0] + assert row["source"] == "gpu_lease_attached" + assert not row["all_rank_worker_verification"] + assert row["lease_span_tokens_observed"] == 9000 + assert analysis.analyze([turn(cached_tokens_reported=8000)], [], [0])["turns"][0]["source"] == "reported_cached" + + +def test_latest_failure_overrides_prior_success_and_duplicates_do_not_make_quorum(): + records = [completion(rank) for rank in range(4)] + records += [completion(3, time_ns=200, outcome="recompute", verified_span_tokens=0)] + report = analysis.analyze([turn()], records + records, [0, 1, 2, 3]) + assert report["turns"][0]["source"] == "unknown" + assert report["summary"]["duplicate_trace_events"] == 5 + assert analysis.analyze([turn()], [completion(0)] * 4, [0, 1, 2, 3])["turns"][0]["source"] == "unknown" + + +def test_conflicting_same_time_and_ambiguous_ids_do_not_verify(): + records = [completion(rank) for rank in range(4)] + [completion(0, outcome="recompute", verified_span_tokens=0)] + row = analysis.analyze([turn()], records, [0, 1, 2, 3])["turns"][0] + assert row["source"] == "unknown" and row["conflicting_ranks"] == [0] + report = analysis.analyze([turn(), turn()], [completion(rank) for rank in range(4)], [0, 1, 2, 3]) + assert report["summary"]["ambiguous_turns"] == 2 + assert all(row["source"] == "unknown" for row in report["turns"]) + + +def test_unmatched_id_not_fuzzily_associated_and_rank_set_explicit(): + report = analysis.analyze([turn()], [completion(0, request_id="chatcmpl-soak-one-other")], [0]) + assert report["summary"]["unmatched_trace_events"] == 1 + with pytest.raises(ValueError, match="explicit"): + analysis.analyze([turn()], [], []) + + +def test_malformed_completion_never_verifies(): + row = analysis.analyze([turn()], [completion(0, requested_span_tokens=[10000])], [0])["turns"][0] + assert row["source"] == "unknown" + row = analysis.analyze([turn()], [completion(0, digest=[])], [0])["turns"][0] + assert row["source"] == "unknown" + + +def test_phase_populations_remain_separate_and_retry_timings_retained(): + turns = [turn(), turn(phase="before", request_id="probe-id", response_id="chatcmpl-probe-id", elapsed_seconds=1)] + report = analysis.analyze(turns, [completion(0, time_ns=50, outcome="recompute", verified_span_tokens=0), completion(0)], [0]) + phases = report["summary"]["by_phase_and_source"] + assert phases["soak"]["verified_all_rank_external_restore"]["median_latency_seconds"] == 5 + assert phases["before"]["unknown"]["median_latency_seconds"] == 1 + assert report["summary"]["per_rank_restore_timings"]["0"]["outcomes"] == {"recompute": 1, "verified": 1} + + +def test_observed_response_id_engine_nonce_joins_strictly(): + response_id = "chatcmpl-soak-cc0bdb28896248beab09f145c9418947" + rows = [completion(rank, request_id=response_id + "-b4884e31") for rank in range(4)] + report = analysis.analyze([turn(response_id=response_id)], rows, [0, 1, 2, 3]) + assert report["turns"][0]["source"] == "verified_all_rank_external_restore" + assert report["summary"]["unmatched_trace_events"] == 0 + for suffix in ("-b4884e3", "-b4884e311", "-b4884e3z", "-b4884e31-more"): + report = analysis.analyze([turn(response_id=response_id)], [completion(0, request_id=response_id + suffix)], [0]) + assert report["turns"][0]["source"] == "unknown" + assert report["summary"]["unmatched_trace_events"] == 1 + + +def test_engine_nonce_requires_observed_response_id_and_refuses_collisions(): + engine_id = "chatcmpl-soak-one-1234abcd" + row = analysis.analyze([turn(response_id=None)], [completion(0, request_id=engine_id)], [0])["turns"][0] + assert row["source"] == "unknown" + turns = [turn(), turn(request_id=engine_id, response_id="other-api-id")] + report = analysis.analyze(turns, [completion(0, request_id=engine_id)], [0]) + assert report["summary"]["ambiguous_turns"] == 2 + assert all(row["source"] == "unknown" for row in report["turns"]) + + +def test_different_engine_requests_cannot_be_combined_into_quorum(): + rows = [completion(0, request_id="chatcmpl-soak-one-1234abcd"), + completion(1, request_id="chatcmpl-soak-one-5678abcd")] + row = analysis.analyze([turn()], rows, [0, 1])["turns"][0] + assert row["source"] == "unknown" + assert not row["all_rank_worker_verification"] diff --git a/performance/harnesses/validation/test_conversation_soak.py b/performance/harnesses/validation/test_conversation_soak.py new file mode 100644 index 00000000..0bb8bb6d --- /dev/null +++ b/performance/harnesses/validation/test_conversation_soak.py @@ -0,0 +1,168 @@ +"""Bounded soak tests use in-memory HTTP fixtures and never contact a host.""" +import importlib.util +import io +import json +from pathlib import Path +import sys + +import pytest + +spec = importlib.util.spec_from_file_location("soak_under_test", Path(__file__).with_name("conversation_soak.py")) +soak = importlib.util.module_from_spec(spec) +spec.loader.exec_module(soak) + + +def config(**overrides): + return {"endpoint": "http://192.0.2.1:8015", "model": "glm-5.3-flash-spark", + "api_key": "test-key", "timeout": 1, "context_limit": 8192, + "temperature": 1, "seed": 20260906, "arm": "mtp3-test", "reasoning_effort": "low", + "probe_reasoning_effort": "low", "chat_template_kwargs": {"enable_thinking": True}, + "probe_tokens": 300, "probe_output_tokens": 64, "probe_repeats": 1, + "start_tokens": [1000], "max_tokens": 64, "tail_tokens": 200, + "reset_tokens": 3000, "concurrency": 2, "duration_seconds": 30, + "max_turns_per_agent": 3, "max_soak_prompt_tokens": 20000, + "image_every": 10, **overrides} + + +def fake_count(messages): + length = 0 + for message in messages: + content = message["content"] + if isinstance(content, list): + length += sum(len(part.get("text", "")) + (500 if part["type"] == "image_url" else 0) + for part in content) + else: + length += len(content) + length += len(message.get("reasoning_content", "")) + 20 + return max(1, length // 5) + + +def fake_http(calls, *, cached=0, missing_usage=False, error=False): + def send(url, payload, key, timeout, request_id=None): + calls.append((url, payload, request_id)) + count = fake_count(payload["messages"]) + if url.endswith("/tokenize"): + return io.BytesIO(json.dumps({"count": count}).encode()) + if error: + raise ValueError("credential echoed by server: test-key") + records = [ + {"id": "chatcmpl-fixture", "choices": [{"delta": {"reasoning_content": "think"}}]}, + {"choices": [{"delta": {"content": "answer"}, "finish_reason": "length"}]}, + ] + if not missing_usage: + records.append({"usage": {"prompt_tokens": count, "completion_tokens": 3, + "prompt_tokens_details": {"cached_tokens": cached}}}) + return io.BytesIO(("".join("data: " + json.dumps(record) + "\n" for record in records) + + "data: [DONE]\n").encode()) + return send + + +def test_calibration_preserves_conversation_and_template(monkeypatch): + calls = [] + monkeypatch.setattr(soak, "request", fake_http(calls)) + history = [{"role": "user", "content": "original"}, {"role": "assistant", "content": "reply"}] + original = json.dumps(history) + messages, count = soak.calibrated_user(config(), history, "seed-agent-1", 1000) + assert abs(count - 1000) <= 10 + assert messages[:2] == history + assert json.dumps(history) == original + assert all(payload["chat_template_kwargs"] == {"enable_thinking": True} for _, payload, _ in calls) + + +def test_stream_records_ids_usage_reasoning_and_delta_clock(monkeypatch): + calls = [] + monkeypatch.setattr(soak, "request", fake_http(calls, cached=1)) + ticks = iter([100, 101, 103, 104, 105]) + record, assistant = soak.stream_turn(config(), [{"role": "user", "content": "fixture"}], + 100, 64, "identity", "soak", request_id="fixture-id", clock=lambda: next(ticks)) + assert record["request_id"] == calls[0][2] == "fixture-id" + assert record["response_id"] == "chatcmpl-fixture" + assert record["cached_tokens_reported"] == 1 + assert record["ttft_seconds"] == 1 + assert record["elapsed_seconds"] == 5 + assert record["content_delta_offsets_seconds"] == [1, 3] + assert record["decode_tokens_per_second_estimate"] == 1 + assert assistant == {"role": "assistant", "content": "answer", "reasoning_content": "think"} + assert record["output_budget_exhausted"] + assert calls[0][1]["reasoning_effort"] == "low" + + +def test_guard_and_missing_usage_fail_closed(monkeypatch): + calls = [] + monkeypatch.setattr(soak, "request", fake_http(calls, missing_usage=True)) + with pytest.raises(ValueError, match="context limit"): + soak.stream_turn(config(), [], 8192, 64, "id", "soak") + assert not calls + with pytest.raises(ValueError, match="authoritative"): + soak.stream_turn(config(), [{"role": "user", "content": "text"}], 50, 64, "id", "soak") + + +def test_bounded_conversations_and_before_after_probes(monkeypatch, tmp_path): + calls = [] + monkeypatch.setattr(soak, "request", fake_http(calls)) + path = tmp_path / "receipt.jsonl" + assert soak.execute(config(), path) == 0 + records = [json.loads(line) for line in path.read_text().splitlines()] + turns = [record for record in records if record["type"] == "turn"] + assert len(turns) == 8 + assert [record["phase"] for record in turns][0] == "before" + assert [record["phase"] for record in turns][-1] == "after" + assert sum(record.get("continuation", False) for record in turns) == 4 + assert len({record["request_id"] for record in turns}) == 8 + assert len({record["prompt_sha256"] for record in turns}) == 8 + assert records[-1]["soak_turns"] == 6 + assert records[-1]["fraction_below_half_cached"] == 1 + assert "test-key" not in path.read_text() + with pytest.raises(FileExistsError): + soak.execute(config(), path) + + +def test_global_admission_budget_counts_all_agents(monkeypatch, tmp_path): + monkeypatch.setattr(soak, "request", fake_http([])) + path = tmp_path / "budget.jsonl" + assert soak.execute(config(max_soak_prompt_tokens=2100), path) == 0 + summary = json.loads(path.read_text().splitlines()[-1]) + assert summary["soak_prompt_tokens_admitted"] <= 2100 + assert summary["soak_turns"] == 2 + assert summary["probes"]["after"]["samples"] == 1 + + +def test_failure_stops_before_soak_and_redacts_error(monkeypatch, tmp_path): + calls = [] + monkeypatch.setattr(soak, "request", fake_http(calls, error=True)) + path = tmp_path / "failed.jsonl" + assert soak.execute(config(), path) == 2 + records = [json.loads(line) for line in path.read_text().splitlines()] + errors = [record for record in records if record["type"] == "error"] + assert len(errors) == 1 and errors[0]["request_id"] + assert records[-1]["soak_turns"] == 0 + assert "test-key" not in path.read_text() + assert sum(url.endswith("/v1/chat/completions") for url, _, _ in calls) == 1 + + +def test_unknown_cache_usage_is_not_a_miss(): + rows = [{"type": "turn", "phase": "soak", "valid": True, "continuation": True, + "cached_fraction_reported": None, "elapsed_seconds": 10}] + summary = soak.summarize(rows) + assert summary["continuations"] == 1 + assert summary["continuations_with_cache_usage"] == 0 + assert summary["fraction_below_half_cached"] is None + + +def test_fixture_seed_reproducible_and_image_added(monkeypatch): + assert soak.fixture("seed-one", 1000) == soak.fixture("seed-one", 1000) + assert soak.fixture("seed-one", 1000) != soak.fixture("seed-two", 1000) + monkeypatch.setattr(soak, "request", fake_http([])) + messages, count = soak.calibrated_user(config(), [], "image-seed", 1000, "data:image/png;base64,fixture") + assert messages[0]["content"][1]["type"] == "image_url" + assert abs(count - 1000) <= 10 + + +def test_plan_requires_no_endpoint_or_key(monkeypatch, capsys): + monkeypatch.setattr(soak, "request", lambda *args, **kwargs: pytest.fail("unexpected HTTP")) + monkeypatch.setattr(sys, "argv", ["conversation_soak.py", "--plan", "--model", "glm-5.3-flash-spark", + "--arm", "baseline-mtp3", "--context-limit", "1m"]) + assert soak.main() == 0 + plan = json.loads(capsys.readouterr().out) + assert plan["max_chat_requests"] == 86 + assert "api_key" not in plan["config"] diff --git a/performance/records/glm53-flash/mtp3-cache-history-observations.json.gz b/performance/records/glm53-flash/mtp3-cache-history-observations.json.gz new file mode 100644 index 00000000..17c3d990 Binary files /dev/null and b/performance/records/glm53-flash/mtp3-cache-history-observations.json.gz differ diff --git a/performance/records/glm53-flash/mtp3-cache-history-validation.md b/performance/records/glm53-flash/mtp3-cache-history-validation.md new file mode 100644 index 00000000..4a007783 --- /dev/null +++ b/performance/records/glm53-flash/mtp3-cache-history-validation.md @@ -0,0 +1,77 @@ +# Persistent-cache pressure with native MTP3 decoding + +Status: **research-only**. This record qualifies only the functional and +measurement conditions below; it is not a production throughput guarantee. + +## Conditions + +GLM-5.3-Flash-NVFP4-Spark revision +`df116c4fb16b1d37ae43d2cfd624de26ffbc832e` ran on four NVIDIA GB10 hosts with +TP4/DCP4, native MTP depth three, 24 GiB KV capacity per rank, and dual-rail +hardware-forwarded RoCE transport. The serving image was +`sha256:75050f7b4dd7287f1ecb3e7e34226d24aa6c5b3e012a8bd6c8399e411cfbd908`, +containing SparkCache `607ccef061d0f511f45a2a8a93f74514c955d3a3` and the source +transforms in `runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/`. + +The transport library SHA-256 was +`056243fad27d224b82e437925ffa2aed42037e6bd29f239f56076a832f6ca5cb`; +the SparkCache placement library SHA-256 was +`2657cdd2e54a097c9544e4c79ae62c0646db6db123ff24e4f0c384238c3a1e8d`. +The persisted cache used a 2 GiB maximum and 1.5 GiB low watermark per rank, +eight restore lanes, and a 16,384-token full-capture interval. Periodic full +capture is opt-in and increases write traffic. + +Four text-only conversations started around 100K tokens and grew by about +2K tokens per turn, with a 512-token response limit and rotation at 160K. +Admission lasted 3,600 seconds; a 100M prompt-token ceiling was not reached. +Thinking was enabled. Three approximately 5.3K-token, 300-output-token probes +ran before and after load. Startup and warmup completed before measurement. + +## Measurement + +The harness `performance/harnesses/validation/conversation_soak.py` records +client monotonic-clock TTFT and stream-delta offsets. Decode rate is estimated +as `(completion_tokens - 1) / (last_delta - first_delta)`; it is not exact +token-level inter-token latency. Reported cached tokens come from API usage. +They do not independently distinguish local hits from external restores. + +The numeric observations are in +[`mtp3-cache-history-observations.json.gz`](mtp3-cache-history-observations.json.gz). +The artifact identifies the raw receipt and harness by SHA-256 and preserves +all 551 response timings and stream offsets. Endpoint addresses, request IDs, +and prompt/generated text are omitted; numeric observations are unchanged. +The artifact verifier recomputes response counts and probe medians. + +## Result + +The run completed 551 responses without request errors: 545 conversation +requests and six probes. It admitted 70,074,391 prompt tokens, including +cached tokens, in 60m58.46s including probes and drain. All 525 continuations +reported at least half their prompt tokens cached. + +| Probe median | Before load | After load | +|---|---:|---:| +| Client TTFT | 2.9436 s | 2.9798 s | +| Estimated decode rate | 51.20 tokens/s | 47.80 tokens/s | + +The decode estimate decreased 6.65%; individual probe ranges overlap. Three +samples per side do not support a precise confidence interval or a claim of +zero performance drift. + +## Conclusion + +This four-rank composition completed the specified sustained cache workload +without request errors or a population of low-reported-cache continuations. +The probes did not reproduce the approximately 35% post-load slowdown +described in SparkCache issue #60 under these conditions. + +## Limitations + +The image combines SparkCache and runtime changes, so this is not an isolated +comparison of any individual patch. No matched original-image or restore-only +control was completed. The 2 GiB test policy is not the original issues' +40 GiB cache policy; multimodal traffic, C8/C16, and near-1M serving are not +qualified. Some resumed schedules still miss safely. One startup attempt +exited for an undetermined reason; the same image passed startup on retry. +This record does not qualify unattended startup availability or close +SparkCache issues #60 or #61. diff --git a/performance/records/glm53-flash/test_mtp3_cache_history_record.py b/performance/records/glm53-flash/test_mtp3_cache_history_record.py new file mode 100644 index 00000000..6487c09f --- /dev/null +++ b/performance/records/glm53-flash/test_mtp3_cache_history_record.py @@ -0,0 +1,22 @@ +import gzip +import json +import statistics +from pathlib import Path + + +def test_mtp3_cache_history_numeric_observations(): + path = Path(__file__).with_name('mtp3-cache-history-observations.json.gz') + record = json.loads(gzip.decompress(path.read_bytes())) + turns = record['turns'] + assert len(turns) == 551 and all(t['valid'] for t in turns) + assert record['summary']['errors'] == 0 + soak = [t for t in turns if t['phase'] == 'soak'] + assert len(soak) == 545 + assert sum(t['tokenized_prompt_tokens'] for t in soak) == 70074391 + assert sum(t['cached_tokens_reported'] > 0 for t in soak) == 525 + for phase in ('before', 'after'): + probes = [t for t in turns if t['phase'] == phase] + assert len(probes) == 3 + expected = record['summary']['probes'][phase] + assert statistics.median(t['ttft_seconds'] for t in probes) == expected['median_ttft_seconds'] + assert statistics.median(t['decode_tokens_per_second_estimate'] for t in probes) == expected['median_decode_tokens_per_second_estimate'] diff --git a/runtime/glm53-flash-jj-r8-gb10/README.md b/runtime/glm53-flash-jj-r8-gb10/README.md index 95bfd2c4..bee09a47 100644 --- a/runtime/glm53-flash-jj-r8-gb10/README.md +++ b/runtime/glm53-flash-jj-r8-gb10/README.md @@ -178,7 +178,8 @@ ports 19006/19007 and secondary ports 19106/19107. The derivation reserves two ports for each admitted capacity Q1024/Q2048/Q4096/Q8192. SIRCL's two transport slots are independent from SparkCache's two 3-GiB -asynchronous page-capture slots and two 256-MiB restore arenas. +asynchronous page-capture slots and sixteen 256-MiB restore arenas (two for +each of eight load lanes). #### Recorded functional evidence @@ -228,6 +229,11 @@ With the connector enabled, `SPARKCACHE_ACCESS_MODE=read-write` restores and publishes persistent entries. `restore-only` reuses compatible entries but does not capture or publish new prompt state. Missing entries are computed by vLLM normally. `store-only` and `disabled` are diagnostic modes. +The environment template sets `SPARKCACHE_ASYNC_PAGE_CAPTURE=auto`: capture +is enabled for `read-write` and `store-only`, and disabled for `restore-only`, +`disabled`, or `SPARKCACHE_ENABLED=0`. Explicit `1` still rejects a mode that +cannot publish. Explicit `0` uses synchronous publication in publishing modes. +The launcher defaults to `0` when no capture setting is supplied. ### Choose the persistent publication format @@ -277,14 +283,43 @@ not discard Triton, TorchInductor, B12X, or vLLM compilation caches. Each rank keeps its own persistent copy under `CACHE_HOST_ROOT`; the four ranks do not write to one network-shared compilation directory. -Set `SPARKCACHE_ASYNC_PAGE_CAPTURE=1` to capture manager pages through the +Set `SPARKCACHE_ASYNC_PAGE_CAPTURE=auto` to capture manager pages through the bounded CUDA ring. `SPARKCACHE_ASYNC_CAPTURE_SLOT_BYTES` defaults to 8 GiB for DCP1, 5 GiB for DCP2, and 3 GiB for DCP4. The DCP4 profile uses two 3 GiB capture slots, so the background publisher can consume one completed capture while a later capture uses the other. Restore separately pipelines bounded -NVMe reads and CUDA placement through two 256 MiB mapped arenas. A third arena -is not part of the profile because the two-stage pipeline has no measured -arena wait that would justify more unified-memory pressure. +NVMe reads and CUDA placement through two 256 MiB mapped arenas **per load +lane**. Eight lanes reserve 4 GiB per rank for restore payloads. With the 6 GiB +capture ring, the DCP4 profile configures 10 GiB per rank (40 GiB across TP4), +in addition to the 24 GiB per-rank KV allocation. Restore-only configures +4 GiB per rank and no capture slots. These figures describe payload capacities; +control arrays, Python objects, shared bases, transport, model weights and +allocator overhead require additional memory. `SPARKCACHE_LOAD_THREADS` defaults +to eight; throughput and memory-pressure effects need hardware measurements. + +### Inspect configured memory before launch + +Status: **implemented**. The launcher can print a JSON allocation plan without +Docker, GPUs, checkpoint files, or cache directories. It sources the same trusted +shell configuration as a launch and resolves its byte counts and capture mode: + +```bash +SPARKRING_PRINT_MEMORY_PLAN=1 bash runtime/glm53-flash-jj-r8-gb10/launch-rank.sh \ + 0 runtime/glm53-flash-jj-r8-gb10/runtime.env.example +``` + +Set `SPARKCACHE_BUFFER_BUDGET_BYTES` in the configuration to reject restore +and capture payload capacities above that per-rank ceiling before host checks +or Docker access. Zero, the default, disables the ceiling. For example, +`10737418240` admits the 10 GiB DCP4 read-write configuration exactly. The +report includes per-rank and topology totals; DCP shards state but does not +reduce the physical TP rank count. A normal launch also logs the plan. + +This ceiling does not cap total process memory or predict whether serving fits. +It excludes model weights, KV allocation, transient reads, retained shared bases, +CUDA control arrays, compilation and transport buffers, and allocator overhead. +The report lists KV separately. A passing offline plan does not qualify a CUDA +allocation or a serving performance result. When `DFLASH_WARMUP=1`, the readiness entrypoint runs `warmup_dflash.py` before Docker reports rank 0 as healthy. The readiness wrapper, diff --git a/runtime/glm53-flash-jj-r8-gb10/launch-rank.sh b/runtime/glm53-flash-jj-r8-gb10/launch-rank.sh index 2090f0a3..bb205a48 100644 --- a/runtime/glm53-flash-jj-r8-gb10/launch-rank.sh +++ b/runtime/glm53-flash-jj-r8-gb10/launch-rank.sh @@ -28,6 +28,11 @@ fi : "${CONTAINER_PREFIX:=glm53-jj-r8-gb10}" : "${SPARKRING_CREATE_ONLY:=0}" : "${SPARKRING_PRINT_CONTAINER_SPEC:=0}" +: "${SPARKRING_PRINT_MEMORY_PLAN:=0}" +case "${SPARKRING_PRINT_MEMORY_PLAN}" in + 0|1) ;; + *) printf 'SPARKRING_PRINT_MEMORY_PLAN must be 0 or 1\n' >&2; exit 78 ;; +esac case "${SPARKRING_PRINT_CONTAINER_SPEC}" in 0|1) ;; *) printf 'SPARKRING_PRINT_CONTAINER_SPEC must be 0 or 1\n' >&2; exit 78 ;; @@ -132,6 +137,7 @@ esac : "${SPARKCACHE_ASYNC_PAGE_CAPTURE:=0}" : "${SPARKCACHE_ASYNC_CAPTURE_SLOT_BYTES:=auto}" : "${SPARKCACHE_ASYNC_CAPTURE_SLOT_COUNT:=2}" +: "${SPARKCACHE_BUFFER_BUDGET_BYTES:=0}" : "${SPARKCACHE_SOURCE_OVERLAY:=}" : "${VLLM_KV_METRICS_OVERLAY:=}" : "${MULTIMODAL_INPUTS:=1}" @@ -184,6 +190,7 @@ do done require_uint SPARKCACHE_LOW_WATERMARK_BYTES require_uint SPARKCACHE_TTL_SECONDS +require_uint SPARKCACHE_BUFFER_BUDGET_BYTES require_uint NCCL_IB_GID_INDEX require_uint MAX_IMAGES_PER_PROMPT require_uint MAX_VIDEOS_PER_PROMPT @@ -297,13 +304,21 @@ case "${ENABLE_PROMPT_TOKENS_DETAILS}" in *) die 'ENABLE_PROMPT_TOKENS_DETAILS must be 0 or 1' ;; esac case "${SPARKCACHE_ASYNC_PAGE_CAPTURE}" in - 0|1) ;; - *) die 'SPARKCACHE_ASYNC_PAGE_CAPTURE must be 0 or 1' ;; + auto|0|1) ;; + *) die 'SPARKCACHE_ASYNC_PAGE_CAPTURE must be auto, 0, or 1' ;; esac case "${SPARKCACHE_ACCESS_MODE}" in read-write|restore-only|store-only|disabled) ;; *) die 'SPARKCACHE_ACCESS_MODE must be read-write, restore-only, store-only, or disabled' ;; esac +if [[ "${SPARKCACHE_ASYNC_PAGE_CAPTURE}" == auto ]]; then + SPARKCACHE_ASYNC_PAGE_CAPTURE=0 + if [[ "${SPARKCACHE_ENABLED}" == 1 ]]; then + case "${SPARKCACHE_ACCESS_MODE}" in + read-write|store-only) SPARKCACHE_ASYNC_PAGE_CAPTURE=1 ;; + esac + fi +fi if [[ "${SPARKCACHE_ASYNC_PAGE_CAPTURE}" == 1 ]]; then [[ "${SPARKCACHE_ENABLED}" == 1 ]] || \ die 'asynchronous page capture requires SPARKCACHE_ENABLED=1' @@ -312,6 +327,69 @@ if [[ "${SPARKCACHE_ASYNC_PAGE_CAPTURE}" == 1 ]]; then *) die 'asynchronous page capture requires a publication-capable access mode' ;; esac fi + +# Resolve the payload buffers before inspecting checkpoints or contacting Docker. +# Python integers avoid overflow when comparing operator-supplied byte budgets. +command -v python3 >/dev/null 2>&1 || die 'python3 is required to resolve the memory plan' +export SPARKCACHE_ENABLED SPARKCACHE_ACCESS_MODE SPARKCACHE_LOAD_THREADS +export SPARKCACHE_CUDA_ARENA_BYTES SPARKCACHE_ASYNC_PAGE_CAPTURE +export SPARKCACHE_ASYNC_CAPTURE_SLOT_BYTES SPARKCACHE_ASYNC_CAPTURE_SLOT_COUNT +export SPARKCACHE_BUFFER_BUDGET_BYTES KV_CACHE_MEMORY_BYTES +export TENSOR_PARALLEL_SIZE PIPELINE_PARALLEL_SIZE DECODE_CONTEXT_PARALLEL_SIZE +memory_plan="$(python3 - <<'PY' +import json +import os +import sys + +def integer(name): + return int(os.environ[name]) + +enabled = os.environ["SPARKCACHE_ENABLED"] == "1" +mode = os.environ["SPARKCACHE_ACCESS_MODE"] +# The pinned manager-page connector caps active placement lanes at eight. +requested_lanes = integer("SPARKCACHE_LOAD_THREADS") +lanes = min(8, requested_lanes) if enabled and mode in ("read-write", "restore-only") else 0 +slots = integer("SPARKCACHE_ASYNC_CAPTURE_SLOT_COUNT") if os.environ["SPARKCACHE_ASYNC_PAGE_CAPTURE"] == "1" else 0 +restore = lanes * 2 * integer("SPARKCACHE_CUDA_ARENA_BYTES") +capture = slots * integer("SPARKCACHE_ASYNC_CAPTURE_SLOT_BYTES") +buffers = restore + capture +budget = integer("SPARKCACHE_BUFFER_BUDGET_BYTES") +ranks = integer("TENSOR_PARALLEL_SIZE") * integer("PIPELINE_PARALLEL_SIZE") +kv = integer("KV_CACHE_MEMORY_BYTES") +print(json.dumps({ + "status": "implemented", + "basis": "configured payload capacities; not measured resident memory", + "access_mode": mode if enabled else "disabled", + "async_page_capture": slots > 0, + "dcp_degree": integer("DECODE_CONTEXT_PARALLEL_SIZE"), + "rank_count": ranks, + "requested_restore_lanes_per_rank": requested_lanes, + "restore_lanes_per_rank": lanes, + "arenas_per_restore_lane": 2, + "capture_slots_per_rank": slots, + "restore_payload_bytes_per_rank": restore, + "capture_payload_bytes_per_rank": capture, + "sparkcache_payload_bytes_per_rank": buffers, + "sparkcache_payload_bytes_all_ranks": buffers * ranks, + "buffer_budget_bytes_per_rank": budget or None, + "within_buffer_budget": budget == 0 or buffers <= budget, + "kv_cache_bytes_per_rank": kv, + "kv_and_payload_bytes_per_rank": kv + buffers, + "kv_and_payload_bytes_all_ranks": (kv + buffers) * ranks, + "excluded": ["model weights", "CUDA control arrays", "Python objects and read buffers", + "shared base retention", "transport", "compilation workspaces", "allocator overhead"], +}, sort_keys=True)) +if budget and buffers > budget: + print(f"SparkCache payload buffers require {buffers} bytes per rank, exceeding " + f"SPARKCACHE_BUFFER_BUDGET_BYTES={budget}", file=sys.stderr) + sys.exit(78) +PY +)" || { printf '%s\n' "${memory_plan}"; exit 78; } +if [[ "${SPARKRING_PRINT_MEMORY_PLAN}" == 1 ]]; then + printf '%s\n' "${memory_plan}" + exit 0 +fi +printf 'sparkcache: memory_plan %s\n' "${memory_plan}" >&2 if [[ -n "${CHAT_TEMPLATE_HOST_PATH}" ]]; then [[ "${CHAT_TEMPLATE_HOST_PATH}" == /* ]] || \ die 'CHAT_TEMPLATE_HOST_PATH must be an absolute host path when set' diff --git a/runtime/glm53-flash-jj-r8-gb10/runtime.env.example b/runtime/glm53-flash-jj-r8-gb10/runtime.env.example index 0931cda2..4b59df75 100644 --- a/runtime/glm53-flash-jj-r8-gb10/runtime.env.example +++ b/runtime/glm53-flash-jj-r8-gb10/runtime.env.example @@ -176,9 +176,15 @@ SPARKCACHE_LOAD_THREADS=8 SPARKCACHE_MAX_PENDING_RESTORES=8 SPARKCACHE_CUDA_RESTORE_IO_WORKERS=8 SPARKCACHE_CUDA_ARENA_BYTES=268435456 +# Each load lane owns two mapped arenas: 8 * 2 * 256 MiB = 4 GiB per rank. +# Optional per-rank ceiling for restore and capture payload buffers, in bytes. +# Zero disables the ceiling. This excludes models, KV, control arrays and reads. +SPARKCACHE_BUFFER_BUDGET_BYTES=0 # Two bounded capture slots let the background publisher consume one capture -# while a later request uses the other. Set to 0 for synchronous publication. -SPARKCACHE_ASYNC_PAGE_CAPTURE=1 +# while a later request uses the other. Auto enables capture only for enabled +# read-write/store-only modes; restore-only and disabled modes allocate no slots. +# Set to 0 for synchronous publication or 1 to require asynchronous capture. +SPARKCACHE_ASYNC_PAGE_CAPTURE='auto' SPARKCACHE_ASYNC_CAPTURE_SLOT_BYTES='auto' SPARKCACHE_ASYNC_CAPTURE_SLOT_COUNT=2 diff --git a/runtime/glm53-flash-jj-r8-gb10/test_image_contract.py b/runtime/glm53-flash-jj-r8-gb10/test_image_contract.py index 0a4a8570..25bcbe36 100644 --- a/runtime/glm53-flash-jj-r8-gb10/test_image_contract.py +++ b/runtime/glm53-flash-jj-r8-gb10/test_image_contract.py @@ -470,7 +470,7 @@ def test_launcher_keeps_gather_workspace_below_native_context_limit() -> None: "KV_CACHE_MEMORY_BYTES='auto'", "B12X_MLA_CKV_GATHER_MAX_TOKENS=524288", "SPARKCACHE_MAX_SPAN_TOKENS=1048576", - "SPARKCACHE_ASYNC_PAGE_CAPTURE=1", + "SPARKCACHE_ASYNC_PAGE_CAPTURE='auto'", "SPARKCACHE_ASYNC_CAPTURE_SLOT_BYTES='auto'", "SPARKCACHE_ASYNC_CAPTURE_SLOT_COUNT=2", "SPARKCACHE_SHARED_PREFIX_LEASE_TTL_SECONDS=300", diff --git a/runtime/glm53-flash-jj-r8-gb10/test_launcher_contract.py b/runtime/glm53-flash-jj-r8-gb10/test_launcher_contract.py index 333e5798..279ae9f4 100644 --- a/runtime/glm53-flash-jj-r8-gb10/test_launcher_contract.py +++ b/runtime/glm53-flash-jj-r8-gb10/test_launcher_contract.py @@ -58,6 +58,7 @@ def test_environment_exposes_reproducible_operator_defaults() -> None: assert values["SPARKCACHE_ENABLED"] == "1" assert values["ENABLE_PROMPT_TOKENS_DETAILS"] == "1" assert values["SPARKCACHE_ACCESS_MODE"] == "read-write" + assert values["SPARKCACHE_ASYNC_PAGE_CAPTURE"] == "auto" assert values["SPARKCACHE_SHARED_PREFIX_LEASE_TTL_SECONDS"] == "300" assert values["SPARKCACHE_CACHE_NAMESPACE"] == ( "glm53-flash-vllm-e02b1746-b12x-9ae41c5c-" @@ -74,8 +75,110 @@ def test_environment_exposes_reproducible_operator_defaults() -> None: ) +def _memory_plan(tmp_path: Path, *settings: str) -> subprocess.CompletedProcess[str]: + config = tmp_path / "memory-plan.env" + # Nonexistent checkpoint and cache paths prove this mode needs no model, + # cache directories, Docker daemon, GPU, or serving host. + config.write_text( + "\n".join(( + f"source '{_bash_path(ENVIRONMENT)}'", + "HOST_IP=rank0.example.net", + "MASTER_ADDR=rank0.example.net", + "TARGET_MODEL_HOST_PATH=/nonexistent-memory-plan/target", + "DFLASH_MODEL_HOST_PATH=/nonexistent-memory-plan/draft", + "CACHE_HOST_ROOT=/nonexistent-memory-plan/cache", + "SPARKRING_PRINT_MEMORY_PLAN=1", + *settings, + )), + encoding="utf-8", + newline="\n", + ) + return subprocess.run( + ["bash", _bash_path(LAUNCHER), "0", _bash_path(config)], + cwd=ROOT, text=True, capture_output=True, check=False, + ) + + +@pytest.mark.parametrize("dcp,slot_gib", [(1, 8), (2, 5), (4, 3)]) +def test_memory_plan_resolves_dcp_and_all_lane_allocations( + tmp_path: Path, dcp: int, slot_gib: int, +) -> None: + result = _memory_plan(tmp_path, f"DECODE_CONTEXT_PARALLEL_SIZE={dcp}") + assert result.returncode == 0, result.stderr + plan = json.loads(result.stdout) + gib = 1024 ** 3 + assert plan["restore_payload_bytes_per_rank"] == 4 * gib + assert plan["capture_payload_bytes_per_rank"] == 2 * slot_gib * gib + assert plan["sparkcache_payload_bytes_all_ranks"] == (4 + 2 * slot_gib) * gib * 4 + assert plan["kv_and_payload_bytes_all_ranks"] == (24 + 4 + 2 * slot_gib) * gib * 4 + assert plan["buffer_budget_bytes_per_rank"] is None + + +@pytest.mark.parametrize("mode,enabled,restore_gib,capture_gib", [ + ("read-write", 1, 4, 6), ("restore-only", 1, 4, 0), + ("store-only", 1, 0, 6), ("disabled", 1, 0, 0), + ("read-write", 0, 0, 0), +]) +def test_auto_capture_respects_access_mode( + tmp_path: Path, mode: str, enabled: int, restore_gib: int, capture_gib: int, +) -> None: + result = _memory_plan(tmp_path, f"SPARKCACHE_ACCESS_MODE={mode}", f"SPARKCACHE_ENABLED={enabled}") + assert result.returncode == 0, result.stderr + plan = json.loads(result.stdout) + assert plan["restore_payload_bytes_per_rank"] == restore_gib * 1024 ** 3 + assert plan["capture_payload_bytes_per_rank"] == capture_gib * 1024 ** 3 + assert plan["async_page_capture"] == bool(capture_gib) + + +@pytest.mark.parametrize("setting,error", [ + ("SPARKCACHE_ACCESS_MODE=restore-only", "publication-capable access mode"), + ("SPARKCACHE_ACCESS_MODE=disabled", "publication-capable access mode"), + ("SPARKCACHE_ENABLED=0", "requires SPARKCACHE_ENABLED=1"), +]) +def test_explicit_capture_contradictions_remain_errors( + tmp_path: Path, setting: str, error: str, +) -> None: + result = _memory_plan(tmp_path, setting, "SPARKCACHE_ASYNC_PAGE_CAPTURE=1") + assert result.returncode == 78 + assert error in result.stderr + + +def test_buffer_budget_rejects_overcommit_before_host_checks(tmp_path: Path) -> None: + required = 10 * 1024 ** 3 + result = _memory_plan(tmp_path, f"SPARKCACHE_BUFFER_BUDGET_BYTES={required - 1}") + assert result.returncode == 78 + assert "exceeding SPARKCACHE_BUFFER_BUDGET_BYTES" in result.stderr + assert json.loads(result.stdout)["within_buffer_budget"] is False + accepted = _memory_plan(tmp_path, f"SPARKCACHE_BUFFER_BUDGET_BYTES={required}") + assert accepted.returncode == 0, accepted.stderr + assert json.loads(accepted.stdout)["within_buffer_budget"] is True + + +def test_explicit_synchronous_capture_and_custom_lane_sizes(tmp_path: Path) -> None: + result = _memory_plan(tmp_path, "SPARKCACHE_ASYNC_PAGE_CAPTURE=0", + "SPARKCACHE_LOAD_THREADS=3", "SPARKCACHE_CUDA_ARENA_BYTES=1048576") + assert result.returncode == 0, result.stderr + plan = json.loads(result.stdout) + assert plan["capture_payload_bytes_per_rank"] == 0 + assert plan["restore_payload_bytes_per_rank"] == 6 * 1048576 + + +def test_memory_plan_uses_connector_lane_limit(tmp_path: Path) -> None: + result = _memory_plan(tmp_path, "SPARKCACHE_LOAD_THREADS=64") + assert result.returncode == 0, result.stderr + plan = json.loads(result.stdout) + assert plan["requested_restore_lanes_per_rank"] == 64 + assert plan["restore_lanes_per_rank"] == 8 + assert plan["restore_payload_bytes_per_rank"] == 4 * 1024 ** 3 + + +@pytest.mark.parametrize("capture_mode,access_mode,capture_enabled", [ + ("0", "read-write", False), + ("auto", "read-write", True), + ("auto", "restore-only", False), +]) def test_launcher_resolves_dcp_profiles_and_prompt_token_details( - tmp_path: Path, + tmp_path: Path, capture_mode: str, access_mode: str, capture_enabled: bool, ) -> None: subprocess.run(["bash", "-n", _bash_path(LAUNCHER)], check=True, cwd=ROOT) fake_bin = tmp_path / "bin" @@ -148,6 +251,8 @@ def test_launcher_resolves_dcp_profiles_and_prompt_token_details( "IMAGE_REF=test-image:r8", f"IMAGE_ID={IMAGE_ID}", f"DECODE_CONTEXT_PARALLEL_SIZE={dcp}", + f"SPARKCACHE_ASYNC_PAGE_CAPTURE={capture_mode}", + f"SPARKCACHE_ACCESS_MODE={access_mode}", ) ), encoding="utf-8", @@ -201,7 +306,8 @@ def test_launcher_resolves_dcp_profiles_and_prompt_token_details( extra = connector["kv_connector_extra_config"] assert extra["spark_cache_publication_schema"] == "tail-cow-v2" assert extra["spark_cache_model_profile"] == "glm53-flash-hybrid" - assert extra["spark_cache_access_mode"] == "read-write" + assert extra["spark_cache_access_mode"] == access_mode + assert extra["spark_cache_async_page_capture"] is capture_enabled assert extra["spark_cache_shared_prefix_lease_ttl_seconds"] == 300 assert "spark_cache_store" not in extra assert "spark_cache_restore" not in extra diff --git a/runtime/glm53-spark-mtp3-mesh/MANAGED_MESH.md b/runtime/glm53-spark-mtp3-mesh/MANAGED_MESH.md index eca6395f..f795f032 100644 --- a/runtime/glm53-spark-mtp3-mesh/MANAGED_MESH.md +++ b/runtime/glm53-spark-mtp3-mesh/MANAGED_MESH.md @@ -362,6 +362,7 @@ The bounded `--run-seconds` mode is for isolated diagnostics only. |---|---| | Child-process and peer checks | 1-second loop; peer HTTP timeout 2 seconds | | Unavailable peer connection | 4-second grace after the first transport failure; degraded health blocks model startup | +| Docker container status | One background query at a time, 3-second timeout; unknown status blocks model startup | | MAC/IP, Ethernet MTU, sysfs GID/netdev, routes, qdiscs, TC state | 5-second periodic check | | Full RDMA active-MTU probe | Startup and approximately every 60 seconds | | Health progress freshness | Readiness rejected after 10 seconds without supervisor progress | @@ -375,6 +376,14 @@ An authentication failure, explicit negative readiness, or changed process generation does not receive transport-error grace: it triggers failure when observed. Local marker exits also trigger failure without that grace. +Docker status queries run outside the fabric-monitor loop. A slow or failed +query reports `docker_status_degraded: true`; it does not declare fabric +failure or interrupt existing serving. Marker, network, and authenticated +peer checks continue. A completed query reporting that the model stopped +still enforces the model-exit policy. Pending queries never reuse a previous +stopped result as proof. Startup and teardown use separate synchronous +checks: unknown Docker state cannot authorize marker or network removal. + These are polling and timeout settings, not zero-window guarantees. Command execution, scheduling, management-network delays, and container-stop time affect detection and containment. In-flight requests can fail; a successful diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/.gitattributes b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/.gitattributes new file mode 100644 index 00000000..66e025bc --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/.gitattributes @@ -0,0 +1,3 @@ +# Transform script hashes include their original line endings. +patch_mtp3_*.py -text whitespace=cr-at-eol +fixtures/*.zip -text diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/README.md b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/README.md new file mode 100644 index 00000000..0feb01c6 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/README.md @@ -0,0 +1,173 @@ +# MTP3 cache-reuse source experiment + +Status: **research-only**. These source transforms coordinate recurrent prefix +reuse, GPU-lease accounting, and the fused indexer's histogram barrier for +GLM-5.3 native-MTP3. `compose.py` produces and verifies their exact Python output +from one specified base image. The package includes source inputs and CPU tests +of the allocation, lookup, retention, and accounting rules. + +The composition tool is implemented. Serving and performance qualification +require separate evidence for the complete image, SparkCache package, native +libraries, and workload. Published runtime pins and production Dockerfiles are +outside this experiment's output. + +The scope is GLM-5.3 native-MTP3 on four GB10 ranks, TP4/DCP4, with 512-token +recurrent/hash pages, 2,048-token attention scheduling alignment, one prefill +lookahead token, and aligned recurrent caching. Other speculation methods, +geometries and images require separate evidence. + +## Exact inputs and outputs + +The supported source input is local image ID +`sha256:2e41b1e934a85ff7c21b780532db2f0a0e978df081e52f4ae2bf11f8992fb24f`. +This is a local immutable image identity, not a registry pull reference. + +This composer and the [indexer-barrier image in PR #226](https://github.com/FujitsuPolycom/sparkring/pull/226) +are separate derivatives of that input. Both add histogram synchronization, but +their indexer output bytes and hashes differ. Do not apply this composer to the +barrier child: its preimage checks reject that source. A combined image needs +one selected indexer output, matching source receipts and cache namespace, and +validation of the complete composition. A source-branch merge alone does not +perform that integration. + +`fixtures/manifest.json` lists the ten input Python files, their paths in +that image, byte counts and SHA-256 values. The 155 KiB fixture archive contains +only those files. Source bytes and their existing SPDX/copyright headers are +preserved. Models, compiled libraries, credentials and site configuration are +excluded. Fixtures are test inputs and do not enter a production image build. + +| Component | Output or required-library SHA-256 | +|---|---| +| vLLM scheduler | `75efa57e7ff5a77c76714b85e2e4d8e1d7f456d9a9eec6c67ebb11ca382942f9` | +| vLLM single-type KV manager | `d2e35b012e0cf45ab3771f545c35ca48f2a5858549c574a352975607369124e2` | +| B12X fused indexer | `b43a4a2802c7dfc4a049bbb5751fc7e7688b05cb06d4ece716ab7a1d91d23d2a` | +| Separately supplied SparkCache placement library | `2657cdd2e54a097c9544e4c79ae62c0646db6db123ff24e4f0c384238c3a1e8d` | +| Required SparkCache capture library | `4398f18b8913e743e7bf1ed8fe29560d4580e61b6a1e2ab8b16684b19b6573b5` | + +The fixture manifest attests each patch script's exact bytes. Each transform +accepts only its declared input SHA-256 or its own output SHA-256. Apply them in +this dependency order: + +1. `patch_mtp3_barrier.py` adds the CTA synchronization before the histogram + arrival signal and increments that kernel's compile-cache revision. +2. `patch_mtp3_lease_accounting.py` includes an attached GPU lease in initial + prefill cache statistics without labeling it an external transfer. +3. `patch_mtp3_local_lease_preference.py` prefers a strictly longer converged + local prefix over a shorter GPU lease and reuses that lookup result. +4. `patch_mtp3_sparse_retention.py` pairs an explicit speculative replay + checkpoint stop with sparse retention of its predecessor state. +5. `patch_mtp3_partial_tail_eligibility.py` decides whether a partial recurrent + tail requires a stop from the recurrent page size, rather than attention's + DCP-scaled scheduling unit. + +Every transform rejects an unknown input and checks its complete output. Its +own output is idempotent; source produced by another transform must still match +one of those two declared hashes. Use `compose.py --verify-candidate` to check +the composed output tree. + +The `original/` directory contains the exact source inputs. The `candidate/` +directory contains transformed output, and `composition.json` records output +hashes and transform receipts. These directory names describe the tool's file +interface, not deployment or qualification status. + +## Offline composition and tests + +Python 3.10 or later is sufficient for composition. Pytest is needed for tests. +No Docker, vLLM installation, Torch, model files or GPU is required: + +```bash +python runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/compose.py --check +python -m pytest runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse -q -rs +``` + +To retain the exact source trees and receipt in a fresh local directory: + +```bash +python runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/compose.py \ + --output-root work/mtp3-cache-reuse-composition +python runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/compose.py \ + --verify-candidate work/mtp3-cache-reuse-composition/candidate +python runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/check_mtp3_checkpoint_allocations.py \ + --source-root work/mtp3-cache-reuse-composition/original/vllm \ + --candidate-root work/mtp3-cache-reuse-composition/candidate/vllm \ + --output work/mtp3-checkpoint-allocations.json +``` + +Existing output directories and result files are rejected. `--source-root` +on the composition command accepts an independently extracted original tree, +with `vllm/` and `b12x/` immediately beneath it; every file is checked against +the same manifest before output creation. + +The allocator checker executes the actual allocation and block-registration +methods against planned running-state and GDN checkpoint writes. It covers +336 fresh and 336 resumed empty-table replay schedules with varied shared +boundaries, chunk budgets and speculative buffer counts. A selected null slot +is skipped by the real registration method. A selected non-null slot whose +planned state does not match its hash boundary fails the check. This tests +metadata consistency, not completion of CUDA writes. Some resumed schedules +can still miss a prompt predecessor; the report distinguishes those safe misses +from stale-state registration. + +The tests also cover lease/API accounting, local-versus-lease selection, +speculative backoff, prompt-length boundaries and recurrent partial-tail +eligibility. One optional test uses the companion SparkCache checkout directly: + +```bash +SPARKCACHE_SOURCE_ROOT=/path/to/sparkcache python -m pytest \ + runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse -q -rs +``` + +That optional test needs the companion checkout's CPU development dependencies. +It is explicitly skipped when `SPARKCACHE_SOURCE_ROOT` is absent. + +## Independent fixture extraction + +Use only the exact base image above on a local Docker host. A stopped container +can expose its files without launching Python, CUDA or serving. The following +Python fragment creates that stopped container, copies only manifest-listed +files, and removes the temporary container. It changes local Docker metadata; +do not run it against a serving container or a remote Docker context. + +```python +import json +from pathlib import Path +import subprocess + +package = Path('runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse') +manifest = json.loads((package / 'fixtures/manifest.json').read_text()) +output = Path('work/mtp3-extracted-original') +output.mkdir(parents=True, exist_ok=False) +image = manifest['base_image_id'] +observed = subprocess.check_output(['docker', 'image', 'inspect', image, + '--format', '{{.Id}}'], text=True).strip() +assert observed == image +container = subprocess.check_output(['docker', 'create', '--entrypoint', + '/bin/true', image], text=True).strip() +try: + for name, record in manifest['files'].items(): + target = output / name + target.parent.mkdir(parents=True, exist_ok=True) + subprocess.run(['docker', 'cp', container + ':' + record['image_path'], + str(target)], check=True) +finally: + subprocess.run(['docker', 'rm', container], check=True) +``` + +Then run `compose.py --source-root work/mtp3-extracted-original --check` to +verify the independently copied inputs and complete transform chain. + +## Serving composition boundary + +To serve with these Python sources, select a SparkCache commit and supply both +native libraries with the hashes listed above. The experimental image also +needs a runtime contract that attests the output scheduler and manager hashes. +The contract is +`sparkcache/runtime_patches/vllm-manager-page-async-contract-55969c16.json`; +the exact scheduler and manager preimage hashes are in the fixture manifest. +Record the resulting contract, full SparkCache source tree, native libraries, +image ID, model identity, topology, namespace and workload settings together. +The source composer does not rewrite that contract or published source pins. + +CPU test success establishes the stated source and metadata checks. Throughput, +deployment readiness, and unmeasured preemption/CUDA schedules require hardware +evidence that identifies the complete image and its test conditions. diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/check_mtp3_checkpoint_allocations.py b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/check_mtp3_checkpoint_allocations.py new file mode 100644 index 00000000..a62855da --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/check_mtp3_checkpoint_allocations.py @@ -0,0 +1,211 @@ +"""Reproduce MTP3 aligned checkpoint allocation and cache registration on CPU. + +Execute allocation and BlockPool.cache_full_blocks from the declared source +fixtures together with the transformed split/retention methods. Synthetic block +state labels track the running endpoint and GDN internal checkpoint scheduled +for GPU writes. This checks metadata/allocator consistency; it does not execute +GPU kernels or prove those writes completed on hardware. +Fresh and resumed cases start with an empty block table (preemption replay). +""" +from __future__ import annotations + +import argparse +import ast +import datetime +import hashlib +import json +import os +from pathlib import Path +from types import SimpleNamespace + +from repro_mtp3_checkpoint_materialization import ( + checkpoint_kernel_metadata, extracted_method, load_scheduler, + partial_hit_eligibility, +) +from repro_mtp3_sparse_retention import MambaSpec, ROOT, execute, load_algorithms + +WORK = Path(__file__).resolve().parent +PATCHED = Path(os.environ.get('MTP3_RETENTION_SOURCE', str(WORK / 'mtp3-retention-patched'))) +EXPECTED_SCHEDULER = '75efa57e7ff5a77c76714b85e2e4d8e1d7f456d9a9eec6c67ebb11ca382942f9' +EXPECTED_MANAGER = 'd2e35b012e0cf45ab3771f545c35ca48f2a5858549c574a352975607369124e2' +EXPECTED_ALLOCATOR = '10846c4994e7860deab8b42c8bcd3315ddc96d14a478d4012c398418cc17a04c' + + +def sha256(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def extracted_classes(): + allocation = ast.parse('class Allocator:\n pass').body[0] + allocation.body = [extracted_method(ROOT / 'v1/core/single_type_kv_cache_manager.py', + 'MambaManager', method) + for method in ('_needs_internal_checkpoint', 'allocate_new_blocks')] + scope = {'MambaSpec': MambaSpec, 'cdiv': lambda a, b: (a + b - 1) // b} + execute([allocation], scope) + algorithms = load_algorithms(ROOT / 'v1/core/kv_cache_utils.py', + manager_source=PATCHED / 'v1/core/single_type_kv_cache_manager.py') + registration = ast.parse('class Registrar:\n pass').body[0] + registration.body = [extracted_method(ROOT / 'v1/core/block_pool.py', 'BlockPool', 'cache_full_blocks')] + register_scope = {'resolve_block_hashes': algorithms.resolve_block_hashes, + 'make_block_hash_with_group_id': lambda value, group: (value, group)} + execute([registration], register_scope) + return scope['Allocator'], register_scope['Registrar'], algorithms + + +def run_case(prompt, total, shared, budget, speculative, classes, Scheduler, checkpoint): + Allocator, Registrar, algorithms = classes + scheduler = Scheduler() + scheduler.cache_config = SimpleNamespace(block_size=512) + scheduler.use_eagle = True + scheduler.mamba_has_prefill_checkpoint_blocks = True + scheduler.hash_block_size = 512 + scheduler.block_size = 2048 + scheduler.need_mamba_block_aligned_split = True + scheduler.kv_cache_manager = SimpleNamespace(coordinator=SimpleNamespace(enable_partial_hash_hits=True)) + scheduler.mamba_partial_cache_hit = partial_hit_eligibility(scheduler, + SimpleNamespace(kv_cache_groups=[SimpleNamespace(kv_cache_spec=MambaSpec(512))]), + PATCHED / 'v1/core/sched/scheduler.py') + scheduler.max_num_scheduled_tokens = 8192 + scheduler.scheduler_config = SimpleNamespace(long_prefill_token_threshold=0) + scheduler.num_prefill_lookahead = 1 + scheduler._recurrent_publication_boundaries = lambda request: ((prompt - 1) // 2048 * 2048,) + request = SimpleNamespace(request_id='replay', num_prompt_tokens=prompt, num_tokens=total, + num_computed_tokens=0, shared_prefix_boundary=shared, + block_hashes=list(range(512, total + 1, 512))) + allocator = Allocator() + allocator.kv_cache_spec = MambaSpec(512) + allocator.kv_cache_spec.num_prefill_checkpoint_blocks = 1 + allocator.mamba_cache_mode = 'align' + allocator.block_size = 512 + allocator.num_speculative_blocks = speculative + allocator.req_to_blocks = {'replay': []} + allocator._num_checkpoint_blocks = {} + allocator._partial_hit_reqs = {} + allocator._allocated_block_reqs = set() + allocator.last_state_block_idx = {} + allocator._null_block = SimpleNamespace(is_null=True, state=None, block_hash=None) + allocator.block_pool = SimpleNamespace(get_new_blocks=lambda n: [ + SimpleNamespace(is_null=False, state=None, block_hash=None, block_hash_num_tokens=None) + for _ in range(n)]) + failures, retained, selected_null, steps = [], set(), 0, [] + registrar = Registrar() + registrar.hash_block_size = 512 + registrar.enable_kv_cache_events = False + + def insert(block_hash, block, *, num_tokens): + if block.state != num_tokens: + failures.append({'expected_tokens': num_tokens, 'state_tokens': block.state, + 'step_start': request.num_computed_tokens}) + else: + retained.add(num_tokens) + block.block_hash, block.block_hash_num_tokens = block_hash, num_tokens + + def remove(block): + block.block_hash, block.block_hash_num_tokens = None, None + return [] + + registrar._insert_block_hash = insert + registrar._remove_cached_block_hashes = remove + registrar._emit_block_removed_events = lambda removed: None + cached = 0 + while request.num_computed_tokens < total: + start = request.num_computed_tokens + count = scheduler._mamba_block_aligned_split(request, min(budget, total - start)) + count = scheduler._reserve_prefill_lookahead(request, start, count) + if count <= 0: + raise AssertionError(f'zero progress: {prompt=}, {total=}, {shared=}, {budget=}, {start=}') + end = start + count + last = allocator.last_state_block_idx.get('replay') + # The source fixture's remove_skipped_blocks retires this state slot. + if last is not None and last < (start + 511) // 512 - 1: + allocator.req_to_blocks['replay'][last] = allocator._null_block + allocator._num_checkpoint_blocks['replay'] = int( + allocator._needs_internal_checkpoint('replay', end, start)) + allocator.allocate_new_blocks('replay', end, end) + blocks = allocator.req_to_blocks['replay'] + running = blocks[(end + 511) // 512 - 1] + assert not running.is_null + running.state = end + internal = checkpoint(start, end, 512) + if internal is not None and not blocks[internal // 512 - 1].is_null: + blocks[internal // 512 - 1].state = internal + mask = algorithms.MambaManager.reachable_block_mask( + start_block=cached, end_block=end // 512, alignment_tokens=2048, + kv_cache_spec=allocator.kv_cache_spec, use_eagle=True, retention_interval=0, + reachable_boundaries=(prompt - 1, shared)) + selected_null += sum(keep and blocks[index].is_null for index, keep in enumerate(mask, cached)) + # Execute cache registration so its null-slot handling participates in + # the check of every hash boundary against the planned state writes. + registrar.cache_full_blocks(request, blocks, cached, end // 512, 512, 1, mask) + steps.append(end) + cached, request.num_computed_tokens = end // 512, end + assert len(steps) < 1024 + required_prompt = ((prompt - 1) // 512 - 1) * 512 + return {'prompt_tokens': prompt, 'num_tokens': total, 'shared_boundary': shared, + 'budget': budget, 'speculative_blocks': speculative, 'steps': steps, + 'retained_boundaries': sorted(retained), 'selected_null_slots': selected_null, + 'stale_or_unwritten_registered_states': failures, + 'prompt_predecessor_retained': required_prompt in retained} + + +def main(): + global ROOT, PATCHED + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--output', type=Path, required=True) + parser.add_argument('--source-root', type=Path, default=ROOT, help='Original vllm directory produced by compose.py') + parser.add_argument('--candidate-root', type=Path, default=PATCHED, help='Candidate vllm directory produced by compose.py') + args = parser.parse_args() + ROOT, PATCHED = args.source_root, args.candidate_root + # The extracted helpers resolve their unchanged support files from this root. + import repro_mtp3_sparse_retention as sparse_helpers + import repro_mtp3_checkpoint_materialization as checkpoint_helpers + sparse_helpers.ROOT = ROOT + checkpoint_helpers.ROOT = ROOT + paths = {'scheduler': PATCHED / 'v1/core/sched/scheduler.py', + 'retention_manager': PATCHED / 'v1/core/single_type_kv_cache_manager.py', + 'allocator_manager': ROOT / 'v1/core/single_type_kv_cache_manager.py', + 'block_pool': ROOT / 'v1/core/block_pool.py', + 'gdn_metadata': ROOT / 'v1/attention/backends/gdn_attn.py', + 'hash_utils': ROOT / 'v1/core/kv_cache_utils.py', + 'coordinator': ROOT / 'v1/core/kv_cache_coordinator.py', + 'algorithm_loader': WORK / 'repro_mtp3_sparse_retention.py', + 'checkpoint_loader': WORK / 'repro_mtp3_checkpoint_materialization.py', + 'checker': Path(__file__).resolve()} + identities = {name: {'path': str(path), 'sha256': sha256(path)} for name, path in paths.items()} + for name, expected in [('scheduler', EXPECTED_SCHEDULER), ('retention_manager', EXPECTED_MANAGER), + ('allocator_manager', EXPECTED_ALLOCATOR)]: + if identities[name]['sha256'] != expected: + raise RuntimeError(f'unexpected {name} SHA-256: {identities[name]["sha256"]}') + classes = extracted_classes() + Scheduler = load_scheduler(paths['scheduler']) + checkpoint = checkpoint_kernel_metadata() + populations = {'fresh': [(value, value) for value in (32768, 32789, 33280, 100968)], + 'resumed': [(32789, 33301), (32789, 40000), (32768, 33301), (32768, 40000)]} + result = {'schema': 'mtp3-checkpoint-allocation-registration-check/v1', + 'time_utc': datetime.datetime.now(datetime.timezone.utc).isoformat(), + 'source_inputs': identities, 'gpu_executed': False, 'populations': {}, + 'limits': ['Checks allocator/retention against planned endpoint/checkpoint writes; no GPU execution', + 'Fresh and resumed requests start from empty block tables, not an injected partial local hit', + 'Only align mode with 512-token recurrent/hash pages, DCP4 retention alignment2048, and lookahead1', + 'Missing retained prompt predecessors are reported safe misses, not treated as corruption']} + for label, pairs in populations.items(): + cases = [run_case(prompt, total, shared, budget, speculative, classes, Scheduler, checkpoint) + for prompt, total in pairs for shared in (8192, 12345, 15872, 16384, 16401, 20000, 30000) + for budget in (8192, 7680, 1024, 512) for speculative in (0, 1, 3)] + summary = {'cases': len(cases), 'selected_null_slots': sum(case['selected_null_slots'] for case in cases), + 'stale_or_unwritten_registered_states': sum(len(case['stale_or_unwritten_registered_states']) for case in cases), + 'cases_without_prompt_predecessor_retention': sum(not case['prompt_predecessor_retained'] for case in cases)} + result['populations'][label] = {'summary': summary, 'cases': cases} + assert summary['cases'] == 336 + result['passed'] = all(not item['summary']['stale_or_unwritten_registered_states'] + for item in result['populations'].values()) + with args.output.open('x', encoding='utf-8') as stream: + json.dump(result, stream, indent=2) + stream.write('\n') + print(json.dumps({'passed': result['passed'], 'output': str(args.output), + 'summary': {name: item['summary'] for name, item in result['populations'].items()}}, indent=2)) + return 0 if result['passed'] else 1 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/compose.py b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/compose.py new file mode 100644 index 00000000..49559a85 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/compose.py @@ -0,0 +1,125 @@ +"""Build hash-verified MTP3 Python source output in an unused local directory. + +The candidate directory is transformed source output. Its hashes verify source +composition; serving qualification requires separate complete-image evidence. +The tool performs no Docker, network, or model operations. +""" +from __future__ import annotations + +import argparse +import hashlib +import json +from pathlib import Path +import shutil +import tempfile +import zipfile + +import patch_mtp3_barrier as barrier +import patch_mtp3_lease_accounting as accounting +import patch_mtp3_local_lease_preference as preference +import patch_mtp3_partial_tail_eligibility as partial +import patch_mtp3_sparse_retention as retention + +HERE = Path(__file__).resolve().parent +FIXTURES = HERE / 'fixtures' +NATIVE_PLACEMENT_SHA256 = '2657cdd2e54a097c9544e4c79ae62c0646db6db123ff24e4f0c384238c3a1e8d' +FINAL_SHA256 = { + 'vllm/v1/core/sched/scheduler.py': partial.AFTER_SHA256, + 'vllm/v1/core/single_type_kv_cache_manager.py': retention.MANAGER_AFTER, + 'b12x/attention/dsa_indexer/fused_indexer.py': barrier.AFTER_SHA256, +} + + +def sha256(data): + return hashlib.sha256(data).hexdigest() + + +def manifest(): + return json.loads((FIXTURES / 'manifest.json').read_text(encoding='utf-8')) + + +def source_bytes(source_root=None): + """Validate every input before creating any output files.""" + metadata = manifest() + if source_root is None: + archive = FIXTURES / 'original-python-sources.zip' + if sha256(archive.read_bytes()) != metadata['archive_sha256']: + raise ValueError('Source fixture archive checksum differs') + with zipfile.ZipFile(archive) as bundle: + if sorted(bundle.namelist()) != sorted(metadata['files']): + raise ValueError('Source fixture archive members differ') + content = {name: bundle.read(name) for name in metadata['files']} + else: + content = {name: (Path(source_root) / name).read_bytes() for name in metadata['files']} + for name, data in content.items(): + if sha256(data) != metadata['files'][name]['sha256']: + raise ValueError(f'Original source checksum differs: {name}') + for name, expected in metadata['transform_script_sha256'].items(): + if sha256((HERE / name).read_bytes()) != expected: + raise ValueError(f'Attested transform script differs: {name}') + return content + + +def verify_candidate(root): + metadata = manifest() + verified = {} + for name, original in metadata['files'].items(): + expected = FINAL_SHA256.get(name, original['sha256']) + observed = sha256((Path(root) / name).read_bytes()) + if observed != expected: + raise ValueError(f'Candidate source checksum differs: {name}') + verified[name] = observed + return verified + + +def compose(output, *, source_root=None): + content = source_bytes(source_root) + output = Path(output) + output.mkdir(parents=True, exist_ok=False) + original, candidate = output / 'original', output / 'candidate' + for name, data in content.items(): + path = original / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(data) + shutil.copytree(original, candidate) + scheduler = candidate / 'vllm/v1/core/sched/scheduler.py' + steps = [barrier.apply_patch(candidate / 'b12x/attention/dsa_indexer/fused_indexer.py'), + accounting.apply_patch(scheduler), preference.apply_patch(scheduler), + retention.apply_patch(candidate / 'vllm'), partial.apply_patch(scheduler)] + result = {'schema': 'sparkring-mtp3-cache-reuse-composition/v1', 'status': 'research-only', + 'base_image_id': manifest()['base_image_id'], 'speculation': 'native MTP3', + 'topology': 'TP4/DCP4, 512-token recurrent/hash pages', + 'candidate_files': verify_candidate(candidate), 'transform_receipts': steps, + 'required_native_placement_sha256': NATIVE_PLACEMENT_SHA256, + 'limits': ['Python source composition only; no SparkCache package or native binary is installed', + 'No serving, transport, GPU, or throughput qualification is implied', + 'Published runtime pins and production Dockerfiles are unchanged']} + (output / 'composition.json').write_text(json.dumps(result, indent=2) + '\n', encoding='utf-8') + return result + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--source-root', type=Path, help='Original fixture layout from the exact base image; defaults to bundled fixtures') + parser.add_argument('--output-root', type=Path, help='Fresh destination; existing directories are rejected') + parser.add_argument('--check', action='store_true', help='Compose and verify only in a temporary directory') + parser.add_argument('--verify-candidate', type=Path, help='Read-only verification of an already composed candidate directory') + args = parser.parse_args() + try: + if args.verify_candidate: + result = {'verified_files': verify_candidate(args.verify_candidate)} + elif args.check: + with tempfile.TemporaryDirectory(prefix='mtp3-cache-reuse-') as temporary: + result = compose(Path(temporary) / 'composition', source_root=args.source_root) + elif args.output_root: + result = compose(args.output_root, source_root=args.source_root) + else: + parser.error('Choose --check, --output-root, or --verify-candidate') + except (ValueError, OSError) as error: + parser.error(str(error)) + print(json.dumps(result, sort_keys=True)) + return 0 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/conftest.py b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/conftest.py new file mode 100644 index 00000000..7c6f36c2 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/conftest.py @@ -0,0 +1,32 @@ +"""Make source-extracted tests self-contained using an attested temporary composition.""" +import os +from pathlib import Path +import sys +import tempfile + +HERE = Path(__file__).resolve().parent +sys.path.insert(0, str(HERE)) +import compose # noqa: E402 + + +def pytest_configure(config): + temporary = tempfile.TemporaryDirectory(prefix='mtp3-cache-reuse-tests-') + config._mtp3_cache_reuse_temporary = temporary + output = Path(temporary.name) / 'composition' + compose.compose(output) + values = {'MTP3_ORIGINAL_SOURCE': str(output / 'original/vllm'), + 'MTP3_RETENTION_SOURCE': str(output / 'candidate/vllm'), + 'MTP3_SCHEDULER_SOURCE': str(output / 'candidate/vllm/v1/core/sched/scheduler.py')} + config._mtp3_cache_reuse_environment = {key: os.environ.get(key) for key in values} + os.environ.update(values) + + +def pytest_unconfigure(config): + for key, value in getattr(config, '_mtp3_cache_reuse_environment', {}).items(): + if value is None: + os.environ.pop(key, None) + else: + os.environ[key] = value + temporary = getattr(config, '_mtp3_cache_reuse_temporary', None) + if temporary: + temporary.cleanup() diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/fixtures/manifest.json b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/fixtures/manifest.json new file mode 100644 index 00000000..2d3de924 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/fixtures/manifest.json @@ -0,0 +1,66 @@ +{ + "schema": "sparkring-mtp3-cache-reuse-fixtures/v1", + "status": "research-only", + "base_image_id": "sha256:2e41b1e934a85ff7c21b780532db2f0a0e978df081e52f4ae2bf11f8992fb24f", + "purpose": "Exact original Python files required by source transforms and extracted CPU tests; not model/runtime assets", + "files": { + "b12x/attention/dsa_indexer/fused_indexer.py": { + "sha256": "893fbcade135b7e1d146b8fb6530cde0650be515f69bf9a17ced0a9c61a141e2", + "bytes": 143038, + "image_path": "/usr/local/lib/python3.12/dist-packages/b12x/attention/dsa_indexer/fused_indexer.py" + }, + "vllm/entrypoints/openai/chat_completion/serving.py": { + "sha256": "9982953285e9df469032a82fffa4095d0e9d86278bede6e2b91d03d02373d182", + "bytes": 56414, + "image_path": "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py" + }, + "vllm/v1/attention/backends/gdn_attn.py": { + "sha256": "fd75fb72efeb762ec558d332364e889df44a9ea483026bd5ded8f77f42be9b7f", + "bytes": 40208, + "image_path": "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/gdn_attn.py" + }, + "vllm/v1/core/block_pool.py": { + "sha256": "ddee56dccb2208411b3a035918e917ce8f56a9858471e9ca12b420d5d79bc69c", + "bytes": 33243, + "image_path": "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/block_pool.py" + }, + "vllm/v1/core/kv_cache_coordinator.py": { + "sha256": "149222b2b1de1ee3714c7c4e0843b9c38c730a605740ce7dc913308df906f20f", + "bytes": 41494, + "image_path": "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py" + }, + "vllm/v1/core/kv_cache_utils.py": { + "sha256": "9b5ec588681252832e322387fc7ecea3b146061305f8abb1b02bb55e219c2a8c", + "bytes": 100179, + "image_path": "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_utils.py" + }, + "vllm/v1/core/sched/scheduler.py": { + "sha256": "122d9722b48f8cec267d2136a18c184b8901a5a0a46187eab7030d6227f1983d", + "bytes": 157770, + "image_path": "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/scheduler.py" + }, + "vllm/v1/core/single_type_kv_cache_manager.py": { + "sha256": "10846c4994e7860deab8b42c8bcd3315ddc96d14a478d4012c398418cc17a04c", + "bytes": 89918, + "image_path": "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/single_type_kv_cache_manager.py" + }, + "vllm/v1/engine/output_processor.py": { + "sha256": "f46abafd45792b4816e75c0702113b510f4b3e15ea33016d155ee00abaa3fb59", + "bytes": 33588, + "image_path": "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py" + }, + "vllm/v1/metrics/stats.py": { + "sha256": "e8c3cbcfb880e7a2a496a1e499d4e4efccf70ba1eef56f4a1a6ee44cc0bf49bc", + "bytes": 22775, + "image_path": "/usr/local/lib/python3.12/dist-packages/vllm/v1/metrics/stats.py" + } + }, + "archive_sha256": "b54fa2cdde26099905be15a3462144877c2c511299642c744daecbcdcdcf7219", + "transform_script_sha256": { + "patch_mtp3_barrier.py": "751cb641cfe42ffa7aa907774a8059edcf69c446d0299612d12b13ed7a1230ec", + "patch_mtp3_lease_accounting.py": "ece6753aa79571eb0f7a9901ac8be69f3bfbf26d10bbdc315134847977d019ae", + "patch_mtp3_local_lease_preference.py": "3489feb646f2b768220422988a6d120a3f82e30972df8c1095d1c4b03fab0bb8", + "patch_mtp3_sparse_retention.py": "a6d27c5804fc4a79b01835beaae01d33a8ef495e98c30b655121d620c62d2afa", + "patch_mtp3_partial_tail_eligibility.py": "2e0609a62a3028841431e92531ea3b5af958cc863274b8518547ce56da7f2c89" + } +} diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/fixtures/original-python-sources.zip b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/fixtures/original-python-sources.zip new file mode 100644 index 00000000..c52e2a83 Binary files /dev/null and b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/fixtures/original-python-sources.zip differ diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/patch_mtp3_barrier.py b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/patch_mtp3_barrier.py new file mode 100644 index 00000000..dbb88a4a --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/patch_mtp3_barrier.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python3 +"""Patch the attested MTP3 B12X histogram barrier, preserving its source bytes.""" + +from __future__ import annotations + +import argparse +import ast +import hashlib +import json +from pathlib import Path + + +BEFORE_SHA256 = "893fbcade135b7e1d146b8fb6530cde0650be515f69bf9a17ced0a9c61a141e2" +AFTER_SHA256 = "b43a4a2802c7dfc4a049bbb5751fc7e7688b05cb06d4ece716ab7a1d91d23d2a" +_BEFORE = ''' """Grid barrier over the group's CTAs on the arrival counter; returns the next phase.""" + arrival_ptr = _fused_state_ptr(state, group_id, Int32(_FUSED_STATE_ARRIVAL)) +'''.replace("\n", "\r\n").encode() +_AFTER = ''' """Grid barrier over the group's CTAs on the arrival counter; returns the next phase.""" + # Every publishing warp must finish before the leader releases this CTA's + # arrival; otherwise peers can scan partial histograms and diverge in rounds. + cute.arch.sync_threads() + arrival_ptr = _fused_state_ptr(state, group_id, Int32(_FUSED_STATE_ARRIVAL)) +'''.replace("\n", "\r\n").encode() +_OLD_CACHE = b'"attention.indexer.fused_indexer", 1, cache_key, labels=labels' +_NEW_CACHE = b'"attention.indexer.fused_indexer", 2, cache_key, labels=labels' + + +def apply_patch(path: Path, *, check_only: bool = False) -> dict: + source = path.read_bytes() + digest = hashlib.sha256(source).hexdigest() + if digest == AFTER_SHA256: + return {"status": "already_patched", "source_sha256": digest, "changed": False} + if digest != BEFORE_SHA256: + raise RuntimeError(f"unsupported MTP3 B12X source preimage: {digest}") + if source.count(_BEFORE) != 1 or source.count(_OLD_CACHE) != 1: + raise RuntimeError("attested barrier or compile revision anchor differs") + patched = source.replace(_BEFORE, _AFTER, 1).replace(_OLD_CACHE, _NEW_CACHE, 1) + if hashlib.sha256(patched).hexdigest() != AFTER_SHA256: + raise RuntimeError("MTP3 B12X transform differs from its expected postimage") + ast.parse(patched.decode("utf-8"), filename=str(path)) + if not check_only: + path.write_bytes(patched) + return {"status": "checked" if check_only else "patched", "before_sha256": digest, + "after_sha256": AFTER_SHA256, "changed": not check_only, "compile_revision": 2} + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("path", type=Path) + parser.add_argument("--check", action="store_true", help="validate the transform without writing") + args = parser.parse_args() + print(json.dumps(apply_patch(args.path, check_only=args.check), sort_keys=True)) diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/patch_mtp3_lease_accounting.py b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/patch_mtp3_lease_accounting.py new file mode 100644 index 00000000..e1db6902 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/patch_mtp3_lease_accounting.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python3 +"""Account for resident GPU lease reuse in the pinned MTP3 prefill statistics.""" + +from __future__ import annotations + +import argparse +import ast +import hashlib +import json +from pathlib import Path + + +BEFORE_SHA256 = "122d9722b48f8cec267d2136a18c184b8901a5a0a46187eab7030d6227f1983d" +AFTER_SHA256 = "3ec6f357fdc770212528ca3f5dfd48234186f972ec5657a514e8b841df61e48b" +_BEFORE = b''' request.num_computed_tokens = min( + attached_tokens, request.num_tokens - 1 + ) + attached = getattr( +''' +_AFTER = b''' request.num_computed_tokens = min( + attached_tokens, request.num_tokens - 1 + ) + # Lease attachment skips hash lookup but still reuses + # resident GPU state. Include it in prefill/API totals + # without reporting a fresh external KV transfer. + if request.prefill_stats and request.num_preemptions <= 0: + request.prefill_stats.set( + num_prompt_tokens=request.num_prompt_tokens, + num_local_cached_tokens=request.num_computed_tokens, + num_external_cached_tokens=0, + ) + attached = getattr( +''' + + +def apply_patch(path: Path, *, check_only: bool = False) -> dict: + source = path.read_bytes() + digest = hashlib.sha256(source).hexdigest() + if digest == AFTER_SHA256: + return {"status": "already_patched", "source_sha256": digest, "changed": False} + if digest != BEFORE_SHA256: + raise RuntimeError(f"unsupported MTP3 scheduler preimage: {digest}") + if source.count(_BEFORE) != 1: + raise RuntimeError("MTP3 lease attachment anchor differs") + patched = source.replace(_BEFORE, _AFTER, 1) + if hashlib.sha256(patched).hexdigest() != AFTER_SHA256: + raise RuntimeError("MTP3 lease accounting postimage differs") + ast.parse(patched.decode("utf-8"), filename=str(path)) + if not check_only: + path.write_bytes(patched) + return {"status": "checked" if check_only else "patched", "changed": not check_only, + "before_sha256": digest, "after_sha256": AFTER_SHA256} + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("path", type=Path) + parser.add_argument("--check", action="store_true") + args = parser.parse_args() + print(json.dumps(apply_patch(args.path, check_only=args.check), sort_keys=True)) diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/patch_mtp3_local_lease_preference.py b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/patch_mtp3_local_lease_preference.py new file mode 100644 index 00000000..9ab4ab94 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/patch_mtp3_local_lease_preference.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 +"""Prefer a strictly longer converged GPU prefix over a shorter shared lease.""" + +from __future__ import annotations + +import argparse +import ast +import hashlib +import json +from pathlib import Path + + +BEFORE_SHA256 = "3ec6f357fdc770212528ca3f5dfd48234186f972ec5657a514e8b841df61e48b" +AFTER_SHA256 = "0df01bf90bbe6ab1e6bc127ca7b15f6286ace944b430c994539d3d81049f2b4a" +TRANSFORMS = ( + (b" did_prefix_cache_lookup = False\n", + b" did_prefix_cache_lookup = False\n local_lease_alternative = None\n"), + (b''' candidate = get_lease(request) if get_lease is not None else None + if candidate is not None: + lease_key, lease_tokens = candidate +''', b''' candidate = get_lease(request) if get_lease is not None else None + if candidate is not None: + lease_key, lease_tokens = candidate + if 0 < lease_tokens <= request.num_tokens: + # This lookup reconciles every KV group and applies + # speculative backoff before any blocks are adopted. + alternative = self.kv_cache_manager.get_computed_blocks(request) + if alternative[1] > min(lease_tokens, request.num_tokens - 1): + local_lease_alternative = (*alternative, False) + candidate = None + # Release this request's follower binding only; + # the verified lease and its other users remain. + rejected = getattr( + self.connector, "shared_prefix_lease_rejected", None + ) + if rejected is not None: + rejected(request_id, lease_key) + if candidate is not None: + lease_key, lease_tokens = candidate +'''), + (b''' ) = self._get_local_prefix_cache_hit(request) +''', b''' ) = ( + local_lease_alternative + if local_lease_alternative is not None + else self._get_local_prefix_cache_hit(request) + ) +'''), +) + + +def apply_patch(path: Path, *, check_only: bool = False) -> dict: + source = path.read_bytes() + digest = hashlib.sha256(source).hexdigest() + if digest == AFTER_SHA256: + return {"status": "already_patched", "source_sha256": digest, "changed": False} + if digest != BEFORE_SHA256: + raise RuntimeError(f"unsupported accounted MTP3 scheduler preimage: {digest}") + patched = source + for before, after in TRANSFORMS: + if patched.count(before) != 1: + raise RuntimeError("MTP3 local/lease selection anchor differs") + patched = patched.replace(before, after, 1) + if hashlib.sha256(patched).hexdigest() != AFTER_SHA256: + raise RuntimeError("MTP3 local/lease selection postimage differs") + ast.parse(patched.decode("utf-8"), filename=str(path)) + if not check_only: + path.write_bytes(patched) + return {"status": "checked" if check_only else "patched", "changed": not check_only, + "before_sha256": digest, "after_sha256": AFTER_SHA256} + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("path", type=Path) + parser.add_argument("--check", action="store_true") + args = parser.parse_args() + print(json.dumps(apply_patch(args.path, check_only=args.check), sort_keys=True)) diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/patch_mtp3_partial_tail_eligibility.py b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/patch_mtp3_partial_tail_eligibility.py new file mode 100644 index 00000000..e5ca8802 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/patch_mtp3_partial_tail_eligibility.py @@ -0,0 +1,63 @@ +#!/usr/bin/env python3 +"""Avoid recurrent partial-tail stops created only by DCP attention geometry.""" + +from __future__ import annotations + +import argparse +import ast +import hashlib +import json +from pathlib import Path + + +BEFORE_SHA256 = "5c0bd785d2d17dce39cdf867a55b5327487070d315851f3c67d9c2cf7a1d4c49" +AFTER_SHA256 = "75efa57e7ff5a77c76714b85e2e4d8e1d7f456d9a9eec6c67ebb11ca382942f9" +BEFORE = b''' # A finer prefix_match_unit is configured: a mamba partial tail entry + # can only be registered by a step ending exactly at the prompt's last + # hash boundary, so the split adds that stop. + self.mamba_partial_cache_hit = ( + self.need_mamba_block_aligned_split + and self.hash_block_size < self.block_size + and self.kv_cache_manager.coordinator.enable_partial_hash_hits + ) +''' +AFTER = b''' # An interior recurrent-page hash needs an explicit tail stop. + # DCP can enlarge attention's scheduling unit without making the + # recurrent page larger than a hash, so inspect Mamba specs directly. + self.mamba_partial_cache_hit = ( + self.need_mamba_block_aligned_split + and any( + isinstance(group.kv_cache_spec, MambaSpec) + and self.hash_block_size < group.kv_cache_spec.block_size + for group in kv_cache_config.kv_cache_groups + ) + and self.kv_cache_manager.coordinator.enable_partial_hash_hits + ) +''' + + +def apply_patch(path: Path, *, check_only=False): + source = path.read_bytes() + digest = hashlib.sha256(source).hexdigest() + if digest == AFTER_SHA256: + return {"status": "already_patched", "source_sha256": digest, "changed": False} + if digest != BEFORE_SHA256: + raise RuntimeError(f"unsupported checkpoint-patched MTP3 scheduler: {digest}") + if source.count(BEFORE) != 1: + raise RuntimeError("MTP3 partial-tail eligibility anchor differs") + patched = source.replace(BEFORE, AFTER, 1) + if hashlib.sha256(patched).hexdigest() != AFTER_SHA256: + raise RuntimeError("MTP3 partial-tail eligibility postimage differs") + ast.parse(patched.decode("utf-8")) + if not check_only: + path.write_bytes(patched) + return {"status": "checked" if check_only else "patched", "changed": not check_only, + "before_sha256": digest, "after_sha256": AFTER_SHA256} + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("path", type=Path) + parser.add_argument("--check", action="store_true") + args = parser.parse_args() + print(json.dumps(apply_patch(args.path, check_only=args.check), sort_keys=True)) diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/patch_mtp3_sparse_retention.py b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/patch_mtp3_sparse_retention.py new file mode 100644 index 00000000..63a2f205 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/patch_mtp3_sparse_retention.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python3 +"""Pair speculative replay checkpoint materialization with sparse retention. + +Attested for the installed scheduler after lease accounting/local preference. +Serving qualification targets native MTP3 with 512-token hash/Mamba pages and +DCP4; no GPU or model execution is performed by this source transformer. +""" + +from __future__ import annotations + +import argparse +import ast +import hashlib +import json +from pathlib import Path + + +SCHEDULER_BEFORE = "0df01bf90bbe6ab1e6bc127ca7b15f6286ace944b430c994539d3d81049f2b4a" +SCHEDULER_AFTER = "5c0bd785d2d17dce39cdf867a55b5327487070d315851f3c67d9c2cf7a1d4c49" +MANAGER_BEFORE = "10846c4994e7860deab8b42c8bcd3315ddc96d14a478d4012c398418cc17a04c" +MANAGER_AFTER = "d2e35b012e0cf45ab3771f545c35ca48f2a5858549c574a352975607369124e2" +SCHEDULER_TRANSFORMS = ( + (b''' if self.use_eagle: + last_cache_position = max(last_cache_position - block_size, 0) +''', b''' if self.use_eagle: + # Lookup excludes the final prompt token, then drops its proof + # block. The corresponding recurrent state needs its own stop. + last_cache_position = max( + (request.num_tokens - 1) // block_size * block_size - block_size, 0 + ) +'''), + (b''' if use_internal_checkpoint: + last_cache_position = 0 +''', b''' if use_internal_checkpoint and not self.use_eagle: + # The internal checkpoint covers the final aligned state only; + # speculative replay may require the preceding state as well. + last_cache_position = 0 +'''), +) +MANAGER_TRANSFORMS = ( + (b''' if start_block <= boundary_block < end_block: + mask[boundary_block - start_block] = True + + return mask + + def remove_skipped_blocks( +''', b''' if start_block <= boundary_block < end_block: + mask[boundary_block - start_block] = True + + if use_eagle: + # Retain the predecessor when materialized, alongside the + # scheduler-aligned fallback. Unmaterialized shared-junction + # slots remain null and cache_full_blocks skips them. + predecessor_block = boundary_tokens // block_size - 2 + if start_block <= predecessor_block < end_block: + mask[predecessor_block - start_block] = True + + return mask + + def remove_skipped_blocks( +'''), +) + + +def transform(source, expected_before, expected_after, transforms): + digest = hashlib.sha256(source).hexdigest() + if digest == expected_after: + return source + if digest != expected_before: + raise RuntimeError(f"unsupported MTP3 checkpoint preimage: {digest}") + for before, after in transforms: + if source.count(before) != 1: + raise RuntimeError("MTP3 checkpoint source anchor differs") + source = source.replace(before, after, 1) + if hashlib.sha256(source).hexdigest() != expected_after: + raise RuntimeError("MTP3 checkpoint postimage differs") + ast.parse(source.decode("utf-8")) + return source + + +def apply_patch(root: Path, *, check_only=False): + files = ( + (root / "v1/core/sched/scheduler.py", SCHEDULER_BEFORE, SCHEDULER_AFTER, SCHEDULER_TRANSFORMS), + (root / "v1/core/single_type_kv_cache_manager.py", MANAGER_BEFORE, MANAGER_AFTER, MANAGER_TRANSFORMS), + ) + prepared = [] + for path, before, after, transforms in files: + original = path.read_bytes() + patched = transform(original, before, after, transforms) + prepared.append((path, original, patched)) + # Validate both preimages before changing either source file. + if not check_only: + for path, original, patched in prepared: + if original != patched: + path.write_bytes(patched) + return {str(path.relative_to(root)): dict(before_sha256=hashlib.sha256(original).hexdigest(), + after_sha256=hashlib.sha256(patched).hexdigest(), changed=original != patched and not check_only) + for path, original, patched in prepared} + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("vllm_root", type=Path) + parser.add_argument("--check", action="store_true") + args = parser.parse_args() + print(json.dumps(apply_patch(args.vllm_root, check_only=args.check), sort_keys=True)) diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/repro_mtp3_checkpoint_materialization.py b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/repro_mtp3_checkpoint_materialization.py new file mode 100644 index 00000000..35993c4c --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/repro_mtp3_checkpoint_materialization.py @@ -0,0 +1,140 @@ +"""Trace MTP3 checkpoint scheduling and retention from exact source fixtures. + +The fixture manifest identifies the image and source inputs. The checker uses +512-token recurrent/hash pages and 2,048-token DCP4 scheduling alignment. +Controlled scheduling budgets expose produced and retained checkpoint +boundaries without GPU execution or a reconstruction of a live request trace. +""" + +import ast +import json +import math +from types import SimpleNamespace + +from repro_mtp3_sparse_retention import ROOT, FullAttentionSpec, MambaSpec, execute, load_algorithms + + +def extracted_method(path, cls_name, method_name): + tree = ast.parse(path.read_text()) + cls = next(node for node in tree.body if isinstance(node, ast.ClassDef) and node.name == cls_name) + return next(node for node in cls.body if isinstance(node, ast.FunctionDef) and node.name == method_name) + + +def load_scheduler(path=None): + path = path or ROOT / "v1/core/sched/scheduler.py" + shell = ast.parse("class Scheduler:\n pass\n").body[0] + shell.body = [extracted_method(path, "Scheduler", name) for name in + ("_mamba_block_aligned_split", "_reserve_prefill_lookahead")] + scope = {} + execute([shell], scope) + return scope["Scheduler"] + + +def partial_hit_eligibility(scheduler, kv_cache_config, path=None): + """Execute the source fixture's recurrent partial-tail eligibility rule.""" + tree = ast.parse((path or ROOT / "v1/core/sched/scheduler.py").read_text()) + nodes = [node for node in ast.walk(tree) if isinstance(node, ast.Assign) + and any(ast.unparse(target) == "self.mamba_partial_cache_hit" for target in node.targets)] + assert len(nodes) == 1 + execute(nodes, dict(self=scheduler, kv_cache_config=kv_cache_config, MambaSpec=MambaSpec)) + return scheduler.mamba_partial_cache_hit + + +def checkpoint_kernel_metadata(): + tree = ast.parse((ROOT / "v1/attention/backends/gdn_attn.py").read_text()) + loops = [node for node in ast.walk(tree) if isinstance(node, ast.For) + and ast.unparse(node.target) == "row" and ast.unparse(node.iter) == "request_rows"] + assert len(loops) == 1 + function = ast.parse("def checkpoint(start, end, block_size):\n pass\n").body[0] + function.body = ast.parse("all_query_lens=[end-start]\nseq_lens=[end]\nrequest_rows=[0]\ncheckpoint_offsets=[]\ncheckpoint_columns=[]").body + function.body.append(loops[0]) + function.body.extend(ast.parse("return start+checkpoint_offsets[0] if checkpoint_offsets[0] else None").body) + scope = {} + execute([function], scope) + return scope["checkpoint"] + + +def main(): + algorithms = load_algorithms(ROOT / "v1/core/kv_cache_utils.py") + utils = ast.parse((ROOT / "v1/core/kv_cache_utils.py").read_text()) + resolve = next(node for node in utils.body if isinstance(node, ast.FunctionDef) + and node.name == "resolve_kv_cache_block_sizes") + scope = dict(math=math, AttentionSpec=FullAttentionSpec, MambaSpec=MambaSpec) + execute([resolve], scope) + recurrent = MambaSpec(512) + recurrent.mamba_cache_mode = "align" + cache = SimpleNamespace(block_size=512, prefix_match_unit=None, enable_prefix_caching=True) + config = SimpleNamespace(cache_config=cache, + parallel_config=SimpleNamespace(decode_context_parallel_size=4), + kv_transfer_config=object()) + groups = SimpleNamespace(kv_cache_groups=[SimpleNamespace(kv_cache_spec=FullAttentionSpec(512)), + SimpleNamespace(kv_cache_spec=recurrent)]) + scheduler_size, hash_size = scope[resolve.name](groups, config) + assert (scheduler_size, hash_size) == (2048, 512) + checkpoint = checkpoint_kernel_metadata() + Scheduler = load_scheduler() + cases = [] + for label, budgets in ( + ("full_8192_token_budgets", [8192] * 8), + ("reduced_budget_materializes_predecessor", [8192, 8192, 8192, 7680, 8192, 8192]), + ): + scheduler = Scheduler() + scheduler.cache_config = cache + scheduler.use_eagle = True + scheduler.mamba_has_prefill_checkpoint_blocks = True + scheduler.mamba_partial_cache_hit = True + scheduler.hash_block_size = hash_size + scheduler.max_num_scheduled_tokens = 8192 + scheduler.scheduler_config = SimpleNamespace(long_prefill_token_threshold=0) + # One native MTP module, repeated three times, uses one prefill lookahead. + scheduler.num_prefill_lookahead = 1 + scheduler._recurrent_publication_boundaries = lambda request: (32768,) + request = SimpleNamespace(num_prompt_tokens=32789, num_tokens=32789, + num_computed_tokens=0, shared_prefix_boundary=0) + steps = [] + materialized = set() + retained = set() + cached_blocks = 0 + for budget in budgets: + start = request.num_computed_tokens + if start == request.num_prompt_tokens: + break + count = scheduler._mamba_block_aligned_split(request, min(budget, request.num_tokens-start)) + count = scheduler._reserve_prefill_lookahead(request, start, count) + assert count > 0 + end = start + count + internal = checkpoint(start, end, recurrent.block_size) + produced = {end} if end % recurrent.block_size == 0 else set() + if internal is not None: + produced.add(internal) + materialized.update(produced) + num_full = end // recurrent.block_size + mask = algorithms.MambaManager.reachable_block_mask( + start_block=cached_blocks, end_block=num_full, + alignment_tokens=scheduler_size, kv_cache_spec=recurrent, + use_eagle=True, retention_interval=0, + reachable_boundaries=(request.num_prompt_tokens-1,), + ) + registered = {(cached_blocks + index + 1) * recurrent.block_size + for index, keep in enumerate(mask) if keep} + retained.update(registered.intersection(materialized)) + steps.append(dict(start=start, end=end, internal_checkpoint=internal, + produced=sorted(produced), newly_retained=sorted(registered))) + cached_blocks = num_full + request.num_computed_tokens = end + assert request.num_computed_tokens == request.num_prompt_tokens + assert retained == {32768} + cases.append(dict(case=label, steps=steps, materialized=sorted(materialized), + retained=sorted(retained), required_predecessor=32256, + predecessor_materialized=32256 in materialized, + predecessor_retained=32256 in retained)) + assert not cases[0]["predecessor_materialized"] + assert cases[1]["predecessor_materialized"] + print(json.dumps(dict(schema="mtp3-installed-checkpoint-trace/v1", gpu_executed=False, + unique_effective_group_sizes=[2048,512], scheduler_block_size=scheduler_size, + hash_block_size=hash_size, lookup_alignment=hash_size, retention_alignment=scheduler_size, + retention_interval=0, num_prefill_checkpoint_blocks=1, cases=cases), sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/repro_mtp3_sparse_retention.py b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/repro_mtp3_sparse_retention.py new file mode 100644 index 00000000..58a70962 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/repro_mtp3_sparse_retention.py @@ -0,0 +1,168 @@ +"""Check prefix lookup and retention using exact vLLM source fixtures on CPU. + +The block pool contains explicitly described synthetic materialized checkpoints. +The result identifies lookup/retention mismatches under the supplied geometry. +It does not establish which intermediate states a live model materializes. +""" + +from __future__ import annotations + +import argparse +import ast +import copy +import hashlib +import itertools +import json +import os +from collections import namedtuple +from collections.abc import Sequence +from pathlib import Path +from types import SimpleNamespace +from typing import overload + + +ROOT = Path(os.environ.get("MTP3_ORIGINAL_SOURCE", str(Path(__file__).resolve().parent / "mtp3-vllm-source"))) + + +class FullAttentionSpec: + def __init__(self, block_size, dcp_replicated=False): + self.block_size = block_size + self.dcp_replicated = dcp_replicated + + +class MambaSpec: + def __init__(self, block_size): + self.block_size = block_size + + +class ChunkedLocalAttentionSpec: + pass + + +def execute(nodes, scope): + tree = ast.Module(body=[ast.ImportFrom(module="__future__", names=[ast.alias(name="annotations")], level=0), + *copy.deepcopy(nodes)], type_ignores=[]) + exec(compile(ast.fix_missing_locations(tree), "", "exec"), scope) + + +def load_algorithms(hash_utils, *, manager_source=None): + scope = dict(FullAttentionSpec=FullAttentionSpec, MambaSpec=MambaSpec, + ChunkedLocalAttentionSpec=ChunkedLocalAttentionSpec, Sequence=Sequence, + itertools=itertools, overload=overload, cdiv=lambda a, b: (a + b - 1) // b) + utils_tree = ast.parse(hash_utils.read_text()) + execute([node for node in utils_tree.body if isinstance(node, (ast.FunctionDef, ast.ClassDef)) + and node.name in {"BlockHashListWithBlockSize", "resolve_block_hashes"}], scope) + manager_tree = ast.parse((manager_source or ROOT / "v1/core/single_type_kv_cache_manager.py").read_text()) + for name, methods in (("FullAttentionManager", {"find_longest_cache_hit"}), + ("MambaManager", {"find_longest_cache_hit", "reachable_block_mask"})): + original = next(node for node in manager_tree.body if isinstance(node, ast.ClassDef) and node.name == name) + shell = ast.parse(f"class {name}:\n supports_fine_grained_hash_lookup = True\n").body[0] + shell.body += [node for node in original.body if isinstance(node, ast.FunctionDef) and node.name in methods] + execute([shell], scope) + coordinator = ast.parse((ROOT / "v1/core/kv_cache_coordinator.py").read_text()) + original = next(node for node in coordinator.body if isinstance(node, ast.ClassDef) + and node.name == "HybridKVCacheCoordinator") + shell = ast.parse("class HybridKVCacheCoordinator:\n pass\n").body[0] + shell.body = [node for node in original.body if isinstance(node, ast.FunctionDef) + and node.name in {"find_longest_cache_hit", "_cache_hit_alignment_tokens"}] + execute([shell], scope) + return SimpleNamespace(**scope) + + +class Pool: + def __init__(self, hash_block_size, retained_mamba, shared_boundary): + self.hash_block_size = hash_block_size + self.null_block = SimpleNamespace(is_null=True, block_hash=None) + self.retained_mamba = set(retained_mamba) + self.shared_boundary = shared_boundary + self.queries = [] + + def get_cached_block(self, block_hash, groups): + self.queries.append((block_hash, tuple(groups))) + if block_hash > self.shared_boundary or any( + group == 1 and block_hash not in self.retained_mamba for group in groups + ): + return None + return [SimpleNamespace(is_null=False, block_hash=(block_hash, group)) for group in groups] + + +def run_case(algorithms, args, *, drop=True, extra_checkpoint=None): + Full = algorithms.FullAttentionManager + Mamba = algorithms.MambaManager + target = FullAttentionSpec(args.page) + state = MambaSpec(args.page) + draft = FullAttentionSpec(args.page, dcp_replicated=True) + mask = Mamba.reachable_block_mask( + start_block=0, end_block=args.shared // args.page, + alignment_tokens=args.scheduler_alignment, kv_cache_spec=state, + use_eagle=False, retention_interval=0, reachable_boundaries=(args.prompt - 1,), + ) + retained = [(index + 1) * args.page for index, keep in enumerate(mask) if keep] + if extra_checkpoint is not None: + retained.append(extra_checkpoint) + pool = Pool(args.hash_unit, retained, args.shared) + coordinator = algorithms.HybridKVCacheCoordinator() + coordinator.kv_cache_config = SimpleNamespace(kv_cache_groups=(target, state, draft)) + coordinator.single_type_managers = ( + SimpleNamespace(block_size=args.page * args.dcp), + SimpleNamespace(block_size=args.page), SimpleNamespace(block_size=args.page), + ) + spec_group = namedtuple("SpecGroup", "spec group_ids manager_cls use_eagle") + coordinator.attention_groups = [spec_group(target, [0], Full, False), spec_group(state, [1], Mamba, False), + spec_group(draft, [2], Full, drop)] + coordinator.block_pool = pool + coordinator.hash_block_size = args.hash_unit + coordinator.scheduler_block_size = args.scheduler_alignment + coordinator.enable_partial_hash_hits = args.fine_hits + coordinator.dcp_world_size = args.dcp + hashes = list(range(args.hash_unit, args.continuation, args.hash_unit)) + separate = [] + for spec, groups, manager, use_eagle in coordinator.attention_groups: + _, hit = manager.find_longest_cache_hit( + block_hashes=hashes, max_length=args.continuation - 1, + kv_cache_group_ids=groups, block_pool=pool, kv_cache_spec=spec, + drop_eagle_block=use_eagle, alignment_tokens=coordinator._cache_hit_alignment_tokens, + dcp_world_size=args.dcp if isinstance(spec, FullAttentionSpec) else 1, + ) + separate.append(hit) + _, reconciled, uncached = coordinator.find_longest_cache_hit(hashes, args.continuation - 1) + return dict(drop_speculative_draft=drop, retained_mamba_tokens=retained, + extra_materialized_checkpoint=extra_checkpoint, + per_group_hit_tokens=separate, reconciled_hit_tokens=reconciled, + uncached_shared_prefix_tokens=uncached) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--hash-utils-source", type=Path, default=ROOT / "v1/core/kv_cache_utils.py") + parser.add_argument("--hash-unit", type=int, default=256) + parser.add_argument("--scheduler-alignment", type=int, default=2048) + parser.add_argument("--page", type=int, default=512) + parser.add_argument("--dcp", type=int, default=4) + parser.add_argument("--prompt", type=int, default=32789) + parser.add_argument("--continuation", type=int, default=34856) + parser.add_argument("--shared", type=int, default=32768) + parser.add_argument("--fine-hits", action=argparse.BooleanOptionalAction, default=True) + args = parser.parse_args() + algorithms = load_algorithms(args.hash_utils_source) + baseline = run_case(algorithms, args) + controls = [run_case(algorithms, args, drop=False)] + draft_hit = baseline["per_group_hit_tokens"][-1] + for unit in sorted({args.page, args.scheduler_alignment}): + checkpoint = draft_hit // unit * unit + if checkpoint > 0: + controls.append(run_case(algorithms, args, extra_checkpoint=checkpoint)) + paths = [ROOT / "v1/core/single_type_kv_cache_manager.py", ROOT / "v1/core/kv_cache_coordinator.py", + args.hash_utils_source] + print(json.dumps(dict(schema="mtp3-sparse-retention-metadata-reproducer/v1", + geometry={key: value for key, value in vars(args).items() if key != "hash_utils_source"}, + synthetic_checkpoint_metadata=True, gpu_executed=False, + sources={str(path): hashlib.sha256(path.read_bytes()).hexdigest() for path in paths}, + baseline=baseline, controls=controls), sort_keys=True)) + if baseline["reconciled_hit_tokens"] == 0 and baseline["per_group_hit_tokens"][0] > 0: + assert controls[0]["reconciled_hit_tokens"] > 0 + assert all(case["reconciled_hit_tokens"] > 0 for case in controls[1:]) + + +if __name__ == "__main__": + main() diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/test_composition.py b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/test_composition.py new file mode 100644 index 00000000..1b4a4d20 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/test_composition.py @@ -0,0 +1,70 @@ +"""Composition tests use bundled source fixtures; no Docker, Torch or GPU required.""" +import json +from pathlib import Path +import subprocess +import sys + +import pytest + +import compose + + +def test_composition_entrypoint_is_self_contained(tmp_path): + output = tmp_path / 'composition' + result = subprocess.run([sys.executable, str(Path(compose.__file__)), '--output-root', str(output)], + text=True, capture_output=True, check=False) + assert result.returncode == 0, result.stderr + receipt = json.loads(result.stdout) + assert receipt['candidate_files']['vllm/v1/core/sched/scheduler.py'].startswith('75efa57e') + assert receipt['candidate_files']['vllm/v1/core/single_type_kv_cache_manager.py'].startswith('d2e35b01') + assert compose.verify_candidate(output / 'candidate') == receipt['candidate_files'] + with pytest.raises(FileExistsError): + compose.compose(output) + + +def test_original_drift_rejected_before_output_created(tmp_path): + output = tmp_path / 'prepared' + compose.compose(output) + source = output / 'original' + (source / 'vllm/v1/core/sched/scheduler.py').write_bytes(b'not the attested source') + rejected = tmp_path / 'rejected' + with pytest.raises(ValueError, match='Original source checksum'): + compose.compose(rejected, source_root=source) + assert not rejected.exists() + + +def test_candidate_drift_rejected(tmp_path): + output = tmp_path / 'prepared' + compose.compose(output) + path = output / 'candidate/vllm/v1/core/sched/scheduler.py' + path.write_bytes(path.read_bytes() + b'\n') + with pytest.raises(ValueError, match='Candidate source checksum'): + compose.verify_candidate(output / 'candidate') + + +def test_each_final_patch_remains_idempotent(tmp_path): + output = tmp_path / 'prepared' + compose.compose(output) + candidate = output / 'candidate' + assert not compose.barrier.apply_patch(candidate / 'b12x/attention/dsa_indexer/fused_indexer.py')['changed'] + assert not compose.partial.apply_patch(candidate / 'vllm/v1/core/sched/scheduler.py')['changed'] + # Earlier chain stages intentionally reject later scheduler postimages. + with pytest.raises(RuntimeError, match='preimage'): + compose.accounting.apply_patch(candidate / 'vllm/v1/core/sched/scheduler.py') + + +def test_actual_allocator_checker_accepts_final_source_composition(tmp_path): + output = tmp_path / 'prepared' + compose.compose(output) + receipt = tmp_path / 'allocator.json' + result = subprocess.run([sys.executable, str(Path(compose.__file__).with_name('check_mtp3_checkpoint_allocations.py')), + '--source-root', str(output / 'original/vllm'), + '--candidate-root', str(output / 'candidate/vllm'), '--output', str(receipt)], + text=True, capture_output=True, check=False) + assert result.returncode == 0, result.stderr + data = json.loads(receipt.read_text()) + assert data['passed'] + assert data['source_inputs']['scheduler']['sha256'] == compose.partial.AFTER_SHA256 + for name in ('fresh', 'resumed'): + assert data['populations'][name]['summary']['cases'] == 336 + assert data['populations'][name]['summary']['stale_or_unwritten_registered_states'] == 0 diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/test_mtp3_lease_accounting.py b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/test_mtp3_lease_accounting.py new file mode 100644 index 00000000..3929aae4 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/test_mtp3_lease_accounting.py @@ -0,0 +1,118 @@ +"""Exercise extracted installed vLLM lease/statistics/API code without a GPU.""" + +import ast +import copy +import os +from dataclasses import dataclass +from pathlib import Path +from types import SimpleNamespace + +import pytest + + +ROOT = Path(os.environ["MTP3_ORIGINAL_SOURCE"]) + + +def compile_nodes(nodes, namespace=None): + scope = {} if namespace is None else dict(namespace) + tree = ast.fix_missing_locations(ast.Module(body=copy.deepcopy(nodes), type_ignores=[])) + exec(compile(tree, "", "exec"), scope) + return scope + + +def extract_class(path, name): + tree = ast.parse(path.read_text()) + return next(node for node in tree.body if isinstance(node, ast.ClassDef) and node.name == name) + + +def wrapper(name, arguments, statements): + tree = ast.parse(f"def {name}({arguments}):\n pass\n") + tree.body[0].body = statements + return tree.body[0] + + +@pytest.fixture +def runtime(): + stats = compile_nodes([extract_class(ROOT / "v1/metrics/stats.py", "PrefillStats")], + {"dataclass": dataclass})["PrefillStats"] + path = Path(os.environ.get("MTP3_SCHEDULER_SOURCE", str(ROOT / "v1/core/sched/scheduler.py"))) + tree = ast.parse(path.read_text()) + candidates = [node for node in ast.walk(tree) if isinstance(node, ast.If) + and ast.unparse(node.test) == "request.num_computed_tokens == 0 and self.connector is not None"] + assert len(candidates) == 1 + setup = ast.parse("request_id = request.request_id").body + apply_lease = compile_nodes([wrapper("apply_lease", "self, request", setup + candidates)])["apply_lease"] + processor = ast.parse((ROOT / "v1/engine/output_processor.py").read_text()) + output = [node for node in ast.walk(processor) if isinstance(node, ast.If) + and ast.unparse(node.test) == "req_state.is_prefilling"] + assert len(output) == 1 + receive = compile_nodes([wrapper("receive", "req_state, engine_core_output", output)])["receive"] + serving = ast.parse((ROOT / "entrypoints/openai/chat_completion/serving.py").read_text()) + helper = next(node for node in serving.body if isinstance(node, ast.FunctionDef) + and node.name == "_make_prompt_tokens_details") + details = compile_nodes([helper], {"PromptTokenUsageInfo": SimpleNamespace})[helper.name] + return SimpleNamespace(Stats=stats, apply=apply_lease, receive=receive, details=details) + + +def attach(runtime, prompt, lease, *, accepted=True, preempted=0, stats=True): + events = [] + request = SimpleNamespace(request_id="request", num_computed_tokens=0, + num_tokens=prompt, num_prompt_tokens=prompt, + num_preemptions=preempted, + prefill_stats=runtime.Stats() if stats else None) + connector = SimpleNamespace(get_shared_prefix_lease_candidate=lambda request: ("lease", lease), + shared_prefix_lease_attached=lambda *args: events.append("attached"), + shared_prefix_lease_rejected=lambda *args: events.append("rejected")) + scheduler = SimpleNamespace(connector=connector, kv_cache_manager=SimpleNamespace( + attach_shared_prefix_lease=lambda *args: lease if accepted else 0, + get_computed_blocks=lambda request: (None, 0, 0))) + runtime.apply(scheduler, request) + return scheduler, request, events + + +@pytest.mark.parametrize("prompt,lease,expected", [(40000, 32768, 32768), (32768, 32768, 32767)]) +def test_attached_gpu_prefix_reaches_api_total_without_external_transfer(runtime, prompt, lease, expected): + scheduler, request, events = attach(runtime, prompt, lease) + stats = request.prefill_stats + assert stats.num_cached_tokens == expected + assert stats.num_local_cached_tokens == expected + assert stats.num_external_cached_tokens == 0 + assert stats.num_computed_tokens == prompt - expected + assert request.num_computed_tokens == expected + # Reentering scheduling cannot double-count the attached prefix. + runtime.apply(scheduler, request) + assert events == ["attached"] + stats.finalize(prompt) + state = SimpleNamespace(is_prefilling=True, num_cached_tokens=0, num_cache_creation_tokens=0) + runtime.receive(state, SimpleNamespace(prefill_stats=stats)) + details = runtime.details(True, state.num_cached_tokens, state.num_cache_creation_tokens, None) + assert details.cached_tokens == expected + assert details.created_cache_tokens == prompt - expected + + +def test_rejected_lease_leaves_normal_lookup_accounting_untouched(runtime): + _, request, events = attach(runtime, 40000, 32768, accepted=False) + assert events == ["rejected"] + assert request.num_computed_tokens == 0 + assert vars(request.prefill_stats) == vars(runtime.Stats()) + + +def test_preemption_does_not_rewrite_first_prefill_stats(runtime): + _, request, events = attach(runtime, 40000, 32768, preempted=1) + assert events == ["attached"] + assert vars(request.prefill_stats) == vars(runtime.Stats()) + + +def test_absent_stats_does_not_change_lease_attachment(runtime): + _, request, events = attach(runtime, 40000, 32768, stats=False) + assert request.prefill_stats is None + assert request.num_computed_tokens == 32768 + assert events == ["attached"] + + +def test_existing_local_plus_external_total_remains_distinct(runtime): + stats = runtime.Stats() + stats.set(num_prompt_tokens=40000, num_local_cached_tokens=8000, num_external_cached_tokens=12000) + details = runtime.details(True, stats.num_cached_tokens, 0, None) + assert (stats.num_local_cached_tokens, stats.num_external_cached_tokens) == (8000, 12000) + assert details.cached_tokens == 20000 diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/test_mtp3_local_lease_preference.py b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/test_mtp3_local_lease_preference.py new file mode 100644 index 00000000..9302f4b2 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/test_mtp3_local_lease_preference.py @@ -0,0 +1,169 @@ +"""GPU-free execution of installed scheduler selection with allocator stubs.""" + +import ast +import os +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from test_mtp3_lease_accounting import ROOT, compile_nodes, wrapper + + +@pytest.fixture +def selection(): + path = Path(os.environ.get("MTP3_SCHEDULER_SOURCE", str(Path(__file__).parent / "mtp3_scheduler_accounted.py"))) + tree = ast.parse(path.read_text()) + lease = next(node for node in ast.walk(tree) if isinstance(node, ast.If) + and ast.unparse(node.test) == "request.num_computed_tokens == 0 and self.connector is not None") + ordinary = min((node for node in ast.walk(tree) if isinstance(node, ast.If) + and ast.unparse(node.test) == "request.num_computed_tokens == 0" + and node.lineno > lease.lineno), key=lambda node: node.lineno) + # Execute the real selection/lease branch and the real local-lookup assignment. + ordinary.body = ordinary.body[:2] + ordinary.orelse = [] + setup = ast.parse("request_id = request.request_id\nlocal_lease_alternative = None\ndid_prefix_cache_lookup = False\nnum_new_local_computed_tokens = 0\nhit_diverged = False").body + result = ast.parse("return did_prefix_cache_lookup, num_new_local_computed_tokens, hit_diverged").body + return compile_nodes([wrapper("select", "self, request", setup + [lease, ordinary] + result)])["select"] + + +def case(selection, *, local, lease=1024, attach=True, stale=False, preempted=False, candidate=True): + calls = [] + members = {"request", "other-follower"} + current = [local] + request = SimpleNamespace(request_id="request", num_computed_tokens=0, num_tokens=2048, + num_prompt_tokens=2048, prefill_stats=None, + num_preemptions=int(preempted), shared_prefix_boundary=0) + + def lookup(request): + calls.append("converged_lookup") + return ("local-blocks", current[0], current[0]) + + def ordinary(request): + calls.append("ordinary_lookup") + return (*lookup(request), False) + + def attach_lease(*args): + calls.append("attach") + if stale: + current[0] = 0 + return lease if attach else 0 + + def reject(*args): + calls.append("reject") + members.discard("request") + + connector = SimpleNamespace(get_shared_prefix_lease_candidate=lambda request: ("lease", lease) if candidate else None, + shared_prefix_lease_attached=lambda *args: calls.append("attached"), + shared_prefix_lease_rejected=reject) + scheduler = SimpleNamespace(connector=connector, + kv_cache_manager=SimpleNamespace(get_computed_blocks=lookup, attach_shared_prefix_lease=attach_lease), + _get_local_prefix_cache_hit=ordinary) + result = selection(scheduler, request) + return result, request, calls, members + + +def test_strictly_longer_converged_local_hit_skips_attachment_and_reuses_lookup(selection): + result, request, calls, members = case(selection, local=1536) + assert result == (True, 1536, False) + assert calls == ["converged_lookup", "reject"] + assert request.num_computed_tokens == 0 # Normal scheduler allocation owns adoption. + assert members == {"other-follower"} + + +@pytest.mark.parametrize("local", [0, 512, 1024]) +def test_equal_or_shorter_local_hit_keeps_lease(selection, local): + result, request, calls, members = case(selection, local=local) + assert not result[0] + assert request.num_computed_tokens == 1024 + assert calls == ["converged_lookup", "attach", "attached"] + assert members == {"request", "other-follower"} + + +def test_rejected_lease_returns_to_normal_lookup(selection): + result, request, calls, members = case(selection, local=512, attach=False) + assert result == (True, 512, False) + assert calls == ["converged_lookup", "attach", "reject", "ordinary_lookup", "converged_lookup"] + assert request.num_computed_tokens == 0 + assert members == {"other-follower"} + + +def test_failed_attachment_does_not_reuse_potentially_stale_probe_blocks(selection): + result, _, calls, _ = case(selection, local=512, attach=False, stale=True) + assert result == (True, 0, False) + assert calls[-2:] == ["ordinary_lookup", "converged_lookup"] + + +def test_preempted_request_still_uses_normal_allocation_for_a_longer_local_hit(selection): + result, request, calls, _ = case(selection, local=1536, preempted=True) + assert result == (True, 1536, False) + assert request.num_computed_tokens == 0 + assert "attach" not in calls + + +def test_no_lease_performs_only_the_ordinary_lookup(selection): + result, _, calls, _ = case(selection, local=1536, candidate=False) + assert result == (True, 1536, False) + assert calls == ["ordinary_lookup", "converged_lookup"] + + +@pytest.mark.skipif(not os.environ.get("SPARKCACHE_SOURCE_ROOT"), reason="optional companion SparkCache source integration") +def test_actual_connector_decline_preserves_verified_lease_and_other_followers(selection, tmp_path): + import sys + sys.path.insert(0, os.environ["SPARKCACHE_SOURCE_ROOT"]) + from sparkcache import test_spark_context_cache_connector as fixtures + + helper = fixtures.AsyncRestoreTests() + connector = helper._cohort_connector(tmp_path) + tokens = list(range(1600)) + digest = helper._offer(connector, tokens) + leader = SimpleNamespace(request_id="leader", prompt_token_ids=tokens) + assert connector.get_num_new_matched_tokens(leader, 0) == (1024, True) + connector.update_state_after_alloc(leader, helper._blocks_stub(), 1024) + connector.build_connector_meta(fixtures._empty_scheduler_output()) + connector.update_connector_output(SimpleNamespace(invalid_block_ids=set(), finished_recving={"leader"})) + assert connector.shared_prefix_lease_published("leader", digest) + other = SimpleNamespace(request_id="other", prompt_token_ids=tokens) + assert connector.get_shared_prefix_lease_candidate(other) == (digest, 1024) + request = SimpleNamespace(request_id="prefer-local", prompt_token_ids=tokens, + num_computed_tokens=0, num_tokens=len(tokens), num_prompt_tokens=len(tokens), + num_preemptions=0, prefill_stats=None, shared_prefix_boundary=0) + scheduler = SimpleNamespace(connector=connector, kv_cache_manager=SimpleNamespace( + get_computed_blocks=lambda request: ("local-blocks", 1280, 1280), + attach_shared_prefix_lease=lambda *args: pytest.fail("short lease was attached")), + _get_local_prefix_cache_hit=lambda request: pytest.fail("lookup repeated")) + try: + assert selection(scheduler, request) == (True, 1280, False) + assert "prefer-local" not in connector._restore_flight_followers + assert "other" in connector._restore_flight_followers + assert connector._restore_flights[digest].lease_published + assert connector.get_shared_prefix_lease_candidate(other) == (digest, 1024) + finally: + connector.shutdown() + + +def test_speculative_backoff_is_applied_before_comparing_with_lease(selection): + # Execute the installed full-attention finder with equal hash/page units. + # In this geometry resolve_block_hashes is its identity branch. + import itertools + from collections.abc import Sequence + from types import SimpleNamespace + + tree = ast.parse((ROOT / "v1/core/single_type_kv_cache_manager.py").read_text()) + original = next(node for node in tree.body if isinstance(node, ast.ClassDef) and node.name == "FullAttentionManager") + method = next(node for node in original.body if isinstance(node, ast.FunctionDef) and node.name == "find_longest_cache_hit") + shell = ast.parse("class FullAttentionManager:\n supports_fine_grained_hash_lookup = True\n").body[0] + shell.body.append(method) + future = ast.parse("from __future__ import annotations").body[0] + scope = compile_nodes([future, shell], dict(FullAttentionSpec=SimpleNamespace, + ChunkedLocalAttentionSpec=type("ChunkedLocalAttentionSpec", (), {}), + resolve_block_hashes=lambda values, *args, **kwargs: values, + itertools=itertools, Sequence=Sequence, cdiv=lambda a, b: (a + b - 1) // b)) + pool = SimpleNamespace(hash_block_size=256, get_cached_block=lambda block_hash, groups: [block_hash]) + _, backed_off = scope["FullAttentionManager"].find_longest_cache_hit( + [1, 2, 3, 4, 5], max_length=1280, kv_cache_group_ids=[0], block_pool=pool, + kv_cache_spec=SimpleNamespace(block_size=256), drop_eagle_block=True, alignment_tokens=256) + assert backed_off == 1024 # Raw 1280-token evidence is not a reusable 1280-token hit. + _, request, calls, _ = case(selection, local=backed_off, lease=1024) + assert request.num_computed_tokens == 1024 + assert "attached" in calls diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/test_mtp3_partial_tail_eligibility.py b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/test_mtp3_partial_tail_eligibility.py new file mode 100644 index 00000000..835ddb13 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/test_mtp3_partial_tail_eligibility.py @@ -0,0 +1,42 @@ +"""Execute the actual recurrent partial-tail predicate and scheduling path.""" + +from types import SimpleNamespace + +import pytest + +from repro_mtp3_checkpoint_materialization import partial_hit_eligibility +from repro_mtp3_sparse_retention import MambaSpec +from test_mtp3_sparse_retention import SOURCE, trace + + +@pytest.mark.parametrize("hash_size,page_sizes,alignment,fine,expected", [ + (512, [512], 2048, True, False), + (256, [512], 2048, True, True), + (512, [1024], 2048, True, True), + (512, [], 2048, True, False), + (256, [512], 2048, False, False), +]) +def test_predicate_uses_actual_mamba_pages(hash_size, page_sizes, alignment, fine, expected): + scheduler = SimpleNamespace(need_mamba_block_aligned_split=True, + hash_block_size=hash_size, block_size=alignment, + kv_cache_manager=SimpleNamespace(coordinator=SimpleNamespace(enable_partial_hash_hits=fine))) + config = SimpleNamespace(kv_cache_groups=[SimpleNamespace(kv_cache_spec=MambaSpec(size)) for size in page_sizes]) + assert partial_hit_eligibility(scheduler, config, SOURCE / "v1/core/sched/scheduler.py") is expected + + +@pytest.mark.parametrize("prompt,expected", [ + (5306, [4096, 4608, 5306]), + (32789, [8192, 16384, 24576, 32256, 32768, 32789]), +]) +def test_unneeded_stop_removed_but_publication_and_predecessor_remain(prompt, expected): + _, _, retained, steps = trace(prompt) + assert steps == expected + assert (prompt - 1) // 2048 * 2048 in retained + assert ((prompt - 1) // 512 - 1) * 512 in retained + + +def test_successful_100k_case_drops_only_the_unneeded_fine_tail_stop(): + _, _, retained, steps = trace(100968) + assert 100352 in steps and 100864 not in steps + assert steps[-1] == 100968 + assert 100352 in retained diff --git a/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/test_mtp3_sparse_retention.py b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/test_mtp3_sparse_retention.py new file mode 100644 index 00000000..7a162478 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/experiments/cache-reuse/test_mtp3_sparse_retention.py @@ -0,0 +1,128 @@ +"""Installed-algorithm tests for MTP3 checkpoint production and reuse.""" + +import os +from collections import namedtuple +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from repro_mtp3_checkpoint_materialization import checkpoint_kernel_metadata, load_scheduler, partial_hit_eligibility +from repro_mtp3_sparse_retention import ROOT, FullAttentionSpec, MambaSpec, Pool, load_algorithms + + +SOURCE = Path(os.environ.get("MTP3_RETENTION_SOURCE", str(Path(__file__).parent / "mtp3-retention-patched"))) + + +def trace(prompt, *, budget=8192, eagle=True, scheduler_source=None, manager_source=None): + algorithms = load_algorithms(ROOT / "v1/core/kv_cache_utils.py", + manager_source=manager_source or SOURCE / "v1/core/single_type_kv_cache_manager.py") + Scheduler = load_scheduler(scheduler_source or SOURCE / "v1/core/sched/scheduler.py") + scheduler = Scheduler() + scheduler.cache_config = SimpleNamespace(block_size=512) + scheduler.use_eagle = eagle + scheduler.mamba_has_prefill_checkpoint_blocks = True + scheduler.hash_block_size = 512 + scheduler.block_size = 2048 + scheduler.need_mamba_block_aligned_split = True + scheduler.kv_cache_manager = SimpleNamespace(coordinator=SimpleNamespace(enable_partial_hash_hits=True)) + partial_hit_eligibility(scheduler, + SimpleNamespace(kv_cache_groups=[SimpleNamespace(kv_cache_spec=MambaSpec(512))]), + scheduler_source or SOURCE / "v1/core/sched/scheduler.py") + scheduler.max_num_scheduled_tokens = 8192 + scheduler.scheduler_config = SimpleNamespace(long_prefill_token_threshold=0) + scheduler.num_prefill_lookahead = 1 + publication = (prompt - 1) // 2048 * 2048 + scheduler._recurrent_publication_boundaries = lambda request: (publication,) if publication else () + request = SimpleNamespace(num_prompt_tokens=prompt, num_tokens=prompt, + num_computed_tokens=0, shared_prefix_boundary=0) + spec = MambaSpec(512) + produced = set() + retained = set() + steps = [] + cached = 0 + checkpoint = checkpoint_kernel_metadata() + while request.num_computed_tokens < prompt: + start = request.num_computed_tokens + n = scheduler._mamba_block_aligned_split(request, min(budget, prompt - start)) + n = scheduler._reserve_prefill_lookahead(request, start, n) + assert n > 0 + end = start + n + point = checkpoint(start, end, 512) + if end % 512 == 0: + produced.add(end) + if point is not None: + produced.add(point) + num_full = end // 512 + mask = algorithms.MambaManager.reachable_block_mask(start_block=cached, + end_block=num_full, alignment_tokens=2048, kv_cache_spec=spec, + use_eagle=eagle, retention_interval=0, reachable_boundaries=(prompt - 1,)) + retained.update((cached + index + 1) * 512 for index, keep in enumerate(mask) + if keep and (cached + index + 1) * 512 in produced) + cached = num_full + request.num_computed_tokens = end + steps.append(end) + assert len(steps) < 100 + return algorithms, produced, retained, steps + + +def replay(algorithms, prompt, retained, *, eagle=True): + target, recurrent = FullAttentionSpec(512), MambaSpec(512) + pool = Pool(512, retained, prompt // 512 * 512) + coordinator = algorithms.HybridKVCacheCoordinator() + coordinator.kv_cache_config = SimpleNamespace(kv_cache_groups=(target, recurrent)) + coordinator.single_type_managers = (SimpleNamespace(block_size=2048), SimpleNamespace(block_size=512)) + group = namedtuple("SpecGroup", "spec group_ids manager_cls use_eagle") + coordinator.attention_groups = [group(target, [0], algorithms.FullAttentionManager, eagle), + group(recurrent, [1], algorithms.MambaManager, eagle)] + coordinator.block_pool = pool + coordinator.hash_block_size = 512 + coordinator.scheduler_block_size = 2048 + coordinator.enable_partial_hash_hits = True + coordinator.dcp_world_size = 4 + hashes = list(range(512, prompt + 1, 512)) + return coordinator.find_longest_cache_hit(hashes, prompt - 1)[1] + + +@pytest.mark.parametrize("prompt", [32768, 32769, 32789, 33280, 33281, 100968]) +@pytest.mark.parametrize("budget", [8192, 7680]) +def test_speculative_predecessor_is_materialized_retained_and_reusable(prompt, budget): + algorithms, produced, retained, _ = trace(prompt, budget=budget) + required = ((prompt - 1) // 512 - 1) * 512 + assert required in produced + assert required in retained + assert replay(algorithms, prompt, retained) == required + # Preserve the scheduler-aligned publication checkpoint as well. + assert (prompt - 1) // 2048 * 2048 in retained + + +def test_known_successful_100k_prompt_keeps_its_existing_reuse(): + algorithms, _, retained, _ = trace(100968) + assert replay(algorithms, 100968, retained) == 100352 + + +def test_non_speculative_internal_checkpoint_behavior_is_unchanged(): + _, produced, retained, steps = trace(32789, eagle=False) + assert steps == [8192, 16384, 24576, 32768, 32789] + assert 32256 not in produced + assert retained == {32768} + + +def test_dense_retention_remains_dense(): + algorithms = load_algorithms(ROOT / "v1/core/kv_cache_utils.py", + manager_source=SOURCE / "v1/core/single_type_kv_cache_manager.py") + assert algorithms.MambaManager.reachable_block_mask(start_block=0, end_block=64, + alignment_tokens=2048, kv_cache_spec=MambaSpec(512), use_eagle=True, + retention_interval=None, reachable_boundaries=(32788,)) is None + + +def test_retention_change_alone_cannot_create_the_missing_checkpoint(): + _, produced, retained, _ = trace(32789, scheduler_source=ROOT / "v1/core/sched/scheduler.py") + assert 32256 not in produced + assert 32256 not in retained + + +def test_materialization_change_alone_does_not_preserve_the_checkpoint(): + _, produced, retained, _ = trace(32789, manager_source=ROOT / "v1/core/single_type_kv_cache_manager.py") + assert 32256 in produced + assert 32256 not in retained diff --git a/runtime/glm53-spark-mtp3-mesh/managed_service.py b/runtime/glm53-spark-mtp3-mesh/managed_service.py index 3aab600b..9f1076cb 100644 --- a/runtime/glm53-spark-mtp3-mesh/managed_service.py +++ b/runtime/glm53-spark-mtp3-mesh/managed_service.py @@ -126,7 +126,8 @@ def validate_group(rows): generations = {str(row['rank']): row['generation'] for row in rows} view = digest(generations) if any(row.get('phase') != 'armed' or row.get('view_digest') != view - or row.get('peer_health_degraded', False) for row in rows): + or row.get('peer_health_degraded', False) + or row.get('docker_status_degraded', False) for row in rows): raise RuntimeError('Mesh ranks have not armed the same process generation set') return view @@ -167,10 +168,41 @@ def docker_running(name): result = subprocess.run(['docker', 'inspect', '--format', '{{.State.Running}}', name], capture_output=True, text=True, timeout=3) if result.returncode: - if 'No such' in result.stderr: + if re.search(r'No such (?:object|container): ' + re.escape(name) + r'(?:\s|$)', result.stderr): return False raise RuntimeError('Cannot establish the dependent container state') - return result.stdout.strip() == 'true' + state = result.stdout.strip() + if state not in ('true', 'false'): + raise RuntimeError('Docker returned an unknown dependent container state') + return state == 'true' + + +class DockerStatePoll: + """One bounded background query; unknown state never proves model exit.""" + + def __init__(self, name): + self.name = name + self.executor = concurrent.futures.ThreadPoolExecutor(max_workers=1) + self.pending = None + self.error = None + + def poll(self): + running = None + if self.pending is not None and self.pending.done(): + try: + running = self.pending.result() + self.error = None + except (subprocess.TimeoutExpired, OSError, RuntimeError) as error: + self.error = str(error) + self.pending = None + if self.pending is None: + self.pending = self.executor.submit(docker_running, self.name) + # Do not reuse a previous False while a query is pending: a container + # can start between samples. Cleanup uses a separate synchronous proof. + return running + + def close(self): + self.executor.shutdown(wait=False, cancel_futures=True) def stop_model(name): @@ -405,6 +437,7 @@ def run(self): fcntl.flock(service_lock, fcntl.LOCK_EX | fcntl.LOCK_NB) for sig in (signal.SIGTERM, signal.SIGINT): signal.signal(sig, lambda *_: self.stop.set()) + docker_watch = None try: if docker_running(self.model): raise RuntimeError('Stop the dependent model before starting mesh ownership') @@ -420,6 +453,7 @@ def run(self): peer_watch = PeerWatch() last_network = time.monotonic() last_full_network = last_network + docker_watch = DockerStatePoll(self.model) while not self.stop.is_set(): if any(child.poll() is not None for child in self.children): raise RuntimeError('A managed source marker exited') @@ -429,6 +463,9 @@ def run(self): if full: last_full_network = time.monotonic() last_network = time.monotonic() + model_running = docker_watch.poll() + self.publish(docker_status_degraded=model_running is None, + docker_status_error=docker_watch.error) try: rows = group_check(self.site, self.config, self.key, self.identity) view = peer_watch.observe(rows) @@ -436,21 +473,22 @@ def run(self): except OSError: if peer_watch.generations is not None: peer_watch.transport_error(time.monotonic()) - elif docker_running(self.model): + elif model_running is True: raise self.publish(peer_health_degraded=True) except Exception: - if peer_watch.generations is not None or docker_running(self.model): + if peer_watch.generations is not None or model_running is True: raise - if self.state['phase'] != 'armed' and docker_running(self.model): + if self.state['phase'] != 'armed' and model_running is True: raise RuntimeError('Dependent model started before four-rank readiness') intent_path = self.state_dir / 'model-intent.json' if intent_path.exists(): intent = json.loads(intent_path.read_text()) if intent.get('generation') == self.generation and intent.get('active') is True: - if docker_running(self.model): + if model_running is True: self.model_seen = True - elif self.model_seen or time.monotonic() > intent['deadline_monotonic']: + elif model_running is False and ( + self.model_seen or time.monotonic() > intent['deadline_monotonic']): raise RuntimeError('Dependent model exited or failed to start') else: self.model_seen = False @@ -462,6 +500,8 @@ def run(self): self.publish(best_effort=True, local_ready=False, phase='failed', error=str(error)) print(json.dumps({'event': 'mesh_failure', 'rank': self.rank, 'error': str(error)}), flush=True) finally: + if docker_watch is not None: + docker_watch.close() self.publish(best_effort=True, local_ready=False, phase='failed' if self.failed else 'stopping') while self.owns_guard: try: @@ -534,7 +574,8 @@ def model_intent(config_path, active): raise RuntimeError('Mesh service has no readiness state') return status = json.loads(status_path.read_text()) - if active and (status.get('phase') != 'armed' or status.get('local_ready') is not True): + if active and (status.get('phase') != 'armed' or status.get('local_ready') is not True + or status.get('docker_status_degraded', False)): raise RuntimeError('Mesh is not armed for model startup') record = {'generation': status['generation'], 'active': active, 'deadline_monotonic': time.monotonic() + 15} diff --git a/runtime/glm53-spark-mtp3-mesh/test_managed_service.py b/runtime/glm53-spark-mtp3-mesh/test_managed_service.py index f2b6db54..fcdbbac4 100644 --- a/runtime/glm53-spark-mtp3-mesh/test_managed_service.py +++ b/runtime/glm53-spark-mtp3-mesh/test_managed_service.py @@ -3,6 +3,8 @@ from pathlib import Path import threading import time +from concurrent.futures import Future +from types import SimpleNamespace import pytest @@ -167,6 +169,197 @@ class Result: assert calls == [['docker', 'kill', 'a' * 64]] +def test_docker_inspect_timeout_is_unknown_without_stalling_fabric_monitor(monkeypatch): + pending = Future() + submitted = [] + + class Executor: + def submit(self, function, name): + submitted.append((function, name)) + return pending + + def shutdown(self, **kwargs): + pass + + monkeypatch.setattr(service.concurrent.futures, 'ThreadPoolExecutor', lambda **kw: Executor()) + watcher = service.DockerStatePoll('a' * 64) + assert watcher.poll() is None + for _ in range(20): + assert watcher.poll() is None + assert len(submitted) == 1 + command = ['docker', 'inspect', '--format', '{{.State.Running}}', 'a' * 64] + pending.set_exception(service.subprocess.TimeoutExpired(command, 3)) + assert watcher.poll() is None + assert '3 seconds' in watcher.error + assert len(submitted) == 2 + watcher.close() + + +def test_docker_status_recovers_without_reusing_stale_stopped_evidence(monkeypatch): + futures = [Future(), Future(), Future()] + pending = iter(futures) + executor = SimpleNamespace(submit=lambda *args: next(pending), shutdown=lambda **kw: None) + monkeypatch.setattr(service.concurrent.futures, 'ThreadPoolExecutor', lambda **kw: executor) + watcher = service.DockerStatePoll('a' * 64) + assert watcher.poll() is None + futures[0].set_result(False) + assert watcher.poll() is False + assert watcher.poll() is None + futures[1].set_result(True) + assert watcher.poll() is True + assert watcher.error is None + watcher.close() + + +def test_unknown_docker_status_blocks_model_admission(): + changed = rows() + changed[0]['docker_status_degraded'] = True + with pytest.raises(RuntimeError): + service.validate_group(changed) + + +def test_model_arm_rechecks_local_docker_status_after_group_gate(tmp_path, monkeypatch): + monkeypatch.setattr(service, 'load_config', lambda path: ({'state_dir': str(tmp_path)},)) + (tmp_path / 'status.json').write_bytes(service.canonical({ + 'phase': 'armed', 'local_ready': True, 'generation': 'g', 'docker_status_degraded': True, + })) + with pytest.raises(RuntimeError, match='not armed'): + service.model_intent('unused', True) + assert not (tmp_path / 'model-intent.json').exists() + + +@pytest.mark.parametrize('message, missing', [ + ('Error: No such object: ' + 'a' * 64, True), + ('Error: No such object: ' + 'b' * 64, False), + ('No such file or directory: Docker socket', False), +]) +def test_only_pinned_missing_container_proves_absence(monkeypatch, message, missing): + monkeypatch.setattr(service.subprocess, 'run', lambda *args, **kw: + SimpleNamespace(returncode=1, stdout='', stderr=message)) + if missing: + assert service.docker_running('a' * 64) is False + else: + with pytest.raises(RuntimeError): + service.docker_running('a' * 64) + + +@pytest.mark.parametrize('output', ['', 'unexpected', 'False']) +def test_unknown_docker_reply_cannot_prove_model_stopped(monkeypatch, output): + monkeypatch.setattr(service.subprocess, 'run', lambda *args, **kw: + SimpleNamespace(returncode=0, stdout=output, stderr='')) + with pytest.raises(RuntimeError): + service.stop_model('a' * 64) + + +def test_docker_timeout_cannot_pass_model_stop_barrier(monkeypatch): + calls = [] + + def timeout(argv, **kwargs): + calls.append((argv, kwargs['timeout'])) + raise service.subprocess.TimeoutExpired(argv, kwargs['timeout']) + + monkeypatch.setattr(service.subprocess, 'run', timeout) + with pytest.raises(service.subprocess.TimeoutExpired): + service.stop_model('a' * 64) + assert len(calls) == 1 + assert calls[0][1] == 3 + + +@pytest.mark.parametrize('peer_failure', [False, True]) +def test_monitor_keeps_fabric_checks_live_when_docker_is_unknown(tmp_path, monkeypatch, peer_failure): + result = owner() + result.rank, result.generation, result.model = 0, 'g', 'a' * 64 + result.config = {'site_path': '/unused'} + result.site, result.identity, result.key = {}, 'identity', b'k' * 32 + result.state_dir, result.network, result.server = tmp_path, None, None + result.marker_records, result.logfiles = [], [] + result.failed, result.owns_guard, result.model_seen = False, False, True + events = [] + samples = iter([True, None, True]) + clock = [100.0] + monitor_round = [0] + + class Stop: + def is_set(self): + return monitor_round[0] >= 3 + + def wait(self, seconds): + monitor_round[0] += 1 + clock[0] += 6 + + result.stop = Stop() + result.start_markers = lambda: None + result.start_server = lambda: None + result.publish = lambda **changes: result.state.update(changes) + result.children = [SimpleNamespace( + poll=lambda: None, + terminate=lambda: events.append('marker-stop'), + wait=lambda **kw: None, + ) for _ in range(2)] + (tmp_path / 'model-intent.json').write_bytes(service.canonical({ + 'generation': 'g', 'active': True, 'deadline_monotonic': 0, + })) + monkeypatch.setattr(service.os, 'geteuid', lambda: 0, raising=False) + original_lstat = type(tmp_path).lstat + + def fixture_lstat(path): + info = original_lstat(path) + if path == tmp_path: + return SimpleNamespace(st_mode=info.st_mode, st_uid=0) + return info + + # Model the root-owned service directory without changing host ownership. + monkeypatch.setattr(type(tmp_path), 'lstat', fixture_lstat) + monkeypatch.setattr(service.signal, 'signal', lambda *args: None) + monkeypatch.setitem(service.sys.modules, 'fcntl', SimpleNamespace( + flock=lambda *args: None, LOCK_EX=1, LOCK_NB=2, + )) + monkeypatch.setitem(service.sys.modules, 'managed_network', SimpleNamespace( + NetworkManager=lambda *args: SimpleNamespace( + up=lambda: None, + check=lambda **kw: events.append('network-check'), + down=lambda: events.append('network-down') or {'clean': True}, + ) + )) + monkeypatch.setattr(service.time, 'monotonic', lambda: clock[0]) + monkeypatch.setattr(service.time, 'sleep', lambda seconds: events.append('retain-markers')) + monkeypatch.setattr(service, 'notify', lambda message: events.append(message)) + monkeypatch.setattr(service, 'docker_running', lambda name: events.append('startup-inspect') or False) + monkeypatch.setattr(service, 'DockerStatePoll', lambda name: SimpleNamespace( + poll=lambda: next(samples), error='Docker inspect timed out after 3 seconds', close=lambda: None, + )) + + def group_check(*args): + events.append('peer-check') + if peer_failure and monitor_round[0] == 1: + raise RuntimeError('Authenticated peer is not locally ready') + return rows() + + stop_attempts = [0] + + def stop_model(name): + stop_attempts[0] += 1 + if stop_attempts[0] == 1: + assert 'marker-stop' not in events and 'network-down' not in events + raise service.subprocess.TimeoutExpired(['docker', 'inspect', name], 3) + events.append('model-stop-confirmed') + + monkeypatch.setattr(service, 'group_check', group_check) + monkeypatch.setattr(service, 'stop_model', stop_model) + assert result.run() == 1 # The explicit stop barrier needed a retry. + assert events.count('startup-inspect') == 1 + assert events.count('peer-check') == (2 if peer_failure else 3) + assert 'network-check' in events + assert events.index('retain-markers') < events.index('model-stop-confirmed') + assert events.index('model-stop-confirmed') < events.index('marker-stop') + assert events.index('marker-stop') < events.index('network-down') + if peer_failure: + assert 'not locally ready' in result.state['error'] + else: + assert 'error' not in result.state + assert result.state['docker_status_degraded'] is False + + def test_units_bind_model_and_disable_automatic_recovery(): rendered = units.unit_text('/opt/sparkring/managed-mesh', '/etc/sparkring/managed-mesh', 'a' * 64) mesh, model = rendered['sparkring-mesh.service'], rendered['sparkring-mesh-model.service']