From edd0f825edfee9969b07167fac98d04337bb328e Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 05:38:58 -0500 Subject: [PATCH 01/67] Bench(docs[orchestration]): Specify active scale why: Define truth and safety contracts before a benchmark that can create 40,000 live panes. what: - Separate destructive setup from repeatable active-topology phases - Specify deterministic Rich workloads and resource-aware ramping - Define correctness, raw-sample, resource, and cleanup evidence --- docs/experimental/index.md | 4 + docs/experimental/orchestration-benchmark.md | 265 +++++++++++++++++++ 2 files changed, 269 insertions(+) create mode 100644 docs/experimental/orchestration-benchmark.md diff --git a/docs/experimental/index.md b/docs/experimental/index.md index 0fcf402f74..baa604a296 100644 --- a/docs/experimental/index.md +++ b/docs/experimental/index.md @@ -23,6 +23,9 @@ result. - [Engines](engines.md) explains how to choose a transport without changing the operation contract. - [Plans](plans.md) covers deferred execution, planners, and the fluent builder. +- [Active orchestration benchmark](orchestration-benchmark.md) specifies the + persistent, high-cardinality workload used to measure construction, mutation, + waiting, enumeration, capture, and search under pane activity. ## Run one operation @@ -50,4 +53,5 @@ operations/index results engines plans +orchestration-benchmark ``` diff --git a/docs/experimental/orchestration-benchmark.md b/docs/experimental/orchestration-benchmark.md new file mode 100644 index 0000000000..cb93757021 --- /dev/null +++ b/docs/experimental/orchestration-benchmark.md @@ -0,0 +1,265 @@ +# Huge Active Orchestration Benchmark Design + +## Purpose + +Build a hermetic benchmark for long-lived tmux servers containing 80 to 100 +sessions, 20 to 100 windows per session, and one to four panes per window. Every +pane remains active while the harness measures construction, mutation, waits, +enumeration, capture, and search. + +The harness supports the full `100x100x4` topology: 100 sessions, 10,000 +windows, and 40,000 panes. A successful smaller ramp step is not evidence that +the maximum completed. Reports distinguish an implemented ceiling, an attempted +shape, a completed shape, and a host-resource cutoff. + +## Deliverables + +- `scripts/orchestration_fuzzer.py`: a self-contained PEP 723 Rich workload + generator with `preview` and `serve` commands. +- `scripts/bench_orchestration.py`: a self-contained PEP 723 benchmark with + `plan`, `run`, and `ramp` commands. +- Focused unit and live-tmux tests for workload determinism, delayed matching, + topology truth, phase correctness, cleanup, and machine-readable results. +- JSON results containing raw samples and environment/resource metadata. +- A Markdown summary generated from the JSON evidence after the large runs. + +The new scripts remain separate from `scripts/bench_engines.py`. That script +measures repeated construction of short-lived topologies; this benchmark builds +one persistent active topology and performs several distinct workloads on it. + +## Topology and scale + +`SxWxP` denotes sessions, windows per session, and panes per window. Every +dimension is positive. The supported ranges are: + +- sessions: 80 through 100 for large runs; +- windows per session: 20 through 100; +- panes per window: one through four; +- small shapes remain accepted for tests and smoke runs. + +The canonical ramp is ordered by expected live-pane pressure: + +| Shape | Sessions | Windows | Panes | +| --- | ---: | ---: | ---: | +| `80x20x1` | 80 | 1,600 | 1,600 | +| `100x20x1` | 100 | 2,000 | 2,000 | +| `80x20x2` | 80 | 1,600 | 3,200 | +| `80x50x1` | 80 | 4,000 | 4,000 | +| `80x20x4` | 80 | 1,600 | 6,400 | +| `80x100x1` | 80 | 8,000 | 8,000 | +| `100x50x2` | 100 | 5,000 | 10,000 | +| `100x100x2` | 100 | 10,000 | 20,000 | +| `100x100x4` | 100 | 10,000 | 40,000 | + +`ramp` attempts shapes in order and cleans each disposable server before the +next. It records a structured cutoff when a resource guard trips or a topology +phase fails. An explicit `run --shape 100x100x4` remains available; the harness +does not silently replace it with a smaller scenario. + +## Activity model + +Launching Rich or `uv` in every pane would make interpreter memory the dominant +limit. Instead, one central Python process renders deterministic Rich frames +into append-only stream files. Pane processes use a lightweight `tail` command +to follow one stream. This keeps every pane's PTY active while retaining a +single controlled workload clock. + +The fuzzer exposes four modes distributed round-robin by stable pane ordinal: + +- `editor`: source paths, line numbers, cursor/status state, and scrolling + libtmux source excerpts; +- `dev-server`: request, rebuild, warning, error, and recovery records; +- `installer`: resolve, download, build, and install progress; +- `delayed-match`: ordinary scrolling output followed by a unique sentinel at a + configured monotonic delay, then more output. + +The generator accepts a seed, frame rate, duration, output directory, delayed +match interval, and sentinel. `preview` renders the same frames interactively. +`serve` starts paused, writes a ready marker, and begins only after the benchmark +releases its gate. The generator records scheduled and actual sentinel times so +the wait benchmark reports detection overhead separately from intentional +delay. + +One selected pane follows a dedicated delayed-match stream. That makes the +sentinel unique across the server for pane-content search. Every other pane +follows one of the shared mode streams. Stream files stay bounded for the +benchmark duration; rotation is outside this benchmark's scope. + +## Isolation and lifecycle + +Every run uses a unique short socket path, an empty tmux configuration, and a +private scratch directory. The benchmark unsets `TMUX` and `TMUX_PANE` before +importing libtmux. It never contacts the ambient tmux server. + +The lifecycle is: + +1. validate the scenario and record the host resource envelope; +2. start the paused fuzzer and wait for its ready marker; +3. create the tmux server and complete the selected topology; +4. verify exact session, window, and pane counts plus pane-process liveness; +5. release the fuzzer and prove every pane displays the current activity epoch; +6. run the measured phases while activity continues; +7. stop the fuzzer, kill the isolated server, and remove scratch state; +8. verify the process, socket, and scratch directory are gone. + +SIGINT, SIGTERM, phase errors, and resource cutoffs use the same cleanup path. +The JSON report is written atomically after cleanup and includes a failed phase +and error summary when the run does not complete. + +## Engines and execution lanes + +The primary lanes are sync and async subprocess engines and sync and async +control-mode engines. All lanes use the same typed operations and validate the +same live postconditions. The default large run uses async control mode because +it can keep one persistent client and attribute pipelined requests individually. + +The classic ORM remains an optional reference for small or explicitly selected +large shapes. It is not mixed into engine speedup claims when its request graph +differs. Setup reports planner steps, engine batches, tmux requests, and known +process starts separately. + +Activity files feed pane processes directly, so the fuzzer does not use the +engine under measurement. The tmux server still bears the intended PTY output +load. + +## Measured phases + +### Setup + +Setup starts with a fresh server and ends after the exact topology exists. The +fuzzer remains paused, so setup measures construction rather than output load. +The report records one raw duration per fresh-server attempt, command counts, +and the resulting resource snapshot. Setup samples are never pooled with the +repeatable phase samples. + +### Activity stabilization + +This untimed correctness gate releases the fuzzer and waits until every pane's +capture contains the current epoch marker. It also checks `pane_dead`, pane PID, +fuzzer heartbeat, and sentinel uniqueness. Failure rejects the run before any +performance phase is reported as valid. + +### Bulk mutation + +Each iteration mutates the largest selected session while activity continues: + +1. set a generation user option on the session; +2. rename every window by stable window ID; +3. set every pane title by stable pane ID; +4. verify the generation and names; +5. restore canonical window names and clear the generation marker. + +The mutation count and target cardinalities are recorded. A failed or partial +request invalidates the sample. + +### Delayed text wait + +The waiter starts before the fuzzer's delayed stream is released. Two strategies +run as separately named cells: + +- `capture-poll`: bounded `capture-pane` polling with a fixed cadence; +- `control-stream`: decoded control-mode output matched as it arrives. + +Each result reports configured delay, actual emission time, detection time, +detection overhead, poll/frame count, timeout, dropped notifications, and exact +sentinel match. Deliberate delay is not counted as library overhead. + +### Enumeration + +Separate cells measure typed `ListSessions`, `ListWindows`, and `ListPanes` +operations. Optional ORM cells measure `server.sessions`, `server.windows`, and +`server.panes`. Every sample asserts the exact row count and a stable checksum +of object IDs before it is accepted. + +The repeated “list panes” requirement is interpreted as listing all three tmux +hierarchy levels: sessions, windows, and panes. + +### Capture contents + +The harness captures a bounded visible/history range from every pane. It +measures serial and engine-batched request strategies separately, records total +bytes and lines, and verifies that every pane contains a current activity epoch. +Batching does not imply fewer subprocesses on the subprocess transport. + +### Search + +Search uses known targets placed at the beginning, middle, and end of stable +topology order. Separate cells measure: + +- tmux server-side format filtering for a session, window, and pane; +- Python `QueryList` filtering over an already materialized snapshot; +- end-to-end list plus Python filtering; +- content search across captured pane text for the unique sentinel. + +Each cell requires exactly the expected object and reports the scanned row or +pane count. In-memory filtering time is not presented as equivalent to an +end-to-end server query. + +## Sampling and reporting + +Repeatable phases use configurable warmups and timed iterations. Strategy order +is deterministically interleaved rather than running every sample of one +strategy first. Reports retain all raw nanosecond samples and render count, +minimum, mean, median, p90, p95, p99, and maximum. Large setup runs report their +individual durations rather than manufacturing percentiles from one attempt. + +Each phase records: + +- requested and observed topology; +- engine, mode, and strategy; +- operations, batches, tmux requests, and returned rows; +- bytes, lines, frames, polls, and dropped notifications where applicable; +- tmux-server RSS, process count, file descriptors, and host available memory; +- tmux and Python versions, CPU count, seed, command line, and git revision; +- correctness and cleanup status. + +The Markdown report distinguishes local descriptive evidence from causal or +machine-independent performance claims. + +## Resource guards + +`plan` performs no tmux writes. It prints exact topology totals, predicted pane +processes, and a conservative request count. `run` and `ramp` sample available +memory, process limits, file-descriptor limits, and tmux-server liveness before +and after each major phase. + +A ramp stops after cleaning the current server when: + +- available memory falls below a configurable floor; +- pane or fuzzer processes die; +- setup makes no progress for the configured watchdog interval; +- exact topology or activity verification fails; +- the user interrupts the run. + +The cutoff is evidence, not success. `--force-extreme` relaxes predictive +preflight refusal but does not disable cleanup, watchdogs, or correctness checks. + +## Tests and verification + +Pure tests cover scenario parsing, ramp order, deterministic workload frames, +sentinel timing metadata, sample statistics, resource-cutoff records, and JSON +serialization. + +Live tests use small disposable topologies to prove: + +- every fuzzer mode scrolls visible pane content; +- the delayed sentinel appears only after its configured gate and delay; +- both wait strategies find the same sentinel without false matches; +- setup, mutation, enumeration, capture, and search return exact counts; +- activity continues during mutation and query phases; +- failure and cancellation remove the server, fuzzer, tail processes, socket, + and scratch directory; +- a hostile user tmux configuration cannot affect the run. + +The final branch gate is formatting, Ruff, mypy, the complete pytest suite, and +the documentation build. Large benchmark artifacts are accepted only after a +validator confirms expected phase names, raw-sample counts, topology checks, +cleanup, and the absence of failed rows. + +## Out of scope + +- Claiming that one local machine establishes universal engine speedups. +- Treating an unattempted `100x100x4` topology as completed. +- Running one Python or `uv` process per pane. +- Changing libtmux's public wait or search API solely for benchmark convenience. +- Benchmarking remote tmux servers or multiple hosts in this iteration. From caeaede743f47aa54dc881149e0fe50b77d7cc32 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 05:45:30 -0500 Subject: [PATCH 02/67] Bench(docs[orchestration]): Repeat wait gates why: Repeated wait samples need distinct emission evidence without restarting the active topology. what: - Define run-scoped sentinel request and evidence markers - Add supervisor recovery and explicit resource floor contracts --- docs/experimental/orchestration-benchmark.md | 26 ++++++++++++++------ 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/docs/experimental/orchestration-benchmark.md b/docs/experimental/orchestration-benchmark.md index cb93757021..c94681f9ba 100644 --- a/docs/experimental/orchestration-benchmark.md +++ b/docs/experimental/orchestration-benchmark.md @@ -74,11 +74,15 @@ The fuzzer exposes four modes distributed round-robin by stable pane ordinal: configured monotonic delay, then more output. The generator accepts a seed, frame rate, duration, output directory, delayed -match interval, and sentinel. `preview` renders the same frames interactively. -`serve` starts paused, writes a ready marker, and begins only after the benchmark -releases its gate. The generator records scheduled and actual sentinel times so -the wait benchmark reports detection overhead separately from intentional -delay. +match interval, and sentinel prefix. `preview` renders the same frames +interactively. `serve` starts paused, writes a ready marker, and begins only +after the benchmark releases its gate. Each wait sample writes an atomic, +run-scoped request with a unique request ID and sentinel. The generator appends +that sentinel after the requested monotonic delay and atomically publishes +request-specific scheduling and emission evidence. This permits independent +warmup and timed wait samples without restarting the active topology, and lets +the report separate configured delay, generator scheduling lateness, and waiter +detection overhead. One selected pane follows a dedicated delayed-match stream. That makes the sentinel unique across the server for pane-content search. Every other pane @@ -103,8 +107,12 @@ The lifecycle is: 8. verify the process, socket, and scratch directory are gone. SIGINT, SIGTERM, phase errors, and resource cutoffs use the same cleanup path. -The JSON report is written atomically after cleanup and includes a failed phase -and error summary when the run does not complete. +A small supervisor process owns the terminal report and monitors an isolated +worker process with progress events and identity-checked process records. This +lets the benchmark recover when a worker is cancelled or an engine call stops +making progress. The JSON report is written atomically at startup, after each +checkpoint, and after cleanup; it includes the failed phase and error summary +when the run does not complete. ## Engines and execution lanes @@ -233,6 +241,10 @@ A ramp stops after cleaning the current server when: The cutoff is evidence, not success. `--force-extreme` relaxes predictive preflight refusal but does not disable cleanup, watchdogs, or correctness checks. +The default PID reserve is the larger of 1,024 processes and 15 percent of the +detected cgroup limit. The default memory floor is the larger of 4 GiB and 15 +percent of detected physical memory. Missing resource probes remain explicit +unknown values rather than being treated as zero. ## Tests and verification From ccff71aa7352330109facbf477b4d44bba309a32 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 06:01:52 -0500 Subject: [PATCH 03/67] Bench(feat[fuzzer]): Generate active streams why: Give the orchestration benchmark deterministic active pane output without starting a Python process in every pane. what: - Add a paused, run-scoped stream service with atomic control markers - Add deterministic frame rendering and request-specific sentinel evidence - Cover gates, invalid markers, repeated waits, and graceful shutdown --- scripts/orchestration_fuzzer.py | 544 +++++++++++++++++++++++++++++ tests/test_orchestration_fuzzer.py | 265 ++++++++++++++ 2 files changed, 809 insertions(+) create mode 100644 scripts/orchestration_fuzzer.py create mode 100644 tests/test_orchestration_fuzzer.py diff --git a/scripts/orchestration_fuzzer.py b/scripts/orchestration_fuzzer.py new file mode 100644 index 0000000000..d5ea63fb90 --- /dev/null +++ b/scripts/orchestration_fuzzer.py @@ -0,0 +1,544 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.10" +# dependencies = ["rich>=13"] +# /// +"""Generate deterministic append-only streams for orchestration benchmarks.""" + +from __future__ import annotations + +import argparse +import contextlib +import dataclasses +import enum +import json +import os +import pathlib +import random +import signal +import tempfile +import time +import typing as t + + +class StreamMode(str, enum.Enum): + """A deterministic active-output stream category.""" + + EDITOR = "editor" + DEV_SERVER = "dev-server" + INSTALLER = "installer" + DELAYED_MATCH = "delayed-match" + + +@dataclasses.dataclass(frozen=True) +class WorkloadOptions: + """Configuration for one fuzzer process. + + Attributes + ---------- + output_dir : pathlib.Path + Directory used for markers and append-only streams. + run_id : str + Run identity carried by every control marker. + source_root : pathlib.Path + Repository root containing ``src/libtmux`` source files. + seed : int + Seed that fixes corpus and frame selection. + frame_rate_hz : float + Number of frames emitted per stream each second. + duration_s : float + Maximum serving duration after activation. + delayed_match_after_s : float + Delay between an accepted request and its sentinel emission. + sentinel_prefix : str + Human-readable prefix for each unique sentinel. + heartbeat_interval_s : float + Maximum interval between heartbeat marker updates. + """ + + output_dir: pathlib.Path + run_id: str + source_root: pathlib.Path + seed: int + frame_rate_hz: float + duration_s: float + delayed_match_after_s: float + sentinel_prefix: str + heartbeat_interval_s: float + + +@dataclasses.dataclass(frozen=True) +class WorkloadPaths: + """Filesystem locations owned by one fuzzer process. + + Attributes + ---------- + root : pathlib.Path + Exclusive output directory. + streams : pathlib.Path + Directory containing mode-specific append-only streams. + requests : pathlib.Path + Directory containing benchmark request markers. + sentinels : pathlib.Path + Directory containing request-specific emission evidence. + ready : pathlib.Path + Marker published after all service paths exist. + gate : pathlib.Path + Marker that releases the paused service. + heartbeat : pathlib.Path + Current fuzzer liveness marker. + stop : pathlib.Path + Marker requesting a graceful stop. + """ + + root: pathlib.Path + streams: pathlib.Path + requests: pathlib.Path + sentinels: pathlib.Path + ready: pathlib.Path + gate: pathlib.Path + heartbeat: pathlib.Path + stop: pathlib.Path + + +@dataclasses.dataclass(frozen=True) +class Frame: + """One rendered stream record. + + Attributes + ---------- + mode : StreamMode + Stream receiving the record. + epoch : int + Monotonic frame sequence number. + text : str + Newline-terminated record appended to the stream. + """ + + mode: StreamMode + epoch: int + text: str + + +@dataclasses.dataclass(frozen=True) +class SentinelEvidence: + """Request-specific sentinel scheduling and emission facts. + + Attributes + ---------- + schema_version : int + Marker schema version. + run_id : str + Run identity that owns the evidence. + request_id : str + Unique request identity. + scheduled_monotonic_ns : int + Requested emission deadline on the monotonic clock. + emitted_monotonic_ns : int + Actual append time on the monotonic clock. + sentinel : str + Canonical text appended to the delayed stream. + """ + + schema_version: int + run_id: str + request_id: str + scheduled_monotonic_ns: int + emitted_monotonic_ns: int + sentinel: str + + +def stream_for_pane(ordinal: int, delayed_ordinal: int) -> StreamMode: + """Return the stable stream assignment for a pane ordinal. + + >>> stream_for_pane(2, 2) + + >>> stream_for_pane(3, 2) + + """ + if ordinal == delayed_ordinal: + return StreamMode.DELAYED_MATCH + shared = (StreamMode.EDITOR, StreamMode.DEV_SERVER, StreamMode.INSTALLER) + compacted = ordinal - int(ordinal > delayed_ordinal) + return shared[compacted % len(shared)] + + +def sentinel_text(run_id: str, request_id: str, value: str) -> str: + """Return a sentinel that identifies its owning run and request. + + >>> sentinel_text("run-7", "sample-03", "READY") + 'LIBTMUX_SENTINEL run=run-7 request=sample-03 value=READY' + """ + return f"LIBTMUX_SENTINEL run={run_id} request={request_id} value={value}" + + +def source_lines(source_root: pathlib.Path, seed: int) -> tuple[str, ...]: + """Read and stably shuffle labeled ``src/libtmux`` source lines. + + >>> source_lines(pathlib.Path("missing"), seed=1) + () + """ + source_dir = source_root / "src" / "libtmux" + lines: list[str] = [] + for path in sorted(source_dir.rglob("*.py")): + relative = path.relative_to(source_root) + decoded = path.read_bytes().decode("utf-8", errors="replace") + lines.extend( + f"{relative}:{number}: {line}" + for number, line in enumerate(decoded.splitlines(), start=1) + ) + random.Random(seed).shuffle(lines) + return tuple(lines) + + +def render_frame( + mode: StreamMode, + epoch: int, + corpus: tuple[str, ...], + seed: int, +) -> Frame: + r"""Render one deterministic newline-terminated activity record. + + >>> render_frame(StreamMode.DEV_SERVER, 4, (), 11).text + '[dev-server epoch=4] request=GET /sessions status=200 elapsed_ms=5\\n' + """ + if mode is StreamMode.EDITOR: + source = corpus[(seed + epoch) % len(corpus)] if corpus else "" + text = f"[editor epoch={epoch}] {source}\n" + elif mode is StreamMode.DEV_SERVER: + records = ( + "request=GET /sessions status=200 elapsed_ms=5", + "rebuild target=workspace state=complete modules=12", + "warning code=W001 source=watcher action=retry", + "recovery service=api state=ready", + ) + text = f"[dev-server epoch={epoch}] {records[(seed + epoch) % len(records)]}\n" + elif mode is StreamMode.INSTALLER: + phases = ("resolve", "download", "build", "install") + phase = phases[(seed + epoch) % len(phases)] + text = f"[installer epoch={epoch}] install phase={phase} unit={epoch + 1}/8\n" + else: + text = f"[delayed-match epoch={epoch}] scan state=waiting cursor={epoch}\n" + return Frame(mode=mode, epoch=epoch, text=text) + + +def prepare_output(options: WorkloadOptions) -> WorkloadPaths: + """Create the exclusive marker tree and empty append-only stream files. + + Raises + ------ + FileExistsError + If another process already owns the requested output directory. + """ + root = options.output_dir + root.mkdir(mode=0o700, parents=True, exist_ok=False) + streams = root / "streams" + requests = root / "requests" + sentinels = root / "sentinels" + for directory in (streams, requests, sentinels): + directory.mkdir() + for mode in StreamMode: + (streams / f"{mode.value}.log").touch(exist_ok=False) + return WorkloadPaths( + root=root, + streams=streams, + requests=requests, + sentinels=sentinels, + ready=root / "ready.json", + gate=root / "gate.json", + heartbeat=root / "heartbeat.json", + stop=root / "stop.json", + ) + + +def write_json_atomic(path: pathlib.Path, data: t.Mapping[str, t.Any]) -> None: + """Replace a marker only after its JSON bytes are durable on disk. + + The sibling temporary file and replacement make readers see either the + previous complete marker or the next complete marker, never a partial one. + """ + with tempfile.NamedTemporaryFile( + mode="w", + encoding="utf-8", + dir=path.parent, + prefix=f".{path.name}.", + suffix=".tmp", + delete=False, + ) as temporary: + temporary_path = pathlib.Path(temporary.name) + json.dump(data, temporary, sort_keys=True, separators=(",", ":")) + temporary.write("\n") + temporary.flush() + os.fsync(temporary.fileno()) + try: + os.replace(temporary_path, path) # noqa: PTH105 + directory_flags = os.O_RDONLY + if hasattr(os, "O_DIRECTORY"): + directory_flags |= os.O_DIRECTORY + directory_fd = os.open(path.parent, directory_flags) + try: + os.fsync(directory_fd) + finally: + os.close(directory_fd) + except BaseException: + with contextlib.suppress(FileNotFoundError): + temporary_path.unlink() + raise + + +def read_control_marker(path: pathlib.Path, run_id: str) -> dict[str, t.Any] | None: + """Return a matching schema-v1 marker, ignoring absent or malformed files. + + >>> read_control_marker(pathlib.Path("missing.json"), "run-7") is None + True + """ + try: + parsed = json.loads(path.read_text(encoding="utf-8")) + except (FileNotFoundError, json.JSONDecodeError, OSError): + return None + if not isinstance(parsed, dict): + return None + marker = t.cast(dict[str, t.Any], parsed) + if marker.get("schema_version") != 1 or marker.get("run_id") != run_id: + return None + return marker + + +def _stream_path(paths: WorkloadPaths, mode: StreamMode) -> pathlib.Path: + """Return the append-only stream path for one mode.""" + return paths.streams / f"{mode.value}.log" + + +def _append_text(path: pathlib.Path, text: str) -> int: + """Append one complete text record and return its UTF-8 byte count.""" + encoded = text.encode("utf-8") + with path.open("ab") as stream: + stream.write(encoded) + stream.flush() + return len(encoded) + + +def _request_from_marker( + marker: dict[str, t.Any], + path: pathlib.Path, + options: WorkloadOptions, +) -> tuple[str, int, str] | None: + """Validate one request marker and return its delayed sentinel inputs.""" + request_id = marker.get("request_id") + requested = marker.get("requested_monotonic_ns") + value = marker.get("value", options.sentinel_prefix) + if ( + not isinstance(request_id, str) + or not request_id + or path.stem != request_id + or isinstance(requested, bool) + or not isinstance(requested, int) + or isinstance(value, bool) + or not isinstance(value, str) + or not value + ): + return None + return request_id, requested, value + + +def run_serve(options: WorkloadOptions) -> int: + """Serve paused deterministic streams until a matching stop or lifecycle exit.""" + if options.frame_rate_hz <= 0: + message = "frame_rate_hz must be positive" + raise ValueError(message) + if options.duration_s <= 0: + message = "duration_s must be positive" + raise ValueError(message) + if options.delayed_match_after_s < 0: + message = "delayed_match_after_s must be non-negative" + raise ValueError(message) + if options.heartbeat_interval_s <= 0: + message = "heartbeat_interval_s must be positive" + raise ValueError(message) + + paths = prepare_output(options) + corpus = source_lines(options.source_root, options.seed) + write_json_atomic(paths.ready, {"schema_version": 1, "run_id": options.run_id}) + + stopping = False + + def request_stop(_signal_number: int, _frame: t.Any) -> None: + """Record a process signal for the serving loop.""" + nonlocal stopping + stopping = True + + previous_sigint = signal.signal(signal.SIGINT, request_stop) + previous_sigterm = signal.signal(signal.SIGTERM, request_stop) + frame_interval_ns = max(1, int(1_000_000_000 / options.frame_rate_hz)) + delay_ns = int(options.delayed_match_after_s * 1_000_000_000) + heartbeat_interval_ns = int(options.heartbeat_interval_s * 1_000_000_000) + bytes_since_heartbeat = 0 + last_heartbeat_ns = 0 + activated_at_ns: int | None = None + next_frame_ns: int | None = None + epoch = 0 + seen_requests: set[str] = set() + pending_request: tuple[str, int, str] | None = None + + def publish_heartbeat(state: str, now_ns: int, force: bool = False) -> None: + """Publish bounded liveness state when time or emitted bytes require it.""" + nonlocal bytes_since_heartbeat, last_heartbeat_ns + if not force and ( + now_ns - last_heartbeat_ns < heartbeat_interval_ns + and bytes_since_heartbeat < 65536 + ): + return + write_json_atomic( + paths.heartbeat, + { + "schema_version": 1, + "run_id": options.run_id, + "state": state, + "epoch": epoch, + "monotonic_ns": now_ns, + "bytes_written": bytes_since_heartbeat, + }, + ) + bytes_since_heartbeat = 0 + last_heartbeat_ns = now_ns + + try: + while not stopping: + now_ns = time.monotonic_ns() + if read_control_marker(paths.stop, options.run_id) is not None: + break + + if activated_at_ns is None: + if read_control_marker(paths.gate, options.run_id) is not None: + activated_at_ns = now_ns + next_frame_ns = now_ns + else: + publish_heartbeat("paused", now_ns) + time.sleep(0.005) + continue + + assert activated_at_ns is not None + assert next_frame_ns is not None + if now_ns - activated_at_ns >= int(options.duration_s * 1_000_000_000): + break + + while now_ns >= next_frame_ns: + for mode in StreamMode: + frame = render_frame(mode, epoch, corpus, options.seed) + bytes_since_heartbeat += _append_text( + _stream_path(paths, mode), frame.text + ) + epoch += 1 + next_frame_ns += frame_interval_ns + + if pending_request is None: + for request_path in sorted(paths.requests.glob("*.json")): + if request_path.name in seen_requests: + continue + marker = read_control_marker(request_path, options.run_id) + if marker is None: + continue + request = _request_from_marker(marker, request_path, options) + if request is None: + continue + request_id, requested_ns, value = request + pending_request = ( + request_id, + requested_ns + delay_ns, + sentinel_text(options.run_id, request_id, value), + ) + seen_requests.add(request_path.name) + break + + if pending_request is not None and now_ns >= pending_request[1]: + request_id, scheduled_ns, sentinel = pending_request + bytes_since_heartbeat += _append_text( + _stream_path(paths, StreamMode.DELAYED_MATCH), f"{sentinel}\n" + ) + emitted_ns = time.monotonic_ns() + evidence = SentinelEvidence( + schema_version=1, + run_id=options.run_id, + request_id=request_id, + scheduled_monotonic_ns=scheduled_ns, + emitted_monotonic_ns=emitted_ns, + sentinel=sentinel, + ) + write_json_atomic( + paths.sentinels / f"{request_id}.json", dataclasses.asdict(evidence) + ) + pending_request = None + + publish_heartbeat("active", now_ns) + sleep_ns = max(0, next_frame_ns - time.monotonic_ns()) + time.sleep(min(0.005, sleep_ns / 1_000_000_000)) + finally: + now_ns = time.monotonic_ns() + publish_heartbeat("stopped", now_ns, force=True) + signal.signal(signal.SIGINT, previous_sigint) + signal.signal(signal.SIGTERM, previous_sigterm) + return 0 + + +def run_preview(options: WorkloadOptions) -> int: + """Render the deterministic frames interactively without importing Rich at load.""" + if options.frame_rate_hz <= 0: + message = "frame_rate_hz must be positive" + raise ValueError(message) + import rich.live + import rich.text + + corpus = source_lines(options.source_root, options.seed) + deadline = time.monotonic() + options.duration_s + epoch = 0 + with rich.live.Live(refresh_per_second=10) as live: + while time.monotonic() < deadline: + rendered = [ + render_frame(mode, epoch, corpus, options.seed).text.rstrip() + for mode in StreamMode + ] + live.update(rich.text.Text("\n".join(rendered))) + epoch += 1 + time.sleep(1 / options.frame_rate_hz) + return 0 + + +def _options_from_namespace(arguments: argparse.Namespace) -> WorkloadOptions: + """Convert parsed command-line values into the typed workload configuration.""" + return WorkloadOptions( + output_dir=pathlib.Path(arguments.output_dir), + run_id=arguments.run_id, + source_root=pathlib.Path(arguments.source_root), + seed=arguments.seed, + frame_rate_hz=arguments.frame_rate, + duration_s=arguments.duration, + delayed_match_after_s=arguments.delayed_match_after, + sentinel_prefix=arguments.sentinel_prefix, + heartbeat_interval_s=arguments.heartbeat_interval, + ) + + +def main(argv: t.Sequence[str] | None = None) -> int: + """Run the ``serve`` or Rich ``preview`` command.""" + parser = argparse.ArgumentParser(prog="orchestration_fuzzer.py") + commands = parser.add_subparsers(dest="command", required=True) + for command in ("serve", "preview"): + command_parser = commands.add_parser(command) + command_parser.add_argument("--output-dir", default="orchestration-fuzzer") + command_parser.add_argument("--run-id", default="run-0") + command_parser.add_argument("--source-root", default=".") + command_parser.add_argument("--seed", type=int, default=0) + command_parser.add_argument("--frame-rate", type=float, default=10.0) + command_parser.add_argument("--duration", type=float, default=60.0) + command_parser.add_argument("--delayed-match-after", type=float, default=1.0) + command_parser.add_argument("--sentinel-prefix", default="READY") + command_parser.add_argument("--heartbeat-interval", type=float, default=0.25) + arguments = parser.parse_args(argv) + options = _options_from_namespace(arguments) + if arguments.command == "serve": + return run_serve(options) + return run_preview(options) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_orchestration_fuzzer.py b/tests/test_orchestration_fuzzer.py new file mode 100644 index 0000000000..12eaed70a0 --- /dev/null +++ b/tests/test_orchestration_fuzzer.py @@ -0,0 +1,265 @@ +"""Behavioral checks for the active orchestration stream service.""" + +from __future__ import annotations + +import importlib.util +import json +import os +import pathlib +import subprocess +import sys +import time +import types +import typing as t + +import pytest + + +@pytest.fixture() +def fuzzer_module() -> types.ModuleType: + """Load the standalone fuzzer script without requiring Rich.""" + script = pathlib.Path(__file__).parents[1] / "scripts" / "orchestration_fuzzer.py" + spec = importlib.util.spec_from_file_location("orchestration_fuzzer", script) + assert spec is not None + assert spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +def test_stream_for_pane_reserves_only_selected_ordinal( + fuzzer_module: types.ModuleType, +) -> None: + """Only the chosen pane ordinal receives the delayed-match stream.""" + observed = [ + fuzzer_module.stream_for_pane(ordinal, delayed_ordinal=2).value + for ordinal in range(8) + ] + + assert observed == [ + "editor", + "dev-server", + "delayed-match", + "installer", + "editor", + "dev-server", + "installer", + "editor", + ] + + +def test_sentinel_text_is_unique_to_run_and_request( + fuzzer_module: types.ModuleType, +) -> None: + """A delayed match cannot be confused with another run or request.""" + assert ( + fuzzer_module.sentinel_text("run-7", "sample-03", "READY") + == "LIBTMUX_SENTINEL run=run-7 request=sample-03 value=READY" + ) + + +def test_source_lines_uses_sorted_paths_and_a_private_seeded_shuffle( + fuzzer_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """Corpus ordering is repeatable even when the filesystem order differs.""" + source_root = tmp_path / "project" + source = source_root / "src" / "libtmux" + source.mkdir(parents=True) + (source / "zeta.py").write_bytes(b"zeta one\nzeta two\n") + (source / "alpha.py").write_bytes(b"alpha one\n") + + assert fuzzer_module.source_lines(source_root, seed=7) == ( + "src/libtmux/zeta.py:2: zeta two", + "src/libtmux/alpha.py:1: alpha one", + "src/libtmux/zeta.py:1: zeta one", + ) + + +def test_render_frame_uses_only_its_mode_epoch_and_seeded_corpus( + fuzzer_module: types.ModuleType, +) -> None: + """Frame rendering has stable mode-specific output for a fixed input.""" + corpus = ("src/libtmux/alpha.py:1: alpha", "src/libtmux/zeta.py:2: zeta") + + editor = fuzzer_module.render_frame( + fuzzer_module.StreamMode.EDITOR, + epoch=4, + corpus=corpus, + seed=11, + ) + installer = fuzzer_module.render_frame( + fuzzer_module.StreamMode.INSTALLER, + epoch=4, + corpus=corpus, + seed=11, + ) + + assert editor.text == "[editor epoch=4] src/libtmux/zeta.py:2: zeta\n" + assert installer.text == "[installer epoch=4] install phase=install unit=5/8\n" + + +def write_marker(path: pathlib.Path, data: dict[str, t.Any]) -> None: + """Publish one complete marker without exposing a partial JSON document.""" + temporary = path.with_name(f".{path.name}.tmp") + temporary.write_text(json.dumps(data), encoding="utf-8") + temporary.replace(path) + + +def wait_for(predicate: t.Callable[[], bool], timeout_s: float = 3.0) -> None: + """Wait for an observable external condition or fail with a useful timeout.""" + deadline = time.monotonic() + timeout_s + while time.monotonic() < deadline: + if predicate(): + return + time.sleep(0.01) + pytest.fail("timed out waiting for fuzzer marker state") + + +def read_json(path: pathlib.Path) -> dict[str, t.Any]: + """Read one completed JSON marker.""" + return t.cast(dict[str, t.Any], json.loads(path.read_text(encoding="utf-8"))) + + +def start_serve(output_dir: pathlib.Path) -> subprocess.Popen[str]: + """Start the standalone fuzzer with a short deterministic service cadence.""" + root = pathlib.Path(__file__).parents[1] + environment = os.environ.copy() + environment.pop("VIRTUAL_ENV", None) + return subprocess.Popen( + ( + sys.executable, + str(root / "scripts" / "orchestration_fuzzer.py"), + "serve", + "--output-dir", + str(output_dir), + "--run-id", + "run-7", + "--source-root", + str(root), + "--seed", + "11", + "--frame-rate", + "100", + "--duration", + "10", + "--delayed-match-after", + "0.02", + "--sentinel-prefix", + "READY", + "--heartbeat-interval", + "0.01", + ), + cwd=root, + env=environment, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + + +def finish_process(process: subprocess.Popen[str]) -> None: + """Ensure a failed service assertion cannot leak a fuzzer process.""" + if process.poll() is None: + process.terminate() + process.wait(timeout=3) + + +def test_serve_pauses_until_a_matching_gate_and_handles_repeated_requests( + tmp_path: pathlib.Path, +) -> None: + """A gated service emits each request sentinel once and exits on matching stop.""" + output_dir = tmp_path / "fuzzer-output" + process = start_serve(output_dir) + + try: + ready = output_dir / "ready.json" + streams = output_dir / "streams" + requests = output_dir / "requests" + sentinels = output_dir / "sentinels" + wait_for(ready.exists) + + assert read_json(ready) == {"schema_version": 1, "run_id": "run-7"} + stream_paths = [ + streams / f"{mode}.log" + for mode in ( + "editor", + "dev-server", + "installer", + "delayed-match", + ) + ] + assert all(path.read_bytes() == b"" for path in stream_paths) + + write_marker(output_dir / "gate.json", {"schema_version": 1, "run_id": "other"}) + time.sleep(0.05) + assert all(path.read_bytes() == b"" for path in stream_paths) + + (output_dir / "gate.json").write_text("{not-json", encoding="utf-8") + time.sleep(0.05) + assert all(path.read_bytes() == b"" for path in stream_paths) + + write_marker(output_dir / "gate.json", {"schema_version": 1, "run_id": "run-7"}) + wait_for(lambda: all(path.read_bytes() for path in stream_paths)) + + write_marker( + requests / "wrong-run.json", + { + "schema_version": 1, + "run_id": "other", + "request_id": "wrong-run", + "requested_monotonic_ns": time.monotonic_ns(), + "value": "READY", + }, + ) + (requests / "malformed.json").write_text("[", encoding="utf-8") + time.sleep(0.05) + assert not (sentinels / "wrong-run.json").exists() + assert not (sentinels / "malformed.json").exists() + + request_ids = ("sample-00", "sample-01") + for request_id in request_ids: + requested = time.monotonic_ns() + write_marker( + requests / f"{request_id}.json", + { + "schema_version": 1, + "run_id": "run-7", + "request_id": request_id, + "requested_monotonic_ns": requested, + "value": "READY", + }, + ) + evidence_path = sentinels / f"{request_id}.json" + wait_for(evidence_path.exists) + evidence = read_json(evidence_path) + assert evidence["schema_version"] == 1 + assert evidence["run_id"] == "run-7" + assert evidence["request_id"] == request_id + assert evidence["scheduled_monotonic_ns"] >= requested + assert ( + evidence["emitted_monotonic_ns"] >= evidence["scheduled_monotonic_ns"] + ) + assert evidence["sentinel"] == ( + f"LIBTMUX_SENTINEL run=run-7 request={request_id} value=READY" + ) + delayed_stream = (streams / "delayed-match.log").read_text(encoding="utf-8") + assert delayed_stream.count(evidence["sentinel"]) == 1 + + def ordinary_frame_follows(sentinel: str = evidence["sentinel"]) -> bool: + """Check that the service resumed normal delayed-stream output.""" + return (streams / "delayed-match.log").read_text( + encoding="utf-8" + ).rstrip().splitlines()[-1] != sentinel + + wait_for(ordinary_frame_follows) + + write_marker(output_dir / "stop.json", {"schema_version": 1, "run_id": "run-7"}) + assert process.wait(timeout=3) == 0 + heartbeat = read_json(output_dir / "heartbeat.json") + assert heartbeat["schema_version"] == 1 + assert heartbeat["run_id"] == "run-7" + assert heartbeat["state"] == "stopped" + finally: + finish_process(process) From 9558e379c9f6580f487dd48324bf7d93a2a640ac Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 06:19:28 -0500 Subject: [PATCH 04/67] Bench(fix[fuzzer]): Record sentinel timing why: Let benchmark reports separate configured wait delay from generator scheduling lateness and reject ambiguous control marker schemas. what: - Record request time, configured delay, and scheduling lateness in evidence - Require an exact integer schema version for every control marker - Add timing, schema, and executable documentation coverage --- scripts/orchestration_fuzzer.py | 116 ++++++++++++++++++++++++++--- tests/test_orchestration_fuzzer.py | 102 +++++++++++++++++++++++++ 2 files changed, 206 insertions(+), 12 deletions(-) diff --git a/scripts/orchestration_fuzzer.py b/scripts/orchestration_fuzzer.py index d5ea63fb90..74b6220773 100644 --- a/scripts/orchestration_fuzzer.py +++ b/scripts/orchestration_fuzzer.py @@ -132,10 +132,16 @@ class SentinelEvidence: Run identity that owns the evidence. request_id : str Unique request identity. + requested_monotonic_ns : int + Request receipt timestamp supplied by the benchmark. + configured_delay_ns : int + Configured delay applied to the request timestamp. scheduled_monotonic_ns : int Requested emission deadline on the monotonic clock. emitted_monotonic_ns : int Actual append time on the monotonic clock. + scheduling_lateness_ns : int + Difference between the actual append time and the scheduled deadline. sentinel : str Canonical text appended to the delayed stream. """ @@ -143,8 +149,11 @@ class SentinelEvidence: schema_version: int run_id: str request_id: str + requested_monotonic_ns: int + configured_delay_ns: int scheduled_monotonic_ns: int emitted_monotonic_ns: int + scheduling_lateness_ns: int sentinel: str @@ -200,7 +209,7 @@ def render_frame( r"""Render one deterministic newline-terminated activity record. >>> render_frame(StreamMode.DEV_SERVER, 4, (), 11).text - '[dev-server epoch=4] request=GET /sessions status=200 elapsed_ms=5\\n' + '[dev-server epoch=4] recovery service=api state=ready\n' """ if mode is StreamMode.EDITOR: source = corpus[(seed + epoch) % len(corpus)] if corpus else "" @@ -225,6 +234,17 @@ def render_frame( def prepare_output(options: WorkloadOptions) -> WorkloadPaths: """Create the exclusive marker tree and empty append-only stream files. + Examples + -------- + >>> with tempfile.TemporaryDirectory() as temporary: + ... root = pathlib.Path(temporary) + ... options = WorkloadOptions( + ... root / "output", "run-7", root, 0, 1.0, 1.0, 0.0, "READY", 1.0 + ... ) + ... paths = prepare_output(options) + ... sorted(path.name for path in paths.streams.iterdir()) + ['delayed-match.log', 'dev-server.log', 'editor.log', 'installer.log'] + Raises ------ FileExistsError @@ -252,10 +272,18 @@ def prepare_output(options: WorkloadOptions) -> WorkloadPaths: def write_json_atomic(path: pathlib.Path, data: t.Mapping[str, t.Any]) -> None: - """Replace a marker only after its JSON bytes are durable on disk. + r"""Replace a marker only after its JSON bytes are durable on disk. The sibling temporary file and replacement make readers see either the previous complete marker or the next complete marker, never a partial one. + + Examples + -------- + >>> with tempfile.TemporaryDirectory() as temporary: + ... marker = pathlib.Path(temporary) / "marker.json" + ... write_json_atomic(marker, {"schema_version": 1}) + ... marker.read_text(encoding="utf-8") + '{"schema_version":1}\n' """ with tempfile.NamedTemporaryFile( mode="w", @@ -299,7 +327,8 @@ def read_control_marker(path: pathlib.Path, run_id: str) -> dict[str, t.Any] | N if not isinstance(parsed, dict): return None marker = t.cast(dict[str, t.Any], parsed) - if marker.get("schema_version") != 1 or marker.get("run_id") != run_id: + version = marker.get("schema_version") + if type(version) is not int or version != 1 or marker.get("run_id") != run_id: return None return marker @@ -310,7 +339,15 @@ def _stream_path(paths: WorkloadPaths, mode: StreamMode) -> pathlib.Path: def _append_text(path: pathlib.Path, text: str) -> int: - """Append one complete text record and return its UTF-8 byte count.""" + r"""Append one complete text record and return its UTF-8 byte count. + + Examples + -------- + >>> with tempfile.TemporaryDirectory() as temporary: + ... stream = pathlib.Path(temporary) / "stream.log" + ... _append_text(stream, "pi\n"), stream.read_text(encoding="utf-8") + (3, 'pi\n') + """ encoded = text.encode("utf-8") with path.open("ab") as stream: stream.write(encoded) @@ -323,7 +360,21 @@ def _request_from_marker( path: pathlib.Path, options: WorkloadOptions, ) -> tuple[str, int, str] | None: - """Validate one request marker and return its delayed sentinel inputs.""" + """Validate one request marker and return its delayed sentinel inputs. + + Examples + -------- + >>> options = WorkloadOptions( + ... pathlib.Path("out"), "run-7", pathlib.Path("."), 0, 1.0, 1.0, + ... 0.0, "READY", 1.0, + ... ) + >>> _request_from_marker( + ... {"request_id": "sample", "requested_monotonic_ns": 7}, + ... pathlib.Path("sample.json"), + ... options, + ... ) + ('sample', 7, 'READY') + """ request_id = marker.get("request_id") requested = marker.get("requested_monotonic_ns") value = marker.get("value", options.sentinel_prefix) @@ -342,7 +393,13 @@ def _request_from_marker( def run_serve(options: WorkloadOptions) -> int: - """Serve paused deterministic streams until a matching stop or lifecycle exit.""" + """Serve paused deterministic streams until a matching stop or lifecycle exit. + + Notes + ----- + The service owns signals and a real marker tree, so its gate, timing, and + shutdown behavior is exercised in ``tests/test_orchestration_fuzzer.py``. + """ if options.frame_rate_hz <= 0: message = "frame_rate_hz must be positive" raise ValueError(message) @@ -378,7 +435,7 @@ def request_stop(_signal_number: int, _frame: t.Any) -> None: next_frame_ns: int | None = None epoch = 0 seen_requests: set[str] = set() - pending_request: tuple[str, int, str] | None = None + pending_request: tuple[str, int, int, int, str] | None = None def publish_heartbeat(state: str, now_ns: int, force: bool = False) -> None: """Publish bounded liveness state when time or emitted bytes require it.""" @@ -444,14 +501,22 @@ def publish_heartbeat(state: str, now_ns: int, force: bool = False) -> None: request_id, requested_ns, value = request pending_request = ( request_id, + requested_ns, + delay_ns, requested_ns + delay_ns, sentinel_text(options.run_id, request_id, value), ) seen_requests.add(request_path.name) break - if pending_request is not None and now_ns >= pending_request[1]: - request_id, scheduled_ns, sentinel = pending_request + if pending_request is not None and now_ns >= pending_request[3]: + ( + request_id, + requested_ns, + configured_delay_ns, + scheduled_ns, + sentinel, + ) = pending_request bytes_since_heartbeat += _append_text( _stream_path(paths, StreamMode.DELAYED_MATCH), f"{sentinel}\n" ) @@ -460,8 +525,11 @@ def publish_heartbeat(state: str, now_ns: int, force: bool = False) -> None: schema_version=1, run_id=options.run_id, request_id=request_id, + requested_monotonic_ns=requested_ns, + configured_delay_ns=configured_delay_ns, scheduled_monotonic_ns=scheduled_ns, emitted_monotonic_ns=emitted_ns, + scheduling_lateness_ns=emitted_ns - scheduled_ns, sentinel=sentinel, ) write_json_atomic( @@ -481,7 +549,14 @@ def publish_heartbeat(state: str, now_ns: int, force: bool = False) -> None: def run_preview(options: WorkloadOptions) -> int: - """Render the deterministic frames interactively without importing Rich at load.""" + """Render deterministic frames interactively without importing Rich at load. + + Notes + ----- + This terminal UI requires a live Rich console. Its import boundary is + exercised by the module-import tests; service rendering is covered by the + dedicated functional test file. + """ if options.frame_rate_hz <= 0: message = "frame_rate_hz must be positive" raise ValueError(message) @@ -504,7 +579,18 @@ def run_preview(options: WorkloadOptions) -> int: def _options_from_namespace(arguments: argparse.Namespace) -> WorkloadOptions: - """Convert parsed command-line values into the typed workload configuration.""" + """Convert parsed command-line values into the typed workload configuration. + + Examples + -------- + >>> arguments = argparse.Namespace( + ... output_dir="out", run_id="run-7", source_root=".", seed=3, + ... frame_rate=2.0, duration=4.0, delayed_match_after=0.5, + ... sentinel_prefix="READY", heartbeat_interval=1.0, + ... ) + >>> _options_from_namespace(arguments).run_id + 'run-7' + """ return WorkloadOptions( output_dir=pathlib.Path(arguments.output_dir), run_id=arguments.run_id, @@ -519,7 +605,13 @@ def _options_from_namespace(arguments: argparse.Namespace) -> WorkloadOptions: def main(argv: t.Sequence[str] | None = None) -> int: - """Run the ``serve`` or Rich ``preview`` command.""" + """Run the ``serve`` or Rich ``preview`` command. + + Notes + ----- + The real ``serve`` command is invoked through ``sys.executable`` in the + dedicated functional test file so its process lifecycle stays observable. + """ parser = argparse.ArgumentParser(prog="orchestration_fuzzer.py") commands = parser.add_subparsers(dest="command", required=True) for command in ("serve", "preview"): diff --git a/tests/test_orchestration_fuzzer.py b/tests/test_orchestration_fuzzer.py index 12eaed70a0..3955fc0f47 100644 --- a/tests/test_orchestration_fuzzer.py +++ b/tests/test_orchestration_fuzzer.py @@ -166,6 +166,108 @@ def finish_process(process: subprocess.Popen[str]) -> None: process.wait(timeout=3) +def test_serve_evidence_preserves_request_delay_and_lateness( + tmp_path: pathlib.Path, +) -> None: + """Evidence separates requested delay from scheduling lateness.""" + output_dir = tmp_path / "evidence-output" + process = start_serve(output_dir) + + try: + wait_for(lambda: (output_dir / "ready.json").exists()) + write_marker(output_dir / "gate.json", {"schema_version": 1, "run_id": "run-7"}) + requested = time.monotonic_ns() + write_marker( + output_dir / "requests" / "sample-delay.json", + { + "schema_version": 1, + "run_id": "run-7", + "request_id": "sample-delay", + "requested_monotonic_ns": requested, + "value": "READY", + }, + ) + evidence_path = output_dir / "sentinels" / "sample-delay.json" + wait_for(evidence_path.exists) + evidence = read_json(evidence_path) + + assert evidence["requested_monotonic_ns"] == requested + assert evidence["configured_delay_ns"] == 20_000_000 + assert evidence["scheduled_monotonic_ns"] == requested + 20_000_000 + assert evidence["scheduling_lateness_ns"] == ( + evidence["emitted_monotonic_ns"] - evidence["scheduled_monotonic_ns"] + ) + finally: + write_marker(output_dir / "stop.json", {"schema_version": 1, "run_id": "run-7"}) + finish_process(process) + + +def test_serve_ignores_boolean_and_float_schema_versions( + tmp_path: pathlib.Path, +) -> None: + """Only an integer schema version activates gates, requests, or stops.""" + output_dir = tmp_path / "schema-output" + process = start_serve(output_dir) + + try: + ready = output_dir / "ready.json" + streams = output_dir / "streams" + requests = output_dir / "requests" + sentinels = output_dir / "sentinels" + wait_for(ready.exists) + stream_paths = [ + streams / f"{mode}.log" + for mode in ( + "editor", + "dev-server", + "installer", + "delayed-match", + ) + ] + + for invalid_version in (True, 1.0): + write_marker( + output_dir / "gate.json", + {"schema_version": invalid_version, "run_id": "run-7"}, + ) + time.sleep(0.05) + assert all(path.read_bytes() == b"" for path in stream_paths) + + write_marker(output_dir / "gate.json", {"schema_version": 1, "run_id": "run-7"}) + wait_for(lambda: all(path.read_bytes() for path in stream_paths)) + + for invalid_version, request_id in ( + (True, "bool-request"), + (1.0, "float-request"), + ): + write_marker( + requests / f"{request_id}.json", + { + "schema_version": invalid_version, + "run_id": "run-7", + "request_id": request_id, + "requested_monotonic_ns": time.monotonic_ns(), + "value": "READY", + }, + ) + time.sleep(0.1) + assert not (sentinels / "bool-request.json").exists() + assert not (sentinels / "float-request.json").exists() + + for invalid_version in (True, 1.0): + write_marker( + output_dir / "stop.json", + {"schema_version": invalid_version, "run_id": "run-7"}, + ) + time.sleep(0.05) + assert process.poll() is None + + write_marker(output_dir / "stop.json", {"schema_version": 1, "run_id": "run-7"}) + assert process.wait(timeout=3) == 0 + finally: + finish_process(process) + + def test_serve_pauses_until_a_matching_gate_and_handles_repeated_requests( tmp_path: pathlib.Path, ) -> None: From 8b64f2e351803ee3ad63979e723eb8c61c17e74e Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 06:23:36 -0500 Subject: [PATCH 05/67] Bench(docs[fuzzer]): Exercise service examples why: Keep the fuzzer's long-running public entry points verified as executable documentation. what: - Add temporary gated serve doctest - Add redirected Rich preview and CLI doctests --- scripts/orchestration_fuzzer.py | 52 +++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/scripts/orchestration_fuzzer.py b/scripts/orchestration_fuzzer.py index 74b6220773..e750ec5ec8 100644 --- a/scripts/orchestration_fuzzer.py +++ b/scripts/orchestration_fuzzer.py @@ -395,6 +395,29 @@ def _request_from_marker( def run_serve(options: WorkloadOptions) -> int: """Serve paused deterministic streams until a matching stop or lifecycle exit. + Examples + -------- + >>> with tempfile.TemporaryDirectory() as temporary: + ... root = pathlib.Path(temporary) + ... import threading + ... options = WorkloadOptions( + ... root / "output", "run-7", root, 0, 1000.0, 0.001, 0.0, + ... "READY", 0.001, + ... ) + ... def release_gate() -> None: + ... while not options.output_dir.exists(): + ... time.sleep(0.001) + ... write_json_atomic( + ... options.output_dir / "gate.json", + ... {"schema_version": 1, "run_id": "run-7"}, + ... ) + ... release = threading.Thread(target=release_gate) + ... release.start() + ... result = run_serve(options) + ... release.join() + ... result + 0 + Notes ----- The service owns signals and a real marker tree, so its gate, timing, and @@ -551,6 +574,20 @@ def publish_heartbeat(state: str, now_ns: int, force: bool = False) -> None: def run_preview(options: WorkloadOptions) -> int: """Render deterministic frames interactively without importing Rich at load. + Examples + -------- + >>> with tempfile.TemporaryDirectory() as temporary: + ... root = pathlib.Path(temporary) + ... import io + ... options = WorkloadOptions( + ... root / "output", "run-7", root, 0, 1000.0, 0.001, 0.0, + ... "READY", 0.001, + ... ) + ... with contextlib.redirect_stdout(io.StringIO()): + ... result = run_preview(options) + ... result + 0 + Notes ----- This terminal UI requires a live Rich console. Its import boundary is @@ -607,6 +644,21 @@ def _options_from_namespace(arguments: argparse.Namespace) -> WorkloadOptions: def main(argv: t.Sequence[str] | None = None) -> int: """Run the ``serve`` or Rich ``preview`` command. + Examples + -------- + >>> with tempfile.TemporaryDirectory() as temporary: + ... root = pathlib.Path(temporary) + ... import io + ... with contextlib.redirect_stdout(io.StringIO()): + ... result = main([ + ... "preview", "--output-dir", str(root / "output"), + ... "--run-id", "run-7", "--source-root", str(root), + ... "--frame-rate", "1000", "--duration", "0.001", + ... "--delayed-match-after", "0", "--heartbeat-interval", "0.001", + ... ]) + ... result + 0 + Notes ----- The real ``serve`` command is invoked through ``sys.executable`` in the From b20b30433429641ea34419531c657d99b3ac1d45 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 06:35:32 -0500 Subject: [PATCH 06/67] Bench(feat[model]): Guard active topology why: Keep large orchestration runs within observed host limits and retain valid evidence when they stop early. what: - Add immutable topology, resource guard, and report records - Provide atomic JSON validation and a tmux-free plan command - Cover cgroup probing, admission, summaries, and artifacts --- scripts/bench_orchestration.py | 930 +++++++++++++++++++++++ tests/test_bench_orchestration_script.py | 396 ++++++++++ 2 files changed, 1326 insertions(+) create mode 100644 scripts/bench_orchestration.py create mode 100644 tests/test_bench_orchestration_script.py diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py new file mode 100644 index 0000000000..981ea86458 --- /dev/null +++ b/scripts/bench_orchestration.py @@ -0,0 +1,930 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.10" +# dependencies = ["rich>=13"] +# /// +"""Plan and report hermetic active-tmux orchestration benchmark runs. + +The ``plan`` command intentionally depends only on host files. It does not +import libtmux or start a tmux server. +""" + +from __future__ import annotations + +import argparse +import dataclasses +import json +import math +import pathlib +import resource +import statistics +import tempfile +import typing as t + + +@dataclasses.dataclass(frozen=True) +class Topology: + """Requested active tmux hierarchy. + + Attributes + ---------- + sessions : int + Number of tmux sessions. + windows_per_session : int + Windows created in each session. + panes_per_window : int + Panes created in each window. + """ + + sessions: int + windows_per_session: int + panes_per_window: int + + @property + def windows(self) -> int: + """Return the exact number of windows. + + >>> Topology(2, 3, 4).windows + 6 + """ + return self.sessions * self.windows_per_session + + @property + def panes(self) -> int: + """Return the exact number of panes. + + >>> Topology(2, 3, 4).panes + 24 + """ + return self.windows * self.panes_per_window + + def __str__(self) -> str: + """Return the portable ``SxWxP`` shape. + + >>> str(Topology(2, 3, 4)) + '2x3x4' + """ + return f"{self.sessions}x{self.windows_per_session}x{self.panes_per_window}" + + +class Reader(t.Protocol): + """Read host limits without binding the pure model to the live filesystem.""" + + def read_text(self, path: str) -> str: + """Return the decoded contents at ``path``.""" + + def getrlimit(self, kind: int) -> tuple[int, int]: + """Return the current process's soft and hard limit for ``kind``.""" + + +class ProcessReader: + """Read the host information used by the side-effect-free plan command.""" + + def read_text(self, path: str) -> str: + """Read a procfs or cgroup text file without starting another process. + + >>> isinstance(ProcessReader().read_text("/proc/meminfo"), str) + True + """ + return pathlib.Path(path).read_text(encoding="utf-8") + + def getrlimit(self, kind: int) -> tuple[int, int]: + """Return the process resource limit requested by the probe. + + >>> ProcessReader().getrlimit(resource.RLIMIT_NOFILE)[0] > 0 + True + """ + return resource.getrlimit(kind) + + +@dataclasses.dataclass(frozen=True) +class HostSnapshot: + """Resource values observed before or during one benchmark phase. + + Attributes + ---------- + available_memory_bytes : int | None + Host ``MemAvailable`` capacity, when procfs exposed it. + physical_memory_bytes : int | None + Host physical memory from ``MemTotal``. + memory_current_bytes : int | None + Unified-cgroup current memory consumption. + memory_max_bytes : int | None + Unified-cgroup memory limit; ``None`` means unknown or unlimited. + pids_current : int | None + Unified-cgroup current process count. + pids_max : int | None + Unified-cgroup PID limit; ``None`` means unknown or unlimited. + nofile_soft_limit : int | None + Current process soft file-descriptor limit. + nofile_hard_limit : int | None + Current process hard file-descriptor limit. + memory_pressure_some_avg10 : float | None + Ten-second cgroup memory pressure average. + source_errors : dict[str, str] + Source-specific errors retained instead of replacing missing data with zero. + """ + + available_memory_bytes: int | None = None + physical_memory_bytes: int | None = None + memory_current_bytes: int | None = None + memory_max_bytes: int | None = None + pids_current: int | None = None + pids_max: int | None = None + nofile_soft_limit: int | None = None + nofile_hard_limit: int | None = None + memory_pressure_some_avg10: float | None = None + source_errors: dict[str, str] = dataclasses.field(default_factory=dict) + + +@dataclasses.dataclass(frozen=True) +class ResourcePolicy: + """Conservative admission thresholds for a proposed topology. + + Attributes + ---------- + persistent_clients : int + Persistent tmux clients declared by the selected execution lane. + pid_reserve : int | None + Explicit remaining-PID reserve, or the documented dynamic default. + memory_floor_bytes : int | None + Explicit available-memory floor, or the documented dynamic default. + """ + + persistent_clients: int = 1 + pid_reserve: int | None = None + memory_floor_bytes: int | None = None + + +@dataclasses.dataclass(frozen=True) +class GuardDecision: + """The admission or runtime decision and the snapshot supporting it. + + Attributes + ---------- + allowed : bool + Whether the benchmark may proceed at this checkpoint. + kind : {"ok", "predictive_refusal", "runtime_cutoff"} + Decision category. + rule : str | None + Named guard that made the decision. + observed : int | float | None + Measured or projected value that triggered the rule. + limit : int | float | None + Applicable guard limit. + forceable : bool + Whether ``--force-extreme`` may override the decision. + snapshot : HostSnapshot + Resource observation used by the guard. + """ + + allowed: bool + kind: t.Literal["ok", "predictive_refusal", "runtime_cutoff"] + rule: str | None + observed: int | float | None + limit: int | float | None + forceable: bool + snapshot: HostSnapshot + + +def _source_error(errors: dict[str, str], label: str, exc: Exception) -> None: + """Record one probe failure without substituting a fabricated numeric value. + + >>> errors: dict[str, str] = {} + >>> _source_error(errors, "pids.current", FileNotFoundError("missing")) + >>> "pids.current" in errors + True + """ + errors[label] = f"{type(exc).__name__}: {exc}" + + +def _parse_meminfo(text: str) -> dict[str, int]: + r"""Parse selected ``/proc/meminfo`` values as bytes. + + >>> _parse_meminfo("MemTotal: 2 kB\nMemAvailable: 1 kB\n") + {'MemTotal': 2048, 'MemAvailable': 1024} + """ + values: dict[str, int] = {} + for line in text.splitlines(): + name, separator, raw_value = line.partition(":") + fields = raw_value.split() + if not separator or len(fields) < 2 or fields[1] != "kB": + continue + try: + values[name] = int(fields[0]) * 1024 + except ValueError: + continue + return values + + +def _unified_cgroup_path(cgroup: str, mountinfo: str) -> str: + r"""Resolve this process's cgroup-v2 directory from literal procfs data. + + >>> _unified_cgroup_path("0::/scope\n", "1 0 0:1 / /cg rw - cgroup2 cgroup rw\n") + '/cg/scope' + """ + relative: str | None = None + for line in cgroup.splitlines(): + hierarchy, separator, path = line.partition("::") + if separator and hierarchy == "0": + relative = path + break + if relative is None: + message = "unified cgroup path is unavailable" + raise ValueError(message) + for line in mountinfo.splitlines(): + before, separator, after = line.partition(" - ") + if not separator or not after.split() or after.split()[0] != "cgroup2": + continue + fields = before.split() + if len(fields) < 5: + continue + root, mountpoint = fields[3], fields[4] + suffix = relative.removeprefix(root).lstrip("/") + return str(pathlib.PurePosixPath(mountpoint, suffix)) + message = "cgroup2 mount is unavailable" + raise ValueError(message) + + +def _read_int(reader: Reader, path: str, errors: dict[str, str]) -> int | None: + r"""Read a finite integer limit, preserving failures in ``errors``. + + >>> _read_int(_DoctestReader({"/value": "7\n"}), "/value", {}) + 7 + """ + try: + value = reader.read_text(path).strip() + return None if value == "max" else int(value) + except (OSError, ValueError, KeyError) as exc: + _source_error(errors, pathlib.PurePosixPath(path).name, exc) + return None + + +class _DoctestReader: + """Small in-memory reader used only by the executable helper example.""" + + def __init__(self, values: dict[str, str]) -> None: + self.values = values + + def read_text(self, path: str) -> str: + """Return a configured path's content.""" + return self.values[path] + + def getrlimit(self, kind: int) -> tuple[int, int]: + """Return a harmless finite descriptor limit.""" + return (1, 1) + + +def _pressure_some_avg10(text: str) -> float | None: + r"""Extract the cgroup memory pressure ``some`` ten-second average. + + >>> _pressure_some_avg10("some avg10=0.25 avg60=0.1 total=2\n") + 0.25 + """ + for line in text.splitlines(): + fields = line.split() + if not fields or fields[0] != "some": + continue + for field in fields[1:]: + if field.startswith("avg10="): + return float(field.removeprefix("avg10=")) + return None + + +def probe_host(reader: Reader) -> HostSnapshot: + r"""Probe host and unified-cgroup resources through an injectable reader. + + Missing or malformed telemetry remains ``None`` and records a source error. + + >>> snapshot = probe_host(_DoctestReader({ + ... "/proc/meminfo": "MemTotal: 2 kB\nMemAvailable: 1 kB\n", + ... "/proc/self/cgroup": "0::/scope\n", + ... "/proc/self/mountinfo": "1 0 0:1 / /cg rw - cgroup2 cgroup rw\n", + ... "/cg/scope/pids.current": "1\n", "/cg/scope/pids.max": "2\n", + ... "/cg/scope/memory.current": "3\n", "/cg/scope/memory.max": "4\n", + ... "/cg/scope/memory.pressure": "some avg10=0.0 total=0\n", + ... })) + >>> (snapshot.available_memory_bytes, snapshot.pids_max) + (1024, 2) + """ + errors: dict[str, str] = {} + memory: dict[str, int] = {} + try: + memory = _parse_meminfo(reader.read_text("/proc/meminfo")) + except (OSError, ValueError, KeyError) as exc: + _source_error(errors, "meminfo", exc) + if "MemAvailable" not in memory: + errors.setdefault("MemAvailable", "unavailable from /proc/meminfo") + if "MemTotal" not in memory: + errors.setdefault("MemTotal", "unavailable from /proc/meminfo") + + cgroup_path: str | None = None + try: + cgroup_path = _unified_cgroup_path( + reader.read_text("/proc/self/cgroup"), + reader.read_text("/proc/self/mountinfo"), + ) + except (OSError, ValueError, KeyError) as exc: + _source_error(errors, "cgroup2", exc) + + if cgroup_path is None: + return HostSnapshot( + available_memory_bytes=memory.get("MemAvailable"), + physical_memory_bytes=memory.get("MemTotal"), + source_errors=errors, + ) + + pids_current = _read_int(reader, f"{cgroup_path}/pids.current", errors) + pids_max = _read_int(reader, f"{cgroup_path}/pids.max", errors) + memory_current = _read_int(reader, f"{cgroup_path}/memory.current", errors) + memory_max = _read_int(reader, f"{cgroup_path}/memory.max", errors) + pressure: float | None = None + try: + pressure = _pressure_some_avg10( + reader.read_text(f"{cgroup_path}/memory.pressure") + ) + except (OSError, ValueError, KeyError) as exc: + _source_error(errors, f"{cgroup_path}/memory.pressure", exc) + nofile_soft: int | None + nofile_hard: int | None + try: + nofile_soft, nofile_hard = reader.getrlimit(resource.RLIMIT_NOFILE) + except (OSError, ValueError) as exc: + _source_error(errors, "RLIMIT_NOFILE", exc) + nofile_soft = nofile_hard = None + return HostSnapshot( + available_memory_bytes=memory.get("MemAvailable"), + physical_memory_bytes=memory.get("MemTotal"), + memory_current_bytes=memory_current, + memory_max_bytes=memory_max, + pids_current=pids_current, + pids_max=pids_max, + nofile_soft_limit=nofile_soft, + nofile_hard_limit=nofile_hard, + memory_pressure_some_avg10=pressure, + source_errors=errors, + ) + + +def _default_pid_reserve(snapshot: HostSnapshot) -> int | None: + """Return the documented dynamic PID reserve when a limit is known. + + >>> _default_pid_reserve(HostSnapshot(pids_max=10_000)) + 1500 + """ + if snapshot.pids_max is None: + return None + return max(1024, math.ceil(snapshot.pids_max * 0.15)) + + +def _default_memory_floor(snapshot: HostSnapshot) -> int | None: + """Return the documented dynamic memory floor when physical memory is known. + + >>> _default_memory_floor(HostSnapshot(physical_memory_bytes=10 * 2**30)) + 4294967296 + """ + if snapshot.physical_memory_bytes is None: + return None + return max(4 * 2**30, math.ceil(snapshot.physical_memory_bytes * 0.15)) + + +def predict_resources( + topology: Topology, + snapshot: HostSnapshot, + policy: ResourcePolicy | None = None, +) -> GuardDecision: + """Apply forceable preflight guards without estimating uncalibrated memory. + + >>> snapshot = HostSnapshot(pids_current=1, pids_max=2000) + >>> decision = predict_resources(Topology(1, 1, 1), snapshot) + >>> decision.kind + 'ok' + """ + policy = policy or ResourcePolicy() + reserve = ( + policy.pid_reserve + if policy.pid_reserve is not None + else _default_pid_reserve(snapshot) + ) + if ( + snapshot.pids_current is not None + and snapshot.pids_max is not None + and reserve is not None + ): + projected = ( + snapshot.pids_current + topology.panes + 2 + policy.persistent_clients + ) + usable_limit = snapshot.pids_max - reserve + if projected > usable_limit: + return GuardDecision( + allowed=False, + kind="predictive_refusal", + rule="pid_reserve", + observed=projected, + limit=usable_limit, + forceable=True, + snapshot=snapshot, + ) + floor = ( + policy.memory_floor_bytes + if policy.memory_floor_bytes is not None + else _default_memory_floor(snapshot) + ) + if ( + snapshot.available_memory_bytes is not None + and floor is not None + and snapshot.available_memory_bytes < floor + ): + return GuardDecision( + allowed=False, + kind="predictive_refusal", + rule="memory_floor", + observed=snapshot.available_memory_bytes, + limit=floor, + forceable=True, + snapshot=snapshot, + ) + return GuardDecision(True, "ok", None, None, None, False, snapshot) + + +def check_runtime_guard( + snapshot: HostSnapshot, + *, + policy: ResourcePolicy | None = None, + processes_alive: bool = True, + topology_verified: bool = True, + watchdog_ok: bool = True, + cleanup_complete: bool = True, + force_extreme: bool = False, +) -> GuardDecision: + """Apply non-forceable guards to live benchmark observations. + + ``force_extreme`` is accepted only to make explicit that it cannot relax a + runtime cutoff. + + >>> check_runtime_guard(HostSnapshot(pids_current=2000, pids_max=2000)).kind + 'runtime_cutoff' + """ + del force_extreme + policy = policy or ResourcePolicy() + reserve = ( + policy.pid_reserve + if policy.pid_reserve is not None + else _default_pid_reserve(snapshot) + ) + if ( + snapshot.pids_current is not None + and snapshot.pids_max is not None + and reserve is not None + ): + usable_limit = snapshot.pids_max - reserve + if snapshot.pids_current > usable_limit: + return GuardDecision( + False, + "runtime_cutoff", + "pid_reserve", + snapshot.pids_current, + usable_limit, + False, + snapshot, + ) + floor = ( + policy.memory_floor_bytes + if policy.memory_floor_bytes is not None + else _default_memory_floor(snapshot) + ) + if ( + snapshot.available_memory_bytes is not None + and floor is not None + and snapshot.available_memory_bytes < floor + ): + return GuardDecision( + False, + "runtime_cutoff", + "memory_floor", + snapshot.available_memory_bytes, + floor, + False, + snapshot, + ) + for rule, valid in ( + ("dead_process", processes_alive), + ("topology", topology_verified), + ("watchdog", watchdog_ok), + ("cleanup", cleanup_complete), + ): + if not valid: + return GuardDecision( + False, "runtime_cutoff", rule, None, None, False, snapshot + ) + return GuardDecision(True, "ok", None, None, None, False, snapshot) + + +@dataclasses.dataclass(frozen=True) +class RawSample: + """One timed phase result retained before statistical aggregation. + + Attributes + ---------- + duration_ns : int | None + Measured integer duration, if the timing completed. + accepted : bool + Whether phase correctness accepted this value for a summary. + error : str | None + Failure detail for a rejected sample. + """ + + duration_ns: int | None + accepted: bool + error: str | None = None + + +@dataclasses.dataclass(frozen=True) +class PhaseReport: + """Raw and summarized evidence for one named benchmark cell. + + Attributes + ---------- + name : str + Stable phase and strategy name. + requested_topology : Topology + Shape requested for this phase. + observed_topology : Topology | None + Shape verified at the phase boundary. + samples : tuple[RawSample, ...] + Timed results, including explicitly rejected rows. + summary : dict[str, int | float] | None + Statistics recomputed from accepted rows only. + """ + + name: str + requested_topology: Topology + observed_topology: Topology | None + samples: tuple[RawSample, ...] = () + summary: dict[str, int | float] | None = None + + +@dataclasses.dataclass(frozen=True) +class CleanupReport: + """Evidence that all resources owned by a run were removed. + + Attributes + ---------- + complete : bool + Whether process, socket, and scratch cleanup verification passed. + errors : tuple[str, ...] + Cleanup verification failures. + """ + + complete: bool + errors: tuple[str, ...] = () + + +@dataclasses.dataclass(frozen=True) +class RampStep: + """Outcome recorded for one canonical ramp shape. + + Attributes + ---------- + shape : str + Canonical ``SxWxP`` shape. + status : {"completed", "refused", "failed", "cutoff", "not_attempted"} + Terminal result for the step. + """ + + shape: str + status: t.Literal["completed", "refused", "failed", "cutoff", "not_attempted"] + + +@dataclasses.dataclass(frozen=True) +class RunReport: + """Machine-readable evidence for one benchmark run or plan. + + Attributes + ---------- + requested_topology : Topology + Topology selected by the user. + observed_topology : Topology | None + Last exact topology verified by the worker. + status : {"in_progress", "completed", "refused", "failed", "cutoff"} + Current report lifecycle status. + phases : tuple[PhaseReport, ...] + Named phase records. + cleanup : CleanupReport + Cleanup evidence; required complete for terminal status. + maximum_completed : bool + True only after exact requested and observed ``100x100x4`` completion. + ramp : tuple[RampStep, ...] + Ordered ramp outcomes, if this is a ramp run. + guard_decision : GuardDecision | None + Effective decision after an optional predictive override. + original_guard_decision : GuardDecision | None + Decision before an optional predictive override. + schema_version : int + Stable artifact schema version. + """ + + requested_topology: Topology + observed_topology: Topology | None = None + status: t.Literal["in_progress", "completed", "refused", "failed", "cutoff"] = ( + "in_progress" + ) + phases: tuple[PhaseReport, ...] = () + cleanup: CleanupReport = CleanupReport(complete=False) + maximum_completed: bool = False + ramp: tuple[RampStep, ...] = () + guard_decision: GuardDecision | None = None + original_guard_decision: GuardDecision | None = None + schema_version: int = 1 + + +def _json_value(value: object) -> object: + """Convert frozen report records to JSON-native values with stable keys. + + >>> _json_value(Topology(1, 2, 3)) + {'sessions': 1, 'windows_per_session': 2, 'panes_per_window': 3} + """ + if dataclasses.is_dataclass(value) and not isinstance(value, type): + return { + field.name: _json_value(getattr(value, field.name)) + for field in dataclasses.fields(value) + } + if isinstance(value, tuple): + return [_json_value(item) for item in value] + if isinstance(value, dict): + return {str(key): _json_value(item) for key, item in value.items()} + return value + + +def write_json_atomic(path: pathlib.Path, value: object) -> None: + r"""Atomically replace ``path`` with one complete JSON value. + + >>> with tempfile.TemporaryDirectory() as directory: + ... target = pathlib.Path(directory) / "value.json" + ... write_json_atomic(target, {"schema_version": 1}) + ... target.read_text(encoding="utf-8") + '{"schema_version":1}\n' + """ + path.parent.mkdir(parents=True, exist_ok=True) + encoded = ( + json.dumps(_json_value(value), separators=(",", ":"), sort_keys=True) + "\n" + ) + with tempfile.NamedTemporaryFile( + mode="w", + encoding="utf-8", + dir=path.parent, + prefix=f".{path.name}.", + suffix=".tmp", + delete=False, + ) as temporary: + temporary.write(encoded) + temporary.flush() + temporary_path = pathlib.Path(temporary.name) + temporary_path.replace(path) + + +def validate_report(report: RunReport) -> None: + """Reject internally inconsistent or incomplete terminal benchmark evidence. + + >>> report = RunReport(Topology(1, 1, 1)) + >>> validate_report(report) + """ + if report.schema_version != 1: + message = "unsupported report schema_version" + raise ValueError(message) + terminal = {"completed", "refused", "failed", "cutoff"} + if report.status in terminal and not report.cleanup.complete: + message = "terminal report requires complete cleanup" + raise ValueError(message) + for phase in report.phases: + accepted: list[int] = [] + for sample in phase.samples: + if sample.accepted and ( + sample.duration_ns is None or sample.duration_ns < 0 + ): + message = "accepted sample requires a nonnegative duration" + raise ValueError(message) + if sample.accepted: + assert sample.duration_ns is not None + accepted.append(sample.duration_ns) + if phase.summary is not None and ( + not accepted or phase.summary != summarize_ns(accepted) + ): + message = "phase summary must match accepted samples" + raise ValueError(message) + cutoff_seen = False + for step in report.ramp: + if cutoff_seen and step.status != "not_attempted": + message = "ramp shapes after cutoff must be not_attempted" + raise ValueError(message) + cutoff_seen = cutoff_seen or step.status == "cutoff" + maximum = Topology(100, 100, 4) + if report.maximum_completed and ( + report.status != "completed" + or report.requested_topology != maximum + or report.observed_topology != maximum + ): + message = "maximum_completed requires exact requested and observed 100x100x4" + raise ValueError(message) + + +def _forced_decision(decision: GuardDecision, force_extreme: bool) -> GuardDecision: + """Return the only allowed override: a predictive preflight admission. + + >>> original = GuardDecision( + ... False, "predictive_refusal", "pid_reserve", 2, 1, True, HostSnapshot() + ... ) + >>> _forced_decision(original, True).allowed + True + """ + if force_extreme and decision.kind == "predictive_refusal" and decision.forceable: + return dataclasses.replace( + decision, allowed=True, kind="ok", rule="force_extreme" + ) + return decision + + +def plan_payload( + topology: Topology, + snapshot: HostSnapshot, + *, + force_extreme: bool = False, +) -> dict[str, object]: + """Build the side-effect-free JSON payload emitted by the plan command. + + >>> plan_payload(Topology(1, 1, 1), HostSnapshot())["predicted_pane_processes"] + 1 + """ + original = predict_resources(topology, snapshot) + decision = _forced_decision(original, force_extreme) + return t.cast( + dict[str, object], + _json_value( + { + "schema_version": 1, + "command": "plan", + "requested_topology": topology, + "predicted_pane_processes": topology.panes, + "guard_decision": decision, + "original_guard_decision": original, + "force_extreme": force_extreme, + } + ), + ) + + +def run_plan(shape: str, output: pathlib.Path | None, force_extreme: bool) -> int: + """Print one host-only plan and optionally persist its JSON evidence. + + >>> import contextlib, io + >>> captured = io.StringIO() + >>> with contextlib.redirect_stdout(captured): + ... result = run_plan("1x1x1", None, False) + >>> result + 0 + """ + from rich.console import Console + from rich.table import Table + + topology = parse_topology(shape) + payload = plan_payload( + topology, probe_host(ProcessReader()), force_extreme=force_extreme + ) + decision = t.cast(dict[str, object], payload["guard_decision"]) + table = Table(title="Orchestration benchmark plan") + table.add_column("Metric") + table.add_column("Value", justify="right") + table.add_row("Sessions", str(topology.sessions)) + table.add_row("Windows", str(topology.windows)) + table.add_row("Panes", str(topology.panes)) + table.add_row("Guard decision", str(decision["kind"])) + table.add_row("Allowed", str(decision["allowed"])) + Console().print(table) + if output is not None: + write_json_atomic(output, payload) + return 0 + + +def parse_topology(shape: str) -> Topology: + """Parse a positive ``SxWxP`` topology string. + + >>> parse_topology("2x3x4") + Topology(sessions=2, windows_per_session=3, panes_per_window=4) + + Parameters + ---------- + shape : str + Topology in lower-case ``SxWxP`` notation. + + Returns + ------- + Topology + Parsed topology with every dimension positive. + + Raises + ------ + ValueError + If the shape is malformed or has a nonpositive dimension. + """ + pieces = shape.split("x") + if len(pieces) != 3: + message = "topology must use SxWxP notation" + raise ValueError(message) + try: + values = tuple(int(piece) for piece in pieces) + except ValueError as exc: + message = "topology must use SxWxP notation" + raise ValueError(message) from exc + if any(value <= 0 for value in values): + message = "topology dimensions must be positive" + raise ValueError(message) + return Topology(*values) + + +def canonical_ramp() -> tuple[Topology, ...]: + """Return the specified progression ordered by expected pane pressure. + + >>> tuple(str(shape) for shape in canonical_ramp())[:3] + ('80x20x1', '100x20x1', '80x20x2') + """ + return ( + Topology(80, 20, 1), + Topology(100, 20, 1), + Topology(80, 20, 2), + Topology(80, 50, 1), + Topology(80, 20, 4), + Topology(80, 100, 1), + Topology(100, 50, 2), + Topology(100, 100, 2), + Topology(100, 100, 4), + ) + + +def summarize_ns(samples: t.Sequence[int]) -> dict[str, int | float]: + """Return descriptive statistics for accepted integer-nanosecond samples. + + Percentiles use the nearest-rank index ``ceil(p * count) - 1``. + + >>> summarize_ns((1, 2, 3, 4))["p90_ns"] + 4 + + Parameters + ---------- + samples : collections.abc.Sequence[int] + Accepted duration measurements in nanoseconds. + + Returns + ------- + dict[str, int | float] + Count, extrema, mean, median, and p90/p95/p99 values. + + Raises + ------ + ValueError + If no samples were accepted. + """ + if not samples: + message = "cannot summarize empty samples" + raise ValueError(message) + ordered = sorted(samples) + + def percentile(percent: float) -> int: + return ordered[math.ceil(percent * len(ordered)) - 1] + + return { + "count": len(ordered), + "min_ns": ordered[0], + "mean_ns": statistics.mean(ordered), + "median_ns": statistics.median(ordered), + "p90_ns": percentile(0.90), + "p95_ns": percentile(0.95), + "p99_ns": percentile(0.99), + "max_ns": ordered[-1], + } + + +def main(argv: t.Sequence[str] | None = None) -> int: + """Run the side-effect-free benchmark planning command. + + >>> import contextlib, io + >>> captured = io.StringIO() + >>> with contextlib.redirect_stdout(captured): + ... result = main(["plan", "--shape", "1x1x1"]) + >>> result + 0 + """ + parser = argparse.ArgumentParser(description=__doc__) + commands = parser.add_subparsers(dest="command", required=True) + plan_parser = commands.add_parser("plan", help="inspect topology and host limits") + plan_parser.add_argument("--shape", required=True) + plan_parser.add_argument("--output", type=pathlib.Path) + plan_parser.add_argument("--force-extreme", action="store_true") + arguments = parser.parse_args(argv) + if arguments.command == "plan": + return run_plan(arguments.shape, arguments.output, arguments.force_extreme) + message = "argparse selected an unsupported command" + raise AssertionError(message) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py new file mode 100644 index 0000000000..7912d02352 --- /dev/null +++ b/tests/test_bench_orchestration_script.py @@ -0,0 +1,396 @@ +"""Behavioral checks for the orchestration benchmark's pure model.""" + +from __future__ import annotations + +import dataclasses +import importlib.util +import json +import pathlib +import subprocess +import sys +import types +import typing as t + +import pytest + + +@pytest.fixture() +def benchmark_module() -> types.ModuleType: + """Load the standalone benchmark script without contacting tmux.""" + script = pathlib.Path(__file__).parents[1] / "scripts" / "bench_orchestration.py" + spec = importlib.util.spec_from_file_location("bench_orchestration", script) + assert spec is not None + assert spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +def test_parse_topology_derives_exact_totals( + benchmark_module: types.ModuleType, +) -> None: + """Changing any topology multiplier would misstate the active workload.""" + topology = benchmark_module.parse_topology("100x100x4") + + assert topology.sessions == 100 + assert topology.windows == 10_000 + assert topology.panes == 40_000 + + +@pytest.mark.parametrize("shape", ("0x1x1", "1x0x1", "1x1x0", "-1x1x1")) +def test_parse_topology_rejects_nonpositive_dimensions( + benchmark_module: types.ModuleType, shape: str +) -> None: + """Accepting a nonpositive dimension would create a false workload plan.""" + with pytest.raises(ValueError, match="positive"): + benchmark_module.parse_topology(shape) + + +@pytest.mark.parametrize("shape", ("100", "100x20", "100x20x1x2", "100X20X1", "one")) +def test_parse_topology_rejects_malformed_shape( + benchmark_module: types.ModuleType, shape: str +) -> None: + """Malformed topology syntax must not silently select another scenario.""" + with pytest.raises(ValueError, match="SxWxP"): + benchmark_module.parse_topology(shape) + + +def test_summarize_ns_uses_nearest_rank_percentiles( + benchmark_module: types.ModuleType, +) -> None: + """Changing percentile ranks would distort tail latency evidence.""" + samples = tuple(range(1, 101)) + + assert benchmark_module.summarize_ns(samples) == { + "count": 100, + "min_ns": 1, + "mean_ns": 50.5, + "median_ns": 50.5, + "p90_ns": 90, + "p95_ns": 95, + "p99_ns": 99, + "max_ns": 100, + } + + +def test_summarize_ns_rejects_empty_samples(benchmark_module: types.ModuleType) -> None: + """An empty timing cell has no defensible descriptive summary.""" + with pytest.raises(ValueError, match="empty"): + benchmark_module.summarize_ns(()) + + +def test_canonical_ramp_is_ordered_by_live_pane_pressure( + benchmark_module: types.ModuleType, +) -> None: + """A reordered ramp would make resource cutoff evidence incomparable.""" + assert tuple(str(shape) for shape in benchmark_module.canonical_ramp()) == ( + "80x20x1", + "100x20x1", + "80x20x2", + "80x50x1", + "80x20x4", + "80x100x1", + "100x50x2", + "100x100x2", + "100x100x4", + ) + + +class LiteralReader: + """Inject complete procfs and cgroup text without reading the live host.""" + + def __init__(self, files: dict[str, str]) -> None: + self.files = files + + def read_text(self, path: str) -> str: + """Return the literal content for one absolute procfs path.""" + return self.files[path] + + def getrlimit(self, kind: int) -> tuple[int, int]: + """Return a finite nofile limit without consulting the process.""" + assert kind >= 0 + return (65_536, 65_536) + + +def complete_procfs_files() -> dict[str, str]: + """Return a hand-derived unified-cgroup host fixture.""" + cgroup_path = "/user.slice/user-1000.slice/user@1000.service/app.slice/bench.scope" + cgroup_root = "/sys/fs/cgroup" + cgroup_path + return { + "/proc/meminfo": ( + "MemTotal: 33554432 kB\n" + "MemFree: 1048576 kB\n" + "MemAvailable: 25165824 kB\n" + "Buffers: 123456 kB\n" + "Cached: 2345678 kB\n" + ), + "/proc/self/cgroup": f"0::{cgroup_path}\n", + "/proc/self/mountinfo": ( + "29 23 0:26 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime " + "- cgroup2 cgroup rw\n" + ), + f"{cgroup_root}/pids.current": "472\n", + f"{cgroup_root}/pids.max": "45343\n", + f"{cgroup_root}/memory.current": "8589934592\n", + f"{cgroup_root}/memory.max": "34359738368\n", + f"{cgroup_root}/memory.pressure": ( + "some avg10=0.00 avg60=0.00 avg300=0.00 total=7\n" + "full avg10=0.00 avg60=0.00 avg300=0.00 total=7\n" + ), + } + + +def test_probe_host_resolves_unified_cgroup_and_literal_limits( + benchmark_module: types.ModuleType, +) -> None: + """A wrong cgroup join would hide the real container resource envelope.""" + snapshot = benchmark_module.probe_host(LiteralReader(complete_procfs_files())) + + assert snapshot.available_memory_bytes == 25_165_824 * 1024 + assert snapshot.physical_memory_bytes == 33_554_432 * 1024 + assert snapshot.pids_current == 472 + assert snapshot.pids_max == 45_343 + assert snapshot.memory_current_bytes == 8_589_934_592 + assert snapshot.memory_max_bytes == 34_359_738_368 + assert snapshot.nofile_soft_limit == 65_536 + assert snapshot.memory_pressure_some_avg10 == 0.0 + assert snapshot.source_errors == {} + + +def test_probe_host_preserves_missing_telemetry_as_unknown( + benchmark_module: types.ModuleType, +) -> None: + """Treating an unreadable cgroup value as zero would make false admissions.""" + files = complete_procfs_files() + del files[ + "/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/bench.scope/pids.current" + ] + + snapshot = benchmark_module.probe_host(LiteralReader(files)) + + assert snapshot.pids_current is None + assert "pids.current" in snapshot.source_errors + + +def test_predict_resources_refuses_when_projected_pids_break_reserve( + benchmark_module: types.ModuleType, +) -> None: + """Ignoring the PID reserve would let the 40,000-pane plan exhaust cgroups.""" + snapshot = benchmark_module.probe_host(LiteralReader(complete_procfs_files())) + + decision = benchmark_module.predict_resources( + benchmark_module.parse_topology("100x100x4"), snapshot + ) + + assert decision.allowed is False + assert decision.kind == "predictive_refusal" + assert decision.rule == "pid_reserve" + assert decision.observed == 40_475 + assert decision.limit == 38_541 + assert decision.forceable is True + + +def test_predict_resources_admits_small_shape_with_same_host_limits( + benchmark_module: types.ModuleType, +) -> None: + """A guard that over-refuses would prevent the canonical ramp from starting.""" + snapshot = benchmark_module.probe_host(LiteralReader(complete_procfs_files())) + + decision = benchmark_module.predict_resources( + benchmark_module.parse_topology("80x20x1"), snapshot + ) + + assert decision.allowed is True + assert decision.kind == "ok" + assert decision.rule is None + + +def completed_report(benchmark_module: types.ModuleType) -> t.Any: + """Build a literal completed artifact with one failed raw sample excluded.""" + topology = benchmark_module.parse_topology("100x100x4") + phase = benchmark_module.PhaseReport( + name="enumeration.sessions", + requested_topology=topology, + observed_topology=topology, + samples=( + benchmark_module.RawSample(duration_ns=10, accepted=True), + benchmark_module.RawSample(duration_ns=30, accepted=True), + benchmark_module.RawSample( + duration_ns=999, accepted=False, error="lost row count" + ), + ), + summary={ + "count": 2, + "min_ns": 10, + "mean_ns": 20, + "median_ns": 20.0, + "p90_ns": 30, + "p95_ns": 30, + "p99_ns": 30, + "max_ns": 30, + }, + ) + return benchmark_module.RunReport( + status="completed", + requested_topology=topology, + observed_topology=topology, + phases=(phase,), + cleanup=benchmark_module.CleanupReport(complete=True), + maximum_completed=True, + ) + + +def test_write_json_atomic_replaces_complete_report( + benchmark_module: types.ModuleType, tmp_path: pathlib.Path +) -> None: + """A torn report replacement would destroy evidence after a phase checkpoint.""" + report_path = tmp_path / "report.json" + report_path.write_text('{"old":true}\n', encoding="utf-8") + report = completed_report(benchmark_module) + + benchmark_module.write_json_atomic(report_path, report) + + payload = json.loads(report_path.read_text(encoding="utf-8")) + assert payload["schema_version"] == 1 + assert payload["status"] == "completed" + assert payload["requested_topology"] == { + "sessions": 100, + "windows_per_session": 100, + "panes_per_window": 4, + } + + +def test_validate_report_recomputes_only_accepted_samples( + benchmark_module: types.ModuleType, +) -> None: + """Including a failed duration would poison the reported benchmark statistic.""" + benchmark_module.validate_report(completed_report(benchmark_module)) + + +def test_validate_report_rejects_summary_with_failed_sample( + benchmark_module: types.ModuleType, +) -> None: + """A summary that includes a failed timing must not be publishable evidence.""" + report = completed_report(benchmark_module) + bad_phase = dataclasses.replace( + report.phases[0], + summary={ + "count": 3, + "min_ns": 10, + "mean_ns": 346.3333333333333, + "median_ns": 30, + "p90_ns": 999, + "p95_ns": 999, + "p99_ns": 999, + "max_ns": 999, + }, + ) + + with pytest.raises(ValueError, match="summary"): + benchmark_module.validate_report( + dataclasses.replace(report, phases=(bad_phase,)) + ) + + +@pytest.mark.parametrize("status", ("refused", "failed", "cutoff")) +def test_validate_report_requires_cleanup_for_terminal_status( + benchmark_module: types.ModuleType, status: str +) -> None: + """Terminal evidence is invalid if it leaves benchmark-owned state behind.""" + report = dataclasses.replace( + completed_report(benchmark_module), + status=status, + cleanup=benchmark_module.CleanupReport( + complete=False, errors=("socket remains",) + ), + maximum_completed=False, + ) + + with pytest.raises(ValueError, match="cleanup"): + benchmark_module.validate_report(report) + + +def test_validate_report_requires_unattempted_shapes_after_cutoff( + benchmark_module: types.ModuleType, +) -> None: + """Continuing after a cutoff would misrepresent the ramp's host evidence.""" + report = dataclasses.replace( + completed_report(benchmark_module), + status="cutoff", + maximum_completed=False, + ramp=( + benchmark_module.RampStep("80x20x1", "completed"), + benchmark_module.RampStep("100x20x1", "cutoff"), + benchmark_module.RampStep("80x20x2", "completed"), + ), + ) + + with pytest.raises(ValueError, match="not_attempted"): + benchmark_module.validate_report(report) + + +@pytest.mark.parametrize( + ("requested", "observed"), + (("100x100x2", "100x100x2"), ("100x100x4", "100x100x2")), +) +def test_validate_report_rejects_false_maximum_completion( + benchmark_module: types.ModuleType, requested: str, observed: str +) -> None: + """A smaller or incomplete shape cannot be presented as the maximum completed.""" + report = dataclasses.replace( + completed_report(benchmark_module), + requested_topology=benchmark_module.parse_topology(requested), + observed_topology=benchmark_module.parse_topology(observed), + ) + + with pytest.raises(ValueError, match="maximum_completed"): + benchmark_module.validate_report(report) + + +def test_runtime_guard_never_allows_force_override( + benchmark_module: types.ModuleType, +) -> None: + """Forcing a runtime cutoff would bypass actual liveness and cleanup safety.""" + snapshot = benchmark_module.HostSnapshot(pids_current=45_000, pids_max=45_343) + + decision = benchmark_module.check_runtime_guard(snapshot, force_extreme=True) + + assert decision.allowed is False + assert decision.kind == "runtime_cutoff" + assert decision.forceable is False + + +def test_plan_writes_original_predictive_decision_without_talking_to_tmux( + tmp_path: pathlib.Path, +) -> None: + """The planning path must remain an offline inspection even when forced.""" + script = pathlib.Path(__file__).parents[1] / "scripts" / "bench_orchestration.py" + output = tmp_path / "plan.json" + + completed = subprocess.run( + ( + sys.executable, + str(script), + "plan", + "--shape", + "1x1x1", + "--output", + str(output), + "--force-extreme", + ), + cwd=tmp_path, + text=True, + capture_output=True, + check=False, + ) + + assert completed.returncode == 0, completed.stderr + assert "Sessions" in completed.stdout + assert not list(tmp_path.glob("*.sock")) + payload = json.loads(output.read_text(encoding="utf-8")) + assert payload["command"] == "plan" + assert ( + payload["original_guard_decision"] == payload["guard_decision"] + or payload["original_guard_decision"]["kind"] == "predictive_refusal" + ) From 9e41c45b51cd41add36e98dc511bb137d14a70fa Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 06:43:36 -0500 Subject: [PATCH 07/67] Bench(fix[model]): Harden run evidence why: Preserve durable and immutable benchmark evidence across resource failures. what: - Sync atomic report replacement and freeze public mappings - Retain independent host probes and validate verified samples - Bind ramp attempts to declared topology sequences --- scripts/bench_orchestration.py | 170 +++++++++++++++++------ tests/test_bench_orchestration_script.py | 132 +++++++++++++++++- 2 files changed, 254 insertions(+), 48 deletions(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index 981ea86458..a545213eb2 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -12,13 +12,16 @@ from __future__ import annotations import argparse +import collections.abc as cabc import dataclasses import json import math +import os import pathlib import resource import statistics import tempfile +import types import typing as t @@ -121,7 +124,7 @@ class HostSnapshot: Current process hard file-descriptor limit. memory_pressure_some_avg10 : float | None Ten-second cgroup memory pressure average. - source_errors : dict[str, str] + source_errors : collections.abc.Mapping[str, str] Source-specific errors retained instead of replacing missing data with zero. """ @@ -134,7 +137,18 @@ class HostSnapshot: nofile_soft_limit: int | None = None nofile_hard_limit: int | None = None memory_pressure_some_avg10: float | None = None - source_errors: dict[str, str] = dataclasses.field(default_factory=dict) + source_errors: cabc.Mapping[str, str] = dataclasses.field(default_factory=dict) + + def __post_init__(self) -> None: + """Freeze source errors independently of their caller-provided mapping. + + >>> snapshot = HostSnapshot(source_errors={"proc": "missing"}) + >>> snapshot.source_errors["proc"] + 'missing' + """ + object.__setattr__( + self, "source_errors", types.MappingProxyType(dict(self.source_errors)) + ) @dataclasses.dataclass(frozen=True) @@ -327,24 +341,21 @@ def probe_host(reader: Reader) -> HostSnapshot: except (OSError, ValueError, KeyError) as exc: _source_error(errors, "cgroup2", exc) - if cgroup_path is None: - return HostSnapshot( - available_memory_bytes=memory.get("MemAvailable"), - physical_memory_bytes=memory.get("MemTotal"), - source_errors=errors, - ) - - pids_current = _read_int(reader, f"{cgroup_path}/pids.current", errors) - pids_max = _read_int(reader, f"{cgroup_path}/pids.max", errors) - memory_current = _read_int(reader, f"{cgroup_path}/memory.current", errors) - memory_max = _read_int(reader, f"{cgroup_path}/memory.max", errors) + pids_current = pids_max = memory_current = memory_max = None pressure: float | None = None - try: - pressure = _pressure_some_avg10( - reader.read_text(f"{cgroup_path}/memory.pressure") - ) - except (OSError, ValueError, KeyError) as exc: - _source_error(errors, f"{cgroup_path}/memory.pressure", exc) + if cgroup_path is not None: + pids_current = _read_int(reader, f"{cgroup_path}/pids.current", errors) + pids_max = _read_int(reader, f"{cgroup_path}/pids.max", errors) + memory_current = _read_int(reader, f"{cgroup_path}/memory.current", errors) + memory_max = _read_int(reader, f"{cgroup_path}/memory.max", errors) + try: + pressure = _pressure_some_avg10( + reader.read_text(f"{cgroup_path}/memory.pressure") + ) + if pressure is None: + errors["memory.pressure"] = "ValueError: missing some avg10" + except (OSError, ValueError, KeyError) as exc: + _source_error(errors, "memory.pressure", exc) nofile_soft: int | None nofile_hard: int | None try: @@ -532,11 +543,14 @@ class RawSample: Whether phase correctness accepted this value for a summary. error : str | None Failure detail for a rejected sample. + verified : bool + Whether the phase's correctness check accepted this timing. """ duration_ns: int | None accepted: bool error: str | None = None + verified: bool = False @dataclasses.dataclass(frozen=True) @@ -553,7 +567,7 @@ class PhaseReport: Shape verified at the phase boundary. samples : tuple[RawSample, ...] Timed results, including explicitly rejected rows. - summary : dict[str, int | float] | None + summary : collections.abc.Mapping[str, int | float] | None Statistics recomputed from accepted rows only. """ @@ -561,7 +575,21 @@ class PhaseReport: requested_topology: Topology observed_topology: Topology | None samples: tuple[RawSample, ...] = () - summary: dict[str, int | float] | None = None + summary: cabc.Mapping[str, int | float] | None = None + + def __post_init__(self) -> None: + """Freeze a copied summary so callers cannot alter recorded evidence. + + >>> report = PhaseReport( + ... "x", Topology(1, 1, 1), Topology(1, 1, 1), summary={"count": 1} + ... ) + >>> report.summary["count"] if report.summary is not None else None + 1 + """ + if self.summary is not None: + object.__setattr__( + self, "summary", types.MappingProxyType(dict(self.summary)) + ) @dataclasses.dataclass(frozen=True) @@ -586,14 +614,17 @@ class RampStep: Attributes ---------- - shape : str - Canonical ``SxWxP`` shape. + shape : Topology + Declared ramp shape. status : {"completed", "refused", "failed", "cutoff", "not_attempted"} Terminal result for the step. + reason : str | None + Required reason for an unattempted shape after a terminal step. """ - shape: str + shape: Topology status: t.Literal["completed", "refused", "failed", "cutoff", "not_attempted"] + reason: str | None = None @dataclasses.dataclass(frozen=True) @@ -616,6 +647,8 @@ class RunReport: True only after exact requested and observed ``100x100x4`` completion. ramp : tuple[RampStep, ...] Ordered ramp outcomes, if this is a ramp run. + requested_shapes : tuple[Topology, ...] + Exact selected ramp sequence, including valid custom smoke ramps. guard_decision : GuardDecision | None Effective decision after an optional predictive override. original_guard_decision : GuardDecision | None @@ -633,6 +666,7 @@ class RunReport: cleanup: CleanupReport = CleanupReport(complete=False) maximum_completed: bool = False ramp: tuple[RampStep, ...] = () + requested_shapes: tuple[Topology, ...] = () guard_decision: GuardDecision | None = None original_guard_decision: GuardDecision | None = None schema_version: int = 1 @@ -651,12 +685,18 @@ def _json_value(value: object) -> object: } if isinstance(value, tuple): return [_json_value(item) for item in value] - if isinstance(value, dict): + if isinstance(value, cabc.Mapping): return {str(key): _json_value(item) for key, item in value.items()} return value -def write_json_atomic(path: pathlib.Path, value: object) -> None: +def write_json_atomic( + path: pathlib.Path, + value: object, + *, + fsync: t.Callable[[int], None] = os.fsync, + replace: t.Callable[[str | pathlib.Path, str | pathlib.Path], None] = os.replace, +) -> None: r"""Atomically replace ``path`` with one complete JSON value. >>> with tempfile.TemporaryDirectory() as directory: @@ -664,23 +704,47 @@ def write_json_atomic(path: pathlib.Path, value: object) -> None: ... write_json_atomic(target, {"schema_version": 1}) ... target.read_text(encoding="utf-8") '{"schema_version":1}\n' + + Parameters + ---------- + path : pathlib.Path + Artifact destination in an existing or creatable parent directory. + value : object + JSON-native value or immutable report record. + fsync : collections.abc.Callable[[int], None] + Injectable durability boundary for the temporary file and parent directory. + replace : collections.abc.Callable[[str | pathlib.Path, str | pathlib.Path], None] + Injectable atomic replacement boundary. """ path.parent.mkdir(parents=True, exist_ok=True) encoded = ( json.dumps(_json_value(value), separators=(",", ":"), sort_keys=True) + "\n" ) - with tempfile.NamedTemporaryFile( - mode="w", - encoding="utf-8", - dir=path.parent, - prefix=f".{path.name}.", - suffix=".tmp", - delete=False, - ) as temporary: - temporary.write(encoded) - temporary.flush() - temporary_path = pathlib.Path(temporary.name) - temporary_path.replace(path) + temporary_path: pathlib.Path | None = None + try: + with tempfile.NamedTemporaryFile( + mode="w", + encoding="utf-8", + dir=path.parent, + prefix=f".{path.name}.", + suffix=".tmp", + delete=False, + ) as temporary: + temporary_path = pathlib.Path(temporary.name) + temporary.write(encoded) + temporary.flush() + fsync(temporary.fileno()) + replace(temporary_path, path) + temporary_path = None + parent_fd = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY) + try: + fsync(parent_fd) + finally: + os.close(parent_fd) + except Exception: + if temporary_path is not None: + temporary_path.unlink(missing_ok=True) + raise def validate_report(report: RunReport) -> None: @@ -699,6 +763,9 @@ def validate_report(report: RunReport) -> None: for phase in report.phases: accepted: list[int] = [] for sample in phase.samples: + if sample.accepted and (sample.error is not None or not sample.verified): + message = "accepted sample requires verified success without an error" + raise ValueError(message) if sample.accepted and ( sample.duration_ns is None or sample.duration_ns < 0 ): @@ -707,17 +774,34 @@ def validate_report(report: RunReport) -> None: if sample.accepted: assert sample.duration_ns is not None accepted.append(sample.duration_ns) + if (accepted or report.status == "completed") and ( + phase.observed_topology is None + or phase.observed_topology != phase.requested_topology + ): + message = "accepted or completed phase requires exact observed topology" + raise ValueError(message) if phase.summary is not None and ( not accepted or phase.summary != summarize_ns(accepted) ): message = "phase summary must match accepted samples" raise ValueError(message) - cutoff_seen = False - for step in report.ramp: - if cutoff_seen and step.status != "not_attempted": - message = "ramp shapes after cutoff must be not_attempted" + if len(report.ramp) != len(report.requested_shapes): + message = "ramp attempts must match requested shapes cardinality" + raise ValueError(message) + terminal_seen = False + for shape, step in zip(report.requested_shapes, report.ramp, strict=True): + if step.shape != shape: + message = "ramp attempts must match requested shapes in order" + raise ValueError(message) + if report.status == "completed" and step.status != "completed": + message = "completed report may contain completed ramp attempts only" + raise ValueError(message) + if terminal_seen and (step.status != "not_attempted" or step.reason is None): + message = ( + "ramp shapes after terminal step must be not_attempted with reason" + ) raise ValueError(message) - cutoff_seen = cutoff_seen or step.status == "cutoff" + terminal_seen = terminal_seen or step.status in {"refused", "failed", "cutoff"} maximum = Topology(100, 100, 4) if report.maximum_completed and ( report.status != "completed" diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index 7912d02352..27dfba0f8a 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -214,8 +214,8 @@ def completed_report(benchmark_module: types.ModuleType) -> t.Any: requested_topology=topology, observed_topology=topology, samples=( - benchmark_module.RawSample(duration_ns=10, accepted=True), - benchmark_module.RawSample(duration_ns=30, accepted=True), + benchmark_module.RawSample(duration_ns=10, accepted=True, verified=True), + benchmark_module.RawSample(duration_ns=30, accepted=True, verified=True), benchmark_module.RawSample( duration_ns=999, accepted=False, error="lost row count" ), @@ -261,6 +261,118 @@ def test_write_json_atomic_replaces_complete_report( } +def test_write_json_atomic_syncs_file_then_parent( + benchmark_module: types.ModuleType, tmp_path: pathlib.Path +) -> None: + """Skipping either durability barrier could lose a completed checkpoint.""" + target = tmp_path / "report.json" + calls: list[str] = [] + + def record_fsync(fd: int) -> None: + """Record the real descriptor type while preserving fsync behavior.""" + calls.append( + "directory" if pathlib.Path(f"/proc/self/fd/{fd}").is_dir() else "file" + ) + benchmark_module.os.fsync(fd) + + def record_replace( + source: str | pathlib.Path, destination: str | pathlib.Path + ) -> None: + """Record atomic replacement while preserving the filesystem effect.""" + calls.append("replace") + pathlib.Path(source).replace(destination) + + benchmark_module.write_json_atomic( + target, {"ok": True}, fsync=record_fsync, replace=record_replace + ) + + assert calls == ["file", "replace", "directory"] + assert json.loads(target.read_text(encoding="utf-8")) == {"ok": True} + + +def test_report_collections_are_deeply_immutable( + benchmark_module: types.ModuleType, +) -> None: + """Mutable source errors or summaries could silently rewrite retained evidence.""" + snapshot = benchmark_module.HostSnapshot(source_errors={"proc": "missing"}) + phase = benchmark_module.PhaseReport( + "phase", + benchmark_module.Topology(1, 1, 1), + benchmark_module.Topology(1, 1, 1), + summary={"count": 1}, + ) + + with pytest.raises(TypeError): + snapshot.source_errors["proc"] = "changed" + assert phase.summary is not None + with pytest.raises(TypeError): + phase.summary["count"] = 2 + assert benchmark_module._json_value(snapshot)["source_errors"] == { + "proc": "missing" + } + + +def test_probe_host_keeps_rlimit_when_cgroup_resolution_fails( + benchmark_module: types.ModuleType, +) -> None: + """A broken cgroup mount must not erase an independent descriptor limit.""" + files = complete_procfs_files() + files["/proc/self/mountinfo"] = "bad mountinfo\n" + snapshot = benchmark_module.probe_host(LiteralReader(files)) + + assert snapshot.nofile_soft_limit == 65_536 + assert "cgroup2" in snapshot.source_errors + + +def test_probe_host_marks_malformed_pressure_and_partial_reads( + benchmark_module: types.ModuleType, +) -> None: + """A partial cgroup fixture must retain every unavailable source explicitly.""" + files = complete_procfs_files() + files[ + "/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/bench.scope/memory.pressure" + ] = "some total=1\n" + del files[ + "/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/bench.scope/memory.max" + ] + snapshot = benchmark_module.probe_host(LiteralReader(files)) + + assert snapshot.memory_max_bytes is None + assert {"memory.max", "memory.pressure"} <= set(snapshot.source_errors) + + +@pytest.mark.parametrize( + "sample", + ( + {"duration_ns": 1, "accepted": True, "error": "failure", "verified": True}, + {"duration_ns": 1, "accepted": True, "verified": False}, + ), +) +def test_validate_report_rejects_unverified_accepted_samples( + benchmark_module: types.ModuleType, sample: dict[str, t.Any] +) -> None: + """An accepted timing without verified success is not benchmark evidence.""" + report = completed_report(benchmark_module) + phase = dataclasses.replace( + report.phases[0], samples=(benchmark_module.RawSample(**sample),) + ) + with pytest.raises(ValueError, match="verified"): + benchmark_module.validate_report(dataclasses.replace(report, phases=(phase,))) + + +def test_validate_report_rejects_missing_or_mismatched_observed_topology( + benchmark_module: types.ModuleType, +) -> None: + """A completed phase requires an exact observed topology check.""" + report = completed_report(benchmark_module) + for observed in (None, benchmark_module.Topology(1, 1, 1)): + phase = dataclasses.replace(report.phases[0], observed_topology=observed) + with pytest.raises(ValueError, match="observed topology"): + benchmark_module.validate_report( + dataclasses.replace(report, phases=(phase,)) + ) + + def test_validate_report_recomputes_only_accepted_samples( benchmark_module: types.ModuleType, ) -> None: @@ -320,9 +432,19 @@ def test_validate_report_requires_unattempted_shapes_after_cutoff( status="cutoff", maximum_completed=False, ramp=( - benchmark_module.RampStep("80x20x1", "completed"), - benchmark_module.RampStep("100x20x1", "cutoff"), - benchmark_module.RampStep("80x20x2", "completed"), + benchmark_module.RampStep( + benchmark_module.parse_topology("80x20x1"), "completed" + ), + benchmark_module.RampStep( + benchmark_module.parse_topology("100x20x1"), "cutoff" + ), + benchmark_module.RampStep( + benchmark_module.parse_topology("80x20x2"), "completed" + ), + ), + requested_shapes=tuple( + benchmark_module.parse_topology(shape) + for shape in ("80x20x1", "100x20x1", "80x20x2") ), ) From e56a3f277b1ee83cfc24010d8d1b8c4fc8de8d96 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 06:47:14 -0500 Subject: [PATCH 08/67] Bench(fix[model]): Validate ramp state why: Prevent contradictory ramp artifacts from being accepted as benchmark evidence. what: - Add explicit single, canonical, and custom ramp state - Bind terminal attempts to declared sequence outcomes - Cover contradictory and noncanonical ramp reports --- scripts/bench_orchestration.py | 17 ++++++++++- tests/test_bench_orchestration_script.py | 37 ++++++++++++++++++++++++ 2 files changed, 53 insertions(+), 1 deletion(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index a545213eb2..e8f4861896 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -649,6 +649,8 @@ class RunReport: Ordered ramp outcomes, if this is a ramp run. requested_shapes : tuple[Topology, ...] Exact selected ramp sequence, including valid custom smoke ramps. + ramp_kind : {"none", "canonical", "custom"} + Whether this record is a single run, canonical ramp, or declared custom ramp. guard_decision : GuardDecision | None Effective decision after an optional predictive override. original_guard_decision : GuardDecision | None @@ -667,6 +669,7 @@ class RunReport: maximum_completed: bool = False ramp: tuple[RampStep, ...] = () requested_shapes: tuple[Topology, ...] = () + ramp_kind: t.Literal["none", "canonical", "custom"] = "none" guard_decision: GuardDecision | None = None original_guard_decision: GuardDecision | None = None schema_version: int = 1 @@ -785,7 +788,19 @@ def validate_report(report: RunReport) -> None: ): message = "phase summary must match accepted samples" raise ValueError(message) - if len(report.ramp) != len(report.requested_shapes): + if report.ramp_kind == "none" and (report.ramp or report.requested_shapes): + message = "none ramp kind must not carry attempts or requested shapes" + raise ValueError(message) + if report.ramp_kind == "canonical" and report.requested_shapes != canonical_ramp(): + message = "canonical ramp kind requires canonical requested shapes" + raise ValueError(message) + if report.ramp_kind == "custom" and ( + not report.requested_shapes + or len(set(report.requested_shapes)) != len(report.requested_shapes) + ): + message = "custom ramp kind requires nonempty unique requested shapes" + raise ValueError(message) + if report.ramp_kind != "none" and len(report.ramp) != len(report.requested_shapes): message = "ramp attempts must match requested shapes cardinality" raise ValueError(message) terminal_seen = False diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index 27dfba0f8a..3a345c790e 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -431,6 +431,7 @@ def test_validate_report_requires_unattempted_shapes_after_cutoff( completed_report(benchmark_module), status="cutoff", maximum_completed=False, + ramp_kind="custom", ramp=( benchmark_module.RampStep( benchmark_module.parse_topology("80x20x1"), "completed" @@ -516,3 +517,39 @@ def test_plan_writes_original_predictive_decision_without_talking_to_tmux( payload["original_guard_decision"] == payload["guard_decision"] or payload["original_guard_decision"]["kind"] == "predictive_refusal" ) + + +def test_validate_report_rejects_contradictory_ramp_terminal_sequence( + benchmark_module: types.ModuleType, +) -> None: + """A ramp cannot report completion after recording a cutoff.""" + shapes = tuple(benchmark_module.Topology(1, 1, panes) for panes in (1, 2, 3)) + report = dataclasses.replace( + completed_report(benchmark_module), + status="cutoff", + maximum_completed=False, + ramp_kind="custom", + requested_shapes=shapes, + ramp=( + benchmark_module.RampStep(shapes[0], "completed"), + benchmark_module.RampStep(shapes[1], "cutoff", "pid_reserve"), + benchmark_module.RampStep(shapes[2], "completed"), + ), + ) + with pytest.raises(ValueError, match="not_attempted"): + benchmark_module.validate_report(report) + + +def test_validate_report_rejects_canonical_kind_with_custom_shapes( + benchmark_module: types.ModuleType, +) -> None: + """Canonical evidence must retain every specified canonical ramp shape.""" + shape = benchmark_module.Topology(1, 1, 1) + report = dataclasses.replace( + completed_report(benchmark_module), + ramp_kind="canonical", + requested_shapes=(shape,), + ramp=(benchmark_module.RampStep(shape, "completed"),), + ) + with pytest.raises(ValueError, match="canonical"): + benchmark_module.validate_report(report) From 57b03ae9079d52ff89a26531edcd4f6d70b08ad6 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 06:50:09 -0500 Subject: [PATCH 09/67] Bench(fix[model]): Enforce ramp transitions why: Report validation must reject contradictory runtime ramp data. what: - Validate ramp kinds and attempt status vocabulary - Enforce one terminal transition with matching later reasons - Add invalid discriminator coverage --- scripts/bench_orchestration.py | 37 ++++++++++++++++++++---- tests/test_bench_orchestration_script.py | 11 +++++++ 2 files changed, 42 insertions(+), 6 deletions(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index e8f4861896..623c594a58 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -788,6 +788,12 @@ def validate_report(report: RunReport) -> None: ): message = "phase summary must match accepted samples" raise ValueError(message) + ramp_kinds = {"none", "canonical", "custom"} + terminal_statuses = {"refused", "cutoff", "failed"} + attempt_statuses = {"completed", "not_attempted", *terminal_statuses} + if report.ramp_kind not in ramp_kinds: + message = "invalid ramp_kind" + raise ValueError(message) if report.ramp_kind == "none" and (report.ramp or report.requested_shapes): message = "none ramp kind must not carry attempts or requested shapes" raise ValueError(message) @@ -803,20 +809,39 @@ def validate_report(report: RunReport) -> None: if report.ramp_kind != "none" and len(report.ramp) != len(report.requested_shapes): message = "ramp attempts must match requested shapes cardinality" raise ValueError(message) - terminal_seen = False for shape, step in zip(report.requested_shapes, report.ramp, strict=True): if step.shape != shape: message = "ramp attempts must match requested shapes in order" raise ValueError(message) - if report.status == "completed" and step.status != "completed": - message = "completed report may contain completed ramp attempts only" + if step.status not in attempt_statuses: + message = "invalid ramp attempt status" raise ValueError(message) - if terminal_seen and (step.status != "not_attempted" or step.reason is None): + if ( + report.ramp_kind != "none" + and report.status == "completed" + and any(step.status != "completed" for step in report.ramp) + ): + message = "completed report may contain completed ramp attempts only" + raise ValueError(message) + if report.ramp_kind != "none" and report.status in terminal_statuses: + terminals = [step for step in report.ramp if step.status in terminal_statuses] + if len(terminals) != 1 or terminals[0].status != report.status: + message = "terminal report requires exactly one matching terminal attempt" + raise ValueError(message) + terminal_index = report.ramp.index(terminals[0]) + reason = terminals[0].reason + if ( + reason is None + or any(step.status != "completed" for step in report.ramp[:terminal_index]) + or any( + step.status != "not_attempted" or step.reason != reason + for step in report.ramp[terminal_index + 1 :] + ) + ): message = ( - "ramp shapes after terminal step must be not_attempted with reason" + "invalid terminal ramp sequence: later attempts must be not_attempted" ) raise ValueError(message) - terminal_seen = terminal_seen or step.status in {"refused", "failed", "cutoff"} maximum = Topology(100, 100, 4) if report.maximum_completed and ( report.status != "completed" diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index 3a345c790e..99226a8cfb 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -553,3 +553,14 @@ def test_validate_report_rejects_canonical_kind_with_custom_shapes( ) with pytest.raises(ValueError, match="canonical"): benchmark_module.validate_report(report) + + +def test_validate_report_rejects_invalid_runtime_ramp_kind( + benchmark_module: types.ModuleType, +) -> None: + """Deserialized report data must not bypass the Literal ramp vocabulary.""" + report = dataclasses.replace( + completed_report(benchmark_module), ramp_kind="invalid" + ) + with pytest.raises(ValueError, match="ramp_kind"): + benchmark_module.validate_report(report) From cc146b2c18be3b229e1a89d37a3a76192f650b6a Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 06:58:52 -0500 Subject: [PATCH 10/67] Bench(fix[model]): Complete model audit why: Runtime artifact types and generated API docs must reject invalid state and describe each callable accurately. what: - Validate report and guard decision discriminator vocabularies - Cover exact terminal ramp and shape declaration state - Add NumPy sections and doctests across benchmark callables --- scripts/bench_orchestration.py | 398 ++++++++++++++++++++++- tests/test_bench_orchestration_script.py | 111 +++++++ 2 files changed, 497 insertions(+), 12 deletions(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index 623c594a58..c3e2ecfb6f 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -49,6 +49,11 @@ def windows(self) -> int: >>> Topology(2, 3, 4).windows 6 + + Returns + ------- + int + Sessions multiplied by windows per session. """ return self.sessions * self.windows_per_session @@ -58,6 +63,11 @@ def panes(self) -> int: >>> Topology(2, 3, 4).panes 24 + + Returns + ------- + int + Total windows multiplied by panes per window. """ return self.windows * self.panes_per_window @@ -66,6 +76,11 @@ def __str__(self) -> str: >>> str(Topology(2, 3, 4)) '2x3x4' + + Returns + ------- + str + Lower-case ``SxWxP`` representation. """ return f"{self.sessions}x{self.windows_per_session}x{self.panes_per_window}" @@ -74,10 +89,52 @@ class Reader(t.Protocol): """Read host limits without binding the pure model to the live filesystem.""" def read_text(self, path: str) -> str: - """Return the decoded contents at ``path``.""" + """Return the decoded contents at ``path``. + + >>> reader: Reader = _DoctestReader({"/value": "contents"}) + >>> reader.read_text("/value") + 'contents' + + Parameters + ---------- + path : str + Absolute procfs or cgroup path to read. + + Returns + ------- + str + Decoded text from the requested path. + + Raises + ------ + OSError + If the host path cannot be read. + """ def getrlimit(self, kind: int) -> tuple[int, int]: - """Return the current process's soft and hard limit for ``kind``.""" + """Return the current process's soft and hard limit for ``kind``. + + >>> reader: Reader = _DoctestReader({}) + >>> reader.getrlimit(resource.RLIMIT_NOFILE) + (1, 1) + + Parameters + ---------- + kind : int + Platform resource-limit identifier. + + Returns + ------- + tuple[int, int] + Soft and hard limits for the requested resource. + + Raises + ------ + OSError + If the process resource limit cannot be read. + ValueError + If ``kind`` is not a recognized resource. + """ class ProcessReader: @@ -88,6 +145,23 @@ def read_text(self, path: str) -> str: >>> isinstance(ProcessReader().read_text("/proc/meminfo"), str) True + + Parameters + ---------- + path : str + Absolute procfs or cgroup path to read. + + Returns + ------- + str + UTF-8 text read from the host path. + + Raises + ------ + OSError + If the path cannot be read. + UnicodeError + If the file is not valid UTF-8 text. """ return pathlib.Path(path).read_text(encoding="utf-8") @@ -96,6 +170,21 @@ def getrlimit(self, kind: int) -> tuple[int, int]: >>> ProcessReader().getrlimit(resource.RLIMIT_NOFILE)[0] > 0 True + + Parameters + ---------- + kind : int + Platform resource-limit identifier. + + Returns + ------- + tuple[int, int] + Soft and hard limits for the requested resource. + + Raises + ------ + ValueError + If ``kind`` is not a recognized resource. """ return resource.getrlimit(kind) @@ -208,6 +297,15 @@ def _source_error(errors: dict[str, str], label: str, exc: Exception) -> None: >>> _source_error(errors, "pids.current", FileNotFoundError("missing")) >>> "pids.current" in errors True + + Parameters + ---------- + errors : dict[str, str] + Mutable destination for source-specific probe failures. + label : str + Stable source name stored in the error mapping. + exc : Exception + Failure whose type and message are retained. """ errors[label] = f"{type(exc).__name__}: {exc}" @@ -217,6 +315,16 @@ def _parse_meminfo(text: str) -> dict[str, int]: >>> _parse_meminfo("MemTotal: 2 kB\nMemAvailable: 1 kB\n") {'MemTotal': 2048, 'MemAvailable': 1024} + + Parameters + ---------- + text : str + Literal contents of ``/proc/meminfo``. + + Returns + ------- + dict[str, int] + Recognized memory fields converted from kibibytes to bytes. """ values: dict[str, int] = {} for line in text.splitlines(): @@ -236,6 +344,23 @@ def _unified_cgroup_path(cgroup: str, mountinfo: str) -> str: >>> _unified_cgroup_path("0::/scope\n", "1 0 0:1 / /cg rw - cgroup2 cgroup rw\n") '/cg/scope' + + Parameters + ---------- + cgroup : str + Literal contents of ``/proc/self/cgroup``. + mountinfo : str + Literal contents of ``/proc/self/mountinfo``. + + Returns + ------- + str + Absolute path to the process's unified-cgroup directory. + + Raises + ------ + ValueError + If the unified hierarchy or cgroup-v2 mount cannot be resolved. """ relative: str | None = None for line in cgroup.splitlines(): @@ -265,6 +390,20 @@ def _read_int(reader: Reader, path: str, errors: dict[str, str]) -> int | None: >>> _read_int(_DoctestReader({"/value": "7\n"}), "/value", {}) 7 + + Parameters + ---------- + reader : Reader + Injectable host-data source. + path : str + Absolute path containing an integer or ``max``. + errors : dict[str, str] + Mutable destination for a source-specific failure. + + Returns + ------- + int | None + Finite integer value, or ``None`` for unlimited or unreadable data. """ try: value = reader.read_text(path).strip() @@ -275,17 +414,66 @@ def _read_int(reader: Reader, path: str, errors: dict[str, str]) -> int | None: class _DoctestReader: - """Small in-memory reader used only by the executable helper example.""" + """Small in-memory reader used only by executable helper examples. + + Attributes + ---------- + values : dict[str, str] + Literal content keyed by absolute source path. + """ def __init__(self, values: dict[str, str]) -> None: + """Store literal host data for an executable example. + + >>> _DoctestReader({"/value": "7"}).values["/value"] + '7' + + Parameters + ---------- + values : dict[str, str] + Literal content keyed by absolute source path. + """ self.values = values def read_text(self, path: str) -> str: - """Return a configured path's content.""" + """Return a configured path's content. + + >>> _DoctestReader({"/value": "7"}).read_text("/value") + '7' + + Parameters + ---------- + path : str + Configured absolute source path. + + Returns + ------- + str + Literal content configured for ``path``. + + Raises + ------ + KeyError + If ``path`` was not configured. + """ return self.values[path] def getrlimit(self, kind: int) -> tuple[int, int]: - """Return a harmless finite descriptor limit.""" + """Return a harmless finite descriptor limit. + + >>> _DoctestReader({}).getrlimit(resource.RLIMIT_NOFILE) + (1, 1) + + Parameters + ---------- + kind : int + Resource identifier accepted for protocol compatibility. + + Returns + ------- + tuple[int, int] + Fixed soft and hard limits for examples. + """ return (1, 1) @@ -294,6 +482,21 @@ def _pressure_some_avg10(text: str) -> float | None: >>> _pressure_some_avg10("some avg10=0.25 avg60=0.1 total=2\n") 0.25 + + Parameters + ---------- + text : str + Literal contents of a cgroup pressure file. + + Returns + ------- + float | None + Ten-second ``some`` pressure average, or ``None`` when absent. + + Raises + ------ + ValueError + If the ``avg10`` value is not numeric. """ for line in text.splitlines(): fields = line.split() @@ -320,6 +523,16 @@ def probe_host(reader: Reader) -> HostSnapshot: ... })) >>> (snapshot.available_memory_bytes, snapshot.pids_max) (1024, 2) + + Parameters + ---------- + reader : Reader + Injectable source for procfs, cgroup, and process-limit data. + + Returns + ------- + HostSnapshot + Observed values plus source-specific errors for unavailable telemetry. """ errors: dict[str, str] = {} memory: dict[str, int] = {} @@ -382,6 +595,16 @@ def _default_pid_reserve(snapshot: HostSnapshot) -> int | None: >>> _default_pid_reserve(HostSnapshot(pids_max=10_000)) 1500 + + Parameters + ---------- + snapshot : HostSnapshot + Host observation containing the unified-cgroup PID limit. + + Returns + ------- + int | None + Larger of 1,024 processes and 15 percent of the limit, or ``None``. """ if snapshot.pids_max is None: return None @@ -393,6 +616,16 @@ def _default_memory_floor(snapshot: HostSnapshot) -> int | None: >>> _default_memory_floor(HostSnapshot(physical_memory_bytes=10 * 2**30)) 4294967296 + + Parameters + ---------- + snapshot : HostSnapshot + Host observation containing detected physical memory. + + Returns + ------- + int | None + Larger of 4 GiB and 15 percent of physical memory, or ``None``. """ if snapshot.physical_memory_bytes is None: return None @@ -410,6 +643,20 @@ def predict_resources( >>> decision = predict_resources(Topology(1, 1, 1), snapshot) >>> decision.kind 'ok' + + Parameters + ---------- + topology : Topology + Proposed active tmux hierarchy. + snapshot : HostSnapshot + Host resource observation used for admission. + policy : ResourcePolicy | None + Explicit resource thresholds, or documented dynamic defaults. + + Returns + ------- + GuardDecision + Predictive admission or forceable refusal with supporting evidence. """ policy = policy or ResourcePolicy() reserve = ( @@ -475,6 +722,28 @@ def check_runtime_guard( >>> check_runtime_guard(HostSnapshot(pids_current=2000, pids_max=2000)).kind 'runtime_cutoff' + + Parameters + ---------- + snapshot : HostSnapshot + Live host resource observation. + policy : ResourcePolicy | None + Explicit resource thresholds, or documented dynamic defaults. + processes_alive : bool + Whether every benchmark-owned pane and fuzzer process is alive. + topology_verified : bool + Whether the observed tmux hierarchy exactly matches the request. + watchdog_ok : bool + Whether the active phase remains within its progress deadline. + cleanup_complete : bool + Whether terminal cleanup removed every benchmark-owned resource. + force_extreme : bool + Predictive override flag, accepted here but never applied at runtime. + + Returns + ------- + GuardDecision + Non-forceable cutoff for the first failed runtime guard, otherwise success. """ del force_extreme policy = policy or ResourcePolicy() @@ -680,6 +949,16 @@ def _json_value(value: object) -> object: >>> _json_value(Topology(1, 2, 3)) {'sessions': 1, 'windows_per_session': 2, 'panes_per_window': 3} + + Parameters + ---------- + value : object + Dataclass, tuple, mapping, or already JSON-native scalar to convert. + + Returns + ------- + object + Equivalent value composed from JSON-native containers and scalars. """ if dataclasses.is_dataclass(value) and not isinstance(value, type): return { @@ -718,6 +997,13 @@ def write_json_atomic( Injectable durability boundary for the temporary file and parent directory. replace : collections.abc.Callable[[str | pathlib.Path, str | pathlib.Path], None] Injectable atomic replacement boundary. + + Raises + ------ + OSError + If directory creation, writing, synchronization, or replacement fails. + TypeError + If ``value`` cannot be serialized to JSON. """ path.parent.mkdir(parents=True, exist_ok=True) encoded = ( @@ -755,10 +1041,29 @@ def validate_report(report: RunReport) -> None: >>> report = RunReport(Topology(1, 1, 1)) >>> validate_report(report) + + Parameters + ---------- + report : RunReport + Immutable run artifact to validate before publication. + + Raises + ------ + ValueError + If a discriminator, phase, cleanup, ramp, or maximum claim is inconsistent. """ if report.schema_version != 1: message = "unsupported report schema_version" raise ValueError(message) + report_statuses = {"in_progress", "completed", "refused", "failed", "cutoff"} + if report.status not in report_statuses: + message = "invalid report status" + raise ValueError(message) + guard_kinds = {"ok", "predictive_refusal", "runtime_cutoff"} + for decision in (report.guard_decision, report.original_guard_decision): + if decision is not None and decision.kind not in guard_kinds: + message = "invalid guard decision kind" + raise ValueError(message) terminal = {"completed", "refused", "failed", "cutoff"} if report.status in terminal and not report.cleanup.complete: message = "terminal report requires complete cleanup" @@ -860,6 +1165,18 @@ def _forced_decision(decision: GuardDecision, force_extreme: bool) -> GuardDecis ... ) >>> _forced_decision(original, True).allowed True + + Parameters + ---------- + decision : GuardDecision + Original predictive admission result. + force_extreme : bool + Whether to override a forceable predictive refusal. + + Returns + ------- + GuardDecision + Effective decision, preserving runtime cutoffs and non-forceable results. """ if force_extreme and decision.kind == "predictive_refusal" and decision.forceable: return dataclasses.replace( @@ -878,6 +1195,20 @@ def plan_payload( >>> plan_payload(Topology(1, 1, 1), HostSnapshot())["predicted_pane_processes"] 1 + + Parameters + ---------- + topology : Topology + Proposed active tmux hierarchy. + snapshot : HostSnapshot + Host observation used for predictive admission. + force_extreme : bool + Whether to override a forceable predictive refusal in the effective result. + + Returns + ------- + dict[str, object] + JSON-native plan containing original and effective guard decisions. """ original = predict_resources(topology, snapshot) decision = _forced_decision(original, force_extreme) @@ -906,6 +1237,27 @@ def run_plan(shape: str, output: pathlib.Path | None, force_extreme: bool) -> in ... result = run_plan("1x1x1", None, False) >>> result 0 + + Parameters + ---------- + shape : str + Topology in lower-case ``SxWxP`` notation. + output : pathlib.Path | None + Optional destination for atomic JSON plan evidence. + force_extreme : bool + Whether to override a forceable predictive refusal in the displayed plan. + + Returns + ------- + int + Zero after the plan is printed and optional evidence is written. + + Raises + ------ + ValueError + If ``shape`` is malformed or has a nonpositive dimension. + OSError + If the optional output path cannot be written. """ from rich.console import Console from rich.table import Table @@ -970,6 +1322,11 @@ def canonical_ramp() -> tuple[Topology, ...]: >>> tuple(str(shape) for shape in canonical_ramp())[:3] ('80x20x1', '100x20x1', '80x20x2') + + Returns + ------- + tuple[Topology, ...] + Exact canonical progression ordered by expected live-pane pressure. """ return ( Topology(80, 20, 1), @@ -1012,17 +1369,15 @@ def summarize_ns(samples: t.Sequence[int]) -> dict[str, int | float]: raise ValueError(message) ordered = sorted(samples) - def percentile(percent: float) -> int: - return ordered[math.ceil(percent * len(ordered)) - 1] - + count = len(ordered) return { - "count": len(ordered), + "count": count, "min_ns": ordered[0], "mean_ns": statistics.mean(ordered), "median_ns": statistics.median(ordered), - "p90_ns": percentile(0.90), - "p95_ns": percentile(0.95), - "p99_ns": percentile(0.99), + "p90_ns": ordered[math.ceil(0.90 * count) - 1], + "p95_ns": ordered[math.ceil(0.95 * count) - 1], + "p99_ns": ordered[math.ceil(0.99 * count) - 1], "max_ns": ordered[-1], } @@ -1036,6 +1391,25 @@ def main(argv: t.Sequence[str] | None = None) -> int: ... result = main(["plan", "--shape", "1x1x1"]) >>> result 0 + + Parameters + ---------- + argv : collections.abc.Sequence[str] | None + Explicit command arguments, or process arguments when omitted. + + Returns + ------- + int + Zero after the selected command completes. + + Raises + ------ + SystemExit + If command-line arguments are invalid. + ValueError + If the selected plan topology is malformed or nonpositive. + OSError + If the selected plan output cannot be written. """ parser = argparse.ArgumentParser(description=__doc__) commands = parser.add_subparsers(dest="command", required=True) diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index 99226a8cfb..55d896d432 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -564,3 +564,114 @@ def test_validate_report_rejects_invalid_runtime_ramp_kind( ) with pytest.raises(ValueError, match="ramp_kind"): benchmark_module.validate_report(report) + + +def test_validate_report_rejects_invalid_runtime_report_status( + benchmark_module: types.ModuleType, +) -> None: + """Deserialized lifecycle state must remain inside the report vocabulary.""" + report = dataclasses.replace(completed_report(benchmark_module), status="invalid") + + with pytest.raises(ValueError, match="status"): + benchmark_module.validate_report(report) + + +@pytest.mark.parametrize("field", ("guard_decision", "original_guard_decision")) +def test_validate_report_rejects_invalid_runtime_guard_kind( + benchmark_module: types.ModuleType, field: str +) -> None: + """Deserialized guard evidence must remain inside its Literal vocabulary.""" + invalid_guard = benchmark_module.GuardDecision( + True, + "invalid", + None, + None, + None, + False, + benchmark_module.HostSnapshot(), + ) + report = dataclasses.replace( + completed_report(benchmark_module), **{field: invalid_guard} + ) + + with pytest.raises(ValueError, match="guard decision kind"): + benchmark_module.validate_report(report) + + +@pytest.mark.parametrize("status", ("refused", "failed", "cutoff")) +def test_validate_report_accepts_exact_terminal_ramp_sequence( + benchmark_module: types.ModuleType, status: str +) -> None: + """Each terminal ramp state has one matching stop and one shared reason.""" + shapes = tuple(benchmark_module.Topology(1, 1, panes) for panes in (1, 2, 3)) + reason = f"{status}_reason" + report = dataclasses.replace( + completed_report(benchmark_module), + status=status, + maximum_completed=False, + ramp_kind="custom", + requested_shapes=shapes, + ramp=( + benchmark_module.RampStep(shapes[0], "completed"), + benchmark_module.RampStep(shapes[1], status, reason), + benchmark_module.RampStep(shapes[2], "not_attempted", reason), + ), + ) + + benchmark_module.validate_report(report) + + +def test_validate_report_rejects_invalid_runtime_attempt_status( + benchmark_module: types.ModuleType, +) -> None: + """Deserialized attempt state must remain inside the ramp vocabulary.""" + shape = benchmark_module.Topology(1, 1, 1) + report = dataclasses.replace( + completed_report(benchmark_module), + maximum_completed=False, + ramp_kind="custom", + requested_shapes=(shape,), + ramp=(benchmark_module.RampStep(shape, "invalid"),), + ) + + with pytest.raises(ValueError, match="attempt status"): + benchmark_module.validate_report(report) + + +def test_validate_report_rejects_none_ramp_kind_with_shapes( + benchmark_module: types.ModuleType, +) -> None: + """Single-run evidence cannot smuggle in an undeclared ramp sequence.""" + shape = benchmark_module.Topology(1, 1, 1) + report = dataclasses.replace( + completed_report(benchmark_module), + maximum_completed=False, + requested_shapes=(shape,), + ramp=(benchmark_module.RampStep(shape, "completed"),), + ) + + with pytest.raises(ValueError, match="none ramp kind"): + benchmark_module.validate_report(report) + + +@pytest.mark.parametrize("duplicate", (False, True)) +def test_validate_report_rejects_invalid_custom_shape_declaration( + benchmark_module: types.ModuleType, duplicate: bool +) -> None: + """A custom ramp must declare at least one shape and cannot repeat one.""" + shape = benchmark_module.Topology(1, 1, 1) + requested_shapes = (shape, shape) if duplicate else () + ramp = tuple( + benchmark_module.RampStep(requested, "completed") + for requested in requested_shapes + ) + report = dataclasses.replace( + completed_report(benchmark_module), + maximum_completed=False, + ramp_kind="custom", + requested_shapes=requested_shapes, + ramp=ramp, + ) + + with pytest.raises(ValueError, match="custom ramp kind"): + benchmark_module.validate_report(report) From cfd2e2e7484438325b41838cda9a3b16e49fcc97 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 07:06:55 -0500 Subject: [PATCH 11/67] Bench(fix[model]): Enforce ramp checkpoints why: In-progress ramps could retain terminal outcomes, completed work past pending entries, or reasons on unattempted shapes. what: - Enforce the completed-prefix and pending-suffix checkpoint grammar - Require terminal attempts to finalize the report lifecycle - Cover malformed and valid in-progress ramp sequences --- scripts/bench_orchestration.py | 16 ++- tests/test_bench_orchestration_script.py | 141 +++++++++++++++++++++++ 2 files changed, 156 insertions(+), 1 deletion(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index c3e2ecfb6f..66d888ce66 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -1037,7 +1037,7 @@ def write_json_atomic( def validate_report(report: RunReport) -> None: - """Reject internally inconsistent or incomplete terminal benchmark evidence. + """Reject internally inconsistent benchmark evidence. >>> report = RunReport(Topology(1, 1, 1)) >>> validate_report(report) @@ -1121,6 +1121,20 @@ def validate_report(report: RunReport) -> None: if step.status not in attempt_statuses: message = "invalid ramp attempt status" raise ValueError(message) + if report.ramp_kind != "none" and report.status == "in_progress": + pending = False + for step in report.ramp: + if step.status in terminal_statuses: + message = "in-progress ramp cannot contain a terminal attempt" + raise ValueError(message) + if step.status == "not_attempted": + if step.reason is not None: + message = "in-progress pending attempt reason must be None" + raise ValueError(message) + pending = True + elif pending: + message = "in-progress ramp requires a completed prefix" + raise ValueError(message) if ( report.ramp_kind != "none" and report.status == "completed" diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index 55d896d432..28724045ef 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -675,3 +675,144 @@ def test_validate_report_rejects_invalid_custom_shape_declaration( with pytest.raises(ValueError, match="custom ramp kind"): benchmark_module.validate_report(report) + + +def test_validate_report_rejects_in_progress_ramp_with_cutoff_attempt( + benchmark_module: types.ModuleType, +) -> None: + """A runtime cutoff requires finalizing the report at the same checkpoint.""" + shapes = ( + benchmark_module.Topology(1, 1, 1), + benchmark_module.Topology(1, 1, 2), + benchmark_module.Topology(1, 1, 3), + ) + report = benchmark_module.RunReport( + requested_topology=shapes[-1], + status="in_progress", + ramp_kind="custom", + requested_shapes=shapes, + ramp=( + benchmark_module.RampStep(shapes[0], "completed"), + benchmark_module.RampStep(shapes[1], "cutoff", "pid_reserve"), + benchmark_module.RampStep(shapes[2], "not_attempted", "pid_reserve"), + ), + ) + + with pytest.raises(ValueError, match=r"in-progress.*terminal"): + benchmark_module.validate_report(report) + + +def test_validate_report_rejects_in_progress_ramp_with_different_terminals( + benchmark_module: types.ModuleType, +) -> None: + """An unfinished report cannot retain competing terminal outcomes.""" + shapes = ( + benchmark_module.Topology(1, 1, 1), + benchmark_module.Topology(1, 1, 2), + benchmark_module.Topology(1, 1, 3), + ) + report = benchmark_module.RunReport( + requested_topology=shapes[-1], + status="in_progress", + ramp_kind="custom", + requested_shapes=shapes, + ramp=( + benchmark_module.RampStep(shapes[0], "refused", "pid_reserve"), + benchmark_module.RampStep(shapes[1], "failed", "tmux_exit"), + benchmark_module.RampStep(shapes[2], "not_attempted"), + ), + ) + + with pytest.raises(ValueError, match=r"in-progress.*terminal"): + benchmark_module.validate_report(report) + + +def test_validate_report_rejects_in_progress_ramp_completed_after_pending( + benchmark_module: types.ModuleType, +) -> None: + """Completed ramp work must remain a prefix of an unfinished checkpoint.""" + shapes = ( + benchmark_module.Topology(1, 1, 1), + benchmark_module.Topology(1, 1, 2), + benchmark_module.Topology(1, 1, 3), + ) + report = benchmark_module.RunReport( + requested_topology=shapes[-1], + status="in_progress", + ramp_kind="custom", + requested_shapes=shapes, + ramp=( + benchmark_module.RampStep(shapes[0], "completed"), + benchmark_module.RampStep(shapes[1], "not_attempted"), + benchmark_module.RampStep(shapes[2], "completed"), + ), + ) + + with pytest.raises(ValueError, match="completed prefix"): + benchmark_module.validate_report(report) + + +def test_validate_report_rejects_in_progress_ramp_pending_reason( + benchmark_module: types.ModuleType, +) -> None: + """Pending attempts use ``None`` until a terminal reason exists.""" + shapes = benchmark_module.canonical_ramp() + report = benchmark_module.RunReport( + requested_topology=shapes[-1], + status="in_progress", + ramp_kind="canonical", + requested_shapes=shapes, + ramp=tuple( + benchmark_module.RampStep( + shape, + "not_attempted", + "waiting" if index == 0 else None, + ) + for index, shape in enumerate(shapes) + ), + ) + + with pytest.raises(ValueError, match=r"pending.*reason"): + benchmark_module.validate_report(report) + + +def test_validate_report_accepts_initial_in_progress_ramp( + benchmark_module: types.ModuleType, +) -> None: + """A canonical ramp may checkpoint before attempting its first shape.""" + shapes = benchmark_module.canonical_ramp() + report = benchmark_module.RunReport( + requested_topology=shapes[-1], + status="in_progress", + ramp_kind="canonical", + requested_shapes=shapes, + ramp=tuple( + benchmark_module.RampStep(shape, "not_attempted") for shape in shapes + ), + ) + + benchmark_module.validate_report(report) + + +def test_validate_report_accepts_in_progress_completed_prefix( + benchmark_module: types.ModuleType, +) -> None: + """An unfinished custom ramp may retain completed work before pending shapes.""" + shapes = ( + benchmark_module.Topology(1, 1, 1), + benchmark_module.Topology(1, 1, 2), + benchmark_module.Topology(1, 1, 3), + ) + report = benchmark_module.RunReport( + requested_topology=shapes[-1], + status="in_progress", + ramp_kind="custom", + requested_shapes=shapes, + ramp=( + benchmark_module.RampStep(shapes[0], "completed"), + benchmark_module.RampStep(shapes[1], "completed"), + benchmark_module.RampStep(shapes[2], "not_attempted"), + ), + ) + + benchmark_module.validate_report(report) From 995649aebf44eec81ef566a15a18a7056bb9e7a6 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 07:37:13 -0500 Subject: [PATCH 12/67] Bench(feat[topology]): Build active servers why: Exercise every engine lane against an exact active tmux topology without contacting ambient servers or leaking owned processes. what: - Build and stabilize one typed WorkspaceSet across all four lanes - Verify exact snapshots, activity epochs, and process identities - Clean fuzzer, engine, server, followers, socket, and scratch state --- scripts/bench_orchestration.py | 1724 ++++++++++++++++++++++ tests/test_bench_orchestration_script.py | 166 +++ 2 files changed, 1890 insertions(+) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index 66d888ce66..0a00910495 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -12,18 +12,64 @@ from __future__ import annotations import argparse +import asyncio import collections.abc as cabc +import contextlib import dataclasses +import enum import json import math import os import pathlib import resource +import shlex +import shutil +import signal import statistics +import subprocess +import sys import tempfile +import time import types import typing as t +if t.TYPE_CHECKING: + from libtmux.experimental.engines.base import AsyncTmuxEngine, TmuxEngine + from libtmux.experimental.models import ( + ClientSnapshot, + PaneSnapshot, + SessionSnapshot, + WindowSnapshot, + ) + from libtmux.experimental.workspace import WorkspaceSet + from libtmux.server import Server + + +class EngineLane(str, enum.Enum): + """Transport family used for one live benchmark server. + + Examples + -------- + >>> EngineLane("control") is EngineLane.CONTROL + True + """ + + SUBPROCESS = "subprocess" + CONTROL = "control" + + +class ExecutionMode(str, enum.Enum): + """Synchronous or asynchronous operation execution. + + Examples + -------- + >>> ExecutionMode("async") is ExecutionMode.ASYNC + True + """ + + SYNC = "sync" + ASYNC = "async" + @dataclasses.dataclass(frozen=True) class Topology: @@ -877,6 +923,247 @@ class CleanupReport: errors: tuple[str, ...] = () +@dataclasses.dataclass(frozen=True) +class ProcessIdentity: + """One benchmark-owned process protected against PID reuse. + + Attributes + ---------- + role : str + Stable owner label such as ``fuzzer``, ``server``, or ``pane``. + pid : int + Positive process identifier. + start_time : int + Linux procfs start-time tick from field 22 of ``/proc/PID/stat``. + + Examples + -------- + >>> ProcessIdentity("pane", 42, 100).role + 'pane' + """ + + role: str + pid: int + start_time: int + + +@dataclasses.dataclass(frozen=True) +class TopologyTotals: + """Flat session, window, and pane totals used in mismatch evidence. + + Attributes + ---------- + sessions : int + Number of observed or requested sessions. + windows : int + Number of observed or requested windows. + panes : int + Number of observed or requested panes. + + Examples + -------- + >>> TopologyTotals.from_topology(Topology(2, 3, 4)) + TopologyTotals(sessions=2, windows=6, panes=24) + """ + + sessions: int + windows: int + panes: int + + @classmethod + def from_topology(cls, topology: Topology) -> TopologyTotals: + """Return the exact flat totals for a requested topology. + + >>> TopologyTotals.from_topology(Topology(1, 2, 3)).panes + 6 + + Parameters + ---------- + topology : Topology + Requested hierarchy dimensions. + + Returns + ------- + TopologyTotals + Sessions, windows, and panes after multiplication. + """ + return cls(topology.sessions, topology.windows, topology.panes) + + +@dataclasses.dataclass(frozen=True) +class TopologySnapshot: + """Concrete typed snapshots returned by all three list operations. + + Attributes + ---------- + sessions : tuple[SessionSnapshot, ...] + Rows returned by typed ``ListSessions``. + windows : tuple[WindowSnapshot, ...] + Rows returned by typed ``ListWindows(all_windows=True)``. + panes : tuple[PaneSnapshot, ...] + Rows returned by typed ``ListPanes(all_panes=True)``. + + Examples + -------- + >>> TopologySnapshot((), (), ()).totals + TopologyTotals(sessions=0, windows=0, panes=0) + """ + + sessions: tuple[SessionSnapshot, ...] + windows: tuple[WindowSnapshot, ...] + panes: tuple[PaneSnapshot, ...] + + @property + def totals(self) -> TopologyTotals: + """Return flat counts without contacting tmux. + + >>> TopologySnapshot((), (), ()).totals.panes + 0 + + Returns + ------- + TopologyTotals + Counts derived from the stored immutable rows. + """ + return TopologyTotals(len(self.sessions), len(self.windows), len(self.panes)) + + +class TopologyVerificationError(RuntimeError): + """Exact topology verification failed. + + Examples + -------- + >>> error = TopologyVerificationError( + ... TopologyTotals(2, 4, 8), TopologyTotals(2, 4, 7), "pane count" + ... ) + >>> error.requested.panes, error.observed.panes + (8, 7) + """ + + def __init__( + self, + requested: TopologyTotals, + observed: TopologyTotals, + detail: str, + ) -> None: + """Retain both totals and the failed invariant. + + >>> str(TopologyVerificationError( + ... TopologyTotals(1, 1, 1), TopologyTotals(1, 1, 0), "missing pane" + ... )) + 'topology verification failed: missing pane; requested=1/1/1 observed=1/1/0' + + Parameters + ---------- + requested : TopologyTotals + Exact totals required by the run. + observed : TopologyTotals + Flat totals returned by the live server. + detail : str + First failed topology invariant. + """ + self.requested = requested + self.observed = observed + self.detail = detail + super().__init__( + "topology verification failed: " + f"{detail}; requested={requested.sessions}/{requested.windows}/" + f"{requested.panes} observed={observed.sessions}/{observed.windows}/" + f"{observed.panes}" + ) + + +@dataclasses.dataclass +class RunContext: + """Single owner for one live topology and all disposable resources. + + Attributes + ---------- + topology : Topology + Requested hierarchy. + lane : EngineLane + Selected transport family. + mode : ExecutionMode + Sync or async dispatch mode. + run_id : str + Identity carried by fuzzer control markers. + scratch : pathlib.Path + Exclusive directory removed during cleanup. + socket_path : pathlib.Path + Explicit isolated tmux socket path. + server : Server + Classic server value used only to bind engines to the socket. + engine : TmuxEngine or AsyncTmuxEngine + Active transport engine. + fuzzer : subprocess.Popen[bytes] + Paused central stream generator. + streams : tuple[pathlib.Path, ...] + Precreated activity streams in mode order. + delayed_ordinal : int + Global pane ordinal assigned the delayed-match stream. + expected_session_names : tuple[str, ...] + Exact declared session names. + expected_window_names : tuple[str, ...] + Exact declared window names. + setup_duration_ns : int + Timed construction duration excluding activity stabilization. + processes : tuple[ProcessIdentity, ...] + Fuzzer, server, and pane-follower identities. + session_ids : tuple[str, ...] + Verified stable session identifiers. + window_ids : tuple[str, ...] + Verified stable window identifiers. + pane_ids : tuple[str, ...] + Verified stable pane identifiers. + delayed_pane_id : str or None + Verified pane following the unique delayed stream. + topology_verified : bool + Whether exact live verification succeeded. + activity_epoch : int or None + Released run-scoped activity epoch. + activity_marker : str or None + Exact marker required in every pane. + activity_pane_ids : tuple[str, ...] + Panes that captured the released marker. + heartbeat_epoch : int + Last monotonic fuzzer heartbeat epoch observed. + ambient_tmux_environment : tuple[str | None, str | None] + Original ``TMUX`` and ``TMUX_PANE`` values restored after cleanup. + + Examples + -------- + >>> required = {"session_ids", "pane_ids", "processes", "activity_epoch"} + >>> required <= {field.name for field in dataclasses.fields(RunContext)} + True + """ + + topology: Topology + lane: EngineLane + mode: ExecutionMode + run_id: str + scratch: pathlib.Path + socket_path: pathlib.Path + server: Server + engine: TmuxEngine | AsyncTmuxEngine + fuzzer: subprocess.Popen[bytes] + streams: tuple[pathlib.Path, ...] + delayed_ordinal: int + expected_session_names: tuple[str, ...] + expected_window_names: tuple[str, ...] + setup_duration_ns: int + processes: tuple[ProcessIdentity, ...] + session_ids: tuple[str, ...] = () + window_ids: tuple[str, ...] = () + pane_ids: tuple[str, ...] = () + delayed_pane_id: str | None = None + topology_verified: bool = False + activity_epoch: int | None = None + activity_marker: str | None = None + activity_pane_ids: tuple[str, ...] = () + heartbeat_epoch: int = -1 + ambient_tmux_environment: tuple[str | None, str | None] = (None, None) + + @dataclasses.dataclass(frozen=True) class RampStep: """Outcome recorded for one canonical ramp shape. @@ -1295,6 +1582,1443 @@ def run_plan(shape: str, output: pathlib.Path | None, force_extreme: bool) -> in return 0 +def _process_start_time(pid: int) -> int | None: + """Read one Linux process start-time tick without following a PID blindly. + + >>> value = _process_start_time(os.getpid()) + >>> isinstance(value, int) and value > 0 + True + + Parameters + ---------- + pid : int + Positive process identifier. + + Returns + ------- + int | None + Procfs field 22, or ``None`` if the identity is absent or unreadable. + """ + if pid <= 0: + return None + try: + stat = pathlib.Path(f"/proc/{pid}/stat").read_text(encoding="utf-8") + close = stat.rindex(")") + fields = stat[close + 2 :].split() + return int(fields[19]) + except (FileNotFoundError, OSError, ValueError, IndexError): + return None + + +def _record_process(role: str, pid: int) -> ProcessIdentity: + """Capture a positive PID and its current procfs start time. + + >>> _record_process("self", os.getpid()).role + 'self' + + Parameters + ---------- + role : str + Stable resource-owner label. + pid : int + Process identifier to bind. + + Returns + ------- + ProcessIdentity + Identity safe to compare before later escalation. + + Raises + ------ + RuntimeError + If procfs cannot prove the process identity. + """ + start_time = _process_start_time(pid) + if start_time is None: + message = f"cannot record {role} process identity for pid {pid}" + raise RuntimeError(message) + return ProcessIdentity(role, pid, start_time) + + +def process_identity_matches(identity: ProcessIdentity) -> bool: + """Return whether a PID still names the exact process that was recorded. + + >>> current = _record_process("self", os.getpid()) + >>> process_identity_matches(current) + True + >>> process_identity_matches(dataclasses.replace(current, start_time=-1)) + False + + Parameters + ---------- + identity : ProcessIdentity + PID and start time captured while the process was owned by this run. + + Returns + ------- + bool + True only while both PID and procfs start time still match. + """ + return _process_start_time(identity.pid) == identity.start_time + + +def _stream_name(ordinal: int, delayed_ordinal: int) -> str: + """Return the Task 1 stream name for one global stable pane ordinal. + + >>> [_stream_name(index, 2) for index in range(4)] + ['editor', 'dev-server', 'delayed-match', 'installer'] + + Parameters + ---------- + ordinal : int + Zero-based pane position across every session and window. + delayed_ordinal : int + Position reserved for the unique delayed stream. + + Returns + ------- + str + One fuzzer stream basename without its ``.log`` suffix. + """ + if ordinal == delayed_ordinal: + return "delayed-match" + shared = ("editor", "dev-server", "installer") + compacted = ordinal - int(ordinal > delayed_ordinal) + return shared[compacted % len(shared)] + + +def _tail_command(stream: pathlib.Path) -> str: + """Render the portable follower command as one shell-safe string. + + >>> _tail_command(pathlib.Path("a stream.log")) + "exec tail -n 0 -f 'a stream.log'" + + Parameters + ---------- + stream : pathlib.Path + Precreated append-only activity file. + + Returns + ------- + str + ``exec tail -n 0 -f`` plus a safely quoted path. + """ + return shlex.join(("exec", "tail", "-n", "0", "-f", str(stream))) + + +def _session_name(run_id: str, index: int) -> str: + """Return one globally distinct, stable benchmark session name. + + >>> _session_name("run-7", 2) + 'bench-run-7-s002' + + Parameters + ---------- + run_id : str + Validated run identity. + index : int + Zero-based session ordinal. + + Returns + ------- + str + Exact tmux session name. + """ + return f"bench-{run_id}-s{index:03d}" + + +def _window_name(run_id: str, session_index: int, window_index: int) -> str: + """Return one globally distinct, stable benchmark window name. + + >>> _window_name("run-7", 2, 3) + 'bench-run-7-s002-w003' + + Parameters + ---------- + run_id : str + Validated run identity. + session_index : int + Zero-based session ordinal. + window_index : int + Zero-based window ordinal within the session. + + Returns + ------- + str + Exact tmux window name. + """ + return f"{_session_name(run_id, session_index)}-w{window_index:03d}" + + +def _only_control_client_name(clients: tuple[ClientSnapshot, ...]) -> str: + """Return the sole client attached to an isolated control server. + + >>> try: + ... _only_control_client_name(()) + ... except RuntimeError as error: + ... print(error) + isolated control server must have one attached client + + Parameters + ---------- + clients : tuple[ClientSnapshot, ...] + Concrete rows returned by typed ``ListClients``. + + Returns + ------- + str + Exact tmux client name accepted by ``SwitchClient``. + + Raises + ------ + RuntimeError + If bootstrap did not leave exactly one persistent control client. + """ + if len(clients) != 1: + message = "isolated control server must have one attached client" + raise RuntimeError(message) + return clients[0].name + + +def build_workspaces( + topology: Topology, + streams_dir: pathlib.Path, + run_id: str, + *, + delayed_ordinal: int, +) -> WorkspaceSet: + """Declare one workspace per session with active commands in every pane. + + The returned set is one compilation unit. Every first pane receives its + command through ``Window.window_shell`` and every split receives the same + command through ``Pane.shell``. + + Examples + -------- + >>> with tempfile.TemporaryDirectory() as directory: + ... streams = pathlib.Path(directory) + ... for name in ("editor", "dev-server", "installer", "delayed-match"): + ... (streams / f"{name}.log").touch() + ... workspace_set = build_workspaces( + ... Topology(1, 1, 2), streams, "run-7", delayed_ordinal=1 + ... ) + ... len(workspace_set.workspaces[0].windows[0].panes) + 2 + + Parameters + ---------- + topology : Topology + Positive hierarchy to declare. + streams_dir : pathlib.Path + Directory containing all four precreated Task 1 streams. + run_id : str + Run identity used in stable tmux names. + delayed_ordinal : int + Unique global pane ordinal assigned the delayed stream. + + Returns + ------- + WorkspaceSet + All sessions wrapped in one declarative collection. + + Raises + ------ + ValueError + If the delayed ordinal is outside the requested pane range. + FileNotFoundError + If any selected stream was not precreated by the fuzzer. + """ + from libtmux.experimental.workspace import Pane, Window, Workspace, WorkspaceSet + + if not 0 <= delayed_ordinal < topology.panes: + message = "delayed pane ordinal must identify a requested pane" + raise ValueError(message) + workspaces: list[Workspace] = [] + ordinal = 0 + for session_index in range(topology.sessions): + windows: list[Window] = [] + for window_index in range(topology.windows_per_session): + panes: list[Pane] = [] + commands: list[str] = [] + for _pane_index in range(topology.panes_per_window): + stream = streams_dir / f"{_stream_name(ordinal, delayed_ordinal)}.log" + if not stream.is_file(): + raise FileNotFoundError(stream) + command = _tail_command(stream) + commands.append(command) + panes.append(Pane(shell=command)) + ordinal += 1 + windows.append( + Window( + name=_window_name(run_id, session_index, window_index), + window_shell=commands[0], + panes=tuple(panes), + ) + ) + workspaces.append( + Workspace( + name=_session_name(run_id, session_index), + windows=tuple(windows), + ) + ) + return WorkspaceSet(workspaces) + + +def start_fuzzer( + scratch: pathlib.Path, + run_id: str, + *, + ready_timeout_s: float = 5.0, + frame_rate_hz: float = 40.0, + duration_s: float = 300.0, +) -> subprocess.Popen[bytes]: + """Start the paused Task 1 service and wait for its exact ready marker. + + Examples + -------- + >>> try: + ... start_fuzzer(pathlib.Path("."), "", ready_timeout_s=1.0) + ... except ValueError as error: + ... print(error) + run_id must be nonempty + + Parameters + ---------- + scratch : pathlib.Path + Existing exclusive directory that will own ``fuzzer/``. + run_id : str + Identity required in every lifecycle marker. + ready_timeout_s : float + Maximum wait for the service's completed ready marker. + frame_rate_hz : float + Active frames per stream per second after gate release. + duration_s : float + Maximum active service duration. + + Returns + ------- + subprocess.Popen[bytes] + Paused fuzzer process whose PID remains owned by the caller. + + Raises + ------ + ValueError + If the run identity or timeout is invalid. + RuntimeError + If the service exits or misses its ready deadline. + """ + if not run_id: + message = "run_id must be nonempty" + raise ValueError(message) + if ready_timeout_s <= 0: + message = "ready timeout must be positive" + raise ValueError(message) + output_dir = scratch / "fuzzer" + script = pathlib.Path(__file__).with_name("orchestration_fuzzer.py") + environment = os.environ.copy() + environment.pop("TMUX", None) + environment.pop("TMUX_PANE", None) + environment.pop("VIRTUAL_ENV", None) + process = subprocess.Popen( + ( + sys.executable, + str(script), + "serve", + "--output-dir", + str(output_dir), + "--run-id", + run_id, + "--source-root", + str(pathlib.Path(__file__).parents[1]), + "--seed", + "11", + "--frame-rate", + str(frame_rate_hz), + "--duration", + str(duration_s), + "--delayed-match-after", + "0.05", + "--sentinel-prefix", + "READY", + "--heartbeat-interval", + "0.02", + ), + cwd=pathlib.Path(__file__).parents[1], + env=environment, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + ready = output_dir / "ready.json" + deadline = time.monotonic() + ready_timeout_s + while time.monotonic() < deadline: + if process.poll() is not None: + message = f"fuzzer exited before ready with code {process.returncode}" + raise RuntimeError(message) + try: + marker = json.loads(ready.read_text(encoding="utf-8")) + except (FileNotFoundError, json.JSONDecodeError, OSError): + time.sleep(0.01) + continue + if marker == {"schema_version": 1, "run_id": run_id}: + return process + time.sleep(0.01) + if process.poll() is None: + process.terminate() + with contextlib.suppress(subprocess.TimeoutExpired): + process.wait(timeout=1.0) + message = f"fuzzer did not become ready within {ready_timeout_s}s" + raise RuntimeError(message) + + +def _prepare_context( + topology: Topology, + lane: EngineLane, + mode: ExecutionMode, + scratch: pathlib.Path, + *, + socket_path: pathlib.Path | None, + run_id: str, + delayed_ordinal: int, +) -> RunContext: + """Create isolated host resources and an engine without starting setup timing. + + >>> try: + ... _prepare_context( + ... Topology(1, 1, 1), EngineLane.SUBPROCESS, ExecutionMode.SYNC, + ... pathlib.Path("unused"), socket_path=None, run_id="run-7", + ... delayed_ordinal=2, + ... ) + ... except ValueError as error: + ... print(error) + delayed pane ordinal must identify a requested pane + + Parameters + ---------- + topology : Topology + Requested live hierarchy. + lane : EngineLane + Selected engine transport. + mode : ExecutionMode + Sync or async dispatch. + scratch : pathlib.Path + New exclusive directory for this run. + socket_path : pathlib.Path | None + Explicit socket path, defaulting inside ``scratch``. + run_id : str + Safe identifier used in markers and tmux names. + delayed_ordinal : int + Unique pane assigned the delayed stream. + + Returns + ------- + RunContext + Paused fuzzer, isolated server value, and unstarted engine. + + Raises + ------ + ValueError + If identifiers, paths, or the delayed ordinal are invalid. + FileExistsError + If ``scratch`` already exists. + """ + if not 0 <= delayed_ordinal < topology.panes: + message = "delayed pane ordinal must identify a requested pane" + raise ValueError(message) + if not run_id or any( + not (character.isascii() and (character.isalnum() or character in "-_")) + for character in run_id + ): + message = ( + "run_id must contain only ASCII letters, digits, hyphens, or underscores" + ) + raise ValueError(message) + resolved_scratch = scratch.resolve() + resolved_socket = (socket_path or scratch / "tmux.sock").resolve() + if not resolved_socket.is_relative_to(resolved_scratch): + message = "socket path must stay inside the run scratch directory" + raise ValueError(message) + ambient_tmux_environment = ( + os.environ.pop("TMUX", None), + os.environ.pop("TMUX_PANE", None), + ) + scratch_created = False + fuzzer: subprocess.Popen[bytes] | None = None + try: + from libtmux.experimental.engines import ( + AsyncControlModeEngine, + AsyncSubprocessEngine, + ControlModeEngine, + SubprocessEngine, + ) + from libtmux.server import Server + + scratch.mkdir(parents=True, exist_ok=False) + scratch_created = True + fuzzer = start_fuzzer(scratch, run_id) + server = Server(socket_path=resolved_socket, config_file=os.devnull) + if mode is ExecutionMode.SYNC: + engine: TmuxEngine | AsyncTmuxEngine + engine = ( + SubprocessEngine.for_server(server) + if lane is EngineLane.SUBPROCESS + else ControlModeEngine.for_server(server) + ) + else: + engine = ( + AsyncSubprocessEngine.for_server(server) + if lane is EngineLane.SUBPROCESS + else AsyncControlModeEngine.for_server(server) + ) + streams_dir = scratch / "fuzzer" / "streams" + streams = tuple( + streams_dir / f"{name}.log" + for name in ("editor", "dev-server", "installer", "delayed-match") + ) + return RunContext( + topology=topology, + lane=lane, + mode=mode, + run_id=run_id, + scratch=scratch, + socket_path=resolved_socket, + server=server, + engine=engine, + fuzzer=fuzzer, + streams=streams, + delayed_ordinal=delayed_ordinal, + expected_session_names=tuple( + _session_name(run_id, index) for index in range(topology.sessions) + ), + expected_window_names=tuple( + _window_name(run_id, session_index, window_index) + for session_index in range(topology.sessions) + for window_index in range(topology.windows_per_session) + ), + setup_duration_ns=0, + processes=(_record_process("fuzzer", fuzzer.pid),), + ambient_tmux_environment=ambient_tmux_environment, + ) + except BaseException: + if fuzzer is not None and fuzzer.poll() is None: + fuzzer.terminate() + with contextlib.suppress(subprocess.TimeoutExpired): + fuzzer.wait(timeout=1.0) + if scratch_created: + shutil.rmtree(scratch, ignore_errors=True) + for name, value in zip( + ("TMUX", "TMUX_PANE"), ambient_tmux_environment, strict=True + ): + if value is None: + os.environ.pop(name, None) + else: + os.environ[name] = value + raise + + +def setup_sync( + topology: Topology, + lane: EngineLane, + scratch: pathlib.Path, + *, + socket_path: pathlib.Path | None = None, + run_id: str = "run-0", + delayed_ordinal: int = 0, +) -> RunContext: + """Build and exactly verify one synchronous live topology. + + Control mode attaches to a disposable keepalive before timing. The returned + ``setup_duration_ns`` ends after that keepalive is removed and excludes the + activity gate and stabilization polls. + + Examples + -------- + >>> try: + ... setup_sync( + ... Topology(1, 1, 1), EngineLane.SUBPROCESS, pathlib.Path("unused"), + ... delayed_ordinal=2, + ... ) + ... except ValueError as error: + ... print(error) + delayed pane ordinal must identify a requested pane + + Parameters + ---------- + topology : Topology + Requested hierarchy. + lane : EngineLane + Synchronous subprocess or control transport. + scratch : pathlib.Path + New exclusive run directory. + socket_path : pathlib.Path | None + Explicit socket path inside ``scratch``. + run_id : str + Marker and topology identity. + delayed_ordinal : int + Unique pane assigned the delayed stream. + + Returns + ------- + RunContext + Verified topology with stable IDs and process identities. + """ + from libtmux.experimental.engines import SubprocessEngine + from libtmux.experimental.ops import ( + BatchingPlanner, + DisplayMessage, + KillSession, + ListClients, + ListSessions, + NameRef, + NewSession, + SetOption, + SwitchClient, + run, + ) + + context = _prepare_context( + topology, + lane, + ExecutionMode.SYNC, + scratch, + socket_path=socket_path, + run_id=run_id, + delayed_ordinal=delayed_ordinal, + ) + engine = t.cast("TmuxEngine", context.engine) + keepalive = f"bench-{run_id}-keepalive" + try: + if lane is EngineLane.CONTROL: + bootstrap = SubprocessEngine.for_server(context.server) + run( + NewSession( + session_name=keepalive, + window_shell="exec tail -n 0 -f /dev/null", + ), + bootstrap, + ).raise_for_status() + run( + SetOption(server=True, option="exit-empty", value="off"), + bootstrap, + ).raise_for_status() + run(ListSessions(), engine).raise_for_status() + + workspaces = build_workspaces( + topology, + context.scratch / "fuzzer" / "streams", + run_id, + delayed_ordinal=delayed_ordinal, + ) + started_ns = time.perf_counter_ns() + build_result = workspaces.build( + engine, + preflight=False, + planner=BatchingPlanner(), + ).raise_for_status() + if lane is EngineLane.CONTROL: + clients = run(ListClients(), engine).raise_for_status().clients + run( + SwitchClient( + client=_only_control_client_name(clients), + to_session=build_result.bindings[0], + ), + engine, + ).raise_for_status() + run(KillSession(target=NameRef(keepalive)), engine).raise_for_status() + context.setup_duration_ns = time.perf_counter_ns() - started_ns + server_pid_result = run(DisplayMessage(message="#{pid}"), engine) + server_pid_result.raise_for_status() + server_pid = int(server_pid_result.text) + context.processes = (*context.processes, _record_process("server", server_pid)) + verify_topology(context, snapshot_topology_sync(context)) + except BaseException: + asyncio.run(cleanup_run(context)) + raise + else: + return context + + +async def setup_async( + topology: Topology, + lane: EngineLane, + scratch: pathlib.Path, + *, + socket_path: pathlib.Path | None = None, + run_id: str = "run-0", + delayed_ordinal: int = 0, +) -> RunContext: + """Build and exactly verify one asynchronous live topology. + + Examples + -------- + >>> async def invalid_setup(): + ... try: + ... await setup_async( + ... Topology(1, 1, 1), EngineLane.SUBPROCESS, + ... pathlib.Path("unused"), delayed_ordinal=2, + ... ) + ... except ValueError as error: + ... return str(error) + >>> asyncio.run(invalid_setup()) + 'delayed pane ordinal must identify a requested pane' + + Parameters + ---------- + topology : Topology + Requested hierarchy. + lane : EngineLane + Asynchronous subprocess or control transport. + scratch : pathlib.Path + New exclusive run directory. + socket_path : pathlib.Path | None + Explicit socket path inside ``scratch``. + run_id : str + Marker and topology identity. + delayed_ordinal : int + Unique pane assigned the delayed stream. + + Returns + ------- + RunContext + Verified topology with stable IDs and process identities. + """ + from libtmux.experimental.engines import ( + AsyncControlModeEngine, + AsyncSubprocessEngine, + ) + from libtmux.experimental.ops import ( + BatchingPlanner, + DisplayMessage, + KillSession, + ListClients, + ListSessions, + NameRef, + NewSession, + SetOption, + SwitchClient, + arun, + ) + + context = _prepare_context( + topology, + lane, + ExecutionMode.ASYNC, + scratch, + socket_path=socket_path, + run_id=run_id, + delayed_ordinal=delayed_ordinal, + ) + engine = t.cast("AsyncTmuxEngine", context.engine) + keepalive = f"bench-{run_id}-keepalive" + try: + if lane is EngineLane.CONTROL: + bootstrap = AsyncSubprocessEngine.for_server(context.server) + ( + await arun( + NewSession( + session_name=keepalive, + window_shell="exec tail -n 0 -f /dev/null", + ), + bootstrap, + ) + ).raise_for_status() + ( + await arun( + SetOption(server=True, option="exit-empty", value="off"), + bootstrap, + ) + ).raise_for_status() + control = t.cast(AsyncControlModeEngine, engine) + await control.start() + (await arun(ListSessions(), engine)).raise_for_status() + + workspaces = build_workspaces( + topology, + context.scratch / "fuzzer" / "streams", + run_id, + delayed_ordinal=delayed_ordinal, + ) + started_ns = time.perf_counter_ns() + build_result = ( + await workspaces.abuild( + engine, + preflight=False, + planner=BatchingPlanner(), + ) + ).raise_for_status() + if lane is EngineLane.CONTROL: + clients = (await arun(ListClients(), engine)).raise_for_status().clients + real_session_id = build_result.bindings[0] + ( + await arun( + SwitchClient( + client=_only_control_client_name(clients), + to_session=real_session_id, + ), + engine, + ) + ).raise_for_status() + control.set_attach_targets([real_session_id]) + ( + await arun(KillSession(target=NameRef(keepalive)), engine) + ).raise_for_status() + context.setup_duration_ns = time.perf_counter_ns() - started_ns + server_pid_result = await arun(DisplayMessage(message="#{pid}"), engine) + server_pid_result.raise_for_status() + server_pid = int(server_pid_result.text) + context.processes = (*context.processes, _record_process("server", server_pid)) + verify_topology(context, await snapshot_topology_async(context)) + except BaseException: + await cleanup_run(context) + raise + else: + return context + + +def snapshot_topology_sync(context: RunContext) -> TopologySnapshot: + """Read exact sync session, window, and pane snapshots through typed ops. + + >>> snapshot_topology_sync.__name__ + 'snapshot_topology_sync' + + Parameters + ---------- + context : RunContext + Synchronous live run. + + Returns + ------- + TopologySnapshot + Concrete typed rows from three independent list operations. + + Raises + ------ + ValueError + If called with an asynchronous context. + ~libtmux.experimental.ops.exc.TmuxCommandError + If a list operation fails. + """ + from libtmux.experimental.ops import ListPanes, ListSessions, ListWindows, run + + if context.mode is not ExecutionMode.SYNC: + message = "sync snapshot requires a synchronous run context" + raise ValueError(message) + engine = t.cast("TmuxEngine", context.engine) + sessions = run(ListSessions(), engine).raise_for_status().sessions + windows = run(ListWindows(all_windows=True), engine).raise_for_status().windows + panes = run(ListPanes(all_panes=True), engine).raise_for_status().panes + return TopologySnapshot(sessions, windows, panes) + + +async def snapshot_topology_async(context: RunContext) -> TopologySnapshot: + """Read exact async session, window, and pane snapshots through typed ops. + + >>> snapshot_topology_async.__name__ + 'snapshot_topology_async' + + Parameters + ---------- + context : RunContext + Asynchronous live run. + + Returns + ------- + TopologySnapshot + Concrete typed rows from three independent list operations. + + Raises + ------ + ValueError + If called with a synchronous context. + ~libtmux.experimental.ops.exc.TmuxCommandError + If a list operation fails. + """ + from libtmux.experimental.ops import ListPanes, ListSessions, ListWindows, arun + + if context.mode is not ExecutionMode.ASYNC: + message = "async snapshot requires an asynchronous run context" + raise ValueError(message) + engine = t.cast("AsyncTmuxEngine", context.engine) + sessions = (await arun(ListSessions(), engine)).raise_for_status().sessions + windows = ( + (await arun(ListWindows(all_windows=True), engine)).raise_for_status().windows + ) + panes = (await arun(ListPanes(all_panes=True), engine)).raise_for_status().panes + return TopologySnapshot(sessions, windows, panes) + + +def verify_topology( + context: RunContext, + snapshot: TopologySnapshot, +) -> TopologySnapshot: + """Verify exact shape, names, liveness, process identities, and delayed pane. + + >>> verify_topology.__name__ + 'verify_topology' + + Parameters + ---------- + context : RunContext + Run whose declared shape and names are authoritative. + snapshot : TopologySnapshot + Typed live rows to verify. + + Returns + ------- + TopologySnapshot + The accepted snapshot for fluent callers. + + Raises + ------ + TopologyVerificationError + If any exact topology or process invariant differs. + """ + requested = TopologyTotals.from_topology(context.topology) + observed = snapshot.totals + + def fail(detail: str) -> t.NoReturn: + raise TopologyVerificationError(requested, observed, detail) + + if observed != requested: + fail("flat totals differ") + session_ids = tuple(session.session_id for session in snapshot.sessions) + window_ids = tuple(window.window_id for window in snapshot.windows) + pane_ids = tuple(pane.pane_id for pane in snapshot.panes) + if len(set(session_ids)) != requested.sessions or any( + not value.startswith("$") for value in session_ids + ): + fail("session identifiers are not unique concrete ids") + if len(set(window_ids)) != requested.windows or any( + not value.startswith("@") for value in window_ids + ): + fail("window identifiers are not unique concrete ids") + if len(set(pane_ids)) != requested.panes or any( + not value.startswith("%") for value in pane_ids + ): + fail("pane identifiers are not unique concrete ids") + if tuple(session.name for session in snapshot.sessions) != ( + context.expected_session_names + ): + fail("session names differ from the declaration") + if {window.name for window in snapshot.windows} != set( + context.expected_window_names + ): + fail("window names differ from the declaration") + + window_counts = dict.fromkeys(session_ids, 0) + for window in snapshot.windows: + if window.session_id not in window_counts: + fail("window refers to an unknown session") + window_counts[window.session_id] += 1 + if set(window_counts.values()) != {context.topology.windows_per_session}: + fail("a session has the wrong window count") + pane_counts = dict.fromkeys(window_ids, 0) + for pane in snapshot.panes: + if pane.window_id not in pane_counts or pane.session_id not in window_counts: + fail("pane refers to an unknown owner") + pane_counts[pane.window_id] += 1 + if set(pane_counts.values()) != {context.topology.panes_per_window}: + fail("a window has the wrong pane count") + + pane_pids = tuple(pane.pid for pane in snapshot.panes) + if any(pid is None or pid <= 0 for pid in pane_pids): + fail("a pane has no positive follower pid") + positive_pids = t.cast(tuple[int, ...], pane_pids) + if len(set(positive_pids)) != requested.panes: + fail("pane follower pids are not unique") + if any(pane.fields.get("pane_dead") != "0" for pane in snapshot.panes): + fail("a pane is dead") + delayed_path = str(context.streams[-1]) + delayed = tuple( + pane + for pane in snapshot.panes + if delayed_path in pane.fields.get("pane_start_command", "") + ) + if len(delayed) != 1: + fail("delayed stream does not have exactly one follower") + if any( + "exec tail -n 0 -f" not in pane.fields.get("pane_start_command", "") + for pane in snapshot.panes + ): + fail("a pane is not running the portable follower command") + try: + pane_processes = tuple(_record_process("pane", pid) for pid in positive_pids) + except RuntimeError as error: + fail(str(error)) + if context.fuzzer.poll() is not None: + fail("fuzzer exited before activity release") + + context.session_ids = session_ids + context.window_ids = window_ids + context.pane_ids = pane_ids + context.delayed_pane_id = delayed[0].pane_id + context.processes = ( + *(process for process in context.processes if process.role != "pane"), + *pane_processes, + ) + context.topology_verified = True + return snapshot + + +def _read_run_marker(path: pathlib.Path, run_id: str) -> dict[str, t.Any] | None: + """Return a complete schema-v1 marker owned by ``run_id``. + + >>> _read_run_marker(pathlib.Path("missing.json"), "run-7") is None + True + + Parameters + ---------- + path : pathlib.Path + Candidate JSON marker. + run_id : str + Required owner identity. + + Returns + ------- + dict[str, typing.Any] | None + Matching mapping, or ``None`` for missing, malformed, or foreign data. + """ + try: + parsed = json.loads(path.read_text(encoding="utf-8")) + except (FileNotFoundError, json.JSONDecodeError, OSError): + return None + if not isinstance(parsed, dict): + return None + marker = t.cast(dict[str, t.Any], parsed) + version = marker.get("schema_version") + if type(version) is not int or version != 1 or marker.get("run_id") != run_id: + return None + return marker + + +def _heartbeat_epoch(context: RunContext) -> tuple[str | None, int | None]: + """Return the current matching fuzzer state and integer epoch. + + >>> _heartbeat_epoch.__name__ + '_heartbeat_epoch' + + Parameters + ---------- + context : RunContext + Run whose heartbeat marker is authoritative. + + Returns + ------- + tuple[str | None, int | None] + Marker state and epoch, or two ``None`` values before publication. + """ + marker = _read_run_marker( + context.scratch / "fuzzer" / "heartbeat.json", context.run_id + ) + if marker is None: + return None, None + state = marker.get("state") + epoch = marker.get("epoch") + return ( + state if isinstance(state, str) else None, + epoch if type(epoch) is int and epoch >= 0 else None, + ) + + +def release_activity_gate(context: RunContext) -> int: + """Atomically release one verified topology and publish its shared epoch. + + The fuzzer's heartbeat independently proves continuing activity. The shared + run marker establishes one exact stabilization target across streams whose + ordinary frame prefixes otherwise differ by mode. + + >>> release_activity_gate.__name__ + 'release_activity_gate' + + Parameters + ---------- + context : RunContext + Exact verified live topology. + + Returns + ------- + int + Monotonic epoch carried by the shared pane marker. + + Raises + ------ + RuntimeError + If topology verification has not completed or the fuzzer exited. + """ + if not context.topology_verified: + message = "activity cannot start before exact topology verification" + raise RuntimeError(message) + if context.fuzzer.poll() is not None: + message = "fuzzer exited before activity release" + raise RuntimeError(message) + if context.activity_epoch is not None: + return context.activity_epoch + _state, observed_epoch = _heartbeat_epoch(context) + epoch = max(context.heartbeat_epoch, observed_epoch or 0) + 1 + marker = f"LIBTMUX_EPOCH run={context.run_id} epoch={epoch}" + write_json_atomic( + context.scratch / "fuzzer" / "gate.json", + {"schema_version": 1, "run_id": context.run_id, "epoch": epoch}, + ) + encoded = f"{marker}\n".encode() + for stream in context.streams: + with stream.open("ab") as destination: + destination.write(encoded) + destination.flush() + os.fsync(destination.fileno()) + context.activity_epoch = epoch + context.activity_marker = marker + context.heartbeat_epoch = observed_epoch or 0 + return epoch + + +def verify_activity_sync( + context: RunContext, + *, + timeout_s: float = 8.0, + no_progress_timeout_s: float = 3.0, + poll_interval_s: float = 0.02, +) -> int: + """Poll typed pane captures until every pane contains the released marker. + + >>> verify_activity_sync.__name__ + 'verify_activity_sync' + + Parameters + ---------- + context : RunContext + Released synchronous live run. + timeout_s : float + Overall stabilization deadline. + no_progress_timeout_s : float + Deadline reset whenever another pane verifies. + poll_interval_s : float + Delay between incomplete capture passes. + + Returns + ------- + int + Verified shared activity epoch. + + Raises + ------ + RuntimeError + If the gate is absent, the fuzzer exits, or heartbeat epochs regress. + TimeoutError + If overall or no-progress stabilization expires. + """ + from libtmux.experimental.ops import CapturePane, PaneId, run + + if context.mode is not ExecutionMode.SYNC: + message = "sync activity verification requires a synchronous context" + raise ValueError(message) + if context.activity_epoch is None or context.activity_marker is None: + message = "activity gate has not been released" + raise RuntimeError(message) + if timeout_s <= 0 or no_progress_timeout_s <= 0 or poll_interval_s <= 0: + message = "activity verification timeouts and cadence must be positive" + raise ValueError(message) + engine = t.cast("TmuxEngine", context.engine) + remaining = set(context.pane_ids) + deadline = time.monotonic() + timeout_s + progress_deadline = time.monotonic() + no_progress_timeout_s + heartbeat_active = False + while remaining or not heartbeat_active: + if context.fuzzer.poll() is not None: + message = "fuzzer exited during activity stabilization" + raise RuntimeError(message) + state, heartbeat_epoch = _heartbeat_epoch(context) + if heartbeat_epoch is not None: + if heartbeat_epoch < context.heartbeat_epoch: + message = "fuzzer heartbeat epoch moved backwards" + raise RuntimeError(message) + context.heartbeat_epoch = heartbeat_epoch + heartbeat_active = ( + state == "active" + and heartbeat_epoch is not None + and heartbeat_epoch >= context.activity_epoch + ) + before = len(remaining) + for pane_id in tuple(remaining): + result = run( + CapturePane(target=PaneId(pane_id), start=-5000), + engine, + ).raise_for_status() + if context.activity_marker in "\n".join(result.lines): + remaining.remove(pane_id) + now = time.monotonic() + if len(remaining) < before: + progress_deadline = now + no_progress_timeout_s + if not remaining and heartbeat_active: + break + if now >= deadline: + message = ( + f"activity stabilization timed out with {len(remaining)} panes pending" + ) + raise TimeoutError(message) + if now >= progress_deadline: + message = ( + "activity stabilization made no progress with " + f"{len(remaining)} panes pending" + ) + raise TimeoutError(message) + time.sleep(poll_interval_s) + context.activity_pane_ids = context.pane_ids + return context.activity_epoch + + +async def verify_activity_async( + context: RunContext, + *, + timeout_s: float = 8.0, + no_progress_timeout_s: float = 3.0, + poll_interval_s: float = 0.02, +) -> int: + """Async sibling of :func:`verify_activity_sync` over typed captures. + + >>> verify_activity_async.__name__ + 'verify_activity_async' + + Parameters + ---------- + context : RunContext + Released asynchronous live run. + timeout_s : float + Overall stabilization deadline. + no_progress_timeout_s : float + Deadline reset whenever another pane verifies. + poll_interval_s : float + Delay between incomplete capture passes. + + Returns + ------- + int + Verified shared activity epoch. + + Raises + ------ + RuntimeError + If the gate is absent, the fuzzer exits, or heartbeat epochs regress. + TimeoutError + If overall or no-progress stabilization expires. + """ + from libtmux.experimental.ops import CapturePane, PaneId, arun + + if context.mode is not ExecutionMode.ASYNC: + message = "async activity verification requires an asynchronous context" + raise ValueError(message) + if context.activity_epoch is None or context.activity_marker is None: + message = "activity gate has not been released" + raise RuntimeError(message) + if timeout_s <= 0 or no_progress_timeout_s <= 0 or poll_interval_s <= 0: + message = "activity verification timeouts and cadence must be positive" + raise ValueError(message) + engine = t.cast("AsyncTmuxEngine", context.engine) + remaining = set(context.pane_ids) + loop = asyncio.get_running_loop() + deadline = loop.time() + timeout_s + progress_deadline = loop.time() + no_progress_timeout_s + heartbeat_active = False + while remaining or not heartbeat_active: + if context.fuzzer.poll() is not None: + message = "fuzzer exited during activity stabilization" + raise RuntimeError(message) + state, heartbeat_epoch = _heartbeat_epoch(context) + if heartbeat_epoch is not None: + if heartbeat_epoch < context.heartbeat_epoch: + message = "fuzzer heartbeat epoch moved backwards" + raise RuntimeError(message) + context.heartbeat_epoch = heartbeat_epoch + heartbeat_active = ( + state == "active" + and heartbeat_epoch is not None + and heartbeat_epoch >= context.activity_epoch + ) + before = len(remaining) + for pane_id in tuple(remaining): + result = ( + await arun( + CapturePane(target=PaneId(pane_id), start=-5000), + engine, + ) + ).raise_for_status() + if context.activity_marker in "\n".join(result.lines): + remaining.remove(pane_id) + now = loop.time() + if len(remaining) < before: + progress_deadline = now + no_progress_timeout_s + if not remaining and heartbeat_active: + break + if now >= deadline: + message = ( + f"activity stabilization timed out with {len(remaining)} panes pending" + ) + raise TimeoutError(message) + if now >= progress_deadline: + message = ( + "activity stabilization made no progress with " + f"{len(remaining)} panes pending" + ) + raise TimeoutError(message) + await asyncio.sleep(poll_interval_s) + context.activity_pane_ids = context.pane_ids + return context.activity_epoch + + +async def _wait_for_process_absence( + identities: tuple[ProcessIdentity, ...], + *, + timeout_s: float, + poll_child: cabc.Callable[[], object] | None = None, +) -> tuple[ProcessIdentity, ...]: + """Wait for recorded identities to disappear without broad process scans. + + >>> asyncio.run(_wait_for_process_absence((), timeout_s=0.01)) + () + + Parameters + ---------- + identities : tuple[ProcessIdentity, ...] + Exact PID and procfs start-time pairs owned by one run. + timeout_s : float + Maximum monotonic wait before returning survivors. + poll_child : collections.abc.Callable[[], object] | None + Optional child poll used to reap the directly owned fuzzer process. + + Returns + ------- + tuple[ProcessIdentity, ...] + Identity-matched processes still alive at the deadline. + """ + deadline = asyncio.get_running_loop().time() + timeout_s + while True: + if poll_child is not None: + poll_child() + remaining = tuple( + identity for identity in identities if process_identity_matches(identity) + ) + if not remaining or asyncio.get_running_loop().time() >= deadline: + return remaining + await asyncio.sleep(0.02) + + +async def cleanup_run( + context: RunContext, + *, + grace_s: float = 2.0, +) -> CleanupReport: + """Close one engine and remove only identity-matched run resources. + + Cleanup first requests a graceful fuzzer stop, closes a persistent engine, + kills the server through its configured ``Server`` connection, and waits for + every recorded identity. Escalation sends signals only after re-reading an + equal procfs start time. + + >>> cleanup_run.__name__ + 'cleanup_run' + + Parameters + ---------- + context : RunContext + Single owner of fuzzer, engine, server, pane followers, socket, and scratch. + grace_s : float + Wait after each graceful or escalated cleanup step. + + Returns + ------- + CleanupReport + Complete only when all identities and filesystem resources are absent. + """ + if grace_s <= 0: + message = "cleanup grace must be positive" + raise ValueError(message) + errors: list[str] = [] + stop_path = context.scratch / "fuzzer" / "stop.json" + if stop_path.parent.exists(): + try: + write_json_atomic( + stop_path, + {"schema_version": 1, "run_id": context.run_id}, + ) + except Exception as error: # noqa: BLE001 + errors.append(f"fuzzer stop marker: {type(error).__name__}: {error}") + + async_closer = getattr(context.engine, "aclose", None) + sync_closer = getattr(context.engine, "close", None) + try: + if callable(async_closer): + await async_closer() + elif callable(sync_closer): + sync_closer() + except Exception as error: # noqa: BLE001 + errors.append(f"engine close: {type(error).__name__}: {error}") + + try: + context.server.kill() + except Exception as error: # noqa: BLE001 + errors.append(f"server kill: {type(error).__name__}: {error}") + + survivors = await _wait_for_process_absence( + context.processes, + timeout_s=grace_s, + poll_child=context.fuzzer.poll, + ) + for signal_number in (signal.SIGTERM, signal.SIGKILL): + if not survivors: + break + for identity in survivors: + if not process_identity_matches(identity): + continue + try: + os.kill(identity.pid, signal_number) + except ProcessLookupError: + continue + except OSError as error: + errors.append( + f"{identity.role} pid {identity.pid} signal " + f"{signal_number}: {type(error).__name__}: {error}" + ) + survivors = await _wait_for_process_absence( + context.processes, + timeout_s=grace_s, + poll_child=context.fuzzer.poll, + ) + + with contextlib.suppress(subprocess.TimeoutExpired): + context.fuzzer.wait(timeout=0.1) + errors.extend( + f"{identity.role} pid {identity.pid} with start time " + f"{identity.start_time} remains" + for identity in survivors + if process_identity_matches(identity) + ) + try: + if context.scratch.exists(): + shutil.rmtree(context.scratch) + except OSError as error: + errors.append(f"scratch removal: {type(error).__name__}: {error}") + if context.socket_path.exists(): + errors.append(f"socket remains: {context.socket_path}") + if context.scratch.exists(): + errors.append(f"scratch remains: {context.scratch}") + for identity in context.processes: + if process_identity_matches(identity) and not any( + f"pid {identity.pid} " in error for error in errors + ): + errors.append( + f"{identity.role} pid {identity.pid} with start time " + f"{identity.start_time} remains" + ) + for name, value in zip( + ("TMUX", "TMUX_PANE"), context.ambient_tmux_environment, strict=True + ): + if value is None: + os.environ.pop(name, None) + else: + os.environ[name] = value + return CleanupReport(complete=not errors, errors=tuple(errors)) + + def parse_topology(shape: str) -> Topology: """Parse a positive ``SxWxP`` topology string. diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index 28724045ef..1c80f04c4e 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -2,9 +2,11 @@ from __future__ import annotations +import asyncio import dataclasses import importlib.util import json +import os import pathlib import subprocess import sys @@ -816,3 +818,167 @@ def test_validate_report_accepts_in_progress_completed_prefix( ) benchmark_module.validate_report(report) + + +def test_lifecycle_marker_reader_rejects_boolean_schema_version( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """A boolean schema value must not release a run as integer version one.""" + marker = tmp_path / "marker.json" + marker.write_text('{"schema_version":true,"run_id":"run-7"}\n', encoding="utf-8") + + assert benchmark_module._read_run_marker(marker, "run-7") is None + + +def test_live_topology_setup_preserves_preexisting_scratch( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """A rejected run must never delete a directory it did not create.""" + scratch = tmp_path / "already-owned" + scratch.mkdir() + marker = scratch / "owner.txt" + marker.write_text("keep\n", encoding="utf-8") + + with pytest.raises(FileExistsError): + benchmark_module.setup_sync( + benchmark_module.Topology(1, 1, 1), + benchmark_module.EngineLane.SUBPROCESS, + scratch, + run_id="preexisting", + ) + + assert marker.read_text(encoding="utf-8") == "keep\n" + + +@pytest.mark.parametrize( + ("lane_name", "mode_name"), + ( + ("subprocess", "sync"), + ("subprocess", "async"), + ("control", "sync"), + ("control", "async"), + ), +) +def test_live_topology_lifecycle_cleans_each_engine_lane( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, + lane_name: str, + mode_name: str, +) -> None: + """A lane must build, activate, and remove only its isolated live topology.""" + monkeypatch.setenv("TMUX", "ambient-server") + monkeypatch.setenv("TMUX_PANE", "%ambient") + topology = benchmark_module.Topology(2, 2, 2) + lane = benchmark_module.EngineLane(lane_name) + scratch = tmp_path / f"{lane_name}-{mode_name}" + socket_path = scratch / "tmux.sock" + run_id = f"live-{lane_name}-{mode_name}" + context = None + snapshot = None + cleanup = None + captured_activity: dict[str, str] = {} + + async def exercise_async() -> None: + """Keep async engines on one event loop through their final close.""" + nonlocal context, snapshot, cleanup + context = await benchmark_module.setup_async( + topology, + lane, + scratch, + socket_path=socket_path, + run_id=run_id, + delayed_ordinal=5, + ) + try: + assert "TMUX" not in os.environ + assert "TMUX_PANE" not in os.environ + snapshot = await benchmark_module.snapshot_topology_async(context) + benchmark_module.verify_topology(context, snapshot) + epoch = benchmark_module.release_activity_gate(context) + assert await benchmark_module.verify_activity_async(context) == epoch + for pane_id in context.pane_ids: + result = context.server.cmd( + "capture-pane", "-t", pane_id, "-p", "-S", "-5000" + ) + assert result.returncode == 0, result.stderr + captured_activity[pane_id] = "\n".join(result.stdout) + finally: + cleanup = await benchmark_module.cleanup_run(context) + + if mode_name == "async": + asyncio.run(exercise_async()) + else: + context = benchmark_module.setup_sync( + topology, + lane, + scratch, + socket_path=socket_path, + run_id=run_id, + delayed_ordinal=5, + ) + try: + assert "TMUX" not in os.environ + assert "TMUX_PANE" not in os.environ + snapshot = benchmark_module.snapshot_topology_sync(context) + benchmark_module.verify_topology(context, snapshot) + epoch = benchmark_module.release_activity_gate(context) + assert benchmark_module.verify_activity_sync(context) == epoch + for pane_id in context.pane_ids: + result = context.server.cmd( + "capture-pane", "-t", pane_id, "-p", "-S", "-5000" + ) + assert result.returncode == 0, result.stderr + captured_activity[pane_id] = "\n".join(result.stdout) + finally: + cleanup = asyncio.run(benchmark_module.cleanup_run(context)) + + assert context is not None + assert snapshot is not None + assert cleanup is not None + assert context.server.config_file == os.devnull + assert context.socket_path == socket_path + assert context.setup_duration_ns > 0 + assert len(snapshot.sessions) == 2 + assert len(snapshot.windows) == 4 + assert len(snapshot.panes) == 8 + assert len(context.session_ids) == len(set(context.session_ids)) == 2 + assert len(context.window_ids) == len(set(context.window_ids)) == 4 + assert len(context.pane_ids) == len(set(context.pane_ids)) == 8 + assert len({pane.pid for pane in snapshot.panes}) == 8 + assert all(pane.pid is not None and pane.pid > 0 for pane in snapshot.panes) + assert all(pane.fields["pane_dead"] == "0" for pane in snapshot.panes) + delayed = [ + pane + for pane in snapshot.panes + if "delayed-match.log" in pane.fields["pane_start_command"] + ] + assert [pane.pane_id for pane in delayed] == [context.delayed_pane_id] + assert context.activity_pane_ids == context.pane_ids + assert context.activity_marker == ( + f"LIBTMUX_EPOCH run={run_id} epoch={context.activity_epoch}" + ) + assert set(captured_activity) == set(context.pane_ids) + assert all( + context.activity_marker in captured_activity[pane_id] + for pane_id in context.pane_ids + ) + assert context.heartbeat_epoch >= context.activity_epoch + + recorded_processes = context.processes + assert ( + len([process for process in recorded_processes if process.role == "pane"]) == 8 + ) + assert cleanup.complete + assert cleanup.errors == () + assert context.fuzzer.poll() is not None + assert all( + not benchmark_module.process_identity_matches(process) + for process in recorded_processes + ) + assert not socket_path.exists() + assert not scratch.exists() + assert os.environ["TMUX"] == "ambient-server" + assert os.environ["TMUX_PANE"] == "%ambient" From f672ee36c544ff6d7c9917dbcead283ece90cb0b Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 08:09:14 -0500 Subject: [PATCH 13/67] Bench(fix[topology]): Harden live ownership why: Partial setup failures could lose cleanup evidence, permissive directories exposed run state, and exact counts did not prove parentage. what: - Reap pre-return children and surface typed cleanup evidence - Enforce private scratch and bounded owned socket paths - Verify declared topology parentage with adversarial coverage --- scripts/bench_orchestration.py | 620 ++++++++++++++++++++--- tests/test_bench_orchestration_script.py | 357 +++++++++++++ 2 files changed, 920 insertions(+), 57 deletions(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index 0a00910495..3f012a8209 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -25,6 +25,7 @@ import shlex import shutil import signal +import stat import statistics import subprocess import sys @@ -32,6 +33,7 @@ import time import types import typing as t +import uuid if t.TYPE_CHECKING: from libtmux.experimental.engines.base import AsyncTmuxEngine, TmuxEngine @@ -923,6 +925,61 @@ class CleanupReport: errors: tuple[str, ...] = () +class SetupCleanupError(RuntimeError): + """A setup failure whose required cleanup also failed. + + Attributes + ---------- + setup_error : BaseException + Original setup exception retained as the chained cause. + cleanup_report : CleanupReport + Structured evidence from the unsuccessful cleanup attempt. + + Examples + -------- + >>> cause = ValueError("setup") + >>> error = SetupCleanupError(cause, CleanupReport(False, ("socket remains",))) + >>> error.setup_error is cause, error.cleanup_report.complete + (True, False) + """ + + def __init__( + self, + setup_error: BaseException, + cleanup_report: CleanupReport, + ) -> None: + """Retain the original setup error and cleanup evidence. + + >>> error = SetupCleanupError( + ... RuntimeError("build"), CleanupReport(False, ("pid remains",)) + ... ) + >>> str(error) + 'setup failed with RuntimeError; cleanup incomplete: pid remains' + + Parameters + ---------- + setup_error : BaseException + Failure that triggered partial-run cleanup. + cleanup_report : CleanupReport + Incomplete cleanup result with accessible error evidence. + + Raises + ------ + ValueError + If the supplied cleanup report claims completion. + """ + if cleanup_report.complete: + message = "setup cleanup error requires an incomplete cleanup report" + raise ValueError(message) + self.setup_error = setup_error + self.cleanup_report = cleanup_report + detail = "; ".join(cleanup_report.errors) or "unspecified cleanup failure" + super().__init__( + f"setup failed with {type(setup_error).__name__}; " + f"cleanup incomplete: {detail}" + ) + + @dataclasses.dataclass(frozen=True) class ProcessIdentity: """One benchmark-owned process protected against PID reuse. @@ -1091,6 +1148,8 @@ class RunContext: Exclusive directory removed during cleanup. socket_path : pathlib.Path Explicit isolated tmux socket path. + socket_root : pathlib.Path or None + Short private directory owned only when the socket path is implicit. server : Server Classic server value used only to bind engines to the socket. engine : TmuxEngine or AsyncTmuxEngine @@ -1105,6 +1164,8 @@ class RunContext: Exact declared session names. expected_window_names : tuple[str, ...] Exact declared window names. + expected_window_parents : tuple[tuple[str, str], ...] + Declared window-name to session-name ownership pairs. setup_duration_ns : int Timed construction duration excluding activity stabilization. processes : tuple[ProcessIdentity, ...] @@ -1115,6 +1176,10 @@ class RunContext: Verified stable window identifiers. pane_ids : tuple[str, ...] Verified stable pane identifiers. + session_bindings : tuple[tuple[str, str], ...] + Verified session-name to session-ID bindings. + window_bindings : tuple[tuple[str, str, str], ...] + Verified window-name, window-ID, and parent-session-ID bindings. delayed_pane_id : str or None Verified pane following the unique delayed stream. topology_verified : bool @@ -1143,6 +1208,7 @@ class RunContext: run_id: str scratch: pathlib.Path socket_path: pathlib.Path + socket_root: pathlib.Path | None server: Server engine: TmuxEngine | AsyncTmuxEngine fuzzer: subprocess.Popen[bytes] @@ -1150,11 +1216,14 @@ class RunContext: delayed_ordinal: int expected_session_names: tuple[str, ...] expected_window_names: tuple[str, ...] + expected_window_parents: tuple[tuple[str, str], ...] setup_duration_ns: int processes: tuple[ProcessIdentity, ...] session_ids: tuple[str, ...] = () window_ids: tuple[str, ...] = () pane_ids: tuple[str, ...] = () + session_bindings: tuple[tuple[str, str], ...] = () + window_bindings: tuple[tuple[str, str, str], ...] = () delayed_pane_id: str | None = None topology_verified: bool = False activity_epoch: int | None = None @@ -1662,6 +1731,165 @@ def process_identity_matches(identity: ProcessIdentity) -> bool: return _process_start_time(identity.pid) == identity.start_time +_UNIX_SOCKET_PATH_MAX_BYTES = 107 + + +def _validate_socket_path(path: pathlib.Path) -> pathlib.Path: + """Validate one pathname against the Linux Unix-socket ABI ceiling. + + Linux defines ``sockaddr_un.sun_path`` as 108 bytes. Pathname sockets need + one trailing NUL byte, leaving 107 encoded filesystem bytes for tmux's + socket path. + + >>> boundary = pathlib.Path("/" + "é" * 53) + >>> len(os.fsencode(boundary)), _validate_socket_path(boundary) == boundary + (107, True) + + Parameters + ---------- + path : pathlib.Path + Candidate tmux socket pathname. + + Returns + ------- + pathlib.Path + The accepted path unchanged. + + Raises + ------ + ValueError + If the encoded path contains NUL or exceeds 107 bytes. + """ + encoded = os.fsencode(path) + if b"\0" in encoded: + message = "tmux socket path must not contain NUL" + raise ValueError(message) + if len(encoded) > _UNIX_SOCKET_PATH_MAX_BYTES: + message = f"tmux socket path exceeds 107 encoded bytes: {len(encoded)} bytes" + raise ValueError(message) + return path + + +def _acquire_private_directory(directory: pathlib.Path) -> pathlib.Path: + """Create and verify an exclusively acquired mode-0700 directory. + + >>> with tempfile.TemporaryDirectory() as temporary: + ... run_dir = pathlib.Path(temporary) / "run" + ... acquired = _acquire_private_directory(run_dir) + ... oct(stat.S_IMODE(acquired.stat().st_mode)) + '0o700' + + Parameters + ---------- + directory : pathlib.Path + New directory whose pre-existence rejects acquisition. + + Returns + ------- + pathlib.Path + The verified private directory. + + Raises + ------ + FileExistsError + If another owner already created the directory. + OSError + If mode 0700 cannot be applied or verified. + """ + directory.mkdir(mode=0o700, parents=True, exist_ok=False) + directory.chmod(0o700) + observed_mode = stat.S_IMODE(directory.stat().st_mode) + if observed_mode != 0o700: + message = f"private run directory mode is {oct(observed_mode)}, expected 0o700" + raise OSError(message) + return directory + + +def _stop_owned_process( + process: subprocess.Popen[bytes] | subprocess.Popen[str], + identity: ProcessIdentity, + *, + grace_s: float = 1.0, +) -> CleanupReport: + """Boundedly stop and reap one child without signaling a reused PID. + + >>> child = subprocess.Popen((sys.executable, "-c", "pass")) + >>> child.wait(timeout=1.0) + 0 + >>> _stop_owned_process( + ... child, ProcessIdentity("finished", child.pid, -1), grace_s=0.01 + ... ).complete + True + + Parameters + ---------- + process : subprocess.Popen[bytes] or subprocess.Popen[str] + Directly owned child handle used for bounded waits and reaping. + identity : ProcessIdentity + PID and procfs start time recorded immediately after spawn. + grace_s : float + Maximum wait after TERM and again after identity-checked KILL. + + Returns + ------- + CleanupReport + Complete only when the child is reaped and its identity is absent. + + Raises + ------ + ValueError + If the cleanup grace is not positive. + """ + if grace_s <= 0: + message = "process cleanup grace must be positive" + raise ValueError(message) + errors: list[str] = [] + process.poll() + if process.returncode is None: + if process_identity_matches(identity): + try: + os.kill(identity.pid, signal.SIGTERM) + except ProcessLookupError: + pass + except OSError as error: + errors.append( + f"{identity.role} pid {identity.pid} SIGTERM: " + f"{type(error).__name__}: {error}" + ) + else: + errors.append( + f"{identity.role} pid {identity.pid} identity changed before SIGTERM" + ) + try: + process.wait(timeout=grace_s) + except subprocess.TimeoutExpired: + if process_identity_matches(identity): + try: + os.kill(identity.pid, signal.SIGKILL) + except ProcessLookupError: + pass + except OSError as error: + errors.append( + f"{identity.role} pid {identity.pid} SIGKILL: " + f"{type(error).__name__}: {error}" + ) + else: + errors.append( + f"{identity.role} pid {identity.pid} identity changed " + "before SIGKILL" + ) + try: + process.wait(timeout=grace_s) + except subprocess.TimeoutExpired: + errors.append(f"{identity.role} pid {identity.pid} was not reaped") + if process_identity_matches(identity): + errors.append( + f"{identity.role} pid {identity.pid} with start time " + f"{identity.start_time} remains" + ) + return CleanupReport(complete=not errors, errors=tuple(errors)) + + def _stream_name(ordinal: int, delayed_ordinal: int) -> str: """Return the Task 1 stream name for one global stable pane ordinal. @@ -1780,6 +2008,36 @@ def _only_control_client_name(clients: tuple[ClientSnapshot, ...]) -> str: return clients[0].name +def _only_process_identity( + identities: list[ProcessIdentity], +) -> ProcessIdentity: + """Return the sole process identity transferred by a child starter. + + >>> identity = ProcessIdentity("fuzzer", 42, 100) + >>> _only_process_identity([identity]) is identity + True + + Parameters + ---------- + identities : list[ProcessIdentity] + Mutable handoff populated before child startup returns. + + Returns + ------- + ProcessIdentity + The single transferred owner identity. + + Raises + ------ + RuntimeError + If startup did not transfer exactly one identity. + """ + if len(identities) != 1: + message = "child startup did not transfer exactly one process identity" + raise RuntimeError(message) + return identities[0] + + def build_workspaces( topology: Topology, streams_dir: pathlib.Path, @@ -1864,6 +2122,64 @@ def build_workspaces( return WorkspaceSet(workspaces) +def _wait_for_fuzzer_ready( + process: subprocess.Popen[bytes], + ready: pathlib.Path, + run_id: str, + *, + timeout_s: float, +) -> None: + """Wait boundedly for one exact fuzzer readiness marker. + + >>> finished = subprocess.Popen((sys.executable, "-c", "pass")) + >>> finished.wait(timeout=1.0) + 0 + >>> try: + ... _wait_for_fuzzer_ready( + ... finished, pathlib.Path("missing.json"), "run-7", timeout_s=0.01 + ... ) + ... except RuntimeError as error: + ... print(str(error).startswith("fuzzer exited before ready")) + True + + Parameters + ---------- + process : subprocess.Popen[bytes] + Live central fuzzer child. + ready : pathlib.Path + Atomic readiness marker path. + run_id : str + Exact owner required in the marker. + timeout_s : float + Maximum monotonic wait. + + Returns + ------- + None + After the exact marker is visible. + + Raises + ------ + RuntimeError + If the child exits or the marker misses its deadline. + """ + deadline = time.monotonic() + timeout_s + while time.monotonic() < deadline: + if process.poll() is not None: + message = f"fuzzer exited before ready with code {process.returncode}" + raise RuntimeError(message) + try: + marker = json.loads(ready.read_text(encoding="utf-8")) + except (FileNotFoundError, json.JSONDecodeError, OSError): + time.sleep(0.01) + continue + if marker == {"schema_version": 1, "run_id": run_id}: + return + time.sleep(0.01) + message = f"fuzzer did not become ready within {timeout_s}s" + raise RuntimeError(message) + + def start_fuzzer( scratch: pathlib.Path, run_id: str, @@ -1871,6 +2187,7 @@ def start_fuzzer( ready_timeout_s: float = 5.0, frame_rate_hz: float = 40.0, duration_s: float = 300.0, + _identity_out: list[ProcessIdentity] | None = None, ) -> subprocess.Popen[bytes]: """Start the paused Task 1 service and wait for its exact ready marker. @@ -1894,6 +2211,9 @@ def start_fuzzer( Active frames per stream per second after gate release. duration_s : float Maximum active service duration. + _identity_out : list[ProcessIdentity] | None + Internal ownership handoff populated with the identity captured at + spawn time before this function returns. Returns ------- @@ -1906,6 +2226,8 @@ def start_fuzzer( If the run identity or timeout is invalid. RuntimeError If the service exits or misses its ready deadline. + SetupCleanupError + If a startup failure is followed by incomplete child cleanup. """ if not run_id: message = "run_id must be nonempty" @@ -1950,25 +2272,49 @@ def start_fuzzer( stderr=subprocess.DEVNULL, ) ready = output_dir / "ready.json" - deadline = time.monotonic() + ready_timeout_s - while time.monotonic() < deadline: - if process.poll() is not None: - message = f"fuzzer exited before ready with code {process.returncode}" - raise RuntimeError(message) - try: - marker = json.loads(ready.read_text(encoding="utf-8")) - except (FileNotFoundError, json.JSONDecodeError, OSError): - time.sleep(0.01) - continue - if marker == {"schema_version": 1, "run_id": run_id}: - return process - time.sleep(0.01) - if process.poll() is None: - process.terminate() - with contextlib.suppress(subprocess.TimeoutExpired): - process.wait(timeout=1.0) - message = f"fuzzer did not become ready within {ready_timeout_s}s" - raise RuntimeError(message) + identity: ProcessIdentity | None = None + try: + identity = _record_process("fuzzer", process.pid) + _wait_for_fuzzer_ready( + process, + ready, + run_id, + timeout_s=ready_timeout_s, + ) + if _identity_out is not None: + _identity_out.append(identity) + except BaseException as startup_error: + if identity is None: + start_time = _process_start_time(process.pid) + if start_time is not None: + identity = ProcessIdentity("fuzzer", process.pid, start_time) + if identity is None: + process.poll() + if process.returncode is None: + process.terminate() + try: + process.wait(timeout=1.0) + except subprocess.TimeoutExpired: + cleanup_error = ( + f"fuzzer pid {process.pid} identity unavailable; " + "child was not reaped" + ) + cleanup_report = CleanupReport( + complete=False, + errors=(cleanup_error,), + ) + else: + cleanup_report = CleanupReport(complete=True) + else: + process.wait() + cleanup_report = CleanupReport(complete=True) + else: + cleanup_report = _stop_owned_process(process, identity) + if not cleanup_report.complete: + raise SetupCleanupError(startup_error, cleanup_report) from startup_error + raise + else: + return process def _prepare_context( @@ -2004,7 +2350,7 @@ def _prepare_context( scratch : pathlib.Path New exclusive directory for this run. socket_path : pathlib.Path | None - Explicit socket path, defaulting inside ``scratch``. + Explicit scratch-contained path, or ``None`` for a short owned root. run_id : str Safe identifier used in markers and tmux names. delayed_ordinal : int @@ -2020,7 +2366,11 @@ def _prepare_context( ValueError If identifiers, paths, or the delayed ordinal are invalid. FileExistsError - If ``scratch`` already exists. + If ``scratch`` or the generated socket root already exists. + OSError + If a private mode cannot be applied or resources cannot be created. + SetupCleanupError + If partial resource acquisition cannot be completely cleaned. """ if not 0 <= delayed_ordinal < topology.panes: message = "delayed pane ordinal must identify a requested pane" @@ -2034,16 +2384,26 @@ def _prepare_context( ) raise ValueError(message) resolved_scratch = scratch.resolve() - resolved_socket = (socket_path or scratch / "tmux.sock").resolve() - if not resolved_socket.is_relative_to(resolved_scratch): - message = "socket path must stay inside the run scratch directory" - raise ValueError(message) + socket_root: pathlib.Path | None + if socket_path is None: + socket_root = pathlib.Path(tempfile.gettempdir()).resolve() / ( + f"libtmux-bench-{os.getpid():x}-{uuid.uuid4().hex[:8]}" + ) + resolved_socket = _validate_socket_path(socket_root / "tmux.sock") + else: + socket_root = None + resolved_socket = _validate_socket_path(socket_path.resolve()) + if not resolved_socket.is_relative_to(resolved_scratch): + message = "explicit socket path must stay inside the run scratch directory" + raise ValueError(message) ambient_tmux_environment = ( os.environ.pop("TMUX", None), os.environ.pop("TMUX_PANE", None), ) scratch_created = False + socket_root_created = False fuzzer: subprocess.Popen[bytes] | None = None + fuzzer_identity: ProcessIdentity | None = None try: from libtmux.experimental.engines import ( AsyncControlModeEngine, @@ -2053,9 +2413,18 @@ def _prepare_context( ) from libtmux.server import Server - scratch.mkdir(parents=True, exist_ok=False) + _acquire_private_directory(resolved_scratch) scratch_created = True - fuzzer = start_fuzzer(scratch, run_id) + if socket_root is not None: + _acquire_private_directory(socket_root) + socket_root_created = True + fuzzer_identities: list[ProcessIdentity] = [] + fuzzer = start_fuzzer( + resolved_scratch, + run_id, + _identity_out=fuzzer_identities, + ) + fuzzer_identity = _only_process_identity(fuzzer_identities) server = Server(socket_path=resolved_socket, config_file=os.devnull) if mode is ExecutionMode.SYNC: engine: TmuxEngine | AsyncTmuxEngine @@ -2070,7 +2439,7 @@ def _prepare_context( if lane is EngineLane.SUBPROCESS else AsyncControlModeEngine.for_server(server) ) - streams_dir = scratch / "fuzzer" / "streams" + streams_dir = resolved_scratch / "fuzzer" / "streams" streams = tuple( streams_dir / f"{name}.log" for name in ("editor", "dev-server", "installer", "delayed-match") @@ -2080,8 +2449,9 @@ def _prepare_context( lane=lane, mode=mode, run_id=run_id, - scratch=scratch, + scratch=resolved_scratch, socket_path=resolved_socket, + socket_root=socket_root, server=server, engine=engine, fuzzer=fuzzer, @@ -2095,17 +2465,36 @@ def _prepare_context( for session_index in range(topology.sessions) for window_index in range(topology.windows_per_session) ), + expected_window_parents=tuple( + ( + _window_name(run_id, session_index, window_index), + _session_name(run_id, session_index), + ) + for session_index in range(topology.sessions) + for window_index in range(topology.windows_per_session) + ), setup_duration_ns=0, - processes=(_record_process("fuzzer", fuzzer.pid),), + processes=(fuzzer_identity,), ambient_tmux_environment=ambient_tmux_environment, ) - except BaseException: - if fuzzer is not None and fuzzer.poll() is None: - fuzzer.terminate() - with contextlib.suppress(subprocess.TimeoutExpired): - fuzzer.wait(timeout=1.0) - if scratch_created: - shutil.rmtree(scratch, ignore_errors=True) + except BaseException as setup_error: + cleanup_errors: list[str] = [] + if fuzzer is not None and fuzzer_identity is not None: + cleanup_errors.extend(_stop_owned_process(fuzzer, fuzzer_identity).errors) + for label, directory, acquired in ( + ("socket root", socket_root, socket_root_created), + ("scratch", resolved_scratch, scratch_created), + ): + if directory is None or not acquired: + continue + try: + shutil.rmtree(directory) + except OSError as cleanup_error: + cleanup_errors.append( + f"{label} removal: {type(cleanup_error).__name__}: {cleanup_error}" + ) + if directory.exists(): + cleanup_errors.append(f"{label} remains: {directory}") for name, value in zip( ("TMUX", "TMUX_PANE"), ambient_tmux_environment, strict=True ): @@ -2113,6 +2502,12 @@ def _prepare_context( os.environ.pop(name, None) else: os.environ[name] = value + cleanup_report = CleanupReport( + complete=not cleanup_errors, + errors=tuple(cleanup_errors), + ) + if not cleanup_report.complete: + raise SetupCleanupError(setup_error, cleanup_report) from setup_error raise @@ -2161,6 +2556,15 @@ def setup_sync( ------- RunContext Verified topology with stable IDs and process identities. + + Raises + ------ + ValueError + If topology identity, socket path, or delayed ordinal is invalid. + FileExistsError + If the run cannot exclusively acquire its scratch directory. + SetupCleanupError + If setup fails and cleanup also reports errors. """ from libtmux.experimental.engines import SubprocessEngine from libtmux.experimental.ops import ( @@ -2231,8 +2635,18 @@ def setup_sync( server_pid = int(server_pid_result.text) context.processes = (*context.processes, _record_process("server", server_pid)) verify_topology(context, snapshot_topology_sync(context)) - except BaseException: - asyncio.run(cleanup_run(context)) + except BaseException as setup_error: + try: + cleanup_report = asyncio.run(cleanup_run(context)) + except BaseException as cleanup_error: # noqa: BLE001 + cleanup_report = CleanupReport( + complete=False, + errors=( + f"cleanup raised {type(cleanup_error).__name__}: {cleanup_error}", + ), + ) + if not cleanup_report.complete: + raise SetupCleanupError(setup_error, cleanup_report) from setup_error raise else: return context @@ -2281,6 +2695,15 @@ async def setup_async( ------- RunContext Verified topology with stable IDs and process identities. + + Raises + ------ + ValueError + If topology identity, socket path, or delayed ordinal is invalid. + FileExistsError + If the run cannot exclusively acquire its scratch directory. + SetupCleanupError + If setup fails and cleanup also reports errors. """ from libtmux.experimental.engines import ( AsyncControlModeEngine, @@ -2368,8 +2791,18 @@ async def setup_async( server_pid = int(server_pid_result.text) context.processes = (*context.processes, _record_process("server", server_pid)) verify_topology(context, await snapshot_topology_async(context)) - except BaseException: - await cleanup_run(context) + except BaseException as setup_error: + try: + cleanup_report = await cleanup_run(context) + except BaseException as cleanup_error: # noqa: BLE001 + cleanup_report = CleanupReport( + complete=False, + errors=( + f"cleanup raised {type(cleanup_error).__name__}: {cleanup_error}", + ), + ) + if not cleanup_report.complete: + raise SetupCleanupError(setup_error, cleanup_report) from setup_error raise else: return context @@ -2378,8 +2811,12 @@ async def setup_async( def snapshot_topology_sync(context: RunContext) -> TopologySnapshot: """Read exact sync session, window, and pane snapshots through typed ops. - >>> snapshot_topology_sync.__name__ - 'snapshot_topology_sync' + >>> wrong_mode = types.SimpleNamespace(mode=ExecutionMode.ASYNC) + >>> try: + ... snapshot_topology_sync(wrong_mode) + ... except ValueError as error: + ... print(error) + sync snapshot requires a synchronous run context Parameters ---------- @@ -2413,8 +2850,12 @@ def snapshot_topology_sync(context: RunContext) -> TopologySnapshot: async def snapshot_topology_async(context: RunContext) -> TopologySnapshot: """Read exact async session, window, and pane snapshots through typed ops. - >>> snapshot_topology_async.__name__ - 'snapshot_topology_async' + >>> wrong_mode = types.SimpleNamespace(mode=ExecutionMode.SYNC) + >>> try: + ... asyncio.run(snapshot_topology_async(wrong_mode)) + ... except ValueError as error: + ... print(error) + async snapshot requires an asynchronous run context Parameters ---------- @@ -2453,8 +2894,12 @@ def verify_topology( ) -> TopologySnapshot: """Verify exact shape, names, liveness, process identities, and delayed pane. - >>> verify_topology.__name__ - 'verify_topology' + >>> empty_context = types.SimpleNamespace(topology=Topology(1, 1, 1)) + >>> try: + ... verify_topology(empty_context, TopologySnapshot((), (), ())) + ... except TopologyVerificationError as error: + ... print(error.detail) + flat totals differ Parameters ---------- @@ -2505,6 +2950,16 @@ def fail(detail: str) -> t.NoReturn: ): fail("window names differ from the declaration") + session_ids_by_name = { + t.cast(str, session.name): session.session_id for session in snapshot.sessions + } + expected_window_parents = dict(context.expected_window_parents) + windows_by_name = {t.cast(str, window.name): window for window in snapshot.windows} + for window_name, expected_session_name in context.expected_window_parents: + window = windows_by_name[window_name] + if window.session_id != session_ids_by_name[expected_session_name]: + fail("window parent differs from the declaration") + window_counts = dict.fromkeys(session_ids, 0) for window in snapshot.windows: if window.session_id not in window_counts: @@ -2513,9 +2968,12 @@ def fail(detail: str) -> t.NoReturn: if set(window_counts.values()) != {context.topology.windows_per_session}: fail("a session has the wrong window count") pane_counts = dict.fromkeys(window_ids, 0) + windows_by_id = {window.window_id: window for window in snapshot.windows} for pane in snapshot.panes: if pane.window_id not in pane_counts or pane.session_id not in window_counts: fail("pane refers to an unknown owner") + if pane.session_id != windows_by_id[pane.window_id].session_id: + fail("pane session differs from its window parent") pane_counts[pane.window_id] += 1 if set(pane_counts.values()) != {context.topology.panes_per_window}: fail("a window has the wrong pane count") @@ -2551,6 +3009,17 @@ def fail(detail: str) -> t.NoReturn: context.session_ids = session_ids context.window_ids = window_ids context.pane_ids = pane_ids + context.session_bindings = tuple( + (name, session_ids_by_name[name]) for name in context.expected_session_names + ) + context.window_bindings = tuple( + ( + name, + windows_by_name[name].window_id, + session_ids_by_name[expected_window_parents[name]], + ) + for name in context.expected_window_names + ) context.delayed_pane_id = delayed[0].pane_id context.processes = ( *(process for process in context.processes if process.role != "pane"), @@ -2594,8 +3063,16 @@ def _read_run_marker(path: pathlib.Path, run_id: str) -> dict[str, t.Any] | None def _heartbeat_epoch(context: RunContext) -> tuple[str | None, int | None]: """Return the current matching fuzzer state and integer epoch. - >>> _heartbeat_epoch.__name__ - '_heartbeat_epoch' + >>> with tempfile.TemporaryDirectory() as temporary: + ... scratch = pathlib.Path(temporary) + ... (scratch / "fuzzer").mkdir() + ... write_json_atomic( + ... scratch / "fuzzer" / "heartbeat.json", + ... {"schema_version": 1, "run_id": "run-7", "state": "active", + ... "epoch": 3}, + ... ) + ... _heartbeat_epoch(types.SimpleNamespace(scratch=scratch, run_id="run-7")) + ('active', 3) Parameters ---------- @@ -2627,8 +3104,11 @@ def release_activity_gate(context: RunContext) -> int: run marker establishes one exact stabilization target across streams whose ordinary frame prefixes otherwise differ by mode. - >>> release_activity_gate.__name__ - 'release_activity_gate' + >>> try: + ... release_activity_gate(types.SimpleNamespace(topology_verified=False)) + ... except RuntimeError as error: + ... print(error) + activity cannot start before exact topology verification Parameters ---------- @@ -2681,8 +3161,11 @@ def verify_activity_sync( ) -> int: """Poll typed pane captures until every pane contains the released marker. - >>> verify_activity_sync.__name__ - 'verify_activity_sync' + >>> try: + ... verify_activity_sync(types.SimpleNamespace(mode=ExecutionMode.ASYNC)) + ... except ValueError as error: + ... print(error) + sync activity verification requires a synchronous context Parameters ---------- @@ -2702,6 +3185,8 @@ def verify_activity_sync( Raises ------ + ValueError + If the context mode or timeout values are invalid. RuntimeError If the gate is absent, the fuzzer exits, or heartbeat epochs regress. TimeoutError @@ -2776,8 +3261,12 @@ async def verify_activity_async( ) -> int: """Async sibling of :func:`verify_activity_sync` over typed captures. - >>> verify_activity_async.__name__ - 'verify_activity_async' + >>> wrong_mode = types.SimpleNamespace(mode=ExecutionMode.SYNC) + >>> try: + ... asyncio.run(verify_activity_async(wrong_mode)) + ... except ValueError as error: + ... print(error) + async activity verification requires an asynchronous context Parameters ---------- @@ -2797,6 +3286,8 @@ async def verify_activity_async( Raises ------ + ValueError + If the context mode or timeout values are invalid. RuntimeError If the gate is absent, the fuzzer exits, or heartbeat epochs regress. TimeoutError @@ -2914,8 +3405,11 @@ async def cleanup_run( every recorded identity. Escalation sends signals only after re-reading an equal procfs start time. - >>> cleanup_run.__name__ - 'cleanup_run' + >>> try: + ... asyncio.run(cleanup_run(types.SimpleNamespace(), grace_s=0)) + ... except ValueError as error: + ... print(error) + cleanup grace must be positive Parameters ---------- @@ -2928,6 +3422,11 @@ async def cleanup_run( ------- CleanupReport Complete only when all identities and filesystem resources are absent. + + Raises + ------ + ValueError + If the cleanup grace is not positive. """ if grace_s <= 0: message = "cleanup grace must be positive" @@ -2997,10 +3496,17 @@ async def cleanup_run( shutil.rmtree(context.scratch) except OSError as error: errors.append(f"scratch removal: {type(error).__name__}: {error}") + try: + if context.socket_root is not None and context.socket_root.exists(): + shutil.rmtree(context.socket_root) + except OSError as error: + errors.append(f"socket root removal: {type(error).__name__}: {error}") if context.socket_path.exists(): errors.append(f"socket remains: {context.socket_path}") if context.scratch.exists(): errors.append(f"scratch remains: {context.scratch}") + if context.socket_root is not None and context.socket_root.exists(): + errors.append(f"socket root remains: {context.socket_root}") for identity in context.processes: if process_identity_matches(identity) and not any( f"pid {identity.pid} " in error for error in errors diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index 1c80f04c4e..97e258b0e7 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -8,6 +8,8 @@ import json import os import pathlib +import signal +import stat import subprocess import sys import types @@ -15,6 +17,8 @@ import pytest +from libtmux.experimental.models import PaneSnapshot, SessionSnapshot, WindowSnapshot + @pytest.fixture() def benchmark_module() -> types.ModuleType: @@ -852,6 +856,359 @@ def test_live_topology_setup_preserves_preexisting_scratch( assert marker.read_text(encoding="utf-8") == "keep\n" +def test_owned_process_cleanup_kills_sigterm_ignoring_child( + benchmark_module: types.ModuleType, +) -> None: + """A child that ignores graceful stop must be identity-killed and reaped.""" + process = subprocess.Popen( + ( + sys.executable, + "-c", + ( + "import signal,sys,time; " + "signal.signal(signal.SIGTERM, signal.SIG_IGN); " + "print('ready', flush=True); time.sleep(30)" + ), + ), + stdout=subprocess.PIPE, + text=True, + ) + try: + assert process.stdout is not None + assert process.stdout.readline() == "ready\n" + identity = benchmark_module._record_process("test-child", process.pid) + + report = benchmark_module._stop_owned_process( + process, + identity, + grace_s=0.05, + ) + + assert report.complete + assert report.errors == () + assert process.poll() == -signal.SIGKILL + assert not benchmark_module.process_identity_matches(identity) + finally: + if process.poll() is None: + process.kill() + process.wait(timeout=2.0) + + +def test_start_fuzzer_reaps_child_after_readiness_timeout( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """A pre-return timeout must leave no fuzzer child for the caller to own.""" + run_id = f"readiness-timeout-{os.getpid()}" + + with pytest.raises(RuntimeError, match="did not become ready"): + benchmark_module.start_fuzzer( + tmp_path, + run_id, + ready_timeout_s=1e-9, + ) + + processes = subprocess.run( + ("ps", "-eo", "args="), + check=True, + capture_output=True, + text=True, + ).stdout + assert run_id not in processes + + +def test_prepare_context_keeps_start_fuzzer_identity_for_partial_cleanup( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Later setup must reuse the identity captured while starting the child.""" + real_record_process = benchmark_module._record_process + real_start_fuzzer = benchmark_module.start_fuzzer + fuzzer_identities: list[t.Any] = [] + spawned: list[subprocess.Popen[bytes]] = [] + context = None + + def record_fuzzer_once(role: str, pid: int) -> t.Any: + if role == "fuzzer": + if fuzzer_identities: + message = "fuzzer identity was captured twice" + raise RuntimeError(message) + identity = real_record_process(role, pid) + fuzzer_identities.append(identity) + return identity + return real_record_process(role, pid) + + def capture_fuzzer(*args: t.Any, **kwargs: t.Any) -> subprocess.Popen[bytes]: + process = t.cast( + subprocess.Popen[bytes], + real_start_fuzzer(*args, **kwargs), + ) + spawned.append(process) + return process + + monkeypatch.setattr(benchmark_module, "_record_process", record_fuzzer_once) + monkeypatch.setattr(benchmark_module, "start_fuzzer", capture_fuzzer) + try: + context = benchmark_module._prepare_context( + benchmark_module.Topology(1, 1, 1), + benchmark_module.EngineLane.SUBPROCESS, + benchmark_module.ExecutionMode.SYNC, + tmp_path / "identity-owner", + socket_path=None, + run_id="identity-owner", + delayed_ordinal=0, + ) + + assert len(fuzzer_identities) == 1 + assert context.processes == (fuzzer_identities[0],) + finally: + if context is not None: + report = asyncio.run(benchmark_module.cleanup_run(context)) + assert report.complete, report.errors + elif spawned: + report = benchmark_module._stop_owned_process( + spawned[0], + fuzzer_identities[0], + ) + assert report.complete, report.errors + + +@pytest.mark.parametrize("mode_name", ("sync", "async")) +def test_setup_surfaces_original_and_cleanup_failures( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, + mode_name: str, +) -> None: + """A partial setup must retain its cause and unsuccessful cleanup evidence.""" + captured_contexts: list[t.Any] = [] + real_cleanup = benchmark_module.cleanup_run + + def fail_verification(_context: t.Any, _snapshot: t.Any) -> t.NoReturn: + message = "injected setup failure" + raise RuntimeError(message) + + async def cleanup_with_injected_failure(context: t.Any) -> t.Any: + captured_contexts.append(context) + report = await real_cleanup(context) + assert report.complete, report.errors + return benchmark_module.CleanupReport( + complete=False, + errors=("injected cleanup failure",), + ) + + monkeypatch.setattr(benchmark_module, "verify_topology", fail_verification) + monkeypatch.setattr( + benchmark_module, + "cleanup_run", + cleanup_with_injected_failure, + ) + scratch = tmp_path / mode_name + + with pytest.raises(benchmark_module.SetupCleanupError) as raised: + if mode_name == "sync": + benchmark_module.setup_sync( + benchmark_module.Topology(1, 1, 1), + benchmark_module.EngineLane.SUBPROCESS, + scratch, + run_id=f"cleanup-evidence-{mode_name}", + ) + else: + asyncio.run( + benchmark_module.setup_async( + benchmark_module.Topology(1, 1, 1), + benchmark_module.EngineLane.SUBPROCESS, + scratch, + run_id=f"cleanup-evidence-{mode_name}", + ) + ) + + error = raised.value + assert isinstance(error.setup_error, RuntimeError) + assert str(error.setup_error) == "injected setup failure" + assert error.__cause__ is error.setup_error + assert error.cleanup_report.errors == ("injected cleanup failure",) + assert len(captured_contexts) == 1 + assert not scratch.exists() + assert all( + not benchmark_module.process_identity_matches(identity) + for identity in captured_contexts[0].processes + ) + + +def test_setup_acquires_private_scratch_under_permissive_umask( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """Run scratch must be mode 0700 even when the ambient umask permits 0777.""" + scratch = tmp_path / "private-scratch" + context = None + previous_umask = os.umask(0) + try: + context = benchmark_module.setup_sync( + benchmark_module.Topology(1, 1, 1), + benchmark_module.EngineLane.SUBPROCESS, + scratch, + run_id="private-scratch", + ) + finally: + os.umask(previous_umask) + try: + assert stat.S_IMODE(scratch.stat().st_mode) == 0o700 + finally: + if context is not None: + cleanup = asyncio.run(benchmark_module.cleanup_run(context)) + assert cleanup.complete, cleanup.errors + + +def test_socket_path_limit_counts_encoded_bytes( + benchmark_module: types.ModuleType, +) -> None: + """The pathname ceiling must reserve the sockaddr_un terminator byte.""" + at_limit = pathlib.Path("/" + "é" * 53) + over_limit = pathlib.Path("/" + "é" * 53 + "s") + assert len(os.fsencode(at_limit)) == 107 + assert len(os.fsencode(over_limit)) == 108 + + assert benchmark_module._validate_socket_path(at_limit) == at_limit + with pytest.raises(ValueError, match="107 encoded bytes"): + benchmark_module._validate_socket_path(over_limit) + + +def test_explicit_long_socket_is_rejected_before_owned_side_effects( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """An invalid explicit socket must not start a fuzzer or acquire scratch.""" + scratch = tmp_path / "must-not-exist" + socket_path = scratch / ("s" * 200) + + def unexpected_fuzzer(*_args: object, **_kwargs: object) -> t.NoReturn: + message = "fuzzer started before socket validation" + raise AssertionError(message) + + monkeypatch.setattr(benchmark_module, "start_fuzzer", unexpected_fuzzer) + + with pytest.raises(ValueError, match="107 encoded bytes"): + benchmark_module.setup_sync( + benchmark_module.Topology(1, 1, 1), + benchmark_module.EngineLane.SUBPROCESS, + scratch, + socket_path=socket_path, + run_id="long-socket", + ) + + assert not scratch.exists() + + +def test_default_socket_uses_short_owned_root_for_long_scratch( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """An owned socket remains ABI-safe when the caller's scratch path is long.""" + scratch = tmp_path / ("long-scratch-" + "s" * 100) + context = benchmark_module.setup_sync( + benchmark_module.Topology(1, 1, 1), + benchmark_module.EngineLane.SUBPROCESS, + scratch, + run_id="short-owned-socket", + ) + socket_root = context.socket_root + try: + assert len(os.fsencode(context.socket_path)) <= 107 + assert socket_root is not None + assert context.socket_path.parent == socket_root + assert not context.socket_path.is_relative_to(scratch) + finally: + cleanup = asyncio.run(benchmark_module.cleanup_run(context)) + assert cleanup.complete, cleanup.errors + assert not socket_root.exists() + + +def test_verify_topology_rejects_swapped_window_parents( + benchmark_module: types.ModuleType, +) -> None: + """Correct per-session counts must not hide windows attached to wrong parents.""" + session_names = ("bench-parent-s000", "bench-parent-s001") + window_names = ( + "bench-parent-s000-w000", + "bench-parent-s001-w000", + ) + context = types.SimpleNamespace( + topology=benchmark_module.Topology(2, 1, 1), + expected_session_names=session_names, + expected_window_names=window_names, + expected_window_parents=( + (window_names[0], session_names[0]), + (window_names[1], session_names[1]), + ), + streams=(pathlib.Path("editor.log"), pathlib.Path("delayed-match.log")), + ) + snapshot = benchmark_module.TopologySnapshot( + sessions=( + SessionSnapshot(session_id="$0", name=session_names[0]), + SessionSnapshot(session_id="$1", name=session_names[1]), + ), + windows=( + WindowSnapshot(window_id="@0", name=window_names[0], session_id="$1"), + WindowSnapshot(window_id="@1", name=window_names[1], session_id="$0"), + ), + panes=( + PaneSnapshot(pane_id="%0", window_id="@0", session_id="$1", pid=9_999_991), + PaneSnapshot(pane_id="%1", window_id="@1", session_id="$0", pid=9_999_992), + ), + ) + + with pytest.raises( + benchmark_module.TopologyVerificationError, + match="window parent differs from the declaration", + ): + benchmark_module.verify_topology(context, snapshot) + + +def test_verify_topology_rejects_pane_session_parent_mismatch( + benchmark_module: types.ModuleType, +) -> None: + """A pane must name the same session as the window that owns its ID.""" + session_names = ("bench-parent-s000", "bench-parent-s001") + window_names = ( + "bench-parent-s000-w000", + "bench-parent-s001-w000", + ) + context = types.SimpleNamespace( + topology=benchmark_module.Topology(2, 1, 1), + expected_session_names=session_names, + expected_window_names=window_names, + expected_window_parents=( + (window_names[0], session_names[0]), + (window_names[1], session_names[1]), + ), + streams=(pathlib.Path("editor.log"), pathlib.Path("delayed-match.log")), + ) + snapshot = benchmark_module.TopologySnapshot( + sessions=( + SessionSnapshot(session_id="$0", name=session_names[0]), + SessionSnapshot(session_id="$1", name=session_names[1]), + ), + windows=( + WindowSnapshot(window_id="@0", name=window_names[0], session_id="$0"), + WindowSnapshot(window_id="@1", name=window_names[1], session_id="$1"), + ), + panes=( + PaneSnapshot(pane_id="%0", window_id="@0", session_id="$1", pid=9_999_991), + PaneSnapshot(pane_id="%1", window_id="@1", session_id="$0", pid=9_999_992), + ), + ) + + with pytest.raises( + benchmark_module.TopologyVerificationError, + match="pane session differs from its window parent", + ): + benchmark_module.verify_topology(context, snapshot) + + @pytest.mark.parametrize( ("lane_name", "mode_name"), ( From e47e16d8e163ee58f79f3a4c9c16e1cb627da837 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 08:22:17 -0500 Subject: [PATCH 14/67] Bench(fix[cleanup]): Roll back private dirs why: A chmod or stat failure after mkdir could bypass caller ownership flags and leak a scratch or socket directory. what: - Roll back newly created directories with exact Path.rmdir calls - Preserve acquisition and rollback failures in a typed exception - Cover scratch and socket-root chmod and stat failure paths --- scripts/bench_orchestration.py | 111 ++++++++++++++++++-- tests/test_bench_orchestration_script.py | 127 +++++++++++++++++++++++ 2 files changed, 231 insertions(+), 7 deletions(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index 3f012a8209..351681e38e 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -980,6 +980,63 @@ def __init__( ) +class PrivateDirectoryAcquisitionError(RuntimeError): + """A private directory failed acquisition and exact rollback. + + Attributes + ---------- + directory : pathlib.Path + Exact newly created directory whose rollback failed. + acquisition_error : BaseException + Original chmod, stat, or mode-verification failure. + rollback_error : BaseException + Failure raised by exact non-recursive ``Path.rmdir`` rollback. + + Examples + -------- + >>> acquire = OSError("chmod") + >>> rollback = OSError("not empty") + >>> error = PrivateDirectoryAcquisitionError( + ... pathlib.Path("run"), acquire, rollback + ... ) + >>> error.acquisition_error is acquire, error.rollback_error is rollback + (True, True) + """ + + def __init__( + self, + directory: pathlib.Path, + acquisition_error: BaseException, + rollback_error: BaseException, + ) -> None: + """Retain the exact directory and both acquisition failures. + + >>> error = PrivateDirectoryAcquisitionError( + ... pathlib.Path("run"), OSError("stat"), OSError("rmdir") + ... ) + >>> "acquisition failed for run" in str(error) + True + + Parameters + ---------- + directory : pathlib.Path + Exact directory created by the failed acquisition. + acquisition_error : BaseException + Failure that prevented mode-0700 verification. + rollback_error : BaseException + Failure from exact non-recursive rollback. + """ + self.directory = directory + self.acquisition_error = acquisition_error + self.rollback_error = rollback_error + super().__init__( + f"private directory acquisition failed for {directory}: " + f"{type(acquisition_error).__name__}: {acquisition_error}; " + "exact rollback failed: " + f"{type(rollback_error).__name__}: {rollback_error}" + ) + + @dataclasses.dataclass(frozen=True) class ProcessIdentity: """One benchmark-owned process protected against PID reuse. @@ -1795,14 +1852,54 @@ def _acquire_private_directory(directory: pathlib.Path) -> pathlib.Path: If another owner already created the directory. OSError If mode 0700 cannot be applied or verified. + PrivateDirectoryAcquisitionError + If mode acquisition and exact non-recursive rollback both fail. """ directory.mkdir(mode=0o700, parents=True, exist_ok=False) - directory.chmod(0o700) + try: + directory.chmod(0o700) + _verify_private_directory_mode(directory) + except BaseException as acquisition_error: + try: + directory.rmdir() + except Exception as rollback_error: # noqa: BLE001 + raise PrivateDirectoryAcquisitionError( + directory, + acquisition_error, + rollback_error, + ) from acquisition_error + raise + else: + return directory + + +def _verify_private_directory_mode(directory: pathlib.Path) -> None: + """Require one acquired directory to have exact mode 0700. + + >>> with tempfile.TemporaryDirectory() as temporary: + ... directory = pathlib.Path(temporary) + ... directory.chmod(0o700) + ... _verify_private_directory_mode(directory) + + Parameters + ---------- + directory : pathlib.Path + Existing directory whose effective permission bits are verified. + + Returns + ------- + None + After exact mode 0700 is observed. + + Raises + ------ + OSError + If stat fails or the observed permission bits differ from 0700. + """ observed_mode = stat.S_IMODE(directory.stat().st_mode) if observed_mode != 0o700: message = f"private run directory mode is {oct(observed_mode)}, expected 0o700" raise OSError(message) - return directory def _stop_owned_process( @@ -2502,11 +2599,11 @@ def _prepare_context( os.environ.pop(name, None) else: os.environ[name] = value - cleanup_report = CleanupReport( - complete=not cleanup_errors, - errors=tuple(cleanup_errors), - ) - if not cleanup_report.complete: + if cleanup_errors: + cleanup_report = CleanupReport( + complete=False, + errors=tuple(cleanup_errors), + ) raise SetupCleanupError(setup_error, cleanup_report) from setup_error raise diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index 97e258b0e7..3efb4a8332 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -1062,6 +1062,133 @@ def test_setup_acquires_private_scratch_under_permissive_umask( assert cleanup.complete, cleanup.errors +@pytest.mark.parametrize( + ("target_kind", "failure_kind"), + ( + ("scratch", "chmod"), + ("scratch", "stat"), + ("socket-root", "chmod"), + ("socket-root", "stat"), + ), +) +def test_private_directory_acquisition_rolls_back_post_mkdir_failure( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, + target_kind: str, + failure_kind: str, +) -> None: + """A post-mkdir acquisition failure must remove the exact new directory.""" + token = f"{target_kind[:3]}{failure_kind[:4]}0" + scratch = tmp_path / f"{target_kind}-{failure_kind}" + socket_root = pathlib.Path(benchmark_module.tempfile.gettempdir()) / ( + f"libtmux-bench-{os.getpid():x}-{token}" + ) + target = scratch if target_kind == "scratch" else socket_root + real_chmod = pathlib.Path.chmod + real_stat = pathlib.Path.stat + injected = False + + def fail_target_chmod(path: pathlib.Path, *args: t.Any, **kwargs: t.Any) -> None: + nonlocal injected + if path == target: + injected = True + message = f"injected {target_kind} chmod failure" + raise OSError(message) + real_chmod(path, *args, **kwargs) + + def fail_target_stat( + path: pathlib.Path, + *args: t.Any, + **kwargs: t.Any, + ) -> os.stat_result: + nonlocal injected + if path == target and not injected: + injected = True + message = f"injected {target_kind} stat failure" + raise OSError(message) + return real_stat(path, *args, **kwargs) + + def unexpected_fuzzer(*_args: object, **_kwargs: object) -> t.NoReturn: + message = "fuzzer started after failed directory acquisition" + raise AssertionError(message) + + monkeypatch.setattr( + benchmark_module.uuid, + "uuid4", + lambda: types.SimpleNamespace(hex=token), + ) + monkeypatch.setattr(benchmark_module, "start_fuzzer", unexpected_fuzzer) + if failure_kind == "chmod": + monkeypatch.setattr(pathlib.Path, "chmod", fail_target_chmod) + else: + monkeypatch.setattr(pathlib.Path, "stat", fail_target_stat) + + try: + with pytest.raises( + OSError, + match=f"injected {target_kind} {failure_kind} failure", + ): + benchmark_module._prepare_context( + benchmark_module.Topology(1, 1, 1), + benchmark_module.EngineLane.SUBPROCESS, + benchmark_module.ExecutionMode.SYNC, + scratch, + socket_path=None, + run_id=f"acquire-{target_kind}-{failure_kind}", + delayed_ordinal=0, + ) + + assert injected + assert not scratch.exists() + assert not socket_root.exists() + finally: + if socket_root.exists(): + socket_root.rmdir() + if scratch.exists(): + scratch.rmdir() + + +def test_private_directory_acquisition_surfaces_exact_rollback_failure( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A nonempty rollback must preserve its path and both failure objects.""" + directory = tmp_path / "rollback-failure" + marker = directory / "do-not-delete" + acquisition_error = OSError("injected chmod failure") + real_chmod = pathlib.Path.chmod + + def fail_after_writing_marker( + path: pathlib.Path, + *args: t.Any, + **kwargs: t.Any, + ) -> None: + if path == directory: + marker.write_text("preserved\n", encoding="utf-8") + raise acquisition_error + real_chmod(path, *args, **kwargs) + + monkeypatch.setattr(pathlib.Path, "chmod", fail_after_writing_marker) + try: + with pytest.raises(RuntimeError) as raised: + benchmark_module._acquire_private_directory(directory) + + error = raised.value + assert isinstance(error, benchmark_module.PrivateDirectoryAcquisitionError) + assert error.directory == directory + assert error.acquisition_error is acquisition_error + assert isinstance(error.rollback_error, OSError) + assert error.__cause__ is acquisition_error + assert marker.read_text(encoding="utf-8") == "preserved\n" + finally: + if marker.exists(): + marker.unlink() + if directory.exists(): + directory.rmdir() + + def test_socket_path_limit_counts_encoded_bytes( benchmark_module: types.ModuleType, ) -> None: From c40c1c50a19bea037cd00661f7db5cf6a9de82ed Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 08:48:21 -0500 Subject: [PATCH 15/67] Bench(feat[phases]): Measure active queries why: Measure active bulk mutation and query strategies against one persistent verified tmux topology. what: - Add typed mutation, enumeration, capture, and search phase results - Interleave repeatable strategies and reject invalid timing samples - Cover sync and async subprocess and control-mode phase parity --- scripts/bench_orchestration.py | 2040 ++++++++++++++++++++++ tests/test_bench_orchestration_script.py | 445 +++++ 2 files changed, 2485 insertions(+) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index 351681e38e..809ccce969 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -17,10 +17,13 @@ import contextlib import dataclasses import enum +import hashlib +import inspect import json import math import os import pathlib +import random import resource import shlex import shutil @@ -43,6 +46,8 @@ SessionSnapshot, WindowSnapshot, ) + from libtmux.experimental.ops.operation import Operation + from libtmux.experimental.ops.plan import LazyPlan from libtmux.experimental.workspace import WorkspaceSet from libtmux.server import Server @@ -862,12 +867,314 @@ class RawSample: Failure detail for a rejected sample. verified : bool Whether the phase's correctness check accepted this timing. + strategy : str | None + Strategy that produced the accepted sample. + ordinal : int | None + Zero-based timed ordinal within that strategy. + resources_before : HostSnapshot | None + Resource observation immediately before execution. + resources_after : HostSnapshot | None + Resource observation after typed and live validation. + + Examples + -------- + >>> RawSample(7, True, verified=True, strategy="serial").duration_ns + 7 """ duration_ns: int | None accepted: bool error: str | None = None verified: bool = False + strategy: str | None = None + ordinal: int | None = None + resources_before: HostSnapshot | None = None + resources_after: HostSnapshot | None = None + + +@dataclasses.dataclass(frozen=True) +class ExecutionMetrics: + """Logical request and transport work for one phase measurement. + + Attributes + ---------- + operations : int + Typed operations in the phase's measured plan or cell. + planner_steps : int + Planner steps used to dispatch those operations. + engine_batches : int + Engine dispatch calls made for the measured operations. + tmux_requests : int + Individually attributable tmux requests. + process_starts : int + Subprocess transport starts, one per request; zero for control mode. + + Examples + -------- + >>> ExecutionMetrics(12, 1, 1, 12, 0).tmux_requests + 12 + """ + + operations: int + planner_steps: int + engine_batches: int + tmux_requests: int + process_starts: int + + +@dataclasses.dataclass(frozen=True) +class MutationResult: + """Verified and restored bulk mutation measurement. + + Attributes + ---------- + duration_ns : int + Mutation plan duration excluding verification and restoration. + metrics : ExecutionMetrics + Logical operation, planning, request, and transport counts. + session_id : str + Concrete session ID selected by maximum window cardinality. + window_ids : tuple[str, ...] + Concrete window IDs renamed by the plan. + pane_ids : tuple[str, ...] + Concrete pane IDs titled by the plan. + generation : str + Exact generation option value set during verification. + verified : bool + Whether all mutated live values matched the plan. + restored : bool + Whether canonical names, titles, and option state were restored. + activity_before_epoch : int + Fuzzer heartbeat epoch before the mutation. + activity_after_epoch : int + Later heartbeat epoch observed after restoration. + + Examples + -------- + >>> result = MutationResult( + ... 5, ExecutionMetrics(3, 1, 1, 3, 0), "$1", ("@2",), ("%3",), + ... "7", True, True, 4, 5, + ... ) + >>> result.restored and result.activity_after_epoch > result.activity_before_epoch + True + """ + + duration_ns: int + metrics: ExecutionMetrics + session_id: str + window_ids: tuple[str, ...] + pane_ids: tuple[str, ...] + generation: str + verified: bool + restored: bool + activity_before_epoch: int + activity_after_epoch: int + + +EnumerationKind: t.TypeAlias = t.Literal["sessions", "windows", "panes"] + + +@dataclasses.dataclass(frozen=True) +class EnumerationResult: + """One exact typed hierarchy enumeration. + + Attributes + ---------- + duration_ns : int + Time spent executing and parsing the typed list operation. + metrics : ExecutionMetrics + One-operation dispatch counts for this list level. + kind : {"sessions", "windows", "panes"} + Hierarchy level enumerated. + row_count : int + Exact number of typed rows returned. + ids : tuple[str, ...] + Concrete IDs in verified stable order. + id_checksum : str + SHA-256 over NUL-separated concrete IDs. + verified : bool + Whether row count, IDs, and checksum match the live run context. + + Examples + -------- + >>> EnumerationResult( + ... 3, ExecutionMetrics(1, 1, 1, 1, 0), "sessions", 1, + ... ("$0",), "digest", True, + ... ).row_count + 1 + """ + + duration_ns: int + metrics: ExecutionMetrics + kind: EnumerationKind + row_count: int + ids: tuple[str, ...] + id_checksum: str + verified: bool + + +@dataclasses.dataclass(frozen=True) +class PaneCapture: + """Retained typed capture lines for one concrete pane. + + Attributes + ---------- + pane_id : str + Concrete pane ID captured. + lines : tuple[str, ...] + Typed capture lines in display order. + + Examples + -------- + >>> PaneCapture("%1", ("line",)).lines + ('line',) + """ + + pane_id: str + lines: tuple[str, ...] + + +CaptureStrategy: t.TypeAlias = t.Literal["serial", "batched"] + + +@dataclasses.dataclass(frozen=True) +class CaptureResult: + """All-pane capture measurement retaining content for later search. + + Attributes + ---------- + duration_ns : int + Time spent executing the identical capture operation graph. + metrics : ExecutionMetrics + Planner and transport counts for the selected strategy. + strategy : {"serial", "batched"} + Planner policy used for the operation graph. + operations : tuple[Operation, ...] + Exact immutable capture operations used by the planner. + captures : tuple[PaneCapture, ...] + Typed content associated with each concrete pane ID. + line_count : int + Total typed lines returned across panes. + byte_count : int + Total UTF-8 bytes in those lines, excluding removed delimiters. + epoch : int + Current activity epoch found in every pane. + verified : bool + Whether every capture succeeded and contained the current marker. + + Examples + -------- + >>> CaptureResult( + ... 2, ExecutionMetrics(1, 1, 1, 1, 0), "batched", (), + ... (PaneCapture("%1", ("x",)),), 1, 1, 4, True, + ... ).byte_count + 1 + """ + + duration_ns: int + metrics: ExecutionMetrics + strategy: CaptureStrategy + operations: tuple[Operation[t.Any], ...] + captures: tuple[PaneCapture, ...] + line_count: int + byte_count: int + epoch: int + verified: bool + + +SearchFamily: t.TypeAlias = t.Literal[ + "server-side", "snapshot", "end-to-end", "contents" +] + + +@dataclasses.dataclass(frozen=True) +class SearchResult: + """One semantically explicit metadata or retained-content search. + + Attributes + ---------- + duration_ns : int + Time spent in the named search family only. + family : {"server-side", "snapshot", "end-to-end", "contents"} + Search source and timing boundary. + kind : {"sessions", "windows", "panes"} + Object kind scanned or returned. + scanned_count : int + Candidate rows or panes scanned. + target : str + Exact concrete ID required from the result. + matched_ids : tuple[str, ...] + Exact concrete IDs returned by the search. + token : str | None + Exact retained content token, only for content search. + verified : bool + Whether exactly the requested ID or token matched. + + Examples + -------- + >>> SearchResult(4, "snapshot", "panes", 2, "%1", ("%1",), verified=True).verified + True + """ + + duration_ns: int + family: SearchFamily + kind: EnumerationKind + scanned_count: int + target: str + matched_ids: tuple[str, ...] + token: str | None = None + verified: bool = False + + +@dataclasses.dataclass(frozen=True) +class RepeatablePhaseFailure: + """Failure metadata that terminates deterministic phase execution. + + Attributes + ---------- + stage : {"warmup", "timed"} + Whether failure happened before or during timed sampling. + strategy : str + Strategy whose callable or postcondition failed. + ordinal : int + Zero-based ordinal within the stage. + error : str + Exception type and message without a fabricated duration. + + Examples + -------- + >>> RepeatablePhaseFailure("timed", "serial", 2, "RuntimeError: failed").ordinal + 2 + """ + + stage: t.Literal["warmup", "timed"] + strategy: str + ordinal: int + error: str + + +@dataclasses.dataclass(frozen=True) +class RepeatablePhaseResult: + """Accepted samples, deterministic call order, and terminal failure. + + Attributes + ---------- + samples : tuple[RawSample, ...] + Accepted timed rows only; warmups and failures never appear. + order : tuple[str, ...] + Strategy names in actual warmup-then-timed invocation order. + failure : RepeatablePhaseFailure | None + First failure metadata, or ``None`` when every invocation passed. + + Examples + -------- + >>> RepeatablePhaseResult((), ("serial",), None).failure is None + True + """ + + samples: tuple[RawSample, ...] + order: tuple[str, ...] + failure: RepeatablePhaseFailure | None = None @dataclasses.dataclass(frozen=True) @@ -3453,6 +3760,1739 @@ async def verify_activity_async( return context.activity_epoch +_GENERATION_OPTION = "@libtmux_bench_generation" + + +def _phase_metrics( + context: RunContext, + *, + operations: int, + planner_steps: int, +) -> ExecutionMetrics: + """Derive comparable logical and transport counts for one measured cell. + + >>> context = types.SimpleNamespace(lane=EngineLane.SUBPROCESS) + >>> _phase_metrics(context, operations=4, planner_steps=1).process_starts + 4 + + Parameters + ---------- + context : RunContext + Live run whose transport determines modeled process starts. + operations : int + Individually attributable tmux requests. + planner_steps : int + Engine dispatch groups selected by the planner. + + Returns + ------- + ExecutionMetrics + Separate operation, planning, request, and process-start quantities. + """ + return ExecutionMetrics( + operations=operations, + planner_steps=planner_steps, + engine_batches=planner_steps, + tmux_requests=operations, + process_starts=(operations if context.lane is EngineLane.SUBPROCESS else 0), + ) + + +def _require_active_phase_context( + context: RunContext, + mode: ExecutionMode, +) -> None: + """Reject phase work before exact topology and activity verification. + + >>> inactive = types.SimpleNamespace( + ... mode=ExecutionMode.SYNC, topology_verified=False, + ... activity_epoch=None, activity_marker=None, + ... ) + >>> try: + ... _require_active_phase_context(inactive, ExecutionMode.SYNC) + ... except RuntimeError as error: + ... print(error) + measured phases require verified active topology + + Parameters + ---------- + context : RunContext + Candidate live run context. + mode : ExecutionMode + Required synchronous or asynchronous execution mode. + + Raises + ------ + ValueError + If the context belongs to the other execution mode. + RuntimeError + If topology or activity stabilization has not completed. + """ + if context.mode is not mode: + message = f"{mode.value} phase requires a {mode.value} run context" + raise ValueError(message) + if ( + not context.topology_verified + or context.activity_epoch is None + or context.activity_marker is None + ): + message = "measured phases require verified active topology" + raise RuntimeError(message) + + +def _current_activity_epoch(context: RunContext) -> int: + """Return and retain an active, non-regressing fuzzer heartbeat epoch. + + >>> context = types.SimpleNamespace(fuzzer=types.SimpleNamespace(poll=lambda: 1)) + >>> try: + ... _current_activity_epoch(context) + ... except RuntimeError as error: + ... print(error) + fuzzer exited during measured phase + + Parameters + ---------- + context : RunContext + Active run whose heartbeat is authoritative. + + Returns + ------- + int + Current active heartbeat epoch. + + Raises + ------ + RuntimeError + If the fuzzer exited, heartbeat is inactive, or its epoch regressed. + """ + if context.fuzzer.poll() is not None: + message = "fuzzer exited during measured phase" + raise RuntimeError(message) + state, epoch = _heartbeat_epoch(context) + if state != "active" or epoch is None: + message = "fuzzer heartbeat is not active during measured phase" + raise RuntimeError(message) + if epoch < context.heartbeat_epoch: + message = "fuzzer heartbeat epoch moved backwards" + raise RuntimeError(message) + context.heartbeat_epoch = epoch + return epoch + + +def _wait_activity_advance_sync( + context: RunContext, + baseline: int, + *, + timeout_s: float = 2.0, +) -> int: + """Wait until the live fuzzer heartbeat advances past ``baseline``. + + >>> try: + ... _wait_activity_advance_sync(types.SimpleNamespace(), 0, timeout_s=0) + ... except ValueError as error: + ... print(error) + activity advance timeout must be positive + + Parameters + ---------- + context : RunContext + Active synchronous run. + baseline : int + Epoch that the fuzzer must surpass. + timeout_s : float + Maximum monotonic wait. + + Returns + ------- + int + First observed later epoch. + + Raises + ------ + ValueError + If ``timeout_s`` is not positive. + TimeoutError + If activity does not advance before the deadline. + """ + if timeout_s <= 0: + message = "activity advance timeout must be positive" + raise ValueError(message) + deadline = time.monotonic() + timeout_s + while time.monotonic() < deadline: + epoch = _current_activity_epoch(context) + if epoch > baseline: + return epoch + time.sleep(0.005) + message = f"activity did not advance past epoch {baseline}" + raise TimeoutError(message) + + +async def _wait_activity_advance_async( + context: RunContext, + baseline: int, + *, + timeout_s: float = 2.0, +) -> int: + """Asynchronously wait for a later active heartbeat epoch. + + >>> async def invalid_wait(): + ... try: + ... await _wait_activity_advance_async( + ... types.SimpleNamespace(), 0, timeout_s=0 + ... ) + ... except ValueError as error: + ... return str(error) + >>> asyncio.run(invalid_wait()) + 'activity advance timeout must be positive' + + Parameters + ---------- + context : RunContext + Active asynchronous run. + baseline : int + Epoch that the fuzzer must surpass. + timeout_s : float + Maximum event-loop wait. + + Returns + ------- + int + First observed later epoch. + + Raises + ------ + ValueError + If ``timeout_s`` is not positive. + TimeoutError + If activity does not advance before the deadline. + """ + if timeout_s <= 0: + message = "activity advance timeout must be positive" + raise ValueError(message) + loop = asyncio.get_running_loop() + deadline = loop.time() + timeout_s + while loop.time() < deadline: + epoch = _current_activity_epoch(context) + if epoch > baseline: + return epoch + await asyncio.sleep(0.005) + message = f"activity did not advance past epoch {baseline}" + raise TimeoutError(message) + + +def _mutation_targets( + context: RunContext, + snapshot: TopologySnapshot, +) -> tuple[str, tuple[str, ...], tuple[str, ...]]: + """Select one largest session and its stable concrete descendants. + + >>> from libtmux.experimental.models import SessionSnapshot + >>> context = types.SimpleNamespace( + ... session_ids=("$0", "$1"), window_ids=(), pane_ids=() + ... ) + >>> snapshot = TopologySnapshot( + ... (SessionSnapshot(session_id="$0"), SessionSnapshot(session_id="$1")), + ... (), (), + ... ) + >>> _mutation_targets(context, snapshot) + ('$0', (), ()) + + Parameters + ---------- + context : RunContext + Verified stable ID order. + snapshot : TopologySnapshot + Current typed hierarchy ownership rows. + + Returns + ------- + tuple[str, tuple[str, ...], tuple[str, ...]] + Session ID, owned window IDs, and owned pane IDs in stable order. + """ + window_counts = dict.fromkeys(context.session_ids, 0) + for window in snapshot.windows: + if window.session_id in window_counts: + window_counts[window.session_id] += 1 + session_id = max(context.session_ids, key=window_counts.__getitem__) + window_id_set = { + window.window_id + for window in snapshot.windows + if window.session_id == session_id + } + window_ids = tuple( + window_id for window_id in context.window_ids if window_id in window_id_set + ) + pane_id_set = { + pane.pane_id for pane in snapshot.panes if pane.window_id in window_id_set + } + pane_ids = tuple(pane_id for pane_id in context.pane_ids if pane_id in pane_id_set) + return session_id, window_ids, pane_ids + + +def _build_mutation_plans( + context: RunContext, + *, + generation: str, + session_id: str, + window_ids: tuple[str, ...], + pane_ids: tuple[str, ...], +) -> tuple[LazyPlan, LazyPlan, dict[str, str], dict[str, str]]: + """Build the shared mutation graph and its untimed restoration graph. + + >>> context = types.SimpleNamespace( + ... run_id="run-7", window_bindings=(("canonical", "@1", "$0"),) + ... ) + >>> mutation, restoration, names, titles = _build_mutation_plans( + ... context, generation="3", session_id="$0", window_ids=("@1",), + ... pane_ids=("%2",), + ... ) + >>> len(mutation), len(restoration), names["@1"], titles["%2"] + (3, 3, 'canonical-g3', 'bench-run-7-g3-p000') + + Parameters + ---------- + context : RunContext + Run whose canonical window bindings and identity are authoritative. + generation : str + Exact user-option value and mutation suffix. + session_id : str + Concrete target session. + window_ids : tuple[str, ...] + Concrete windows to rename. + pane_ids : tuple[str, ...] + Concrete panes to title. + + Returns + ------- + tuple[LazyPlan, LazyPlan, dict[str, str], dict[str, str]] + Mutation, restoration, expected mutated names, and expected titles. + """ + from libtmux.experimental.ops import ( + LazyPlan, + PaneId, + RenameWindow, + SelectPane, + SessionId, + SetOption, + WindowId, + ) + + canonical_names = { + window_id: name for name, window_id, _session_id in context.window_bindings + } + mutated_names = { + window_id: f"{canonical_names[window_id]}-g{generation}" + for window_id in window_ids + } + titles = { + pane_id: f"bench-{context.run_id}-g{generation}-p{ordinal:03d}" + for ordinal, pane_id in enumerate(pane_ids) + } + mutation = LazyPlan() + mutation.add( + SetOption( + target=SessionId(session_id), + option=_GENERATION_OPTION, + value=generation, + ) + ) + for window_id in window_ids: + mutation.add( + RenameWindow( + target=WindowId(window_id), + name=mutated_names[window_id], + ) + ) + for pane_id in pane_ids: + mutation.add(SelectPane(target=PaneId(pane_id), title=titles[pane_id])) + + restoration = LazyPlan() + restoration.add( + SetOption( + target=SessionId(session_id), + option=_GENERATION_OPTION, + unset=True, + ) + ) + for window_id in window_ids: + restoration.add( + RenameWindow( + target=WindowId(window_id), + name=canonical_names[window_id], + ) + ) + for pane_id in pane_ids: + restoration.add(SelectPane(target=PaneId(pane_id), title="")) + return mutation, restoration, mutated_names, titles + + +def _verify_mutation_state( + snapshot: TopologySnapshot, + *, + generation: str, + options: cabc.Mapping[str, str], + mutated_names: cabc.Mapping[str, str], + titles: cabc.Mapping[str, str], +) -> None: + """Require every generation, window-name, and pane-title mutation. + + >>> from libtmux.experimental.models import PaneSnapshot, WindowSnapshot + >>> snapshot = TopologySnapshot( + ... (), (WindowSnapshot(window_id="@1", name="renamed"),), + ... (PaneSnapshot(pane_id="%1", title="t", fields={"pane_title": "t"}),), + ... ) + >>> _verify_mutation_state( + ... snapshot, generation="4", options={_GENERATION_OPTION: "4"}, + ... mutated_names={"@1": "renamed"}, titles={"%1": "t"}, + ... ) + + Parameters + ---------- + snapshot : TopologySnapshot + Typed live state read after the timed plan. + generation : str + Exact expected generation option value. + options : collections.abc.Mapping[str, str] + Typed ``ShowOptions`` mapping for the selected session. + mutated_names : collections.abc.Mapping[str, str] + Expected window names keyed by concrete ID. + titles : collections.abc.Mapping[str, str] + Expected pane titles keyed by concrete ID. + + Raises + ------ + RuntimeError + If any mutation is absent or partial. + """ + if options.get(_GENERATION_OPTION) != generation: + message = "generation option did not match the mutation" + raise RuntimeError(message) + observed_names = {window.window_id: window.name for window in snapshot.windows} + if any(observed_names.get(key) != value for key, value in mutated_names.items()): + message = "window rename verification failed" + raise RuntimeError(message) + observed_titles = { + pane.pane_id: pane.fields.get("pane_title", pane.title) + for pane in snapshot.panes + } + if any(observed_titles.get(key) != value for key, value in titles.items()): + message = "pane title verification failed" + raise RuntimeError(message) + + +def _verify_restored_state( + context: RunContext, + snapshot: TopologySnapshot, + *, + options: cabc.Mapping[str, str], + window_ids: tuple[str, ...], + pane_ids: tuple[str, ...], +) -> None: + """Require canonical names, cleared titles, and an absent generation option. + + >>> from libtmux.experimental.models import PaneSnapshot, WindowSnapshot + >>> context = types.SimpleNamespace( + ... window_bindings=(("canonical", "@1", "$0"),) + ... ) + >>> snapshot = TopologySnapshot( + ... (), (WindowSnapshot(window_id="@1", name="canonical"),), + ... (PaneSnapshot(pane_id="%1", fields={"pane_title": ""}),), + ... ) + >>> _verify_restored_state( + ... context, snapshot, options={}, window_ids=("@1",), + ... pane_ids=("%1",), + ... ) + + Parameters + ---------- + context : RunContext + Run containing canonical window bindings. + snapshot : TopologySnapshot + Typed live state after the restoration plan. + options : collections.abc.Mapping[str, str] + Typed options after unsetting the generation marker. + window_ids : tuple[str, ...] + Windows that must have canonical names. + pane_ids : tuple[str, ...] + Panes whose explicit titles must be absent or empty. + + Raises + ------ + RuntimeError + If restoration is absent or partial. + """ + if _GENERATION_OPTION in options: + message = "generation option remained after restoration" + raise RuntimeError(message) + canonical_names = { + window_id: name for name, window_id, _session_id in context.window_bindings + } + observed_names = {window.window_id: window.name for window in snapshot.windows} + if any( + observed_names.get(window_id) != canonical_names[window_id] + for window_id in window_ids + ): + message = "canonical window restoration failed" + raise RuntimeError(message) + observed_titles = { + pane.pane_id: pane.fields.get("pane_title", pane.title) + for pane in snapshot.panes + } + title_mismatches = { + pane_id: observed_titles.get(pane_id) + for pane_id in pane_ids + if observed_titles.get(pane_id) not in {None, ""} + } + if title_mismatches: + message = f"pane title restoration failed: {title_mismatches!r}" + raise RuntimeError(message) + + +def mutate_sync(context: RunContext, *, generation: int | str) -> MutationResult: + """Mutate one largest session through a batched plan, verify, and restore. + + >>> try: + ... mutate_sync(types.SimpleNamespace(mode=ExecutionMode.ASYNC), generation=1) + ... except ValueError as error: + ... print(error) + sync phase requires a sync run context + + Parameters + ---------- + context : RunContext + Verified active synchronous topology. + generation : int | str + Exact nonempty marker value for this mutation iteration. + + Returns + ------- + MutationResult + Timed plan duration plus verified restoration and activity evidence. + + Raises + ------ + ValueError + If context mode or generation is invalid. + RuntimeError + If mutation, restoration, or live activity verification fails. + """ + from libtmux.experimental.ops import BatchingPlanner, SessionId, ShowOptions, run + + _require_active_phase_context(context, ExecutionMode.SYNC) + generation_text = str(generation) + if not generation_text: + message = "mutation generation must be nonempty" + raise ValueError(message) + engine = t.cast("TmuxEngine", context.engine) + initial = snapshot_topology_sync(context) + session_id, window_ids, pane_ids = _mutation_targets(context, initial) + mutation, restoration, mutated_names, titles = _build_mutation_plans( + context, + generation=generation_text, + session_id=session_id, + window_ids=window_ids, + pane_ids=pane_ids, + ) + planner = BatchingPlanner() + baseline = _current_activity_epoch(context) + verified = False + restored = False + duration_ns = 0 + try: + started_ns = time.perf_counter_ns() + mutation_result = mutation.execute(engine, planner=planner) + duration_ns = time.perf_counter_ns() - started_ns + mutation_result.raise_for_status() + mutated_snapshot = snapshot_topology_sync(context) + option_result = run( + ShowOptions(target=SessionId(session_id)), engine + ).raise_for_status() + _verify_mutation_state( + mutated_snapshot, + generation=generation_text, + options=option_result.options, + mutated_names=mutated_names, + titles=titles, + ) + verified = True + finally: + restoration.execute(engine, planner=planner).raise_for_status() + restored_snapshot = snapshot_topology_sync(context) + restored_options = run( + ShowOptions(target=SessionId(session_id)), engine + ).raise_for_status() + _verify_restored_state( + context, + restored_snapshot, + options=restored_options.options, + window_ids=window_ids, + pane_ids=pane_ids, + ) + restored = True + activity_after = _wait_activity_advance_sync(context, baseline) + steps = len(mutation.explain(planner)) + return MutationResult( + duration_ns=duration_ns, + metrics=_phase_metrics(context, operations=len(mutation), planner_steps=steps), + session_id=session_id, + window_ids=window_ids, + pane_ids=pane_ids, + generation=generation_text, + verified=verified, + restored=restored, + activity_before_epoch=baseline, + activity_after_epoch=activity_after, + ) + + +async def mutate_async( + context: RunContext, + *, + generation: int | str, +) -> MutationResult: + """Async sibling of :func:`mutate_sync` over the identical operation graph. + + >>> async def invalid_mutation(): + ... try: + ... await mutate_async( + ... types.SimpleNamespace(mode=ExecutionMode.SYNC), generation=1 + ... ) + ... except ValueError as error: + ... return str(error) + >>> asyncio.run(invalid_mutation()) + 'async phase requires a async run context' + + Parameters + ---------- + context : RunContext + Verified active asynchronous topology. + generation : int | str + Exact nonempty marker value for this mutation iteration. + + Returns + ------- + MutationResult + Timed plan duration plus verified restoration and activity evidence. + + Raises + ------ + ValueError + If context mode or generation is invalid. + RuntimeError + If mutation, restoration, or live activity verification fails. + """ + from libtmux.experimental.ops import BatchingPlanner, SessionId, ShowOptions, arun + + _require_active_phase_context(context, ExecutionMode.ASYNC) + generation_text = str(generation) + if not generation_text: + message = "mutation generation must be nonempty" + raise ValueError(message) + engine = t.cast("AsyncTmuxEngine", context.engine) + initial = await snapshot_topology_async(context) + session_id, window_ids, pane_ids = _mutation_targets(context, initial) + mutation, restoration, mutated_names, titles = _build_mutation_plans( + context, + generation=generation_text, + session_id=session_id, + window_ids=window_ids, + pane_ids=pane_ids, + ) + planner = BatchingPlanner() + baseline = _current_activity_epoch(context) + verified = False + restored = False + duration_ns = 0 + try: + started_ns = time.perf_counter_ns() + mutation_result = await mutation.aexecute(engine, planner=planner) + duration_ns = time.perf_counter_ns() - started_ns + mutation_result.raise_for_status() + mutated_snapshot = await snapshot_topology_async(context) + option_result = ( + await arun(ShowOptions(target=SessionId(session_id)), engine) + ).raise_for_status() + _verify_mutation_state( + mutated_snapshot, + generation=generation_text, + options=option_result.options, + mutated_names=mutated_names, + titles=titles, + ) + verified = True + finally: + (await restoration.aexecute(engine, planner=planner)).raise_for_status() + restored_snapshot = await snapshot_topology_async(context) + restored_options = ( + await arun(ShowOptions(target=SessionId(session_id)), engine) + ).raise_for_status() + _verify_restored_state( + context, + restored_snapshot, + options=restored_options.options, + window_ids=window_ids, + pane_ids=pane_ids, + ) + restored = True + activity_after = await _wait_activity_advance_async(context, baseline) + steps = len(mutation.explain(planner)) + return MutationResult( + duration_ns=duration_ns, + metrics=_phase_metrics(context, operations=len(mutation), planner_steps=steps), + session_id=session_id, + window_ids=window_ids, + pane_ids=pane_ids, + generation=generation_text, + verified=verified, + restored=restored, + activity_before_epoch=baseline, + activity_after_epoch=activity_after, + ) + + +def _id_checksum(ids: tuple[str, ...]) -> str: + """Return SHA-256 over an unambiguous ordered concrete-ID encoding. + + >>> len(_id_checksum(("$0", "$1"))) + 64 + + Parameters + ---------- + ids : tuple[str, ...] + Stable concrete IDs in accepted hierarchy order. + + Returns + ------- + str + Lower-case hexadecimal SHA-256 digest over NUL-separated IDs. + """ + return hashlib.sha256("\0".join(ids).encode()).hexdigest() + + +def _enumeration_expected( + context: RunContext, + kind: EnumerationKind, +) -> tuple[str, ...]: + """Return the verified ID tuple for one hierarchy level. + + >>> context = types.SimpleNamespace( + ... session_ids=("$0",), window_ids=("@0",), pane_ids=("%0",) + ... ) + >>> _enumeration_expected(context, "windows") + ('@0',) + + Parameters + ---------- + context : RunContext + Verified topology context. + kind : {"sessions", "windows", "panes"} + Hierarchy level requested. + + Returns + ------- + tuple[str, ...] + Stable concrete IDs captured during topology verification. + + Raises + ------ + ValueError + If ``kind`` is outside the closed hierarchy vocabulary. + """ + if kind == "sessions": + return context.session_ids + if kind == "windows": + return context.window_ids + if kind == "panes": + return context.pane_ids + message = f"unknown enumeration kind: {kind}" + raise ValueError(message) + + +def _enumeration_operation(kind: EnumerationKind) -> object: + """Build the one typed list operation for a hierarchy level. + + >>> _enumeration_operation("panes").kind + 'list_panes' + + Parameters + ---------- + kind : {"sessions", "windows", "panes"} + Hierarchy level requested. + + Returns + ------- + object + ``ListSessions``, ``ListWindows``, or ``ListPanes`` operation. + """ + from libtmux.experimental.ops import ListPanes, ListSessions, ListWindows + + if kind == "sessions": + return ListSessions() + if kind == "windows": + return ListWindows(all_windows=True) + return ListPanes(all_panes=True) + + +def _enumeration_ids(result: t.Any, kind: EnumerationKind) -> tuple[str, ...]: + """Extract concrete IDs from a successful typed list result. + + >>> from libtmux.experimental.ops import ListSessions + >>> result = ListSessions().build_result(returncode=0, stdout=()) + >>> _enumeration_ids(result, "sessions") + () + + Parameters + ---------- + result : object + Successful specialized typed operation result. + kind : {"sessions", "windows", "panes"} + Hierarchy level represented by ``result``. + + Returns + ------- + tuple[str, ...] + Concrete IDs in the result's stable row order. + """ + if kind == "sessions": + return tuple(row.session_id for row in result.sessions) + if kind == "windows": + return tuple(row.window_id for row in result.windows) + return tuple(row.pane_id for row in result.panes) + + +def _accepted_enumeration( + context: RunContext, + kind: EnumerationKind, + ids: tuple[str, ...], + duration_ns: int, +) -> EnumerationResult: + """Validate exact row identity before constructing an enumeration sample. + + >>> context = types.SimpleNamespace( + ... lane=EngineLane.CONTROL, session_ids=("$0",), + ... window_ids=(), pane_ids=(), + ... ) + >>> _accepted_enumeration(context, "sessions", ("$0",), 4).verified + True + + Parameters + ---------- + context : RunContext + Verified concrete ID authority. + kind : {"sessions", "windows", "panes"} + Enumerated hierarchy level. + ids : tuple[str, ...] + IDs returned by the typed list operation. + duration_ns : int + Measured typed execution duration. + + Returns + ------- + EnumerationResult + Accepted exact rows and stable checksum. + + Raises + ------ + RuntimeError + If row count, stable order, or checksum differs from verified topology. + """ + expected = _enumeration_expected(context, kind) + observed_checksum = _id_checksum(ids) + expected_checksum = _id_checksum(expected) + if len(ids) != len(expected): + message = f"{kind} enumeration row count mismatch" + raise RuntimeError(message) + if ids != expected or observed_checksum != expected_checksum: + message = f"{kind} enumeration stable ID checksum mismatch" + raise RuntimeError(message) + return EnumerationResult( + duration_ns=duration_ns, + metrics=_phase_metrics(context, operations=1, planner_steps=1), + kind=kind, + row_count=len(ids), + ids=ids, + id_checksum=observed_checksum, + verified=True, + ) + + +def enumerate_sync( + context: RunContext, + *, + kind: EnumerationKind, +) -> EnumerationResult: + """Execute and validate one synchronous typed hierarchy list operation. + + >>> try: + ... enumerate_sync( + ... types.SimpleNamespace(mode=ExecutionMode.ASYNC), kind="panes" + ... ) + ... except ValueError as error: + ... print(error) + sync phase requires a sync run context + + Parameters + ---------- + context : RunContext + Verified active synchronous topology. + kind : {"sessions", "windows", "panes"} + One hierarchy cell to enumerate. + + Returns + ------- + EnumerationResult + Exact row count, concrete IDs, checksum, and timing. + """ + from libtmux.experimental.ops import run + + _require_active_phase_context(context, ExecutionMode.SYNC) + _enumeration_expected(context, kind) + engine = t.cast("TmuxEngine", context.engine) + operation = t.cast("Operation[t.Any]", _enumeration_operation(kind)) + started_ns = time.perf_counter_ns() + result = run(operation, engine) + duration_ns = time.perf_counter_ns() - started_ns + result.raise_for_status() + return _accepted_enumeration( + context, + kind, + _enumeration_ids(result, kind), + duration_ns, + ) + + +async def enumerate_async( + context: RunContext, + *, + kind: EnumerationKind, +) -> EnumerationResult: + """Execute and validate one asynchronous typed hierarchy list operation. + + >>> async def invalid_enumeration(): + ... try: + ... await enumerate_async( + ... types.SimpleNamespace(mode=ExecutionMode.SYNC), kind="panes" + ... ) + ... except ValueError as error: + ... return str(error) + >>> asyncio.run(invalid_enumeration()) + 'async phase requires a async run context' + + Parameters + ---------- + context : RunContext + Verified active asynchronous topology. + kind : {"sessions", "windows", "panes"} + One hierarchy cell to enumerate. + + Returns + ------- + EnumerationResult + Exact row count, concrete IDs, checksum, and timing. + """ + from libtmux.experimental.ops import arun + + _require_active_phase_context(context, ExecutionMode.ASYNC) + _enumeration_expected(context, kind) + engine = t.cast("AsyncTmuxEngine", context.engine) + operation = t.cast("Operation[t.Any]", _enumeration_operation(kind)) + started_ns = time.perf_counter_ns() + result = await arun(operation, engine) + duration_ns = time.perf_counter_ns() - started_ns + result.raise_for_status() + return _accepted_enumeration( + context, + kind, + _enumeration_ids(result, kind), + duration_ns, + ) + + +def _capture_plan(context: RunContext) -> LazyPlan: + """Build the transport-independent all-pane capture operation graph. + + >>> context = types.SimpleNamespace(pane_ids=("%1", "%2")) + >>> [operation.target.value for operation in _capture_plan(context).operations] + ['%1', '%2'] + + Parameters + ---------- + context : RunContext + Verified stable pane ID authority. + + Returns + ------- + LazyPlan + One bounded ``CapturePane`` operation per concrete pane. + """ + from libtmux.experimental.ops import CapturePane, LazyPlan, PaneId + + plan = LazyPlan() + for pane_id in context.pane_ids: + plan.add(CapturePane(target=PaneId(pane_id), start=-5000)) + return plan + + +def _capture_planner(strategy: CaptureStrategy) -> object: + """Return the planner policy named by one capture strategy. + + >>> type(_capture_planner("serial")).__name__ + 'SequentialPlanner' + >>> type(_capture_planner("batched")).__name__ + 'BatchingPlanner' + + Parameters + ---------- + strategy : {"serial", "batched"} + Planner policy name. + + Returns + ------- + object + ``SequentialPlanner`` or ``BatchingPlanner``. + + Raises + ------ + ValueError + If ``strategy`` is outside the closed vocabulary. + """ + from libtmux.experimental.ops import BatchingPlanner, SequentialPlanner + + if strategy == "serial": + return SequentialPlanner() + if strategy == "batched": + return BatchingPlanner() + message = f"unknown capture strategy: {strategy}" + raise ValueError(message) + + +def _accepted_capture( + context: RunContext, + plan: LazyPlan, + plan_result: t.Any, + strategy: CaptureStrategy, + duration_ns: int, +) -> CaptureResult: + """Validate typed current-epoch captures before returning measurement data. + + >>> from libtmux.experimental.ops import CapturePane, LazyPlan, PaneId + >>> plan = LazyPlan() + >>> _ = plan.add(CapturePane(target=PaneId("%1"))) + >>> raw = CapturePane(target=PaneId("%1")).build_result( + ... returncode=0, stdout=("epoch",) + ... ) + >>> context = types.SimpleNamespace( + ... pane_ids=("%1",), activity_marker="epoch", activity_epoch=2, + ... lane=EngineLane.CONTROL, + ... ) + >>> _accepted_capture( + ... context, plan, types.SimpleNamespace(results=(raw,)), "serial", 4 + ... ).line_count + 1 + + Parameters + ---------- + context : RunContext + Active run containing exact pane IDs and current activity marker. + plan : LazyPlan + Identical operation graph used for either planner. + plan_result : object + Successful ``PlanResult`` with specialized capture results. + strategy : {"serial", "batched"} + Planner strategy used for dispatch. + duration_ns : int + Measured graph execution duration. + + Returns + ------- + CaptureResult + Retained lines, totals, counts, and epoch verification. + + Raises + ------ + RuntimeError + If a result is not a typed capture, is empty, or lacks current epoch. + """ + from libtmux.experimental.ops import BatchingPlanner, CapturePaneResult + + captures: list[PaneCapture] = [] + results = tuple(plan_result.results) + if len(results) != len(context.pane_ids): + message = "capture result count did not match stable pane IDs" + raise RuntimeError(message) + for pane_id, result in zip(context.pane_ids, results, strict=True): + if not isinstance(result, CapturePaneResult): + message = f"capture for {pane_id} did not return typed lines" + raise TypeError(message) + result.raise_for_status() + if not result.lines: + message = f"capture for {pane_id} returned no lines" + raise RuntimeError(message) + marker = t.cast(str, context.activity_marker) + if marker not in "\n".join(result.lines): + message = f"capture for {pane_id} lacks current activity epoch" + raise RuntimeError(message) + captures.append(PaneCapture(pane_id, result.lines)) + line_count = sum(len(capture.lines) for capture in captures) + byte_count = sum( + len(line.encode("utf-8")) for capture in captures for line in capture.lines + ) + if line_count <= 0 or byte_count <= 0: + message = "all-pane capture produced no typed content" + raise RuntimeError(message) + planner_steps = ( + len(plan.explain(BatchingPlanner())) if strategy == "batched" else len(plan) + ) + return CaptureResult( + duration_ns=duration_ns, + metrics=_phase_metrics( + context, + operations=len(plan), + planner_steps=planner_steps, + ), + strategy=strategy, + operations=plan.operations, + captures=tuple(captures), + line_count=line_count, + byte_count=byte_count, + epoch=t.cast(int, context.activity_epoch), + verified=True, + ) + + +def capture_all_sync( + context: RunContext, + *, + strategy: CaptureStrategy, +) -> CaptureResult: + """Capture every pane synchronously with serial or batched planning. + + >>> try: + ... capture_all_sync( + ... types.SimpleNamespace(mode=ExecutionMode.ASYNC), strategy="serial" + ... ) + ... except ValueError as error: + ... print(error) + sync phase requires a sync run context + + Parameters + ---------- + context : RunContext + Verified active synchronous topology. + strategy : {"serial", "batched"} + Planner policy; operation graph remains identical. + + Returns + ------- + CaptureResult + Retained typed lines and exact work counts. + """ + from libtmux.experimental.ops import Planner + + _require_active_phase_context(context, ExecutionMode.SYNC) + plan = _capture_plan(context) + planner = t.cast("Planner", _capture_planner(strategy)) + engine = t.cast("TmuxEngine", context.engine) + started_ns = time.perf_counter_ns() + result = plan.execute(engine, planner=planner) + duration_ns = time.perf_counter_ns() - started_ns + result.raise_for_status() + return _accepted_capture(context, plan, result, strategy, duration_ns) + + +async def capture_all_async( + context: RunContext, + *, + strategy: CaptureStrategy, +) -> CaptureResult: + """Capture every pane asynchronously over the same operation graph. + + >>> async def invalid_capture(): + ... try: + ... await capture_all_async( + ... types.SimpleNamespace(mode=ExecutionMode.SYNC), strategy="serial" + ... ) + ... except ValueError as error: + ... return str(error) + >>> asyncio.run(invalid_capture()) + 'async phase requires a async run context' + + Parameters + ---------- + context : RunContext + Verified active asynchronous topology. + strategy : {"serial", "batched"} + Planner policy; operation graph remains identical. + + Returns + ------- + CaptureResult + Retained typed lines and exact work counts. + """ + from libtmux.experimental.ops import Planner + + _require_active_phase_context(context, ExecutionMode.ASYNC) + plan = _capture_plan(context) + planner = t.cast("Planner", _capture_planner(strategy)) + engine = t.cast("AsyncTmuxEngine", context.engine) + started_ns = time.perf_counter_ns() + result = await plan.aexecute(engine, planner=planner) + duration_ns = time.perf_counter_ns() - started_ns + result.raise_for_status() + return _accepted_capture(context, plan, result, strategy, duration_ns) + + +def _search_ids(context: RunContext, kind: EnumerationKind) -> tuple[str, ...]: + """Return stable concrete IDs for one metadata search kind. + + >>> context = types.SimpleNamespace( + ... session_ids=("$0",), window_ids=("@0",), pane_ids=("%0",) + ... ) + >>> _search_ids(context, "panes") + ('%0',) + + Parameters + ---------- + context : RunContext + Verified topology ID authority. + kind : {"sessions", "windows", "panes"} + Search object kind. + + Returns + ------- + tuple[str, ...] + Stable candidate IDs. + """ + return _enumeration_expected(context, kind) + + +def _row_id(row: object, kind: EnumerationKind) -> str: + """Read the concrete ID attribute shared by snapshot and classic rows. + + >>> _row_id(types.SimpleNamespace(window_id="@2"), "windows") + '@2' + + Parameters + ---------- + row : object + Typed snapshot or classic ORM object. + kind : {"sessions", "windows", "panes"} + Object kind controlling the concrete ID attribute. + + Returns + ------- + str + Concrete object ID. + """ + attribute = { + "sessions": "session_id", + "windows": "window_id", + "panes": "pane_id", + }[kind] + value = getattr(row, attribute) + if not isinstance(value, str): + message = f"{kind} row has no concrete ID" + raise TypeError(message) + return value + + +def _search_result( + *, + family: SearchFamily, + kind: EnumerationKind, + scanned_count: int, + target: str, + matches: cabc.Iterable[object], + duration_ns: int, + token: str | None = None, +) -> SearchResult: + """Accept only one exact metadata or content search match. + + >>> _search_result( + ... family="snapshot", kind="sessions", scanned_count=2, target="$1", + ... matches=(types.SimpleNamespace(session_id="$1"),), duration_ns=3, + ... ).matched_ids + ('$1',) + + Parameters + ---------- + family : {"server-side", "snapshot", "end-to-end", "contents"} + Explicit search timing boundary. + kind : {"sessions", "windows", "panes"} + Result object kind. + scanned_count : int + Candidate cardinality scanned. + target : str + Exact concrete ID required. + matches : collections.abc.Iterable[object] + Snapshot or classic rows returned by the search. + duration_ns : int + Search-only duration. + token : str | None + Exact content token for retained capture search. + + Returns + ------- + SearchResult + Verified exact match and scan cardinality. + + Raises + ------ + RuntimeError + If the search did not return exactly the requested ID. + """ + matched_ids = tuple(_row_id(row, kind) for row in matches) + if matched_ids != (target,): + message = f"{family} {kind} search expected {(target,)!r}, got {matched_ids!r}" + raise RuntimeError(message) + return SearchResult( + duration_ns=duration_ns, + family=family, + kind=kind, + scanned_count=scanned_count, + target=target, + matched_ids=matched_ids, + token=token, + verified=True, + ) + + +def search_server_side( + context: RunContext, + *, + kind: EnumerationKind, + target: str, +) -> SearchResult: + """Use classic tmux ``-f`` filtering for one exact concrete metadata ID. + + >>> try: + ... search_server_side( + ... types.SimpleNamespace(session_ids=("$0",), window_ids=(), pane_ids=()), + ... kind="sessions", target="$9", + ... ) + ... except ValueError as error: + ... print(error) + target '$9' is not a verified sessions ID + + Parameters + ---------- + context : RunContext + Verified isolated server and stable IDs. + kind : {"sessions", "windows", "panes"} + Classic server-side search level. + target : str + Exact concrete ID included in the tmux format filter. + + Returns + ------- + SearchResult + One exact match, full candidate cardinality, and search duration. + """ + candidates = _search_ids(context, kind) + if target not in candidates: + message = f"target {target!r} is not a verified {kind} ID" + raise ValueError(message) + format_name = { + "sessions": "session_id", + "windows": "window_id", + "panes": "pane_id", + }[kind] + filter_expression = f"#{{==:#{{{format_name}}},{target}}}" + search = { + "sessions": context.server.search_sessions, + "windows": context.server.search_windows, + "panes": context.server.search_panes, + }[kind] + started_ns = time.perf_counter_ns() + matches = t.cast(cabc.Iterable[object], search(filter=filter_expression)) + duration_ns = time.perf_counter_ns() - started_ns + return _search_result( + family="server-side", + kind=kind, + scanned_count=len(candidates), + target=target, + matches=matches, + duration_ns=duration_ns, + ) + + +def search_snapshot( + rows: object, + *, + kind: EnumerationKind, + target: str, +) -> SearchResult: + """Time only ``QueryList.filter`` over caller-prematerialized snapshot rows. + + >>> from libtmux._internal.query_list import QueryList + >>> from libtmux.experimental.models import SessionSnapshot + >>> rows = QueryList([SessionSnapshot(session_id="$0")]) + >>> search_snapshot(rows, kind="sessions", target="$0").scanned_count + 1 + + Parameters + ---------- + rows : object + Prematerialized :class:`~libtmux._internal.query_list.QueryList`. + kind : {"sessions", "windows", "panes"} + Snapshot object kind. + target : str + Exact concrete ID required. + + Returns + ------- + SearchResult + In-memory filter timing and exact scan cardinality. + + Raises + ------ + TypeError + If ``rows`` is not already a ``QueryList``. + """ + from libtmux._internal.query_list import QueryList + + if not isinstance(rows, QueryList): + message = "snapshot search requires a prematerialized QueryList" + raise TypeError(message) + field = { + "sessions": "session_id", + "windows": "window_id", + "panes": "pane_id", + }[kind] + started_ns = time.perf_counter_ns() + matches = rows.filter(**{field: target}) + duration_ns = time.perf_counter_ns() - started_ns + return _search_result( + family="snapshot", + kind=kind, + scanned_count=len(rows), + target=target, + matches=matches, + duration_ns=duration_ns, + ) + + +def search_end_to_end( + context: RunContext, + *, + kind: EnumerationKind, + target: str, +) -> SearchResult: + """Time classic list materialization plus Python ``QueryList`` filtering. + + >>> context = types.SimpleNamespace(session_ids=("$0",), window_ids=(), pane_ids=()) + >>> try: + ... search_end_to_end(context, kind="sessions", target="$9") + ... except ValueError as error: + ... print(error) + target '$9' is not a verified sessions ID + + Parameters + ---------- + context : RunContext + Verified isolated classic server and stable IDs. + kind : {"sessions", "windows", "panes"} + Object kind to list and filter. + target : str + Exact concrete ID required. + + Returns + ------- + SearchResult + End-to-end materialization and Python-filter timing. + """ + candidates = _search_ids(context, kind) + if target not in candidates: + message = f"target {target!r} is not a verified {kind} ID" + raise ValueError(message) + field = { + "sessions": "session_id", + "windows": "window_id", + "panes": "pane_id", + }[kind] + started_ns = time.perf_counter_ns() + rows = getattr(context.server, kind) + matches = rows.filter(**{field: target}) + duration_ns = time.perf_counter_ns() - started_ns + return _search_result( + family="end-to-end", + kind=kind, + scanned_count=len(rows), + target=target, + matches=matches, + duration_ns=duration_ns, + ) + + +def search_contents( + captures: CaptureResult, + *, + token: str, + expected_pane_id: str | None, +) -> SearchResult: + """Search retained typed capture lines for one exact sentinel token. + + >>> captures = CaptureResult( + ... 1, ExecutionMetrics(1, 1, 1, 1, 0), "serial", (), + ... (PaneCapture("%1", ("token",)), PaneCapture("%2", ("other",))), + ... 2, 10, 1, True, + ... ) + >>> search_contents( + ... captures, token="token", expected_pane_id="%1" + ... ).matched_ids + ('%1',) + + Parameters + ---------- + captures : CaptureResult + Prematerialized typed pane lines; no tmux I/O occurs here. + token : str + Exact sentinel line to match. + expected_pane_id : str | None + Concrete delayed-pane ID required as the sole match. + + Returns + ------- + SearchResult + Retained-content scan timing and exact token/ID evidence. + + Raises + ------ + ValueError + If token or expected pane ID is absent. + RuntimeError + If zero or multiple captures contain the exact sentinel line. + """ + if not token: + message = "content search token must be nonempty" + raise ValueError(message) + if expected_pane_id is None: + message = "content search requires a concrete expected pane ID" + raise ValueError(message) + started_ns = time.perf_counter_ns() + matches = tuple(capture for capture in captures.captures if token in capture.lines) + duration_ns = time.perf_counter_ns() - started_ns + return _search_result( + family="contents", + kind="panes", + scanned_count=len(captures.captures), + target=expected_pane_id, + matches=matches, + duration_ns=duration_ns, + token=token, + ) + + +PhaseMeasurement: t.TypeAlias = ( + MutationResult | EnumerationResult | CaptureResult | SearchResult +) + + +def _validated_phase_measurement(value: object) -> PhaseMeasurement: + """Require a typed successful measurement with positive integer timing. + + >>> result = SearchResult( + ... 3, "snapshot", "sessions", 1, "$0", ("$0",), verified=True + ... ) + >>> _validated_phase_measurement(result) is result + True + + Parameters + ---------- + value : object + Strategy callable result. + + Returns + ------- + MutationResult | EnumerationResult | CaptureResult | SearchResult + Valid typed measurement. + + Raises + ------ + TypeError + If the callable returned another type or non-integer duration. + RuntimeError + If typed verification was false or duration was nonpositive. + """ + accepted_types = ( + MutationResult, + EnumerationResult, + CaptureResult, + SearchResult, + ) + if not isinstance(value, accepted_types): + message = "phase callable did not return a typed measurement" + raise TypeError(message) + if type(value.duration_ns) is not int: + message = "phase duration must be integer nanoseconds" + raise TypeError(message) + if value.duration_ns <= 0: + message = "phase duration must be positive" + raise RuntimeError(message) + if not value.verified: + message = "phase typed result was not verified" + raise RuntimeError(message) + return value + + +async def _await_if_needed(value: object) -> object: + """Await an awaitable strategy value and pass synchronous values through. + + >>> asyncio.run(_await_if_needed(4)) + 4 + >>> asyncio.run(_await_if_needed(asyncio.sleep(0, result=5))) + 5 + + Parameters + ---------- + value : object + Immediate value or awaitable. + + Returns + ------- + object + Resolved value. + """ + if inspect.isawaitable(value): + return await value + return value + + +def _require_live_postcondition(value: object) -> None: + """Require an exact true live postcondition result. + + >>> _require_live_postcondition(True) + >>> try: + ... _require_live_postcondition(False) + ... except RuntimeError as error: + ... print(error) + live postcondition rejected the phase result + + Parameters + ---------- + value : object + Resolved live-postcondition return value. + + Raises + ------ + RuntimeError + If the postcondition did not return exactly ``True``. + """ + if value is not True: + message = "live postcondition rejected the phase result" + raise RuntimeError(message) + + +async def run_repeatable_phase( + strategies: cabc.Mapping[str, cabc.Callable[[], object]], + *, + warmup: int, + runs: int, + seed: int, + snapshot_resources: cabc.Callable[[], HostSnapshot] | None = None, + live_postcondition: cabc.Callable[[PhaseMeasurement], object] | None = None, +) -> RepeatablePhaseResult: + """Deterministically interleave strategies and retain accepted timed rows. + + The seed shuffles one base strategy order. Each subsequent warmup or timed + ordinal rotates that base order by one, giving a deterministic round robin. + Resource observations bracket every invocation. A typed result and its live + postcondition must pass before a timed :class:`RawSample` is appended. + + >>> async def example(): + ... def measured(): + ... return SearchResult( + ... 3, "snapshot", "sessions", 1, "$0", ("$0",), verified=True + ... ) + ... return await run_repeatable_phase( + ... {"one": measured}, warmup=0, runs=1, seed=2, + ... snapshot_resources=lambda: HostSnapshot(), + ... ) + >>> len(asyncio.run(example()).samples) + 1 + + Parameters + ---------- + strategies : collections.abc.Mapping[str, collections.abc.Callable] + Nonempty named sync or async phase callables. + warmup : int + Untimed invocation count per strategy. + runs : int + Timed accepted invocation count requested per strategy. + seed : int + Deterministic base-order shuffle seed. + snapshot_resources : collections.abc.Callable[[], HostSnapshot] | None + Injectable resource sampler; defaults to the live process/cgroup probe. + live_postcondition : collections.abc.Callable | None + Optional sync or async check run after typed validation; defaults true. + + Returns + ------- + RepeatablePhaseResult + Accepted rows, invocation order, and first failure metadata if any. + + Raises + ------ + ValueError + If strategies are empty, names are invalid, or counts are invalid. + """ + if not strategies or any(not name for name in strategies): + message = "repeatable phase requires nonempty named strategies" + raise ValueError(message) + if warmup < 0 or runs <= 0: + message = "warmup must be nonnegative and runs must be positive" + raise ValueError(message) + sampler = snapshot_resources or (lambda: probe_host(ProcessReader())) + base_order = list(strategies) + random.Random(seed).shuffle(base_order) + samples: list[RawSample] = [] + order: list[str] = [] + total_cycles = warmup + runs + for cycle in range(total_cycles): + stage: t.Literal["warmup", "timed"] = "warmup" if cycle < warmup else "timed" + ordinal = cycle if stage == "warmup" else cycle - warmup + rotation = cycle % len(base_order) + cycle_order = (*base_order[rotation:], *base_order[:rotation]) + for strategy in cycle_order: + order.append(strategy) + try: + resources_before = sampler() + produced = strategies[strategy]() + measurement = _validated_phase_measurement( + await _await_if_needed(produced) + ) + resources_after = sampler() + if live_postcondition is not None: + postcondition = await _await_if_needed( + live_postcondition(measurement) + ) + _require_live_postcondition(postcondition) + if stage == "timed": + samples.append( + RawSample( + duration_ns=measurement.duration_ns, + accepted=True, + verified=True, + strategy=strategy, + ordinal=ordinal, + resources_before=resources_before, + resources_after=resources_after, + ) + ) + except Exception as error: # noqa: BLE001 + return RepeatablePhaseResult( + samples=tuple(samples), + order=tuple(order), + failure=RepeatablePhaseFailure( + stage=stage, + strategy=strategy, + ordinal=ordinal, + error=f"{type(error).__name__}: {error}", + ), + ) + return RepeatablePhaseResult(tuple(samples), tuple(order)) + + async def _wait_for_process_absence( identities: tuple[ProcessIdentity, ...], *, diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index 3efb4a8332..c04038fbc3 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -4,6 +4,7 @@ import asyncio import dataclasses +import hashlib import importlib.util import json import os @@ -12,13 +13,22 @@ import stat import subprocess import sys +import time import types import typing as t import pytest +from libtmux._internal.query_list import QueryList from libtmux.experimental.models import PaneSnapshot, SessionSnapshot, WindowSnapshot +_PHASE_LANES = ( + ("subprocess", "sync"), + ("subprocess", "async"), + ("control", "sync"), + ("control", "async"), +) + @pytest.fixture() def benchmark_module() -> types.ModuleType: @@ -1466,3 +1476,438 @@ async def exercise_async() -> None: assert not scratch.exists() assert os.environ["TMUX"] == "ambient-server" assert os.environ["TMUX_PANE"] == "%ambient" + + +def _checksum_ids(ids: tuple[str, ...]) -> str: + """Derive the test oracle without calling the benchmark helper.""" + return hashlib.sha256("\0".join(ids).encode()).hexdigest() + + +@pytest.mark.parametrize(("lane_name", "mode_name"), _PHASE_LANES) +def test_live_mutation_phase_restores_stable_targets_and_keeps_activity_advancing( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + lane_name: str, + mode_name: str, +) -> None: + """A mutation sample must verify live state and restore it outside timing.""" + topology = benchmark_module.Topology(2, 3, 2) + lane = benchmark_module.EngineLane(lane_name) + scratch = tmp_path / f"mutation-{lane_name}-{mode_name}" + context = result = restored = cleanup = None + + async def exercise_async() -> None: + """Keep an async engine on one loop through mutation and cleanup.""" + nonlocal context, result, restored, cleanup + context = await benchmark_module.setup_async( + topology, + lane, + scratch, + run_id=f"mutation-{lane_name}-{mode_name}", + delayed_ordinal=7, + ) + try: + benchmark_module.release_activity_gate(context) + await benchmark_module.verify_activity_async(context) + result = await benchmark_module.mutate_async(context, generation=7) + restored = await benchmark_module.snapshot_topology_async(context) + finally: + cleanup = await benchmark_module.cleanup_run(context) + + if mode_name == "async": + asyncio.run(exercise_async()) + else: + context = benchmark_module.setup_sync( + topology, + lane, + scratch, + run_id=f"mutation-{lane_name}-{mode_name}", + delayed_ordinal=7, + ) + try: + benchmark_module.release_activity_gate(context) + benchmark_module.verify_activity_sync(context) + result = benchmark_module.mutate_sync(context, generation=7) + restored = benchmark_module.snapshot_topology_sync(context) + finally: + cleanup = asyncio.run(benchmark_module.cleanup_run(context)) + + assert context is not None + assert result is not None + assert restored is not None + assert cleanup is not None + assert result.duration_ns > 0 + assert result.generation == "7" + assert result.session_id in context.session_ids + assert len(result.window_ids) == 3 + assert len(result.pane_ids) == 6 + assert result.metrics.operations == 10 + assert result.metrics.planner_steps == 1 + assert result.metrics.engine_batches == 1 + assert result.metrics.tmux_requests == 10 + assert result.metrics.process_starts == ( + 10 if lane is benchmark_module.EngineLane.SUBPROCESS else 0 + ) + assert result.verified + assert result.restored + assert result.activity_after_epoch > result.activity_before_epoch + assert tuple(session.name for session in restored.sessions) == ( + context.expected_session_names + ) + assert {window.name for window in restored.windows} == set( + context.expected_window_names + ) + restored_panes = { + pane.pane_id: pane.fields.get("pane_title") for pane in restored.panes + } + assert all( + not (restored_panes[pane_id] or "").startswith(f"bench-{context.run_id}-g7-") + for pane_id in result.pane_ids + ) + assert cleanup.complete, cleanup.errors + + +@pytest.mark.parametrize(("lane_name", "mode_name"), _PHASE_LANES) +def test_live_enumeration_phase_accepts_exact_rows_and_stable_id_checksums( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + lane_name: str, + mode_name: str, +) -> None: + """An accepted enumeration must match the verified concrete ID tuples.""" + topology = benchmark_module.Topology(2, 3, 2) + lane = benchmark_module.EngineLane(lane_name) + scratch = tmp_path / f"enumeration-{lane_name}-{mode_name}" + context = cleanup = None + observed: dict[str, t.Any] = {} + + async def exercise_async() -> None: + """Keep async enumeration and cleanup on one event loop.""" + nonlocal context, cleanup + context = await benchmark_module.setup_async( + topology, + lane, + scratch, + run_id=f"enumeration-{lane_name}-{mode_name}", + delayed_ordinal=7, + ) + try: + benchmark_module.release_activity_gate(context) + await benchmark_module.verify_activity_async(context) + for kind in ("sessions", "windows", "panes"): + observed[kind] = await benchmark_module.enumerate_async( + context, kind=kind + ) + finally: + cleanup = await benchmark_module.cleanup_run(context) + + if mode_name == "async": + asyncio.run(exercise_async()) + else: + context = benchmark_module.setup_sync( + topology, + lane, + scratch, + run_id=f"enumeration-{lane_name}-{mode_name}", + delayed_ordinal=7, + ) + try: + benchmark_module.release_activity_gate(context) + benchmark_module.verify_activity_sync(context) + for kind in ("sessions", "windows", "panes"): + observed[kind] = benchmark_module.enumerate_sync(context, kind=kind) + finally: + cleanup = asyncio.run(benchmark_module.cleanup_run(context)) + + assert context is not None + expected = { + "sessions": context.session_ids, + "windows": context.window_ids, + "panes": context.pane_ids, + } + assert {kind: result.row_count for kind, result in observed.items()} == { + "sessions": 2, + "windows": 6, + "panes": 12, + } + for kind, ids in expected.items(): + result = observed[kind] + assert result.ids == ids + assert result.id_checksum == _checksum_ids(ids) + assert result.duration_ns > 0 + assert result.metrics.operations == 1 + assert result.metrics.tmux_requests == 1 + assert result.verified + assert cleanup is not None + assert cleanup.complete, cleanup.errors + + +@pytest.mark.parametrize(("lane_name", "mode_name"), _PHASE_LANES) +def test_live_capture_phase_uses_identical_graphs_with_distinct_planners( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + lane_name: str, + mode_name: str, +) -> None: + """Serial and batched capture must return every current active pane.""" + topology = benchmark_module.Topology(2, 3, 2) + lane = benchmark_module.EngineLane(lane_name) + scratch = tmp_path / f"capture-{lane_name}-{mode_name}" + context = serial = batched = cleanup = None + + async def exercise_async() -> None: + """Keep both async planner executions on the engine's owning loop.""" + nonlocal context, serial, batched, cleanup + context = await benchmark_module.setup_async( + topology, + lane, + scratch, + run_id=f"capture-{lane_name}-{mode_name}", + delayed_ordinal=7, + ) + try: + benchmark_module.release_activity_gate(context) + await benchmark_module.verify_activity_async(context) + serial = await benchmark_module.capture_all_async( + context, strategy="serial" + ) + batched = await benchmark_module.capture_all_async( + context, strategy="batched" + ) + finally: + cleanup = await benchmark_module.cleanup_run(context) + + if mode_name == "async": + asyncio.run(exercise_async()) + else: + context = benchmark_module.setup_sync( + topology, + lane, + scratch, + run_id=f"capture-{lane_name}-{mode_name}", + delayed_ordinal=7, + ) + try: + benchmark_module.release_activity_gate(context) + benchmark_module.verify_activity_sync(context) + serial = benchmark_module.capture_all_sync(context, strategy="serial") + batched = benchmark_module.capture_all_sync(context, strategy="batched") + finally: + cleanup = asyncio.run(benchmark_module.cleanup_run(context)) + + assert context is not None + assert serial is not None + assert batched is not None + for result in (serial, batched): + assert tuple(capture.pane_id for capture in result.captures) == ( + context.pane_ids + ) + assert result.duration_ns > 0 + assert result.line_count >= 12 + assert result.byte_count > 0 + assert result.epoch == context.activity_epoch + assert result.metrics.operations == 12 + assert result.metrics.tmux_requests == 12 + assert result.metrics.process_starts == ( + 12 if lane is benchmark_module.EngineLane.SUBPROCESS else 0 + ) + assert result.verified + assert serial.metrics.planner_steps == 12 + assert serial.metrics.engine_batches == 12 + assert batched.metrics.planner_steps == 1 + assert batched.metrics.engine_batches == 1 + assert serial.operations == batched.operations + assert cleanup is not None + assert cleanup.complete, cleanup.errors + + +def test_live_search_phase_keeps_server_snapshot_end_to_end_and_content_distinct( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """Each search family must scan its declared source and return one target.""" + topology = benchmark_module.Topology(2, 3, 2) + scratch = tmp_path / "search" + context = benchmark_module.setup_sync( + topology, + benchmark_module.EngineLane.SUBPROCESS, + scratch, + run_id="search-phase", + delayed_ordinal=7, + ) + cleanup = None + try: + benchmark_module.release_activity_gate(context) + benchmark_module.verify_activity_sync(context) + snapshot = benchmark_module.snapshot_topology_sync(context) + ids_by_kind = { + "sessions": context.session_ids, + "windows": context.window_ids, + "panes": context.pane_ids, + } + for kind, ids in ids_by_kind.items(): + for target in (ids[0], ids[len(ids) // 2], ids[-1]): + result = benchmark_module.search_server_side( + context, kind=kind, target=target + ) + assert result.family == "server-side" + assert result.matched_ids == (target,) + assert result.scanned_count == len(ids) + assert result.verified + + snapshot_rows = QueryList(snapshot.sessions) + snapshot_result = benchmark_module.search_snapshot( + snapshot_rows, + kind="sessions", + target=context.session_ids[1], + ) + end_to_end_result = benchmark_module.search_end_to_end( + context, + kind="windows", + target=context.window_ids[3], + ) + + request_id = "content-search" + requested_ns = time.monotonic_ns() + benchmark_module.write_json_atomic( + scratch / "fuzzer" / "requests" / f"{request_id}.json", + { + "schema_version": 1, + "run_id": context.run_id, + "request_id": request_id, + "requested_monotonic_ns": requested_ns, + "value": "CONTENT-ONLY", + }, + ) + evidence_path = scratch / "fuzzer" / "sentinels" / f"{request_id}.json" + deadline = time.monotonic() + 2.0 + evidence = None + while time.monotonic() < deadline: + try: + evidence = json.loads(evidence_path.read_text(encoding="utf-8")) + except (FileNotFoundError, json.JSONDecodeError): + time.sleep(0.01) + continue + break + assert evidence is not None + sentinel = evidence["sentinel"] + + captures = None + while time.monotonic() < deadline: + candidate = benchmark_module.capture_all_sync(context, strategy="batched") + if any(sentinel in capture.lines for capture in candidate.captures): + captures = candidate + break + time.sleep(0.01) + assert captures is not None + content_result = benchmark_module.search_contents( + captures, + token=sentinel, + expected_pane_id=context.delayed_pane_id, + ) + + assert snapshot_result.family == "snapshot" + assert snapshot_result.matched_ids == (context.session_ids[1],) + assert end_to_end_result.family == "end-to-end" + assert end_to_end_result.matched_ids == (context.window_ids[3],) + assert content_result.family == "contents" + assert content_result.matched_ids == (context.delayed_pane_id,) + assert content_result.token == sentinel + assert { + snapshot_result.scanned_count, + end_to_end_result.scanned_count, + content_result.scanned_count, + } == {2, 6, 12} + finally: + cleanup = asyncio.run(benchmark_module.cleanup_run(context)) + assert cleanup.complete, cleanup.errors + + +def test_run_repeatable_phase_interleaves_and_accepts_only_verified_samples( + benchmark_module: types.ModuleType, +) -> None: + """Only typed results with a true live postcondition become raw samples.""" + calls: list[str] = [] + resource_ordinal = 0 + + def snapshot_resources() -> t.Any: + nonlocal resource_ordinal + resource_ordinal += 1 + return benchmark_module.HostSnapshot(pids_current=resource_ordinal) + + def measured(name: str) -> t.Callable[[], t.Any]: + def run() -> t.Any: + calls.append(name) + return benchmark_module.SearchResult( + duration_ns=17, + family="snapshot", + kind="sessions", + scanned_count=2, + target="$1", + matched_ids=("$1",), + verified=True, + ) + + return run + + result = asyncio.run( + benchmark_module.run_repeatable_phase( + {"alpha": measured("alpha"), "beta": measured("beta")}, + warmup=1, + runs=2, + seed=11, + snapshot_resources=snapshot_resources, + live_postcondition=lambda measurement: measurement.matched_ids == ("$1",), + ) + ) + + assert result.failure is None + assert len(result.samples) == 4 + assert len(result.order) == 6 + assert calls[:2] in (["alpha", "beta"], ["beta", "alpha"]) + assert calls[2:4] == list(reversed(calls[:2])) + assert calls[4:6] == calls[:2] + assert all(sample.duration_ns == 17 for sample in result.samples) + assert all(sample.accepted and sample.verified for sample in result.samples) + assert all(sample.resources_before is not None for sample in result.samples) + assert all(sample.resources_after is not None for sample in result.samples) + + +def test_run_repeatable_phase_stops_without_appending_failed_duration( + benchmark_module: types.ModuleType, +) -> None: + """A phase exception must retain failure metadata but no duration row.""" + attempts = 0 + + def fail_second() -> t.Any: + nonlocal attempts + attempts += 1 + if attempts == 2: + message = "live postcondition lost" + raise RuntimeError(message) + return benchmark_module.SearchResult( + duration_ns=23, + family="snapshot", + kind="sessions", + scanned_count=2, + target="$1", + matched_ids=("$1",), + verified=True, + ) + + result = asyncio.run( + benchmark_module.run_repeatable_phase( + {"only": fail_second}, + warmup=0, + runs=3, + seed=3, + snapshot_resources=lambda: benchmark_module.HostSnapshot(), + ) + ) + + assert len(result.samples) == 1 + assert result.samples[0].duration_ns == 23 + assert result.failure is not None + assert result.failure.strategy == "only" + assert result.failure.ordinal == 1 + assert result.failure.error == "RuntimeError: live postcondition lost" From 5b7c4648eeb97019f067c0d6f91c3db62c5e5573 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 09:23:58 -0500 Subject: [PATCH 16/67] Bench(fix[phases]): Harden live acceptance why: Prevent stale heartbeat, misattributed capture, or cancellation from yielding accepted phase evidence. what: - Require fresh post-restoration activity and mandatory postconditions - Shield async restoration and bind captures to exact pane targets - Exercise explicit search families across all live lanes --- scripts/bench_orchestration.py | 546 +++++++++++++++---- tests/test_bench_orchestration_script.py | 636 ++++++++++++++++++++--- 2 files changed, 1011 insertions(+), 171 deletions(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index 809ccce969..9b28205ccb 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -48,6 +48,7 @@ ) from libtmux.experimental.ops.operation import Operation from libtmux.experimental.ops.plan import LazyPlan + from libtmux.experimental.ops.planner import Planner from libtmux.experimental.workspace import WorkspaceSet from libtmux.server import Server @@ -922,6 +923,30 @@ class ExecutionMetrics: process_starts: int +@dataclasses.dataclass(frozen=True) +class HeartbeatObservation: + """One run-scoped fuzzer heartbeat observed by the benchmark. + + Attributes + ---------- + epoch : int + Active fuzzer epoch published in the atomic heartbeat marker. + published_monotonic_ns : int + Fuzzer monotonic timestamp stored with that epoch. + observed_monotonic_ns : int + Benchmark monotonic timestamp immediately after reading the marker. + + Examples + -------- + >>> HeartbeatObservation(4, 10, 12).epoch + 4 + """ + + epoch: int + published_monotonic_ns: int + observed_monotonic_ns: int + + @dataclasses.dataclass(frozen=True) class MutationResult: """Verified and restored bulk mutation measurement. @@ -944,18 +969,21 @@ class MutationResult: Whether all mutated live values matched the plan. restored : bool Whether canonical names, titles, and option state were restored. - activity_before_epoch : int - Fuzzer heartbeat epoch before the mutation. - activity_after_epoch : int - Later heartbeat epoch observed after restoration. + restoration_verified_monotonic_ns : int + Local timestamp taken after restored-state verification. + activity_baseline : HeartbeatObservation + Fresh heartbeat read only after restoration verification. + activity_after : HeartbeatObservation + Strictly later heartbeat proving continued activity. Examples -------- >>> result = MutationResult( ... 5, ExecutionMetrics(3, 1, 1, 3, 0), "$1", ("@2",), ("%3",), - ... "7", True, True, 4, 5, + ... "7", True, True, 9, HeartbeatObservation(4, 8, 10), + ... HeartbeatObservation(5, 11, 12), ... ) - >>> result.restored and result.activity_after_epoch > result.activity_before_epoch + >>> result.restored and result.activity_after.epoch > result.activity_baseline.epoch True """ @@ -967,8 +995,9 @@ class MutationResult: generation: str verified: bool restored: bool - activity_before_epoch: int - activity_after_epoch: int + restoration_verified_monotonic_ns: int + activity_baseline: HeartbeatObservation + activity_after: HeartbeatObservation EnumerationKind: t.TypeAlias = t.Literal["sessions", "windows", "panes"] @@ -1082,9 +1111,7 @@ class CaptureResult: verified: bool -SearchFamily: t.TypeAlias = t.Literal[ - "server-side", "snapshot", "end-to-end", "contents" -] +SearchFamily: t.TypeAlias = t.Literal["classic", "snapshot", "end-to-end", "contents"] @dataclasses.dataclass(frozen=True) @@ -1095,7 +1122,7 @@ class SearchResult: ---------- duration_ns : int Time spent in the named search family only. - family : {"server-side", "snapshot", "end-to-end", "contents"} + family : {"classic", "snapshot", "end-to-end", "contents"} Search source and timing boundary. kind : {"sessions", "windows", "panes"} Object kind scanned or returned. @@ -1556,6 +1583,8 @@ class RunContext: Panes that captured the released marker. heartbeat_epoch : int Last monotonic fuzzer heartbeat epoch observed. + heartbeat_monotonic_ns : int + Last fuzzer monotonic publication timestamp observed. ambient_tmux_environment : tuple[str | None, str | None] Original ``TMUX`` and ``TMUX_PANE`` values restored after cleanup. @@ -1594,6 +1623,7 @@ class RunContext: activity_marker: str | None = None activity_pane_ids: tuple[str, ...] = () heartbeat_epoch: int = -1 + heartbeat_monotonic_ns: int = -1 ambient_tmux_environment: tuple[str | None, str | None] = (None, None) @@ -3840,12 +3870,16 @@ def _require_active_phase_context( raise RuntimeError(message) -def _current_activity_epoch(context: RunContext) -> int: - """Return and retain an active, non-regressing fuzzer heartbeat epoch. +def _current_activity_heartbeat( + context: RunContext, + *, + max_age_s: float, +) -> HeartbeatObservation: + """Read one fresh active heartbeat with epoch and monotonic timestamps. >>> context = types.SimpleNamespace(fuzzer=types.SimpleNamespace(poll=lambda: 1)) >>> try: - ... _current_activity_epoch(context) + ... _current_activity_heartbeat(context, max_age_s=1) ... except RuntimeError as error: ... print(error) fuzzer exited during measured phase @@ -3854,37 +3888,73 @@ def _current_activity_epoch(context: RunContext) -> int: ---------- context : RunContext Active run whose heartbeat is authoritative. + max_age_s : float + Maximum permitted age of the fuzzer publication timestamp. Returns ------- - int - Current active heartbeat epoch. + HeartbeatObservation + Current active epoch with publication and local observation times. Raises ------ RuntimeError - If the fuzzer exited, heartbeat is inactive, or its epoch regressed. + If the fuzzer exited, heartbeat is inactive, stale, or regressed. + ValueError + If ``max_age_s`` is not positive. """ + if max_age_s <= 0: + message = "heartbeat maximum age must be positive" + raise ValueError(message) if context.fuzzer.poll() is not None: message = "fuzzer exited during measured phase" raise RuntimeError(message) - state, epoch = _heartbeat_epoch(context) - if state != "active" or epoch is None: + marker = _read_run_marker( + context.scratch / "fuzzer" / "heartbeat.json", context.run_id + ) + if marker is None or marker.get("state") != "active": message = "fuzzer heartbeat is not active during measured phase" raise RuntimeError(message) + epoch = marker.get("epoch") + published_monotonic_ns = marker.get("monotonic_ns") + if ( + type(epoch) is not int + or epoch < 0 + or type(published_monotonic_ns) is not int + or published_monotonic_ns < 0 + ): + message = "fuzzer heartbeat has invalid activity evidence" + raise RuntimeError(message) + observed_monotonic_ns = time.monotonic_ns() + age_ns = observed_monotonic_ns - published_monotonic_ns + if age_ns < 0: + message = "fuzzer heartbeat publication is in the future" + raise RuntimeError(message) + if age_ns > int(max_age_s * 1_000_000_000): + message = "fuzzer heartbeat is stale during measured phase" + raise RuntimeError(message) if epoch < context.heartbeat_epoch: message = "fuzzer heartbeat epoch moved backwards" raise RuntimeError(message) + previous_monotonic_ns = getattr(context, "heartbeat_monotonic_ns", -1) + if published_monotonic_ns < previous_monotonic_ns: + message = "fuzzer heartbeat timestamp moved backwards" + raise RuntimeError(message) context.heartbeat_epoch = epoch - return epoch + context.heartbeat_monotonic_ns = published_monotonic_ns + return HeartbeatObservation( + epoch=epoch, + published_monotonic_ns=published_monotonic_ns, + observed_monotonic_ns=observed_monotonic_ns, + ) def _wait_activity_advance_sync( context: RunContext, - baseline: int, + baseline: HeartbeatObservation, *, timeout_s: float = 2.0, -) -> int: +) -> HeartbeatObservation: """Wait until the live fuzzer heartbeat advances past ``baseline``. >>> try: @@ -3897,15 +3967,15 @@ def _wait_activity_advance_sync( ---------- context : RunContext Active synchronous run. - baseline : int - Epoch that the fuzzer must surpass. + baseline : HeartbeatObservation + Post-restoration heartbeat that the fuzzer must surpass. timeout_s : float Maximum monotonic wait. Returns ------- - int - First observed later epoch. + HeartbeatObservation + First observation with a later epoch and publication timestamp. Raises ------ @@ -3917,22 +3987,32 @@ def _wait_activity_advance_sync( if timeout_s <= 0: message = "activity advance timeout must be positive" raise ValueError(message) - deadline = time.monotonic() + timeout_s - while time.monotonic() < deadline: - epoch = _current_activity_epoch(context) - if epoch > baseline: - return epoch + deadline_ns = time.monotonic_ns() + int(timeout_s * 1_000_000_000) + maximum_age_s = ( + timeout_s + + (baseline.observed_monotonic_ns - baseline.published_monotonic_ns) + / 1_000_000_000 + ) + while time.monotonic_ns() < deadline_ns: + heartbeat = _current_activity_heartbeat(context, max_age_s=maximum_age_s) + if ( + heartbeat.epoch > baseline.epoch + and heartbeat.published_monotonic_ns > baseline.published_monotonic_ns + and heartbeat.observed_monotonic_ns > baseline.observed_monotonic_ns + and heartbeat.observed_monotonic_ns <= deadline_ns + ): + return heartbeat time.sleep(0.005) - message = f"activity did not advance past epoch {baseline}" + message = f"activity did not advance past epoch {baseline.epoch}" raise TimeoutError(message) async def _wait_activity_advance_async( context: RunContext, - baseline: int, + baseline: HeartbeatObservation, *, timeout_s: float = 2.0, -) -> int: +) -> HeartbeatObservation: """Asynchronously wait for a later active heartbeat epoch. >>> async def invalid_wait(): @@ -3949,15 +4029,15 @@ async def _wait_activity_advance_async( ---------- context : RunContext Active asynchronous run. - baseline : int - Epoch that the fuzzer must surpass. + baseline : HeartbeatObservation + Post-restoration heartbeat that the fuzzer must surpass. timeout_s : float Maximum event-loop wait. Returns ------- - int - First observed later epoch. + HeartbeatObservation + First observation with a later epoch and publication timestamp. Raises ------ @@ -3969,14 +4049,23 @@ async def _wait_activity_advance_async( if timeout_s <= 0: message = "activity advance timeout must be positive" raise ValueError(message) - loop = asyncio.get_running_loop() - deadline = loop.time() + timeout_s - while loop.time() < deadline: - epoch = _current_activity_epoch(context) - if epoch > baseline: - return epoch + deadline_ns = time.monotonic_ns() + int(timeout_s * 1_000_000_000) + maximum_age_s = ( + timeout_s + + (baseline.observed_monotonic_ns - baseline.published_monotonic_ns) + / 1_000_000_000 + ) + while time.monotonic_ns() < deadline_ns: + heartbeat = _current_activity_heartbeat(context, max_age_s=maximum_age_s) + if ( + heartbeat.epoch > baseline.epoch + and heartbeat.published_monotonic_ns > baseline.published_monotonic_ns + and heartbeat.observed_monotonic_ns > baseline.observed_monotonic_ns + and heartbeat.observed_monotonic_ns <= deadline_ns + ): + return heartbeat await asyncio.sleep(0.005) - message = f"activity did not advance past epoch {baseline}" + message = f"activity did not advance past epoch {baseline.epoch}" raise TimeoutError(message) @@ -4294,9 +4383,9 @@ def mutate_sync(context: RunContext, *, generation: int | str) -> MutationResult pane_ids=pane_ids, ) planner = BatchingPlanner() - baseline = _current_activity_epoch(context) verified = False restored = False + restoration_verified_monotonic_ns = 0 duration_ns = 0 try: started_ns = time.perf_counter_ns() @@ -4329,7 +4418,9 @@ def mutate_sync(context: RunContext, *, generation: int | str) -> MutationResult pane_ids=pane_ids, ) restored = True - activity_after = _wait_activity_advance_sync(context, baseline) + restoration_verified_monotonic_ns = time.monotonic_ns() + activity_baseline = _current_activity_heartbeat(context, max_age_s=2.0) + activity_after = _wait_activity_advance_sync(context, activity_baseline) steps = len(mutation.explain(planner)) return MutationResult( duration_ns=duration_ns, @@ -4340,9 +4431,81 @@ def mutate_sync(context: RunContext, *, generation: int | str) -> MutationResult generation=generation_text, verified=verified, restored=restored, - activity_before_epoch=baseline, - activity_after_epoch=activity_after, + restoration_verified_monotonic_ns=restoration_verified_monotonic_ns, + activity_baseline=activity_baseline, + activity_after=activity_after, + ) + + +async def _restore_mutation_async( + context: RunContext, + restoration: LazyPlan, + planner: Planner, + *, + session_id: str, + window_ids: tuple[str, ...], + pane_ids: tuple[str, ...], +) -> int: + """Execute and verify async restoration within one owned task. + + >>> async def invalid_restoration(): + ... try: + ... await _restore_mutation_async( + ... types.SimpleNamespace(mode=ExecutionMode.SYNC), + ... t.cast("LazyPlan", None), t.cast("Planner", None), + ... session_id="$0", window_ids=(), pane_ids=(), + ... ) + ... except ValueError as error: + ... return str(error) + >>> asyncio.run(invalid_restoration()) + 'async restoration requires an async run context' + + Parameters + ---------- + context : RunContext + Async live run containing canonical topology bindings. + restoration : LazyPlan + Fully constructed canonical-name, option, and title restoration graph. + planner : Planner + Planner used by the corresponding mutation graph. + session_id : str + Concrete session whose generation option must be absent afterward. + window_ids : tuple[str, ...] + Mutated windows that must regain canonical names. + pane_ids : tuple[str, ...] + Mutated panes whose explicit titles must be cleared. + + Returns + ------- + int + Local monotonic timestamp taken after restored-state verification. + + Raises + ------ + ValueError + If the context is not asynchronous. + RuntimeError + If restoration execution or typed live verification fails. + """ + from libtmux.experimental.ops import SessionId, ShowOptions, arun + + if context.mode is not ExecutionMode.ASYNC: + message = "async restoration requires an async run context" + raise ValueError(message) + engine = t.cast("AsyncTmuxEngine", context.engine) + (await restoration.aexecute(engine, planner=planner)).raise_for_status() + restored_snapshot = await snapshot_topology_async(context) + restored_options = ( + await arun(ShowOptions(target=SessionId(session_id)), engine) + ).raise_for_status() + _verify_restored_state( + context, + restored_snapshot, + options=restored_options.options, + window_ids=window_ids, + pane_ids=pane_ids, ) + return time.monotonic_ns() async def mutate_async( @@ -4399,10 +4562,10 @@ async def mutate_async( pane_ids=pane_ids, ) planner = BatchingPlanner() - baseline = _current_activity_epoch(context) verified = False restored = False duration_ns = 0 + primary_error: BaseException | None = None try: started_ns = time.perf_counter_ns() mutation_result = await mutation.aexecute(engine, planner=planner) @@ -4420,21 +4583,43 @@ async def mutate_async( titles=titles, ) verified = True - finally: - (await restoration.aexecute(engine, planner=planner)).raise_for_status() - restored_snapshot = await snapshot_topology_async(context) - restored_options = ( - await arun(ShowOptions(target=SessionId(session_id)), engine) - ).raise_for_status() - _verify_restored_state( + except BaseException as error: # noqa: BLE001 + primary_error = error + + restoration_task = asyncio.create_task( + _restore_mutation_async( context, - restored_snapshot, - options=restored_options.options, + restoration, + planner, + session_id=session_id, window_ids=window_ids, pane_ids=pane_ids, - ) - restored = True - activity_after = await _wait_activity_advance_async(context, baseline) + ), + name="libtmux-mutation-restoration", + ) + while not restoration_task.done(): + try: + await asyncio.shield(restoration_task) + except asyncio.CancelledError as cancellation: # noqa: PERF203 + if not restoration_task.cancelled() and primary_error is None: + primary_error = cancellation + except BaseException: # noqa: BLE001 + break + restoration_error: BaseException | None = None + restoration_verified_monotonic_ns = 0 + try: + restoration_verified_monotonic_ns = restoration_task.result() + except BaseException as error: # noqa: BLE001 + restoration_error = error + if restoration_error is not None: + if primary_error is not None: + raise primary_error from restoration_error + raise restoration_error + restored = True + if primary_error is not None: + raise primary_error + activity_baseline = _current_activity_heartbeat(context, max_age_s=2.0) + activity_after = await _wait_activity_advance_async(context, activity_baseline) steps = len(mutation.explain(planner)) return MutationResult( duration_ns=duration_ns, @@ -4445,8 +4630,9 @@ async def mutate_async( generation=generation_text, verified=verified, restored=restored, - activity_before_epoch=baseline, - activity_after_epoch=activity_after, + restoration_verified_monotonic_ns=restoration_verified_monotonic_ns, + activity_baseline=activity_baseline, + activity_after=activity_after, ) @@ -4766,6 +4952,161 @@ def _capture_planner(strategy: CaptureStrategy) -> object: raise ValueError(message) +def _activity_frame_epochs(lines: tuple[str, ...]) -> tuple[int, ...]: + """Extract fuzzer frame epochs from retained pane lines. + + >>> _activity_frame_epochs(("[editor epoch=3] x", "other")) + (3,) + + Parameters + ---------- + lines : tuple[str, ...] + Captured pane lines in display order. + + Returns + ------- + tuple[int, ...] + Nonnegative epochs from complete bracketed fuzzer frame prefixes. + """ + epochs: list[int] = [] + for line in lines: + prefix, separator, remainder = line.partition(" epoch=") + epoch_text, closing, _tail = remainder.partition("]") + if prefix.startswith("[") and separator and closing and epoch_text.isdigit(): + epochs.append(int(epoch_text)) + return tuple(epochs) + + +def _streams_reached_epoch(context: RunContext, epoch: int) -> bool: + r"""Return whether every source stream has emitted ``epoch`` or newer. + + >>> with tempfile.TemporaryDirectory() as temporary: + ... stream = pathlib.Path(temporary) / "stream.log" + ... _ = stream.write_text("[editor epoch=3] x\n", encoding="utf-8") + ... _streams_reached_epoch(types.SimpleNamespace(streams=(stream,)), 3) + True + + Parameters + ---------- + context : RunContext + Active run owning the source streams. + epoch : int + Exact observed heartbeat epoch required in every stream. + + Returns + ------- + bool + Whether every stream contains an epoch greater than or equal to target. + """ + for stream in context.streams: + try: + lines = tuple(stream.read_text(encoding="utf-8").splitlines()) + except OSError: + return False + if max(_activity_frame_epochs(lines), default=-1) < epoch: + return False + return True + + +def _wait_stream_epoch_sync( + context: RunContext, + epoch: int, + *, + timeout_s: float = 2.0, +) -> None: + """Wait outside capture timing until every source emitted ``epoch``. + + >>> try: + ... _wait_stream_epoch_sync(types.SimpleNamespace(), 1, timeout_s=0) + ... except ValueError as error: + ... print(error) + stream epoch timeout must be positive + + Parameters + ---------- + context : RunContext + Active run owning the source streams and fuzzer. + epoch : int + Exact observed heartbeat epoch required before capture. + timeout_s : float + Maximum monotonic wait outside the measured capture cell. + + Raises + ------ + ValueError + If ``timeout_s`` is not positive. + RuntimeError + If the fuzzer exits before source readiness. + TimeoutError + If a source stream does not reach the epoch in time. + """ + if timeout_s <= 0: + message = "stream epoch timeout must be positive" + raise ValueError(message) + deadline = time.monotonic() + timeout_s + while time.monotonic() < deadline: + if context.fuzzer.poll() is not None: + message = "fuzzer exited before current capture epoch" + raise RuntimeError(message) + if _streams_reached_epoch(context, epoch): + return + time.sleep(0.005) + message = f"source streams did not reach capture epoch {epoch}" + raise TimeoutError(message) + + +async def _wait_stream_epoch_async( + context: RunContext, + epoch: int, + *, + timeout_s: float = 2.0, +) -> None: + """Asynchronously wait outside timing for the observed source epoch. + + >>> async def invalid_stream_wait(): + ... try: + ... await _wait_stream_epoch_async( + ... types.SimpleNamespace(), 1, timeout_s=0 + ... ) + ... except ValueError as error: + ... return str(error) + >>> asyncio.run(invalid_stream_wait()) + 'stream epoch timeout must be positive' + + Parameters + ---------- + context : RunContext + Active run owning the source streams and fuzzer. + epoch : int + Exact observed heartbeat epoch required before capture. + timeout_s : float + Maximum event-loop wait outside the measured capture cell. + + Raises + ------ + ValueError + If ``timeout_s`` is not positive. + RuntimeError + If the fuzzer exits before source readiness. + TimeoutError + If a source stream does not reach the epoch in time. + """ + if timeout_s <= 0: + message = "stream epoch timeout must be positive" + raise ValueError(message) + loop = asyncio.get_running_loop() + deadline = loop.time() + timeout_s + while loop.time() < deadline: + if context.fuzzer.poll() is not None: + message = "fuzzer exited before current capture epoch" + raise RuntimeError(message) + if _streams_reached_epoch(context, epoch): + return + await asyncio.sleep(0.005) + message = f"source streams did not reach capture epoch {epoch}" + raise TimeoutError(message) + + def _accepted_capture( context: RunContext, plan: LazyPlan, @@ -4777,18 +5118,19 @@ def _accepted_capture( >>> from libtmux.experimental.ops import CapturePane, LazyPlan, PaneId >>> plan = LazyPlan() - >>> _ = plan.add(CapturePane(target=PaneId("%1"))) - >>> raw = CapturePane(target=PaneId("%1")).build_result( - ... returncode=0, stdout=("epoch",) + >>> operation = CapturePane(target=PaneId("%1")) + >>> _ = plan.add(operation) + >>> raw = operation.build_result( + ... returncode=0, stdout=("epoch", "[editor epoch=2] current") ... ) >>> context = types.SimpleNamespace( ... pane_ids=("%1",), activity_marker="epoch", activity_epoch=2, - ... lane=EngineLane.CONTROL, + ... heartbeat_epoch=2, lane=EngineLane.CONTROL, ... ) >>> _accepted_capture( ... context, plan, types.SimpleNamespace(results=(raw,)), "serial", 4 ... ).line_count - 1 + 2 Parameters ---------- @@ -4811,15 +5153,42 @@ def _accepted_capture( Raises ------ RuntimeError - If a result is not a typed capture, is empty, or lacks current epoch. + If result attribution, cardinality, content, or current epoch is invalid. """ - from libtmux.experimental.ops import BatchingPlanner, CapturePaneResult + from libtmux.experimental.ops import ( + BatchingPlanner, + CapturePane, + CapturePaneResult, + PaneId, + ) captures: list[PaneCapture] = [] results = tuple(plan_result.results) - if len(results) != len(context.pane_ids): + operations = tuple(plan.operations) + if len(operations) != len(context.pane_ids) or len(results) != len(operations): message = "capture result count did not match stable pane IDs" raise RuntimeError(message) + expected_targets = tuple( + operation.target.value + if isinstance(operation, CapturePane) and isinstance(operation.target, PaneId) + else None + for operation in operations + ) + result_targets = tuple( + result.operation.target.value + if isinstance(result, CapturePaneResult) + and isinstance(result.operation, CapturePane) + and isinstance(result.operation.target, PaneId) + else None + for result in results + ) + if expected_targets != context.pane_ids or result_targets != context.pane_ids: + message = "capture result target order did not match stable pane IDs" + raise RuntimeError(message) + current_epoch = context.heartbeat_epoch + if type(current_epoch) is not int or current_epoch < 0: + message = "capture has no observed current heartbeat epoch" + raise RuntimeError(message) for pane_id, result in zip(context.pane_ids, results, strict=True): if not isinstance(result, CapturePaneResult): message = f"capture for {pane_id} did not return typed lines" @@ -4830,6 +5199,9 @@ def _accepted_capture( raise RuntimeError(message) marker = t.cast(str, context.activity_marker) if marker not in "\n".join(result.lines): + message = f"capture for {pane_id} lacks its run-scoped activity marker" + raise RuntimeError(message) + if max(_activity_frame_epochs(result.lines), default=-1) < current_epoch: message = f"capture for {pane_id} lacks current activity epoch" raise RuntimeError(message) captures.append(PaneCapture(pane_id, result.lines)) @@ -4855,7 +5227,7 @@ def _accepted_capture( captures=tuple(captures), line_count=line_count, byte_count=byte_count, - epoch=t.cast(int, context.activity_epoch), + epoch=current_epoch, verified=True, ) @@ -4887,12 +5259,12 @@ def capture_all_sync( CaptureResult Retained typed lines and exact work counts. """ - from libtmux.experimental.ops import Planner - _require_active_phase_context(context, ExecutionMode.SYNC) plan = _capture_plan(context) planner = t.cast("Planner", _capture_planner(strategy)) engine = t.cast("TmuxEngine", context.engine) + heartbeat = _current_activity_heartbeat(context, max_age_s=2.0) + _wait_stream_epoch_sync(context, heartbeat.epoch) started_ns = time.perf_counter_ns() result = plan.execute(engine, planner=planner) duration_ns = time.perf_counter_ns() - started_ns @@ -4929,12 +5301,12 @@ async def capture_all_async( CaptureResult Retained typed lines and exact work counts. """ - from libtmux.experimental.ops import Planner - _require_active_phase_context(context, ExecutionMode.ASYNC) plan = _capture_plan(context) planner = t.cast("Planner", _capture_planner(strategy)) engine = t.cast("AsyncTmuxEngine", context.engine) + heartbeat = _current_activity_heartbeat(context, max_age_s=2.0) + await _wait_stream_epoch_async(context, heartbeat.epoch) started_ns = time.perf_counter_ns() result = await plan.aexecute(engine, planner=planner) duration_ns = time.perf_counter_ns() - started_ns @@ -5016,7 +5388,7 @@ def _search_result( Parameters ---------- - family : {"server-side", "snapshot", "end-to-end", "contents"} + family : {"classic", "snapshot", "end-to-end", "contents"} Explicit search timing boundary. kind : {"sessions", "windows", "panes"} Result object kind. @@ -5107,7 +5479,7 @@ def search_server_side( matches = t.cast(cabc.Iterable[object], search(filter=filter_expression)) duration_ns = time.perf_counter_ns() - started_ns return _search_result( - family="server-side", + family="classic", kind=kind, scanned_count=len(candidates), target=target, @@ -5389,8 +5761,8 @@ async def run_repeatable_phase( warmup: int, runs: int, seed: int, + live_postcondition: cabc.Callable[[PhaseMeasurement], object], snapshot_resources: cabc.Callable[[], HostSnapshot] | None = None, - live_postcondition: cabc.Callable[[PhaseMeasurement], object] | None = None, ) -> RepeatablePhaseResult: """Deterministically interleave strategies and retain accepted timed rows. @@ -5406,6 +5778,7 @@ async def run_repeatable_phase( ... ) ... return await run_repeatable_phase( ... {"one": measured}, warmup=0, runs=1, seed=2, + ... live_postcondition=lambda measurement: measurement.verified, ... snapshot_resources=lambda: HostSnapshot(), ... ) >>> len(asyncio.run(example()).samples) @@ -5421,10 +5794,10 @@ async def run_repeatable_phase( Timed accepted invocation count requested per strategy. seed : int Deterministic base-order shuffle seed. + live_postcondition : collections.abc.Callable + Required independent sync or async live check run after typed validation. snapshot_resources : collections.abc.Callable[[], HostSnapshot] | None Injectable resource sampler; defaults to the live process/cgroup probe. - live_postcondition : collections.abc.Callable | None - Optional sync or async check run after typed validation; defaults true. Returns ------- @@ -5461,12 +5834,9 @@ async def run_repeatable_phase( measurement = _validated_phase_measurement( await _await_if_needed(produced) ) + postcondition = await _await_if_needed(live_postcondition(measurement)) + _require_live_postcondition(postcondition) resources_after = sampler() - if live_postcondition is not None: - postcondition = await _await_if_needed( - live_postcondition(measurement) - ) - _require_live_postcondition(postcondition) if stage == "timed": samples.append( RawSample( diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index c04038fbc3..a9adc02240 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -1483,6 +1483,88 @@ def _checksum_ids(ids: tuple[str, ...]) -> str: return hashlib.sha256("\0".join(ids).encode()).hexdigest() +def _capture_frame_epochs(lines: tuple[str, ...]) -> tuple[int, ...]: + """Parse literal fuzzer frame epochs without benchmark implementation help.""" + epochs: list[int] = [] + for line in lines: + prefix, separator, remainder = line.partition(" epoch=") + epoch_text, closing, _tail = remainder.partition("]") + if prefix.startswith("[") and separator and closing and epoch_text.isdigit(): + epochs.append(int(epoch_text)) + return tuple(epochs) + + +@pytest.mark.parametrize("mode_name", ("sync", "async")) +def test_activity_advance_rejects_stalled_heartbeat_in_both_modes( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + mode_name: str, +) -> None: + """A fixed live marker must not satisfy post-restoration freshness.""" + heartbeat = tmp_path / "fuzzer" / "heartbeat.json" + heartbeat.parent.mkdir() + benchmark_module.write_json_atomic( + heartbeat, + { + "schema_version": 1, + "run_id": "stalled-run", + "state": "active", + "epoch": 41, + "monotonic_ns": time.monotonic_ns(), + }, + ) + context = types.SimpleNamespace( + scratch=tmp_path, + run_id="stalled-run", + fuzzer=types.SimpleNamespace(poll=lambda: None), + heartbeat_epoch=40, + heartbeat_monotonic_ns=-1, + ) + baseline = benchmark_module._current_activity_heartbeat(context, max_age_s=1.0) + + if mode_name == "async": + with pytest.raises(TimeoutError, match="did not advance"): + asyncio.run( + benchmark_module._wait_activity_advance_async( + context, baseline, timeout_s=0.02 + ) + ) + else: + with pytest.raises(TimeoutError, match="did not advance"): + benchmark_module._wait_activity_advance_sync( + context, baseline, timeout_s=0.02 + ) + + +def test_current_activity_heartbeat_rejects_stale_publication( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """An old active marker must not prove that the fuzzer is still moving.""" + heartbeat = tmp_path / "fuzzer" / "heartbeat.json" + heartbeat.parent.mkdir() + benchmark_module.write_json_atomic( + heartbeat, + { + "schema_version": 1, + "run_id": "stale-run", + "state": "active", + "epoch": 99, + "monotonic_ns": time.monotonic_ns() - 1_000_000_000, + }, + ) + context = types.SimpleNamespace( + scratch=tmp_path, + run_id="stale-run", + fuzzer=types.SimpleNamespace(poll=lambda: None), + heartbeat_epoch=1, + heartbeat_monotonic_ns=-1, + ) + + with pytest.raises(RuntimeError, match="stale"): + benchmark_module._current_activity_heartbeat(context, max_age_s=0.01) + + @pytest.mark.parametrize(("lane_name", "mode_name"), _PHASE_LANES) def test_live_mutation_phase_restores_stable_targets_and_keeps_activity_advancing( benchmark_module: types.ModuleType, @@ -1550,7 +1632,16 @@ async def exercise_async() -> None: ) assert result.verified assert result.restored - assert result.activity_after_epoch > result.activity_before_epoch + assert result.activity_baseline.epoch < result.activity_after.epoch + assert ( + result.activity_baseline.published_monotonic_ns + < result.activity_after.published_monotonic_ns + ) + assert ( + result.restoration_verified_monotonic_ns + <= result.activity_baseline.observed_monotonic_ns + < result.activity_after.observed_monotonic_ns + ) assert tuple(session.name for session in restored.sessions) == ( context.expected_session_names ) @@ -1567,6 +1658,125 @@ async def exercise_async() -> None: assert cleanup.complete, cleanup.errors +@pytest.mark.parametrize("lane_name", ("subprocess", "control")) +def test_async_mutation_shields_restoration_through_repeated_cancellation( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, + lane_name: str, +) -> None: + """Cancellation must wait for real restoration and retain its first cause. + + The benchmark has no public restoration boundary, so this test wraps only + ``LazyPlan.aexecute`` to pause its second real plan dispatch. The actual + async engine, mutation plan, restoration plan, snapshots, and cleanup stay + live. + """ + from libtmux.experimental.ops import LazyPlan, SessionId, ShowOptions, arun + + topology = benchmark_module.Topology(2, 3, 2) + lane = benchmark_module.EngineLane(lane_name) + scratch = tmp_path / f"cancel-restoration-{lane_name}" + original_aexecute = LazyPlan.aexecute + plan_calls = 0 + restoration_started: asyncio.Event + release_restoration: asyncio.Event + + async def gated_aexecute( + plan: t.Any, + engine: t.Any, + *, + version: str | None = None, + planner: t.Any = None, + on_step: t.Any = None, + ) -> t.Any: + nonlocal plan_calls + plan_calls += 1 + if plan_calls == 2: + restoration_started.set() + await release_restoration.wait() + await original_aexecute( + plan, + engine, + version=version, + planner=planner, + on_step=on_step, + ) + message = "restoration evidence failure" + raise RuntimeError(message) + return await original_aexecute( + plan, + engine, + version=version, + planner=planner, + on_step=on_step, + ) + + async def exercise() -> None: + nonlocal plan_calls, restoration_started, release_restoration + restoration_started = asyncio.Event() + release_restoration = asyncio.Event() + context = await benchmark_module.setup_async( + topology, + lane, + scratch, + run_id=f"cancel-restoration-{lane_name}", + delayed_ordinal=7, + ) + cleanup = None + try: + benchmark_module.release_activity_gate(context) + await benchmark_module.verify_activity_async(context) + plan_calls = 0 + monkeypatch.setattr(LazyPlan, "aexecute", gated_aexecute) + mutation = asyncio.create_task( + benchmark_module.mutate_async(context, generation=19), + name="mutation-under-cancellation", + ) + await asyncio.wait_for(restoration_started.wait(), timeout=5.0) + mutation.cancel("original cancellation") + await asyncio.sleep(0) + mutation.cancel("repeated cancellation") + release_restoration.set() + with pytest.raises(asyncio.CancelledError) as cancelled: + await mutation + assert cancelled.value.args == ("original cancellation",) + assert isinstance(cancelled.value.__cause__, RuntimeError) + assert str(cancelled.value.__cause__) == "restoration evidence failure" + + restored = await benchmark_module.snapshot_topology_async(context) + option_result = ( + await arun( + ShowOptions(target=SessionId(context.session_ids[0])), + context.engine, + ) + ).raise_for_status() + assert "@libtmux_bench_generation" not in option_result.options + assert {window.name for window in restored.windows} == set( + context.expected_window_names + ) + restored_window_ids = { + window.window_id + for window in restored.windows + if window.session_id == context.session_ids[0] + } + assert all( + pane.fields.get("pane_title") in {None, ""} + for pane in restored.panes + if pane.window_id in restored_window_ids + ) + assert not any( + task.get_name() == "libtmux-mutation-restoration" and not task.done() + for task in asyncio.all_tasks() + ) + finally: + release_restoration.set() + cleanup = await benchmark_module.cleanup_run(context) + assert cleanup.complete, cleanup.errors + + asyncio.run(exercise()) + + @pytest.mark.parametrize(("lane_name", "mode_name"), _PHASE_LANES) def test_live_enumeration_phase_accepts_exact_rows_and_stable_id_checksums( benchmark_module: types.ModuleType, @@ -1668,12 +1878,20 @@ async def exercise_async() -> None: try: benchmark_module.release_activity_gate(context) await benchmark_module.verify_activity_async(context) + heartbeat = json.loads( + (scratch / "fuzzer" / "heartbeat.json").read_text(encoding="utf-8") + ) serial = await benchmark_module.capture_all_async( context, strategy="serial" ) + assert serial.epoch >= heartbeat["epoch"] + heartbeat = json.loads( + (scratch / "fuzzer" / "heartbeat.json").read_text(encoding="utf-8") + ) batched = await benchmark_module.capture_all_async( context, strategy="batched" ) + assert batched.epoch >= heartbeat["epoch"] finally: cleanup = await benchmark_module.cleanup_run(context) @@ -1690,8 +1908,16 @@ async def exercise_async() -> None: try: benchmark_module.release_activity_gate(context) benchmark_module.verify_activity_sync(context) + heartbeat = json.loads( + (scratch / "fuzzer" / "heartbeat.json").read_text(encoding="utf-8") + ) serial = benchmark_module.capture_all_sync(context, strategy="serial") + assert serial.epoch >= heartbeat["epoch"] + heartbeat = json.loads( + (scratch / "fuzzer" / "heartbeat.json").read_text(encoding="utf-8") + ) batched = benchmark_module.capture_all_sync(context, strategy="batched") + assert batched.epoch >= heartbeat["epoch"] finally: cleanup = asyncio.run(benchmark_module.cleanup_run(context)) @@ -1705,7 +1931,11 @@ async def exercise_async() -> None: assert result.duration_ns > 0 assert result.line_count >= 12 assert result.byte_count > 0 - assert result.epoch == context.activity_epoch + assert result.epoch >= context.activity_epoch + assert all( + max(_capture_frame_epochs(capture.lines), default=-1) >= result.epoch + for capture in result.captures + ) assert result.metrics.operations == 12 assert result.metrics.tmux_requests == 12 assert result.metrics.process_starts == ( @@ -1721,105 +1951,284 @@ async def exercise_async() -> None: assert cleanup.complete, cleanup.errors +@pytest.mark.parametrize( + "result_targets", + (("%2", "%1"), ("%1", "%1"), ("%1", "%9")), + ids=("reordered", "duplicate", "wrong"), +) +def test_capture_rejects_unattributed_typed_results( + benchmark_module: types.ModuleType, + result_targets: tuple[str, str], +) -> None: + """Typed lines from reordered, duplicate, or wrong targets are invalid.""" + from libtmux.experimental.ops import CapturePane, PaneId + + context = types.SimpleNamespace( + pane_ids=("%1", "%2"), + activity_marker="LIBTMUX_EPOCH run=capture-run epoch=1", + activity_epoch=1, + heartbeat_epoch=8, + lane=benchmark_module.EngineLane.CONTROL, + ) + plan = benchmark_module._capture_plan(context) + results = tuple( + CapturePane(target=PaneId(pane_id)).build_result( + returncode=0, + stdout=( + context.activity_marker, + "[editor epoch=8] current", + ), + ) + for pane_id in result_targets + ) + + with pytest.raises(RuntimeError, match="target"): + benchmark_module._accepted_capture( + context, + plan, + types.SimpleNamespace(results=results), + "batched", + 5, + ) + + +def _request_content_sentinel( + benchmark_module: types.ModuleType, + context: t.Any, + request_id: str, +) -> str: + """Request one Task 1 sentinel and wait outside content-search timing.""" + requested_ns = time.monotonic_ns() + benchmark_module.write_json_atomic( + context.scratch / "fuzzer" / "requests" / f"{request_id}.json", + { + "schema_version": 1, + "run_id": context.run_id, + "request_id": request_id, + "requested_monotonic_ns": requested_ns, + "value": "CONTENT-ONLY", + }, + ) + evidence_path = context.scratch / "fuzzer" / "sentinels" / f"{request_id}.json" + deadline = time.monotonic() + 3.0 + while time.monotonic() < deadline: + try: + evidence = json.loads(evidence_path.read_text(encoding="utf-8")) + except (FileNotFoundError, json.JSONDecodeError): + time.sleep(0.01) + continue + assert evidence["run_id"] == context.run_id + assert evidence["request_id"] == request_id + assert evidence["requested_monotonic_ns"] == requested_ns + return t.cast(str, evidence["sentinel"]) + message = "Task 1 sentinel evidence did not arrive" + raise AssertionError(message) + + +@pytest.mark.parametrize(("lane_name", "mode_name"), _PHASE_LANES) +@pytest.mark.parametrize( + ("target_position", "delayed_ordinal"), + (("first", 0), ("middle", 6), ("last", 11)), +) def test_live_search_phase_keeps_server_snapshot_end_to_end_and_content_distinct( benchmark_module: types.ModuleType, tmp_path: pathlib.Path, + lane_name: str, + mode_name: str, + target_position: str, + delayed_ordinal: int, ) -> None: - """Each search family must scan its declared source and return one target.""" + """Every applicable family must find explicit targets in every live lane.""" topology = benchmark_module.Topology(2, 3, 2) - scratch = tmp_path / "search" - context = benchmark_module.setup_sync( - topology, - benchmark_module.EngineLane.SUBPROCESS, - scratch, - run_id="search-phase", - delayed_ordinal=7, - ) - cleanup = None - try: - benchmark_module.release_activity_gate(context) - benchmark_module.verify_activity_sync(context) - snapshot = benchmark_module.snapshot_topology_sync(context) + lane = benchmark_module.EngineLane(lane_name) + run_id = f"s-{lane_name[0]}-{mode_name[0]}-{target_position[0]}" + scratch = tmp_path / f"search-{lane_name}-{mode_name}-{target_position}" + context: t.Any = None + cleanup: t.Any = None + observed: dict[tuple[str, str, str], tuple[int, tuple[str, ...]]] = {} + + def exercise_metadata(snapshot: t.Any) -> None: + """Run classic, pre-materialized, and end-to-end metadata families.""" ids_by_kind = { "sessions": context.session_ids, "windows": context.window_ids, "panes": context.pane_ids, } + rows_by_kind = { + "sessions": QueryList(snapshot.sessions), + "windows": QueryList(snapshot.windows), + "panes": QueryList(snapshot.panes), + } + indexes = {"first": 0, "middle": None, "last": -1} for kind, ids in ids_by_kind.items(): - for target in (ids[0], ids[len(ids) // 2], ids[-1]): - result = benchmark_module.search_server_side( + for position, configured_index in indexes.items(): + index = len(ids) // 2 if configured_index is None else configured_index + target = ids[index] + classic = benchmark_module.search_server_side( context, kind=kind, target=target ) - assert result.family == "server-side" - assert result.matched_ids == (target,) - assert result.scanned_count == len(ids) - assert result.verified - - snapshot_rows = QueryList(snapshot.sessions) - snapshot_result = benchmark_module.search_snapshot( - snapshot_rows, - kind="sessions", - target=context.session_ids[1], + snapshot_only = benchmark_module.search_snapshot( + rows_by_kind[kind], kind=kind, target=target + ) + end_to_end = benchmark_module.search_end_to_end( + context, kind=kind, target=target + ) + assert classic.family == "classic" + assert snapshot_only.family == "snapshot" + assert end_to_end.family == "end-to-end" + for family, result in ( + ("classic", classic), + ("snapshot", snapshot_only), + ("end-to-end", end_to_end), + ): + assert result.target == target + assert result.matched_ids == (target,) + assert result.verified + observed[family, kind, position] = ( + result.scanned_count, + result.matched_ids, + ) + + def assert_explicit_expectations() -> None: + """Compare named family/target cells with hand-derived cardinalities.""" + expected_counts = {"sessions": 2, "windows": 6, "panes": 12} + for family in ("classic", "snapshot", "end-to-end"): + for kind, expected_count in expected_counts.items(): + ids = { + "sessions": context.session_ids, + "windows": context.window_ids, + "panes": context.pane_ids, + }[kind] + for position, target in ( + ("first", ids[0]), + ("middle", ids[len(ids) // 2]), + ("last", ids[-1]), + ): + assert observed[family, kind, position] == ( + expected_count, + (target,), + ) + assert observed["contents", "panes", target_position] == ( + 12, + (context.pane_ids[delayed_ordinal],), ) - end_to_end_result = benchmark_module.search_end_to_end( - context, - kind="windows", - target=context.window_ids[3], + + async def exercise_async() -> None: + """Keep typed async capture and final topology proof on one loop.""" + nonlocal context, cleanup + context = await benchmark_module.setup_async( + topology, + lane, + scratch, + run_id=run_id, + delayed_ordinal=delayed_ordinal, ) + try: + benchmark_module.release_activity_gate(context) + await benchmark_module.verify_activity_async(context) + snapshot = await benchmark_module.snapshot_topology_async(context) + exercise_metadata(snapshot) + sentinel = _request_content_sentinel( + benchmark_module, context, f"content-{target_position}" + ) + deadline = time.monotonic() + 3.0 + captures = None + while time.monotonic() < deadline: + candidate = await benchmark_module.capture_all_async( + context, strategy="batched" + ) + if any(sentinel in capture.lines for capture in candidate.captures): + captures = candidate + break + await asyncio.sleep(0.01) + assert captures is not None + content = benchmark_module.search_contents( + captures, + token=sentinel, + expected_pane_id=context.pane_ids[delayed_ordinal], + ) + observed["contents", "panes", target_position] = ( + content.scanned_count, + content.matched_ids, + ) + final_snapshot = await benchmark_module.snapshot_topology_async(context) + assert tuple(row.session_id for row in final_snapshot.sessions) == ( + context.session_ids + ) + assert tuple(row.window_id for row in final_snapshot.windows) == ( + context.window_ids + ) + assert ( + tuple(row.pane_id for row in final_snapshot.panes) == context.pane_ids + ) + assert ( + benchmark_module._current_activity_heartbeat( + context, max_age_s=2.0 + ).epoch + >= captures.epoch + ) + assert_explicit_expectations() + finally: + cleanup = await benchmark_module.cleanup_run(context) - request_id = "content-search" - requested_ns = time.monotonic_ns() - benchmark_module.write_json_atomic( - scratch / "fuzzer" / "requests" / f"{request_id}.json", - { - "schema_version": 1, - "run_id": context.run_id, - "request_id": request_id, - "requested_monotonic_ns": requested_ns, - "value": "CONTENT-ONLY", - }, + if mode_name == "async": + asyncio.run(exercise_async()) + else: + context = benchmark_module.setup_sync( + topology, + lane, + scratch, + run_id=run_id, + delayed_ordinal=delayed_ordinal, ) - evidence_path = scratch / "fuzzer" / "sentinels" / f"{request_id}.json" - deadline = time.monotonic() + 2.0 - evidence = None - while time.monotonic() < deadline: - try: - evidence = json.loads(evidence_path.read_text(encoding="utf-8")) - except (FileNotFoundError, json.JSONDecodeError): + try: + benchmark_module.release_activity_gate(context) + benchmark_module.verify_activity_sync(context) + snapshot = benchmark_module.snapshot_topology_sync(context) + exercise_metadata(snapshot) + sentinel = _request_content_sentinel( + benchmark_module, context, f"content-{target_position}" + ) + deadline = time.monotonic() + 3.0 + captures = None + while time.monotonic() < deadline: + candidate = benchmark_module.capture_all_sync( + context, strategy="batched" + ) + if any(sentinel in capture.lines for capture in candidate.captures): + captures = candidate + break time.sleep(0.01) - continue - break - assert evidence is not None - sentinel = evidence["sentinel"] - - captures = None - while time.monotonic() < deadline: - candidate = benchmark_module.capture_all_sync(context, strategy="batched") - if any(sentinel in capture.lines for capture in candidate.captures): - captures = candidate - break - time.sleep(0.01) - assert captures is not None - content_result = benchmark_module.search_contents( - captures, - token=sentinel, - expected_pane_id=context.delayed_pane_id, - ) - - assert snapshot_result.family == "snapshot" - assert snapshot_result.matched_ids == (context.session_ids[1],) - assert end_to_end_result.family == "end-to-end" - assert end_to_end_result.matched_ids == (context.window_ids[3],) - assert content_result.family == "contents" - assert content_result.matched_ids == (context.delayed_pane_id,) - assert content_result.token == sentinel - assert { - snapshot_result.scanned_count, - end_to_end_result.scanned_count, - content_result.scanned_count, - } == {2, 6, 12} - finally: - cleanup = asyncio.run(benchmark_module.cleanup_run(context)) + assert captures is not None + content = benchmark_module.search_contents( + captures, + token=sentinel, + expected_pane_id=context.pane_ids[delayed_ordinal], + ) + observed["contents", "panes", target_position] = ( + content.scanned_count, + content.matched_ids, + ) + final_snapshot = benchmark_module.snapshot_topology_sync(context) + assert tuple(row.session_id for row in final_snapshot.sessions) == ( + context.session_ids + ) + assert tuple(row.window_id for row in final_snapshot.windows) == ( + context.window_ids + ) + assert ( + tuple(row.pane_id for row in final_snapshot.panes) == context.pane_ids + ) + assert ( + benchmark_module._current_activity_heartbeat( + context, max_age_s=2.0 + ).epoch + >= captures.epoch + ) + assert_explicit_expectations() + finally: + cleanup = asyncio.run(benchmark_module.cleanup_run(context)) + assert cleanup is not None assert cleanup.complete, cleanup.errors @@ -1873,6 +2282,66 @@ def run() -> t.Any: assert all(sample.resources_after is not None for sample in result.samples) +def test_run_repeatable_phase_requires_live_postcondition_at_call_time( + benchmark_module: types.ModuleType, +) -> None: + """Omitting independent live validation must not create a coroutine.""" + with pytest.raises(TypeError, match="live_postcondition"): + benchmark_module.run_repeatable_phase( + {"only": lambda: None}, + warmup=0, + runs=1, + seed=1, + ) + + +@pytest.mark.parametrize("postcondition_kind", ("false", "exception")) +def test_run_repeatable_phase_rejects_postcondition_before_after_snapshot( + benchmark_module: types.ModuleType, + postcondition_kind: str, +) -> None: + """A false or failing live check must retain no sample or after snapshot.""" + snapshot_calls = 0 + + def snapshot_resources() -> t.Any: + nonlocal snapshot_calls + snapshot_calls += 1 + return benchmark_module.HostSnapshot(pids_current=snapshot_calls) + + def measured() -> t.Any: + return benchmark_module.SearchResult( + duration_ns=19, + family="snapshot", + kind="sessions", + scanned_count=2, + target="$1", + matched_ids=("$1",), + verified=True, + ) + + def live_postcondition(_measurement: t.Any) -> bool: + if postcondition_kind == "exception": + message = "independent live check failed" + raise RuntimeError(message) + return False + + result = asyncio.run( + benchmark_module.run_repeatable_phase( + {"only": measured}, + warmup=0, + runs=1, + seed=1, + snapshot_resources=snapshot_resources, + live_postcondition=live_postcondition, + ) + ) + + assert result.samples == () + assert result.failure is not None + assert result.failure.strategy == "only" + assert snapshot_calls == 1 + + def test_run_repeatable_phase_stops_without_appending_failed_duration( benchmark_module: types.ModuleType, ) -> None: @@ -1902,6 +2371,7 @@ def fail_second() -> t.Any: runs=3, seed=3, snapshot_resources=lambda: benchmark_module.HostSnapshot(), + live_postcondition=lambda _measurement: True, ) ) From 9df37eba33b98d0d57fb9b016246185e917a1ca6 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 09:54:28 -0500 Subject: [PATCH 17/67] Bench(feat[wait]): Time delayed output why: Quantify request-scoped delayed output without restarting the active topology or conflating the configured delay with waiter overhead. what: - Harden schema-v1 sentinel request identity and durable evidence - Add sync/async capture polling and decoded control-stream waits - Verify repeated timing, stale-token, drop, and subscription contracts --- scripts/bench_orchestration.py | 854 ++++++++++++++++++++++- scripts/orchestration_fuzzer.py | 63 +- tests/test_bench_orchestration_script.py | 477 +++++++++++++ tests/test_orchestration_fuzzer.py | 158 +++++ 4 files changed, 1539 insertions(+), 13 deletions(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index 9b28205ccb..47099c135b 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -53,6 +53,10 @@ from libtmux.server import Server +_SENTINEL_DELAY_S = 0.05 +_SENTINEL_DELAY_NS = 50_000_000 + + class EngineLane(str, enum.Enum): """Transport family used for one live benchmark server. @@ -947,6 +951,123 @@ class HeartbeatObservation: observed_monotonic_ns: int +@dataclasses.dataclass(frozen=True) +class SentinelRequest: + """One unique delayed-stream request published by the benchmark. + + Attributes + ---------- + request_id : str + Run-local canonical marker identity. + token : str + Exact canonical sentinel text expected from the delayed pane. + requested_monotonic_ns : int + Monotonic timestamp captured immediately before atomic publication. + configured_delay_ns : int + Fuzzer delay added to the request timestamp. + + Examples + -------- + >>> SentinelRequest("sample-1", "token", 7, 5).configured_delay_ns + 5 + """ + + request_id: str + token: str + requested_monotonic_ns: int + configured_delay_ns: int + + +WaitStrategy: t.TypeAlias = t.Literal["capture-poll", "control-stream"] + + +@dataclasses.dataclass(frozen=True) +class WaitResult: + """Verified delayed-sentinel detection and generator timing evidence. + + Attributes + ---------- + strategy : {"capture-poll", "control-stream"} + Observation mechanism used for this request. + request_id : str + Exact run-local request identity. + token : str + Exact canonical sentinel matched in the selected pane. + pane_id : str + Concrete delayed pane ID observed by the waiter. + configured_delay_ns : int + Fuzzer delay applied after request publication. + requested_monotonic_ns : int + Benchmark timestamp immediately before atomic request publication. + scheduled_monotonic_ns : int + Requested timestamp plus the configured delay. + emitted_monotonic_ns : int + Fuzzer timestamp immediately before the durable stream append. + detected_monotonic_ns : int + Benchmark timestamp when the exact sentinel bytes were detected. + scheduling_lateness_ns : int + Emitted timestamp minus scheduled timestamp. + detection_overhead_ns : int + Detected timestamp minus emitted timestamp. + poll_count : int + Typed capture requests issued by capture polling. + frame_count : int + Matching-pane control output frames consumed by stream waiting. + timeout_s : float + Configured monotonic wait deadline in seconds. + timed_out : bool + Whether the returned result represents a timeout; successful waits are false. + dropped_notification_delta : int + Control-engine notification drops observed during this request. + verified : bool + Whether exact token, evidence, timing, pane, and drop checks passed. + + Examples + -------- + >>> result = WaitResult( + ... "capture-poll", "sample-1", "token", "%1", 5, 7, 12, 14, 17, + ... 2, 3, 1, 0, 1.0, False, 0, True, + ... ) + >>> result.duration_ns + 3 + """ + + strategy: WaitStrategy + request_id: str + token: str + pane_id: str + configured_delay_ns: int + requested_monotonic_ns: int + scheduled_monotonic_ns: int + emitted_monotonic_ns: int + detected_monotonic_ns: int + scheduling_lateness_ns: int + detection_overhead_ns: int + poll_count: int + frame_count: int + timeout_s: float + timed_out: bool + dropped_notification_delta: int + verified: bool + + @property + def duration_ns(self) -> int: + """Return waiter overhead with the deliberate delay excluded. + + >>> WaitResult( + ... "control-stream", "r", "t", "%1", 5, 7, 12, 14, 18, + ... 2, 4, 0, 1, 1.0, False, 0, True, + ... ).duration_ns + 4 + + Returns + ------- + int + Detection timestamp minus actual emission timestamp. + """ + return self.detection_overhead_ns + + @dataclasses.dataclass(frozen=True) class MutationResult: """Verified and restored bulk mutation measurement. @@ -1551,6 +1672,8 @@ class RunContext: Precreated activity streams in mode order. delayed_ordinal : int Global pane ordinal assigned the delayed-match stream. + sentinel_delay_ns : int + Configured fuzzer delay for every request-scoped sentinel. expected_session_names : tuple[str, ...] Exact declared session names. expected_window_names : tuple[str, ...] @@ -1607,6 +1730,7 @@ class RunContext: fuzzer: subprocess.Popen[bytes] streams: tuple[pathlib.Path, ...] delayed_ordinal: int + sentinel_delay_ns: int expected_session_names: tuple[str, ...] expected_window_names: tuple[str, ...] expected_window_parents: tuple[tuple[str, str], ...] @@ -2693,7 +2817,7 @@ def start_fuzzer( "--duration", str(duration_s), "--delayed-match-after", - "0.05", + str(_SENTINEL_DELAY_S), "--sentinel-prefix", "READY", "--heartbeat-interval", @@ -2891,6 +3015,7 @@ def _prepare_context( fuzzer=fuzzer, streams=streams, delayed_ordinal=delayed_ordinal, + sentinel_delay_ns=_SENTINEL_DELAY_NS, expected_session_names=tuple( _session_name(run_id, index) for index in range(topology.sessions) ), @@ -4069,6 +4194,728 @@ async def _wait_activity_advance_async( raise TimeoutError(message) +def request_sentinel( + context: RunContext, + *, + request_id: str, + value: str = "READY", +) -> SentinelRequest: + """Atomically publish one unique request for the active delayed stream. + + Examples + -------- + >>> with tempfile.TemporaryDirectory() as temporary: + ... scratch = pathlib.Path(temporary) + ... (scratch / "fuzzer" / "requests").mkdir(parents=True) + ... (scratch / "fuzzer" / "sentinels").mkdir() + ... context = types.SimpleNamespace( + ... run_id="run-7", scratch=scratch, sentinel_delay_ns=5, + ... delayed_pane_id="%1", fuzzer=types.SimpleNamespace(poll=lambda: None), + ... ) + ... request = request_sentinel(context, request_id="sample-1", value="GO") + ... request.request_id, request.token.endswith("value=GO") + ('sample-1', True) + + Parameters + ---------- + context : RunContext + Active topology whose fuzzer owns the request directory. + request_id : str + Unique ASCII identifier containing only letters, digits, hyphens, or + underscores. + value : str + Nonempty single-line value embedded in the canonical sentinel. + + Returns + ------- + SentinelRequest + Exact request identity, token, publication time, and configured delay. + + Raises + ------ + ValueError + If request identity, value, delay, or delayed pane is invalid. + FileExistsError + If the request or its evidence path already exists. + RuntimeError + If the fuzzer has exited. + OSError + If atomic marker publication fails. + """ + if ( + not request_id + or len(request_id) > 128 + or any( + not (character.isascii() and (character.isalnum() or character in "-_")) + for character in request_id + ) + ): + message = ( + "request_id must contain 1-128 ASCII letters, digits, hyphens, " + "or underscores" + ) + raise ValueError(message) + if not isinstance(value, str) or not value or "\n" in value or "\r" in value: + message = "sentinel value must be a nonempty single line" + raise ValueError(message) + delay_ns = context.sentinel_delay_ns + if type(delay_ns) is not int or delay_ns < 0: + message = "sentinel delay must be a nonnegative integer nanosecond value" + raise ValueError(message) + if not isinstance(context.delayed_pane_id, str) or not context.delayed_pane_id: + message = "wait requires a verified delayed pane ID" + raise ValueError(message) + if context.fuzzer.poll() is not None: + message = "fuzzer exited before sentinel request" + raise RuntimeError(message) + requests = context.scratch / "fuzzer" / "requests" + sentinels = context.scratch / "fuzzer" / "sentinels" + request_path = requests / f"{request_id}.json" + evidence_path = sentinels / f"{request_id}.json" + if request_path.exists() or evidence_path.exists(): + message = f"sentinel request already exists: {request_id}" + raise FileExistsError(message) + token = f"LIBTMUX_SENTINEL run={context.run_id} request={request_id} value={value}" + requested_ns = time.monotonic_ns() + write_json_atomic( + request_path, + { + "schema_version": 1, + "run_id": context.run_id, + "request_id": request_id, + "requested_monotonic_ns": requested_ns, + "value": value, + }, + ) + return SentinelRequest(request_id, token, requested_ns, delay_ns) + + +def _validated_sentinel_evidence( + context: RunContext, + request: SentinelRequest, + marker: cabc.Mapping[str, t.Any], +) -> tuple[int, int, int]: + r"""Validate exact request evidence and return its derived timing fields. + + Examples + -------- + >>> token = "LIBTMUX_SENTINEL run=run-7 request=sample-1 value=GO" + >>> request = SentinelRequest("sample-1", token, 7, 5) + >>> marker = { + ... "schema_version": 1, "run_id": "run-7", "request_id": "sample-1", + ... "requested_monotonic_ns": 7, "configured_delay_ns": 5, + ... "scheduled_monotonic_ns": 12, "emitted_monotonic_ns": 14, + ... "scheduling_lateness_ns": 2, "sentinel": token, + ... "sentinel_sha256": hashlib.sha256(f"{token}\n".encode()).hexdigest(), + ... } + >>> _validated_sentinel_evidence( + ... types.SimpleNamespace(run_id="run-7"), request, marker + ... ) + (12, 14, 2) + + Parameters + ---------- + context : RunContext + Run identity required in the marker. + request : SentinelRequest + Exact request whose stream token was detected. + marker : collections.abc.Mapping[str, typing.Any] + Parsed atomic schema-v1 evidence. + + Returns + ------- + tuple[int, int, int] + Scheduled timestamp, emitted timestamp, and scheduling lateness. + + Raises + ------ + RuntimeError + If identity, token, hash, integer types, or timing arithmetic differs. + """ + integer_fields = ( + "requested_monotonic_ns", + "configured_delay_ns", + "scheduled_monotonic_ns", + "emitted_monotonic_ns", + "scheduling_lateness_ns", + ) + if ( + type(marker.get("schema_version")) is not int + or marker.get("schema_version") != 1 + or marker.get("run_id") != context.run_id + or marker.get("request_id") != request.request_id + or marker.get("sentinel") != request.token + or any( + type(marker.get(field)) is not int or marker[field] < 0 + for field in integer_fields + ) + ): + message = "sentinel evidence does not match request" + raise RuntimeError(message) + requested_ns = t.cast(int, marker["requested_monotonic_ns"]) + configured_delay_ns = t.cast(int, marker["configured_delay_ns"]) + scheduled_ns = t.cast(int, marker["scheduled_monotonic_ns"]) + emitted_ns = t.cast(int, marker["emitted_monotonic_ns"]) + scheduling_lateness_ns = t.cast(int, marker["scheduling_lateness_ns"]) + expected_hash = hashlib.sha256(f"{request.token}\n".encode()).hexdigest() + if ( + requested_ns != request.requested_monotonic_ns + or configured_delay_ns != request.configured_delay_ns + or scheduled_ns != requested_ns + configured_delay_ns + or emitted_ns < scheduled_ns + or scheduling_lateness_ns != emitted_ns - scheduled_ns + or marker.get("sentinel_sha256") != expected_hash + ): + message = "sentinel evidence does not match request" + raise RuntimeError(message) + return scheduled_ns, emitted_ns, scheduling_lateness_ns + + +def _read_sentinel_evidence( + context: RunContext, + request: SentinelRequest, +) -> cabc.Mapping[str, t.Any] | None: + """Read absent evidence as ``None`` and reject every completed mismatch. + + >>> _read_sentinel_evidence( + ... types.SimpleNamespace(scratch=pathlib.Path("missing"), run_id="run-7"), + ... SentinelRequest("sample-1", "token", 7, 5), + ... ) is None + True + + Parameters + ---------- + context : RunContext + Run whose evidence directory is authoritative. + request : SentinelRequest + Exact request already detected in pane output. + + Returns + ------- + collections.abc.Mapping[str, typing.Any] | None + Valid matching evidence, or ``None`` before its atomic publication. + + Raises + ------ + RuntimeError + If a published file is malformed or does not match the request. + TypeError + If a published JSON value is not a mapping. + """ + path = context.scratch / "fuzzer" / "sentinels" / f"{request.request_id}.json" + try: + parsed = json.loads(path.read_text(encoding="utf-8")) + except FileNotFoundError: + return None + except (json.JSONDecodeError, OSError) as error: + message = "sentinel evidence is not a complete JSON marker" + raise RuntimeError(message) from error + if not isinstance(parsed, dict): + message = "sentinel evidence is not a mapping" + raise TypeError(message) + marker = t.cast(dict[str, t.Any], parsed) + _validated_sentinel_evidence(context, request, marker) + return marker + + +def _completed_wait_result( + context: RunContext, + request: SentinelRequest, + evidence: cabc.Mapping[str, t.Any], + *, + strategy: WaitStrategy, + detected_monotonic_ns: int, + poll_count: int, + frame_count: int, + timeout_s: float, + dropped_notification_delta: int, +) -> WaitResult: + r"""Combine exact evidence with detection facts and reject invalid samples. + + Examples + -------- + >>> token = "LIBTMUX_SENTINEL run=run-7 request=sample-1 value=GO" + >>> request = SentinelRequest("sample-1", token, 7, 5) + >>> evidence = { + ... "schema_version": 1, "run_id": "run-7", "request_id": "sample-1", + ... "requested_monotonic_ns": 7, "configured_delay_ns": 5, + ... "scheduled_monotonic_ns": 12, "emitted_monotonic_ns": 14, + ... "scheduling_lateness_ns": 2, "sentinel": token, + ... "sentinel_sha256": hashlib.sha256(f"{token}\n".encode()).hexdigest(), + ... } + >>> result = _completed_wait_result( + ... types.SimpleNamespace(run_id="run-7", delayed_pane_id="%1"), + ... request, evidence, strategy="capture-poll", detected_monotonic_ns=17, + ... poll_count=1, frame_count=0, timeout_s=1.0, + ... dropped_notification_delta=0, + ... ) + >>> result.detection_overhead_ns + 3 + + Parameters + ---------- + context : RunContext + Run containing the concrete delayed pane ID. + request : SentinelRequest + Request whose exact token was detected. + evidence : collections.abc.Mapping[str, typing.Any] + Matching schema-v1 fuzzer evidence. + strategy : {"capture-poll", "control-stream"} + Detection mechanism used. + detected_monotonic_ns : int + Local exact-match timestamp. + poll_count : int + Typed capture operations issued. + frame_count : int + Matching-pane output notifications consumed. + timeout_s : float + Configured monotonic timeout. + dropped_notification_delta : int + Engine drop-counter change during the wait. + + Returns + ------- + WaitResult + Verified timing and request evidence. + + Raises + ------ + RuntimeError + If timing, counters, pane identity, or notification integrity is invalid. + """ + scheduled_ns, emitted_ns, scheduling_lateness_ns = _validated_sentinel_evidence( + context, request, evidence + ) + if detected_monotonic_ns < emitted_ns: + message = "sentinel detection predates fuzzer emission" + raise RuntimeError(message) + if dropped_notification_delta != 0: + message = f"control stream dropped {dropped_notification_delta} notification(s)" + raise RuntimeError(message) + if strategy == "capture-poll": + valid_counts = poll_count > 0 and frame_count == 0 + else: + valid_counts = poll_count == 0 and frame_count > 0 + if not valid_counts: + message = "wait counters do not match the selected strategy" + raise RuntimeError(message) + pane_id = context.delayed_pane_id + if not isinstance(pane_id, str) or not pane_id: + message = "wait lacks a concrete delayed pane ID" + raise RuntimeError(message) + return WaitResult( + strategy=strategy, + request_id=request.request_id, + token=request.token, + pane_id=pane_id, + configured_delay_ns=request.configured_delay_ns, + requested_monotonic_ns=request.requested_monotonic_ns, + scheduled_monotonic_ns=scheduled_ns, + emitted_monotonic_ns=emitted_ns, + detected_monotonic_ns=detected_monotonic_ns, + scheduling_lateness_ns=scheduling_lateness_ns, + detection_overhead_ns=detected_monotonic_ns - emitted_ns, + poll_count=poll_count, + frame_count=frame_count, + timeout_s=float(timeout_s), + timed_out=False, + dropped_notification_delta=dropped_notification_delta, + verified=True, + ) + + +def wait_capture_poll_sync( + context: RunContext, + *, + request_id: str, + value: str = "READY", + timeout_s: float = 3.0, + poll_interval_s: float = 0.01, +) -> WaitResult: + """Request and detect one exact sentinel through typed sync captures. + + Examples + -------- + >>> try: + ... wait_capture_poll_sync(types.SimpleNamespace(mode=ExecutionMode.ASYNC), + ... request_id="sample-1") + ... except ValueError as error: + ... print(error) + sync capture wait requires a synchronous run context + + Parameters + ---------- + context : RunContext + Active synchronous topology with one verified delayed pane. + request_id : str + Unique request identity. + value : str + Unique request value embedded in the exact token. + timeout_s : float + Overall monotonic timeout including configured fuzzer delay. + poll_interval_s : float + Maximum sleep between typed capture operations. + + Returns + ------- + WaitResult + Exact detection and matching fuzzer timing evidence. + + Raises + ------ + ValueError + If mode, timeout, cadence, or request data is invalid. + TimeoutError + If exact detection or matching evidence misses the deadline. + RuntimeError + If the fuzzer exits or matching evidence is invalid. + ~libtmux.experimental.ops.exc.TmuxCommandError + If a typed capture fails. + """ + from libtmux.experimental.ops import CapturePane, PaneId, run + + if context.mode is not ExecutionMode.SYNC: + message = "sync capture wait requires a synchronous run context" + raise ValueError(message) + if timeout_s <= 0 or poll_interval_s <= 0: + message = "capture wait timeout and cadence must be positive" + raise ValueError(message) + engine = t.cast("TmuxEngine", context.engine) + pane_id = context.delayed_pane_id + if not isinstance(pane_id, str) or not pane_id: + message = "capture wait requires a concrete delayed pane ID" + raise ValueError(message) + request = request_sentinel(context, request_id=request_id, value=value) + deadline_ns = request.requested_monotonic_ns + int(timeout_s * 1_000_000_000) + operation = CapturePane(target=PaneId(pane_id), start=-5000, join_wrapped=True) + needle = f"{request.token}\n".encode() + poll_count = 0 + detected_ns: int | None = None + while time.monotonic_ns() < deadline_ns: + if context.fuzzer.poll() is not None: + message = "fuzzer exited during capture wait" + raise RuntimeError(message) + result = run(operation, engine).raise_for_status() + poll_count += 1 + captured = ("\n".join(result.lines) + "\n").encode() + observed_ns = time.monotonic_ns() + if needle in captured and observed_ns <= deadline_ns: + detected_ns = observed_ns + break + remaining_ns = deadline_ns - time.monotonic_ns() + if remaining_ns > 0: + time.sleep(min(poll_interval_s, remaining_ns / 1_000_000_000)) + if detected_ns is None: + message = f"capture wait timed out for request {request.request_id}" + raise TimeoutError(message) + evidence = _read_sentinel_evidence(context, request) + while evidence is None and time.monotonic_ns() < deadline_ns: + if context.fuzzer.poll() is not None: + message = "fuzzer exited before sentinel evidence" + raise RuntimeError(message) + time.sleep(0.001) + evidence = _read_sentinel_evidence(context, request) + if evidence is None: + message = f"sentinel evidence timed out for request {request.request_id}" + raise TimeoutError(message) + return _completed_wait_result( + context, + request, + evidence, + strategy="capture-poll", + detected_monotonic_ns=detected_ns, + poll_count=poll_count, + frame_count=0, + timeout_s=timeout_s, + dropped_notification_delta=0, + ) + + +async def wait_capture_poll_async( + context: RunContext, + *, + request_id: str, + value: str = "READY", + timeout_s: float = 3.0, + poll_interval_s: float = 0.01, +) -> WaitResult: + """Request and detect one exact sentinel through typed async captures. + + Examples + -------- + >>> async def invalid_wait(): + ... try: + ... await wait_capture_poll_async( + ... types.SimpleNamespace(mode=ExecutionMode.SYNC), + ... request_id="sample-1", + ... ) + ... except ValueError as error: + ... return str(error) + >>> asyncio.run(invalid_wait()) + 'async capture wait requires an asynchronous run context' + + Parameters + ---------- + context : RunContext + Active asynchronous topology with one verified delayed pane. + request_id : str + Unique request identity. + value : str + Unique request value embedded in the exact token. + timeout_s : float + Overall monotonic timeout including configured fuzzer delay. + poll_interval_s : float + Maximum event-loop sleep between typed capture operations. + + Returns + ------- + WaitResult + Exact detection and matching fuzzer timing evidence. + + Raises + ------ + ValueError + If mode, timeout, cadence, or request data is invalid. + TimeoutError + If exact detection or matching evidence misses the deadline. + RuntimeError + If the fuzzer exits or matching evidence is invalid. + ~libtmux.experimental.ops.exc.TmuxCommandError + If a typed capture fails. + """ + from libtmux.experimental.ops import CapturePane, PaneId, arun + + if context.mode is not ExecutionMode.ASYNC: + message = "async capture wait requires an asynchronous run context" + raise ValueError(message) + if timeout_s <= 0 or poll_interval_s <= 0: + message = "capture wait timeout and cadence must be positive" + raise ValueError(message) + engine = t.cast("AsyncTmuxEngine", context.engine) + pane_id = context.delayed_pane_id + if not isinstance(pane_id, str) or not pane_id: + message = "capture wait requires a concrete delayed pane ID" + raise ValueError(message) + request = request_sentinel(context, request_id=request_id, value=value) + deadline_ns = request.requested_monotonic_ns + int(timeout_s * 1_000_000_000) + operation = CapturePane(target=PaneId(pane_id), start=-5000, join_wrapped=True) + needle = f"{request.token}\n".encode() + poll_count = 0 + detected_ns: int | None = None + while time.monotonic_ns() < deadline_ns: + if context.fuzzer.poll() is not None: + message = "fuzzer exited during capture wait" + raise RuntimeError(message) + result = (await arun(operation, engine)).raise_for_status() + poll_count += 1 + captured = ("\n".join(result.lines) + "\n").encode() + observed_ns = time.monotonic_ns() + if needle in captured and observed_ns <= deadline_ns: + detected_ns = observed_ns + break + remaining_ns = deadline_ns - time.monotonic_ns() + if remaining_ns > 0: + await asyncio.sleep(min(poll_interval_s, remaining_ns / 1_000_000_000)) + if detected_ns is None: + message = f"capture wait timed out for request {request.request_id}" + raise TimeoutError(message) + evidence = _read_sentinel_evidence(context, request) + while evidence is None and time.monotonic_ns() < deadline_ns: + if context.fuzzer.poll() is not None: + message = "fuzzer exited before sentinel evidence" + raise RuntimeError(message) + await asyncio.sleep(0.001) + evidence = _read_sentinel_evidence(context, request) + if evidence is None: + message = f"sentinel evidence timed out for request {request.request_id}" + raise TimeoutError(message) + return _completed_wait_result( + context, + request, + evidence, + strategy="capture-poll", + detected_monotonic_ns=detected_ns, + poll_count=poll_count, + frame_count=0, + timeout_s=timeout_s, + dropped_notification_delta=0, + ) + + +async def wait_control_stream( + context: RunContext, + *, + request_id: str, + value: str = "READY", + timeout_s: float = 3.0, +) -> WaitResult: + """Detect one exact sentinel in decoded async control output. + + The subscription is advanced before request publication. Only decoded + ``%output`` and ``%extended-output`` bytes for the concrete delayed pane + participate in matching. A suffix of at most ``len(token) - 1`` bytes + carries a possible partial match across notification boundaries. + + Examples + -------- + >>> async def invalid_wait(): + ... try: + ... await wait_control_stream( + ... types.SimpleNamespace( + ... mode=ExecutionMode.SYNC, lane=EngineLane.CONTROL + ... ), + ... request_id="sample-1", + ... ) + ... except ValueError as error: + ... return str(error) + >>> asyncio.run(invalid_wait()) + 'control stream wait requires an asynchronous context' + + Parameters + ---------- + context : RunContext + Active asynchronous control-mode topology. + request_id : str + Unique request identity. + value : str + Unique request value embedded in the exact token. + timeout_s : float + Overall monotonic timeout including configured fuzzer delay. + + Returns + ------- + WaitResult + Exact decoded detection and matching fuzzer timing evidence. + + Raises + ------ + ValueError + If mode, lane, timeout, or request data is invalid. + TypeError + If the context engine is not :class:`AsyncControlModeEngine`. + TimeoutError + If exact detection or matching evidence misses the deadline. + RuntimeError + If the stream ends, the fuzzer exits, evidence differs, or output drops. + """ + from libtmux.experimental.engines.async_control_mode import ( + AsyncControlModeEngine, + ) + + if context.mode is not ExecutionMode.ASYNC: + message = "control stream wait requires an asynchronous context" + raise ValueError(message) + if context.lane is not EngineLane.CONTROL: + message = "control stream wait requires the control engine lane" + raise ValueError(message) + if timeout_s <= 0: + message = "control stream wait timeout must be positive" + raise ValueError(message) + if not isinstance(context.engine, AsyncControlModeEngine): + message = "control stream wait requires AsyncControlModeEngine" + raise TypeError(message) + pane_id = context.delayed_pane_id + if not isinstance(pane_id, str) or not pane_id: + message = "control stream wait requires a concrete delayed pane ID" + raise ValueError(message) + engine = context.engine + dropped_before = engine.dropped_notifications + subscription = t.cast(t.AsyncGenerator[t.Any, None], engine.subscribe()) + next_notification: asyncio.Task[t.Any] | None = asyncio.create_task( + anext(subscription), + name=f"bench-control-wait-{request_id}", + ) + request: SentinelRequest | None = None + evidence: cabc.Mapping[str, t.Any] | None = None + detected_ns: int | None = None + frame_count = 0 + try: + await asyncio.sleep(0) + assert next_notification is not None + if next_notification.done() and not next_notification.cancelled(): + failure = next_notification.exception() + if failure is not None: + message = "control notification stream ended before request" + raise RuntimeError(message) from failure + request = request_sentinel(context, request_id=request_id, value=value) + deadline_ns = request.requested_monotonic_ns + int(timeout_s * 1_000_000_000) + needle = request.token.encode() + suffix = b"" + suffix_limit = max(0, len(needle) - 1) + while time.monotonic_ns() < deadline_ns: + if context.fuzzer.poll() is not None: + message = "fuzzer exited during control stream wait" + raise RuntimeError(message) + remaining_s = (deadline_ns - time.monotonic_ns()) / 1_000_000_000 + if remaining_s <= 0: + break + assert next_notification is not None + try: + notification = await asyncio.wait_for( + next_notification, + timeout=remaining_s, + ) + except asyncio.TimeoutError: + next_notification = None + break + except StopAsyncIteration as error: + next_notification = None + message = "control notification stream ended during wait" + raise RuntimeError(message) from error + next_notification = None + if ( + notification.kind in {"output", "extended-output"} + and notification.pane_id == pane_id + and notification.payload is not None + ): + frame_count += 1 + combined = suffix + notification.payload + observed_ns = time.monotonic_ns() + if needle in combined and observed_ns <= deadline_ns: + detected_ns = observed_ns + break + suffix = combined[-suffix_limit:] if suffix_limit else b"" + next_notification = asyncio.create_task( + anext(subscription), + name=f"bench-control-wait-{request_id}", + ) + if detected_ns is None: + message = f"control stream wait timed out for request {request.request_id}" + raise TimeoutError(message) + evidence = _read_sentinel_evidence(context, request) + while evidence is None and time.monotonic_ns() < deadline_ns: + if context.fuzzer.poll() is not None: + message = "fuzzer exited before sentinel evidence" + raise RuntimeError(message) + await asyncio.sleep(0.001) + evidence = _read_sentinel_evidence(context, request) + if evidence is None: + message = f"sentinel evidence timed out for request {request.request_id}" + raise TimeoutError(message) + finally: + if next_notification is not None: + if not next_notification.done(): + next_notification.cancel() + with contextlib.suppress(asyncio.CancelledError, Exception): + await next_notification + await subscription.aclose() + assert request is not None + assert evidence is not None + assert detected_ns is not None + dropped_delta = engine.dropped_notifications - dropped_before + return _completed_wait_result( + context, + request, + evidence, + strategy="control-stream", + detected_monotonic_ns=detected_ns, + poll_count=0, + frame_count=frame_count, + timeout_s=timeout_s, + dropped_notification_delta=dropped_delta, + ) + + def _mutation_targets( context: RunContext, snapshot: TopologySnapshot, @@ -5656,7 +6503,7 @@ def search_contents( PhaseMeasurement: t.TypeAlias = ( - MutationResult | EnumerationResult | CaptureResult | SearchResult + MutationResult | EnumerationResult | CaptureResult | SearchResult | WaitResult ) @@ -5676,7 +6523,7 @@ def _validated_phase_measurement(value: object) -> PhaseMeasurement: Returns ------- - MutationResult | EnumerationResult | CaptureResult | SearchResult + MutationResult | EnumerationResult | CaptureResult | SearchResult | WaitResult Valid typed measurement. Raises @@ -5691,6 +6538,7 @@ def _validated_phase_measurement(value: object) -> PhaseMeasurement: EnumerationResult, CaptureResult, SearchResult, + WaitResult, ) if not isinstance(value, accepted_types): message = "phase callable did not return a typed measurement" diff --git a/scripts/orchestration_fuzzer.py b/scripts/orchestration_fuzzer.py index e750ec5ec8..cf0b8bf736 100644 --- a/scripts/orchestration_fuzzer.py +++ b/scripts/orchestration_fuzzer.py @@ -11,6 +11,7 @@ import contextlib import dataclasses import enum +import hashlib import json import os import pathlib @@ -139,11 +140,13 @@ class SentinelEvidence: scheduled_monotonic_ns : int Requested emission deadline on the monotonic clock. emitted_monotonic_ns : int - Actual append time on the monotonic clock. + Timestamp immediately before the append on the monotonic clock. scheduling_lateness_ns : int Difference between the actual append time and the scheduled deadline. sentinel : str Canonical text appended to the delayed stream. + sentinel_sha256 : str + SHA-256 of the exact newline-terminated UTF-8 bytes appended. """ schema_version: int @@ -155,6 +158,7 @@ class SentinelEvidence: emitted_monotonic_ns: int scheduling_lateness_ns: int sentinel: str + sentinel_sha256: str def stream_for_pane(ordinal: int, delayed_ordinal: int) -> StreamMode: @@ -338,7 +342,7 @@ def _stream_path(paths: WorkloadPaths, mode: StreamMode) -> pathlib.Path: return paths.streams / f"{mode.value}.log" -def _append_text(path: pathlib.Path, text: str) -> int: +def _append_text(path: pathlib.Path, text: str, *, durable: bool = False) -> int: r"""Append one complete text record and return its UTF-8 byte count. Examples @@ -347,11 +351,27 @@ def _append_text(path: pathlib.Path, text: str) -> int: ... stream = pathlib.Path(temporary) / "stream.log" ... _append_text(stream, "pi\n"), stream.read_text(encoding="utf-8") (3, 'pi\n') + + Parameters + ---------- + path : pathlib.Path + Existing append-only stream. + text : str + Exact text to encode as UTF-8 and append. + durable : bool + Flush and fsync the append before returning when true. + + Returns + ------- + int + Number of exact encoded bytes appended. """ encoded = text.encode("utf-8") with path.open("ab") as stream: stream.write(encoded) stream.flush() + if durable: + os.fsync(stream.fileno()) return len(encoded) @@ -359,6 +379,8 @@ def _request_from_marker( marker: dict[str, t.Any], path: pathlib.Path, options: WorkloadOptions, + *, + seen_request_ids: t.Container[str] = (), ) -> tuple[str, int, str] | None: """Validate one request marker and return its delayed sentinel inputs. @@ -372,6 +394,7 @@ def _request_from_marker( ... {"request_id": "sample", "requested_monotonic_ns": 7}, ... pathlib.Path("sample.json"), ... options, + ... seen_request_ids=set(), ... ) ('sample', 7, 'READY') """ @@ -381,12 +404,21 @@ def _request_from_marker( if ( not isinstance(request_id, str) or not request_id - or path.stem != request_id + or len(request_id) > 128 + or any( + not (character.isascii() and (character.isalnum() or character in "-_")) + for character in request_id + ) + or path.name != f"{request_id}.json" + or request_id in seen_request_ids or isinstance(requested, bool) or not isinstance(requested, int) + or requested < 0 or isinstance(value, bool) or not isinstance(value, str) or not value + or "\n" in value + or "\r" in value ): return None return request_id, requested, value @@ -457,7 +489,7 @@ def request_stop(_signal_number: int, _frame: t.Any) -> None: activated_at_ns: int | None = None next_frame_ns: int | None = None epoch = 0 - seen_requests: set[str] = set() + seen_request_ids: set[str] = set() pending_request: tuple[str, int, int, int, str] | None = None def publish_heartbeat(state: str, now_ns: int, force: bool = False) -> None: @@ -513,15 +545,21 @@ def publish_heartbeat(state: str, now_ns: int, force: bool = False) -> None: if pending_request is None: for request_path in sorted(paths.requests.glob("*.json")): - if request_path.name in seen_requests: - continue marker = read_control_marker(request_path, options.run_id) if marker is None: continue - request = _request_from_marker(marker, request_path, options) + request = _request_from_marker( + marker, + request_path, + options, + seen_request_ids=seen_request_ids, + ) if request is None: continue request_id, requested_ns, value = request + if (paths.sentinels / f"{request_id}.json").exists(): + seen_request_ids.add(request_id) + continue pending_request = ( request_id, requested_ns, @@ -529,7 +567,7 @@ def publish_heartbeat(state: str, now_ns: int, force: bool = False) -> None: requested_ns + delay_ns, sentinel_text(options.run_id, request_id, value), ) - seen_requests.add(request_path.name) + seen_request_ids.add(request_id) break if pending_request is not None and now_ns >= pending_request[3]: @@ -540,10 +578,14 @@ def publish_heartbeat(state: str, now_ns: int, force: bool = False) -> None: scheduled_ns, sentinel, ) = pending_request + sentinel_record = f"{sentinel}\n" + sentinel_bytes = sentinel_record.encode() + emitted_ns = time.monotonic_ns() bytes_since_heartbeat += _append_text( - _stream_path(paths, StreamMode.DELAYED_MATCH), f"{sentinel}\n" + _stream_path(paths, StreamMode.DELAYED_MATCH), + sentinel_record, + durable=True, ) - emitted_ns = time.monotonic_ns() evidence = SentinelEvidence( schema_version=1, run_id=options.run_id, @@ -554,6 +596,7 @@ def publish_heartbeat(state: str, now_ns: int, force: bool = False) -> None: emitted_monotonic_ns=emitted_ns, scheduling_lateness_ns=emitted_ns - scheduled_ns, sentinel=sentinel, + sentinel_sha256=hashlib.sha256(sentinel_bytes).hexdigest(), ) write_json_atomic( paths.sentinels / f"{request_id}.json", dataclasses.asdict(evidence) diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index a9adc02240..1fe46cfac3 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -1992,6 +1992,483 @@ def test_capture_rejects_unattributed_typed_results( ) +def _wait_result_invariants( + benchmark_module: types.ModuleType, + context: t.Any, + result: t.Any, + *, + request_id: str, + value: str, + strategy: str, +) -> None: + """Assert independently recomputable wait timing and identity evidence.""" + assert isinstance(result, benchmark_module.WaitResult) + assert result.strategy == strategy + assert result.request_id == request_id + assert result.token == ( + f"LIBTMUX_SENTINEL run={context.run_id} request={request_id} value={value}" + ) + assert result.pane_id == context.delayed_pane_id + assert result.configured_delay_ns == 50_000_000 + assert result.scheduled_monotonic_ns == ( + result.requested_monotonic_ns + result.configured_delay_ns + ) + assert result.emitted_monotonic_ns >= result.scheduled_monotonic_ns + assert result.scheduling_lateness_ns == ( + result.emitted_monotonic_ns - result.scheduled_monotonic_ns + ) + assert result.scheduling_lateness_ns >= 0 + assert result.detected_monotonic_ns >= result.emitted_monotonic_ns + assert result.detection_overhead_ns == ( + result.detected_monotonic_ns - result.emitted_monotonic_ns + ) + assert result.detection_overhead_ns >= 0 + assert result.duration_ns == result.detection_overhead_ns + assert result.timeout_s == 2.0 + assert result.timed_out is False + assert result.dropped_notification_delta == 0 + assert result.verified + if strategy == "capture-poll": + assert result.poll_count > 0 + assert result.frame_count == 0 + else: + assert result.poll_count == 0 + assert result.frame_count > 0 + + +def _ordinary_delayed_frame_follows_wait(context: t.Any, result: t.Any) -> bool: + """Wait until the delayed pane shows ordinary scrolling after a sentinel.""" + deadline = time.monotonic() + 2.0 + while time.monotonic() < deadline: + captured = context.server.cmd( + "capture-pane", + "-t", + result.pane_id, + "-p", + "-J", + "-S", + "-5000", + ) + if captured.returncode != 0: + return False + lines = captured.stdout + try: + sentinel_index = lines.index(result.token) + except ValueError: + time.sleep(0.005) + continue + if any( + line.startswith("[delayed-match epoch=") + for line in lines[sentinel_index + 1 :] + ): + return True + time.sleep(0.005) + return False + + +def _assert_wait_tokens_only_reach_delayed_pane( + context: t.Any, + results: list[t.Any], +) -> None: + """Require every exact request token only in the selected delayed pane.""" + for pane_id in context.pane_ids: + captured = context.server.cmd( + "capture-pane", + "-t", + pane_id, + "-p", + "-J", + "-S", + "-5000", + ) + assert captured.returncode == 0, captured.stderr + matched = tuple( + result.request_id for result in results if result.token in captured.stdout + ) + if pane_id == context.delayed_pane_id: + assert matched == tuple(result.request_id for result in results) + else: + assert matched == () + + +def test_control_wait_matches_split_target_bytes_and_closes_subscription( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A real engine object isolates deterministic chunk and close semantics.""" + from libtmux.experimental.engines.async_control_mode import ( + AsyncControlModeEngine, + ControlNotification, + ) + + fuzzer_root = tmp_path / "fuzzer" + (fuzzer_root / "requests").mkdir(parents=True) + (fuzzer_root / "sentinels").mkdir() + engine = AsyncControlModeEngine() + events: list[str] = [] + context = types.SimpleNamespace( + mode=benchmark_module.ExecutionMode.ASYNC, + lane=benchmark_module.EngineLane.CONTROL, + engine=engine, + delayed_pane_id="%7", + run_id="split-run", + scratch=tmp_path, + sentinel_delay_ns=0, + fuzzer=types.SimpleNamespace(poll=lambda: None), + ) + + async def notifications() -> t.AsyncIterator[ControlNotification]: + request_path = fuzzer_root / "requests" / "split-request.json" + events.append("subscribed") + assert not request_path.exists() + try: + while not request_path.exists(): + await asyncio.sleep(0) + events.append("requested") + marker = json.loads(request_path.read_text(encoding="utf-8")) + token = "LIBTMUX_SENTINEL run=split-run request=split-request value=RIGHT" + requested_ns = marker["requested_monotonic_ns"] + emitted_ns = time.monotonic_ns() + benchmark_module.write_json_atomic( + fuzzer_root / "sentinels" / "split-request.json", + { + "schema_version": 1, + "run_id": "split-run", + "request_id": "split-request", + "requested_monotonic_ns": requested_ns, + "configured_delay_ns": 0, + "scheduled_monotonic_ns": requested_ns, + "emitted_monotonic_ns": emitted_ns, + "scheduling_lateness_ns": emitted_ns - requested_ns, + "sentinel": token, + "sentinel_sha256": hashlib.sha256( + f"{token}\n".encode() + ).hexdigest(), + }, + ) + yield ControlNotification( + "output", (), "", pane_id="%8", payload=token.encode() + ) + yield ControlNotification( + "output", + (), + "", + pane_id="%7", + payload=b"LIBTMUX_SENTINEL run=split-run request=stale value=WRONG", + ) + split = len(token) // 2 + yield ControlNotification( + "extended-output", + (), + "", + pane_id="%7", + payload=token.encode()[:split], + ) + yield ControlNotification( + "output", + (), + "", + pane_id="%7", + payload=token.encode()[split:], + ) + finally: + events.append("closed") + + monkeypatch.setattr(engine, "subscribe", notifications) + + result = asyncio.run( + benchmark_module.wait_control_stream( + context, + request_id="split-request", + value="RIGHT", + timeout_s=1.0, + ) + ) + + assert result.request_id == "split-request" + assert result.frame_count == 3 + assert result.dropped_notification_delta == 0 + assert events == ["subscribed", "requested", "closed"] + + +def test_control_wait_rejects_drop_delta_and_still_closes_subscription( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A deterministic overflow double proves drops invalidate and close a wait.""" + from libtmux.experimental.engines.async_control_mode import ( + AsyncControlModeEngine, + ControlNotification, + ) + + fuzzer_root = tmp_path / "fuzzer" + (fuzzer_root / "requests").mkdir(parents=True) + (fuzzer_root / "sentinels").mkdir() + engine = AsyncControlModeEngine() + closed = False + context = types.SimpleNamespace( + mode=benchmark_module.ExecutionMode.ASYNC, + lane=benchmark_module.EngineLane.CONTROL, + engine=engine, + delayed_pane_id="%7", + run_id="drop-run", + scratch=tmp_path, + sentinel_delay_ns=0, + fuzzer=types.SimpleNamespace(poll=lambda: None), + ) + + async def notifications() -> t.AsyncIterator[ControlNotification]: + nonlocal closed + request_path = fuzzer_root / "requests" / "drop-request.json" + try: + while not request_path.exists(): + await asyncio.sleep(0) + marker = json.loads(request_path.read_text(encoding="utf-8")) + token = "LIBTMUX_SENTINEL run=drop-run request=drop-request value=RIGHT" + requested_ns = marker["requested_monotonic_ns"] + emitted_ns = time.monotonic_ns() + benchmark_module.write_json_atomic( + fuzzer_root / "sentinels" / "drop-request.json", + { + "schema_version": 1, + "run_id": "drop-run", + "request_id": "drop-request", + "requested_monotonic_ns": requested_ns, + "configured_delay_ns": 0, + "scheduled_monotonic_ns": requested_ns, + "emitted_monotonic_ns": emitted_ns, + "scheduling_lateness_ns": emitted_ns - requested_ns, + "sentinel": token, + "sentinel_sha256": hashlib.sha256( + f"{token}\n".encode() + ).hexdigest(), + }, + ) + engine._dropped_notifications += 1 + yield ControlNotification( + "output", (), "", pane_id="%7", payload=token.encode() + ) + finally: + closed = True + + monkeypatch.setattr(engine, "subscribe", notifications) + + with pytest.raises(RuntimeError, match="dropped 1 notification"): + asyncio.run( + benchmark_module.wait_control_stream( + context, + request_id="drop-request", + value="RIGHT", + timeout_s=1.0, + ) + ) + + assert closed + + +@pytest.mark.parametrize( + ("field", "wrong_value"), + ( + ("run_id", "stale-run"), + ("request_id", "stale-request"), + ("requested_monotonic_ns", 8), + ("sentinel", "LIBTMUX_SENTINEL run=run-7 request=old value=WRONG"), + ("sentinel_sha256", "0" * 64), + ), +) +def test_wait_evidence_rejects_wrong_or_stale_request_identity( + benchmark_module: types.ModuleType, + field: str, + wrong_value: object, +) -> None: + """Evidence from another request or token cannot validate a detection.""" + token = "LIBTMUX_SENTINEL run=run-7 request=request-1 value=RIGHT" + request = benchmark_module.SentinelRequest( + request_id="request-1", + token=token, + requested_monotonic_ns=7, + configured_delay_ns=5, + ) + marker = { + "schema_version": 1, + "run_id": "run-7", + "request_id": "request-1", + "requested_monotonic_ns": 7, + "configured_delay_ns": 5, + "scheduled_monotonic_ns": 12, + "emitted_monotonic_ns": 14, + "scheduling_lateness_ns": 2, + "sentinel": token, + "sentinel_sha256": hashlib.sha256(f"{token}\n".encode()).hexdigest(), + } + marker[field] = wrong_value + + with pytest.raises(RuntimeError, match="does not match request"): + benchmark_module._validated_sentinel_evidence( + types.SimpleNamespace(run_id="run-7"), request, marker + ) + + +def test_repeated_wait_capture_poll_sync_uses_one_active_topology( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """Five sync polls must match fresh requests without restarting resources.""" + context = benchmark_module.setup_sync( + benchmark_module.Topology(1, 2, 2), + benchmark_module.EngineLane.SUBPROCESS, + tmp_path / "repeated-wait-sync", + run_id="wait-sync", + delayed_ordinal=2, + ) + cleanup = None + observed: list[t.Any] = [] + identities_before = context.processes + request_ids = iter(f"sync-{ordinal}" for ordinal in range(5)) + + def measured() -> t.Any: + request_id = next(request_ids) + value = f"TOKEN-{request_id}" + result = benchmark_module.wait_capture_poll_sync( + context, + request_id=request_id, + value=value, + timeout_s=2.0, + poll_interval_s=0.005, + ) + _wait_result_invariants( + benchmark_module, + context, + result, + request_id=request_id, + value=value, + strategy="capture-poll", + ) + observed.append(result) + return result + + try: + benchmark_module.release_activity_gate(context) + benchmark_module.verify_activity_sync(context) + repeated = asyncio.run( + benchmark_module.run_repeatable_phase( + {"capture-poll": measured}, + warmup=2, + runs=3, + seed=11, + live_postcondition=lambda result: _ordinary_delayed_frame_follows_wait( + context, result + ), + snapshot_resources=lambda: benchmark_module.HostSnapshot(), + ) + ) + _assert_wait_tokens_only_reach_delayed_pane(context, observed) + assert repeated.failure is None + assert len(repeated.samples) == 3 + assert len(observed) == 5 + assert len({result.request_id for result in observed}) == 5 + assert len({result.token for result in observed}) == 5 + assert context.fuzzer.pid == identities_before[0].pid + assert context.processes == identities_before + assert len(tuple((context.scratch / "fuzzer" / "requests").glob("*.json"))) == 5 + finally: + cleanup = asyncio.run(benchmark_module.cleanup_run(context)) + assert cleanup.complete, cleanup.errors + + +@pytest.mark.parametrize( + ("lane_name", "strategy"), + (("subprocess", "capture-poll"), ("control", "control-stream")), +) +def test_repeated_wait_async_strategies_use_one_active_topology( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + lane_name: str, + strategy: str, +) -> None: + """Async polling and streaming each keep one active topology for five waits.""" + context = cleanup = repeated = None + observed: list[t.Any] = [] + + async def exercise() -> None: + """Keep the async engine, waits, and cleanup on one event loop.""" + nonlocal context, cleanup, repeated + context = await benchmark_module.setup_async( + benchmark_module.Topology(1, 2, 2), + benchmark_module.EngineLane(lane_name), + tmp_path / f"repeated-wait-{lane_name}", + run_id=f"wait-{lane_name}", + delayed_ordinal=2, + ) + identities_before = context.processes + request_ids = iter(f"{strategy}-{ordinal}" for ordinal in range(5)) + + async def measured() -> t.Any: + request_id = next(request_ids) + value = f"TOKEN-{request_id}" + if strategy == "capture-poll": + result = await benchmark_module.wait_capture_poll_async( + context, + request_id=request_id, + value=value, + timeout_s=2.0, + poll_interval_s=0.005, + ) + else: + result = await benchmark_module.wait_control_stream( + context, + request_id=request_id, + value=value, + timeout_s=2.0, + ) + _wait_result_invariants( + benchmark_module, + context, + result, + request_id=request_id, + value=value, + strategy=strategy, + ) + observed.append(result) + return result + + try: + benchmark_module.release_activity_gate(context) + await benchmark_module.verify_activity_async(context) + repeated = await benchmark_module.run_repeatable_phase( + {strategy: measured}, + warmup=2, + runs=3, + seed=11, + live_postcondition=lambda result: _ordinary_delayed_frame_follows_wait( + context, result + ), + snapshot_resources=lambda: benchmark_module.HostSnapshot(), + ) + _assert_wait_tokens_only_reach_delayed_pane(context, observed) + assert context.fuzzer.pid == identities_before[0].pid + assert context.processes == identities_before + assert ( + len(tuple((context.scratch / "fuzzer" / "requests").glob("*.json"))) + == 5 + ) + finally: + cleanup = await benchmark_module.cleanup_run(context) + + asyncio.run(exercise()) + + assert repeated is not None + assert repeated.failure is None + assert len(repeated.samples) == 3 + assert len(observed) == 5 + assert len({result.request_id for result in observed}) == 5 + assert len({result.token for result in observed}) == 5 + assert cleanup is not None + assert cleanup.complete, cleanup.errors + + def _request_content_sentinel( benchmark_module: types.ModuleType, context: t.Any, diff --git a/tests/test_orchestration_fuzzer.py b/tests/test_orchestration_fuzzer.py index 3955fc0f47..e789b5b44a 100644 --- a/tests/test_orchestration_fuzzer.py +++ b/tests/test_orchestration_fuzzer.py @@ -2,6 +2,7 @@ from __future__ import annotations +import hashlib import importlib.util import json import os @@ -100,6 +101,100 @@ def test_render_frame_uses_only_its_mode_epoch_and_seeded_corpus( assert installer.text == "[installer epoch=4] install phase=install unit=5/8\n" +@pytest.mark.parametrize( + ("filename", "request_id"), + ( + ("nested.json.json", "nested.json"), + (".json", ".json"), + ("space id.json", "space id"), + ("unicode-π.json", "unicode-π"), + ), +) +def test_request_marker_rejects_noncanonical_filename_payload_id( + fuzzer_module: types.ModuleType, + filename: str, + request_id: str, +) -> None: + """Ambiguous request filenames must not become marker or evidence paths.""" + options = fuzzer_module.WorkloadOptions( + pathlib.Path("out"), + "run-7", + pathlib.Path(), + 0, + 1.0, + 1.0, + 0.0, + "READY", + 1.0, + ) + + assert ( + fuzzer_module._request_from_marker( + { + "request_id": request_id, + "requested_monotonic_ns": 7, + "value": "READY", + }, + pathlib.Path(filename), + options, + seen_request_ids=set(), + ) + is None + ) + + +def test_request_marker_rejects_duplicate_payload_id( + fuzzer_module: types.ModuleType, +) -> None: + """A previously accepted request ID cannot be scheduled a second time.""" + options = fuzzer_module.WorkloadOptions( + pathlib.Path("out"), + "run-7", + pathlib.Path(), + 0, + 1.0, + 1.0, + 0.0, + "READY", + 1.0, + ) + + assert ( + fuzzer_module._request_from_marker( + { + "request_id": "sample-1", + "requested_monotonic_ns": 7, + "value": "READY", + }, + pathlib.Path("sample-1.json"), + options, + seen_request_ids={"sample-1"}, + ) + is None + ) + + +def test_durable_stream_append_flushes_and_fsyncs_exact_bytes( + fuzzer_module: types.ModuleType, + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Evidence publication must follow a durable exact sentinel append.""" + stream = tmp_path / "delayed-match.log" + stream.touch() + observed: list[bytes] = [] + + def record_fsync(_file_descriptor: int) -> None: + observed.append(stream.read_bytes()) + + monkeypatch.setattr(fuzzer_module.os, "fsync", record_fsync) + + count = fuzzer_module._append_text(stream, "sentinel\n", durable=True) + + assert count == 9 + assert observed == [b"sentinel\n"] + + def write_marker(path: pathlib.Path, data: dict[str, t.Any]) -> None: """Publish one complete marker without exposing a partial JSON document.""" temporary = path.with_name(f".{path.name}.tmp") @@ -197,6 +292,10 @@ def test_serve_evidence_preserves_request_delay_and_lateness( assert evidence["scheduling_lateness_ns"] == ( evidence["emitted_monotonic_ns"] - evidence["scheduled_monotonic_ns"] ) + assert ( + evidence["sentinel_sha256"] + == hashlib.sha256(f"{evidence['sentinel']}\n".encode()).hexdigest() + ) finally: write_marker(output_dir / "stop.json", {"schema_version": 1, "run_id": "run-7"}) finish_process(process) @@ -346,6 +445,10 @@ def test_serve_pauses_until_a_matching_gate_and_handles_repeated_requests( assert evidence["sentinel"] == ( f"LIBTMUX_SENTINEL run=run-7 request={request_id} value=READY" ) + assert ( + evidence["sentinel_sha256"] + == hashlib.sha256(f"{evidence['sentinel']}\n".encode()).hexdigest() + ) delayed_stream = (streams / "delayed-match.log").read_text(encoding="utf-8") assert delayed_stream.count(evidence["sentinel"]) == 1 @@ -365,3 +468,58 @@ def ordinary_frame_follows(sentinel: str = evidence["sentinel"]) -> bool: assert heartbeat["state"] == "stopped" finally: finish_process(process) + + +def test_serve_processes_sorted_requests_sequentially_and_rejects_duplicate( + tmp_path: pathlib.Path, +) -> None: + """Sorted unseen IDs emit once each even when requests arrive together.""" + output_dir = tmp_path / "ordered-output" + process = start_serve(output_dir) + + try: + wait_for(lambda: (output_dir / "ready.json").exists()) + write_marker(output_dir / "gate.json", {"schema_version": 1, "run_id": "run-7"}) + requested = time.monotonic_ns() + for request_id in ("z-last", "a-first"): + write_marker( + output_dir / "requests" / f"{request_id}.json", + { + "schema_version": 1, + "run_id": "run-7", + "request_id": request_id, + "requested_monotonic_ns": requested, + "value": request_id, + }, + ) + first_path = output_dir / "sentinels" / "a-first.json" + last_path = output_dir / "sentinels" / "z-last.json" + wait_for(lambda: first_path.exists() and last_path.exists()) + first = read_json(first_path) + last = read_json(last_path) + + assert first["emitted_monotonic_ns"] <= last["emitted_monotonic_ns"] + delayed = output_dir / "streams" / "delayed-match.log" + original = delayed.read_text(encoding="utf-8") + assert original.count(first["sentinel"]) == 1 + assert original.count(last["sentinel"]) == 1 + + write_marker( + output_dir / "requests" / "a-first.json", + { + "schema_version": 1, + "run_id": "run-7", + "request_id": "a-first", + "requested_monotonic_ns": time.monotonic_ns(), + "value": "DUPLICATE", + }, + ) + time.sleep(0.1) + + after_duplicate = delayed.read_text(encoding="utf-8") + assert after_duplicate.count(first["sentinel"]) == 1 + assert "value=DUPLICATE" not in after_duplicate + assert read_json(first_path) == first + finally: + write_marker(output_dir / "stop.json", {"schema_version": 1, "run_id": "run-7"}) + finish_process(process) From ec491b328629882487adfc66c79fbd7eec17dae2 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 10:35:47 -0500 Subject: [PATCH 18/67] Bench(fix[wait]): Bound delayed capture why: A blocked typed capture could outlive its request deadline, and unbounded terminal identity could evade exact capture matching. what: - Bound sync subprocess/control and async capture work to the request deadline without leaving owned work behind - Restrict sentinel components, record size, producer rate, and capture history to one documented terminal-safe contract - Cover timeout cleanup, pending cancellation, and maximum-token round trips on the repeated active topology --- scripts/bench_orchestration.py | 278 +++++++++++--- scripts/orchestration_fuzzer.py | 88 ++++- tests/test_bench_orchestration_script.py | 460 +++++++++++++++++++++++ tests/test_orchestration_fuzzer.py | 122 +++++- 4 files changed, 893 insertions(+), 55 deletions(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index 47099c135b..6e633cf8a0 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -55,6 +55,113 @@ _SENTINEL_DELAY_S = 0.05 _SENTINEL_DELAY_NS = 50_000_000 +_TERMINAL_SAFE_COMPONENT_ALPHABET = frozenset( + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789._:-" +) +_SENTINEL_COMPONENT_MAX_BYTES = 128 +_SENTINEL_RECORD_MAX_BYTES = 422 +_WAIT_TIMEOUT_MAX_S = 3.0 +_WAIT_FRAME_RATE_MAX_HZ = 40.0 +# At the maximum supported producer rate and wait duration, 5,000 joined +# history rows retain 120 ordinary delayed frames plus the 422-byte sentinel +# with ample wrapping headroom in the required active 1x2x2 topology. +_WAIT_CAPTURE_HISTORY_LINES = 5_000 + + +def _is_terminal_safe_component(value: object) -> bool: + """Return whether *value* is one bounded printable ASCII component. + + Components contain 1-128 encoded bytes drawn only from letters, digits, + ``.``, ``_``, ``:``, and ``-``. + + >>> _is_terminal_safe_component("run.uuid:sample_1-2") + True + >>> _is_terminal_safe_component("unsafe value") + False + """ + return ( + isinstance(value, str) + and value.isascii() + and 0 < len(value.encode()) <= _SENTINEL_COMPONENT_MAX_BYTES + and all(character in _TERMINAL_SAFE_COMPONENT_ALPHABET for character in value) + ) + + +def _sentinel_token(run_id: str, request_id: str, value: str) -> str: + """Build one terminal-safe sentinel within the capture-history contract. + + >>> _sentinel_token("run-7", "sample-1", "READY") + 'LIBTMUX_SENTINEL run=run-7 request=sample-1 value=READY' + + Raises + ------ + ValueError + If a component is not terminal-safe or the complete record is too long. + """ + components = {"run_id": run_id, "request_id": request_id, "value": value} + for name, component in components.items(): + if not _is_terminal_safe_component(component): + message = ( + f"{name} must be a 1-{_SENTINEL_COMPONENT_MAX_BYTES} byte " + "terminal-safe ASCII component using letters, digits, '.', '_', " + "':', or '-'" + ) + raise ValueError(message) + token = f"LIBTMUX_SENTINEL run={run_id} request={request_id} value={value}" + if len(f"{token}\n".encode()) > _SENTINEL_RECORD_MAX_BYTES: + message = ( + f"sentinel record must be at most {_SENTINEL_RECORD_MAX_BYTES} " + "encoded bytes" + ) + raise ValueError(message) + return token + + +def _validated_wait_timeout(timeout_s: float) -> float: + """Return one finite wait duration covered by retained capture history. + + >>> _validated_wait_timeout(3.0) + 3.0 + + Raises + ------ + ValueError + If the timeout is not finite, positive, or at most three seconds. + """ + if ( + isinstance(timeout_s, bool) + or not isinstance(timeout_s, (int, float)) + or not math.isfinite(timeout_s) + or timeout_s <= 0 + or timeout_s > _WAIT_TIMEOUT_MAX_S + ): + message = ( + f"wait timeout must be positive and at most {_WAIT_TIMEOUT_MAX_S} seconds" + ) + raise ValueError(message) + return float(timeout_s) + + +def _validated_poll_interval(poll_interval_s: float) -> float: + """Return one finite positive capture-poll cadence. + + >>> _validated_poll_interval(0.01) + 0.01 + + Raises + ------ + ValueError + If the cadence is not finite and positive. + """ + if ( + isinstance(poll_interval_s, bool) + or not isinstance(poll_interval_s, (int, float)) + or not math.isfinite(poll_interval_s) + or poll_interval_s <= 0 + ): + message = "capture wait cadence must be finite and positive" + raise ValueError(message) + return float(poll_interval_s) class EngineLane(str, enum.Enum): @@ -2754,8 +2861,8 @@ def start_fuzzer( >>> try: ... start_fuzzer(pathlib.Path("."), "", ready_timeout_s=1.0) ... except ValueError as error: - ... print(error) - run_id must be nonempty + ... str(error).startswith("run_id must be a 1-128 byte terminal-safe") + True Parameters ---------- @@ -2787,12 +2894,26 @@ def start_fuzzer( SetupCleanupError If a startup failure is followed by incomplete child cleanup. """ - if not run_id: - message = "run_id must be nonempty" + if not _is_terminal_safe_component(run_id): + message = ( + f"run_id must be a 1-{_SENTINEL_COMPONENT_MAX_BYTES} byte " + "terminal-safe ASCII component using letters, digits, '.', '_', ':', " + "or '-'" + ) raise ValueError(message) if ready_timeout_s <= 0: message = "ready timeout must be positive" raise ValueError(message) + if ( + not math.isfinite(frame_rate_hz) + or frame_rate_hz <= 0 + or frame_rate_hz > _WAIT_FRAME_RATE_MAX_HZ + ): + message = ( + f"fuzzer frame rate must be positive and at most " + f"{_WAIT_FRAME_RATE_MAX_HZ} frames per second" + ) + raise ValueError(message) output_dir = scratch / "fuzzer" script = pathlib.Path(__file__).with_name("orchestration_fuzzer.py") environment = os.environ.copy() @@ -2933,12 +3054,11 @@ def _prepare_context( if not 0 <= delayed_ordinal < topology.panes: message = "delayed pane ordinal must identify a requested pane" raise ValueError(message) - if not run_id or any( - not (character.isascii() and (character.isalnum() or character in "-_")) - for character in run_id - ): + if not _is_terminal_safe_component(run_id): message = ( - "run_id must contain only ASCII letters, digits, hyphens, or underscores" + f"run_id must be a 1-{_SENTINEL_COMPONENT_MAX_BYTES} byte " + "terminal-safe ASCII component using letters, digits, '.', '_', ':', " + "or '-'" ) raise ValueError(message) resolved_scratch = scratch.resolve() @@ -4221,10 +4341,9 @@ def request_sentinel( context : RunContext Active topology whose fuzzer owns the request directory. request_id : str - Unique ASCII identifier containing only letters, digits, hyphens, or - underscores. + Unique 1-128 byte terminal-safe ASCII component. value : str - Nonempty single-line value embedded in the canonical sentinel. + Terminal-safe 1-128 byte value embedded in the canonical sentinel. Returns ------- @@ -4242,22 +4361,7 @@ def request_sentinel( OSError If atomic marker publication fails. """ - if ( - not request_id - or len(request_id) > 128 - or any( - not (character.isascii() and (character.isalnum() or character in "-_")) - for character in request_id - ) - ): - message = ( - "request_id must contain 1-128 ASCII letters, digits, hyphens, " - "or underscores" - ) - raise ValueError(message) - if not isinstance(value, str) or not value or "\n" in value or "\r" in value: - message = "sentinel value must be a nonempty single line" - raise ValueError(message) + token = _sentinel_token(context.run_id, request_id, value) delay_ns = context.sentinel_delay_ns if type(delay_ns) is not int or delay_ns < 0: message = "sentinel delay must be a nonnegative integer nanosecond value" @@ -4275,7 +4379,6 @@ def request_sentinel( if request_path.exists() or evidence_path.exists(): message = f"sentinel request already exists: {request_id}" raise FileExistsError(message) - token = f"LIBTMUX_SENTINEL run={context.run_id} request={request_id} value={value}" requested_ns = time.monotonic_ns() write_json_atomic( request_path, @@ -4572,22 +4675,36 @@ def wait_capture_poll_sync( ~libtmux.experimental.ops.exc.TmuxCommandError If a typed capture fails. """ - from libtmux.experimental.ops import CapturePane, PaneId, run + from libtmux import exc + from libtmux.experimental.engines.base import CommandRequest, encode_direct_argv + from libtmux.experimental.engines.control_mode import ( + ControlModeEngine, + ControlModeError, + ) + from libtmux.experimental.engines.subprocess import SubprocessEngine + from libtmux.experimental.ops import CapturePane, PaneId if context.mode is not ExecutionMode.SYNC: message = "sync capture wait requires a synchronous run context" raise ValueError(message) - if timeout_s <= 0 or poll_interval_s <= 0: - message = "capture wait timeout and cadence must be positive" - raise ValueError(message) - engine = t.cast("TmuxEngine", context.engine) + timeout_s = _validated_wait_timeout(timeout_s) + poll_interval_s = _validated_poll_interval(poll_interval_s) + engine = context.engine + if not isinstance(engine, (SubprocessEngine, ControlModeEngine)): + message = "sync capture wait requires a subprocess or control engine" + raise TypeError(message) pane_id = context.delayed_pane_id if not isinstance(pane_id, str) or not pane_id: message = "capture wait requires a concrete delayed pane ID" raise ValueError(message) request = request_sentinel(context, request_id=request_id, value=value) deadline_ns = request.requested_monotonic_ns + int(timeout_s * 1_000_000_000) - operation = CapturePane(target=PaneId(pane_id), start=-5000, join_wrapped=True) + operation = CapturePane( + target=PaneId(pane_id), + start=-_WAIT_CAPTURE_HISTORY_LINES, + join_wrapped=True, + ) + rendered = operation.render() needle = f"{request.token}\n".encode() poll_count = 0 detected_ns: int | None = None @@ -4595,7 +4712,62 @@ def wait_capture_poll_sync( if context.fuzzer.poll() is not None: message = "fuzzer exited during capture wait" raise RuntimeError(message) - result = run(operation, engine).raise_for_status() + remaining_s = (deadline_ns - time.monotonic_ns()) / 1_000_000_000 + if remaining_s <= 0: + break + if isinstance(engine, SubprocessEngine): + command = engine.connection.argv(*encode_direct_argv(rendered)) + try: + process = subprocess.Popen( + command, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + encoding="utf-8", + errors="backslashreplace", + start_new_session=True, + ) + except FileNotFoundError: + raise exc.TmuxCommandNotFound from None + try: + stdout, stderr = process.communicate(timeout=remaining_s) + except subprocess.TimeoutExpired as error: + with contextlib.suppress(OSError): + os.killpg(process.pid, signal.SIGKILL) + stdout, stderr = process.communicate() + del stdout, stderr + message = f"capture wait timed out for request {request.request_id}" + raise TimeoutError(message) from error + stdout_lines = stdout.split("\n") + while stdout_lines and stdout_lines[-1] == "": + stdout_lines.pop() + stderr_lines = tuple(line for line in stderr.split("\n") if line) + result = operation.build_result( + argv=rendered, + returncode=( + process.returncode if process.returncode is not None else -1 + ), + stdout=tuple(stdout_lines), + stderr=stderr_lines, + ).raise_for_status() + else: + original_timeout = engine.timeout + engine.timeout = min(original_timeout, remaining_s) + try: + raw = engine.run(CommandRequest(args=rendered)) + except ControlModeError as error: + if time.monotonic_ns() >= deadline_ns: + message = f"capture wait timed out for request {request.request_id}" + raise TimeoutError(message) from error + raise + finally: + engine.timeout = original_timeout + result = operation.build_result( + argv=rendered, + returncode=raw.returncode, + stdout=raw.stdout, + stderr=raw.stderr, + ).raise_for_status() poll_count += 1 captured = ("\n".join(result.lines) + "\n").encode() observed_ns = time.monotonic_ns() @@ -4688,9 +4860,8 @@ async def wait_capture_poll_async( if context.mode is not ExecutionMode.ASYNC: message = "async capture wait requires an asynchronous run context" raise ValueError(message) - if timeout_s <= 0 or poll_interval_s <= 0: - message = "capture wait timeout and cadence must be positive" - raise ValueError(message) + timeout_s = _validated_wait_timeout(timeout_s) + poll_interval_s = _validated_poll_interval(poll_interval_s) engine = t.cast("AsyncTmuxEngine", context.engine) pane_id = context.delayed_pane_id if not isinstance(pane_id, str) or not pane_id: @@ -4698,7 +4869,11 @@ async def wait_capture_poll_async( raise ValueError(message) request = request_sentinel(context, request_id=request_id, value=value) deadline_ns = request.requested_monotonic_ns + int(timeout_s * 1_000_000_000) - operation = CapturePane(target=PaneId(pane_id), start=-5000, join_wrapped=True) + operation = CapturePane( + target=PaneId(pane_id), + start=-_WAIT_CAPTURE_HISTORY_LINES, + join_wrapped=True, + ) needle = f"{request.token}\n".encode() poll_count = 0 detected_ns: int | None = None @@ -4706,7 +4881,24 @@ async def wait_capture_poll_async( if context.fuzzer.poll() is not None: message = "fuzzer exited during capture wait" raise RuntimeError(message) - result = (await arun(operation, engine)).raise_for_status() + remaining_s = (deadline_ns - time.monotonic_ns()) / 1_000_000_000 + if remaining_s <= 0: + break + capture_task = asyncio.create_task( + arun(operation, engine), + name=f"bench-capture-wait-{request.request_id}", + ) + try: + result = ( + await asyncio.wait_for(capture_task, timeout=remaining_s) + ).raise_for_status() + except asyncio.TimeoutError as error: + message = f"capture wait timed out for request {request.request_id}" + raise TimeoutError(message) from error + finally: + if not capture_task.done(): + capture_task.cancel() + await asyncio.gather(capture_task, return_exceptions=True) poll_count += 1 captured = ("\n".join(result.lines) + "\n").encode() observed_ns = time.monotonic_ns() @@ -4808,9 +5000,7 @@ async def wait_control_stream( if context.lane is not EngineLane.CONTROL: message = "control stream wait requires the control engine lane" raise ValueError(message) - if timeout_s <= 0: - message = "control stream wait timeout must be positive" - raise ValueError(message) + timeout_s = _validated_wait_timeout(timeout_s) if not isinstance(context.engine, AsyncControlModeEngine): message = "control stream wait requires AsyncControlModeEngine" raise TypeError(message) diff --git a/scripts/orchestration_fuzzer.py b/scripts/orchestration_fuzzer.py index cf0b8bf736..5126d44438 100644 --- a/scripts/orchestration_fuzzer.py +++ b/scripts/orchestration_fuzzer.py @@ -21,6 +21,14 @@ import time import typing as t +# Sentinel components are printable ASCII terminal atoms. Spaces and every +# control byte are excluded so a captured token remains one literal field. +_TERMINAL_SAFE_COMPONENT_ALPHABET = frozenset( + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789._:-" +) +SENTINEL_COMPONENT_MAX_BYTES = 128 +SENTINEL_RECORD_MAX_BYTES = 422 + class StreamMode(str, enum.Enum): """A deterministic active-output stream category.""" @@ -176,13 +184,72 @@ def stream_for_pane(ordinal: int, delayed_ordinal: int) -> StreamMode: return shared[compacted % len(shared)] +def _is_terminal_safe_component(value: object) -> bool: + """Return whether *value* is one bounded printable ASCII component. + + Components contain 1-128 encoded bytes drawn only from letters, digits, + ``.``, ``_``, ``:``, and ``-``. The bound keeps a complete sentinel within + :data:`SENTINEL_RECORD_MAX_BYTES`. + + >>> _is_terminal_safe_component("run.uuid:sample_1-2") + True + >>> _is_terminal_safe_component("unsafe value") + False + """ + return ( + isinstance(value, str) + and value.isascii() + and 0 < len(value.encode()) <= SENTINEL_COMPONENT_MAX_BYTES + and all(character in _TERMINAL_SAFE_COMPONENT_ALPHABET for character in value) + ) + + def sentinel_text(run_id: str, request_id: str, value: str) -> str: - """Return a sentinel that identifies its owning run and request. + """Return a bounded terminal-safe sentinel for one run and request. >>> sentinel_text("run-7", "sample-03", "READY") 'LIBTMUX_SENTINEL run=run-7 request=sample-03 value=READY' + + Raises + ------ + ValueError + If any component is not a 1-128 byte terminal-safe ASCII atom or the + complete newline-terminated record exceeds 422 encoded bytes. """ - return f"LIBTMUX_SENTINEL run={run_id} request={request_id} value={value}" + components = {"run_id": run_id, "request_id": request_id, "value": value} + for name, component in components.items(): + if not _is_terminal_safe_component(component): + message = ( + f"{name} must be a 1-{SENTINEL_COMPONENT_MAX_BYTES} byte " + "terminal-safe ASCII component using letters, digits, '.', '_', " + "':', or '-'" + ) + raise ValueError(message) + sentinel = f"LIBTMUX_SENTINEL run={run_id} request={request_id} value={value}" + if len(f"{sentinel}\n".encode()) > SENTINEL_RECORD_MAX_BYTES: + message = ( + f"sentinel record must be at most {SENTINEL_RECORD_MAX_BYTES} encoded bytes" + ) + raise ValueError(message) + return sentinel + + +def _validate_workload_identity(options: WorkloadOptions) -> None: + """Reject unsafe service identity before creating output or rendering UI. + + >>> _validate_workload_identity( + ... WorkloadOptions( + ... pathlib.Path("out"), "run-7", pathlib.Path("."), 0, 1.0, 1.0, + ... 0.0, "READY", 1.0, + ... ) + ... ) + + Raises + ------ + ValueError + If the run identity or default sentinel value is not terminal-safe. + """ + sentinel_text(options.run_id, "request", options.sentinel_prefix) def source_lines(source_root: pathlib.Path, seed: int) -> tuple[str, ...]: @@ -254,6 +321,7 @@ def prepare_output(options: WorkloadOptions) -> WorkloadPaths: FileExistsError If another process already owns the requested output directory. """ + _validate_workload_identity(options) root = options.output_dir root.mkdir(mode=0o700, parents=True, exist_ok=False) streams = root / "streams" @@ -404,11 +472,7 @@ def _request_from_marker( if ( not isinstance(request_id, str) or not request_id - or len(request_id) > 128 - or any( - not (character.isascii() and (character.isalnum() or character in "-_")) - for character in request_id - ) + or not _is_terminal_safe_component(request_id) or path.name != f"{request_id}.json" or request_id in seen_request_ids or isinstance(requested, bool) @@ -416,11 +480,13 @@ def _request_from_marker( or requested < 0 or isinstance(value, bool) or not isinstance(value, str) - or not value - or "\n" in value - or "\r" in value + or not _is_terminal_safe_component(value) ): return None + try: + sentinel_text(options.run_id, request_id, value) + except ValueError: + return None return request_id, requested, value @@ -455,6 +521,7 @@ def run_serve(options: WorkloadOptions) -> int: The service owns signals and a real marker tree, so its gate, timing, and shutdown behavior is exercised in ``tests/test_orchestration_fuzzer.py``. """ + _validate_workload_identity(options) if options.frame_rate_hz <= 0: message = "frame_rate_hz must be positive" raise ValueError(message) @@ -637,6 +704,7 @@ def run_preview(options: WorkloadOptions) -> int: exercised by the module-import tests; service rendering is covered by the dedicated functional test file. """ + _validate_workload_identity(options) if options.frame_rate_hz <= 0: message = "frame_rate_hz must be positive" raise ValueError(message) diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index 1fe46cfac3..219a529720 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -13,6 +13,7 @@ import stat import subprocess import sys +import threading import time import types import typing as t @@ -2091,6 +2092,426 @@ def _assert_wait_tokens_only_reach_delayed_pane( assert matched == () +def _isolated_wait_context( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + *, + mode: t.Any, + lane: t.Any, + engine: t.Any, + run_id: str = "wait-run", +) -> t.Any: + """Build the marker boundary needed by deterministic waiter regressions.""" + fuzzer_root = tmp_path / "fuzzer" + (fuzzer_root / "requests").mkdir(parents=True) + (fuzzer_root / "sentinels").mkdir() + return types.SimpleNamespace( + mode=mode, + lane=lane, + engine=engine, + delayed_pane_id="%7", + run_id=run_id, + scratch=tmp_path, + sentinel_delay_ns=0, + fuzzer=types.SimpleNamespace(poll=lambda: None), + ) + + +def _blocking_tmux_binary(tmp_path: pathlib.Path) -> tuple[pathlib.Path, pathlib.Path]: + """Create a real blocking process so timeout cleanup stays observable.""" + executable = tmp_path / "blocking-tmux" + pid_path = tmp_path / "blocking-tmux.pid" + executable.write_text( + "#!/usr/bin/env python3\n" + "import os\n" + "import pathlib\n" + "import time\n" + "pathlib.Path(os.environ['BLOCKING_TMUX_PID']).write_text(\n" + " str(os.getpid()), encoding='ascii'\n" + ")\n" + "time.sleep(0.5)\n", + encoding="utf-8", + ) + executable.chmod(executable.stat().st_mode | stat.S_IXUSR) + return executable, pid_path + + +@pytest.mark.parametrize( + ("field", "unsafe"), + ( + ("run_id", "unsafe run"), + ("run_id", "unsafe\tcontrol"), + ("run_id", "unsafe\nline"), + ("run_id", "unsafe\x1bescape"), + ("run_id", "unsafe\x08backspace"), + ("run_id", "nonascii-π"), + ("run_id", "r" * 129), + ("request_id", "unsafe request"), + ("request_id", "unsafe\tcontrol"), + ("request_id", "unsafe\nline"), + ("request_id", "unsafe\x1bescape"), + ("request_id", "unsafe\x08backspace"), + ("request_id", "nonascii-π"), + ("request_id", "q" * 129), + ("value", "unsafe value"), + ("value", "unsafe\tcontrol"), + ("value", "unsafe\nline"), + ("value", "unsafe\x1bescape"), + ("value", "unsafe\x08backspace"), + ("value", "nonascii-π"), + ("value", "v" * 129), + ), +) +def test_request_sentinel_rejects_unsafe_components_before_publication( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + field: str, + unsafe: str, +) -> None: + """Terminal controls and oversized identities never reach marker storage.""" + context = _isolated_wait_context( + benchmark_module, + tmp_path, + mode=benchmark_module.ExecutionMode.SYNC, + lane=benchmark_module.EngineLane.SUBPROCESS, + engine=object(), + ) + arguments = { + "request_id": "request.safe:1", + "value": "VALUE.safe:1", + } + if field == "run_id": + context.run_id = unsafe + else: + arguments[field] = unsafe + + with pytest.raises(ValueError, match="terminal-safe"): + benchmark_module.request_sentinel(context, **arguments) + + assert tuple((tmp_path / "fuzzer" / "requests").iterdir()) == () + + +def test_request_sentinel_accepts_literal_maximum_terminal_safe_token( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """All documented boundary bytes remain valid and exactly reproducible.""" + context = _isolated_wait_context( + benchmark_module, + tmp_path, + mode=benchmark_module.ExecutionMode.SYNC, + lane=benchmark_module.EngineLane.SUBPROCESS, + engine=object(), + run_id="r" * 128, + ) + + request = benchmark_module.request_sentinel( + context, + request_id="q" * 128, + value="v" * 128, + ) + + assert len(f"{request.token}\n".encode()) == 422 + assert benchmark_module._SENTINEL_RECORD_MAX_BYTES == 422 + marker = json.loads( + next((tmp_path / "fuzzer" / "requests").iterdir()).read_text(encoding="utf-8") + ) + assert marker["run_id"] == "r" * 128 + assert marker["request_id"] == "q" * 128 + assert marker["value"] == "v" * 128 + + +@pytest.mark.parametrize("timeout_s", (3.000_001, float("inf"), float("nan"))) +def test_capture_wait_rejects_unbounded_timeout_before_publication( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + timeout_s: float, +) -> None: + """The history-sized wait ceiling applies before a request is visible.""" + context = _isolated_wait_context( + benchmark_module, + tmp_path, + mode=benchmark_module.ExecutionMode.SYNC, + lane=benchmark_module.EngineLane.SUBPROCESS, + engine=object(), + ) + + with pytest.raises(ValueError, match=r"at most 3\.0 seconds"): + benchmark_module.wait_capture_poll_sync( + context, + request_id="bounded-timeout", + timeout_s=timeout_s, + ) + + assert tuple((tmp_path / "fuzzer" / "requests").iterdir()) == () + + +def test_start_fuzzer_rejects_frame_rate_above_wait_history_bound( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The producer cannot outrun the documented capture history budget.""" + monkeypatch.setattr( + benchmark_module.subprocess, + "Popen", + lambda *_args, **_kwargs: pytest.fail("fuzzer started above frame bound"), + ) + + with pytest.raises(ValueError, match=r"at most 40\.0 frames per second"): + benchmark_module.start_fuzzer( + tmp_path, + "bounded-rate", + frame_rate_hz=40.000_001, + ) + + +def test_sync_subprocess_capture_timeout_kills_and_reaps_exact_child( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A blocked real subprocess cannot outlive the capture wait deadline.""" + from libtmux.experimental.engines import SubprocessEngine + + executable, pid_path = _blocking_tmux_binary(tmp_path) + monkeypatch.setenv("BLOCKING_TMUX_PID", str(pid_path)) + engine = SubprocessEngine(tmux_bin=executable) + context = _isolated_wait_context( + benchmark_module, + tmp_path, + mode=benchmark_module.ExecutionMode.SYNC, + lane=benchmark_module.EngineLane.SUBPROCESS, + engine=engine, + ) + threads_before = frozenset(thread.ident for thread in threading.enumerate()) + started = time.monotonic() + + with pytest.raises( + TimeoutError, + match="capture wait timed out for request sync-blocked", + ): + benchmark_module.wait_capture_poll_sync( + context, + request_id="sync-blocked", + timeout_s=0.05, + poll_interval_s=0.01, + ) + + elapsed = time.monotonic() - started + pid = int(pid_path.read_text(encoding="ascii")) + assert elapsed < 0.25 + assert benchmark_module._process_start_time(pid) is None + assert frozenset(thread.ident for thread in threading.enumerate()) == threads_before + + +def test_sync_control_capture_uses_and_restores_remaining_transport_timeout( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """The supported control engine owns a blocked request until timeout.""" + from libtmux.experimental.engines.control_mode import ( + ControlModeEngine, + ControlModeError, + ) + + class BlockingControlEngine(ControlModeEngine): + def __init__(self) -> None: + super().__init__(timeout=0.5) + self.observed_timeout: float | None = None + + def tmux_version(self) -> None: + return None + + def run(self, request: t.Any) -> t.NoReturn: + del request + self.observed_timeout = self.timeout + threading.Event().wait(self.timeout) + message = f"blocked for {self.timeout}s" + raise ControlModeError(message) + + engine = BlockingControlEngine() + context = _isolated_wait_context( + benchmark_module, + tmp_path, + mode=benchmark_module.ExecutionMode.SYNC, + lane=benchmark_module.EngineLane.CONTROL, + engine=engine, + ) + threads_before = frozenset(thread.ident for thread in threading.enumerate()) + started = time.monotonic() + + with pytest.raises( + TimeoutError, + match="capture wait timed out for request control-blocked", + ): + benchmark_module.wait_capture_poll_sync( + context, + request_id="control-blocked", + timeout_s=0.05, + poll_interval_s=0.01, + ) + + elapsed = time.monotonic() - started + assert elapsed < 0.25 + assert engine.observed_timeout is not None + assert 0 < engine.observed_timeout <= 0.05 + assert engine.timeout == 0.5 + assert frozenset(thread.ident for thread in threading.enumerate()) == threads_before + + +def test_async_subprocess_capture_timeout_drains_task_and_reaps_child( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Cancelling a blocked native async capture leaves no task or child.""" + from libtmux.experimental.engines import AsyncSubprocessEngine + + executable, pid_path = _blocking_tmux_binary(tmp_path) + monkeypatch.setenv("BLOCKING_TMUX_PID", str(pid_path)) + engine = AsyncSubprocessEngine(tmux_bin=executable) + context = _isolated_wait_context( + benchmark_module, + tmp_path, + mode=benchmark_module.ExecutionMode.ASYNC, + lane=benchmark_module.EngineLane.SUBPROCESS, + engine=engine, + ) + + async def exercise() -> tuple[float, tuple[str, ...]]: + started = time.monotonic() + with pytest.raises( + TimeoutError, + match="capture wait timed out for request async-blocked", + ): + await asyncio.wait_for( + benchmark_module.wait_capture_poll_async( + context, + request_id="async-blocked", + timeout_s=0.05, + poll_interval_s=0.01, + ), + timeout=0.3, + ) + await asyncio.sleep(0) + names = tuple( + task.get_name() + for task in asyncio.all_tasks() + if task is not asyncio.current_task() + and ( + task.get_name().startswith("bench-capture-wait-") + or task.get_name().startswith("libtmux-async-subprocess-") + ) + ) + return time.monotonic() - started, names + + elapsed, task_names = asyncio.run(exercise(), debug=True) + pid = int(pid_path.read_text(encoding="ascii")) + assert elapsed < 0.25 + assert task_names == () + assert benchmark_module._process_start_time(pid) is None + + +def test_control_wait_timeout_drains_pending_subscription( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """A pending real subscriber unregisters without loop diagnostics.""" + from libtmux.experimental.engines.async_control_mode import ( + AsyncControlModeEngine, + ) + + engine = AsyncControlModeEngine() + context = _isolated_wait_context( + benchmark_module, + tmp_path, + mode=benchmark_module.ExecutionMode.ASYNC, + lane=benchmark_module.EngineLane.CONTROL, + engine=engine, + ) + + async def exercise() -> tuple[list[dict[str, t.Any]], tuple[str, ...]]: + diagnostics: list[dict[str, t.Any]] = [] + loop = asyncio.get_running_loop() + loop.set_exception_handler(lambda _loop, details: diagnostics.append(details)) + with pytest.raises( + TimeoutError, + match="control stream wait timed out for request pending-timeout", + ): + await benchmark_module.wait_control_stream( + context, + request_id="pending-timeout", + timeout_s=0.03, + ) + await asyncio.sleep(0) + assert engine._subscribers == set() + names = tuple( + task.get_name() + for task in asyncio.all_tasks() + if task is not asyncio.current_task() + and task.get_name().startswith("bench-control-wait-") + ) + return diagnostics, names + + diagnostics, task_names = asyncio.run(exercise(), debug=True) + assert diagnostics == [] + assert task_names == () + + +def test_control_wait_external_cancellation_preserves_payload_and_unregisters( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """Caller cancellation drains pending iteration and remains unchanged.""" + from libtmux.experimental.engines.async_control_mode import ( + AsyncControlModeEngine, + ) + + engine = AsyncControlModeEngine() + context = _isolated_wait_context( + benchmark_module, + tmp_path, + mode=benchmark_module.ExecutionMode.ASYNC, + lane=benchmark_module.EngineLane.CONTROL, + engine=engine, + ) + + async def exercise() -> tuple[list[dict[str, t.Any]], tuple[str, ...]]: + diagnostics: list[dict[str, t.Any]] = [] + loop = asyncio.get_running_loop() + loop.set_exception_handler(lambda _loop, details: diagnostics.append(details)) + waiter = asyncio.create_task( + benchmark_module.wait_control_stream( + context, + request_id="pending-cancel", + timeout_s=1.0, + ), + name="test-pending-control-wait", + ) + deadline = loop.time() + 0.5 + while not engine._subscribers and loop.time() < deadline: + await asyncio.sleep(0) + assert engine._subscribers + waiter.cancel("caller-stop") + with pytest.raises(asyncio.CancelledError) as captured: + await waiter + await asyncio.sleep(0) + assert captured.value.args == ("caller-stop",) + assert engine._subscribers == set() + names = tuple( + task.get_name() + for task in asyncio.all_tasks() + if task is not asyncio.current_task() + and task.get_name().startswith("bench-control-wait-") + ) + return diagnostics, names + + diagnostics, task_names = asyncio.run(exercise(), debug=True) + assert diagnostics == [] + assert task_names == () + + def test_control_wait_matches_split_target_bytes_and_closes_subscription( benchmark_module: types.ModuleType, tmp_path: pathlib.Path, @@ -2311,6 +2732,45 @@ def test_wait_evidence_rejects_wrong_or_stale_request_identity( ) +def test_maximum_wait_token_round_trips_through_one_real_wrapped_pane( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """The longest supported token survives tmux wrapping byte-for-byte.""" + context = benchmark_module.setup_sync( + benchmark_module.Topology(1, 2, 2), + benchmark_module.EngineLane.SUBPROCESS, + tmp_path / "maximum-wait-token", + run_id="r" * 128, + delayed_ordinal=2, + ) + cleanup = None + try: + benchmark_module.release_activity_gate(context) + result = benchmark_module.wait_capture_poll_sync( + context, + request_id="q" * 128, + value="v" * 128, + timeout_s=2.0, + poll_interval_s=0.005, + ) + captured = context.server.cmd( + "capture-pane", + "-t", + context.delayed_pane_id, + "-p", + "-J", + "-S", + str(-benchmark_module._WAIT_CAPTURE_HISTORY_LINES), + ) + assert captured.returncode == 0, captured.stderr + assert result.token in captured.stdout + assert len(f"{result.token}\n".encode()) == 422 + finally: + cleanup = asyncio.run(benchmark_module.cleanup_run(context)) + assert cleanup.complete, cleanup.errors + + def test_repeated_wait_capture_poll_sync_uses_one_active_topology( benchmark_module: types.ModuleType, tmp_path: pathlib.Path, diff --git a/tests/test_orchestration_fuzzer.py b/tests/test_orchestration_fuzzer.py index e789b5b44a..6f0ec0730a 100644 --- a/tests/test_orchestration_fuzzer.py +++ b/tests/test_orchestration_fuzzer.py @@ -2,6 +2,7 @@ from __future__ import annotations +import dataclasses import hashlib import importlib.util import json @@ -60,6 +61,97 @@ def test_sentinel_text_is_unique_to_run_and_request( ) +def test_sentinel_text_accepts_maximum_terminal_safe_components( + fuzzer_module: types.ModuleType, +) -> None: + """The documented component and complete-record boundaries stay literal.""" + sentinel = fuzzer_module.sentinel_text("r" * 128, "q" * 128, "v" * 128) + + assert len(f"{sentinel}\n".encode()) == 422 + assert fuzzer_module.SENTINEL_COMPONENT_MAX_BYTES == 128 + assert fuzzer_module.SENTINEL_RECORD_MAX_BYTES == 422 + + +@pytest.mark.parametrize( + ("component", "unsafe"), + ( + ("run_id", "unsafe run"), + ("run_id", "unsafe\tcontrol"), + ("run_id", "unsafe\nline"), + ("run_id", "unsafe\x1bescape"), + ("run_id", "unsafe\x08backspace"), + ("run_id", "nonascii-π"), + ("run_id", "r" * 129), + ("request_id", "unsafe request"), + ("request_id", "unsafe\tcontrol"), + ("request_id", "unsafe\nline"), + ("request_id", "unsafe\x1bescape"), + ("request_id", "unsafe\x08backspace"), + ("request_id", "nonascii-π"), + ("request_id", "q" * 129), + ("value", "unsafe value"), + ("value", "unsafe\tcontrol"), + ("value", "unsafe\nline"), + ("value", "unsafe\x1bescape"), + ("value", "unsafe\x08backspace"), + ("value", "nonascii-π"), + ("value", "v" * 129), + ), +) +def test_sentinel_text_rejects_unsafe_or_oversized_components( + fuzzer_module: types.ModuleType, + component: str, + unsafe: str, +) -> None: + """No terminal control or unbounded component reaches stream output.""" + values = {"run_id": "run.safe:1", "request_id": "request.safe:1", "value": "READY"} + values[component] = unsafe + + with pytest.raises(ValueError, match="terminal-safe"): + fuzzer_module.sentinel_text(**values) + + +@pytest.mark.parametrize( + ("field", "unsafe"), + ( + ("run_id", "unsafe run"), + ("run_id", "r" * 129), + ("sentinel_prefix", "unsafe\tvalue"), + ("sentinel_prefix", "v" * 129), + ), +) +def test_serve_rejects_unsafe_identity_before_output_publication( + fuzzer_module: types.ModuleType, + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, + field: str, + unsafe: str, +) -> None: + """Invalid service identity fails before the marker tree can be created.""" + options = fuzzer_module.WorkloadOptions( + tmp_path / "output", + "run.safe:1", + tmp_path, + 0, + 1.0, + 1.0, + 0.0, + "READY", + 1.0, + ) + options = dataclasses.replace(options, **{field: unsafe}) + monkeypatch.setattr( + fuzzer_module, + "prepare_output", + lambda _options: pytest.fail("output publication preceded identity validation"), + ) + + with pytest.raises(ValueError, match="terminal-safe"): + fuzzer_module.run_serve(options) + + assert not options.output_dir.exists() + + def test_source_lines_uses_sorted_paths_and_a_private_seeded_shuffle( fuzzer_module: types.ModuleType, tmp_path: pathlib.Path, @@ -104,7 +196,7 @@ def test_render_frame_uses_only_its_mode_epoch_and_seeded_corpus( @pytest.mark.parametrize( ("filename", "request_id"), ( - ("nested.json.json", "nested.json"), + ("nested.json.json", "nested"), (".json", ".json"), ("space id.json", "space id"), ("unicode-π.json", "unicode-π"), @@ -143,6 +235,34 @@ def test_request_marker_rejects_noncanonical_filename_payload_id( ) +def test_request_marker_accepts_terminal_safe_dot_and_colon_identity( + fuzzer_module: types.ModuleType, +) -> None: + """Dots and colons remain canonical terminal-safe identity components.""" + options = fuzzer_module.WorkloadOptions( + pathlib.Path("out"), + "run-7", + pathlib.Path(), + 0, + 1.0, + 1.0, + 0.0, + "READY", + 1.0, + ) + + assert fuzzer_module._request_from_marker( + { + "request_id": "sample.part:1", + "requested_monotonic_ns": 7, + "value": "READY", + }, + pathlib.Path("sample.part:1.json"), + options, + seen_request_ids=set(), + ) == ("sample.part:1", 7, "READY") + + def test_request_marker_rejects_duplicate_payload_id( fuzzer_module: types.ModuleType, ) -> None: From 13d1b1d662ace61d835bf1b6ea8651654d87ecba Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 11:37:41 -0500 Subject: [PATCH 19/67] Bench(feat[runner]): Supervise active runs why: Make active benchmark execution recoverable and preserve validated evidence across completion, refusal, cutoff, cancellation, and failure. what: - Add public run and ramp commands with a hidden checkpointing worker - Add sequence-based supervision and exact identity-checked cleanup - Validate JSON artifacts and render descriptive Markdown summaries - Cover real CLI phase graphs, ramps, recovery, and every engine lane --- scripts/bench_orchestration.py | 3674 +++++++++++++++++++++- tests/test_bench_orchestration_script.py | 842 +++++ 2 files changed, 4403 insertions(+), 113 deletions(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index 6e633cf8a0..437457d8bf 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -17,12 +17,14 @@ import contextlib import dataclasses import enum +import functools import hashlib import inspect import json import math import os import pathlib +import platform import random import resource import shlex @@ -66,6 +68,31 @@ # history rows retain 120 ordinary delayed frames plus the 422-byte sentinel # with ample wrapping headroom in the required active 1x2x2 topology. _WAIT_CAPTURE_HISTORY_LINES = 5_000 +_SEARCH_POSITIONS = ("first", "middle", "last") +_ENUMERATION_KINDS = ("sessions", "windows", "panes") +_SEARCH_FAMILIES = ("classic", "snapshot", "end-to-end") +_RUNNER_REPEATABLE_PHASES = ( + "mutation.bulk", + "wait.capture-poll", + *tuple(f"enumeration.{kind}" for kind in _ENUMERATION_KINDS), + "capture.serial", + "capture.batched", + *tuple( + f"search.{family}.{kind}.{position}" + for family in _SEARCH_FAMILIES + for kind in _ENUMERATION_KINDS + for position in _SEARCH_POSITIONS + ), + "search.contents", +) +_RUNNER_PHASES = ( + "setup", + "stabilization", + "mutation.bulk", + "wait.capture-poll", + "wait.control-stream", + *_RUNNER_REPEATABLE_PHASES[2:], +) def _is_terminal_safe_component(value: object) -> bool: @@ -1432,6 +1459,81 @@ class RepeatablePhaseResult: failure: RepeatablePhaseFailure | None = None +@dataclasses.dataclass(frozen=True) +class PhaseObservation: + """Recomputable correctness and work counts for one accepted timing. + + Attributes + ---------- + ordinal : int + Zero-based timed sample ordinal within the cell. + strategy : str + Stable cell name that produced the observation. + duration_ns : int + Raw integer timing associated with this observation. + metrics : ExecutionMetrics | None + Typed operation and transport counts when the cell contacts tmux. + row_count : int | None + Exact hierarchy rows returned by enumeration. + byte_count : int | None + UTF-8 bytes retained by an all-pane capture. + line_count : int | None + Typed lines retained by an all-pane capture. + poll_count : int | None + Capture requests issued by a delayed-output wait. + frame_count : int | None + Matching control notifications consumed by a delayed-output wait. + dropped_notification_delta : int | None + Control notification drops observed during a delayed-output wait. + configured_delay_ns : int | None + Intentional fuzzer delay excluded from waiter overhead. + scheduling_lateness_ns : int | None + Fuzzer emission lateness beyond its requested schedule. + detection_overhead_ns : int | None + Waiter detection time after actual fuzzer emission. + scanned_count : int | None + Candidate cardinality scanned by a search. + matched_count : int | None + Exact accepted match cardinality for a search. + target : str | None + Concrete search target or delayed pane identifier. + session_count : int | None + Sessions verified or mutated by this observation. + window_count : int | None + Windows verified or mutated by this observation. + pane_count : int | None + Panes verified, mutated, or captured by this observation. + verified : bool + Whether typed and live checks accepted the observation. + + Examples + -------- + >>> PhaseObservation(0, "enumeration.sessions", 7, row_count=1).row_count + 1 + """ + + ordinal: int + strategy: str + duration_ns: int + metrics: ExecutionMetrics | None = None + row_count: int | None = None + byte_count: int | None = None + line_count: int | None = None + poll_count: int | None = None + frame_count: int | None = None + dropped_notification_delta: int | None = None + configured_delay_ns: int | None = None + scheduling_lateness_ns: int | None = None + detection_overhead_ns: int | None = None + scanned_count: int | None = None + matched_count: int | None = None + target: str | None = None + session_count: int | None = None + window_count: int | None = None + pane_count: int | None = None + verified: bool = True + + @dataclasses.dataclass(frozen=True) class PhaseReport: """Raw and summarized evidence for one named benchmark cell. @@ -1448,6 +1550,14 @@ class PhaseReport: Timed results, including explicitly rejected rows. summary : collections.abc.Mapping[str, int | float] | None Statistics recomputed from accepted rows only. + status : {"in_progress", "completed", "failed", "not_applicable"} + Lifecycle or applicability state of this benchmark cell. + warmup : int + Untimed invocations requested for a repeatable cell. + runs : int + Timed invocations requested for a repeatable cell. + observations : tuple[PhaseObservation, ...] + Typed counts corresponding one-to-one with accepted timed samples. """ name: str @@ -1455,6 +1565,12 @@ class PhaseReport: observed_topology: Topology | None samples: tuple[RawSample, ...] = () summary: cabc.Mapping[str, int | float] | None = None + status: t.Literal["in_progress", "completed", "failed", "not_applicable"] = ( + "completed" + ) + warmup: int = 0 + runs: int = 0 + observations: tuple[PhaseObservation, ...] = () def __post_init__(self) -> None: """Freeze a copied summary so callers cannot alter recorded evidence. @@ -1481,10 +1597,19 @@ class CleanupReport: Whether process, socket, and scratch cleanup verification passed. errors : tuple[str, ...] Cleanup verification failures. + processes_absent : bool | None + Whether every recorded PID/start-time identity was absent. + socket_absent : bool | None + Whether the exact isolated socket path was absent. + scratch_absent : bool | None + Whether the exact private scratch directory was absent. """ complete: bool errors: tuple[str, ...] = () + processes_absent: bool | None = None + socket_absent: bool | None = None + scratch_absent: bool | None = None class SetupCleanupError(RuntimeError): @@ -1623,6 +1748,72 @@ class ProcessIdentity: start_time: int +@dataclasses.dataclass(frozen=True) +class ProgressEvent: + """One append-only worker checkpoint observed by the supervisor. + + Attributes + ---------- + run_id : str + Exact run identity shared with the checkpoint report. + sequence : int + Strictly increasing worker-local progress sequence. + checkpoint : str + Stable lifecycle or phase checkpoint name. + monotonic_ns : int + Worker monotonic publication time. + processes : tuple[ProcessIdentity, ...] + Cumulative exact identities known at publication time. + schema_version : int + Progress stream schema version. + + Examples + -------- + >>> ProgressEvent("run-7", 2, "setup", 10).sequence + 2 + """ + + run_id: str + sequence: int + checkpoint: str + monotonic_ns: int + processes: tuple[ProcessIdentity, ...] = () + schema_version: int = 1 + + +@dataclasses.dataclass(frozen=True) +class EnvironmentReport: + """Descriptive local environment attached to one benchmark artifact. + + Attributes + ---------- + python_version : str + Running Python implementation version. + tmux_version : str | None + ``tmux -V`` output when the executable was available. + cpu_count : int | None + Logical CPU count exposed to Python. + seed : int + Deterministic phase-order seed. + command_line : tuple[str, ...] + Exact worker-independent public invocation arguments. + git_revision : str | None + Current checkout revision when Git could resolve it. + + Examples + -------- + >>> EnvironmentReport("3.10", "tmux 3.4", 4, 11, ("run",), "abc").seed + 11 + """ + + python_version: str + tmux_version: str | None + cpu_count: int | None + seed: int + command_line: tuple[str, ...] + git_revision: str | None + + @dataclasses.dataclass(frozen=True) class TopologyTotals: """Flat session, window, and pane totals used in mismatch evidence. @@ -1817,6 +2008,10 @@ class RunContext: Last fuzzer monotonic publication timestamp observed. ambient_tmux_environment : tuple[str | None, str | None] Original ``TMUX`` and ``TMUX_PANE`` values restored after cleanup. + setup_metrics : ExecutionMetrics | None + Exact construction operation and dispatch counts. + process_identity_callback : collections.abc.Callable | None + Private worker hook called as owned identities become known. Examples -------- @@ -1856,6 +2051,8 @@ class RunContext: heartbeat_epoch: int = -1 heartbeat_monotonic_ns: int = -1 ambient_tmux_environment: tuple[str | None, str | None] = (None, None) + setup_metrics: ExecutionMetrics | None = None + process_identity_callback: cabc.Callable[[ProcessIdentity], None] | None = None @dataclasses.dataclass(frozen=True) @@ -1870,11 +2067,23 @@ class RampStep: Terminal result for the step. reason : str | None Required reason for an unattempted shape after a terminal step. + run_id : str | None + Fresh run identity for an attempted shape. + report_path : str | None + Per-shape validated JSON artifact. + scratch_path : str | None + Fresh private scratch path, absent after cleanup. + socket_path : str | None + Fresh isolated socket path, absent after cleanup. """ shape: Topology status: t.Literal["completed", "refused", "failed", "cutoff", "not_attempted"] reason: str | None = None + run_id: str | None = None + report_path: str | None = None + scratch_path: str | None = None + socket_path: str | None = None @dataclasses.dataclass(frozen=True) @@ -1907,6 +2116,32 @@ class RunReport: Decision before an optional predictive override. schema_version : int Stable artifact schema version. + run_id : str | None + Fresh terminal-safe identity for an executable scenario. + lane : {"subprocess", "control"} | None + Selected transport lane for a single executable scenario. + mode : {"sync", "async"} | None + Selected dispatch mode for a single executable scenario. + warmup : int | None + Untimed invocation count per repeatable cell. + runs : int | None + Timed invocation count per repeatable cell. + failed_phase : str | None + Phase or boundary that produced a terminal non-success status. + error : str | None + Concise terminal reason retained by the supervisor. + processes : tuple[ProcessIdentity, ...] + Exact worker, fuzzer, server, and pane identities observed by progress. + scratch_path : str | None + Exact private run path whose absence cleanup verified. + socket_path : str | None + Exact isolated socket path whose absence cleanup verified. + progress_path : str | None + Append-only JSONL progress artifact owned by the supervisor. + progress_sequence : int + Highest worker sequence incorporated into this checkpoint. + environment : EnvironmentReport | None + Descriptive local environment; never a causal performance claim. """ requested_topology: Topology @@ -1923,6 +2158,19 @@ class RunReport: guard_decision: GuardDecision | None = None original_guard_decision: GuardDecision | None = None schema_version: int = 1 + run_id: str | None = None + lane: t.Literal["subprocess", "control"] | None = None + mode: t.Literal["sync", "async"] | None = None + warmup: int | None = None + runs: int | None = None + failed_phase: str | None = None + error: str | None = None + processes: tuple[ProcessIdentity, ...] = () + scratch_path: str | None = None + socket_path: str | None = None + progress_path: str | None = None + progress_sequence: int = -1 + environment: EnvironmentReport | None = None def _json_value(value: object) -> object: @@ -2017,6 +2265,365 @@ def write_json_atomic( raise +def _json_mapping(value: object, label: str) -> dict[str, t.Any]: + """Return one string-keyed JSON mapping or reject the artifact. + + >>> _json_mapping({"value": 1}, "example")["value"] + 1 + + Parameters + ---------- + value : object + Parsed JSON value. + label : str + Human-readable field name used in errors. + + Returns + ------- + dict[str, typing.Any] + Mapping copied from the parsed artifact. + + Raises + ------ + ValueError + If the value is not a mapping with string keys. + """ + if not isinstance(value, dict) or any(not isinstance(key, str) for key in value): + message = f"{label} must be a JSON object" + raise ValueError(message) + return t.cast(dict[str, t.Any], value) + + +def _topology_from_json(value: object) -> Topology: + """Decode one exact topology mapping. + + >>> _topology_from_json( + ... {"sessions": 1, "windows_per_session": 2, "panes_per_window": 3} + ... ) + Topology(sessions=1, windows_per_session=2, panes_per_window=3) + """ + row = _json_mapping(value, "topology") + try: + topology = Topology( + sessions=row["sessions"], + windows_per_session=row["windows_per_session"], + panes_per_window=row["panes_per_window"], + ) + except KeyError as error: + message = f"topology lacks {error.args[0]}" + raise ValueError(message) from error + if any( + type(dimension) is not int or dimension <= 0 + for dimension in ( + topology.sessions, + topology.windows_per_session, + topology.panes_per_window, + ) + ): + message = "topology dimensions must be positive integers" + raise ValueError(message) + return topology + + +def _host_snapshot_from_json(value: object | None) -> HostSnapshot | None: + """Decode an optional host-resource snapshot. + + >>> _host_snapshot_from_json(None) is None + True + """ + if value is None: + return None + row = _json_mapping(value, "host snapshot") + return HostSnapshot( + available_memory_bytes=row.get("available_memory_bytes"), + physical_memory_bytes=row.get("physical_memory_bytes"), + memory_current_bytes=row.get("memory_current_bytes"), + memory_max_bytes=row.get("memory_max_bytes"), + pids_current=row.get("pids_current"), + pids_max=row.get("pids_max"), + nofile_soft_limit=row.get("nofile_soft_limit"), + nofile_hard_limit=row.get("nofile_hard_limit"), + memory_pressure_some_avg10=row.get("memory_pressure_some_avg10"), + source_errors=_json_mapping(row.get("source_errors", {}), "source errors"), + ) + + +def _guard_from_json(value: object | None) -> GuardDecision | None: + """Decode an optional guard decision. + + >>> _guard_from_json(None) is None + True + """ + if value is None: + return None + row = _json_mapping(value, "guard decision") + snapshot = _host_snapshot_from_json(row.get("snapshot")) + if snapshot is None: + message = "guard decision requires a host snapshot" + raise ValueError(message) + return GuardDecision( + allowed=t.cast(bool, row.get("allowed")), + kind=t.cast( + t.Literal["ok", "predictive_refusal", "runtime_cutoff"], + row.get("kind"), + ), + rule=t.cast(str | None, row.get("rule")), + observed=t.cast(int | None, row.get("observed")), + limit=t.cast(int | None, row.get("limit")), + forceable=t.cast(bool, row.get("forceable")), + snapshot=snapshot, + ) + + +def _metrics_from_json(value: object | None) -> ExecutionMetrics | None: + """Decode optional operation and transport counts. + + >>> _metrics_from_json(None) is None + True + """ + if value is None: + return None + row = _json_mapping(value, "execution metrics") + return ExecutionMetrics( + operations=t.cast(int, row.get("operations")), + planner_steps=t.cast(int, row.get("planner_steps")), + engine_batches=t.cast(int, row.get("engine_batches")), + tmux_requests=t.cast(int, row.get("tmux_requests")), + process_starts=t.cast(int, row.get("process_starts")), + ) + + +def _sample_from_json(value: object) -> RawSample: + """Decode one raw timing sample. + + >>> _sample_from_json({"duration_ns": 3, "accepted": True}).duration_ns + 3 + """ + row = _json_mapping(value, "raw sample") + return RawSample( + duration_ns=t.cast(int | None, row.get("duration_ns")), + accepted=t.cast(bool, row.get("accepted")), + error=t.cast(str | None, row.get("error")), + verified=t.cast(bool, row.get("verified", False)), + strategy=t.cast(str | None, row.get("strategy")), + ordinal=t.cast(int | None, row.get("ordinal")), + resources_before=_host_snapshot_from_json(row.get("resources_before")), + resources_after=_host_snapshot_from_json(row.get("resources_after")), + ) + + +def _observation_from_json(value: object) -> PhaseObservation: + """Decode one typed phase observation. + + >>> _observation_from_json( + ... {"ordinal": 0, "strategy": "x", "duration_ns": 2} + ... ).strategy + 'x' + """ + row = _json_mapping(value, "phase observation") + return PhaseObservation( + ordinal=t.cast(int, row.get("ordinal")), + strategy=t.cast(str, row.get("strategy")), + duration_ns=t.cast(int, row.get("duration_ns")), + metrics=_metrics_from_json(row.get("metrics")), + row_count=row.get("row_count"), + byte_count=row.get("byte_count"), + line_count=row.get("line_count"), + poll_count=row.get("poll_count"), + frame_count=row.get("frame_count"), + dropped_notification_delta=row.get("dropped_notification_delta"), + configured_delay_ns=row.get("configured_delay_ns"), + scheduling_lateness_ns=row.get("scheduling_lateness_ns"), + detection_overhead_ns=row.get("detection_overhead_ns"), + scanned_count=row.get("scanned_count"), + matched_count=row.get("matched_count"), + target=row.get("target"), + session_count=row.get("session_count"), + window_count=row.get("window_count"), + pane_count=row.get("pane_count"), + verified=t.cast(bool, row.get("verified", False)), + ) + + +def _phase_from_json(value: object) -> PhaseReport: + """Decode one phase report and its raw observations. + + >>> _phase_from_json({ + ... "name": "stabilization", + ... "requested_topology": { + ... "sessions": 1, "windows_per_session": 1, "panes_per_window": 1 + ... }, + ... "observed_topology": None, + ... }).name + 'stabilization' + """ + row = _json_mapping(value, "phase") + observed = row.get("observed_topology") + summary = row.get("summary") + return PhaseReport( + name=t.cast(str, row.get("name")), + requested_topology=_topology_from_json(row.get("requested_topology")), + observed_topology=(None if observed is None else _topology_from_json(observed)), + samples=tuple(_sample_from_json(item) for item in row.get("samples", [])), + summary=(None if summary is None else _json_mapping(summary, "phase summary")), + status=t.cast( + t.Literal["in_progress", "completed", "failed", "not_applicable"], + row.get("status", "completed"), + ), + warmup=t.cast(int, row.get("warmup", 0)), + runs=t.cast(int, row.get("runs", 0)), + observations=tuple( + _observation_from_json(item) for item in row.get("observations", []) + ), + ) + + +def _cleanup_from_json(value: object) -> CleanupReport: + """Decode exact cleanup evidence. + + >>> _cleanup_from_json({"complete": True, "errors": []}).complete + True + """ + row = _json_mapping(value, "cleanup") + return CleanupReport( + complete=t.cast(bool, row.get("complete")), + errors=tuple(t.cast(t.Iterable[str], row.get("errors", []))), + processes_absent=t.cast(bool | None, row.get("processes_absent")), + socket_absent=t.cast(bool | None, row.get("socket_absent")), + scratch_absent=t.cast(bool | None, row.get("scratch_absent")), + ) + + +def _identity_from_json(value: object) -> ProcessIdentity: + """Decode one PID/start-time identity. + + >>> _identity_from_json({"role": "worker", "pid": 2, "start_time": 3}).pid + 2 + """ + row = _json_mapping(value, "process identity") + return ProcessIdentity( + t.cast(str, row.get("role")), + t.cast(int, row.get("pid")), + t.cast(int, row.get("start_time")), + ) + + +def _environment_from_json(value: object | None) -> EnvironmentReport | None: + """Decode optional descriptive environment evidence. + + >>> _environment_from_json(None) is None + True + """ + if value is None: + return None + row = _json_mapping(value, "environment") + return EnvironmentReport( + python_version=t.cast(str, row.get("python_version")), + tmux_version=t.cast(str | None, row.get("tmux_version")), + cpu_count=t.cast(int | None, row.get("cpu_count")), + seed=t.cast(int, row.get("seed")), + command_line=tuple(t.cast(t.Iterable[str], row.get("command_line", []))), + git_revision=t.cast(str | None, row.get("git_revision")), + ) + + +def run_report_from_json(value: object) -> RunReport: + """Decode a JSON-native report into immutable typed evidence. + + >>> report = run_report_from_json({ + ... "requested_topology": { + ... "sessions": 1, "windows_per_session": 1, "panes_per_window": 1 + ... }, + ... "cleanup": {"complete": False, "errors": []}, + ... }) + >>> report.status + 'in_progress' + + Parameters + ---------- + value : object + Parsed JSON report object. + + Returns + ------- + RunReport + Immutable artifact suitable for :func:`validate_report`. + """ + row = _json_mapping(value, "run report") + observed = row.get("observed_topology") + ramp_rows = [] + for item in row.get("ramp", []): + ramp_row = _json_mapping(item, "ramp step") + ramp_rows.append( + RampStep( + shape=_topology_from_json(ramp_row.get("shape")), + status=t.cast( + t.Literal[ + "completed", + "refused", + "failed", + "cutoff", + "not_attempted", + ], + ramp_row.get("status"), + ), + reason=ramp_row.get("reason"), + run_id=ramp_row.get("run_id"), + report_path=ramp_row.get("report_path"), + scratch_path=ramp_row.get("scratch_path"), + socket_path=ramp_row.get("socket_path"), + ) + ) + return RunReport( + requested_topology=_topology_from_json(row.get("requested_topology")), + observed_topology=(None if observed is None else _topology_from_json(observed)), + status=row.get("status", "in_progress"), + phases=tuple(_phase_from_json(item) for item in row.get("phases", [])), + cleanup=_cleanup_from_json( + row.get("cleanup", {"complete": False, "errors": []}) + ), + maximum_completed=row.get("maximum_completed", False), + ramp=tuple(ramp_rows), + requested_shapes=tuple( + _topology_from_json(item) for item in row.get("requested_shapes", []) + ), + ramp_kind=row.get("ramp_kind", "none"), + guard_decision=_guard_from_json(row.get("guard_decision")), + original_guard_decision=_guard_from_json(row.get("original_guard_decision")), + schema_version=row.get("schema_version", 1), + run_id=row.get("run_id"), + lane=row.get("lane"), + mode=row.get("mode"), + warmup=row.get("warmup"), + runs=row.get("runs"), + failed_phase=row.get("failed_phase"), + error=row.get("error"), + processes=tuple(_identity_from_json(item) for item in row.get("processes", [])), + scratch_path=row.get("scratch_path"), + socket_path=row.get("socket_path"), + progress_path=row.get("progress_path"), + progress_sequence=row.get("progress_sequence", -1), + environment=_environment_from_json(row.get("environment")), + ) + + +def load_run_report(path: pathlib.Path) -> RunReport: + """Load one complete JSON report without accepting partial bytes. + + >>> with tempfile.TemporaryDirectory() as directory: + ... path = pathlib.Path(directory) / "report.json" + ... write_json_atomic(path, RunReport(Topology(1, 1, 1))) + ... load_run_report(path).requested_topology.panes + 1 + """ + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as error: + message = f"report is not complete JSON: {path}" + raise ValueError(message) from error + return run_report_from_json(value) + + def validate_report(report: RunReport) -> None: """Reject internally inconsistent benchmark evidence. @@ -2049,7 +2656,27 @@ def validate_report(report: RunReport) -> None: if report.status in terminal and not report.cleanup.complete: message = "terminal report requires complete cleanup" raise ValueError(message) + phase_statuses = {"in_progress", "completed", "failed", "not_applicable"} + if len({phase.name for phase in report.phases}) != len(report.phases): + message = "phase names must be unique" + raise ValueError(message) for phase in report.phases: + if phase.status not in phase_statuses: + message = "invalid phase status" + raise ValueError(message) + if ( + type(phase.warmup) is not int + or phase.warmup < 0 + or type(phase.runs) is not int + or phase.runs < 0 + ): + message = "phase warmup and runs must be nonnegative integers" + raise ValueError(message) + if phase.status == "not_applicable" and ( + phase.samples or phase.summary is not None or phase.observations + ): + message = "not_applicable phase cannot carry timing evidence" + raise ValueError(message) accepted: list[int] = [] for sample in phase.samples: if sample.accepted and (sample.error is not None or not sample.verified): @@ -2074,6 +2701,17 @@ def validate_report(report: RunReport) -> None: ): message = "phase summary must match accepted samples" raise ValueError(message) + for observation in phase.observations: + if ( + type(observation.ordinal) is not int + or observation.ordinal < 0 + or not observation.strategy + or type(observation.duration_ns) is not int + or observation.duration_ns <= 0 + or not observation.verified + ): + message = "phase observation requires verified positive evidence" + raise ValueError(message) ramp_kinds = {"none", "canonical", "custom"} terminal_statuses = {"refused", "cutoff", "failed"} attempt_statuses = {"completed", "not_attempted", *terminal_statuses} @@ -2150,6 +2788,262 @@ def validate_report(report: RunReport) -> None: ): message = "maximum_completed requires exact requested and observed 100x100x4" raise ValueError(message) + if report.run_id is not None: + _validate_executable_report(report) + if report.ramp_kind != "none" and report.environment is not None: + _validate_executable_ramp(report) + + +def _validate_executable_report(report: RunReport) -> None: + """Validate the stronger contract for a supervisor-owned scenario. + + >>> refused = RunReport( + ... Topology(1, 1, 1), status="refused", cleanup=CleanupReport( + ... True, processes_absent=True, socket_absent=True, + ... scratch_absent=True, + ... ), run_id="run-7", lane="control", mode="async", warmup=0, + ... runs=1, failed_phase="preflight", error="predictive refusal", + ... guard_decision=GuardDecision( + ... False, "predictive_refusal", "pid_reserve", 2, 1, True, + ... HostSnapshot(), + ... ), + ... ) + >>> _validate_executable_report(refused) + + Parameters + ---------- + report : RunReport + Single-scenario report carrying a non-null run identity. + + Returns + ------- + None + After exact metadata, phase, count, and cleanup checks pass. + + Raises + ------ + ValueError + If executable evidence is incomplete or contradictory. + """ + if not _is_terminal_safe_component(report.run_id): + message = "executable report requires a terminal-safe run_id" + raise ValueError(message) + if report.lane not in {lane.value for lane in EngineLane}: + message = "executable report requires a valid lane" + raise ValueError(message) + if report.mode not in {mode.value for mode in ExecutionMode}: + message = "executable report requires a valid mode" + raise ValueError(message) + if ( + type(report.warmup) is not int + or report.warmup < 0 + or type(report.runs) is not int + or report.runs <= 0 + ): + message = "executable report requires nonnegative warmup and positive runs" + raise ValueError(message) + if report.ramp_kind != "none": + message = "one executable scenario cannot also be a ramp report" + raise ValueError(message) + if report.status in {"completed", "refused", "failed", "cutoff"} and ( + report.cleanup.processes_absent is not True + or report.cleanup.socket_absent is not True + or report.cleanup.scratch_absent is not True + ): + message = "terminal executable report requires exact cleanup evidence" + raise ValueError(message) + for identity in report.processes: + if ( + not identity.role + or type(identity.pid) is not int + or identity.pid <= 0 + or type(identity.start_time) is not int + or identity.start_time <= 0 + ): + message = "executable report has an invalid process identity" + raise ValueError(message) + if report.status == "refused": + if ( + report.failed_phase != "preflight" + or not report.error + or report.observed_topology is not None + or report.phases + or report.processes + or report.scratch_path is not None + or report.socket_path is not None + or report.progress_path is not None + or report.guard_decision is None + or report.guard_decision.kind != "predictive_refusal" + or report.guard_decision.allowed + ): + message = "refused executable report contains live-run evidence" + raise ValueError(message) + return + if report.environment is None: + message = "attempted executable report requires environment evidence" + raise ValueError(message) + if not report.scratch_path or not report.socket_path or not report.progress_path: + message = "attempted executable report requires owned resource paths" + raise ValueError(message) + if report.status != "completed": + if not report.failed_phase or not report.error: + message = "terminal unsuccessful report requires phase and error" + raise ValueError(message) + return + if report.failed_phase is not None or report.error is not None: + message = "completed report cannot carry a terminal failure" + raise ValueError(message) + if report.observed_topology != report.requested_topology: + message = "completed report requires exact observed topology" + raise ValueError(message) + if tuple(phase.name for phase in report.phases) != _RUNNER_PHASES: + message = "completed report has an incomplete phase graph" + raise ValueError(message) + phases = {phase.name: phase for phase in report.phases} + setup = phases["setup"] + if ( + setup.status != "completed" + or setup.warmup != 0 + or setup.runs != 1 + or len(setup.samples) != 1 + or len(setup.observations) != 1 + or setup.summary is not None + ): + message = "setup must remain one unsummarized fresh-server observation" + raise ValueError(message) + stabilization = phases["stabilization"] + if ( + stabilization.status != "completed" + or stabilization.samples + or stabilization.summary is not None + or len(stabilization.observations) != 1 + or stabilization.observations[0].pane_count != report.requested_topology.panes + ): + message = "stabilization requires one exact untimed topology observation" + raise ValueError(message) + control_applicable = ( + report.lane == EngineLane.CONTROL.value + and report.mode == ExecutionMode.ASYNC.value + ) + control_phase = phases["wait.control-stream"] + if not control_applicable: + if control_phase.status != "not_applicable": + message = "control-stream must be not_applicable outside async control" + raise ValueError(message) + elif control_phase.status != "completed": + message = "async control report requires control-stream evidence" + raise ValueError(message) + repeatable_names = (*_RUNNER_REPEATABLE_PHASES,) + if control_applicable: + repeatable_names = ( + *repeatable_names[:2], + "wait.control-stream", + *repeatable_names[2:], + ) + for name in repeatable_names: + phase = phases[name] + if ( + phase.status != "completed" + or phase.warmup != report.warmup + or phase.runs != report.runs + or len(phase.samples) != report.runs + or len(phase.observations) != report.runs + or phase.summary is None + or phase.summary.get("count") != report.runs + ): + message = f"repeatable phase {name} has incomplete raw evidence" + raise ValueError(message) + samples_by_ordinal = {sample.ordinal: sample for sample in phase.samples} + for observation in phase.observations: + sample = samples_by_ordinal.get(observation.ordinal) + if ( + sample is None + or sample.strategy != name + or sample.duration_ns != observation.duration_ns + or not sample.accepted + or not sample.verified + ): + message = f"phase {name} observation does not match raw sample" + raise ValueError(message) + topology = report.requested_topology + for kind, expected in ( + ("sessions", topology.sessions), + ("windows", topology.windows), + ("panes", topology.panes), + ): + if any( + observation.row_count != expected + for observation in phases[f"enumeration.{kind}"].observations + ): + message = f"enumeration.{kind} row count differs from topology" + raise ValueError(message) + for strategy in ("serial", "batched"): + if any( + observation.pane_count != topology.panes + or observation.line_count is None + or observation.line_count <= 0 + or observation.byte_count is None + or observation.byte_count <= 0 + for observation in phases[f"capture.{strategy}"].observations + ): + message = f"capture.{strategy} count evidence is incomplete" + raise ValueError(message) + for phase in report.phases: + if ( + phase.name.startswith("search.") + and phase.status == "completed" + and any( + observation.matched_count != 1 + or observation.scanned_count is None + or observation.scanned_count <= 0 + for observation in phase.observations + ) + ): + message = f"{phase.name} search count evidence is incomplete" + raise ValueError(message) + + +def _validate_executable_ramp(report: RunReport) -> None: + """Require fresh per-shape identities and paths in a rendered ramp. + + >>> shape = Topology(1, 1, 1) + >>> ramp = RunReport( + ... shape, status="completed", cleanup=CleanupReport( + ... True, processes_absent=True, socket_absent=True, + ... scratch_absent=True, + ... ), ramp_kind="custom", requested_shapes=(shape,), + ... ramp=(RampStep(shape, "completed", run_id="run-1", + ... report_path="one.json", scratch_path="one", + ... socket_path="one/sock"),), + ... environment=EnvironmentReport("3.10", None, 1, 11, ("ramp",), None), + ... ) + >>> _validate_executable_ramp(ramp) + """ + if report.status in {"completed", "refused", "failed", "cutoff"} and ( + report.cleanup.processes_absent is not True + or report.cleanup.socket_absent is not True + or report.cleanup.scratch_absent is not True + ): + message = "terminal ramp requires exact cleanup evidence" + raise ValueError(message) + if ( + not report.requested_shapes + or report.requested_topology != report.requested_shapes[-1] + ): + message = "ramp requested topology must remain the declared final shape" + raise ValueError(message) + attempted = [step for step in report.ramp if step.status != "not_attempted"] + for attribute in ("run_id", "report_path"): + values = [getattr(step, attribute) for step in attempted] + if any(not value for value in values) or len(set(values)) != len(values): + message = f"ramp attempts require fresh unique {attribute} values" + raise ValueError(message) + resource_attempts = [step for step in attempted if step.status != "refused"] + for attribute in ("scratch_path", "socket_path"): + values = [getattr(step, attribute) for step in resource_attempts] + if any(not value for value in values) or len(set(values)) != len(values): + message = f"ramp attempts require fresh unique {attribute} values" + raise ValueError(message) def _forced_decision(decision: GuardDecision, force_extreme: bool) -> GuardDecision: @@ -2552,7 +3446,11 @@ def _stop_owned_process( f"{identity.role} pid {identity.pid} with start time " f"{identity.start_time} remains" ) - return CleanupReport(complete=not errors, errors=tuple(errors)) + return CleanupReport( + complete=not errors, + errors=tuple(errors), + processes_absent=not process_identity_matches(identity), + ) def _stream_name(ordinal: int, delayed_ordinal: int) -> str: @@ -3005,6 +3903,7 @@ def _prepare_context( socket_path: pathlib.Path | None, run_id: str, delayed_ordinal: int, + _process_identity_callback: cabc.Callable[[ProcessIdentity], None] | None = None, ) -> RunContext: """Create isolated host resources and an engine without starting setup timing. @@ -3034,6 +3933,8 @@ def _prepare_context( Safe identifier used in markers and tmux names. delayed_ordinal : int Unique pane assigned the delayed stream. + _process_identity_callback : collections.abc.Callable | None + Private worker hook invoked as exact process identities become known. Returns ------- @@ -3103,6 +4004,8 @@ def _prepare_context( _identity_out=fuzzer_identities, ) fuzzer_identity = _only_process_identity(fuzzer_identities) + if _process_identity_callback is not None: + _process_identity_callback(fuzzer_identity) server = Server(socket_path=resolved_socket, config_file=os.devnull) if mode is ExecutionMode.SYNC: engine: TmuxEngine | AsyncTmuxEngine @@ -3155,6 +4058,7 @@ def _prepare_context( setup_duration_ns=0, processes=(fuzzer_identity,), ambient_tmux_environment=ambient_tmux_environment, + process_identity_callback=_process_identity_callback, ) except BaseException as setup_error: cleanup_errors: list[str] = [] @@ -3198,6 +4102,7 @@ def setup_sync( socket_path: pathlib.Path | None = None, run_id: str = "run-0", delayed_ordinal: int = 0, + _process_identity_callback: cabc.Callable[[ProcessIdentity], None] | None = None, ) -> RunContext: """Build and exactly verify one synchronous live topology. @@ -3230,6 +4135,8 @@ def setup_sync( Marker and topology identity. delayed_ordinal : int Unique pane assigned the delayed stream. + _process_identity_callback : collections.abc.Callable | None + Private worker hook invoked as exact process identities become known. Returns ------- @@ -3248,6 +4155,7 @@ def setup_sync( from libtmux.experimental.engines import SubprocessEngine from libtmux.experimental.ops import ( BatchingPlanner, + BoundedPlanner, DisplayMessage, KillSession, ListClients, @@ -3267,6 +4175,7 @@ def setup_sync( socket_path=socket_path, run_id=run_id, delayed_ordinal=delayed_ordinal, + _process_identity_callback=_process_identity_callback, ) engine = t.cast("TmuxEngine", context.engine) keepalive = f"bench-{run_id}-keepalive" @@ -3292,6 +4201,15 @@ def setup_sync( run_id, delayed_ordinal=delayed_ordinal, ) + compiled = workspaces.compile() + setup_planner = BoundedPlanner( + BatchingPlanner(), frozenset(compiled.host_after) + ) + context.setup_metrics = _phase_metrics( + context, + operations=len(compiled.plan), + planner_steps=len(compiled.plan.explain(setup_planner)), + ) started_ns = time.perf_counter_ns() build_result = workspaces.build( engine, @@ -3312,7 +4230,10 @@ def setup_sync( server_pid_result = run(DisplayMessage(message="#{pid}"), engine) server_pid_result.raise_for_status() server_pid = int(server_pid_result.text) - context.processes = (*context.processes, _record_process("server", server_pid)) + server_identity = _record_process("server", server_pid) + context.processes = (*context.processes, server_identity) + if context.process_identity_callback is not None: + context.process_identity_callback(server_identity) verify_topology(context, snapshot_topology_sync(context)) except BaseException as setup_error: try: @@ -3339,6 +4260,7 @@ async def setup_async( socket_path: pathlib.Path | None = None, run_id: str = "run-0", delayed_ordinal: int = 0, + _process_identity_callback: cabc.Callable[[ProcessIdentity], None] | None = None, ) -> RunContext: """Build and exactly verify one asynchronous live topology. @@ -3369,6 +4291,8 @@ async def setup_async( Marker and topology identity. delayed_ordinal : int Unique pane assigned the delayed stream. + _process_identity_callback : collections.abc.Callable | None + Private worker hook invoked as exact process identities become known. Returns ------- @@ -3390,6 +4314,7 @@ async def setup_async( ) from libtmux.experimental.ops import ( BatchingPlanner, + BoundedPlanner, DisplayMessage, KillSession, ListClients, @@ -3409,6 +4334,7 @@ async def setup_async( socket_path=socket_path, run_id=run_id, delayed_ordinal=delayed_ordinal, + _process_identity_callback=_process_identity_callback, ) engine = t.cast("AsyncTmuxEngine", context.engine) keepalive = f"bench-{run_id}-keepalive" @@ -3440,8 +4366,17 @@ async def setup_async( run_id, delayed_ordinal=delayed_ordinal, ) - started_ns = time.perf_counter_ns() - build_result = ( + compiled = workspaces.compile() + setup_planner = BoundedPlanner( + BatchingPlanner(), frozenset(compiled.host_after) + ) + context.setup_metrics = _phase_metrics( + context, + operations=len(compiled.plan), + planner_steps=len(compiled.plan.explain(setup_planner)), + ) + started_ns = time.perf_counter_ns() + build_result = ( await workspaces.abuild( engine, preflight=False, @@ -3468,7 +4403,10 @@ async def setup_async( server_pid_result = await arun(DisplayMessage(message="#{pid}"), engine) server_pid_result.raise_for_status() server_pid = int(server_pid_result.text) - context.processes = (*context.processes, _record_process("server", server_pid)) + server_identity = _record_process("server", server_pid) + context.processes = (*context.processes, server_identity) + if context.process_identity_callback is not None: + context.process_identity_callback(server_identity) verify_topology(context, await snapshot_topology_async(context)) except BaseException as setup_error: try: @@ -3704,6 +4642,10 @@ def fail(detail: str) -> t.NoReturn: *(process for process in context.processes if process.role != "pane"), *pane_processes, ) + identity_callback = getattr(context, "process_identity_callback", None) + if identity_callback is not None: + for identity in pane_processes: + identity_callback(identity) context.topology_verified = True return snapshot @@ -6801,6 +7743,10 @@ async def run_repeatable_phase( seed: int, live_postcondition: cabc.Callable[[PhaseMeasurement], object], snapshot_resources: cabc.Callable[[], HostSnapshot] | None = None, + progress_callback: cabc.Callable[ + [str, str, int, PhaseMeasurement, RawSample | None], object + ] + | None = None, ) -> RepeatablePhaseResult: """Deterministically interleave strategies and retain accepted timed rows. @@ -6836,6 +7782,9 @@ async def run_repeatable_phase( Required independent sync or async live check run after typed validation. snapshot_resources : collections.abc.Callable[[], HostSnapshot] | None Injectable resource sampler; defaults to the live process/cgroup probe. + progress_callback : collections.abc.Callable | None + Private orchestration hook called after every accepted warmup or timed + invocation. Timed calls receive the newly retained raw sample. Returns ------- @@ -6875,18 +7824,30 @@ async def run_repeatable_phase( postcondition = await _await_if_needed(live_postcondition(measurement)) _require_live_postcondition(postcondition) resources_after = sampler() + raw_sample: RawSample | None = None if stage == "timed": - samples.append( - RawSample( - duration_ns=measurement.duration_ns, - accepted=True, - verified=True, - strategy=strategy, - ordinal=ordinal, - resources_before=resources_before, - resources_after=resources_after, + raw_sample = RawSample( + duration_ns=measurement.duration_ns, + accepted=True, + verified=True, + strategy=strategy, + ordinal=ordinal, + resources_before=resources_before, + resources_after=resources_after, + ) + samples.append(raw_sample) + if progress_callback is not None: + await _await_if_needed( + progress_callback( + stage, + strategy, + ordinal, + measurement, + raw_sample, ) ) + except RuntimeCutoffError: + raise except Exception as error: # noqa: BLE001 return RepeatablePhaseResult( samples=tuple(samples), @@ -7067,148 +8028,2635 @@ async def cleanup_run( os.environ.pop(name, None) else: os.environ[name] = value - return CleanupReport(complete=not errors, errors=tuple(errors)) + processes_absent = all( + not process_identity_matches(identity) for identity in context.processes + ) + socket_absent = not context.socket_path.exists() + scratch_absent = not context.scratch.exists() and ( + context.socket_root is None or not context.socket_root.exists() + ) + return CleanupReport( + complete=not errors, + errors=tuple(errors), + processes_absent=processes_absent, + socket_absent=socket_absent, + scratch_absent=scratch_absent, + ) -def parse_topology(shape: str) -> Topology: - """Parse a positive ``SxWxP`` topology string. +class RuntimeCutoffError(RuntimeError): + """A non-forceable live resource guard stopped the current run. - >>> parse_topology("2x3x4") - Topology(sessions=2, windows_per_session=3, panes_per_window=4) + Attributes + ---------- + decision : GuardDecision + Exact runtime guard observation that caused the cutoff. + + Examples + -------- + >>> decision = GuardDecision( + ... False, "runtime_cutoff", "watchdog", None, None, False, + ... HostSnapshot(), + ... ) + >>> RuntimeCutoffError(decision).decision.rule + 'watchdog' + """ + + def __init__(self, decision: GuardDecision) -> None: + """Retain one non-forceable runtime decision. + + >>> decision = GuardDecision( + ... False, "runtime_cutoff", "memory_floor", 1, 2, False, + ... HostSnapshot(), + ... ) + >>> str(RuntimeCutoffError(decision)) + 'runtime cutoff: memory_floor' + """ + self.decision = decision + super().__init__(f"runtime cutoff: {decision.rule or 'unknown'}") + + +class PhaseExecutionError(RuntimeError): + """A repeatable phase failed before producing all requested samples. + + Attributes + ---------- + phase : str + Exact cell that failed. + failure : RepeatablePhaseFailure + Stage, ordinal, and original error metadata. + + Examples + -------- + >>> failure = RepeatablePhaseFailure("timed", "cell", 1, "RuntimeError: no") + >>> PhaseExecutionError("cell", failure).phase + 'cell' + """ + + def __init__(self, phase: str, failure: RepeatablePhaseFailure) -> None: + """Retain one typed repeatable failure. + + >>> failure = RepeatablePhaseFailure("warmup", "cell", 0, "ValueError: x") + >>> str(PhaseExecutionError("cell", failure)) + 'cell warmup 0 failed: ValueError: x' + """ + self.phase = phase + self.failure = failure + super().__init__( + f"{phase} {failure.stage} {failure.ordinal} failed: {failure.error}" + ) + + +def _collect_environment( + *, + seed: int, + command_line: t.Sequence[str], + include_tmux: bool = True, +) -> EnvironmentReport: + """Collect descriptive version and checkout facts without contacting a server. + + >>> environment = _collect_environment(seed=11, command_line=("run",)) + >>> environment.seed, environment.command_line + (11, ('run',)) Parameters ---------- - shape : str - Topology in lower-case ``SxWxP`` notation. + seed : int + Deterministic phase-order seed. + command_line : collections.abc.Sequence[str] + Public command arguments represented by the artifact. + include_tmux : bool + Whether admission has occurred and invoking ``tmux -V`` is permitted. Returns ------- - Topology - Parsed topology with every dimension positive. - - Raises - ------ - ValueError - If the shape is malformed or has a nonpositive dimension. + EnvironmentReport + Local descriptive evidence with unavailable values retained as ``None``. """ - pieces = shape.split("x") - if len(pieces) != 3: - message = "topology must use SxWxP notation" - raise ValueError(message) + tmux_version: str | None = None + git_revision: str | None = None + if include_tmux: + try: + completed = subprocess.run( + ("tmux", "-V"), + check=False, + capture_output=True, + text=True, + timeout=2.0, + ) + if completed.returncode == 0: + tmux_version = completed.stdout.strip() or None + except (OSError, subprocess.TimeoutExpired): + pass try: - values = tuple(int(piece) for piece in pieces) - except ValueError as exc: - message = "topology must use SxWxP notation" - raise ValueError(message) from exc - if any(value <= 0 for value in values): - message = "topology dimensions must be positive" - raise ValueError(message) - return Topology(*values) + completed = subprocess.run( + ("git", "rev-parse", "HEAD"), + cwd=pathlib.Path(__file__).parents[1], + check=False, + capture_output=True, + text=True, + timeout=2.0, + ) + if completed.returncode == 0: + git_revision = completed.stdout.strip() or None + except (OSError, subprocess.TimeoutExpired): + pass + return EnvironmentReport( + python_version=platform.python_version(), + tmux_version=tmux_version, + cpu_count=os.cpu_count(), + seed=seed, + command_line=tuple(command_line), + git_revision=git_revision, + ) -def canonical_ramp() -> tuple[Topology, ...]: - """Return the specified progression ordered by expected pane pressure. +def _observation_for_measurement( + strategy: str, + ordinal: int, + measurement: PhaseMeasurement, +) -> PhaseObservation: + """Project one accepted typed measurement into stable JSON evidence. - >>> tuple(str(shape) for shape in canonical_ramp())[:3] - ('80x20x1', '100x20x1', '80x20x2') + >>> measurement = EnumerationResult( + ... 7, ExecutionMetrics(1, 1, 1, 1, 0), "sessions", 2, + ... ("$0", "$1"), "digest", True, + ... ) + >>> _observation_for_measurement("enumeration.sessions", 0, measurement).row_count + 2 + + Parameters + ---------- + strategy : str + Stable full phase name. + ordinal : int + Zero-based timed ordinal. + measurement : PhaseMeasurement + Verified typed Task 4 or Task 5 result. Returns ------- - tuple[Topology, ...] - Exact canonical progression ordered by expected live-pane pressure. - """ - return ( - Topology(80, 20, 1), - Topology(100, 20, 1), - Topology(80, 20, 2), - Topology(80, 50, 1), - Topology(80, 20, 4), - Topology(80, 100, 1), - Topology(100, 50, 2), - Topology(100, 100, 2), - Topology(100, 100, 4), + PhaseObservation + Counts sufficient for artifact validation without serializing operations. + """ + if isinstance(measurement, MutationResult): + return PhaseObservation( + ordinal, + strategy, + measurement.duration_ns, + metrics=measurement.metrics, + session_count=1, + window_count=len(measurement.window_ids), + pane_count=len(measurement.pane_ids), + target=measurement.session_id, + verified=measurement.verified, + ) + if isinstance(measurement, EnumerationResult): + return PhaseObservation( + ordinal, + strategy, + measurement.duration_ns, + metrics=measurement.metrics, + row_count=measurement.row_count, + verified=measurement.verified, + ) + if isinstance(measurement, CaptureResult): + return PhaseObservation( + ordinal, + strategy, + measurement.duration_ns, + metrics=measurement.metrics, + byte_count=measurement.byte_count, + line_count=measurement.line_count, + pane_count=len(measurement.captures), + verified=measurement.verified, + ) + if isinstance(measurement, SearchResult): + return PhaseObservation( + ordinal, + strategy, + measurement.duration_ns, + scanned_count=measurement.scanned_count, + matched_count=len(measurement.matched_ids), + target=measurement.target, + verified=measurement.verified, + ) + return PhaseObservation( + ordinal, + strategy, + measurement.duration_ns, + poll_count=measurement.poll_count, + frame_count=measurement.frame_count, + dropped_notification_delta=measurement.dropped_notification_delta, + configured_delay_ns=measurement.configured_delay_ns, + scheduling_lateness_ns=measurement.scheduling_lateness_ns, + detection_overhead_ns=measurement.detection_overhead_ns, + target=measurement.pane_id, + verified=measurement.verified, ) -def summarize_ns(samples: t.Sequence[int]) -> dict[str, int | float]: - """Return descriptive statistics for accepted integer-nanosecond samples. +def _replace_phase(report: RunReport, phase: PhaseReport) -> RunReport: + """Replace a named phase in place or append it while preserving order. - Percentiles use the nearest-rank index ``ceil(p * count) - 1``. + >>> topology = Topology(1, 1, 1) + >>> report = RunReport(topology, phases=(PhaseReport("a", topology, None),)) + >>> tuple(row.name for row in _replace_phase( + ... report, PhaseReport("b", topology, None) + ... ).phases) + ('a', 'b') + """ + phases = list(report.phases) + for index, existing in enumerate(phases): + if existing.name == phase.name: + phases[index] = phase + break + else: + phases.append(phase) + return dataclasses.replace(report, phases=tuple(phases)) - >>> summarize_ns((1, 2, 3, 4))["p90_ns"] - 4 + +def append_progress_event(path: pathlib.Path, event: ProgressEvent) -> None: + """Durably append one complete JSON line without rewriting prior events. + + >>> with tempfile.TemporaryDirectory() as directory: + ... path = pathlib.Path(directory) / "progress.jsonl" + ... append_progress_event(path, ProgressEvent("run-7", 0, "start", 1)) + ... json.loads(path.read_text(encoding="utf-8"))["sequence"] + 0 Parameters ---------- - samples : collections.abc.Sequence[int] - Accepted duration measurements in nanoseconds. + path : pathlib.Path + Append-only worker-to-supervisor progress stream. + event : ProgressEvent + Complete event with cumulative process identities. Returns ------- - dict[str, int | float] - Count, extrema, mean, median, and p90/p95/p99 values. - - Raises - ------ - ValueError - If no samples were accepted. + None + After bytes and the containing directory are synchronized. """ - if not samples: - message = "cannot summarize empty samples" + path.parent.mkdir(parents=True, exist_ok=True) + encoded = ( + json.dumps(_json_value(event), separators=(",", ":"), sort_keys=True) + "\n" + ).encode() + flags = os.O_APPEND | os.O_CREAT | os.O_WRONLY + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + descriptor = os.open(path, flags, 0o600) + try: + written = os.write(descriptor, encoded) + if written != len(encoded): + message = "short append to progress stream" + raise OSError(message) + os.fsync(descriptor) + finally: + os.close(descriptor) + + +def _progress_event_from_json(value: object) -> ProgressEvent: + """Decode and validate one complete progress line. + + >>> _progress_event_from_json({ + ... "schema_version": 1, "run_id": "run-7", "sequence": 0, + ... "checkpoint": "start", "monotonic_ns": 1, "processes": [], + ... }).checkpoint + 'start' + """ + row = _json_mapping(value, "progress event") + event = ProgressEvent( + run_id=t.cast(str, row.get("run_id")), + sequence=t.cast(int, row.get("sequence")), + checkpoint=t.cast(str, row.get("checkpoint")), + monotonic_ns=t.cast(int, row.get("monotonic_ns")), + processes=tuple(_identity_from_json(item) for item in row.get("processes", [])), + schema_version=t.cast(int, row.get("schema_version", 1)), + ) + if ( + event.schema_version != 1 + or not _is_terminal_safe_component(event.run_id) + or type(event.sequence) is not int + or event.sequence < 0 + or not event.checkpoint + or type(event.monotonic_ns) is not int + or event.monotonic_ns < 0 + ): + message = "invalid progress event" raise ValueError(message) - ordered = sorted(samples) + return event - count = len(ordered) - return { - "count": count, - "min_ns": ordered[0], - "mean_ns": statistics.mean(ordered), - "median_ns": statistics.median(ordered), - "p90_ns": ordered[math.ceil(0.90 * count) - 1], - "p95_ns": ordered[math.ceil(0.95 * count) - 1], - "p99_ns": ordered[math.ceil(0.99 * count) - 1], - "max_ns": ordered[-1], - } +@dataclasses.dataclass +class _WorkerRecorder: + """Own worker checkpoints and the append-only progress sequence. -def main(argv: t.Sequence[str] | None = None) -> int: - """Run the side-effect-free benchmark planning command. + Attributes + ---------- + report : RunReport + Current immutable report snapshot. + checkpoint_path : pathlib.Path + Worker-owned atomic checkpoint artifact. + progress_path : pathlib.Path + Append-only supervisor progress stream. + stall_after : str | None + Private test-harness checkpoint that deliberately stops progress. + sequence : int + Last published sequence number. + identities : list[ProcessIdentity] + Cumulative exact identities published with every event. - >>> import contextlib, io - >>> captured = io.StringIO() - >>> with contextlib.redirect_stdout(captured): - ... result = main(["plan", "--shape", "1x1x1"]) - >>> result + Examples + -------- + >>> with tempfile.TemporaryDirectory() as directory: + ... root = pathlib.Path(directory) + ... recorder = _WorkerRecorder( + ... RunReport(Topology(1, 1, 1)), root / "report.json", + ... root / "progress.jsonl", + ... ) + ... recorder.checkpoint("start") + ... recorder.sequence 0 + """ + + report: RunReport + checkpoint_path: pathlib.Path + progress_path: pathlib.Path + stall_after: str | None = None + sequence: int = -1 + identities: list[ProcessIdentity] = dataclasses.field(default_factory=list) + + def checkpoint(self, name: str) -> None: + """Publish an increasing event before its matching report checkpoint. + + >>> with tempfile.TemporaryDirectory() as directory: + ... root = pathlib.Path(directory) + ... recorder = _WorkerRecorder( + ... RunReport(Topology(1, 1, 1)), root / "report.json", + ... root / "progress.jsonl", + ... ) + ... recorder.checkpoint("one") + ... load_run_report(root / "report.json").progress_sequence + 0 + """ + self.sequence += 1 + self.report = dataclasses.replace( + self.report, + processes=tuple(self.identities), + progress_sequence=self.sequence, + ) + append_progress_event( + self.progress_path, + ProgressEvent( + run_id=t.cast(str, self.report.run_id), + sequence=self.sequence, + checkpoint=name, + monotonic_ns=time.monotonic_ns(), + processes=tuple(self.identities), + ), + ) + write_json_atomic(self.checkpoint_path, self.report) + if self.stall_after == name: + while True: + time.sleep(0.05) + + def record_identity(self, identity: ProcessIdentity) -> None: + """Publish one exact identity once, at the moment it becomes known. + + >>> recorder = _WorkerRecorder( + ... RunReport(Topology(1, 1, 1), run_id="run-7"), + ... pathlib.Path("unused"), pathlib.Path("unused-progress"), + ... ) + >>> recorder.identities.append(ProcessIdentity("worker", 2, 3)) + >>> len(recorder.identities) + 1 + """ + key = (identity.role, identity.pid, identity.start_time) + if key in {(item.role, item.pid, item.start_time) for item in self.identities}: + return + self.identities.append(identity) + self.checkpoint(f"identity.{identity.role}") + + +async def _worker_live_postcondition( + context: RunContext, + _measurement: PhaseMeasurement, + policy: ResourcePolicy, +) -> bool: + """Require live activity, owned processes, topology, and resource headroom. + + >>> async def invalid(): + ... context = types.SimpleNamespace( + ... fuzzer=types.SimpleNamespace(poll=lambda: 1), processes=(), + ... topology_verified=True, + ... ) + ... measurement = SearchResult( + ... 1, "snapshot", "sessions", 1, "$0", ("$0",), verified=True + ... ) + ... try: + ... await _worker_live_postcondition(context, measurement, ResourcePolicy()) + ... except RuntimeError as error: + ... return str(error) + >>> asyncio.run(invalid()) + 'fuzzer exited during measured phase' Parameters ---------- - argv : collections.abc.Sequence[str] | None - Explicit command arguments, or process arguments when omitted. + context : RunContext + Active worker-owned topology. + _measurement : PhaseMeasurement + Independently typed and verified phase result. + policy : ResourcePolicy + Runtime PID and memory reserve configuration. Returns ------- - int - Zero after the selected command completes. + bool + Exactly ``True`` after every independent live check passes. + """ + _current_activity_heartbeat(context, max_age_s=2.0) + snapshot = probe_host(ProcessReader()) + decision = check_runtime_guard( + snapshot, + policy=policy, + processes_alive=all( + process_identity_matches(identity) for identity in context.processes + ), + topology_verified=context.topology_verified, + watchdog_ok=True, + cleanup_complete=True, + ) + if not decision.allowed: + raise RuntimeCutoffError(decision) + return True - Raises - ------ - SystemExit - If command-line arguments are invalid. - ValueError - If the selected plan topology is malformed or nonpositive. - OSError - If the selected plan output cannot be written. + +def _completed_phase(phase: PhaseReport) -> PhaseReport: + """Summarize all accepted raw rows and mark one repeatable cell complete. + + >>> topology = Topology(1, 1, 1) + >>> sample = RawSample(3, True, verified=True, strategy="x", ordinal=0) + >>> _completed_phase(PhaseReport( + ... "x", topology, topology, (sample,), status="in_progress", runs=1 + ... )).summary["count"] + 1 """ - parser = argparse.ArgumentParser(description=__doc__) - commands = parser.add_subparsers(dest="command", required=True) - plan_parser = commands.add_parser("plan", help="inspect topology and host limits") - plan_parser.add_argument("--shape", required=True) - plan_parser.add_argument("--output", type=pathlib.Path) - plan_parser.add_argument("--force-extreme", action="store_true") - arguments = parser.parse_args(argv) - if arguments.command == "plan": - return run_plan(arguments.shape, arguments.output, arguments.force_extreme) + durations = tuple( + t.cast(int, sample.duration_ns) for sample in phase.samples if sample.accepted + ) + return dataclasses.replace( + phase, + status="completed", + summary=summarize_ns(durations), + ) + + +async def _run_worker_group( + recorder: _WorkerRecorder, + context: RunContext, + strategies: cabc.Mapping[str, cabc.Callable[[], object]], + *, + warmup: int, + runs: int, + seed: int, + policy: ResourcePolicy, + latest: dict[str, PhaseMeasurement], + fail_after: str | None, +) -> None: + """Run one deterministically interleaved family and checkpoint each call. + + >>> async def empty_group(): + ... try: + ... await _run_worker_group( + ... t.cast(_WorkerRecorder, None), t.cast(RunContext, None), {}, + ... warmup=0, runs=1, seed=1, policy=ResourcePolicy(), latest={}, + ... fail_after=None, + ... ) + ... except ValueError as error: + ... return str(error) + >>> asyncio.run(empty_group()) + 'worker phase group requires strategies' + + Parameters + ---------- + recorder : _WorkerRecorder + Atomic checkpoint and append-only progress owner. + context : RunContext + Active topology used by every strategy. + strategies : collections.abc.Mapping[str, collections.abc.Callable] + Full phase names mapped to typed sync or async callables. + warmup : int + Untimed calls per cell. + runs : int + Timed accepted calls per cell. + seed : int + Deterministic family-order seed. + policy : ResourcePolicy + Runtime guard thresholds. + latest : dict[str, PhaseMeasurement] + Destination retaining the latest typed result by cell. + fail_after : str | None + Private test-harness phase that raises after its final checkpoint. + + Returns + ------- + None + After all cells retain exactly ``runs`` accepted samples. + """ + if not strategies: + message = "worker phase group requires strategies" + raise ValueError(message) + topology = context.topology + for name in strategies: + recorder.report = _replace_phase( + recorder.report, + PhaseReport( + name=name, + requested_topology=topology, + observed_topology=topology, + status="in_progress", + warmup=warmup, + runs=runs, + ), + ) + recorder.checkpoint(f"{next(iter(strategies))}.started") + + async def on_progress( + stage: str, + strategy: str, + ordinal: int, + measurement: PhaseMeasurement, + sample: RawSample | None, + ) -> None: + latest[strategy] = measurement + phases = {phase.name: phase for phase in recorder.report.phases} + phase = phases[strategy] + if sample is not None: + phase = dataclasses.replace( + phase, + samples=(*phase.samples, sample), + observations=( + *phase.observations, + _observation_for_measurement(strategy, ordinal, measurement), + ), + ) + if ordinal == runs - 1: + phase = _completed_phase(phase) + recorder.report = _replace_phase(recorder.report, phase) + checkpoint = ( + strategy + if sample is not None and ordinal == runs - 1 + else f"{strategy}.{stage}.{ordinal}" + ) + recorder.checkpoint(checkpoint) + if checkpoint == strategy and fail_after == strategy: + message = f"injected phase failure after {strategy}" + raise RuntimeError(message) + + result = await run_repeatable_phase( + strategies, + warmup=warmup, + runs=runs, + seed=seed, + snapshot_resources=lambda: probe_host(ProcessReader()), + live_postcondition=lambda measurement: _worker_live_postcondition( + context, measurement, policy + ), + progress_callback=on_progress, + ) + if result.failure is not None: + phases = {phase.name: phase for phase in recorder.report.phases} + failed = dataclasses.replace( + phases[result.failure.strategy], status="failed", summary=None + ) + recorder.report = _replace_phase(recorder.report, failed) + recorder.checkpoint(f"{result.failure.strategy}.failed") + raise PhaseExecutionError(result.failure.strategy, result.failure) + + +def _position_target(ids: tuple[str, ...], position: str) -> str: + """Return the stable first, middle, or last concrete ID. + + >>> [_position_target(("a", "b", "c"), position) for position in _SEARCH_POSITIONS] + ['a', 'b', 'c'] + + Parameters + ---------- + ids : tuple[str, ...] + Nonempty stable concrete ID sequence. + position : str + One of ``first``, ``middle``, or ``last``. + + Returns + ------- + str + Concrete ID at the requested stable position. + """ + if not ids: + message = "search target sequence must be nonempty" + raise ValueError(message) + indices = {"first": 0, "middle": len(ids) // 2, "last": len(ids) - 1} + try: + return ids[indices[position]] + except KeyError as error: + message = f"unknown search position: {position}" + raise ValueError(message) from error + + +async def run_worker( + topology: Topology, + *, + lane: EngineLane, + mode: ExecutionMode, + runs: int, + warmup: int, + seed: int, + run_id: str, + scratch: pathlib.Path, + socket_path: pathlib.Path, + checkpoint_path: pathlib.Path, + progress_path: pathlib.Path, + guard_decision: GuardDecision, + original_guard_decision: GuardDecision, + policy: ResourcePolicy, + stall_after: str | None = None, + fail_after: str | None = None, + extra_identity: ProcessIdentity | None = None, +) -> RunReport: + """Execute one complete worker phase graph with cleanup in ``finally``. + + The private injection arguments are a subprocess-only test harness. Public + callers cannot select them through ``run`` or ``ramp`` help. + + >>> async def invalid_worker(): + ... try: + ... await run_worker( + ... Topology(1, 1, 1), lane=EngineLane.CONTROL, + ... mode=ExecutionMode.ASYNC, runs=0, warmup=0, seed=11, + ... run_id="run-7", scratch=pathlib.Path("unused"), + ... socket_path=pathlib.Path("unused/sock"), + ... checkpoint_path=pathlib.Path("unused.json"), + ... progress_path=pathlib.Path("unused.jsonl"), + ... guard_decision=GuardDecision( + ... True, "ok", None, None, None, False, HostSnapshot() + ... ), + ... original_guard_decision=GuardDecision( + ... True, "ok", None, None, None, False, HostSnapshot() + ... ), policy=ResourcePolicy(), + ... ) + ... except ValueError as error: + ... return str(error) + >>> asyncio.run(invalid_worker()) + 'worker runs must be positive and warmup nonnegative' + + Parameters + ---------- + topology : Topology + Exact requested hierarchy. + lane : EngineLane + Subprocess or control transport. + mode : ExecutionMode + Synchronous or asynchronous dispatch. + runs : int + Timed samples for every repeatable cell. + warmup : int + Untimed samples for every repeatable cell. + seed : int + Deterministic interleaving seed. + run_id : str + Terminal-safe owner identity. + scratch : pathlib.Path + New exclusive private run directory. + socket_path : pathlib.Path + Exact scratch-contained tmux socket. + checkpoint_path : pathlib.Path + Worker-owned atomic report checkpoint. + progress_path : pathlib.Path + Append-only supervisor progress stream. + guard_decision : GuardDecision + Effective predictive admission decision. + original_guard_decision : GuardDecision + Unmodified predictive admission evidence. + policy : ResourcePolicy + Runtime guard thresholds. + stall_after : str | None + Private test checkpoint that stops emitting progress. + fail_after : str | None + Private test phase that raises after completion. + extra_identity : ProcessIdentity | None + Private test identity published to prove PID-reuse safety. + + Returns + ------- + RunReport + Terminal worker candidate for supervisor validation and ownership. + """ + if type(runs) is not int or runs <= 0 or type(warmup) is not int or warmup < 0: + message = "worker runs must be positive and warmup nonnegative" + raise ValueError(message) + if not guard_decision.allowed: + message = "worker cannot start after predictive refusal" + raise ValueError(message) + environment = _collect_environment( + seed=seed, + command_line=( + "run", + "--shape", + str(topology), + "--lane", + lane.value, + "--mode", + mode.value, + "--runs", + str(runs), + "--warmup", + str(warmup), + ), + ) + initial = RunReport( + requested_topology=topology, + status="in_progress", + cleanup=CleanupReport(False), + guard_decision=guard_decision, + original_guard_decision=original_guard_decision, + run_id=run_id, + lane=lane.value, + mode=mode.value, + warmup=warmup, + runs=runs, + scratch_path=str(scratch), + socket_path=str(socket_path), + progress_path=str(progress_path), + environment=environment, + ) + recorder = _WorkerRecorder( + initial, + checkpoint_path, + progress_path, + stall_after=stall_after, + ) + recorder.identities.append(_record_process("worker", os.getpid())) + recorder.checkpoint("worker.started") + if extra_identity is not None: + recorder.record_identity(extra_identity) + context: RunContext | None = None + terminal_status: t.Literal["completed", "failed", "cutoff"] = "completed" + failed_phase: str | None = None + terminal_error: str | None = None + cleanup = CleanupReport(False) + latest: dict[str, PhaseMeasurement] = {} + + def fail_if_requested(phase: str) -> None: + if fail_after == phase: + message = f"injected phase failure after {phase}" + raise RuntimeError(message) + + try: + resources_before = probe_host(ProcessReader()) + panes_per_session = topology.windows_per_session * topology.panes_per_window + delayed_ordinal = min(topology.panes // 2, panes_per_session - 1) + if mode is ExecutionMode.SYNC: + context = setup_sync( + topology, + lane, + scratch, + socket_path=socket_path, + run_id=run_id, + delayed_ordinal=delayed_ordinal, + _process_identity_callback=recorder.record_identity, + ) + else: + context = await setup_async( + topology, + lane, + scratch, + socket_path=socket_path, + run_id=run_id, + delayed_ordinal=delayed_ordinal, + _process_identity_callback=recorder.record_identity, + ) + resources_after = probe_host(ProcessReader()) + setup_duration = max(1, context.setup_duration_ns) + setup_sample = RawSample( + setup_duration, + True, + verified=True, + strategy="setup", + ordinal=0, + resources_before=resources_before, + resources_after=resources_after, + ) + setup_observation = PhaseObservation( + ordinal=0, + strategy="setup", + duration_ns=setup_duration, + metrics=context.setup_metrics, + session_count=topology.sessions, + window_count=topology.windows, + pane_count=topology.panes, + ) + recorder.report = dataclasses.replace( + recorder.report, + observed_topology=topology, + ) + recorder.report = _replace_phase( + recorder.report, + PhaseReport( + "setup", + topology, + topology, + samples=(setup_sample,), + summary=None, + status="completed", + warmup=0, + runs=1, + observations=(setup_observation,), + ), + ) + recorder.checkpoint("setup") + fail_if_requested("setup") + + stabilization_started = time.perf_counter_ns() + release_activity_gate(context) + if mode is ExecutionMode.SYNC: + verify_activity_sync(context) + else: + await verify_activity_async(context) + stabilization_duration = max(1, time.perf_counter_ns() - stabilization_started) + recorder.report = _replace_phase( + recorder.report, + PhaseReport( + "stabilization", + topology, + topology, + status="completed", + observations=( + PhaseObservation( + 0, + "stabilization", + stabilization_duration, + session_count=topology.sessions, + window_count=topology.windows, + pane_count=len(context.activity_pane_ids), + ), + ), + ), + ) + recorder.checkpoint("stabilization") + fail_if_requested("stabilization") + + mutation_counter = 0 + + def mutation_sync_call() -> MutationResult: + nonlocal mutation_counter + mutation_counter += 1 + return mutate_sync(context, generation=mutation_counter) + + async def mutation_async_call() -> MutationResult: + nonlocal mutation_counter + mutation_counter += 1 + return await mutate_async(context, generation=mutation_counter) + + await _run_worker_group( + recorder, + context, + { + "mutation.bulk": ( + mutation_sync_call + if mode is ExecutionMode.SYNC + else mutation_async_call + ) + }, + warmup=warmup, + runs=runs, + seed=seed, + policy=policy, + latest=latest, + fail_after=fail_after, + ) + + wait_counter = 0 + + def next_request_id(strategy: str) -> str: + nonlocal wait_counter + wait_counter += 1 + compact = strategy.removeprefix("wait.").replace("-", "_") + return f"{compact}-{wait_counter:04d}" + + def wait_sync_call() -> WaitResult: + return wait_capture_poll_sync( + context, + request_id=next_request_id("wait.capture-poll"), + ) + + async def wait_async_call() -> WaitResult: + return await wait_capture_poll_async( + context, + request_id=next_request_id("wait.capture-poll"), + ) + + async def wait_control_call() -> WaitResult: + return await wait_control_stream( + context, + request_id=next_request_id("wait.control-stream"), + ) + + wait_strategies: dict[str, cabc.Callable[[], object]] = { + "wait.capture-poll": ( + wait_sync_call if mode is ExecutionMode.SYNC else wait_async_call + ) + } + control_applicable = lane is EngineLane.CONTROL and mode is ExecutionMode.ASYNC + if control_applicable: + wait_strategies["wait.control-stream"] = wait_control_call + await _run_worker_group( + recorder, + context, + wait_strategies, + warmup=warmup, + runs=runs, + seed=seed + 1, + policy=policy, + latest=latest, + fail_after=fail_after, + ) + if not control_applicable: + recorder.report = _replace_phase( + recorder.report, + PhaseReport( + "wait.control-stream", + topology, + topology, + status="not_applicable", + warmup=warmup, + runs=runs, + ), + ) + recorder.checkpoint("wait.control-stream") + fail_if_requested("wait.control-stream") + + enumeration_strategies: dict[str, cabc.Callable[[], object]] = {} + for kind in _ENUMERATION_KINDS: + phase_name = f"enumeration.{kind}" + enumeration_kind = t.cast("EnumerationKind", kind) + if mode is ExecutionMode.SYNC: + enumeration_strategies[phase_name] = functools.partial( + enumerate_sync, + context, + kind=enumeration_kind, + ) + else: + enumeration_strategies[phase_name] = functools.partial( + enumerate_async, + context, + kind=enumeration_kind, + ) + await _run_worker_group( + recorder, + context, + enumeration_strategies, + warmup=warmup, + runs=runs, + seed=seed + 2, + policy=policy, + latest=latest, + fail_after=fail_after, + ) + + capture_strategies: dict[str, cabc.Callable[[], object]] = {} + for strategy in ("serial", "batched"): + phase_name = f"capture.{strategy}" + if mode is ExecutionMode.SYNC: + capture_strategies[phase_name] = functools.partial( + capture_all_sync, + context, + strategy=strategy, + ) + else: + capture_strategies[phase_name] = functools.partial( + capture_all_async, + context, + strategy=strategy, + ) + await _run_worker_group( + recorder, + context, + capture_strategies, + warmup=warmup, + runs=runs, + seed=seed + 3, + policy=policy, + latest=latest, + fail_after=fail_after, + ) + + from libtmux._internal.query_list import QueryList + + snapshot = ( + snapshot_topology_sync(context) + if mode is ExecutionMode.SYNC + else await snapshot_topology_async(context) + ) + snapshot_rows = { + "sessions": QueryList(snapshot.sessions), + "windows": QueryList(snapshot.windows), + "panes": QueryList(snapshot.panes), + } + ids_by_kind = { + "sessions": context.session_ids, + "windows": context.window_ids, + "panes": context.pane_ids, + } + search_strategies: dict[str, cabc.Callable[[], object]] = {} + for family in _SEARCH_FAMILIES: + for kind in _ENUMERATION_KINDS: + for position in _SEARCH_POSITIONS: + phase_name = f"search.{family}.{kind}.{position}" + target = _position_target(ids_by_kind[kind], position) + search_kind = t.cast("EnumerationKind", kind) + if family == "classic": + search_strategies[phase_name] = functools.partial( + search_server_side, + context, + kind=search_kind, + target=target, + ) + elif family == "snapshot": + search_strategies[phase_name] = functools.partial( + search_snapshot, + snapshot_rows[kind], + kind=search_kind, + target=target, + ) + else: + search_strategies[phase_name] = functools.partial( + search_end_to_end, + context, + kind=search_kind, + target=target, + ) + capture = t.cast(CaptureResult, latest["capture.batched"]) + wait_name = "wait.control-stream" if control_applicable else "wait.capture-poll" + wait_result = t.cast(WaitResult, latest[wait_name]) + search_strategies["search.contents"] = lambda: search_contents( + capture, + token=wait_result.token, + expected_pane_id=context.delayed_pane_id, + ) + await _run_worker_group( + recorder, + context, + search_strategies, + warmup=warmup, + runs=runs, + seed=seed + 4, + policy=policy, + latest=latest, + fail_after=fail_after, + ) + + final_snapshot = ( + snapshot_topology_sync(context) + if mode is ExecutionMode.SYNC + else await snapshot_topology_async(context) + ) + verify_topology(context, final_snapshot) + recorder.checkpoint("verification") + fail_if_requested("verification") + except asyncio.CancelledError as error: + terminal_status = "cutoff" + failed_phase = "cancellation" + terminal_error = f"CancelledError: {error}" + except RuntimeCutoffError as error: + terminal_status = "cutoff" + failed_phase = error.decision.rule or "runtime_guard" + terminal_error = str(error) + recorder.report = dataclasses.replace( + recorder.report, guard_decision=error.decision + ) + except PhaseExecutionError as error: + terminal_status = "failed" + failed_phase = error.phase + terminal_error = str(error) + except BaseException as error: # noqa: BLE001 + terminal_status = "failed" + failed_phase = failed_phase or "setup" + terminal_error = f"{type(error).__name__}: {error}" + finally: + if context is not None: + try: + cleanup = await cleanup_run(context) + except BaseException as error: # noqa: BLE001 + cleanup = CleanupReport( + False, + (f"cleanup raised {type(error).__name__}: {error}",), + processes_absent=False, + socket_absent=not socket_path.exists(), + scratch_absent=not scratch.exists(), + ) + else: + owned = tuple( + identity + for identity in recorder.identities + if identity.role != "worker" and identity is not extra_identity + ) + cleanup = CleanupReport( + complete=( + all(not process_identity_matches(identity) for identity in owned) + and not socket_path.exists() + and not scratch.exists() + ), + errors=(), + processes_absent=all( + not process_identity_matches(identity) for identity in owned + ), + socket_absent=not socket_path.exists(), + scratch_absent=not scratch.exists(), + ) + if not cleanup.complete: + terminal_status = "failed" + failed_phase = failed_phase or "cleanup" + cleanup_detail = "; ".join(cleanup.errors) or "cleanup incomplete" + terminal_error = ( + f"{terminal_error}; {cleanup_detail}" + if terminal_error + else cleanup_detail + ) + recorder.report = dataclasses.replace( + recorder.report, + status=terminal_status, + cleanup=cleanup, + maximum_completed=( + terminal_status == "completed" + and topology == Topology(100, 100, 4) + and recorder.report.observed_topology == topology + ), + failed_phase=failed_phase, + error=terminal_error, + ) + recorder.checkpoint("cleanup") + return recorder.report + + +def _read_progress_chunk( + path: pathlib.Path, + offset: int, + remainder: str, +) -> tuple[tuple[ProgressEvent, ...], int, str]: + """Read only newly appended complete JSONL records. + + >>> with tempfile.TemporaryDirectory() as directory: + ... path = pathlib.Path(directory) / "events.jsonl" + ... append_progress_event(path, ProgressEvent("run-7", 0, "start", 1)) + ... events, offset, remainder = _read_progress_chunk(path, 0, "") + ... (events[0].sequence, offset > 0, remainder) + (0, True, '') + + Parameters + ---------- + path : pathlib.Path + Append-only progress stream. + offset : int + Previously consumed byte offset. + remainder : str + Incomplete decoded line retained from the prior read. + + Returns + ------- + tuple[tuple[ProgressEvent, ...], int, str] + Complete events, new byte offset, and any incomplete final line. + """ + try: + with path.open("r", encoding="utf-8") as stream: + stream.seek(offset) + chunk = stream.read() + new_offset = stream.tell() + except FileNotFoundError: + return (), offset, remainder + combined = remainder + chunk + pieces = combined.split("\n") + trailing = pieces.pop() + events = tuple( + _progress_event_from_json(json.loads(line)) for line in pieces if line + ) + return events, new_offset, trailing + + +def _wait_identity_absence( + identities: t.Iterable[ProcessIdentity], + *, + timeout_s: float, +) -> tuple[ProcessIdentity, ...]: + """Wait boundedly for exact identities without using process-name scans. + + >>> _wait_identity_absence((), timeout_s=0.01) + () + """ + deadline = time.monotonic() + timeout_s + identities_tuple = tuple(identities) + while True: + survivors = tuple( + identity + for identity in identities_tuple + if process_identity_matches(identity) + ) + if not survivors or time.monotonic() >= deadline: + return survivors + time.sleep(0.02) + + +def _signal_worker_group( + worker: subprocess.Popen[bytes], + identity: ProcessIdentity, + signal_number: signal.Signals, +) -> None: + """Signal an isolated worker group only while its leader identity matches. + + >>> finished = subprocess.Popen((sys.executable, "-c", "pass")) + >>> finished.wait(timeout=1) + 0 + >>> _signal_worker_group( + ... finished, ProcessIdentity("worker", finished.pid, -1), signal.SIGTERM + ... ) + """ + worker.poll() + if worker.returncode is not None or not process_identity_matches(identity): + return + with contextlib.suppress(ProcessLookupError): + os.killpg(identity.pid, signal_number) + + +def _remove_supervised_scratch(path: pathlib.Path) -> tuple[str, ...]: + """Remove one exact private directory without following replacement links. + + >>> with tempfile.TemporaryDirectory() as directory: + ... path = pathlib.Path(directory) / "run" + ... _ = _acquire_private_directory(path) + ... _remove_supervised_scratch(path) + () + """ + if not path.exists() and not path.is_symlink(): + return () + try: + status = path.lstat() + if stat.S_ISLNK(status.st_mode) or not stat.S_ISDIR(status.st_mode): + return (f"scratch is not an owned directory: {path}",) + if status.st_uid != os.getuid() or stat.S_IMODE(status.st_mode) != 0o700: + return (f"scratch ownership or mode changed: {path}",) + shutil.rmtree(path) + except OSError as error: + return (f"scratch removal: {type(error).__name__}: {error}",) + return () if not path.exists() else (f"scratch remains: {path}",) + + +def _recover_supervised_run( + worker: subprocess.Popen[bytes], + worker_identity: ProcessIdentity, + identities: tuple[ProcessIdentity, ...], + *, + scratch: pathlib.Path, + socket_path: pathlib.Path, + grace_s: float, +) -> CleanupReport: + """Boundedly stop exact worker-owned processes and verify filesystem absence. + + >>> finished = subprocess.Popen((sys.executable, "-c", "pass")) + >>> finished.wait(timeout=1) + 0 + >>> with tempfile.TemporaryDirectory() as directory: + ... scratch = pathlib.Path(directory) / "absent" + ... report = _recover_supervised_run( + ... finished, ProcessIdentity("worker", finished.pid, -1), (), + ... scratch=scratch, socket_path=scratch / "sock", grace_s=0.01, + ... ) + ... report.complete + True + + Parameters + ---------- + worker : subprocess.Popen[bytes] + Direct child launched in its own session. + worker_identity : ProcessIdentity + Exact group-leader identity captured immediately after spawn. + identities : tuple[ProcessIdentity, ...] + Cumulative progress identities, possibly including already absent PIDs. + scratch : pathlib.Path + Exact private worker directory. + socket_path : pathlib.Path + Exact isolated tmux socket. + grace_s : float + Bounded graceful and escalated wait interval. + + Returns + ------- + CleanupReport + Exact identity, socket, and scratch absence evidence. + """ + if grace_s <= 0: + message = "supervisor cleanup grace must be positive" + raise ValueError(message) + errors: list[str] = [] + _signal_worker_group(worker, worker_identity, signal.SIGTERM) + try: + worker.wait(timeout=grace_s) + except subprocess.TimeoutExpired: + _signal_worker_group(worker, worker_identity, signal.SIGKILL) + try: + worker.wait(timeout=grace_s) + except subprocess.TimeoutExpired: + errors.append("worker process group did not exit") + unique: dict[tuple[int, int], ProcessIdentity] = {} + for identity in (worker_identity, *identities): + unique[(identity.pid, identity.start_time)] = identity + owned = tuple(unique.values()) + survivors = _wait_identity_absence(owned, timeout_s=grace_s) + for signal_number in (signal.SIGTERM, signal.SIGKILL): + if not survivors: + break + for identity in survivors: + if not process_identity_matches(identity): + continue + try: + os.kill(identity.pid, signal_number) + except ProcessLookupError: + continue + except OSError as error: + errors.append( + f"{identity.role} pid {identity.pid} signal {signal_number}: " + f"{type(error).__name__}: {error}" + ) + survivors = _wait_identity_absence(owned, timeout_s=grace_s) + errors.extend( + f"{identity.role} pid {identity.pid} with start time " + f"{identity.start_time} remains" + for identity in survivors + if process_identity_matches(identity) + ) + errors.extend(_remove_supervised_scratch(scratch)) + processes_absent = not any(process_identity_matches(identity) for identity in owned) + socket_absent = not socket_path.exists() + scratch_absent = not scratch.exists() + if not socket_absent: + errors.append(f"socket remains: {socket_path}") + return CleanupReport( + complete=(not errors and processes_absent and socket_absent and scratch_absent), + errors=tuple(errors), + processes_absent=processes_absent, + socket_absent=socket_absent, + scratch_absent=scratch_absent, + ) + + +def _merge_identities( + *groups: t.Iterable[ProcessIdentity], +) -> tuple[ProcessIdentity, ...]: + """Deduplicate identities without collapsing different start times. + + >>> one = ProcessIdentity("worker", 2, 3) + >>> _merge_identities((one,), (one,)) + (ProcessIdentity(role='worker', pid=2, start_time=3),) + """ + merged: dict[tuple[str, int, int], ProcessIdentity] = {} + for group in groups: + for identity in group: + merged[(identity.role, identity.pid, identity.start_time)] = identity + return tuple(merged.values()) + + +def _validate_progress_run_id(event: ProgressEvent, run_id: str) -> None: + """Reject progress from any other supervised run. + + >>> _validate_progress_run_id(ProgressEvent("run", 0, "started", 1), "run") + >>> _validate_progress_run_id(ProgressEvent("other", 0, "started", 1), "run") + Traceback (most recent call last): + ... + RuntimeError: progress event run identity mismatch + """ + if event.run_id != run_id: + message = "progress event run identity mismatch" + raise RuntimeError(message) + + +def _accept_progress_events( + events: t.Iterable[ProgressEvent], + *, + run_id: str, + highest_sequence: int, + identities: tuple[ProcessIdentity, ...], +) -> tuple[int, tuple[ProcessIdentity, ...], bool]: + """Accept only events that advance one supervised run's sequence. + + >>> event = ProgressEvent("run-7", 1, "setup", 10) + >>> _accept_progress_events( + ... (event,), run_id="run-7", highest_sequence=1, identities=() + ... ) + (1, (), False) + + Parameters + ---------- + events : collections.abc.Iterable[ProgressEvent] + Newly appended complete progress rows. + run_id : str + Exact supervised run identity. + highest_sequence : int + Highest sequence previously accepted. + identities : tuple[ProcessIdentity, ...] + Cumulative identities from previously accepted rows. + + Returns + ------- + tuple[int, tuple[ProcessIdentity, ...], bool] + New high-water sequence, identities, and whether progress increased. + + Raises + ------ + RuntimeError + If any event belongs to another run. + """ + advanced = False + for event in events: + _validate_progress_run_id(event, run_id) + if event.sequence <= highest_sequence: + continue + highest_sequence = event.sequence + identities = _merge_identities(identities, event.processes) + advanced = True + return highest_sequence, identities, advanced + + +def supervise_worker( + topology: Topology, + *, + lane: EngineLane, + mode: ExecutionMode, + runs: int, + warmup: int, + seed: int, + run_id: str, + scratch: pathlib.Path, + socket_path: pathlib.Path, + output: pathlib.Path, + markdown_output: pathlib.Path, + guard_decision: GuardDecision, + original_guard_decision: GuardDecision, + policy: ResourcePolicy, + watchdog_s: float, + cleanup_grace_s: float, + _test_stall_after: str | None = None, + _test_fail_after: str | None = None, + _test_extra_identity: ProcessIdentity | None = None, +) -> RunReport: + """Launch and supervise the hidden worker with a sequence-based watchdog. + + The ``_test_*`` arguments are private CLI injection points used only by the + benchmark's subprocess recovery tests. + + >>> try: + ... supervise_worker( + ... Topology(1, 1, 1), lane=EngineLane.CONTROL, + ... mode=ExecutionMode.ASYNC, runs=1, warmup=0, seed=11, + ... run_id="run-7", scratch=pathlib.Path("scratch"), + ... socket_path=pathlib.Path("scratch/sock"), + ... output=pathlib.Path("report.json"), + ... markdown_output=pathlib.Path("report.md"), + ... guard_decision=GuardDecision( + ... True, "ok", None, None, None, False, HostSnapshot() + ... ), + ... original_guard_decision=GuardDecision( + ... True, "ok", None, None, None, False, HostSnapshot() + ... ), policy=ResourcePolicy(), watchdog_s=0, + ... cleanup_grace_s=1, + ... ) + ... except ValueError as error: + ... print(error) + supervisor watchdog must be positive + + Returns + ------- + RunReport + Supervisor-owned terminal report. + """ + if watchdog_s <= 0: + message = "supervisor watchdog must be positive" + raise ValueError(message) + if cleanup_grace_s <= 0: + message = "supervisor cleanup grace must be positive" + raise ValueError(message) + progress_path = output.with_name(f"{output.stem}.{run_id}.progress.jsonl") + checkpoint_path = output.with_name(f".{output.name}.{run_id}.worker.json") + admission_path = output.with_name(f".{output.name}.{run_id}.admission.json") + write_json_atomic( + admission_path, + { + "guard_decision": guard_decision, + "original_guard_decision": original_guard_decision, + }, + ) + command = [ + sys.executable, + str(pathlib.Path(__file__)), + "_worker", + "--shape", + str(topology), + "--lane", + lane.value, + "--mode", + mode.value, + "--runs", + str(runs), + "--warmup", + str(warmup), + "--seed", + str(seed), + "--run-id", + run_id, + "--scratch", + str(scratch), + "--socket-path", + str(socket_path), + "--checkpoint", + str(checkpoint_path), + "--progress", + str(progress_path), + "--admission", + str(admission_path), + "--pid-reserve", + str(policy.pid_reserve) if policy.pid_reserve is not None else "dynamic", + "--memory-floor-bytes", + ( + str(policy.memory_floor_bytes) + if policy.memory_floor_bytes is not None + else "dynamic" + ), + ] + if _test_stall_after is not None: + command.extend(("--_test-stall-after", _test_stall_after)) + if _test_fail_after is not None: + command.extend(("--_test-fail-after", _test_fail_after)) + if _test_extra_identity is not None: + serialized_identity = ":".join( + ( + _test_extra_identity.role, + str(_test_extra_identity.pid), + str(_test_extra_identity.start_time), + ) + ) + command.extend( + ( + "--_test-extra-identity", + serialized_identity, + ) + ) + environment = os.environ.copy() + environment.pop("TMUX", None) + environment.pop("TMUX_PANE", None) + worker = subprocess.Popen( + command, + cwd=pathlib.Path(__file__).parents[1], + env=environment, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + start_new_session=True, + ) + worker_identity = _record_process("worker", worker.pid) + identities: tuple[ProcessIdentity, ...] = (worker_identity,) + highest_sequence = -1 + offset = 0 + remainder = "" + deadline = time.monotonic() + watchdog_s + supervisor_status: t.Literal["failed", "cutoff"] | None = None + failed_phase: str | None = None + terminal_error: str | None = None + try: + while worker.poll() is None: + events, offset, remainder = _read_progress_chunk( + progress_path, offset, remainder + ) + highest_sequence, identities, advanced = _accept_progress_events( + events, + run_id=run_id, + highest_sequence=highest_sequence, + identities=identities, + ) + if advanced: + deadline = time.monotonic() + watchdog_s + if time.monotonic() >= deadline: + supervisor_status = "cutoff" + failed_phase = "watchdog" + terminal_error = f"progress watchdog expired after {watchdog_s} seconds" + break + time.sleep(0.02) + except KeyboardInterrupt: + supervisor_status = "cutoff" + failed_phase = "cancellation" + terminal_error = "KeyboardInterrupt: supervisor interrupted" + except BaseException as error: # noqa: BLE001 + supervisor_status = "failed" + failed_phase = "supervisor" + terminal_error = f"{type(error).__name__}: {error}" + + if supervisor_status is not None: + cleanup = _recover_supervised_run( + worker, + worker_identity, + identities, + scratch=scratch, + socket_path=socket_path, + grace_s=cleanup_grace_s, + ) + else: + worker.wait() + events, offset, remainder = _read_progress_chunk( + progress_path, offset, remainder + ) + highest_sequence, identities, _advanced = _accept_progress_events( + events, + run_id=run_id, + highest_sequence=highest_sequence, + identities=identities, + ) + cleanup = CleanupReport( + complete=( + not any(process_identity_matches(item) for item in identities) + and not socket_path.exists() + and not scratch.exists() + ), + processes_absent=not any( + process_identity_matches(item) for item in identities + ), + socket_absent=not socket_path.exists(), + scratch_absent=not scratch.exists(), + ) + if not cleanup.complete: + cleanup = _recover_supervised_run( + worker, + worker_identity, + identities, + scratch=scratch, + socket_path=socket_path, + grace_s=cleanup_grace_s, + ) + + try: + candidate = load_run_report(checkpoint_path) + except (OSError, ValueError): + candidate = RunReport( + topology, + status="in_progress", + cleanup=CleanupReport(False), + guard_decision=guard_decision, + original_guard_decision=original_guard_decision, + run_id=run_id, + lane=lane.value, + mode=mode.value, + warmup=warmup, + runs=runs, + scratch_path=str(scratch), + socket_path=str(socket_path), + progress_path=str(progress_path), + environment=_collect_environment( + seed=seed, command_line=("run", "--shape", str(topology)) + ), + ) + if supervisor_status is None: + if worker.returncode == 0 and candidate.status == "completed": + final_status: t.Literal["completed", "failed", "cutoff"] = "completed" + elif candidate.status == "cutoff": + final_status = "cutoff" + else: + final_status = "failed" + if final_status != "completed": + failed_phase = candidate.failed_phase or "worker" + terminal_error = candidate.error or f"worker exited {worker.returncode}" + else: + final_status = supervisor_status + if not cleanup.complete: + final_status = "failed" + failed_phase = "cleanup" + detail = "; ".join(cleanup.errors) or "cleanup verification failed" + terminal_error = f"{terminal_error}; {detail}" if terminal_error else detail + final_report = dataclasses.replace( + candidate, + status=final_status, + cleanup=cleanup, + maximum_completed=( + final_status == "completed" + and topology == Topology(100, 100, 4) + and candidate.observed_topology == topology + ), + failed_phase=(None if final_status == "completed" else failed_phase), + error=(None if final_status == "completed" else terminal_error), + processes=identities, + progress_path=str(progress_path), + progress_sequence=highest_sequence, + guard_decision=( + candidate.guard_decision + if candidate.guard_decision is not None + else guard_decision + ), + original_guard_decision=original_guard_decision, + ) + write_json_atomic(output, final_report) + validate_report(final_report) + render_markdown_summary(output, markdown_output) + admission_path.unlink(missing_ok=True) + return final_report + + +def _write_text_atomic(path: pathlib.Path, text: str) -> None: + r"""Atomically and durably replace one UTF-8 text artifact. + + >>> with tempfile.TemporaryDirectory() as directory: + ... path = pathlib.Path(directory) / "summary.md" + ... _write_text_atomic(path, "summary\n") + ... path.read_text(encoding="utf-8") + 'summary\n' + """ + path.parent.mkdir(parents=True, exist_ok=True) + temporary: pathlib.Path | None = None + try: + with tempfile.NamedTemporaryFile( + mode="w", + encoding="utf-8", + dir=path.parent, + prefix=f".{path.name}.", + suffix=".tmp", + delete=False, + ) as stream: + temporary = pathlib.Path(stream.name) + stream.write(text) + stream.flush() + os.fsync(stream.fileno()) + temporary.replace(path) + temporary = None + descriptor = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY) + try: + os.fsync(descriptor) + finally: + os.close(descriptor) + finally: + if temporary is not None: + temporary.unlink(missing_ok=True) + + +def _format_ns(value: float) -> str: + """Render integer nanoseconds as compact descriptive milliseconds. + + >>> _format_ns(1_500_000) + '1.500 ms' + """ + return f"{float(value) / 1_000_000:.3f} ms" + + +def render_markdown_summary( + report_path: pathlib.Path, + output_path: pathlib.Path | None = None, +) -> str: + """Render only a validated JSON artifact as local descriptive evidence. + + >>> with tempfile.TemporaryDirectory() as directory: + ... root = pathlib.Path(directory) + ... path = root / "report.json" + ... report = RunReport( + ... Topology(1, 1, 1), status="refused", + ... cleanup=CleanupReport( + ... True, processes_absent=True, socket_absent=True, + ... scratch_absent=True, + ... ), run_id="run-7", lane="control", mode="async", warmup=0, + ... runs=1, failed_phase="preflight", error="predictive refusal", + ... guard_decision=GuardDecision( + ... False, "predictive_refusal", "pid_reserve", 2, 1, True, + ... HostSnapshot(), + ... ), + ... ) + ... write_json_atomic(path, report) + ... "Local descriptive evidence" in render_markdown_summary(path) + True + + Parameters + ---------- + report_path : pathlib.Path + Complete machine-readable JSON artifact. + output_path : pathlib.Path | None + Optional Markdown destination. + + Returns + ------- + str + Markdown summary explicitly limited to local descriptive evidence. + + Raises + ------ + ValueError + If JSON or its recomputed report contract is invalid. + """ + report = load_run_report(report_path) + validate_report(report) + if report.status == "in_progress": + message = "cannot render an in-progress report" + raise ValueError(message) + lines = [ + "# Active orchestration benchmark", + "", + ( + "> Local descriptive evidence only; these timings are not causal or " + "machine-independent claims." + ), + "", + f"Status: `{report.status}`", + "", + f"Requested topology: `{report.requested_topology}`", + "", + ] + if report.observed_topology is not None: + lines.extend((f"Observed topology: `{report.observed_topology}`", "")) + if report.lane is not None and report.mode is not None: + lines.extend((f"Lane: `{report.lane}/{report.mode}`", "")) + if report.error is not None: + lines.extend((f"Terminal reason: {report.error}", "")) + if report.ramp: + lines.extend( + ( + "## Ramp attempts", + "", + "| Shape | Status | Reason |", + "| --- | --- | --- |", + ) + ) + lines.extend( + f"| `{step.shape}` | `{step.status}` | {step.reason or ''} |" + for step in report.ramp + ) + lines.append("") + if report.phases: + lines.extend( + ( + "## Phase timings", + "", + "| Phase | Status | Samples | Median | p95 |", + "| --- | --- | ---: | ---: | ---: |", + ) + ) + for phase in report.phases: + if phase.name == "setup": + values = ", ".join( + _format_ns(t.cast(int, sample.duration_ns)) + for sample in phase.samples + if sample.accepted + ) + lines.append( + f"| `{phase.name}` | `{phase.status}` | " + f"{len(phase.samples)} individual: {values} | n/a | n/a |" + ) + elif phase.summary is None: + lines.append(f"| `{phase.name}` | `{phase.status}` | 0 | n/a | n/a |") + else: + lines.append( + f"| `{phase.name}` | `{phase.status}` | " + f"{phase.summary['count']} | " + f"{_format_ns(phase.summary['median_ns'])} | " + f"{_format_ns(phase.summary['p95_ns'])} |" + ) + lines.append("") + lines.extend( + ( + "## Cleanup", + "", + f"Verified complete: `{str(report.cleanup.complete).lower()}`", + "", + ) + ) + rendered = "\n".join(lines) + if output_path is not None: + _write_text_atomic(output_path, rendered) + return rendered + + +def run_scenario( + topology: Topology, + *, + lane: EngineLane = EngineLane.CONTROL, + mode: ExecutionMode = ExecutionMode.ASYNC, + runs: int = 100, + warmup: int = 3, + seed: int = 11, + output: pathlib.Path = pathlib.Path("orchestration-report.json"), + markdown_output: pathlib.Path | None = None, + scratch_root: pathlib.Path | None = None, + force_extreme: bool = False, + policy: ResourcePolicy | None = None, + host_snapshot: HostSnapshot | None = None, + watchdog_s: float = 120.0, + cleanup_grace_s: float = 2.0, + _test_stall_after: str | None = None, + _test_fail_after: str | None = None, + _test_extra_identity: ProcessIdentity | None = None, +) -> RunReport: + """Preflight and supervise one fresh active benchmark scenario. + + Private ``_test_*`` arguments are exposed only as suppressed hidden-worker + harness flags and are never part of the documented benchmark interface. + + >>> refusal_path = pathlib.Path(tempfile.gettempdir()) / "unused-refusal.json" + >>> refused = run_scenario( + ... Topology(1, 1, 1), runs=1, warmup=0, + ... output=refusal_path, + ... host_snapshot=HostSnapshot(pids_current=10, pids_max=11), + ... policy=ResourcePolicy(pid_reserve=1), + ... ) + >>> refused.status + 'refused' + >>> refusal_path.unlink(missing_ok=True) + >>> refusal_path.with_suffix(".md").unlink(missing_ok=True) + + Parameters + ---------- + topology : Topology + Exact requested hierarchy. + lane : EngineLane + Subprocess or control transport; control is the default. + mode : ExecutionMode + Sync or async execution; async is the default. + runs : int + Timed samples per repeatable cell. + warmup : int + Untimed samples per repeatable cell. + seed : int + Deterministic cell interleaving seed. + output : pathlib.Path + Supervisor-owned JSON destination. + markdown_output : pathlib.Path | None + Summary destination, defaulting beside ``output``. + scratch_root : pathlib.Path | None + Parent for fresh private run directories. + force_extreme : bool + Override predictive refusal only. + policy : ResourcePolicy | None + Predictive and runtime reserve thresholds. + host_snapshot : HostSnapshot | None + Injectable preflight observation; live probing is the default. + watchdog_s : float + Maximum interval without an increasing progress sequence. + cleanup_grace_s : float + Bounded graceful and escalation waits. + + Returns + ------- + RunReport + Validated supervisor-owned terminal report. + """ + if type(runs) is not int or runs <= 0 or type(warmup) is not int or warmup < 0: + message = "runs must be positive and warmup nonnegative" + raise ValueError(message) + policy = policy or ResourcePolicy( + persistent_clients=(1 if lane is EngineLane.CONTROL else 0) + ) + snapshot = host_snapshot or probe_host(ProcessReader()) + original = predict_resources(topology, snapshot, policy) + decision = _forced_decision(original, force_extreme) + run_id = f"r{uuid.uuid4().hex[:10]}" + markdown_output = markdown_output or output.with_suffix(".md") + if not decision.allowed: + report = RunReport( + topology, + status="refused", + cleanup=CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + guard_decision=original, + original_guard_decision=original, + run_id=run_id, + lane=lane.value, + mode=mode.value, + warmup=warmup, + runs=runs, + failed_phase="preflight", + error=f"predictive refusal: {original.rule or 'unknown'}", + ) + write_json_atomic(output, report) + validate_report(report) + render_markdown_summary(output, markdown_output) + return report + scratch_parent = scratch_root or pathlib.Path(tempfile.gettempdir()) + scratch = scratch_parent.resolve() / f"run-{run_id}" + socket_path = scratch / "tmux.sock" + return supervise_worker( + topology, + lane=lane, + mode=mode, + runs=runs, + warmup=warmup, + seed=seed, + run_id=run_id, + scratch=scratch, + socket_path=socket_path, + output=output, + markdown_output=markdown_output, + guard_decision=decision, + original_guard_decision=original, + policy=policy, + watchdog_s=watchdog_s, + cleanup_grace_s=cleanup_grace_s, + _test_stall_after=_test_stall_after, + _test_fail_after=_test_fail_after, + _test_extra_identity=_test_extra_identity, + ) + + +def run_ramp( + shapes: t.Sequence[Topology], + *, + lane: EngineLane = EngineLane.CONTROL, + mode: ExecutionMode = ExecutionMode.ASYNC, + runs: int = 100, + warmup: int = 3, + seed: int = 11, + output: pathlib.Path = pathlib.Path("orchestration-ramp.json"), + markdown_output: pathlib.Path | None = None, + scratch_root: pathlib.Path | None = None, + force_extreme: bool = False, + policy: ResourcePolicy | None = None, + host_snapshot: HostSnapshot | None = None, + watchdog_s: float = 120.0, + cleanup_grace_s: float = 2.0, + canonical: bool = False, + _test_stall_after: str | None = None, + _test_fail_after: str | None = None, + _test_extra_identity: ProcessIdentity | None = None, +) -> RunReport: + """Run fresh disposable scenarios until completion or the first terminal stop. + + >>> try: + ... run_ramp((), runs=1, warmup=0) + ... except ValueError as error: + ... print(error) + ramp requires at least one unique shape + + Returns + ------- + RunReport + Validated aggregate with later shapes marked ``not_attempted``. + """ + declared = tuple(shapes) + if not declared or len(set(declared)) != len(declared): + message = "ramp requires at least one unique shape" + raise ValueError(message) + if canonical and declared != canonical_ramp(): + message = "canonical ramp must use the exact declared sequence" + raise ValueError(message) + markdown_output = markdown_output or output.with_suffix(".md") + attempts = tuple(RampStep(shape, "not_attempted") for shape in declared) + report = RunReport( + requested_topology=declared[-1], + status="in_progress", + cleanup=CleanupReport(False), + ramp=attempts, + requested_shapes=declared, + ramp_kind="canonical" if canonical else "custom", + environment=_collect_environment( + seed=seed, + command_line=("ramp", "--shapes", ",".join(map(str, declared))), + include_tmux=False, + ), + ) + write_json_atomic(output, report) + child_root = output.with_name(f"{output.stem}.runs") + steps = list(attempts) + last_observed: Topology | None = None + terminal_status: t.Literal["refused", "failed", "cutoff"] | None = None + terminal_reason: str | None = None + for index, shape in enumerate(declared): + child_output = child_root / f"{index:02d}-{shape}.json" + child_markdown = child_output.with_suffix(".md") + child = run_scenario( + shape, + lane=lane, + mode=mode, + runs=runs, + warmup=warmup, + seed=seed + index, + output=child_output, + markdown_output=child_markdown, + scratch_root=scratch_root, + force_extreme=force_extreme, + policy=policy, + host_snapshot=host_snapshot, + watchdog_s=watchdog_s, + cleanup_grace_s=cleanup_grace_s, + _test_stall_after=_test_stall_after, + _test_fail_after=_test_fail_after, + _test_extra_identity=_test_extra_identity, + ) + steps[index] = RampStep( + shape, + t.cast( + t.Literal["completed", "refused", "failed", "cutoff"], + child.status, + ), + child.error, + run_id=child.run_id, + report_path=str(child_output), + scratch_path=child.scratch_path, + socket_path=child.socket_path, + ) + if child.status == "completed": + last_observed = child.observed_topology + else: + terminal_status = t.cast( + t.Literal["refused", "failed", "cutoff"], child.status + ) + terminal_reason = child.error or child.status + for later in range(index + 1, len(declared)): + steps[later] = RampStep( + declared[later], "not_attempted", terminal_reason + ) + break + report = dataclasses.replace( + report, + observed_topology=last_observed, + ramp=tuple(steps), + ) + write_json_atomic(output, report) + cleanup_complete = all( + step.status == "not_attempted" + or load_run_report(pathlib.Path(t.cast(str, step.report_path))).cleanup.complete + for step in steps + ) + cleanup = CleanupReport( + cleanup_complete, + processes_absent=cleanup_complete, + socket_absent=cleanup_complete, + scratch_absent=cleanup_complete, + ) + final_status: t.Literal["completed", "refused", "failed", "cutoff"] = ( + terminal_status or "completed" + ) + report = dataclasses.replace( + report, + status=final_status, + observed_topology=last_observed, + cleanup=cleanup, + ramp=tuple(steps), + error=terminal_reason, + ) + write_json_atomic(output, report) + validate_report(report) + render_markdown_summary(output, markdown_output) + return report + + +def parse_topology(shape: str) -> Topology: + """Parse a positive ``SxWxP`` topology string. + + >>> parse_topology("2x3x4") + Topology(sessions=2, windows_per_session=3, panes_per_window=4) + + Parameters + ---------- + shape : str + Topology in lower-case ``SxWxP`` notation. + + Returns + ------- + Topology + Parsed topology with every dimension positive. + + Raises + ------ + ValueError + If the shape is malformed or has a nonpositive dimension. + """ + pieces = shape.split("x") + if len(pieces) != 3: + message = "topology must use SxWxP notation" + raise ValueError(message) + try: + values = tuple(int(piece) for piece in pieces) + except ValueError as exc: + message = "topology must use SxWxP notation" + raise ValueError(message) from exc + if any(value <= 0 for value in values): + message = "topology dimensions must be positive" + raise ValueError(message) + return Topology(*values) + + +def canonical_ramp() -> tuple[Topology, ...]: + """Return the specified progression ordered by expected pane pressure. + + >>> tuple(str(shape) for shape in canonical_ramp())[:3] + ('80x20x1', '100x20x1', '80x20x2') + + Returns + ------- + tuple[Topology, ...] + Exact canonical progression ordered by expected live-pane pressure. + """ + return ( + Topology(80, 20, 1), + Topology(100, 20, 1), + Topology(80, 20, 2), + Topology(80, 50, 1), + Topology(80, 20, 4), + Topology(80, 100, 1), + Topology(100, 50, 2), + Topology(100, 100, 2), + Topology(100, 100, 4), + ) + + +def summarize_ns(samples: t.Sequence[int]) -> dict[str, int | float]: + """Return descriptive statistics for accepted integer-nanosecond samples. + + Percentiles use the nearest-rank index ``ceil(p * count) - 1``. + + >>> summarize_ns((1, 2, 3, 4))["p90_ns"] + 4 + + Parameters + ---------- + samples : collections.abc.Sequence[int] + Accepted duration measurements in nanoseconds. + + Returns + ------- + dict[str, int | float] + Count, extrema, mean, median, and p90/p95/p99 values. + + Raises + ------ + ValueError + If no samples were accepted. + """ + if not samples: + message = "cannot summarize empty samples" + raise ValueError(message) + ordered = sorted(samples) + + count = len(ordered) + return { + "count": count, + "min_ns": ordered[0], + "mean_ns": statistics.mean(ordered), + "median_ns": statistics.median(ordered), + "p90_ns": ordered[math.ceil(0.90 * count) - 1], + "p95_ns": ordered[math.ceil(0.95 * count) - 1], + "p99_ns": ordered[math.ceil(0.99 * count) - 1], + "max_ns": ordered[-1], + } + + +def _load_host_snapshot(path: pathlib.Path | None) -> HostSnapshot | None: + """Load a private test-harness preflight snapshot. + + >>> _load_host_snapshot(None) is None + True + + Parameters + ---------- + path : pathlib.Path | None + Hidden CLI JSON fixture, or ``None`` for live probing. + + Returns + ------- + HostSnapshot | None + Decoded fixture or ``None``. + """ + if path is None: + return None + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as error: + message = f"test host snapshot is not complete JSON: {path}" + raise ValueError(message) from error + snapshot = _host_snapshot_from_json(value) + if snapshot is None: + message = "test host snapshot cannot be null" + raise ValueError(message) + return snapshot + + +def _parse_optional_limit(value: str) -> int | None: + """Parse a positive integer or the hidden worker's ``dynamic`` marker. + + >>> _parse_optional_limit("dynamic") is None + True + >>> _parse_optional_limit("1024") + 1024 + """ + if value == "dynamic": + return None + try: + parsed = int(value) + except ValueError as error: + message = "resource limit must be a positive integer or dynamic" + raise ValueError(message) from error + if parsed <= 0: + message = "resource limit must be a positive integer or dynamic" + raise ValueError(message) + return parsed + + +def _parse_extra_identity(value: str | None) -> ProcessIdentity | None: + """Decode the private PID-reuse test identity. + + >>> _parse_extra_identity("pane:42:100") + ProcessIdentity(role='pane', pid=42, start_time=100) + >>> _parse_extra_identity(None) is None + True + """ + if value is None: + return None + pieces = value.split(":") + if len(pieces) != 3 or not pieces[0]: + message = "test identity must use role:pid:start_time" + raise ValueError(message) + try: + identity = ProcessIdentity(pieces[0], int(pieces[1]), int(pieces[2])) + except ValueError as error: + message = "test identity must use role:pid:start_time" + raise ValueError(message) from error + if identity.pid <= 0 or identity.start_time <= 0: + message = "test identity requires positive pid and start_time" + raise ValueError(message) + return identity + + +async def _run_worker_with_signals(**kwargs: t.Any) -> RunReport: + """Translate worker SIGINT/SIGTERM into cancellation and awaited cleanup. + + >>> async def invalid(): + ... try: + ... await _run_worker_with_signals() + ... except TypeError as error: + ... return "topology" in str(error) + >>> asyncio.run(invalid()) + True + """ + loop = asyncio.get_running_loop() + task = asyncio.create_task(run_worker(**kwargs), name="orchestration-worker") + installed: list[signal.Signals] = [] + for signal_number in (signal.SIGINT, signal.SIGTERM): + try: + loop.add_signal_handler( + signal_number, + task.cancel, + f"received {signal_number.name}", + ) + except (NotImplementedError, RuntimeError): + continue + installed.append(signal_number) + try: + return await task + finally: + for signal_number in installed: + loop.remove_signal_handler(signal_number) + + +def _run_hidden_worker(arguments: argparse.Namespace) -> int: + """Decode hidden CLI state and execute exactly one worker process. + + >>> _run_hidden_worker.__name__ + '_run_hidden_worker' + """ + admission = _json_mapping( + json.loads(arguments.admission.read_text(encoding="utf-8")), "admission" + ) + guard = _guard_from_json(admission.get("guard_decision")) + original = _guard_from_json(admission.get("original_guard_decision")) + if guard is None or original is None: + message = "worker admission requires both guard decisions" + raise ValueError(message) + policy = ResourcePolicy( + persistent_clients=(1 if arguments.lane == "control" else 0), + pid_reserve=_parse_optional_limit(arguments.pid_reserve), + memory_floor_bytes=_parse_optional_limit(arguments.memory_floor_bytes), + ) + report = asyncio.run( + _run_worker_with_signals( + topology=parse_topology(arguments.shape), + lane=EngineLane(arguments.lane), + mode=ExecutionMode(arguments.mode), + runs=arguments.runs, + warmup=arguments.warmup, + seed=arguments.seed, + run_id=arguments.run_id, + scratch=arguments.scratch, + socket_path=arguments.socket_path, + checkpoint_path=arguments.checkpoint, + progress_path=arguments.progress, + guard_decision=guard, + original_guard_decision=original, + policy=policy, + stall_after=arguments.test_stall_after, + fail_after=arguments.test_fail_after, + extra_identity=_parse_extra_identity(arguments.test_extra_identity), + ) + ) + return 0 if report.status == "completed" else 2 + + +def _add_execution_arguments(parser: argparse.ArgumentParser) -> None: + """Add the shared public run/ramp execution flags. + + >>> parser = argparse.ArgumentParser() + >>> _add_execution_arguments(parser) + >>> parser.parse_args(["--runs", "2"]).runs + 2 + """ + parser.add_argument( + "--lane", choices=tuple(lane.value for lane in EngineLane), default="control" + ) + parser.add_argument( + "--mode", choices=tuple(mode.value for mode in ExecutionMode), default="async" + ) + parser.add_argument("--runs", type=int, default=100) + parser.add_argument("--warmup", type=int, default=3) + parser.add_argument("--seed", type=int, default=11) + parser.add_argument("--output", type=pathlib.Path) + parser.add_argument("--markdown-output", type=pathlib.Path) + parser.add_argument("--scratch-root", type=pathlib.Path) + parser.add_argument("--force-extreme", action="store_true") + parser.add_argument("--pid-reserve", type=int) + parser.add_argument("--memory-floor-bytes", type=int) + parser.add_argument("--watchdog-seconds", type=float, default=120.0) + parser.add_argument("--cleanup-grace-seconds", type=float, default=2.0) + parser.add_argument( + "--_test-host-snapshot", + dest="test_host_snapshot", + type=pathlib.Path, + help=argparse.SUPPRESS, + ) + parser.add_argument( + "--_test-stall-after", + dest="test_stall_after", + help=argparse.SUPPRESS, + ) + parser.add_argument( + "--_test-fail-after", + dest="test_fail_after", + help=argparse.SUPPRESS, + ) + parser.add_argument( + "--_test-extra-identity", + dest="test_extra_identity", + help=argparse.SUPPRESS, + ) + + +def _hidden_worker_parser() -> argparse.ArgumentParser: + """Build the private supervisor-to-worker protocol parser. + + Keeping this parser outside the public subcommand collection prevents the + implementation protocol and its test harness from appearing in help. + + >>> parser = _hidden_worker_parser() + >>> parser.prog + 'bench_orchestration.py _worker' + + Returns + ------- + argparse.ArgumentParser + Parser for arguments emitted only by :func:`supervise_worker`. + """ + parser = argparse.ArgumentParser(prog="bench_orchestration.py _worker") + parser.add_argument("--shape", required=True) + parser.add_argument("--lane", required=True, choices=("subprocess", "control")) + parser.add_argument("--mode", required=True, choices=("sync", "async")) + parser.add_argument("--runs", required=True, type=int) + parser.add_argument("--warmup", required=True, type=int) + parser.add_argument("--seed", required=True, type=int) + parser.add_argument("--run-id", required=True) + parser.add_argument("--scratch", required=True, type=pathlib.Path) + parser.add_argument("--socket-path", required=True, type=pathlib.Path) + parser.add_argument("--checkpoint", required=True, type=pathlib.Path) + parser.add_argument("--progress", required=True, type=pathlib.Path) + parser.add_argument("--admission", required=True, type=pathlib.Path) + parser.add_argument("--pid-reserve", required=True) + parser.add_argument("--memory-floor-bytes", required=True) + parser.add_argument( + "--_test-stall-after", dest="test_stall_after", help=argparse.SUPPRESS + ) + parser.add_argument( + "--_test-fail-after", dest="test_fail_after", help=argparse.SUPPRESS + ) + parser.add_argument( + "--_test-extra-identity", + dest="test_extra_identity", + help=argparse.SUPPRESS, + ) + return parser + + +def main(argv: t.Sequence[str] | None = None) -> int: + """Run planning, one supervised scenario, a ramp, or the hidden worker. + + >>> import contextlib, io + >>> captured = io.StringIO() + >>> with contextlib.redirect_stdout(captured): + ... result = main(["plan", "--shape", "1x1x1"]) + >>> result + 0 + + Parameters + ---------- + argv : collections.abc.Sequence[str] | None + Explicit command arguments, or process arguments when omitted. + + Returns + ------- + int + Zero only after the selected public work completes successfully. + + Raises + ------ + SystemExit + If command-line arguments are invalid. + ValueError + If the selected plan topology is malformed or nonpositive. + OSError + If the selected plan output cannot be written. + """ + raw_arguments = tuple(sys.argv[1:] if argv is None else argv) + if raw_arguments[:1] == ("_worker",): + worker_arguments = _hidden_worker_parser().parse_args(raw_arguments[1:]) + return _run_hidden_worker(worker_arguments) + parser = argparse.ArgumentParser(description=__doc__) + commands = parser.add_subparsers(dest="command", required=True) + plan_parser = commands.add_parser("plan", help="inspect topology and host limits") + plan_parser.add_argument("--shape", required=True) + plan_parser.add_argument("--output", type=pathlib.Path) + plan_parser.add_argument("--force-extreme", action="store_true") + run_parser = commands.add_parser("run", help="run one active topology") + run_parser.add_argument("--shape", required=True) + _add_execution_arguments(run_parser) + ramp_parser = commands.add_parser("ramp", help="run fresh topologies in order") + ramp_parser.add_argument("--shapes") + _add_execution_arguments(ramp_parser) + arguments = parser.parse_args(raw_arguments) + if arguments.command == "plan": + return run_plan(arguments.shape, arguments.output, arguments.force_extreme) + policy = ResourcePolicy( + persistent_clients=(1 if arguments.lane == "control" else 0), + pid_reserve=arguments.pid_reserve, + memory_floor_bytes=arguments.memory_floor_bytes, + ) + host_snapshot = _load_host_snapshot(arguments.test_host_snapshot) + extra_identity = _parse_extra_identity(arguments.test_extra_identity) + if arguments.command == "run": + output = arguments.output or pathlib.Path("orchestration-report.json") + report = run_scenario( + parse_topology(arguments.shape), + lane=EngineLane(arguments.lane), + mode=ExecutionMode(arguments.mode), + runs=arguments.runs, + warmup=arguments.warmup, + seed=arguments.seed, + output=output, + markdown_output=arguments.markdown_output, + scratch_root=arguments.scratch_root, + force_extreme=arguments.force_extreme, + policy=policy, + host_snapshot=host_snapshot, + watchdog_s=arguments.watchdog_seconds, + cleanup_grace_s=arguments.cleanup_grace_seconds, + _test_stall_after=arguments.test_stall_after, + _test_fail_after=arguments.test_fail_after, + _test_extra_identity=extra_identity, + ) + return 0 if report.status == "completed" else 2 + if arguments.command == "ramp": + shapes = ( + canonical_ramp() + if arguments.shapes is None + else tuple(parse_topology(shape) for shape in arguments.shapes.split(",")) + ) + output = arguments.output or pathlib.Path("orchestration-ramp.json") + report = run_ramp( + shapes, + lane=EngineLane(arguments.lane), + mode=ExecutionMode(arguments.mode), + runs=arguments.runs, + warmup=arguments.warmup, + seed=arguments.seed, + output=output, + markdown_output=arguments.markdown_output, + scratch_root=arguments.scratch_root, + force_extreme=arguments.force_extreme, + policy=policy, + host_snapshot=host_snapshot, + watchdog_s=arguments.watchdog_seconds, + cleanup_grace_s=arguments.cleanup_grace_seconds, + canonical=arguments.shapes is None, + _test_stall_after=arguments.test_stall_after, + _test_fail_after=arguments.test_fail_after, + _test_extra_identity=extra_identity, + ) + return 0 if report.status == "completed" else 2 message = "argparse selected an unsupported command" raise AssertionError(message) diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index 219a529720..a1fcb74d21 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -30,6 +30,112 @@ ("control", "async"), ) +_RUNNER_REPEATABLE_PHASES = ( + "mutation.bulk", + "wait.capture-poll", + "enumeration.sessions", + "enumeration.windows", + "enumeration.panes", + "capture.serial", + "capture.batched", + "search.classic.sessions.first", + "search.classic.sessions.middle", + "search.classic.sessions.last", + "search.classic.windows.first", + "search.classic.windows.middle", + "search.classic.windows.last", + "search.classic.panes.first", + "search.classic.panes.middle", + "search.classic.panes.last", + "search.snapshot.sessions.first", + "search.snapshot.sessions.middle", + "search.snapshot.sessions.last", + "search.snapshot.windows.first", + "search.snapshot.windows.middle", + "search.snapshot.windows.last", + "search.snapshot.panes.first", + "search.snapshot.panes.middle", + "search.snapshot.panes.last", + "search.end-to-end.sessions.first", + "search.end-to-end.sessions.middle", + "search.end-to-end.sessions.last", + "search.end-to-end.windows.first", + "search.end-to-end.windows.middle", + "search.end-to-end.windows.last", + "search.end-to-end.panes.first", + "search.end-to-end.panes.middle", + "search.end-to-end.panes.last", + "search.contents", +) + + +def _benchmark_script() -> pathlib.Path: + """Return the real standalone benchmark entry point.""" + return pathlib.Path(__file__).parents[1] / "scripts" / "bench_orchestration.py" + + +def _run_cli(*arguments: str, cwd: pathlib.Path) -> subprocess.CompletedProcess[str]: + """Run the real benchmark process with no inherited tmux coordinates.""" + environment = os.environ.copy() + environment.pop("TMUX", None) + environment.pop("TMUX_PANE", None) + return subprocess.run( + (sys.executable, str(_benchmark_script()), *arguments), + cwd=cwd, + env=environment, + text=True, + capture_output=True, + check=False, + timeout=120, + ) + + +def _start_cli(*arguments: str, cwd: pathlib.Path) -> subprocess.Popen[str]: + """Start the real benchmark so a test can interrupt its supervisor.""" + environment = os.environ.copy() + environment.pop("TMUX", None) + environment.pop("TMUX_PANE", None) + return subprocess.Popen( + (sys.executable, str(_benchmark_script()), *arguments), + cwd=cwd, + env=environment, + text=True, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + + +def _process_start_time(pid: int) -> int: + """Read one exact Linux process start-time identity.""" + raw = pathlib.Path(f"/proc/{pid}/stat").read_text(encoding="utf-8") + close = raw.rindex(")") + return int(raw[close + 2 :].split()[19]) + + +def _identity_still_matches(identity: dict[str, t.Any]) -> bool: + """Return whether procfs still names the exact serialized process.""" + try: + raw = pathlib.Path(f"/proc/{identity['pid']}/stat").read_text(encoding="utf-8") + close = raw.rindex(")") + return int(raw[close + 2 :].split()[19]) == t.cast(int, identity["start_time"]) + except (FileNotFoundError, OSError, ValueError, IndexError): + return False + + +def _assert_terminal_cleanup(payload: dict[str, t.Any]) -> None: + """Assert the supervisor's exact terminal cleanup evidence.""" + assert payload["cleanup"] == { + "complete": True, + "errors": [], + "processes_absent": True, + "scratch_absent": True, + "socket_absent": True, + } + assert not pathlib.Path(payload["scratch_path"]).exists() + assert not pathlib.Path(payload["socket_path"]).exists() + assert all(not _identity_still_matches(row) for row in payload["processes"]) + @pytest.fixture() def benchmark_module() -> types.ModuleType: @@ -3318,3 +3424,739 @@ def fail_second() -> t.Any: assert result.failure.strategy == "only" assert result.failure.ordinal == 1 assert result.failure.error == "RuntimeError: live postcondition lost" + + +def test_cli_run_executes_every_phase_and_writes_validated_artifacts( + tmp_path: pathlib.Path, +) -> None: + """Skipping a phase or requested sample would publish incomplete evidence.""" + report_path = tmp_path / "run.json" + markdown_path = tmp_path / "run.md" + scratch_root = tmp_path / "scratch" + + completed = _run_cli( + "run", + "--shape", + "2x2x2", + "--runs", + "2", + "--warmup", + "1", + "--output", + str(report_path), + "--markdown-output", + str(markdown_path), + "--scratch-root", + str(scratch_root), + "--watchdog-seconds", + "30", + cwd=tmp_path, + ) + + assert completed.returncode == 0, completed.stderr + payload = json.loads(report_path.read_text(encoding="utf-8")) + assert payload["status"] == "completed" + assert payload["requested_topology"] == { + "sessions": 2, + "windows_per_session": 2, + "panes_per_window": 2, + } + assert payload["observed_topology"] == payload["requested_topology"] + assert payload["lane"] == "control" + assert payload["mode"] == "async" + phases = {phase["name"]: phase for phase in payload["phases"]} + assert tuple(phases) == ( + "setup", + "stabilization", + *_RUNNER_REPEATABLE_PHASES[:2], + "wait.control-stream", + *_RUNNER_REPEATABLE_PHASES[2:], + ) + assert phases["setup"]["summary"] is None + assert len(phases["setup"]["samples"]) == 1 + assert phases["stabilization"]["samples"] == [] + for phase_name in (*_RUNNER_REPEATABLE_PHASES, "wait.control-stream"): + phase = phases[phase_name] + assert phase["status"] == "completed" + assert phase["warmup"] == 1 + assert phase["runs"] == 2 + assert len(phase["samples"]) == 2 + assert phase["summary"]["count"] == 2 + assert len(phase["observations"]) == 2 + assert all(row["accepted"] and row["verified"] for row in phase["samples"]) + assert payload["cleanup"] == { + "complete": True, + "errors": [], + "processes_absent": True, + "scratch_absent": True, + "socket_absent": True, + } + assert not pathlib.Path(payload["scratch_path"]).exists() + assert not pathlib.Path(payload["socket_path"]).exists() + assert all(not _identity_still_matches(row) for row in payload["processes"]) + progress = pathlib.Path(payload["progress_path"]) + events = [json.loads(line) for line in progress.read_text().splitlines()] + assert [event["sequence"] for event in events] == list(range(len(events))) + assert "Local descriptive evidence" in markdown_path.read_text(encoding="utf-8") + + +def test_cli_ramp_uses_fresh_owned_resources_for_every_shape( + tmp_path: pathlib.Path, +) -> None: + """Reusing a server between shapes would invalidate fresh-setup evidence.""" + report_path = tmp_path / "ramp.json" + markdown_path = tmp_path / "ramp.md" + + completed = _run_cli( + "ramp", + "--shapes", + "1x1x1,2x2x1", + "--runs", + "1", + "--warmup", + "0", + "--output", + str(report_path), + "--markdown-output", + str(markdown_path), + "--scratch-root", + str(tmp_path / "scratch"), + "--watchdog-seconds", + "30", + cwd=tmp_path, + ) + + assert completed.returncode == 0, completed.stderr + payload = json.loads(report_path.read_text(encoding="utf-8")) + assert payload["status"] == "completed" + assert [step["status"] for step in payload["ramp"]] == [ + "completed", + "completed", + ] + assert [step["shape"] for step in payload["ramp"]] == [ + {"sessions": 1, "windows_per_session": 1, "panes_per_window": 1}, + {"sessions": 2, "windows_per_session": 2, "panes_per_window": 1}, + ] + run_ids = [step["run_id"] for step in payload["ramp"]] + scratch_paths = [step["scratch_path"] for step in payload["ramp"]] + socket_paths = [step["socket_path"] for step in payload["ramp"]] + assert len(set(run_ids)) == 2 + assert len(set(scratch_paths)) == 2 + assert len(set(socket_paths)) == 2 + assert all(not pathlib.Path(path).exists() for path in scratch_paths) + assert all(not pathlib.Path(path).exists() for path in socket_paths) + child_reports = [ + json.loads(pathlib.Path(step["report_path"]).read_text(encoding="utf-8")) + for step in payload["ramp"] + ] + assert [child["run_id"] for child in child_reports] == run_ids + assert [child["status"] for child in child_reports] == [ + "completed", + "completed", + ] + server_identities = [ + next(row for row in child["processes"] if row["role"] == "server") + for child in child_reports + ] + assert len({(row["pid"], row["start_time"]) for row in server_identities}) == 2 + assert all(not _identity_still_matches(row) for row in server_identities) + assert "1x1x1" in markdown_path.read_text(encoding="utf-8") + assert "2x2x1" in markdown_path.read_text(encoding="utf-8") + + +def test_cli_predictive_refusal_writes_terminal_report_without_worker( + tmp_path: pathlib.Path, +) -> None: + """A refused preflight must not create any live benchmark-owned resource.""" + report_path = tmp_path / "refused.json" + markdown_path = tmp_path / "refused.md" + host_path = tmp_path / "host.json" + host_path.write_text( + json.dumps( + { + "available_memory_bytes": 16 * 1024**3, + "physical_memory_bytes": 32 * 1024**3, + "memory_current_bytes": 1024, + "memory_max_bytes": 32 * 1024**3, + "pids_current": 10, + "pids_max": 12, + "nofile_soft_limit": 65536, + "nofile_hard_limit": 65536, + "memory_pressure_some_avg10": 0.0, + "source_errors": {}, + } + ), + encoding="utf-8", + ) + + completed = _run_cli( + "run", + "--shape", + "2x2x2", + "--runs", + "1", + "--warmup", + "0", + "--output", + str(report_path), + "--markdown-output", + str(markdown_path), + "--scratch-root", + str(tmp_path / "scratch"), + "--pid-reserve", + "1", + "--_test-host-snapshot", + str(host_path), + cwd=tmp_path, + ) + + assert completed.returncode != 0 + payload = json.loads(report_path.read_text(encoding="utf-8")) + assert payload["status"] == "refused" + assert payload["failed_phase"] == "preflight" + assert payload["guard_decision"]["kind"] == "predictive_refusal" + assert payload["processes"] == [] + assert payload["progress_path"] is None + assert payload["scratch_path"] is None + assert payload["socket_path"] is None + assert payload["cleanup"]["complete"] is True + assert not (tmp_path / "scratch").exists() + assert not list(tmp_path.glob("*.progress.jsonl")) + assert "refused" in markdown_path.read_text(encoding="utf-8") + + +def test_cli_watchdog_recovers_stalled_worker_and_exact_resources( + tmp_path: pathlib.Path, +) -> None: + """A stopped progress stream must trigger bounded supervisor recovery.""" + report_path = tmp_path / "watchdog.json" + + completed = _run_cli( + "run", + "--shape", + "1x1x1", + "--runs", + "1", + "--warmup", + "0", + "--output", + str(report_path), + "--scratch-root", + str(tmp_path / "scratch"), + "--watchdog-seconds", + "1.0", + "--cleanup-grace-seconds", + "0.3", + "--_test-stall-after", + "setup", + cwd=tmp_path, + ) + + assert completed.returncode != 0 + payload = json.loads(report_path.read_text(encoding="utf-8")) + assert payload["status"] == "cutoff" + assert payload["failed_phase"] == "watchdog" + assert "progress watchdog expired" in payload["error"] + assert payload["progress_sequence"] >= 0 + assert {row["role"] for row in payload["processes"]} >= { + "worker", + "fuzzer", + "server", + "pane", + } + _assert_terminal_cleanup(payload) + + +def test_watchdog_progress_requires_an_increasing_sequence( + benchmark_module: types.ModuleType, +) -> None: + """Duplicate or older progress lines must not refresh the watchdog.""" + identity = benchmark_module.ProcessIdentity("fuzzer", 2, 3) + stalled = ( + benchmark_module.ProgressEvent("run-7", 3, "same", 10, (identity,)), + benchmark_module.ProgressEvent("run-7", 2, "older", 11), + benchmark_module.ProgressEvent("run-7", 3, "same-again", 12), + ) + + highest, identities, advanced = benchmark_module._accept_progress_events( + stalled, + run_id="run-7", + highest_sequence=3, + identities=(), + ) + + assert highest == 3 + assert identities == () + assert advanced is False + advanced_event = benchmark_module.ProgressEvent("run-7", 4, "next", 13, (identity,)) + highest, identities, advanced = benchmark_module._accept_progress_events( + (advanced_event,), + run_id="run-7", + highest_sequence=highest, + identities=identities, + ) + assert highest == 4 + assert identities == (identity,) + assert advanced is True + + +def test_worker_progress_precedes_matching_report_checkpoint( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """An identity event must be durable before its checkpoint can expose it.""" + calls: list[str] = [] + monkeypatch.setattr( + benchmark_module, + "append_progress_event", + lambda _path, _event: calls.append("progress"), + ) + monkeypatch.setattr( + benchmark_module, + "write_json_atomic", + lambda _path, _report: calls.append("checkpoint"), + ) + recorder = benchmark_module._WorkerRecorder( + benchmark_module.RunReport(benchmark_module.Topology(1, 1, 1), run_id="run-7"), + tmp_path / "checkpoint.json", + tmp_path / "progress.jsonl", + ) + + recorder.checkpoint("worker.started") + + assert calls == ["progress", "checkpoint"] + + +def test_cli_phase_failure_uses_supervisor_cleanup_contract( + tmp_path: pathlib.Path, +) -> None: + """A measured phase exception must retain failure and cleanup evidence.""" + report_path = tmp_path / "failure.json" + + completed = _run_cli( + "run", + "--shape", + "1x1x1", + "--runs", + "1", + "--warmup", + "0", + "--output", + str(report_path), + "--scratch-root", + str(tmp_path / "scratch"), + "--watchdog-seconds", + "30", + "--_test-fail-after", + "mutation.bulk", + cwd=tmp_path, + ) + + assert completed.returncode != 0 + payload = json.loads(report_path.read_text(encoding="utf-8")) + assert payload["status"] == "failed" + assert payload["failed_phase"] == "mutation.bulk" + assert "injected phase failure" in payload["error"] + phases = {phase["name"]: phase for phase in payload["phases"]} + assert phases["mutation.bulk"]["status"] == "failed" + _assert_terminal_cleanup(payload) + + +def test_cli_runtime_cutoff_is_not_relaxed_by_force_extreme( + tmp_path: pathlib.Path, +) -> None: + """A predictive override must not turn a live memory cutoff into failure.""" + report_path = tmp_path / "runtime-cutoff.json" + + completed = _run_cli( + "run", + "--shape", + "1x1x1", + "--runs", + "1", + "--warmup", + "0", + "--output", + str(report_path), + "--scratch-root", + str(tmp_path / "scratch"), + "--memory-floor-bytes", + str(10**18), + "--force-extreme", + "--watchdog-seconds", + "30", + cwd=tmp_path, + ) + + assert completed.returncode != 0 + payload = json.loads(report_path.read_text(encoding="utf-8")) + assert payload["status"] == "cutoff" + assert payload["failed_phase"] == "memory_floor" + assert payload["guard_decision"]["kind"] == "runtime_cutoff" + assert payload["original_guard_decision"]["kind"] == "predictive_refusal" + _assert_terminal_cleanup(payload) + + +def test_cli_cancellation_uses_supervisor_cleanup_contract( + tmp_path: pathlib.Path, +) -> None: + """SIGINT to the public supervisor must clean its isolated worker run.""" + report_path = tmp_path / "cancelled.json" + process = _start_cli( + "run", + "--shape", + "1x1x1", + "--runs", + "1", + "--warmup", + "0", + "--output", + str(report_path), + "--scratch-root", + str(tmp_path / "scratch"), + "--watchdog-seconds", + "30", + "--cleanup-grace-seconds", + "0.3", + "--_test-stall-after", + "setup", + cwd=tmp_path, + ) + stdout = stderr = "" + try: + deadline = time.monotonic() + 20 + while time.monotonic() < deadline: + streams = tuple(tmp_path.glob("*.progress.jsonl")) + if streams and '"checkpoint":"setup"' in streams[0].read_text( + encoding="utf-8" + ): + break + if process.poll() is not None: + break + time.sleep(0.02) + else: + pytest.fail("worker never reached the injected setup stall") + assert process.poll() is None + process.send_signal(signal.SIGINT) + stdout, stderr = process.communicate(timeout=20) + finally: + if process.poll() is None: + process.kill() + stdout, stderr = process.communicate(timeout=5) + + assert process.returncode != 0, (stdout, stderr) + payload = json.loads(report_path.read_text(encoding="utf-8")) + assert payload["status"] == "cutoff" + assert payload["failed_phase"] == "cancellation" + assert "supervisor interrupted" in payload["error"] + _assert_terminal_cleanup(payload) + + +def test_cli_process_identity_mismatch_never_signals_unrelated_pid( + tmp_path: pathlib.Path, +) -> None: + """A reused PID with another start time must remain outside recovery.""" + unrelated = subprocess.Popen( + (sys.executable, "-c", "import time; time.sleep(60)"), + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + start_new_session=True, + ) + try: + start_time = _process_start_time(unrelated.pid) + report_path = tmp_path / "identity.json" + completed = _run_cli( + "run", + "--shape", + "1x1x1", + "--runs", + "1", + "--warmup", + "0", + "--output", + str(report_path), + "--scratch-root", + str(tmp_path / "scratch"), + "--watchdog-seconds", + "0.2", + "--cleanup-grace-seconds", + "0.3", + "--_test-extra-identity", + f"unrelated:{unrelated.pid}:{start_time + 1}", + "--_test-stall-after", + "identity.unrelated", + cwd=tmp_path, + ) + + assert completed.returncode != 0 + assert unrelated.poll() is None + assert _process_start_time(unrelated.pid) == start_time + payload = json.loads(report_path.read_text(encoding="utf-8")) + assert payload["status"] == "cutoff" + assert any( + row + == { + "role": "unrelated", + "pid": unrelated.pid, + "start_time": start_time + 1, + } + for row in payload["processes"] + ) + _assert_terminal_cleanup(payload) + finally: + unrelated.terminate() + unrelated.wait(timeout=5) + + +def test_cli_ramp_refusal_marks_later_shapes_not_attempted( + tmp_path: pathlib.Path, +) -> None: + """A refused shape must stop the ramp and preserve one reason downstream.""" + report_path = tmp_path / "ramp-refused.json" + markdown_path = tmp_path / "ramp-refused.md" + host_path = tmp_path / "host.json" + host_path.write_text( + json.dumps( + { + "available_memory_bytes": 16 * 1024**3, + "physical_memory_bytes": 32 * 1024**3, + "memory_current_bytes": 1024, + "memory_max_bytes": 32 * 1024**3, + "pids_current": 10, + "pids_max": 12, + "nofile_soft_limit": 65536, + "nofile_hard_limit": 65536, + "memory_pressure_some_avg10": 0.0, + "source_errors": {}, + } + ), + encoding="utf-8", + ) + + completed = _run_cli( + "ramp", + "--shapes", + "1x1x1,2x2x2", + "--runs", + "1", + "--warmup", + "0", + "--output", + str(report_path), + "--markdown-output", + str(markdown_path), + "--scratch-root", + str(tmp_path / "scratch"), + "--pid-reserve", + "1", + "--_test-host-snapshot", + str(host_path), + cwd=tmp_path, + ) + + assert completed.returncode != 0 + payload = json.loads(report_path.read_text(encoding="utf-8")) + assert payload["status"] == "refused" + assert [step["status"] for step in payload["ramp"]] == [ + "refused", + "not_attempted", + ] + assert payload["ramp"][0]["reason"] == payload["ramp"][1]["reason"] + assert payload["ramp"][0]["run_id"] is not None + assert payload["ramp"][0]["report_path"] is not None + assert payload["ramp"][0]["scratch_path"] is None + assert payload["ramp"][0]["socket_path"] is None + assert payload["ramp"][1]["run_id"] is None + assert payload["cleanup"]["complete"] is True + assert not (tmp_path / "scratch").exists() + assert "not_attempted" in markdown_path.read_text(encoding="utf-8") + + +def test_cli_ramp_predictive_refusal_never_executes_tmux_binary( + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Aggregate environment capture must not contact tmux before admission.""" + marker = tmp_path / "tmux-executed" + fake_bin = tmp_path / "bin" + fake_bin.mkdir() + fake_tmux = fake_bin / "tmux" + fake_tmux.write_text( + "#!/usr/bin/env python3\n" + "import pathlib\n" + f"pathlib.Path({str(marker)!r}).touch()\n" + "print('tmux fake')\n", + encoding="utf-8", + ) + fake_tmux.chmod(fake_tmux.stat().st_mode | stat.S_IXUSR) + monkeypatch.setenv("PATH", f"{fake_bin}{os.pathsep}{os.environ['PATH']}") + host_path = tmp_path / "host.json" + host_path.write_text( + json.dumps( + { + "available_memory_bytes": 16 * 1024**3, + "physical_memory_bytes": 32 * 1024**3, + "memory_current_bytes": 1024, + "memory_max_bytes": 32 * 1024**3, + "pids_current": 10, + "pids_max": 12, + "nofile_soft_limit": 65536, + "nofile_hard_limit": 65536, + "memory_pressure_some_avg10": 0.0, + "source_errors": {}, + } + ), + encoding="utf-8", + ) + + completed = _run_cli( + "ramp", + "--shapes", + "1x1x1,2x2x2", + "--runs", + "1", + "--warmup", + "0", + "--output", + str(tmp_path / "ramp.json"), + "--pid-reserve", + "1", + "--_test-host-snapshot", + str(host_path), + cwd=tmp_path, + ) + + assert completed.returncode != 0 + assert not marker.exists() + + +def test_cli_plan_is_a_real_read_only_subprocess(tmp_path: pathlib.Path) -> None: + """Planning must print admission evidence without creating owned state.""" + completed = _run_cli("plan", "--shape", "2x2x2", cwd=tmp_path) + + assert completed.returncode == 0, completed.stderr + assert "Sessions" in completed.stdout + assert "Windows" in completed.stdout + assert "Panes" in completed.stdout + assert "Allowed" in completed.stdout + assert tuple(tmp_path.iterdir()) == () + + +def test_markdown_renderer_refuses_invalid_json_evidence( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """Rendering must validate JSON before replacing an existing summary.""" + report_path = tmp_path / "invalid.json" + markdown_path = tmp_path / "summary.md" + report = completed_report(benchmark_module) + benchmark_module.write_json_atomic(report_path, report) + payload = json.loads(report_path.read_text(encoding="utf-8")) + payload["phases"][0]["summary"]["count"] = 99 + report_path.write_text(json.dumps(payload), encoding="utf-8") + markdown_path.write_text("retained\n", encoding="utf-8") + + with pytest.raises(ValueError, match="summary"): + benchmark_module.render_markdown_summary(report_path, markdown_path) + + assert markdown_path.read_text(encoding="utf-8") == "retained\n" + + +def test_cli_help_hides_worker_and_private_test_harness_flags( + tmp_path: pathlib.Path, +) -> None: + """The worker protocol and injection hooks must not be public CLI surface.""" + root_help = _run_cli("--help", cwd=tmp_path) + run_help = _run_cli("run", "--help", cwd=tmp_path) + + assert root_help.returncode == 0 + assert "_worker" not in root_help.stdout + assert "_test" not in root_help.stdout + assert run_help.returncode == 0 + assert "_test" not in run_help.stdout + + +@pytest.mark.parametrize(("lane", "mode"), _PHASE_LANES) +def test_cli_run_supports_all_four_engine_mode_lanes( + tmp_path: pathlib.Path, + lane: str, + mode: str, +) -> None: + """Every explicit engine/mode lane must execute the same phase graph.""" + report_path = tmp_path / f"{lane}-{mode}.json" + + completed = _run_cli( + "run", + "--shape", + "1x1x1", + "--lane", + lane, + "--mode", + mode, + "--runs", + "1", + "--warmup", + "0", + "--output", + str(report_path), + "--scratch-root", + str(tmp_path / "scratch"), + "--watchdog-seconds", + "30", + cwd=tmp_path, + ) + + assert completed.returncode == 0, completed.stderr + payload = json.loads(report_path.read_text(encoding="utf-8")) + assert payload["status"] == "completed" + assert (payload["lane"], payload["mode"]) == (lane, mode) + phases = {phase["name"]: phase for phase in payload["phases"]} + expected_control = ( + "completed" if (lane, mode) == ("control", "async") else ("not_applicable") + ) + assert phases["wait.control-stream"]["status"] == expected_control + assert phases["wait.capture-poll"]["status"] == "completed" + _assert_terminal_cleanup(payload) + + +def test_cli_ramp_phase_failure_marks_later_shapes_not_attempted( + tmp_path: pathlib.Path, +) -> None: + """A worker failure must stop later ramp attempts after exact cleanup.""" + report_path = tmp_path / "ramp-failed.json" + + completed = _run_cli( + "ramp", + "--shapes", + "1x1x1,2x1x1", + "--runs", + "1", + "--warmup", + "0", + "--output", + str(report_path), + "--scratch-root", + str(tmp_path / "scratch"), + "--watchdog-seconds", + "30", + "--_test-fail-after", + "setup", + cwd=tmp_path, + ) + + assert completed.returncode != 0 + payload = json.loads(report_path.read_text(encoding="utf-8")) + assert payload["status"] == "failed" + assert [step["status"] for step in payload["ramp"]] == [ + "failed", + "not_attempted", + ] + assert payload["ramp"][0]["reason"] == payload["ramp"][1]["reason"] + child = json.loads( + pathlib.Path(payload["ramp"][0]["report_path"]).read_text(encoding="utf-8") + ) + assert child["failed_phase"] == "setup" + _assert_terminal_cleanup(child) + assert payload["cleanup"]["complete"] is True From e12ad7ba8ae2d93dac829a3559376b43d9fc94a5 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 13:00:12 -0500 Subject: [PATCH 20/67] Bench(fix[runner]): Harden supervised recovery why: Recovery must bind process ownership without PID races and remain durable through journal corruption or repeated cancellation. what: - Retain Linux pidfds for every published identity and drain identity deltas throughout bounded recovery - Make progress appends durable, fail closed on corrupt terminal tails, and preserve evidence until cleanup is proven - Shield lifecycle finalization through repeated cancellation and tighten phase and cleanup validation --- scripts/bench_orchestration.py | 1827 +++++++++++++++++----- tests/test_bench_orchestration_script.py | 1202 +++++++++++++- 2 files changed, 2617 insertions(+), 412 deletions(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index 437457d8bf..e5c52edf88 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -35,6 +35,7 @@ import subprocess import sys import tempfile +import threading import time import types import typing as t @@ -1556,6 +1557,8 @@ class PhaseReport: Untimed invocations requested for a repeatable cell. runs : int Timed invocations requested for a repeatable cell. + warmup_observations : tuple[PhaseObservation, ...] + Typed counts retained for every untimed warmup invocation. observations : tuple[PhaseObservation, ...] Typed counts corresponding one-to-one with accepted timed samples. """ @@ -1570,6 +1573,7 @@ class PhaseReport: ) warmup: int = 0 runs: int = 0 + warmup_observations: tuple[PhaseObservation, ...] = () observations: tuple[PhaseObservation, ...] = () def __post_init__(self) -> None: @@ -1763,7 +1767,7 @@ class ProgressEvent: monotonic_ns : int Worker monotonic publication time. processes : tuple[ProcessIdentity, ...] - Cumulative exact identities known at publication time. + Exact identities first learned at this checkpoint; never cumulative. schema_version : int Progress stream schema version. @@ -2012,6 +2016,8 @@ class RunContext: Exact construction operation and dispatch counts. process_identity_callback : collections.abc.Callable | None Private worker hook called as owned identities become known. + process_handles : _PidfdRegistry + Worker-owned stable handles retained as identities become known. Examples -------- @@ -2052,7 +2058,10 @@ class RunContext: heartbeat_monotonic_ns: int = -1 ambient_tmux_environment: tuple[str | None, str | None] = (None, None) setup_metrics: ExecutionMetrics | None = None - process_identity_callback: cabc.Callable[[ProcessIdentity], None] | None = None + process_identity_callback: ( + cabc.Callable[[tuple[ProcessIdentity, ...]], None] | None + ) = None + process_handles: _PidfdRegistry | None = None @dataclasses.dataclass(frozen=True) @@ -2472,6 +2481,9 @@ def _phase_from_json(value: object) -> PhaseReport: ), warmup=t.cast(int, row.get("warmup", 0)), runs=t.cast(int, row.get("runs", 0)), + warmup_observations=tuple( + _observation_from_json(item) for item in row.get("warmup_observations", []) + ), observations=tuple( _observation_from_json(item) for item in row.get("observations", []) ), @@ -2653,13 +2665,33 @@ def validate_report(report: RunReport) -> None: message = "invalid guard decision kind" raise ValueError(message) terminal = {"completed", "refused", "failed", "cutoff"} - if report.status in terminal and not report.cleanup.complete: + cleanup_facts_complete = ( + report.cleanup.complete is True + and report.cleanup.processes_absent is True + and report.cleanup.socket_absent is True + and report.cleanup.scratch_absent is True + and not report.cleanup.errors + ) + if report.cleanup.complete is True and not cleanup_facts_complete: + message = "cleanup complete requires all absence flags and no errors" + raise ValueError(message) + if report.status in {"completed", "refused", "cutoff"} and not ( + cleanup_facts_complete + ): message = "terminal report requires complete cleanup" raise ValueError(message) phase_statuses = {"in_progress", "completed", "failed", "not_applicable"} if len({phase.name for phase in report.phases}) != len(report.phases): message = "phase names must be unique" raise ValueError(message) + if report.status in terminal: + unfinished_seen = False + for phase in report.phases: + if phase.status in {"failed", "in_progress"}: + unfinished_seen = True + elif unfinished_seen and phase.status in {"completed", "not_applicable"}: + message = "terminal report has an invalid phase status prefix" + raise ValueError(message) for phase in report.phases: if phase.status not in phase_statuses: message = "invalid phase status" @@ -2673,10 +2705,33 @@ def validate_report(report: RunReport) -> None: message = "phase warmup and runs must be nonnegative integers" raise ValueError(message) if phase.status == "not_applicable" and ( - phase.samples or phase.summary is not None or phase.observations + phase.samples + or phase.summary is not None + or phase.warmup_observations + or phase.observations ): message = "not_applicable phase cannot carry timing evidence" raise ValueError(message) + if phase.name == "setup" and phase.status == "completed": + setup_sample = phase.samples[0] if len(phase.samples) == 1 else None + setup_observation = ( + phase.observations[0] if len(phase.observations) == 1 else None + ) + if ( + setup_sample is None + or setup_observation is None + or setup_sample.accepted is not True + or setup_sample.verified is not True + or setup_sample.error is not None + or setup_sample.ordinal != 0 + or setup_sample.strategy != "setup" + or setup_observation.verified is not True + or setup_observation.ordinal != 0 + or setup_observation.strategy != "setup" + or setup_sample.duration_ns != setup_observation.duration_ns + ): + message = "completed setup requires one accepted verified observation" + raise ValueError(message) accepted: list[int] = [] for sample in phase.samples: if sample.accepted and (sample.error is not None or not sample.verified): @@ -2712,6 +2767,56 @@ def validate_report(report: RunReport) -> None: ): message = "phase observation requires verified positive evidence" raise ValueError(message) + for observation in phase.warmup_observations: + if ( + type(observation.ordinal) is not int + or observation.ordinal < 0 + or not observation.strategy + or type(observation.duration_ns) is not int + or observation.duration_ns <= 0 + or not observation.verified + ): + message = "warmup observation requires verified positive evidence" + raise ValueError(message) + if ( + phase.runs > 0 + and phase.name not in {"setup", "stabilization"} + and (phase.status != "not_applicable") + ): + timed_ordinals = tuple(sample.ordinal for sample in phase.samples) + observation_ordinals = tuple( + observation.ordinal for observation in phase.observations + ) + expected_timed = tuple( + range(phase.runs if phase.status == "completed" else len(phase.samples)) + ) + if ( + timed_ordinals != expected_timed + or observation_ordinals != expected_timed + or any(sample.strategy != phase.name for sample in phase.samples) + or any( + observation.strategy != phase.name + for observation in phase.observations + ) + ): + message = f"phase {phase.name} has invalid timed ordinals" + raise ValueError(message) + warmup_ordinals = tuple( + observation.ordinal for observation in phase.warmup_observations + ) + expected_warmup = tuple( + range( + phase.warmup + if phase.status == "completed" + else len(phase.warmup_observations) + ) + ) + if warmup_ordinals != expected_warmup or any( + observation.strategy != phase.name + for observation in phase.warmup_observations + ): + message = f"phase {phase.name} has invalid warmup ordinals" + raise ValueError(message) ramp_kinds = {"none", "canonical", "custom"} terminal_statuses = {"refused", "cutoff", "failed"} attempt_statuses = {"completed", "not_attempted", *terminal_statuses} @@ -2845,7 +2950,7 @@ def _validate_executable_report(report: RunReport) -> None: if report.ramp_kind != "none": message = "one executable scenario cannot also be a ramp report" raise ValueError(message) - if report.status in {"completed", "refused", "failed", "cutoff"} and ( + if report.status in {"completed", "refused", "cutoff"} and ( report.cleanup.processes_absent is not True or report.cleanup.socket_absent is not True or report.cleanup.scratch_absent is not True @@ -2885,6 +2990,10 @@ def _validate_executable_report(report: RunReport) -> None: if not report.scratch_path or not report.socket_path or not report.progress_path: message = "attempted executable report requires owned resource paths" raise ValueError(message) + phase_names = tuple(phase.name for phase in report.phases) + if phase_names != _RUNNER_PHASES[: len(phase_names)]: + message = "attempted executable report phases must be a runner phase prefix" + raise ValueError(message) if report.status != "completed": if not report.failed_phase or not report.error: message = "terminal unsuccessful report requires phase and error" @@ -2907,7 +3016,16 @@ def _validate_executable_report(report: RunReport) -> None: or setup.runs != 1 or len(setup.samples) != 1 or len(setup.observations) != 1 + or setup.warmup_observations or setup.summary is not None + or not setup.samples[0].accepted + or not setup.samples[0].verified + or not setup.observations[0].verified + or setup.samples[0].ordinal != 0 + or setup.observations[0].ordinal != 0 + or setup.samples[0].strategy != "setup" + or setup.observations[0].strategy != "setup" + or setup.samples[0].duration_ns != setup.observations[0].duration_ns ): message = "setup must remain one unsummarized fresh-server observation" raise ValueError(message) @@ -3019,7 +3137,7 @@ def _validate_executable_ramp(report: RunReport) -> None: ... ) >>> _validate_executable_ramp(ramp) """ - if report.status in {"completed", "refused", "failed", "cutoff"} and ( + if report.status in {"completed", "refused", "cutoff"} and ( report.cleanup.processes_absent is not True or report.cleanup.socket_absent is not True or report.cleanup.scratch_absent is not True @@ -3250,6 +3368,199 @@ def process_identity_matches(identity: ProcessIdentity) -> bool: return _process_start_time(identity.pid) == identity.start_time +def _pidfd_capability_error() -> str | None: + """Return why stable process signaling is unavailable, if applicable. + + >>> error = _pidfd_capability_error() + >>> error is None or "pidfd" in error + True + + Returns + ------- + str | None + ``None`` only when Linux pidfd open and signal APIs are callable. + """ + if platform.system() != "Linux": + return "pidfd signaling requires Linux" + if not callable(getattr(os, "pidfd_open", None)): + return "os.pidfd_open is unavailable" + if not callable(getattr(signal, "pidfd_send_signal", None)): + return "signal.pidfd_send_signal is unavailable" + return None + + +@dataclasses.dataclass(frozen=True) +class _PidfdHandle: + """One stable kernel handle bound to an exact recorded identity. + + Attributes + ---------- + identity : ProcessIdentity + Identity verified immediately before and after opening the handle. + descriptor : int + Owned pidfd closed by :class:`_PidfdRegistry`. + + Examples + -------- + >>> _PidfdHandle(ProcessIdentity("worker", 2, 3), 4).descriptor + 4 + """ + + identity: ProcessIdentity + descriptor: int + + +class _PidfdRegistry: + """Retain and signal stable handles without check-to-signal PID races. + + >>> registry = _PidfdRegistry() + >>> current = _record_process("self", os.getpid()) + >>> registry.retain(current) + True + >>> registry.retained == (current,) + True + >>> registry.close() + + Parameters + ---------- + _start_time_reader : collections.abc.Callable | None + Private deterministic test seam for the two identity reads. + """ + + def __init__( + self, + *, + _start_time_reader: cabc.Callable[[int], int | None] | None = None, + ) -> None: + """Create an empty registry after checking platform capability. + + >>> registry = _PidfdRegistry() + >>> registry.retained + () + >>> registry.close() + """ + capability_error = _pidfd_capability_error() + if capability_error is not None: + raise RuntimeError(capability_error) + self._start_time_reader = _start_time_reader or _process_start_time + self._handles: dict[tuple[int, int], _PidfdHandle] = {} + self.errors: list[str] = [] + + @property + def retained(self) -> tuple[ProcessIdentity, ...]: + """Return identities with live registry-owned handles. + + >>> registry = _PidfdRegistry() + >>> registry.retained + () + >>> registry.close() + """ + return tuple(handle.identity for handle in self._handles.values()) + + def retain(self, identity: ProcessIdentity) -> bool: + """Bind one identity across pre-open and post-open start-time checks. + + >>> registry = _PidfdRegistry() + >>> registry.retain(dataclasses.replace( + ... _record_process("self", os.getpid()), start_time=-1 + ... )) + False + >>> registry.close() + + Parameters + ---------- + identity : ProcessIdentity + Exact PID and start time learned from owned progress. + + Returns + ------- + bool + Whether a stable handle is retained for the exact identity. + """ + key = (identity.pid, identity.start_time) + if key in self._handles: + return True + if self._start_time_reader(identity.pid) != identity.start_time: + return False + try: + descriptor = os.pidfd_open(identity.pid, 0) + except OSError as error: + self.errors.append( + f"{identity.role} pid {identity.pid} pidfd_open: " + f"{type(error).__name__}: {error}" + ) + return False + if self._start_time_reader(identity.pid) != identity.start_time: + os.close(descriptor) + return False + self._handles[key] = _PidfdHandle(identity, descriptor) + return True + + def retain_many(self, identities: t.Iterable[ProcessIdentity]) -> None: + """Attempt to retain every newly learned identity. + + >>> registry = _PidfdRegistry() + >>> registry.retain_many(()) + >>> registry.retained + () + >>> registry.close() + """ + for identity in identities: + self.retain(identity) + + def signal(self, identity: ProcessIdentity, number: signal.Signals) -> bool: + """Signal an exact retained handle without consulting its numeric PID. + + >>> registry = _PidfdRegistry() + >>> registry.signal(ProcessIdentity("absent", 2, 3), signal.SIGTERM) + False + >>> registry.close() + + Parameters + ---------- + identity : ProcessIdentity + Previously retained exact identity. + number : signal.Signals + Signal delivered through the stable pidfd. + + Returns + ------- + bool + Whether the kernel accepted the pidfd signal. + """ + handle = self._handles.get((identity.pid, identity.start_time)) + if handle is None: + return False + try: + signal.pidfd_send_signal( + handle.descriptor, + number, + None, + 0, + ) + except ProcessLookupError: + return False + except OSError as error: + self.errors.append( + f"{identity.role} pid {identity.pid} pidfd signal {number}: " + f"{type(error).__name__}: {error}" + ) + return False + return True + + def close(self) -> None: + """Close every retained descriptor exactly once. + + >>> registry = _PidfdRegistry() + >>> registry.close() + >>> registry.close() + """ + handles = tuple(self._handles.values()) + self._handles.clear() + for handle in handles: + os.close(handle.descriptor) + + _UNIX_SOCKET_PATH_MAX_BYTES = 107 @@ -3403,44 +3714,30 @@ def _stop_owned_process( message = "process cleanup grace must be positive" raise ValueError(message) errors: list[str] = [] - process.poll() - if process.returncode is None: - if process_identity_matches(identity): - try: - os.kill(identity.pid, signal.SIGTERM) - except ProcessLookupError: - pass - except OSError as error: - errors.append( - f"{identity.role} pid {identity.pid} SIGTERM: " - f"{type(error).__name__}: {error}" - ) - else: - errors.append( - f"{identity.role} pid {identity.pid} identity changed before SIGTERM" - ) - try: - process.wait(timeout=grace_s) - except subprocess.TimeoutExpired: - if process_identity_matches(identity): - try: - os.kill(identity.pid, signal.SIGKILL) - except ProcessLookupError: - pass - except OSError as error: - errors.append( - f"{identity.role} pid {identity.pid} SIGKILL: " - f"{type(error).__name__}: {error}" - ) - else: - errors.append( - f"{identity.role} pid {identity.pid} identity changed " - "before SIGKILL" - ) + try: + registry = _PidfdRegistry() + except RuntimeError as error: + return CleanupReport( + complete=False, + errors=(str(error),), + processes_absent=not process_identity_matches(identity), + ) + try: + registry.retain(identity) + process.poll() + if process.returncode is None: + registry.signal(identity, signal.SIGTERM) try: process.wait(timeout=grace_s) except subprocess.TimeoutExpired: - errors.append(f"{identity.role} pid {identity.pid} was not reaped") + registry.signal(identity, signal.SIGKILL) + try: + process.wait(timeout=grace_s) + except subprocess.TimeoutExpired: + errors.append(f"{identity.role} pid {identity.pid} was not reaped") + errors.extend(registry.errors) + finally: + registry.close() if process_identity_matches(identity): errors.append( f"{identity.role} pid {identity.pid} with start time " @@ -3453,6 +3750,126 @@ def _stop_owned_process( ) +def _kill_exact_tmux_socket( + socket_path: pathlib.Path, + *, + timeout_s: float, +) -> tuple[str, ...]: + """Boundedly request ``kill-server`` on one exact isolated socket. + + >>> _kill_exact_tmux_socket(pathlib.Path("missing.sock"), timeout_s=0.01) + () + + Parameters + ---------- + socket_path : pathlib.Path + Exact isolated socket owned by one benchmark run. + timeout_s : float + Maximum wait for the helper and each stable-handle escalation. + + Returns + ------- + tuple[str, ...] + Empty only when the socket is absent after the bounded request. + + Raises + ------ + ValueError + If the timeout is not positive. + """ + if timeout_s <= 0: + message = "exact socket cleanup timeout must be positive" + raise ValueError(message) + if not socket_path.exists(): + return () + environment = os.environ.copy() + environment.pop("TMUX", None) + environment.pop("TMUX_PANE", None) + errors: list[str] = [] + try: + helper = subprocess.Popen( + ("tmux", "-S", str(socket_path), "kill-server"), + env=environment, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + except OSError as error: + return (f"exact socket kill-server: {type(error).__name__}: {error}",) + try: + identity = _record_process("tmux-kill-server", helper.pid) + except RuntimeError: + try: + helper.wait(timeout=timeout_s) + except subprocess.TimeoutExpired: + errors.append("exact socket kill-server identity unavailable") + else: + registry = _PidfdRegistry() + try: + registry.retain(identity) + try: + helper.wait(timeout=timeout_s) + except subprocess.TimeoutExpired: + registry.signal(identity, signal.SIGTERM) + try: + helper.wait(timeout=timeout_s) + except subprocess.TimeoutExpired: + registry.signal(identity, signal.SIGKILL) + try: + helper.wait(timeout=timeout_s) + except subprocess.TimeoutExpired: + errors.append("exact socket kill-server helper remains") + errors.extend(registry.errors) + finally: + registry.close() + if helper.returncode == 0 and socket_path.exists(): + try: + socket_status = socket_path.lstat() + if not stat.S_ISSOCK(socket_status.st_mode): + errors.append(f"configured socket was replaced: {socket_path}") + else: + socket_path.unlink() + except OSError as error: + errors.append(f"exact socket path removal: {type(error).__name__}: {error}") + return tuple(errors) + + +def _remove_proven_stale_socket( + socket_path: pathlib.Path, + identities: t.Iterable[ProcessIdentity], +) -> tuple[str, ...]: + """Remove a stale socket only after its recorded server is absent. + + >>> _remove_proven_stale_socket(pathlib.Path("missing.sock"), ()) + () + + Parameters + ---------- + socket_path : pathlib.Path + Exact configured socket after a bounded ``kill-server`` attempt. + identities : collections.abc.Iterable[ProcessIdentity] + Recorded ownership evidence including the exact server identity. + + Returns + ------- + tuple[str, ...] + Empty only when the path is absent or safely removed. + """ + if not socket_path.exists(): + return () + servers = tuple(identity for identity in identities if identity.role == "server") + if not servers or any(process_identity_matches(identity) for identity in servers): + return (f"socket ownership is not proven absent: {socket_path}",) + try: + status = socket_path.lstat() + if not stat.S_ISSOCK(status.st_mode) or status.st_uid != os.getuid(): + return (f"configured socket was replaced: {socket_path}",) + socket_path.unlink() + except OSError as error: + return (f"stale socket removal: {type(error).__name__}: {error}",) + return () if not socket_path.exists() else (f"socket remains: {socket_path}",) + + def _stream_name(ordinal: int, delayed_ordinal: int) -> str: """Return the Task 1 stream name for one global stable pane ordinal. @@ -3751,6 +4168,9 @@ def start_fuzzer( frame_rate_hz: float = 40.0, duration_s: float = 300.0, _identity_out: list[ProcessIdentity] | None = None, + _identity_callback: ( + cabc.Callable[[tuple[ProcessIdentity, ...]], None] | None + ) = None, ) -> subprocess.Popen[bytes]: """Start the paused Task 1 service and wait for its exact ready marker. @@ -3777,6 +4197,9 @@ def start_fuzzer( _identity_out : list[ProcessIdentity] | None Internal ownership handoff populated with the identity captured at spawn time before this function returns. + _identity_callback : collections.abc.Callable | None + Private worker hook invoked immediately after the child identity is + captured and before readiness is awaited. Returns ------- @@ -3852,14 +4275,16 @@ def start_fuzzer( identity: ProcessIdentity | None = None try: identity = _record_process("fuzzer", process.pid) + if _identity_out is not None: + _identity_out.append(identity) + if _identity_callback is not None: + _identity_callback((identity,)) _wait_for_fuzzer_ready( process, ready, run_id, timeout_s=ready_timeout_s, ) - if _identity_out is not None: - _identity_out.append(identity) except BaseException as startup_error: if identity is None: start_time = _process_start_time(process.pid) @@ -3868,23 +4293,21 @@ def start_fuzzer( if identity is None: process.poll() if process.returncode is None: - process.terminate() - try: - process.wait(timeout=1.0) - except subprocess.TimeoutExpired: - cleanup_error = ( - f"fuzzer pid {process.pid} identity unavailable; " - "child was not reaped" - ) - cleanup_report = CleanupReport( - complete=False, - errors=(cleanup_error,), - ) - else: - cleanup_report = CleanupReport(complete=True) + cleanup_error = ( + f"fuzzer pid {process.pid} identity unavailable; " + "refused unsafe PID signaling" + ) + cleanup_report = CleanupReport( + complete=False, + errors=(cleanup_error,), + processes_absent=False, + ) else: process.wait() - cleanup_report = CleanupReport(complete=True) + cleanup_report = CleanupReport( + complete=True, + processes_absent=True, + ) else: cleanup_report = _stop_owned_process(process, identity) if not cleanup_report.complete: @@ -3903,7 +4326,9 @@ def _prepare_context( socket_path: pathlib.Path | None, run_id: str, delayed_ordinal: int, - _process_identity_callback: cabc.Callable[[ProcessIdentity], None] | None = None, + _process_identity_callback: ( + cabc.Callable[[tuple[ProcessIdentity, ...]], None] | None + ) = None, ) -> RunContext: """Create isolated host resources and an engine without starting setup timing. @@ -3983,6 +4408,7 @@ def _prepare_context( socket_root_created = False fuzzer: subprocess.Popen[bytes] | None = None fuzzer_identity: ProcessIdentity | None = None + process_handles: _PidfdRegistry | None = None try: from libtmux.experimental.engines import ( AsyncControlModeEngine, @@ -3997,15 +4423,21 @@ def _prepare_context( if socket_root is not None: _acquire_private_directory(socket_root) socket_root_created = True + process_handles = _PidfdRegistry() + + def publish_identities(delta: tuple[ProcessIdentity, ...]) -> None: + process_handles.retain_many(delta) + if _process_identity_callback is not None: + _process_identity_callback(delta) + fuzzer_identities: list[ProcessIdentity] = [] fuzzer = start_fuzzer( resolved_scratch, run_id, _identity_out=fuzzer_identities, + _identity_callback=publish_identities, ) fuzzer_identity = _only_process_identity(fuzzer_identities) - if _process_identity_callback is not None: - _process_identity_callback(fuzzer_identity) server = Server(socket_path=resolved_socket, config_file=os.devnull) if mode is ExecutionMode.SYNC: engine: TmuxEngine | AsyncTmuxEngine @@ -4058,12 +4490,16 @@ def _prepare_context( setup_duration_ns=0, processes=(fuzzer_identity,), ambient_tmux_environment=ambient_tmux_environment, - process_identity_callback=_process_identity_callback, + process_identity_callback=publish_identities, + process_handles=process_handles, ) except BaseException as setup_error: cleanup_errors: list[str] = [] if fuzzer is not None and fuzzer_identity is not None: cleanup_errors.extend(_stop_owned_process(fuzzer, fuzzer_identity).errors) + if process_handles is not None: + cleanup_errors.extend(process_handles.errors) + process_handles.close() for label, directory, acquired in ( ("socket root", socket_root, socket_root_created), ("scratch", resolved_scratch, scratch_created), @@ -4102,7 +4538,9 @@ def setup_sync( socket_path: pathlib.Path | None = None, run_id: str = "run-0", delayed_ordinal: int = 0, - _process_identity_callback: cabc.Callable[[ProcessIdentity], None] | None = None, + _process_identity_callback: ( + cabc.Callable[[tuple[ProcessIdentity, ...]], None] | None + ) = None, ) -> RunContext: """Build and exactly verify one synchronous live topology. @@ -4180,19 +4618,25 @@ def setup_sync( engine = t.cast("TmuxEngine", context.engine) keepalive = f"bench-{run_id}-keepalive" try: + bootstrap = SubprocessEngine.for_server(context.server) + run( + NewSession( + session_name=keepalive, + window_shell="exec tail -n 0 -f /dev/null", + ), + bootstrap, + ).raise_for_status() + run( + SetOption(server=True, option="exit-empty", value="off"), + bootstrap, + ).raise_for_status() + server_pid_result = run(DisplayMessage(message="#{pid}"), bootstrap) + server_pid_result.raise_for_status() + server_identity = _record_process("server", int(server_pid_result.text)) + context.processes = (*context.processes, server_identity) + if context.process_identity_callback is not None: + context.process_identity_callback((server_identity,)) if lane is EngineLane.CONTROL: - bootstrap = SubprocessEngine.for_server(context.server) - run( - NewSession( - session_name=keepalive, - window_shell="exec tail -n 0 -f /dev/null", - ), - bootstrap, - ).raise_for_status() - run( - SetOption(server=True, option="exit-empty", value="off"), - bootstrap, - ).raise_for_status() run(ListSessions(), engine).raise_for_status() workspaces = build_workspaces( @@ -4225,15 +4669,8 @@ def setup_sync( ), engine, ).raise_for_status() - run(KillSession(target=NameRef(keepalive)), engine).raise_for_status() + run(KillSession(target=NameRef(keepalive)), engine).raise_for_status() context.setup_duration_ns = time.perf_counter_ns() - started_ns - server_pid_result = run(DisplayMessage(message="#{pid}"), engine) - server_pid_result.raise_for_status() - server_pid = int(server_pid_result.text) - server_identity = _record_process("server", server_pid) - context.processes = (*context.processes, server_identity) - if context.process_identity_callback is not None: - context.process_identity_callback(server_identity) verify_topology(context, snapshot_topology_sync(context)) except BaseException as setup_error: try: @@ -4260,7 +4697,9 @@ async def setup_async( socket_path: pathlib.Path | None = None, run_id: str = "run-0", delayed_ordinal: int = 0, - _process_identity_callback: cabc.Callable[[ProcessIdentity], None] | None = None, + _process_identity_callback: ( + cabc.Callable[[tuple[ProcessIdentity, ...]], None] | None + ) = None, ) -> RunContext: """Build and exactly verify one asynchronous live topology. @@ -4339,23 +4778,29 @@ async def setup_async( engine = t.cast("AsyncTmuxEngine", context.engine) keepalive = f"bench-{run_id}-keepalive" try: + bootstrap = AsyncSubprocessEngine.for_server(context.server) + ( + await arun( + NewSession( + session_name=keepalive, + window_shell="exec tail -n 0 -f /dev/null", + ), + bootstrap, + ) + ).raise_for_status() + ( + await arun( + SetOption(server=True, option="exit-empty", value="off"), + bootstrap, + ) + ).raise_for_status() + server_pid_result = await arun(DisplayMessage(message="#{pid}"), bootstrap) + server_pid_result.raise_for_status() + server_identity = _record_process("server", int(server_pid_result.text)) + context.processes = (*context.processes, server_identity) + if context.process_identity_callback is not None: + context.process_identity_callback((server_identity,)) if lane is EngineLane.CONTROL: - bootstrap = AsyncSubprocessEngine.for_server(context.server) - ( - await arun( - NewSession( - session_name=keepalive, - window_shell="exec tail -n 0 -f /dev/null", - ), - bootstrap, - ) - ).raise_for_status() - ( - await arun( - SetOption(server=True, option="exit-empty", value="off"), - bootstrap, - ) - ).raise_for_status() control = t.cast(AsyncControlModeEngine, engine) await control.start() (await arun(ListSessions(), engine)).raise_for_status() @@ -4396,17 +4841,8 @@ async def setup_async( ) ).raise_for_status() control.set_attach_targets([real_session_id]) - ( - await arun(KillSession(target=NameRef(keepalive)), engine) - ).raise_for_status() + (await arun(KillSession(target=NameRef(keepalive)), engine)).raise_for_status() context.setup_duration_ns = time.perf_counter_ns() - started_ns - server_pid_result = await arun(DisplayMessage(message="#{pid}"), engine) - server_pid_result.raise_for_status() - server_pid = int(server_pid_result.text) - server_identity = _record_process("server", server_pid) - context.processes = (*context.processes, server_identity) - if context.process_identity_callback is not None: - context.process_identity_callback(server_identity) verify_topology(context, await snapshot_topology_async(context)) except BaseException as setup_error: try: @@ -4644,8 +5080,7 @@ def fail(detail: str) -> t.NoReturn: ) identity_callback = getattr(context, "process_identity_callback", None) if identity_callback is not None: - for identity in pane_processes: - identity_callback(identity) + identity_callback(pane_processes) context.topology_verified = True return snapshot @@ -5659,6 +6094,7 @@ def wait_capture_poll_sync( break if isinstance(engine, SubprocessEngine): command = engine.connection.argv(*encode_direct_argv(rendered)) + handles = _PidfdRegistry() try: process = subprocess.Popen( command, @@ -5670,16 +6106,37 @@ def wait_capture_poll_sync( start_new_session=True, ) except FileNotFoundError: + handles.close() raise exc.TmuxCommandNotFound from None try: - stdout, stderr = process.communicate(timeout=remaining_s) - except subprocess.TimeoutExpired as error: - with contextlib.suppress(OSError): - os.killpg(process.pid, signal.SIGKILL) - stdout, stderr = process.communicate() - del stdout, stderr - message = f"capture wait timed out for request {request.request_id}" - raise TimeoutError(message) from error + try: + recorded_identity = _record_process("capture-wait", process.pid) + except RuntimeError: + identity = None + else: + identity = recorded_identity + handles.retain(recorded_identity) + try: + stdout, stderr = process.communicate(timeout=remaining_s) + except subprocess.TimeoutExpired as error: + if identity is None or not handles.signal(identity, signal.SIGKILL): + message = ( + "capture wait could not safely stop its timed-out " + "subprocess" + ) + raise RuntimeError(message) from error + try: + stdout, stderr = process.communicate(timeout=0.1) + except subprocess.TimeoutExpired as cleanup_error: + message = ( + "capture wait subprocess did not exit after escalation" + ) + raise RuntimeError(message) from cleanup_error + del stdout, stderr + message = f"capture wait timed out for request {request.request_id}" + raise TimeoutError(message) from error + finally: + handles.close() stdout_lines = stdout.split("\n") while stdout_lines and stdout_lines[-1] == "": stdout_lines.pop() @@ -7747,6 +8204,7 @@ async def run_repeatable_phase( [str, str, int, PhaseMeasurement, RawSample | None], object ] | None = None, + boundary_callback: cabc.Callable[[str], object] | None = None, ) -> RepeatablePhaseResult: """Deterministically interleave strategies and retain accepted timed rows. @@ -7785,6 +8243,8 @@ async def run_repeatable_phase( progress_callback : collections.abc.Callable | None Private orchestration hook called after every accepted warmup or timed invocation. Timed calls receive the newly retained raw sample. + boundary_callback : collections.abc.Callable | None + Private hook called with the exact strategy before every invocation. Returns ------- @@ -7816,6 +8276,8 @@ async def run_repeatable_phase( for strategy in cycle_order: order.append(strategy) try: + if boundary_callback is not None: + await _await_if_needed(boundary_callback(strategy)) resources_before = sampler() produced = strategies[strategy]() measurement = _validated_phase_measurement( @@ -7938,6 +8400,10 @@ async def cleanup_run( message = "cleanup grace must be positive" raise ValueError(message) errors: list[str] = [] + registry = getattr(context, "process_handles", None) + if registry is None: + registry = _PidfdRegistry() + registry.retain_many(context.processes) stop_path = context.scratch / "fuzzer" / "stop.json" if stop_path.parent.exists(): try: @@ -7962,6 +8428,8 @@ async def cleanup_run( context.server.kill() except Exception as error: # noqa: BLE001 errors.append(f"server kill: {type(error).__name__}: {error}") + if context.socket_path.exists(): + errors.extend(_kill_exact_tmux_socket(context.socket_path, timeout_s=grace_s)) survivors = await _wait_for_process_absence( context.processes, @@ -7972,17 +8440,7 @@ async def cleanup_run( if not survivors: break for identity in survivors: - if not process_identity_matches(identity): - continue - try: - os.kill(identity.pid, signal_number) - except ProcessLookupError: - continue - except OSError as error: - errors.append( - f"{identity.role} pid {identity.pid} signal " - f"{signal_number}: {type(error).__name__}: {error}" - ) + registry.signal(identity, signal_number) survivors = await _wait_for_process_absence( context.processes, timeout_s=grace_s, @@ -7997,16 +8455,30 @@ async def cleanup_run( for identity in survivors if process_identity_matches(identity) ) - try: - if context.scratch.exists(): - shutil.rmtree(context.scratch) - except OSError as error: - errors.append(f"scratch removal: {type(error).__name__}: {error}") - try: - if context.socket_root is not None and context.socket_root.exists(): - shutil.rmtree(context.socket_root) - except OSError as error: - errors.append(f"socket root removal: {type(error).__name__}: {error}") + processes_absent = all( + not process_identity_matches(identity) for identity in context.processes + ) + if processes_absent and context.socket_path.exists(): + errors.extend( + _remove_proven_stale_socket(context.socket_path, context.processes) + ) + socket_absent = not context.socket_path.exists() + if processes_absent and socket_absent: + try: + if context.scratch.exists(): + shutil.rmtree(context.scratch) + except OSError as error: + errors.append(f"scratch removal: {type(error).__name__}: {error}") + try: + if context.socket_root is not None and context.socket_root.exists(): + shutil.rmtree(context.socket_root) + except OSError as error: + errors.append(f"socket root removal: {type(error).__name__}: {error}") + else: + if not processes_absent: + errors.append("owned processes remain; retained scratch evidence") + if not socket_absent: + errors.append("configured socket remains; retained scratch evidence") if context.socket_path.exists(): errors.append(f"socket remains: {context.socket_path}") if context.scratch.exists(): @@ -8028,6 +8500,8 @@ async def cleanup_run( os.environ.pop(name, None) else: os.environ[name] = value + errors.extend(registry.errors) + registry.close() processes_absent = all( not process_identity_matches(identity) for identity in context.processes ) @@ -8036,7 +8510,7 @@ async def cleanup_run( context.socket_root is None or not context.socket_root.exists() ) return CleanupReport( - complete=not errors, + complete=(not errors and processes_absent and socket_absent and scratch_absent), errors=tuple(errors), processes_absent=processes_absent, socket_absent=socket_absent, @@ -8290,7 +8764,7 @@ def append_progress_event(path: pathlib.Path, event: ProgressEvent) -> None: path : pathlib.Path Append-only worker-to-supervisor progress stream. event : ProgressEvent - Complete event with cumulative process identities. + Complete event with only the identities first learned at this checkpoint. Returns ------- @@ -8301,18 +8775,32 @@ def append_progress_event(path: pathlib.Path, event: ProgressEvent) -> None: encoded = ( json.dumps(_json_value(event), separators=(",", ":"), sort_keys=True) + "\n" ).encode() - flags = os.O_APPEND | os.O_CREAT | os.O_WRONLY + flags = os.O_APPEND | os.O_WRONLY if hasattr(os, "O_NOFOLLOW"): flags |= os.O_NOFOLLOW - descriptor = os.open(path, flags, 0o600) + created = False + try: + descriptor = os.open(path, flags | os.O_CREAT | os.O_EXCL, 0o600) + created = True + except FileExistsError: + descriptor = os.open(path, flags) try: - written = os.write(descriptor, encoded) - if written != len(encoded): - message = "short append to progress stream" - raise OSError(message) + remaining = memoryview(encoded) + while remaining: + written = os.write(descriptor, remaining) + if written <= 0: + message = "zero-byte append to progress stream" + raise OSError(message) + remaining = remaining[written:] os.fsync(descriptor) finally: os.close(descriptor) + if created: + parent_descriptor = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY) + try: + os.fsync(parent_descriptor) + finally: + os.close(parent_descriptor) def _progress_event_from_json(value: object) -> ProgressEvent: @@ -8364,7 +8852,7 @@ class _WorkerRecorder: sequence : int Last published sequence number. identities : list[ProcessIdentity] - Cumulative exact identities published with every event. + Cumulative exact identities retained for one terminal report copy. Examples -------- @@ -8386,7 +8874,12 @@ class _WorkerRecorder: sequence: int = -1 identities: list[ProcessIdentity] = dataclasses.field(default_factory=list) - def checkpoint(self, name: str) -> None: + def checkpoint( + self, + name: str, + *, + identity_delta: tuple[ProcessIdentity, ...] = (), + ) -> None: """Publish an increasing event before its matching report checkpoint. >>> with tempfile.TemporaryDirectory() as directory: @@ -8402,7 +8895,6 @@ def checkpoint(self, name: str) -> None: self.sequence += 1 self.report = dataclasses.replace( self.report, - processes=tuple(self.identities), progress_sequence=self.sequence, ) append_progress_event( @@ -8412,7 +8904,7 @@ def checkpoint(self, name: str) -> None: sequence=self.sequence, checkpoint=name, monotonic_ns=time.monotonic_ns(), - processes=tuple(self.identities), + processes=identity_delta, ), ) write_json_atomic(self.checkpoint_path, self.report) @@ -8431,11 +8923,51 @@ def record_identity(self, identity: ProcessIdentity) -> None: >>> len(recorder.identities) 1 """ - key = (identity.role, identity.pid, identity.start_time) - if key in {(item.role, item.pid, item.start_time) for item in self.identities}: + self.record_identities((identity,), checkpoint=f"identity.{identity.role}") + + def record_identities( + self, + identities: t.Iterable[ProcessIdentity], + *, + checkpoint: str | None = None, + ) -> None: + """Publish one constant-count or batched identity delta exactly once. + + >>> recorder = _WorkerRecorder( + ... RunReport(Topology(1, 1, 1), run_id="run-7"), + ... pathlib.Path("unused"), pathlib.Path("unused-progress"), + ... ) + >>> recorder.identities.extend((ProcessIdentity("worker", 2, 3),)) + >>> len(recorder.identities) + 1 + + Parameters + ---------- + identities : collections.abc.Iterable[ProcessIdentity] + Newly learned exact identities, possibly one verified pane batch. + checkpoint : str | None + Stable delta checkpoint label. When omitted, a homogeneous delta + uses its role and a mixed delta uses ``identities``. + """ + known = { + (identity.role, identity.pid, identity.start_time) + for identity in self.identities + } + delta: list[ProcessIdentity] = [] + for identity in identities: + key = (identity.role, identity.pid, identity.start_time) + if key in known: + continue + known.add(key) + delta.append(identity) + if not delta: return - self.identities.append(identity) - self.checkpoint(f"identity.{identity.role}") + self.identities.extend(delta) + if checkpoint is None: + roles = {identity.role for identity in delta} + role = roles.pop() if len(roles) == 1 else "identities" + checkpoint = f"identity.{role}" + self.checkpoint(checkpoint, identity_delta=tuple(delta)) async def _worker_live_postcondition( @@ -8522,6 +9054,7 @@ async def _run_worker_group( policy: ResourcePolicy, latest: dict[str, PhaseMeasurement], fail_after: str | None, + active_phase: list[str], ) -> None: """Run one deterministically interleaved family and checkpoint each call. @@ -8530,7 +9063,7 @@ async def _run_worker_group( ... await _run_worker_group( ... t.cast(_WorkerRecorder, None), t.cast(RunContext, None), {}, ... warmup=0, runs=1, seed=1, policy=ResourcePolicy(), latest={}, - ... fail_after=None, + ... fail_after=None, active_phase=["setup"], ... ) ... except ValueError as error: ... return str(error) @@ -8557,6 +9090,8 @@ async def _run_worker_group( Destination retaining the latest typed result by cell. fail_after : str | None Private test-harness phase that raises after its final checkpoint. + active_phase : list[str] + Single-item mutable exact boundary owned by :func:`run_worker`. Returns ------- @@ -8566,6 +9101,7 @@ async def _run_worker_group( if not strategies: message = "worker phase group requires strategies" raise ValueError(message) + active_phase[0] = next(iter(strategies)) topology = context.topology for name in strategies: recorder.report = _replace_phase( @@ -8591,14 +9127,17 @@ async def on_progress( latest[strategy] = measurement phases = {phase.name: phase for phase in recorder.report.phases} phase = phases[strategy] - if sample is not None: + observation = _observation_for_measurement(strategy, ordinal, measurement) + if stage == "warmup": + phase = dataclasses.replace( + phase, + warmup_observations=(*phase.warmup_observations, observation), + ) + elif sample is not None: phase = dataclasses.replace( phase, samples=(*phase.samples, sample), - observations=( - *phase.observations, - _observation_for_measurement(strategy, ordinal, measurement), - ), + observations=(*phase.observations, observation), ) if ordinal == runs - 1: phase = _completed_phase(phase) @@ -8623,6 +9162,7 @@ async def on_progress( context, measurement, policy ), progress_callback=on_progress, + boundary_callback=lambda strategy: active_phase.__setitem__(0, strategy), ) if result.failure is not None: phases = {phase.name: phase for phase in recorder.report.phases} @@ -8663,6 +9203,52 @@ def _position_target(ids: tuple[str, ...], position: str) -> str: raise ValueError(message) from error +_ShieldedResult = t.TypeVar("_ShieldedResult") + + +async def _drain_task_under_repeated_cancellation( + task: asyncio.Task[_ShieldedResult], + *, + initial_cancellation: asyncio.CancelledError | None = None, +) -> tuple[_ShieldedResult, asyncio.CancelledError | None]: + """Drain an independently owned task before re-raising first cancellation. + + >>> async def example(): + ... task = asyncio.create_task(asyncio.sleep(0, result=7)) + ... result, cancellation = await _drain_task_under_repeated_cancellation(task) + ... return result, cancellation + >>> asyncio.run(example()) + (7, None) + + Parameters + ---------- + task : asyncio.Task + Independently owned finalization task that must reach a terminal state. + initial_cancellation : asyncio.CancelledError | None + Cancellation already caught before finalization began. + + Returns + ------- + tuple[typing.Any, asyncio.CancelledError | None] + Finalization result and the first cancellation to re-raise directly. + + Raises + ------ + BaseException + Finalization failure after the task has been drained. + """ + cancellation = initial_cancellation + while not task.done(): + try: + await asyncio.shield(task) + except asyncio.CancelledError as error: # noqa: PERF203 + if cancellation is None: + cancellation = error + except BaseException: # noqa: BLE001 + break + return task.result(), cancellation + + async def run_worker( topology: Topology, *, @@ -8795,8 +9381,10 @@ async def run_worker( progress_path, stall_after=stall_after, ) - recorder.identities.append(_record_process("worker", os.getpid())) - recorder.checkpoint("worker.started") + recorder.record_identities( + (_record_process("worker", os.getpid()),), + checkpoint="worker.started", + ) if extra_identity is not None: recorder.record_identity(extra_identity) context: RunContext | None = None @@ -8805,6 +9393,8 @@ async def run_worker( terminal_error: str | None = None cleanup = CleanupReport(False) latest: dict[str, PhaseMeasurement] = {} + active_phase = ["setup"] + cancellation: asyncio.CancelledError | None = None def fail_if_requested(phase: str) -> None: if fail_after == phase: @@ -8823,7 +9413,7 @@ def fail_if_requested(phase: str) -> None: socket_path=socket_path, run_id=run_id, delayed_ordinal=delayed_ordinal, - _process_identity_callback=recorder.record_identity, + _process_identity_callback=recorder.record_identities, ) else: context = await setup_async( @@ -8833,7 +9423,7 @@ def fail_if_requested(phase: str) -> None: socket_path=socket_path, run_id=run_id, delayed_ordinal=delayed_ordinal, - _process_identity_callback=recorder.record_identity, + _process_identity_callback=recorder.record_identities, ) resources_after = probe_host(ProcessReader()) setup_duration = max(1, context.setup_duration_ns) @@ -8876,6 +9466,7 @@ def fail_if_requested(phase: str) -> None: recorder.checkpoint("setup") fail_if_requested("setup") + active_phase[0] = "stabilization" stabilization_started = time.perf_counter_ns() release_activity_gate(context) if mode is ExecutionMode.SYNC: @@ -8933,6 +9524,7 @@ async def mutation_async_call() -> MutationResult: policy=policy, latest=latest, fail_after=fail_after, + active_phase=active_phase, ) wait_counter = 0 @@ -8979,8 +9571,10 @@ async def wait_control_call() -> WaitResult: policy=policy, latest=latest, fail_after=fail_after, + active_phase=active_phase, ) if not control_applicable: + active_phase[0] = "wait.control-stream" recorder.report = _replace_phase( recorder.report, PhaseReport( @@ -9021,6 +9615,7 @@ async def wait_control_call() -> WaitResult: policy=policy, latest=latest, fail_after=fail_after, + active_phase=active_phase, ) capture_strategies: dict[str, cabc.Callable[[], object]] = {} @@ -9048,10 +9643,12 @@ async def wait_control_call() -> WaitResult: policy=policy, latest=latest, fail_after=fail_after, + active_phase=active_phase, ) from libtmux._internal.query_list import QueryList + active_phase[0] = "search.classic.sessions.first" snapshot = ( snapshot_topology_sync(context) if mode is ExecutionMode.SYNC @@ -9113,8 +9710,10 @@ async def wait_control_call() -> WaitResult: policy=policy, latest=latest, fail_after=fail_after, + active_phase=active_phase, ) + active_phase[0] = "verification" final_snapshot = ( snapshot_topology_sync(context) if mode is ExecutionMode.SYNC @@ -9124,6 +9723,7 @@ async def wait_control_call() -> WaitResult: recorder.checkpoint("verification") fail_if_requested("verification") except asyncio.CancelledError as error: + cancellation = error terminal_status = "cutoff" failed_phase = "cancellation" terminal_error = f"CancelledError: {error}" @@ -9140,61 +9740,74 @@ async def wait_control_call() -> WaitResult: terminal_error = str(error) except BaseException as error: # noqa: BLE001 terminal_status = "failed" - failed_phase = failed_phase or "setup" + failed_phase = failed_phase or active_phase[0] terminal_error = f"{type(error).__name__}: {error}" finally: - if context is not None: - try: - cleanup = await cleanup_run(context) - except BaseException as error: # noqa: BLE001 - cleanup = CleanupReport( - False, - (f"cleanup raised {type(error).__name__}: {error}",), - processes_absent=False, - socket_absent=not socket_path.exists(), - scratch_absent=not scratch.exists(), + + async def finalize_worker() -> None: + nonlocal cleanup, terminal_status, failed_phase, terminal_error + if context is not None: + try: + cleanup = await cleanup_run(context) + except BaseException as error: # noqa: BLE001 + cleanup = CleanupReport( + False, + (f"cleanup raised {type(error).__name__}: {error}",), + processes_absent=False, + socket_absent=not socket_path.exists(), + scratch_absent=not scratch.exists(), + ) + else: + owned = tuple( + identity + for identity in recorder.identities + if identity.role != "worker" and identity is not extra_identity ) - else: - owned = tuple( - identity - for identity in recorder.identities - if identity.role != "worker" and identity is not extra_identity - ) - cleanup = CleanupReport( - complete=( - all(not process_identity_matches(identity) for identity in owned) - and not socket_path.exists() - and not scratch.exists() - ), - errors=(), - processes_absent=all( + processes_absent = all( not process_identity_matches(identity) for identity in owned + ) + socket_absent = not socket_path.exists() + scratch_absent = not scratch.exists() + cleanup = CleanupReport( + complete=(processes_absent and socket_absent and scratch_absent), + errors=(), + processes_absent=processes_absent, + socket_absent=socket_absent, + scratch_absent=scratch_absent, + ) + if not cleanup.complete: + terminal_status = "failed" + failed_phase = failed_phase or "cleanup" + cleanup_detail = "; ".join(cleanup.errors) or "cleanup incomplete" + terminal_error = ( + f"{terminal_error}; {cleanup_detail}" + if terminal_error + else cleanup_detail + ) + recorder.report = dataclasses.replace( + recorder.report, + status=terminal_status, + cleanup=cleanup, + maximum_completed=( + terminal_status == "completed" + and topology == Topology(100, 100, 4) + and recorder.report.observed_topology == topology ), - socket_absent=not socket_path.exists(), - scratch_absent=not scratch.exists(), + failed_phase=failed_phase, + error=terminal_error, ) - if not cleanup.complete: - terminal_status = "failed" - failed_phase = failed_phase or "cleanup" - cleanup_detail = "; ".join(cleanup.errors) or "cleanup incomplete" - terminal_error = ( - f"{terminal_error}; {cleanup_detail}" - if terminal_error - else cleanup_detail - ) - recorder.report = dataclasses.replace( - recorder.report, - status=terminal_status, - cleanup=cleanup, - maximum_completed=( - terminal_status == "completed" - and topology == Topology(100, 100, 4) - and recorder.report.observed_topology == topology - ), - failed_phase=failed_phase, - error=terminal_error, + recorder.checkpoint("cleanup") + + finalization = asyncio.create_task( + finalize_worker(), + name="orchestration-worker-finalization", ) - recorder.checkpoint("cleanup") + _result, deferred_cancellation = await _drain_task_under_repeated_cancellation( + finalization, + initial_cancellation=cancellation, + ) + if deferred_cancellation is not None: + raise deferred_cancellation return recorder.report @@ -9202,6 +9815,8 @@ def _read_progress_chunk( path: pathlib.Path, offset: int, remainder: str, + *, + terminal: bool = False, ) -> tuple[tuple[ProgressEvent, ...], int, str]: """Read only newly appended complete JSONL records. @@ -9220,6 +9835,8 @@ def _read_progress_chunk( Previously consumed byte offset. remainder : str Incomplete decoded line retained from the prior read. + terminal : bool + Whether the producer has exited and any incomplete tail is corruption. Returns ------- @@ -9231,14 +9848,24 @@ def _read_progress_chunk( stream.seek(offset) chunk = stream.read() new_offset = stream.tell() - except FileNotFoundError: + except FileNotFoundError as error: + if terminal: + message = "progress journal is missing at terminal drain" + raise ValueError(message) from error return (), offset, remainder combined = remainder + chunk pieces = combined.split("\n") trailing = pieces.pop() - events = tuple( - _progress_event_from_json(json.loads(line)) for line in pieces if line - ) + if terminal and trailing: + message = "progress journal has a torn terminal record" + raise ValueError(message) + try: + events = tuple( + _progress_event_from_json(json.loads(line)) for line in pieces if line + ) + except (json.JSONDecodeError, TypeError, ValueError) as error: + message = "progress journal contains an invalid event" + raise ValueError(message) from error return events, new_offset, trailing @@ -9265,27 +9892,6 @@ def _wait_identity_absence( time.sleep(0.02) -def _signal_worker_group( - worker: subprocess.Popen[bytes], - identity: ProcessIdentity, - signal_number: signal.Signals, -) -> None: - """Signal an isolated worker group only while its leader identity matches. - - >>> finished = subprocess.Popen((sys.executable, "-c", "pass")) - >>> finished.wait(timeout=1) - 0 - >>> _signal_worker_group( - ... finished, ProcessIdentity("worker", finished.pid, -1), signal.SIGTERM - ... ) - """ - worker.poll() - if worker.returncode is not None or not process_identity_matches(identity): - return - with contextlib.suppress(ProcessLookupError): - os.killpg(identity.pid, signal_number) - - def _remove_supervised_scratch(path: pathlib.Path) -> tuple[str, ...]: """Remove one exact private directory without following replacement links. @@ -9312,7 +9918,7 @@ def _remove_supervised_scratch(path: pathlib.Path) -> tuple[str, ...]: def _recover_supervised_run( worker: subprocess.Popen[bytes], worker_identity: ProcessIdentity, - identities: tuple[ProcessIdentity, ...], + progress: _ProgressTracker, *, scratch: pathlib.Path, socket_path: pathlib.Path, @@ -9324,11 +9930,19 @@ def _recover_supervised_run( >>> finished.wait(timeout=1) 0 >>> with tempfile.TemporaryDirectory() as directory: - ... scratch = pathlib.Path(directory) / "absent" + ... root = pathlib.Path(directory) + ... journal = root / "progress.jsonl" + ... identity = ProcessIdentity("worker", finished.pid, -1) + ... append_progress_event( + ... journal, ProgressEvent("run-7", 0, "worker.started", 1) + ... ) + ... registry = _PidfdRegistry() + ... progress = _ProgressTracker(journal, "run-7", registry) ... report = _recover_supervised_run( - ... finished, ProcessIdentity("worker", finished.pid, -1), (), - ... scratch=scratch, socket_path=scratch / "sock", grace_s=0.01, + ... finished, identity, progress, scratch=root / "absent", + ... socket_path=root / "absent" / "sock", grace_s=0.01, ... ) + ... registry.close() ... report.complete True @@ -9338,8 +9952,8 @@ def _recover_supervised_run( Direct child launched in its own session. worker_identity : ProcessIdentity Exact group-leader identity captured immediately after spawn. - identities : tuple[ProcessIdentity, ...] - Cumulative progress identities, possibly including already absent PIDs. + progress : _ProgressTracker + Live journal state and stable handles for newly learned identities. scratch : pathlib.Path Exact private worker directory. socket_path : pathlib.Path @@ -9356,35 +9970,43 @@ def _recover_supervised_run( message = "supervisor cleanup grace must be positive" raise ValueError(message) errors: list[str] = [] - _signal_worker_group(worker, worker_identity, signal.SIGTERM) - try: - worker.wait(timeout=grace_s) - except subprocess.TimeoutExpired: - _signal_worker_group(worker, worker_identity, signal.SIGKILL) + progress.handles.retain(worker_identity) + + def drain(*, terminal: bool = False) -> bool: try: - worker.wait(timeout=grace_s) - except subprocess.TimeoutExpired: - errors.append("worker process group did not exit") + return progress.drain(terminal=terminal) + except (RuntimeError, ValueError) as error: + detail = f"progress journal: {type(error).__name__}: {error}" + if detail not in errors: + errors.append(detail) + return False + + def wait_worker() -> bool: + deadline = time.monotonic() + grace_s + while worker.poll() is None and time.monotonic() < deadline: + drain() + time.sleep(0.02) + drain() + return worker.poll() is not None + + drain() + progress.handles.signal(worker_identity, signal.SIGTERM) + if not wait_worker(): + progress.handles.signal(worker_identity, signal.SIGKILL) + if not wait_worker(): + errors.append("worker process did not exit") + drain(terminal=worker.poll() is not None) unique: dict[tuple[int, int], ProcessIdentity] = {} - for identity in (worker_identity, *identities): + for identity in (worker_identity, *progress.identities): unique[(identity.pid, identity.start_time)] = identity owned = tuple(unique.values()) + progress.handles.retain_many(owned) survivors = _wait_identity_absence(owned, timeout_s=grace_s) for signal_number in (signal.SIGTERM, signal.SIGKILL): if not survivors: break for identity in survivors: - if not process_identity_matches(identity): - continue - try: - os.kill(identity.pid, signal_number) - except ProcessLookupError: - continue - except OSError as error: - errors.append( - f"{identity.role} pid {identity.pid} signal {signal_number}: " - f"{type(error).__name__}: {error}" - ) + progress.handles.signal(identity, signal_number) survivors = _wait_identity_absence(owned, timeout_s=grace_s) errors.extend( f"{identity.role} pid {identity.pid} with start time " @@ -9392,9 +10014,19 @@ def _recover_supervised_run( for identity in survivors if process_identity_matches(identity) ) - errors.extend(_remove_supervised_scratch(scratch)) processes_absent = not any(process_identity_matches(identity) for identity in owned) + if socket_path.exists(): + errors.extend(_kill_exact_tmux_socket(socket_path, timeout_s=grace_s)) + if processes_absent and socket_path.exists(): + errors.extend(_remove_proven_stale_socket(socket_path, owned)) socket_absent = not socket_path.exists() + errors.extend(progress.handles.errors) + if progress.journal_error is not None and not any( + progress.journal_error in error for error in errors + ): + errors.append(f"progress journal: {progress.journal_error}") + if not errors and processes_absent and socket_absent: + errors.extend(_remove_supervised_scratch(scratch)) scratch_absent = not scratch.exists() if not socket_absent: errors.append(f"socket remains: {socket_path}") @@ -9478,12 +10110,374 @@ def _accept_progress_events( _validate_progress_run_id(event, run_id) if event.sequence <= highest_sequence: continue + if event.sequence != highest_sequence + 1: + message = ( + "progress journal sequence gap: expected " + f"{highest_sequence + 1}, observed {event.sequence}" + ) + raise RuntimeError(message) highest_sequence = event.sequence identities = _merge_identities(identities, event.processes) advanced = True return highest_sequence, identities, advanced +@dataclasses.dataclass +class _ProgressTracker: + """Incrementally merge one journal while retaining newly learned pidfds. + + Attributes + ---------- + path : pathlib.Path + Append-only progress journal. + run_id : str + Exact producer identity accepted from every event. + handles : _PidfdRegistry + Registry that binds each accepted identity delta. + identities : tuple[ProcessIdentity, ...] + Cumulative identities retained once for the terminal report. + highest_sequence : int + Last strictly increasing accepted sequence. + offset : int + Consumed journal byte offset. + remainder : str + Incomplete active tail retained between reads. + journal_error : str | None + First corruption detected while draining. + + Examples + -------- + >>> with tempfile.TemporaryDirectory() as directory: + ... root = pathlib.Path(directory) + ... registry = _PidfdRegistry() + ... tracker = _ProgressTracker(root / "missing", "run-7", registry) + ... tracker.drain() + ... registry.close() + False + """ + + path: pathlib.Path + run_id: str + handles: _PidfdRegistry + identities: tuple[ProcessIdentity, ...] = () + highest_sequence: int = -1 + offset: int = 0 + remainder: str = "" + journal_error: str | None = None + + def drain(self, *, terminal: bool = False) -> bool: + """Accept new complete records and bind every accepted identity delta. + + >>> with tempfile.TemporaryDirectory() as directory: + ... root = pathlib.Path(directory) + ... path = root / "progress.jsonl" + ... append_progress_event(path, ProgressEvent("run-7", 0, "start", 1)) + ... registry = _PidfdRegistry() + ... tracker = _ProgressTracker(path, "run-7", registry) + ... tracker.drain(terminal=True) + ... registry.close() + True + + Parameters + ---------- + terminal : bool + Whether a missing journal or nonempty final tail is corruption. + + Returns + ------- + bool + Whether the highest accepted sequence increased. + + Raises + ------ + RuntimeError + If event ownership or sequence continuity is invalid. + ValueError + If terminal JSONL evidence is missing, torn, or malformed. + """ + try: + events, offset, remainder = _read_progress_chunk( + self.path, + self.offset, + self.remainder, + terminal=terminal, + ) + highest, identities, advanced = _accept_progress_events( + events, + run_id=self.run_id, + highest_sequence=self.highest_sequence, + identities=self.identities, + ) + except (RuntimeError, ValueError) as error: + if self.journal_error is None: + self.journal_error = f"{type(error).__name__}: {error}" + raise + self.offset = offset + self.remainder = remainder + self.highest_sequence = highest + self.identities = identities + self.handles.retain_many( + identity for event in events for identity in event.processes + ) + return advanced + + +_FinalizerResult = t.TypeVar("_FinalizerResult") + + +def _drain_finalizer_thread( + callback: cabc.Callable[[], _FinalizerResult], + *, + initial_interrupt: KeyboardInterrupt | None = None, +) -> tuple[_FinalizerResult, KeyboardInterrupt | None]: + """Drain independent synchronous finalization through repeated interrupts. + + >>> result, interruption = _drain_finalizer_thread(lambda: 7) + >>> result, interruption is None + (7, True) + + Parameters + ---------- + callback : collections.abc.Callable + Recovery, report, or ramp finalization that must run to completion. + initial_interrupt : KeyboardInterrupt | None + First interruption already translated into terminal report state. + + Returns + ------- + tuple[typing.Any, KeyboardInterrupt | None] + Finalization result and the first observed interruption. + + Raises + ------ + BaseException + Callback failure after its independently owned thread terminates. + """ + results: list[_FinalizerResult] = [] + failures: list[BaseException] = [] + completed = threading.Event() + + def run() -> None: + try: + results.append(callback()) + except BaseException as error: # noqa: BLE001 + failures.append(error) + finally: + completed.set() + + thread = threading.Thread( + target=run, + name="orchestration-finalization", + ) + thread.start() + interruption = initial_interrupt + while not completed.is_set(): + try: + completed.wait(timeout=0.02) + except KeyboardInterrupt as error: # noqa: PERF203 + if interruption is None: + interruption = error + while thread.is_alive(): + try: + thread.join(timeout=0.02) + except KeyboardInterrupt as error: # noqa: PERF203 + if interruption is None: + interruption = error + except RuntimeError: + pass + if failures: + if interruption is not None: + raise interruption from failures[0] + raise failures[0] + return results[0], interruption + + +def _finalize_supervised_worker( + worker: subprocess.Popen[bytes], + worker_identity: ProcessIdentity, + progress: _ProgressTracker, + *, + topology: Topology, + lane: EngineLane, + mode: ExecutionMode, + runs: int, + warmup: int, + seed: int, + run_id: str, + scratch: pathlib.Path, + socket_path: pathlib.Path, + output: pathlib.Path, + markdown_output: pathlib.Path, + checkpoint_path: pathlib.Path, + admission_path: pathlib.Path, + guard_decision: GuardDecision, + original_guard_decision: GuardDecision, + cleanup_grace_s: float, + supervisor_status: t.Literal["failed", "cutoff"] | None, + failed_phase: str | None, + terminal_error: str | None, +) -> RunReport: + """Recover, terminally drain, validate, and render one supervised worker. + + >>> _finalize_supervised_worker.__name__ + '_finalize_supervised_worker' + + Returns + ------- + RunReport + Durable supervisor-owned terminal artifact. + """ + try: + if supervisor_status is not None: + cleanup = _recover_supervised_run( + worker, + worker_identity, + progress, + scratch=scratch, + socket_path=socket_path, + grace_s=cleanup_grace_s, + ) + else: + worker.wait() + try: + progress.drain(terminal=True) + except (RuntimeError, ValueError) as error: + supervisor_status = "failed" + failed_phase = "supervisor" + terminal_error = f"{type(error).__name__}: {error}" + if supervisor_status is not None: + cleanup = _recover_supervised_run( + worker, + worker_identity, + progress, + scratch=scratch, + socket_path=socket_path, + grace_s=cleanup_grace_s, + ) + else: + processes_absent = not any( + process_identity_matches(item) for item in progress.identities + ) + socket_absent = not socket_path.exists() + scratch_absent = not scratch.exists() + cleanup = CleanupReport( + complete=( + processes_absent + and socket_absent + and scratch_absent + and not progress.handles.errors + ), + errors=tuple(progress.handles.errors), + processes_absent=processes_absent, + socket_absent=socket_absent, + scratch_absent=scratch_absent, + ) + if not cleanup.complete: + cleanup = _recover_supervised_run( + worker, + worker_identity, + progress, + scratch=scratch, + socket_path=socket_path, + grace_s=cleanup_grace_s, + ) + + try: + candidate = load_run_report(checkpoint_path) + except (OSError, ValueError): + candidate = RunReport( + topology, + status="in_progress", + cleanup=CleanupReport(False), + guard_decision=guard_decision, + original_guard_decision=original_guard_decision, + run_id=run_id, + lane=lane.value, + mode=mode.value, + warmup=warmup, + runs=runs, + scratch_path=str(scratch), + socket_path=str(socket_path), + progress_path=str(progress.path), + environment=_collect_environment( + seed=seed, command_line=("run", "--shape", str(topology)) + ), + ) + if supervisor_status is None: + if worker.returncode == 0 and candidate.status == "completed": + final_status: t.Literal["completed", "failed", "cutoff"] = "completed" + elif candidate.status == "cutoff": + final_status = "cutoff" + else: + final_status = "failed" + if final_status != "completed": + failed_phase = candidate.failed_phase or "worker" + terminal_error = candidate.error or f"worker exited {worker.returncode}" + else: + final_status = supervisor_status + if not cleanup.complete: + final_status = "failed" + failed_phase = "cleanup" + detail = "; ".join(cleanup.errors) or "cleanup verification failed" + terminal_error = f"{terminal_error}; {detail}" if terminal_error else detail + final_report = dataclasses.replace( + candidate, + status=final_status, + cleanup=cleanup, + maximum_completed=( + final_status == "completed" + and topology == Topology(100, 100, 4) + and candidate.observed_topology == topology + ), + failed_phase=(None if final_status == "completed" else failed_phase), + error=(None if final_status == "completed" else terminal_error), + processes=progress.identities, + progress_path=str(progress.path), + progress_sequence=progress.highest_sequence, + guard_decision=( + candidate.guard_decision + if candidate.guard_decision is not None + else guard_decision + ), + original_guard_decision=original_guard_decision, + ) + write_json_atomic(output, final_report) + validate_report(final_report) + render_markdown_summary(output, markdown_output) + admission_path.unlink(missing_ok=True) + return final_report + finally: + progress.handles.close() + + +def _publish_interrupted_supervisor_report( + report: RunReport, + output: pathlib.Path, + markdown_output: pathlib.Path, +) -> RunReport: + """Replace a post-work completion with durable cancellation evidence. + + >>> _publish_interrupted_supervisor_report.__name__ + '_publish_interrupted_supervisor_report' + + Returns + ------- + RunReport + Validated cutoff report with the already verified cleanup evidence. + """ + interrupted = dataclasses.replace( + report, + status="cutoff", + maximum_completed=False, + failed_phase="cancellation", + error="KeyboardInterrupt: supervisor interrupted during finalization", + ) + write_json_atomic(output, interrupted) + validate_report(interrupted) + render_markdown_summary(output, markdown_output) + return interrupted + + def supervise_worker( topology: Topology, *, @@ -9620,25 +10614,22 @@ def supervise_worker( start_new_session=True, ) worker_identity = _record_process("worker", worker.pid) - identities: tuple[ProcessIdentity, ...] = (worker_identity,) - highest_sequence = -1 - offset = 0 - remainder = "" + handles = _PidfdRegistry() + handles.retain(worker_identity) + progress = _ProgressTracker( + progress_path, + run_id, + handles, + identities=(worker_identity,), + ) deadline = time.monotonic() + watchdog_s supervisor_status: t.Literal["failed", "cutoff"] | None = None failed_phase: str | None = None terminal_error: str | None = None + interruption: KeyboardInterrupt | None = None try: while worker.poll() is None: - events, offset, remainder = _read_progress_chunk( - progress_path, offset, remainder - ) - highest_sequence, identities, advanced = _accept_progress_events( - events, - run_id=run_id, - highest_sequence=highest_sequence, - identities=identities, - ) + advanced = progress.drain() if advanced: deadline = time.monotonic() + watchdog_s if time.monotonic() >= deadline: @@ -9647,7 +10638,8 @@ def supervise_worker( terminal_error = f"progress watchdog expired after {watchdog_s} seconds" break time.sleep(0.02) - except KeyboardInterrupt: + except KeyboardInterrupt as error: + interruption = error supervisor_status = "cutoff" failed_phase = "cancellation" terminal_error = "KeyboardInterrupt: supervisor interrupted" @@ -9656,111 +10648,42 @@ def supervise_worker( failed_phase = "supervisor" terminal_error = f"{type(error).__name__}: {error}" - if supervisor_status is not None: - cleanup = _recover_supervised_run( + final_report, _interruption = _drain_finalizer_thread( + lambda: _finalize_supervised_worker( worker, worker_identity, - identities, + progress, + topology=topology, + lane=lane, + mode=mode, + runs=runs, + warmup=warmup, + seed=seed, + run_id=run_id, scratch=scratch, socket_path=socket_path, - grace_s=cleanup_grace_s, - ) - else: - worker.wait() - events, offset, remainder = _read_progress_chunk( - progress_path, offset, remainder - ) - highest_sequence, identities, _advanced = _accept_progress_events( - events, - run_id=run_id, - highest_sequence=highest_sequence, - identities=identities, - ) - cleanup = CleanupReport( - complete=( - not any(process_identity_matches(item) for item in identities) - and not socket_path.exists() - and not scratch.exists() - ), - processes_absent=not any( - process_identity_matches(item) for item in identities - ), - socket_absent=not socket_path.exists(), - scratch_absent=not scratch.exists(), - ) - if not cleanup.complete: - cleanup = _recover_supervised_run( - worker, - worker_identity, - identities, - scratch=scratch, - socket_path=socket_path, - grace_s=cleanup_grace_s, - ) - - try: - candidate = load_run_report(checkpoint_path) - except (OSError, ValueError): - candidate = RunReport( - topology, - status="in_progress", - cleanup=CleanupReport(False), + output=output, + markdown_output=markdown_output, + checkpoint_path=checkpoint_path, + admission_path=admission_path, guard_decision=guard_decision, original_guard_decision=original_guard_decision, - run_id=run_id, - lane=lane.value, - mode=mode.value, - warmup=warmup, - runs=runs, - scratch_path=str(scratch), - socket_path=str(socket_path), - progress_path=str(progress_path), - environment=_collect_environment( - seed=seed, command_line=("run", "--shape", str(topology)) - ), - ) - if supervisor_status is None: - if worker.returncode == 0 and candidate.status == "completed": - final_status: t.Literal["completed", "failed", "cutoff"] = "completed" - elif candidate.status == "cutoff": - final_status = "cutoff" - else: - final_status = "failed" - if final_status != "completed": - failed_phase = candidate.failed_phase or "worker" - terminal_error = candidate.error or f"worker exited {worker.returncode}" - else: - final_status = supervisor_status - if not cleanup.complete: - final_status = "failed" - failed_phase = "cleanup" - detail = "; ".join(cleanup.errors) or "cleanup verification failed" - terminal_error = f"{terminal_error}; {detail}" if terminal_error else detail - final_report = dataclasses.replace( - candidate, - status=final_status, - cleanup=cleanup, - maximum_completed=( - final_status == "completed" - and topology == Topology(100, 100, 4) - and candidate.observed_topology == topology - ), - failed_phase=(None if final_status == "completed" else failed_phase), - error=(None if final_status == "completed" else terminal_error), - processes=identities, - progress_path=str(progress_path), - progress_sequence=highest_sequence, - guard_decision=( - candidate.guard_decision - if candidate.guard_decision is not None - else guard_decision + cleanup_grace_s=cleanup_grace_s, + supervisor_status=supervisor_status, + failed_phase=failed_phase, + terminal_error=terminal_error, ), - original_guard_decision=original_guard_decision, + initial_interrupt=interruption, ) - write_json_atomic(output, final_report) - validate_report(final_report) - render_markdown_summary(output, markdown_output) - admission_path.unlink(missing_ok=True) + if _interruption is not None and final_report.status == "completed": + final_report, _interruption = _drain_finalizer_thread( + lambda: _publish_interrupted_supervisor_report( + final_report, + output, + markdown_output, + ), + initial_interrupt=_interruption, + ) return final_report @@ -10015,6 +10938,17 @@ def run_scenario( snapshot = host_snapshot or probe_host(ProcessReader()) original = predict_resources(topology, snapshot, policy) decision = _forced_decision(original, force_extreme) + capability_error = _pidfd_capability_error() + if decision.allowed and capability_error is not None: + decision = GuardDecision( + False, + "predictive_refusal", + "pidfd_capability", + None, + None, + False, + snapshot, + ) run_id = f"r{uuid.uuid4().hex[:10]}" markdown_output = markdown_output or output.with_suffix(".md") if not decision.allowed: @@ -10027,7 +10961,7 @@ def run_scenario( socket_absent=True, scratch_absent=True, ), - guard_decision=original, + guard_decision=decision, original_guard_decision=original, run_id=run_id, lane=lane.value, @@ -10035,7 +10969,11 @@ def run_scenario( warmup=warmup, runs=runs, failed_phase="preflight", - error=f"predictive refusal: {original.rule or 'unknown'}", + error=( + capability_error + if capability_error is not None and original.allowed + else f"predictive refusal: {original.rule or 'unknown'}" + ), ) write_json_atomic(output, report) validate_report(report) @@ -10067,6 +11005,92 @@ def run_scenario( ) +def _finalize_ramp_aggregation( + report: RunReport, + steps: t.Sequence[RampStep], + *, + last_observed: Topology | None, + terminal_status: t.Literal["refused", "failed", "cutoff"] | None, + terminal_reason: str | None, + output: pathlib.Path, + markdown_output: pathlib.Path, +) -> RunReport: + """Validate child cleanup and durably publish one terminal ramp aggregate. + + >>> _finalize_ramp_aggregation.__name__ + '_finalize_ramp_aggregation' + + Returns + ------- + RunReport + Validated aggregate retaining every declared step. + """ + cleanup_complete = all( + step.status == "not_attempted" + or load_run_report(pathlib.Path(t.cast(str, step.report_path))).cleanup.complete + for step in steps + ) + cleanup = CleanupReport( + cleanup_complete, + processes_absent=cleanup_complete, + socket_absent=cleanup_complete, + scratch_absent=cleanup_complete, + ) + final_status: t.Literal["completed", "refused", "failed", "cutoff"] = ( + terminal_status or "completed" + ) + terminal = dataclasses.replace( + report, + status=final_status, + observed_topology=last_observed, + cleanup=cleanup, + ramp=tuple(steps), + error=terminal_reason, + ) + write_json_atomic(output, terminal) + validate_report(terminal) + render_markdown_summary(output, markdown_output) + return terminal + + +def _publish_interrupted_ramp_report( + report: RunReport, + output: pathlib.Path, + markdown_output: pathlib.Path, +) -> RunReport: + """Publish cancellation that arrived while a completed ramp aggregated. + + >>> _publish_interrupted_ramp_report.__name__ + '_publish_interrupted_ramp_report' + + Returns + ------- + RunReport + Validated cutoff aggregate with the final attempt marked cutoff. + """ + reason = "KeyboardInterrupt: ramp interrupted during aggregation" + steps = list(report.ramp) + completed_indices = tuple( + index for index, step in enumerate(steps) if step.status == "completed" + ) + if not completed_indices: + return report + final_index = completed_indices[-1] + final = steps[final_index] + steps[final_index] = dataclasses.replace(final, status="cutoff", reason=reason) + interrupted = dataclasses.replace( + report, + status="cutoff", + maximum_completed=False, + ramp=tuple(steps), + error=reason, + ) + write_json_atomic(output, interrupted) + validate_report(interrupted) + render_markdown_summary(output, markdown_output) + return interrupted + + def run_ramp( shapes: t.Sequence[Topology], *, @@ -10181,31 +11205,26 @@ def run_ramp( ramp=tuple(steps), ) write_json_atomic(output, report) - cleanup_complete = all( - step.status == "not_attempted" - or load_run_report(pathlib.Path(t.cast(str, step.report_path))).cleanup.complete - for step in steps - ) - cleanup = CleanupReport( - cleanup_complete, - processes_absent=cleanup_complete, - socket_absent=cleanup_complete, - scratch_absent=cleanup_complete, - ) - final_status: t.Literal["completed", "refused", "failed", "cutoff"] = ( - terminal_status or "completed" - ) - report = dataclasses.replace( - report, - status=final_status, - observed_topology=last_observed, - cleanup=cleanup, - ramp=tuple(steps), - error=terminal_reason, + report, interruption = _drain_finalizer_thread( + lambda: _finalize_ramp_aggregation( + report, + steps, + last_observed=last_observed, + terminal_status=terminal_status, + terminal_reason=terminal_reason, + output=output, + markdown_output=markdown_output, + ) ) - write_json_atomic(output, report) - validate_report(report) - render_markdown_summary(output, markdown_output) + if interruption is not None and report.status == "completed": + report, _interruption = _drain_finalizer_thread( + lambda: _publish_interrupted_ramp_report( + report, + output, + markdown_output, + ), + initial_interrupt=interruption, + ) return report diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index a1fcb74d21..85218f01fb 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -359,7 +359,12 @@ def completed_report(benchmark_module: types.ModuleType) -> t.Any: requested_topology=topology, observed_topology=topology, phases=(phase,), - cleanup=benchmark_module.CleanupReport(complete=True), + cleanup=benchmark_module.CleanupReport( + complete=True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), maximum_completed=True, ) @@ -528,11 +533,11 @@ def test_validate_report_rejects_summary_with_failed_sample( ) -@pytest.mark.parametrize("status", ("refused", "failed", "cutoff")) +@pytest.mark.parametrize("status", ("refused", "cutoff")) def test_validate_report_requires_cleanup_for_terminal_status( benchmark_module: types.ModuleType, status: str ) -> None: - """Terminal evidence is invalid if it leaves benchmark-owned state behind.""" + """Successful refusal or cutoff cleanup cannot leave owned state behind.""" report = dataclasses.replace( completed_report(benchmark_module), status=status, @@ -1011,6 +1016,174 @@ def test_owned_process_cleanup_kills_sigterm_ignoring_child( process.wait(timeout=2.0) +def test_pidfd_registry_signals_only_the_retained_real_child( + benchmark_module: types.ModuleType, +) -> None: + """A retained pidfd must terminate its exact child without a PID lookup.""" + assert benchmark_module._pidfd_capability_error() is None + target = subprocess.Popen( + (sys.executable, "-c", "import time; time.sleep(60)"), + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + unrelated = subprocess.Popen( + (sys.executable, "-c", "import time; time.sleep(60)"), + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + registry = benchmark_module._PidfdRegistry() + try: + identity = benchmark_module._record_process("target", target.pid) + assert registry.retain(identity) + assert registry.signal(identity, signal.SIGTERM) + assert target.wait(timeout=2.0) == -signal.SIGTERM + assert unrelated.poll() is None + finally: + registry.close() + if target.poll() is None: + target.kill() + target.wait(timeout=2.0) + if unrelated.poll() is None: + unrelated.terminate() + unrelated.wait(timeout=2.0) + + +def test_pidfd_registry_rejects_reuse_between_open_checks( + benchmark_module: types.ModuleType, +) -> None: + """A PID reused between precheck/open/postcheck must never be signaled.""" + child = subprocess.Popen( + (sys.executable, "-c", "import time; time.sleep(60)"), + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + identity = benchmark_module._record_process("target", child.pid) + observed = iter((identity.start_time, identity.start_time + 1)) + registry = benchmark_module._PidfdRegistry( + _start_time_reader=lambda _pid: next(observed), + ) + try: + assert not registry.retain(identity) + assert not registry.signal(identity, signal.SIGKILL) + assert child.poll() is None + assert registry.retained == () + finally: + registry.close() + child.terminate() + child.wait(timeout=2.0) + + +def test_pidfd_registry_rejects_an_already_mismatched_identity( + benchmark_module: types.ModuleType, +) -> None: + """An unrelated process at a recorded PID must remain untouched.""" + child = subprocess.Popen( + (sys.executable, "-c", "import time; time.sleep(60)"), + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + current = benchmark_module._record_process("unrelated", child.pid) + stale = dataclasses.replace(current, start_time=current.start_time + 1) + registry = benchmark_module._PidfdRegistry() + try: + assert not registry.retain(stale) + assert not registry.signal(stale, signal.SIGKILL) + assert child.poll() is None + finally: + registry.close() + child.terminate() + child.wait(timeout=2.0) + + +def test_exact_socket_fallback_kills_only_the_configured_tmux_server( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """Fallback cleanup must address one socket and preserve another daemon.""" + target = tmp_path / "target.sock" + unrelated = tmp_path / "unrelated.sock" + for socket_path in (target, unrelated): + created = subprocess.run( + ( + "tmux", + "-S", + str(socket_path), + "new-session", + "-d", + "-s", + "keepalive", + ), + check=False, + capture_output=True, + text=True, + ) + assert created.returncode == 0, created.stderr + try: + errors = benchmark_module._kill_exact_tmux_socket(target, timeout_s=1.0) + + assert errors == () + assert not target.exists() + still_alive = subprocess.run( + ("tmux", "-S", str(unrelated), "list-sessions"), + check=False, + capture_output=True, + text=True, + ) + assert still_alive.returncode == 0, still_alive.stderr + finally: + subprocess.run( + ("tmux", "-S", str(unrelated), "kill-server"), + check=False, + capture_output=True, + text=True, + ) + + +def test_run_scenario_refuses_when_pidfds_are_unavailable( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """Unsupported hosts must fail closed before the hidden worker is spawned.""" + monkeypatch.setattr( + benchmark_module, + "_pidfd_capability_error", + lambda: "pidfd signaling unavailable", + ) + + def reject_spawn(*_args: t.Any, **_kwargs: t.Any) -> t.NoReturn: + pytest.fail("unsupported pidfd host spawned a worker") + + monkeypatch.setattr(benchmark_module.subprocess, "Popen", reject_spawn) + output = tmp_path / "unsupported.json" + report = benchmark_module.run_scenario( + benchmark_module.Topology(1, 1, 1), + runs=1, + warmup=0, + output=output, + host_snapshot=benchmark_module.HostSnapshot( + available_memory_bytes=16 * 1024**3, + pids_current=1, + pids_max=100_000, + nofile_soft_limit=65_536, + ), + policy=benchmark_module.ResourcePolicy( + pid_reserve=1, + memory_floor_bytes=1, + ), + ) + + assert report.status == "refused" + assert report.failed_phase == "preflight" + assert report.error == "pidfd signaling unavailable" + assert report.cleanup.complete + assert output.exists() + + def test_start_fuzzer_reaps_child_after_readiness_timeout( benchmark_module: types.ModuleType, tmp_path: pathlib.Path, @@ -1034,6 +1207,36 @@ def test_start_fuzzer_reaps_child_after_readiness_timeout( assert run_id not in processes +def test_start_fuzzer_publishes_identity_before_readiness_wait( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A startup stall cannot hide the already spawned fuzzer from recovery.""" + published: list[t.Any] = [] + + def stop_at_readiness(*_args: t.Any, **_kwargs: t.Any) -> t.NoReturn: + assert [identity.role for identity in published] == ["fuzzer"] + message = "injected readiness stop" + raise RuntimeError(message) + + monkeypatch.setattr( + benchmark_module, + "_wait_for_fuzzer_ready", + stop_at_readiness, + ) + + with pytest.raises(RuntimeError, match="injected readiness stop"): + benchmark_module.start_fuzzer( + tmp_path, + "early-fuzzer", + _identity_callback=lambda delta: published.extend(delta), + ) + + assert len(published) == 1 + assert not benchmark_module.process_identity_matches(published[0]) + + def test_prepare_context_keeps_start_fuzzer_identity_for_partial_cleanup( benchmark_module: types.ModuleType, tmp_path: pathlib.Path, @@ -1091,6 +1294,57 @@ def capture_fuzzer(*args: t.Any, **kwargs: t.Any) -> subprocess.Popen[bytes]: assert report.complete, report.errors +def test_setup_publishes_early_constant_deltas_and_one_pane_batch( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """Server recovery identity must precede build and panes must publish once.""" + topology = benchmark_module.Topology(1, 2, 2) + scratch = tmp_path / "identity-deltas" + socket_path = scratch / "tmux.sock" + deltas: list[tuple[t.Any, ...]] = [] + context = cleanup = None + + def publish(delta: tuple[t.Any, ...]) -> None: + deltas.append(delta) + if tuple(identity.role for identity in delta) == ("server",): + listed = subprocess.run( + ( + "tmux", + "-S", + str(socket_path), + "list-sessions", + "-F", + "#{session_name}", + ), + check=True, + capture_output=True, + text=True, + ) + assert listed.stdout.splitlines() == ["bench-identity-deltas-keepalive"] + + try: + context = benchmark_module.setup_sync( + topology, + benchmark_module.EngineLane.SUBPROCESS, + scratch, + socket_path=socket_path, + run_id="identity-deltas", + delayed_ordinal=1, + _process_identity_callback=publish, + ) + finally: + if context is not None: + cleanup = asyncio.run(benchmark_module.cleanup_run(context)) + + assert cleanup is not None and cleanup.complete, getattr(cleanup, "errors", ()) + assert [tuple(identity.role for identity in delta) for delta in deltas] == [ + ("fuzzer",), + ("server",), + ("pane", "pane", "pane", "pane"), + ] + + @pytest.mark.parametrize("mode_name", ("sync", "async")) def test_setup_surfaces_original_and_cleanup_failures( benchmark_module: types.ModuleType, @@ -1220,7 +1474,7 @@ def fail_target_stat( **kwargs: t.Any, ) -> os.stat_result: nonlocal injected - if path == target and not injected: + if path == target and os.path.lexists(path) and not injected: injected = True message = f"injected {target_kind} stat failure" raise OSError(message) @@ -1847,9 +2101,15 @@ async def exercise() -> None: release_restoration.set() with pytest.raises(asyncio.CancelledError) as cancelled: await mutation - assert cancelled.value.args == ("original cancellation",) - assert isinstance(cancelled.value.__cause__, RuntimeError) - assert str(cancelled.value.__cause__) == "restoration evidence failure" + expected_args = ( + () if sys.version_info[:2] == (3, 10) else ("original cancellation",) + ) + assert cancelled.value.args == expected_args + if sys.version_info[:2] == (3, 10): + assert cancelled.value.__cause__ is None + else: + assert isinstance(cancelled.value.__cause__, RuntimeError) + assert str(cancelled.value.__cause__) == "restoration evidence failure" restored = await benchmark_module.snapshot_topology_async(context) option_result = ( @@ -2382,6 +2642,11 @@ def test_sync_subprocess_capture_timeout_kills_and_reaps_exact_child( executable, pid_path = _blocking_tmux_binary(tmp_path) monkeypatch.setenv("BLOCKING_TMUX_PID", str(pid_path)) + monkeypatch.setattr( + benchmark_module.os, + "killpg", + lambda *_args: pytest.fail("capture timeout used an unstable process group"), + ) engine = SubprocessEngine(tmux_bin=executable) context = _isolated_wait_context( benchmark_module, @@ -2603,7 +2868,8 @@ async def exercise() -> tuple[list[dict[str, t.Any]], tuple[str, ...]]: with pytest.raises(asyncio.CancelledError) as captured: await waiter await asyncio.sleep(0) - assert captured.value.args == ("caller-stop",) + expected_args = () if sys.version_info[:2] == (3, 10) else ("caller-stop",) + assert captured.value.args == expected_args assert engine._subscribers == set() names = tuple( task.get_name() @@ -3480,6 +3746,7 @@ def test_cli_run_executes_every_phase_and_writes_validated_artifacts( assert phase["status"] == "completed" assert phase["warmup"] == 1 assert phase["runs"] == 2 + assert [row["ordinal"] for row in phase["warmup_observations"]] == [0] assert len(phase["samples"]) == 2 assert phase["summary"]["count"] == 2 assert len(phase["observations"]) == 2 @@ -3728,6 +3995,562 @@ def test_worker_progress_precedes_matching_report_checkpoint( assert calls == ["progress", "checkpoint"] +def test_progress_append_retries_short_writes_and_syncs_new_parent( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """A short kernel append must not leave a torn identity record.""" + path = tmp_path / "journal" / "progress.jsonl" + real_write = os.write + real_fsync = os.fsync + write_sizes: list[int] = [] + sync_kinds: list[str] = [] + + def short_write(descriptor: int, payload: bytes | bytearray | memoryview) -> int: + chunk = bytes(payload[:7]) + write_sizes.append(len(chunk)) + return real_write(descriptor, chunk) + + def tracked_fsync(descriptor: int) -> None: + mode = os.fstat(descriptor).st_mode + sync_kinds.append("directory" if stat.S_ISDIR(mode) else "file") + real_fsync(descriptor) + + monkeypatch.setattr(benchmark_module.os, "write", short_write) + monkeypatch.setattr(benchmark_module.os, "fsync", tracked_fsync) + event = benchmark_module.ProgressEvent( + "run-7", + 0, + "identity.fuzzer", + 1, + (benchmark_module.ProcessIdentity("fuzzer", 42, 100),), + ) + + benchmark_module.append_progress_event(path, event) + + first = json.loads(path.read_text(encoding="utf-8")) + assert first["checkpoint"] == "identity.fuzzer" + assert len(write_sizes) > 1 + assert sync_kinds == ["file", "directory"] + + benchmark_module.append_progress_event( + path, + benchmark_module.ProgressEvent("run-7", 1, "setup", 2), + ) + assert len(path.read_text(encoding="utf-8").splitlines()) == 2 + assert sync_kinds == ["file", "directory", "file"] + + +@pytest.mark.parametrize( + "tail", + ( + '{"schema_version":1', + '{"schema_version":1}\n', + ), +) +def test_terminal_progress_drain_rejects_torn_or_corrupt_tail( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + tail: str, +) -> None: + """Terminal recovery cannot silently discard malformed identity evidence.""" + path = tmp_path / "progress.jsonl" + path.write_text(tail, encoding="utf-8") + + with pytest.raises(ValueError, match="progress journal"): + benchmark_module._read_progress_chunk(path, 0, "", terminal=True) + + +def test_progress_sequence_gap_is_journal_corruption( + benchmark_module: types.ModuleType, +) -> None: + """Losing one identity delta must not look like valid later progress.""" + events = ( + benchmark_module.ProgressEvent("run-7", 0, "worker.started", 1), + benchmark_module.ProgressEvent("run-7", 2, "setup", 2), + ) + + with pytest.raises(RuntimeError, match="sequence gap"): + benchmark_module._accept_progress_events( + events, + run_id="run-7", + highest_sequence=-1, + identities=(), + ) + + +def test_recovery_drains_and_stops_identity_published_during_grace( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """Recovery must merge a late identity instead of freezing its first read.""" + scratch = tmp_path / "late-identity" + scratch.mkdir(mode=0o700) + progress = scratch / "progress.jsonl" + worker = subprocess.Popen( + ( + sys.executable, + "-c", + ( + "import signal,time; " + "signal.signal(signal.SIGTERM, signal.SIG_IGN); " + "print('ready', flush=True); time.sleep(60)" + ), + ), + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, + text=True, + ) + assert worker.stdout is not None + assert worker.stdout.readline() == "ready\n" + worker_identity = benchmark_module._record_process("worker", worker.pid) + benchmark_module.append_progress_event( + progress, + benchmark_module.ProgressEvent( + "run-7", 0, "worker.started", 1, (worker_identity,) + ), + ) + registry = benchmark_module._PidfdRegistry() + registry.retain(worker_identity) + tracker = benchmark_module._ProgressTracker( + progress, + "run-7", + registry, + identities=(worker_identity,), + ) + assert tracker.drain() + late_identities: list[t.Any] = [] + + def publish_late_identity() -> None: + time.sleep(0.05) + late = subprocess.Popen( + ( + sys.executable, + "-c", + ( + "import signal,time; " + "signal.signal(signal.SIGTERM, signal.SIG_IGN); time.sleep(60)" + ), + ), + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + identity = benchmark_module._record_process("fuzzer", late.pid) + late_identities.append(identity) + benchmark_module.append_progress_event( + progress, + benchmark_module.ProgressEvent( + "run-7", 1, "identity.fuzzer", 2, (identity,) + ), + ) + late.wait(timeout=5.0) + + publisher = threading.Thread(target=publish_late_identity) + publisher.start() + try: + cleanup = benchmark_module._recover_supervised_run( + worker, + worker_identity, + tracker, + scratch=scratch, + socket_path=scratch / "tmux.sock", + grace_s=0.15, + ) + finally: + publisher.join(timeout=5.0) + registry.close() + if worker.poll() is None: + worker.kill() + worker.wait(timeout=2.0) + + assert not publisher.is_alive() + assert len(late_identities) == 1 + assert late_identities[0] in tracker.identities + assert cleanup.complete, cleanup.errors + assert not scratch.exists() + + +@pytest.mark.parametrize("journal_state", ("missing", "torn")) +def test_recovery_retains_evidence_when_terminal_journal_is_incomplete( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + journal_state: str, +) -> None: + """Missing or torn terminal ownership evidence must make cleanup incomplete.""" + scratch = tmp_path / journal_state + scratch.mkdir(mode=0o700) + progress = scratch / "progress.jsonl" + if journal_state == "torn": + progress.write_text('{"schema_version":1', encoding="utf-8") + worker = subprocess.Popen((sys.executable, "-c", "import time; time.sleep(1)")) + worker_identity = benchmark_module._record_process("worker", worker.pid) + worker.terminate() + worker.wait(timeout=2.0) + registry = benchmark_module._PidfdRegistry() + tracker = benchmark_module._ProgressTracker( + progress, + "run-7", + registry, + identities=(worker_identity,), + ) + try: + cleanup = benchmark_module._recover_supervised_run( + worker, + worker_identity, + tracker, + scratch=scratch, + socket_path=scratch / "tmux.sock", + grace_s=0.05, + ) + finally: + registry.close() + + assert not cleanup.complete + assert any("progress journal" in error for error in cleanup.errors) + assert scratch.exists() + + +def test_progress_identity_deltas_scale_linearly( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """Checkpoint count must not multiply a large identity set on disk.""" + events: list[t.Any] = [] + reports: list[dict[str, t.Any]] = [] + monkeypatch.setattr( + benchmark_module, + "append_progress_event", + lambda _path, event: events.append(event), + ) + monkeypatch.setattr( + benchmark_module, + "write_json_atomic", + lambda _path, report: reports.append( + t.cast(dict[str, t.Any], benchmark_module._json_value(report)) + ), + ) + recorder = benchmark_module._WorkerRecorder( + benchmark_module.RunReport(benchmark_module.Topology(1, 1, 1), run_id="run-7"), + tmp_path / "checkpoint.json", + tmp_path / "progress.jsonl", + ) + identities = tuple( + benchmark_module.ProcessIdentity("pane", 100_000 + ordinal, ordinal + 1) + for ordinal in range(40_000) + ) + + recorder.record_identities(identities, checkpoint="identity.panes") + for ordinal in range(100): + recorder.checkpoint(f"timed.{ordinal}") + + assert len(events) == 101 + assert sum(len(event.processes) for event in events) == 40_000 + assert len(events[0].processes) == 40_000 + assert all(event.processes == () for event in events[1:]) + assert all(report["processes"] == [] for report in reports) + + +@pytest.mark.parametrize( + "cleanup", + ( + {"complete": True, "errors": ("socket remains",), "flags": (True, True, True)}, + {"complete": True, "errors": (), "flags": (False, True, True)}, + {"complete": True, "errors": (), "flags": (True, False, True)}, + {"complete": True, "errors": (), "flags": (True, True, False)}, + ), +) +def test_validator_rejects_false_cleanup_success( + benchmark_module: types.ModuleType, + cleanup: dict[str, t.Any], +) -> None: + """A success bit cannot override errors or any surviving resource class.""" + processes_absent, socket_absent, scratch_absent = cleanup["flags"] + report = benchmark_module.RunReport( + benchmark_module.Topology(1, 1, 1), + status="failed", + cleanup=benchmark_module.CleanupReport( + cleanup["complete"], + cleanup["errors"], + processes_absent=processes_absent, + socket_absent=socket_absent, + scratch_absent=scratch_absent, + ), + ) + + with pytest.raises(ValueError, match="cleanup"): + benchmark_module.validate_report(report) + + +def test_validator_accepts_explicit_terminal_cleanup_failure_evidence( + benchmark_module: types.ModuleType, +) -> None: + """A failed cleanup must remain publishable without claiming completion.""" + topology = benchmark_module.Topology(1, 1, 1) + report = benchmark_module.RunReport( + topology, + status="failed", + cleanup=benchmark_module.CleanupReport( + False, + ("socket remains",), + processes_absent=True, + socket_absent=False, + scratch_absent=False, + ), + run_id="run-7", + lane="control", + mode="async", + warmup=0, + runs=1, + failed_phase="cleanup", + error="socket remains", + scratch_path="scratch", + socket_path="scratch/tmux.sock", + progress_path="progress.jsonl", + environment=benchmark_module.EnvironmentReport( + "3.10", None, 1, 11, ("run",), None + ), + ) + + benchmark_module.validate_report(report) + + +def test_validator_rejects_nonprefix_terminal_phase_graph( + benchmark_module: types.ModuleType, +) -> None: + """A failed run cannot invent a later phase after omitting stabilization.""" + topology = benchmark_module.Topology(1, 1, 1) + setup_sample = benchmark_module.RawSample( + 1, + True, + verified=True, + strategy="setup", + ordinal=0, + ) + report = benchmark_module.RunReport( + topology, + observed_topology=topology, + status="failed", + phases=( + benchmark_module.PhaseReport( + "setup", + topology, + topology, + samples=(setup_sample,), + status="completed", + runs=1, + observations=(benchmark_module.PhaseObservation(0, "setup", 1),), + ), + benchmark_module.PhaseReport( + "mutation.bulk", + topology, + topology, + status="failed", + runs=1, + ), + ), + cleanup=benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + run_id="run-7", + lane="subprocess", + mode="sync", + warmup=0, + runs=1, + failed_phase="mutation.bulk", + error="injected failure", + scratch_path="scratch", + socket_path="scratch/socket", + progress_path="progress.jsonl", + environment=benchmark_module.EnvironmentReport( + "3.10", None, 1, 11, ("run",), None + ), + ) + + with pytest.raises(ValueError, match="phase prefix"): + benchmark_module.validate_report(report) + + +def test_validator_rejects_completed_phase_after_failed_prefix( + benchmark_module: types.ModuleType, +) -> None: + """No later completed row may follow the first failed phase boundary.""" + topology = benchmark_module.Topology(1, 1, 1) + report = benchmark_module.RunReport( + topology, + status="failed", + phases=( + benchmark_module.PhaseReport("setup", topology, topology, status="failed"), + benchmark_module.PhaseReport( + "stabilization", topology, topology, status="completed" + ), + ), + cleanup=benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + ) + + with pytest.raises(ValueError, match="phase status prefix"): + benchmark_module.validate_report(report) + + +def test_validator_requires_accepted_verified_setup_observation( + benchmark_module: types.ModuleType, +) -> None: + """A completed setup row cannot claim success with rejected raw evidence.""" + topology = benchmark_module.Topology(1, 1, 1) + report = benchmark_module.RunReport( + topology, + observed_topology=topology, + status="completed", + phases=( + benchmark_module.PhaseReport( + "setup", + topology, + topology, + samples=( + benchmark_module.RawSample( + 1, + False, + error="not verified", + strategy="setup", + ordinal=0, + ), + ), + status="completed", + runs=1, + observations=( + benchmark_module.PhaseObservation( + 0, + "setup", + 1, + verified=False, + ), + ), + ), + ), + cleanup=benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + ) + + with pytest.raises(ValueError, match=r"setup.*accepted.*verified"): + benchmark_module.validate_report(report) + + +def test_validator_rejects_duplicate_timed_ordinals( + benchmark_module: types.ModuleType, +) -> None: + """Duplicate timed rows cannot satisfy a declared two-run cell.""" + topology = benchmark_module.Topology(1, 1, 1) + samples = tuple( + benchmark_module.RawSample( + duration, + True, + verified=True, + strategy="mutation.bulk", + ordinal=0, + ) + for duration in (10, 20) + ) + observations = tuple( + benchmark_module.PhaseObservation( + 0, + "mutation.bulk", + duration, + ) + for duration in (10, 20) + ) + report = benchmark_module.RunReport( + topology, + status="in_progress", + phases=( + benchmark_module.PhaseReport( + "mutation.bulk", + topology, + topology, + samples=samples, + summary={ + "count": 2, + "min_ns": 10, + "mean_ns": 15, + "median_ns": 15.0, + "p90_ns": 20, + "p95_ns": 20, + "p99_ns": 20, + "max_ns": 20, + }, + status="completed", + warmup=0, + runs=2, + observations=observations, + ), + ), + ) + + with pytest.raises(ValueError, match="timed ordinals"): + benchmark_module.validate_report(report) + + +def test_validator_requires_exact_warmup_observation_ordinals( + benchmark_module: types.ModuleType, +) -> None: + """A completed cell must retain every unique declared warmup ordinal.""" + topology = benchmark_module.Topology(1, 1, 1) + phase = benchmark_module.PhaseReport( + "mutation.bulk", + topology, + topology, + samples=( + benchmark_module.RawSample( + 10, + True, + verified=True, + strategy="mutation.bulk", + ordinal=0, + ), + ), + summary={ + "count": 1, + "min_ns": 10, + "mean_ns": 10, + "median_ns": 10, + "p90_ns": 10, + "p95_ns": 10, + "p99_ns": 10, + "max_ns": 10, + }, + status="completed", + warmup=2, + runs=1, + warmup_observations=( + benchmark_module.PhaseObservation(0, "mutation.bulk", 5), + benchmark_module.PhaseObservation(0, "mutation.bulk", 6), + ), + observations=(benchmark_module.PhaseObservation(0, "mutation.bulk", 10),), + ) + report = benchmark_module.RunReport( + topology, + status="in_progress", + phases=(phase,), + ) + + with pytest.raises(ValueError, match="warmup ordinals"): + benchmark_module.validate_report(report) + + def test_cli_phase_failure_uses_supervisor_cleanup_contract( tmp_path: pathlib.Path, ) -> None: @@ -3763,6 +4586,199 @@ def test_cli_phase_failure_uses_supervisor_cleanup_contract( _assert_terminal_cleanup(payload) +def test_worker_reports_the_exact_active_stabilization_boundary( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """An unexpected lifecycle error must not be mislabeled as setup.""" + context = types.SimpleNamespace( + setup_duration_ns=1, + setup_metrics=None, + activity_pane_ids=(), + ) + + async def fake_setup(*_args: t.Any, **_kwargs: t.Any) -> t.Any: + return context + + async def fake_cleanup(_context: t.Any) -> t.Any: + return benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ) + + def fail_stabilization(_context: t.Any) -> t.NoReturn: + message = "injected stabilization boundary failure" + raise RuntimeError(message) + + monkeypatch.setattr(benchmark_module, "setup_async", fake_setup) + monkeypatch.setattr(benchmark_module, "cleanup_run", fake_cleanup) + monkeypatch.setattr(benchmark_module, "release_activity_gate", fail_stabilization) + decision = benchmark_module.GuardDecision( + True, + "ok", + None, + None, + None, + False, + benchmark_module.HostSnapshot(), + ) + + report = asyncio.run( + benchmark_module.run_worker( + benchmark_module.Topology(1, 1, 1), + lane=benchmark_module.EngineLane.SUBPROCESS, + mode=benchmark_module.ExecutionMode.ASYNC, + runs=1, + warmup=0, + seed=11, + run_id="active-stabilization", + scratch=tmp_path / "scratch", + socket_path=tmp_path / "scratch" / "tmux.sock", + checkpoint_path=tmp_path / "checkpoint.json", + progress_path=tmp_path / "progress.jsonl", + guard_decision=decision, + original_guard_decision=decision, + policy=benchmark_module.ResourcePolicy(), + ) + ) + + assert report.status == "failed" + assert report.failed_phase == "stabilization" + assert "injected stabilization boundary failure" in t.cast(str, report.error) + + +def test_cli_worker_reports_exact_not_applicable_control_boundary( + tmp_path: pathlib.Path, +) -> None: + """A non-applicable wait boundary cannot inherit the prior phase label.""" + report_path = tmp_path / "not-applicable-failure.json" + + completed = _run_cli( + "run", + "--shape", + "1x1x1", + "--lane", + "subprocess", + "--mode", + "async", + "--runs", + "1", + "--warmup", + "0", + "--output", + str(report_path), + "--scratch-root", + str(tmp_path / "scratch"), + "--watchdog-seconds", + "30", + "--_test-fail-after", + "wait.control-stream", + cwd=tmp_path, + ) + + assert completed.returncode != 0 + payload = json.loads(report_path.read_text(encoding="utf-8")) + assert payload["status"] == "failed" + assert payload["failed_phase"] == "wait.control-stream" + _assert_terminal_cleanup(payload) + + +def test_worker_drains_cleanup_through_repeated_cancellation( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """Repeated cancellation must wait for durable cleanup and preserve payload.""" + + async def exercise() -> None: + activity_started = asyncio.Event() + cleanup_started = asyncio.Event() + release_cleanup = asyncio.Event() + context = types.SimpleNamespace( + setup_duration_ns=1, + setup_metrics=None, + activity_pane_ids=(), + ) + + async def fake_setup(*_args: t.Any, **_kwargs: t.Any) -> t.Any: + return context + + async def block_activity(_context: t.Any) -> t.NoReturn: + activity_started.set() + await asyncio.Event().wait() + message = "unreachable" + raise AssertionError(message) + + async def delayed_cleanup(_context: t.Any) -> t.Any: + cleanup_started.set() + await release_cleanup.wait() + return benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ) + + monkeypatch.setattr(benchmark_module, "setup_async", fake_setup) + monkeypatch.setattr(benchmark_module, "release_activity_gate", lambda _ctx: 1) + monkeypatch.setattr(benchmark_module, "verify_activity_async", block_activity) + monkeypatch.setattr(benchmark_module, "cleanup_run", delayed_cleanup) + decision = benchmark_module.GuardDecision( + True, + "ok", + None, + None, + None, + False, + benchmark_module.HostSnapshot(), + ) + checkpoint = tmp_path / "cancel-checkpoint.json" + task = asyncio.create_task( + benchmark_module.run_worker( + benchmark_module.Topology(1, 1, 1), + lane=benchmark_module.EngineLane.SUBPROCESS, + mode=benchmark_module.ExecutionMode.ASYNC, + runs=1, + warmup=0, + seed=11, + run_id="worker-cancellation", + scratch=tmp_path / "scratch", + socket_path=tmp_path / "scratch" / "tmux.sock", + checkpoint_path=checkpoint, + progress_path=tmp_path / "cancel-progress.jsonl", + guard_decision=decision, + original_guard_decision=decision, + policy=benchmark_module.ResourcePolicy(), + ), + name="worker-cancellation-test", + ) + await activity_started.wait() + task.cancel("original cancellation") + await cleanup_started.wait() + task.cancel("repeated cancellation") + await asyncio.sleep(0) + assert not task.done() + release_cleanup.set() + with pytest.raises(asyncio.CancelledError) as cancelled: + await task + assert isinstance(cancelled.value, asyncio.CancelledError) + report = benchmark_module.load_run_report(checkpoint) + assert report.status == "cutoff" + assert report.failed_phase == "cancellation" + assert "original cancellation" in t.cast(str, report.error) + assert report.cleanup.complete + assert not [ + pending + for pending in asyncio.all_tasks() + if pending is not asyncio.current_task() and not pending.done() + ] + + asyncio.run(exercise()) + + def test_cli_runtime_cutoff_is_not_relaxed_by_force_extreme( tmp_path: pathlib.Path, ) -> None: @@ -3839,6 +4855,9 @@ def test_cli_cancellation_uses_supervisor_cleanup_contract( pytest.fail("worker never reached the injected setup stall") assert process.poll() is None process.send_signal(signal.SIGINT) + time.sleep(0.05) + if process.poll() is None: + process.send_signal(signal.SIGINT) stdout, stderr = process.communicate(timeout=20) finally: if process.poll() is None: @@ -3853,6 +4872,83 @@ def test_cli_cancellation_uses_supervisor_cleanup_contract( _assert_terminal_cleanup(payload) +def test_supervisor_cancellation_during_final_report_write_is_durable( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """An interrupt during the atomic terminal write must be reported afterward.""" + report_path = tmp_path / "final-write.json" + markdown_path = tmp_path / "final-write.md" + entered_write = threading.Event() + release_write = threading.Event() + real_write = benchmark_module.write_json_atomic + + def delayed_terminal_write(path: pathlib.Path, value: t.Any) -> None: + if ( + path == report_path + and isinstance(value, benchmark_module.RunReport) + and value.status == "completed" + ): + entered_write.set() + assert release_write.wait(timeout=20.0) + real_write(path, value) + + monkeypatch.setattr( + benchmark_module, + "write_json_atomic", + delayed_terminal_write, + ) + + def interrupt_final_write() -> None: + assert entered_write.wait(timeout=30.0) + os.kill(os.getpid(), signal.SIGINT) + time.sleep(0.05) + release_write.set() + + interrupter = threading.Thread(target=interrupt_final_write) + interrupter.start() + decision = benchmark_module.GuardDecision( + True, + "ok", + None, + None, + None, + False, + benchmark_module.HostSnapshot(), + ) + try: + report = benchmark_module.supervise_worker( + benchmark_module.Topology(1, 1, 1), + lane=benchmark_module.EngineLane.SUBPROCESS, + mode=benchmark_module.ExecutionMode.SYNC, + runs=1, + warmup=0, + seed=11, + run_id="cancel-final-write", + scratch=tmp_path / "scratch", + socket_path=tmp_path / "scratch" / "tmux.sock", + output=report_path, + markdown_output=markdown_path, + guard_decision=decision, + original_guard_decision=decision, + policy=benchmark_module.ResourcePolicy(), + watchdog_s=30.0, + cleanup_grace_s=0.3, + ) + finally: + release_write.set() + interrupter.join(timeout=5.0) + + assert not interrupter.is_alive() + assert report.status == "cutoff" + assert report.failed_phase == "cancellation" + assert report.cleanup.complete + durable = benchmark_module.load_run_report(report_path) + assert durable == report + assert "cutoff" in markdown_path.read_text(encoding="utf-8") + + def test_cli_process_identity_mismatch_never_signals_unrelated_pid( tmp_path: pathlib.Path, ) -> None: @@ -3974,6 +5070,96 @@ def test_cli_ramp_refusal_marks_later_shapes_not_attempted( assert "not_attempted" in markdown_path.read_text(encoding="utf-8") +def test_ramp_cancellation_during_aggregation_is_durable( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """Aggregation interrupted after child cleanup must publish a terminal ramp.""" + output = tmp_path / "ramp-cancel.json" + markdown = tmp_path / "ramp-cancel.md" + entered_write = threading.Event() + release_write = threading.Event() + real_write = benchmark_module.write_json_atomic + next_run = 0 + + def fake_run_scenario(shape: t.Any, **kwargs: t.Any) -> t.Any: + nonlocal next_run + next_run += 1 + child = benchmark_module.RunReport( + shape, + observed_topology=shape, + status="completed", + cleanup=benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + run_id=f"child-{next_run}", + lane="control", + mode="async", + warmup=0, + runs=1, + scratch_path=f"scratch-{next_run}", + socket_path=f"scratch-{next_run}/tmux.sock", + progress_path=f"progress-{next_run}.jsonl", + environment=benchmark_module.EnvironmentReport( + "3.10", None, 1, 11, ("run",), None + ), + ) + real_write(kwargs["output"], child) + return child + + def delayed_aggregate_write(path: pathlib.Path, value: t.Any) -> None: + if ( + path == output + and isinstance(value, benchmark_module.RunReport) + and value.status == "completed" + ): + entered_write.set() + assert release_write.wait(timeout=10.0) + real_write(path, value) + + monkeypatch.setattr(benchmark_module, "run_scenario", fake_run_scenario) + monkeypatch.setattr( + benchmark_module, + "write_json_atomic", + delayed_aggregate_write, + ) + + def interrupt_aggregation() -> None: + assert entered_write.wait(timeout=10.0) + os.kill(os.getpid(), signal.SIGINT) + time.sleep(0.05) + release_write.set() + + interrupter = threading.Thread(target=interrupt_aggregation) + interrupter.start() + report = None + try: + report = benchmark_module.run_ramp( + ( + benchmark_module.Topology(1, 1, 1), + benchmark_module.Topology(2, 1, 1), + ), + runs=1, + warmup=0, + output=output, + markdown_output=markdown, + ) + finally: + release_write.set() + interrupter.join(timeout=5.0) + + assert report is not None + assert report.status == "cutoff" + assert report.ramp[-1].status == "cutoff" + assert report.cleanup.complete + assert benchmark_module.load_run_report(output) == report + assert "cutoff" in markdown.read_text(encoding="utf-8") + + def test_cli_ramp_predictive_refusal_never_executes_tmux_binary( tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch, From 0f6a3324f1c18622fba1f489d4c9afbb0d06f5fe Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 14:05:30 -0500 Subject: [PATCH 21/67] Bench(fix[runner]): Prove safe handoff why: Runtime symbol checks and unprotected spawn setup could admit a run without proving stable signaling or preserve an interruption before cleanup completed. what: - Probe pidfd signaling before any live-run side effect - Bind worker identity under an exact temporary signal mask - Drain supervisor and ramp finalization through cancellation - Validate completed evidence preceding terminal failures --- scripts/bench_orchestration.py | 679 ++++++++++++++------- tests/test_bench_orchestration_script.py | 724 ++++++++++++++++++++++- 2 files changed, 1172 insertions(+), 231 deletions(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index e5c52edf88..db6eb37883 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -2994,48 +2994,55 @@ def _validate_executable_report(report: RunReport) -> None: if phase_names != _RUNNER_PHASES[: len(phase_names)]: message = "attempted executable report phases must be a runner phase prefix" raise ValueError(message) - if report.status != "completed": - if not report.failed_phase or not report.error: - message = "terminal unsuccessful report requires phase and error" - raise ValueError(message) - return - if report.failed_phase is not None or report.error is not None: - message = "completed report cannot carry a terminal failure" - raise ValueError(message) - if report.observed_topology != report.requested_topology: - message = "completed report requires exact observed topology" - raise ValueError(message) - if tuple(phase.name for phase in report.phases) != _RUNNER_PHASES: - message = "completed report has an incomplete phase graph" + terminal_unsuccessful = report.status in {"failed", "cutoff"} + if terminal_unsuccessful and (not report.failed_phase or not report.error): + message = "terminal unsuccessful report requires phase and error" raise ValueError(message) + if report.status == "completed": + if report.failed_phase is not None or report.error is not None: + message = "completed report cannot carry a terminal failure" + raise ValueError(message) + if report.observed_topology != report.requested_topology: + message = "completed report requires exact observed topology" + raise ValueError(message) + if tuple(phase.name for phase in report.phases) != _RUNNER_PHASES: + message = "completed report has an incomplete phase graph" + raise ValueError(message) phases = {phase.name: phase for phase in report.phases} - setup = phases["setup"] + setup = phases.get("setup") if ( - setup.status != "completed" - or setup.warmup != 0 - or setup.runs != 1 - or len(setup.samples) != 1 - or len(setup.observations) != 1 - or setup.warmup_observations - or setup.summary is not None - or not setup.samples[0].accepted - or not setup.samples[0].verified - or not setup.observations[0].verified - or setup.samples[0].ordinal != 0 - or setup.observations[0].ordinal != 0 - or setup.samples[0].strategy != "setup" - or setup.observations[0].strategy != "setup" - or setup.samples[0].duration_ns != setup.observations[0].duration_ns + setup is not None + and setup.status == "completed" + and ( + setup.warmup != 0 + or setup.runs != 1 + or len(setup.samples) != 1 + or len(setup.observations) != 1 + or setup.warmup_observations + or setup.summary is not None + or not setup.samples[0].accepted + or not setup.samples[0].verified + or not setup.observations[0].verified + or setup.samples[0].ordinal != 0 + or setup.observations[0].ordinal != 0 + or setup.samples[0].strategy != "setup" + or setup.observations[0].strategy != "setup" + or setup.samples[0].duration_ns != setup.observations[0].duration_ns + ) ): message = "setup must remain one unsummarized fresh-server observation" raise ValueError(message) - stabilization = phases["stabilization"] + stabilization = phases.get("stabilization") if ( - stabilization.status != "completed" - or stabilization.samples - or stabilization.summary is not None - or len(stabilization.observations) != 1 - or stabilization.observations[0].pane_count != report.requested_topology.panes + stabilization is not None + and stabilization.status == "completed" + and ( + stabilization.samples + or stabilization.summary is not None + or len(stabilization.observations) != 1 + or stabilization.observations[0].pane_count + != report.requested_topology.panes + ) ): message = "stabilization requires one exact untimed topology observation" raise ValueError(message) @@ -3043,33 +3050,68 @@ def _validate_executable_report(report: RunReport) -> None: report.lane == EngineLane.CONTROL.value and report.mode == ExecutionMode.ASYNC.value ) - control_phase = phases["wait.control-stream"] - if not control_applicable: - if control_phase.status != "not_applicable": + control_phase = phases.get("wait.control-stream") + if control_phase is not None: + if not control_applicable and control_phase.status != "not_applicable": message = "control-stream must be not_applicable outside async control" raise ValueError(message) - elif control_phase.status != "completed": - message = "async control report requires control-stream evidence" + if ( + control_applicable + and report.status == "completed" + and control_phase.status != "completed" + ): + message = "async control report requires control-stream evidence" + raise ValueError(message) + elif report.status == "completed": + message = "completed report is missing control-stream disposition" raise ValueError(message) - repeatable_names = (*_RUNNER_REPEATABLE_PHASES,) + active = tuple( + phase for phase in report.phases if phase.status in {"failed", "in_progress"} + ) + if terminal_unsuccessful and len(active) > 1: + message = "terminal unsuccessful report has multiple active phases" + raise ValueError(message) + failed_rows = tuple(phase for phase in active if phase.status == "failed") + if failed_rows and report.failed_phase != failed_rows[0].name: + message = "failed_phase must match the active failed phase" + raise ValueError(message) + repeatable_names = set(_RUNNER_REPEATABLE_PHASES) if control_applicable: - repeatable_names = ( - *repeatable_names[:2], - "wait.control-stream", - *repeatable_names[2:], - ) + repeatable_names.add("wait.control-stream") for name in repeatable_names: - phase = phases[name] - if ( - phase.status != "completed" - or phase.warmup != report.warmup - or phase.runs != report.runs - or len(phase.samples) != report.runs - or len(phase.observations) != report.runs - or phase.summary is None - or phase.summary.get("count") != report.runs - ): - message = f"repeatable phase {name} has incomplete raw evidence" + phase = phases.get(name) + if phase is None: + continue + if phase.status == "not_applicable": + if name != "wait.control-stream" or control_applicable: + message = f"repeatable phase {name} cannot be not_applicable" + raise ValueError(message) + continue + if phase.warmup != report.warmup or phase.runs != report.runs: + message = f"repeatable phase {name} count declaration differs from report" + raise ValueError(message) + if phase.status == "completed": + if ( + len(phase.warmup_observations) != report.warmup + or len(phase.samples) != report.runs + or len(phase.observations) != report.runs + or phase.summary is None + or phase.summary.get("count") != report.runs + ): + message = f"repeatable phase {name} has incomplete raw evidence" + raise ValueError(message) + elif phase.status in {"failed", "in_progress"}: + if ( + len(phase.warmup_observations) > report.warmup + or len(phase.samples) > report.runs + or len(phase.observations) > report.runs + or phase.summary is not None + or (phase.samples and len(phase.warmup_observations) != report.warmup) + ): + message = f"active repeatable phase {name} has invalid partial evidence" + raise ValueError(message) + else: + message = f"repeatable phase {name} has an invalid status" raise ValueError(message) samples_by_ordinal = {sample.ordinal: sample for sample in phase.samples} for observation in phase.observations: @@ -3089,20 +3131,30 @@ def _validate_executable_report(report: RunReport) -> None: ("windows", topology.windows), ("panes", topology.panes), ): - if any( - observation.row_count != expected - for observation in phases[f"enumeration.{kind}"].observations + enumeration = phases.get(f"enumeration.{kind}") + if ( + enumeration is not None + and enumeration.status == "completed" + and any( + observation.row_count != expected + for observation in enumeration.observations + ) ): message = f"enumeration.{kind} row count differs from topology" raise ValueError(message) for strategy in ("serial", "batched"): - if any( - observation.pane_count != topology.panes - or observation.line_count is None - or observation.line_count <= 0 - or observation.byte_count is None - or observation.byte_count <= 0 - for observation in phases[f"capture.{strategy}"].observations + capture = phases.get(f"capture.{strategy}") + if ( + capture is not None + and capture.status == "completed" + and any( + observation.pane_count != topology.panes + or observation.line_count is None + or observation.line_count <= 0 + or observation.byte_count is None + or observation.byte_count <= 0 + for observation in capture.observations + ) ): message = f"capture.{strategy} count evidence is incomplete" raise ValueError(message) @@ -3119,6 +3171,23 @@ def _validate_executable_report(report: RunReport) -> None: ): message = f"{phase.name} search count evidence is incomplete" raise ValueError(message) + if report.status != "completed": + return + if control_applicable and control_phase is not None: + if control_phase.status != "completed": + message = "async control report requires control-stream evidence" + raise ValueError(message) + elif control_phase is not None and control_phase.status != "not_applicable": + message = "control-stream must be not_applicable outside async control" + raise ValueError(message) + for name in _RUNNER_REPEATABLE_PHASES: + phase = phases[name] + if phase.status != "completed": + message = f"repeatable phase {name} has incomplete raw evidence" + raise ValueError(message) + if control_applicable and phases["wait.control-stream"].status != "completed": + message = "async control report requires control-stream evidence" + raise ValueError(message) def _validate_executable_ramp(report: RunReport) -> None: @@ -3219,6 +3288,7 @@ def plan_payload( """ original = predict_resources(topology, snapshot) decision = _forced_decision(original, force_extreme) + capability_error = _pidfd_capability_error() return t.cast( dict[str, object], _json_value( @@ -3230,6 +3300,10 @@ def plan_payload( "guard_decision": decision, "original_guard_decision": original, "force_extreme": force_extreme, + "pidfd_capability": { + "available": capability_error is None, + "reason": capability_error, + }, } ), ) @@ -3282,6 +3356,8 @@ def run_plan(shape: str, output: pathlib.Path | None, force_extreme: bool) -> in table.add_row("Panes", str(topology.panes)) table.add_row("Guard decision", str(decision["kind"])) table.add_row("Allowed", str(decision["allowed"])) + capability = t.cast(dict[str, object], payload["pidfd_capability"]) + table.add_row("Stable signaling", str(capability["available"])) Console().print(table) if output is not None: write_json_atomic(output, payload) @@ -3368,17 +3444,17 @@ def process_identity_matches(identity: ProcessIdentity) -> bool: return _process_start_time(identity.pid) == identity.start_time -def _pidfd_capability_error() -> str | None: - """Return why stable process signaling is unavailable, if applicable. +def _pidfd_api_error() -> str | None: + """Return why the interpreter cannot expose Linux stable signaling. - >>> error = _pidfd_capability_error() + >>> error = _pidfd_api_error() >>> error is None or "pidfd" in error True Returns ------- str | None - ``None`` only when Linux pidfd open and signal APIs are callable. + ``None`` only when both required Linux pidfd APIs are callable. """ if platform.system() != "Linux": return "pidfd signaling requires Linux" @@ -3386,9 +3462,50 @@ def _pidfd_capability_error() -> str | None: return "os.pidfd_open is unavailable" if not callable(getattr(signal, "pidfd_send_signal", None)): return "signal.pidfd_send_signal is unavailable" + if not callable(getattr(signal, "pthread_sigmask", None)): + return "signal.pthread_sigmask is unavailable for the pidfd handoff" return None +def _pidfd_capability_error() -> str | None: + """Probe whether stable signaling actually works for the current process. + + >>> error = _pidfd_capability_error() + >>> error is None or "pidfd" in error + True + + Returns + ------- + str | None + ``None`` only after opening a self pidfd and sending signal zero. + """ + api_error = _pidfd_api_error() + if api_error is not None: + return api_error + try: + descriptor = os.pidfd_open(os.getpid(), 0) + except Exception as error: # noqa: BLE001 + return f"os.pidfd_open self-probe failed: {type(error).__name__}: {error}" + failure: str | None = None + try: + signal.pidfd_send_signal(descriptor, 0, None, 0) + except Exception as error: # noqa: BLE001 + failure = ( + "signal.pidfd_send_signal signal-zero self-probe failed: " + f"{type(error).__name__}: {error}" + ) + finally: + try: + os.close(descriptor) + except OSError as error: + if failure is None: + failure = ( + "pidfd stable-signaling probe close failed: " + f"{type(error).__name__}: {error}" + ) + return failure + + @dataclasses.dataclass(frozen=True) class _PidfdHandle: """One stable kernel handle bound to an exact recorded identity. @@ -3439,7 +3556,7 @@ def __init__( () >>> registry.close() """ - capability_error = _pidfd_capability_error() + capability_error = _pidfd_api_error() if capability_error is not None: raise RuntimeError(capability_error) self._start_time_reader = _start_time_reader or _process_start_time @@ -8803,6 +8920,41 @@ def append_progress_event(path: pathlib.Path, event: ProgressEvent) -> None: os.close(parent_descriptor) +def _initialize_progress_journal(path: pathlib.Path) -> None: + """Durably create an empty supervisor-owned progress journal. + + >>> with tempfile.TemporaryDirectory() as directory: + ... path = pathlib.Path(directory) / "progress.jsonl" + ... _initialize_progress_journal(path) + ... path.read_bytes() + b'' + + Parameters + ---------- + path : pathlib.Path + Fresh journal path created before the worker handoff begins. + + Returns + ------- + None + After the empty file and its parent directory are synchronized. + """ + path.parent.mkdir(parents=True, exist_ok=True) + flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + descriptor = os.open(path, flags, 0o600) + try: + os.fsync(descriptor) + finally: + os.close(descriptor) + parent_descriptor = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY) + try: + os.fsync(parent_descriptor) + finally: + os.close(parent_descriptor) + + def _progress_event_from_json(value: object) -> ProgressEvent: """Decode and validate one complete progress line. @@ -9916,8 +10068,8 @@ def _remove_supervised_scratch(path: pathlib.Path) -> tuple[str, ...]: def _recover_supervised_run( - worker: subprocess.Popen[bytes], - worker_identity: ProcessIdentity, + worker: subprocess.Popen[bytes] | None, + worker_identity: ProcessIdentity | None, progress: _ProgressTracker, *, scratch: pathlib.Path, @@ -9948,10 +10100,10 @@ def _recover_supervised_run( Parameters ---------- - worker : subprocess.Popen[bytes] - Direct child launched in its own session. - worker_identity : ProcessIdentity - Exact group-leader identity captured immediately after spawn. + worker : subprocess.Popen[bytes] | None + Direct child launched in its own session, or ``None`` when spawn failed. + worker_identity : ProcessIdentity | None + Exact group-leader identity captured immediately after spawn, if available. progress : _ProgressTracker Live journal state and stable handles for newly learned identities. scratch : pathlib.Path @@ -9970,7 +10122,8 @@ def _recover_supervised_run( message = "supervisor cleanup grace must be positive" raise ValueError(message) errors: list[str] = [] - progress.handles.retain(worker_identity) + if worker_identity is not None: + progress.handles.retain(worker_identity) def drain(*, terminal: bool = False) -> bool: try: @@ -9982,6 +10135,8 @@ def drain(*, terminal: bool = False) -> bool: return False def wait_worker() -> bool: + if worker is None: + return True deadline = time.monotonic() + grace_s while worker.poll() is None and time.monotonic() < deadline: drain() @@ -9990,14 +10145,24 @@ def wait_worker() -> bool: return worker.poll() is not None drain() - progress.handles.signal(worker_identity, signal.SIGTERM) - if not wait_worker(): + if worker is not None and worker.poll() is None: + if worker_identity is None: + errors.append("worker identity unavailable; exact signaling is impossible") + else: + progress.handles.signal(worker_identity, signal.SIGTERM) + if not wait_worker() and worker_identity is not None: progress.handles.signal(worker_identity, signal.SIGKILL) if not wait_worker(): errors.append("worker process did not exit") - drain(terminal=worker.poll() is not None) + worker_absent = worker is None or worker.poll() is not None + drain(terminal=worker_absent) unique: dict[tuple[int, int], ProcessIdentity] = {} - for identity in (worker_identity, *progress.identities): + initial_identities = ( + progress.identities + if worker_identity is None + else (worker_identity, *progress.identities) + ) + for identity in initial_identities: unique[(identity.pid, identity.start_time)] = identity owned = tuple(unique.values()) progress.handles.retain_many(owned) @@ -10014,7 +10179,9 @@ def wait_worker() -> bool: for identity in survivors if process_identity_matches(identity) ) - processes_absent = not any(process_identity_matches(identity) for identity in owned) + processes_absent = worker_absent and not any( + process_identity_matches(identity) for identity in owned + ) if socket_path.exists(): errors.extend(_kill_exact_tmux_socket(socket_path, timeout_s=grace_s)) if processes_absent and socket_path.exists(): @@ -10269,8 +10436,17 @@ def run() -> None: target=run, name="orchestration-finalization", ) - thread.start() interruption = initial_interrupt + started = False + while not started: + try: + thread.start() + except KeyboardInterrupt as error: # noqa: PERF203 + if interruption is None: + interruption = error + started = thread.ident is not None + else: + started = True while not completed.is_set(): try: completed.wait(timeout=0.02) @@ -10293,8 +10469,8 @@ def run() -> None: def _finalize_supervised_worker( - worker: subprocess.Popen[bytes], - worker_identity: ProcessIdentity, + worker: subprocess.Popen[bytes] | None, + worker_identity: ProcessIdentity | None, progress: _ProgressTracker, *, topology: Topology, @@ -10337,7 +10513,7 @@ def _finalize_supervised_worker( socket_path=socket_path, grace_s=cleanup_grace_s, ) - else: + elif worker is not None: worker.wait() try: progress.drain(terminal=True) @@ -10381,6 +10557,18 @@ def _finalize_supervised_worker( socket_path=socket_path, grace_s=cleanup_grace_s, ) + else: + supervisor_status = "failed" + failed_phase = failed_phase or "supervisor" + terminal_error = terminal_error or "worker was not spawned" + cleanup = _recover_supervised_run( + worker, + worker_identity, + progress, + scratch=scratch, + socket_path=socket_path, + grace_s=cleanup_grace_s, + ) try: candidate = load_run_report(checkpoint_path) @@ -10404,6 +10592,7 @@ def _finalize_supervised_worker( ), ) if supervisor_status is None: + assert worker is not None if worker.returncode == 0 and candidate.status == "completed": final_status: t.Literal["completed", "failed", "cutoff"] = "completed" elif candidate.status == "cutoff": @@ -10539,13 +10728,6 @@ def supervise_worker( progress_path = output.with_name(f"{output.stem}.{run_id}.progress.jsonl") checkpoint_path = output.with_name(f".{output.name}.{run_id}.worker.json") admission_path = output.with_name(f".{output.name}.{run_id}.admission.json") - write_json_atomic( - admission_path, - { - "guard_decision": guard_decision, - "original_guard_decision": original_guard_decision, - }, - ) command = [ sys.executable, str(pathlib.Path(__file__)), @@ -10604,30 +10786,87 @@ def supervise_worker( environment = os.environ.copy() environment.pop("TMUX", None) environment.pop("TMUX_PANE", None) - worker = subprocess.Popen( - command, - cwd=pathlib.Path(__file__).parents[1], - env=environment, - stdin=subprocess.DEVNULL, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - start_new_session=True, - ) - worker_identity = _record_process("worker", worker.pid) + capability_error = _pidfd_capability_error() + if capability_error is not None: + raise RuntimeError(capability_error) + if threading.current_thread() is not threading.main_thread(): + message = "supervisor worker handoff requires the main thread" + raise RuntimeError(message) + worker: subprocess.Popen[bytes] | None = None + worker_identity: ProcessIdentity | None = None handles = _PidfdRegistry() - handles.retain(worker_identity) progress = _ProgressTracker( progress_path, run_id, handles, - identities=(worker_identity,), ) deadline = time.monotonic() + watchdog_s supervisor_status: t.Literal["failed", "cutoff"] | None = None failed_phase: str | None = None terminal_error: str | None = None interruption: KeyboardInterrupt | None = None + previous_sigterm_handler = signal.getsignal(signal.SIGTERM) + + def request_termination( + signal_number: int, + _frame: types.FrameType | None, + ) -> t.NoReturn: + message = f"received {signal.Signals(signal_number).name}" + raise KeyboardInterrupt(message) + + signal.signal(signal.SIGTERM, request_termination) try: + _initialize_progress_journal(progress_path) + write_json_atomic( + admission_path, + { + "guard_decision": guard_decision, + "original_guard_decision": original_guard_decision, + }, + ) + blocked = frozenset({signal.SIGINT, signal.SIGTERM}) + previous_mask = signal.pthread_sigmask(signal.SIG_BLOCK, blocked) + handoff_failure: BaseException | None = None + try: + worker = subprocess.Popen( + command, + cwd=pathlib.Path(__file__).parents[1], + env=environment, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + start_new_session=True, + ) + worker_identity = _record_process("worker", worker.pid) + if not handles.retain(worker_identity): + message = "cannot retain stable worker pidfd during handoff" + raise RuntimeError(message) # noqa: TRY301 + progress.identities = (worker_identity,) + except BaseException as error: # noqa: BLE001 + handoff_failure = error + if worker is not None: + try: + if worker_identity is None: + worker_identity = _record_process("worker", worker.pid) + if handles.retain(worker_identity): + progress.identities = _merge_identities( + progress.identities, + (worker_identity,), + ) + except BaseException as recovery_error: # noqa: BLE001 + handles.errors.append( + "worker pidfd handoff recovery: " + f"{type(recovery_error).__name__}: {recovery_error}" + ) + finally: + try: + signal.pthread_sigmask(signal.SIG_SETMASK, previous_mask) + except BaseException as error: # noqa: BLE001 + if handoff_failure is None: + handoff_failure = error + if handoff_failure is not None: + raise handoff_failure # noqa: TRY301 + assert worker is not None while worker.poll() is None: advanced = progress.drain() if advanced: @@ -10648,43 +10887,55 @@ def supervise_worker( failed_phase = "supervisor" terminal_error = f"{type(error).__name__}: {error}" - final_report, _interruption = _drain_finalizer_thread( - lambda: _finalize_supervised_worker( - worker, - worker_identity, - progress, - topology=topology, - lane=lane, - mode=mode, - runs=runs, - warmup=warmup, - seed=seed, - run_id=run_id, - scratch=scratch, - socket_path=socket_path, - output=output, - markdown_output=markdown_output, - checkpoint_path=checkpoint_path, - admission_path=admission_path, - guard_decision=guard_decision, - original_guard_decision=original_guard_decision, - cleanup_grace_s=cleanup_grace_s, - supervisor_status=supervisor_status, - failed_phase=failed_phase, - terminal_error=terminal_error, - ), - initial_interrupt=interruption, - ) - if _interruption is not None and final_report.status == "completed": - final_report, _interruption = _drain_finalizer_thread( - lambda: _publish_interrupted_supervisor_report( - final_report, - output, - markdown_output, + try: + final_report, deferred_interruption = _drain_finalizer_thread( + lambda: _finalize_supervised_worker( + worker, + worker_identity, + progress, + topology=topology, + lane=lane, + mode=mode, + runs=runs, + warmup=warmup, + seed=seed, + run_id=run_id, + scratch=scratch, + socket_path=socket_path, + output=output, + markdown_output=markdown_output, + checkpoint_path=checkpoint_path, + admission_path=admission_path, + guard_decision=guard_decision, + original_guard_decision=original_guard_decision, + cleanup_grace_s=cleanup_grace_s, + supervisor_status=supervisor_status, + failed_phase=failed_phase, + terminal_error=terminal_error, ), - initial_interrupt=_interruption, + initial_interrupt=interruption, ) - return final_report + if deferred_interruption is not None and ( + final_report.cleanup.complete + and not any(phase.status == "failed" for phase in final_report.phases) + and ( + final_report.status != "cutoff" + or final_report.failed_phase != "cancellation" + ) + ): + final_report, deferred_interruption = _drain_finalizer_thread( + lambda: _publish_interrupted_supervisor_report( + final_report, + output, + markdown_output, + ), + initial_interrupt=deferred_interruption, + ) + if deferred_interruption is not None: + raise deferred_interruption + return final_report + finally: + signal.signal(signal.SIGTERM, previous_sigterm_handler) def _write_text_atomic(path: pathlib.Path, text: str) -> None: @@ -10971,7 +11222,7 @@ def run_scenario( failed_phase="preflight", error=( capability_error - if capability_error is not None and original.allowed + if capability_error is not None and decision.rule == "pidfd_capability" else f"predictive refusal: {original.rule or 'unknown'}" ), ) @@ -11153,28 +11404,33 @@ def run_ramp( last_observed: Topology | None = None terminal_status: t.Literal["refused", "failed", "cutoff"] | None = None terminal_reason: str | None = None + interruption: KeyboardInterrupt | None = None for index, shape in enumerate(declared): child_output = child_root / f"{index:02d}-{shape}.json" child_markdown = child_output.with_suffix(".md") - child = run_scenario( - shape, - lane=lane, - mode=mode, - runs=runs, - warmup=warmup, - seed=seed + index, - output=child_output, - markdown_output=child_markdown, - scratch_root=scratch_root, - force_extreme=force_extreme, - policy=policy, - host_snapshot=host_snapshot, - watchdog_s=watchdog_s, - cleanup_grace_s=cleanup_grace_s, - _test_stall_after=_test_stall_after, - _test_fail_after=_test_fail_after, - _test_extra_identity=_test_extra_identity, - ) + try: + child = run_scenario( + shape, + lane=lane, + mode=mode, + runs=runs, + warmup=warmup, + seed=seed + index, + output=child_output, + markdown_output=child_markdown, + scratch_root=scratch_root, + force_extreme=force_extreme, + policy=policy, + host_snapshot=host_snapshot, + watchdog_s=watchdog_s, + cleanup_grace_s=cleanup_grace_s, + _test_stall_after=_test_stall_after, + _test_fail_after=_test_fail_after, + _test_extra_identity=_test_extra_identity, + ) + except KeyboardInterrupt as error: + interruption = error + child = load_run_report(child_output) steps[index] = RampStep( shape, t.cast( @@ -11205,7 +11461,7 @@ def run_ramp( ramp=tuple(steps), ) write_json_atomic(output, report) - report, interruption = _drain_finalizer_thread( + report, deferred_interruption = _drain_finalizer_thread( lambda: _finalize_ramp_aggregation( report, steps, @@ -11214,17 +11470,20 @@ def run_ramp( terminal_reason=terminal_reason, output=output, markdown_output=markdown_output, - ) + ), + initial_interrupt=interruption, ) - if interruption is not None and report.status == "completed": - report, _interruption = _drain_finalizer_thread( + if deferred_interruption is not None and report.status == "completed": + report, deferred_interruption = _drain_finalizer_thread( lambda: _publish_interrupted_ramp_report( report, output, markdown_output, ), - initial_interrupt=interruption, + initial_interrupt=deferred_interruption, ) + if deferred_interruption is not None: + raise deferred_interruption return report @@ -11429,6 +11688,8 @@ async def _run_worker_with_signals(**kwargs: t.Any) -> RunReport: except (NotImplementedError, RuntimeError): continue installed.append(signal_number) + if installed and callable(getattr(signal, "pthread_sigmask", None)): + signal.pthread_sigmask(signal.SIG_UNBLOCK, installed) try: return await task finally: @@ -11628,25 +11889,28 @@ def main(argv: t.Sequence[str] | None = None) -> int: extra_identity = _parse_extra_identity(arguments.test_extra_identity) if arguments.command == "run": output = arguments.output or pathlib.Path("orchestration-report.json") - report = run_scenario( - parse_topology(arguments.shape), - lane=EngineLane(arguments.lane), - mode=ExecutionMode(arguments.mode), - runs=arguments.runs, - warmup=arguments.warmup, - seed=arguments.seed, - output=output, - markdown_output=arguments.markdown_output, - scratch_root=arguments.scratch_root, - force_extreme=arguments.force_extreme, - policy=policy, - host_snapshot=host_snapshot, - watchdog_s=arguments.watchdog_seconds, - cleanup_grace_s=arguments.cleanup_grace_seconds, - _test_stall_after=arguments.test_stall_after, - _test_fail_after=arguments.test_fail_after, - _test_extra_identity=extra_identity, - ) + try: + report = run_scenario( + parse_topology(arguments.shape), + lane=EngineLane(arguments.lane), + mode=ExecutionMode(arguments.mode), + runs=arguments.runs, + warmup=arguments.warmup, + seed=arguments.seed, + output=output, + markdown_output=arguments.markdown_output, + scratch_root=arguments.scratch_root, + force_extreme=arguments.force_extreme, + policy=policy, + host_snapshot=host_snapshot, + watchdog_s=arguments.watchdog_seconds, + cleanup_grace_s=arguments.cleanup_grace_seconds, + _test_stall_after=arguments.test_stall_after, + _test_fail_after=arguments.test_fail_after, + _test_extra_identity=extra_identity, + ) + except KeyboardInterrupt: + return 130 return 0 if report.status == "completed" else 2 if arguments.command == "ramp": shapes = ( @@ -11655,26 +11919,29 @@ def main(argv: t.Sequence[str] | None = None) -> int: else tuple(parse_topology(shape) for shape in arguments.shapes.split(",")) ) output = arguments.output or pathlib.Path("orchestration-ramp.json") - report = run_ramp( - shapes, - lane=EngineLane(arguments.lane), - mode=ExecutionMode(arguments.mode), - runs=arguments.runs, - warmup=arguments.warmup, - seed=arguments.seed, - output=output, - markdown_output=arguments.markdown_output, - scratch_root=arguments.scratch_root, - force_extreme=arguments.force_extreme, - policy=policy, - host_snapshot=host_snapshot, - watchdog_s=arguments.watchdog_seconds, - cleanup_grace_s=arguments.cleanup_grace_seconds, - canonical=arguments.shapes is None, - _test_stall_after=arguments.test_stall_after, - _test_fail_after=arguments.test_fail_after, - _test_extra_identity=extra_identity, - ) + try: + report = run_ramp( + shapes, + lane=EngineLane(arguments.lane), + mode=ExecutionMode(arguments.mode), + runs=arguments.runs, + warmup=arguments.warmup, + seed=arguments.seed, + output=output, + markdown_output=arguments.markdown_output, + scratch_root=arguments.scratch_root, + force_extreme=arguments.force_extreme, + policy=policy, + host_snapshot=host_snapshot, + watchdog_s=arguments.watchdog_seconds, + cleanup_grace_s=arguments.cleanup_grace_seconds, + canonical=arguments.shapes is None, + _test_stall_after=arguments.test_stall_after, + _test_fail_after=arguments.test_fail_after, + _test_extra_identity=extra_identity, + ) + except KeyboardInterrupt: + return 130 return 0 if report.status == "completed" else 2 message = "argparse selected an unsupported command" raise AssertionError(message) diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index 85218f01fb..cbaf7df133 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -4,6 +4,7 @@ import asyncio import dataclasses +import errno import hashlib import importlib.util import json @@ -645,6 +646,10 @@ def test_plan_writes_original_predictive_decision_without_talking_to_tmux( payload["original_guard_decision"] == payload["guard_decision"] or payload["original_guard_decision"]["kind"] == "predictive_refusal" ) + assert payload["pidfd_capability"] == { + "available": True, + "reason": None, + } def test_validate_report_rejects_contradictory_ramp_terminal_sequence( @@ -1020,7 +1025,9 @@ def test_pidfd_registry_signals_only_the_retained_real_child( benchmark_module: types.ModuleType, ) -> None: """A retained pidfd must terminate its exact child without a PID lookup.""" - assert benchmark_module._pidfd_capability_error() is None + capability_error = benchmark_module._pidfd_capability_error() + if capability_error is not None: + pytest.skip(capability_error) target = subprocess.Popen( (sys.executable, "-c", "import time; time.sleep(60)"), stdin=subprocess.DEVNULL, @@ -1050,6 +1057,147 @@ def test_pidfd_registry_signals_only_the_retained_real_child( unrelated.wait(timeout=2.0) +def test_pidfd_capability_probe_opens_signals_and_closes_self( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Capability success requires a real signal-zero operation on a self pidfd.""" + calls: list[tuple[t.Any, ...]] = [] + descriptor = os.open("/proc/self/stat", os.O_RDONLY) + + def open_pidfd(pid: int, flags: int) -> int: + calls.append(("open", pid, flags)) + return descriptor + + def send_signal( + received: int, + number: int, + siginfo: object, + flags: int, + ) -> None: + calls.append(("signal", received, number, siginfo, flags)) + + monkeypatch.setattr(benchmark_module.os, "pidfd_open", open_pidfd, raising=False) + monkeypatch.setattr( + benchmark_module.signal, + "pidfd_send_signal", + send_signal, + raising=False, + ) + + assert benchmark_module._pidfd_capability_error() is None + assert calls == [ + ("open", os.getpid(), 0), + ("signal", descriptor, 0, None, 0), + ] + with pytest.raises(OSError, match="Bad file descriptor"): + os.fstat(descriptor) + + +@pytest.mark.parametrize( + ("operation", "error_number"), + ( + ("open", errno.ENOSYS), + ("open", errno.EACCES), + ("signal", errno.EPERM), + ("signal", errno.EINVAL), + ), +) +def test_run_scenario_refuses_failed_pidfd_self_probe_before_side_effects( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, + operation: str, + error_number: int, +) -> None: + """An unusable stable-signal syscall must refuse before any run resource.""" + descriptor: int | None = None + + def fail_open(pid: int, flags: int) -> int: + nonlocal descriptor + if operation == "open": + raise OSError(error_number, os.strerror(error_number)) + descriptor = os.open("/proc/self/stat", os.O_RDONLY) + return descriptor + + def fail_signal( + descriptor: int, + number: int, + siginfo: object, + flags: int, + ) -> None: + if operation == "signal": + raise OSError(error_number, os.strerror(error_number)) + pytest.fail("open failure reached pidfd_send_signal") + + monkeypatch.setattr(benchmark_module.os, "pidfd_open", fail_open, raising=False) + monkeypatch.setattr( + benchmark_module.signal, + "pidfd_send_signal", + fail_signal, + raising=False, + ) + marker = tmp_path / "tmux-executed" + fake_bin = tmp_path / "bin" + fake_bin.mkdir() + fake_tmux = fake_bin / "tmux" + fake_tmux.write_text( + "#!/usr/bin/env python3\n" + "import pathlib\n" + f"pathlib.Path({str(marker)!r}).touch()\n", + encoding="utf-8", + ) + fake_tmux.chmod(fake_tmux.stat().st_mode | stat.S_IXUSR) + monkeypatch.setenv("PATH", f"{fake_bin}{os.pathsep}{os.environ['PATH']}") + + def reject_spawn(*_args: t.Any, **_kwargs: t.Any) -> t.NoReturn: + pytest.fail("failed pidfd preflight spawned a worker") + + monkeypatch.setattr(benchmark_module.subprocess, "Popen", reject_spawn) + output = tmp_path / f"pidfd-{operation}.json" + scratch_root = tmp_path / "scratch" + + report = benchmark_module.run_scenario( + benchmark_module.Topology(1, 1, 1), + runs=1, + warmup=0, + output=output, + scratch_root=scratch_root, + host_snapshot=benchmark_module.HostSnapshot( + available_memory_bytes=16 * 1024**3, + pids_current=1, + pids_max=100_000, + nofile_soft_limit=65_536, + ), + policy=benchmark_module.ResourcePolicy( + pid_reserve=1, + memory_floor_bytes=1, + ), + ) + + assert report.status == "refused" + assert report.failed_phase == "preflight" + assert "pidfd" in t.cast(str, report.error) + assert os.strerror(error_number) in t.cast(str, report.error) + assert not scratch_root.exists() + assert report.scratch_path is None + assert report.socket_path is None + assert report.processes == () + assert not marker.exists() + if descriptor is not None: + with pytest.raises(OSError, match="Bad file descriptor"): + os.fstat(descriptor) + + +def test_pidfd_capability_real_self_probe( + benchmark_module: types.ModuleType, +) -> None: + """The supported test host must prove a stable signal-zero pidfd operation.""" + if not hasattr(os, "pidfd_open") or not hasattr(signal, "pidfd_send_signal"): + pytest.skip("interpreter does not expose Linux pidfd APIs") + assert benchmark_module._pidfd_capability_error() is None + + def test_pidfd_registry_rejects_reuse_between_open_checks( benchmark_module: types.ModuleType, ) -> None: @@ -4551,6 +4699,167 @@ def test_validator_requires_exact_warmup_observation_ordinals( benchmark_module.validate_report(report) +def _failed_report_with_completed_mutation_prefix( + benchmark_module: types.ModuleType, + *, + phase_warmup: int = 2, + phase_runs: int = 2, + warmup_ordinals: tuple[int, ...] = (0, 1), + timed_ordinals: tuple[int, ...] = (0, 1), +) -> t.Any: + """Build a valid active failure after one configurable completed cell.""" + topology = benchmark_module.Topology(1, 1, 1) + setup = benchmark_module.PhaseReport( + "setup", + topology, + topology, + samples=( + benchmark_module.RawSample( + 1, + True, + verified=True, + strategy="setup", + ordinal=0, + ), + ), + status="completed", + runs=1, + observations=(benchmark_module.PhaseObservation(0, "setup", 1),), + ) + stabilization = benchmark_module.PhaseReport( + "stabilization", + topology, + topology, + status="completed", + observations=( + benchmark_module.PhaseObservation( + 0, + "stabilization", + 1, + pane_count=topology.panes, + ), + ), + ) + samples = tuple( + benchmark_module.RawSample( + 10 + ordinal, + True, + verified=True, + strategy="mutation.bulk", + ordinal=ordinal, + ) + for ordinal in timed_ordinals + ) + durations = tuple(t.cast(int, sample.duration_ns) for sample in samples) + mutation = benchmark_module.PhaseReport( + "mutation.bulk", + topology, + topology, + samples=samples, + summary=(benchmark_module.summarize_ns(durations) if durations else None), + status="completed", + warmup=phase_warmup, + runs=phase_runs, + warmup_observations=tuple( + benchmark_module.PhaseObservation( + ordinal, + "mutation.bulk", + 2 + ordinal, + ) + for ordinal in warmup_ordinals + ), + observations=tuple( + benchmark_module.PhaseObservation( + t.cast(int, sample.ordinal), + "mutation.bulk", + t.cast(int, sample.duration_ns), + ) + for sample in samples + ), + ) + active_failure = benchmark_module.PhaseReport( + "wait.capture-poll", + topology, + topology, + status="failed", + warmup=2, + runs=2, + warmup_observations=( + benchmark_module.PhaseObservation(0, "wait.capture-poll", 3), + ), + ) + return benchmark_module.RunReport( + topology, + observed_topology=topology, + status="failed", + phases=(setup, stabilization, mutation, active_failure), + cleanup=benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + run_id="run-7", + lane="subprocess", + mode="sync", + warmup=2, + runs=2, + failed_phase="wait.capture-poll", + error="injected active wait failure", + scratch_path="scratch", + socket_path="scratch/socket", + progress_path="progress.jsonl", + environment=benchmark_module.EnvironmentReport( + "3.10", None, 1, 11, ("run",), None + ), + ) + + +@pytest.mark.parametrize( + "report_changes", + ( + {"phase_warmup": 1, "warmup_ordinals": (0,)}, + {"phase_runs": 1, "timed_ordinals": (0,)}, + {"phase_warmup": 2, "warmup_ordinals": (0, 0)}, + {"phase_runs": 2, "timed_ordinals": (0, 0)}, + ), +) +def test_validator_rejects_incomplete_completed_cell_before_active_failure( + benchmark_module: types.ModuleType, + report_changes: dict[str, t.Any], +) -> None: + """A later valid failure cannot excuse malformed completed-prefix evidence.""" + report = _failed_report_with_completed_mutation_prefix( + benchmark_module, + **report_changes, + ) + + with pytest.raises(ValueError, match=r"mutation\.bulk"): + benchmark_module.validate_report(report) + + +def test_validator_accepts_partial_active_failure_after_complete_prefix( + benchmark_module: types.ModuleType, +) -> None: + """The one active failed cell may retain a valid incomplete warmup prefix.""" + benchmark_module.validate_report( + _failed_report_with_completed_mutation_prefix(benchmark_module) + ) + + +def test_validator_rejects_failed_phase_that_disagrees_with_active_row( + benchmark_module: types.ModuleType, +) -> None: + """Terminal metadata must name the exact active failed phase.""" + report = dataclasses.replace( + _failed_report_with_completed_mutation_prefix(benchmark_module), + failed_phase="capture.serial", + ) + + with pytest.raises(ValueError, match="failed_phase"): + benchmark_module.validate_report(report) + + def test_cli_phase_failure_uses_supervisor_cleanup_contract( tmp_path: pathlib.Path, ) -> None: @@ -4864,7 +5173,7 @@ def test_cli_cancellation_uses_supervisor_cleanup_contract( process.kill() stdout, stderr = process.communicate(timeout=5) - assert process.returncode != 0, (stdout, stderr) + assert process.returncode == 130, (stdout, stderr) payload = json.loads(report_path.read_text(encoding="utf-8")) assert payload["status"] == "cutoff" assert payload["failed_phase"] == "cancellation" @@ -4918,35 +5227,328 @@ def interrupt_final_write() -> None: benchmark_module.HostSnapshot(), ) try: - report = benchmark_module.supervise_worker( + with pytest.raises(KeyboardInterrupt) as cancellation: + benchmark_module.supervise_worker( + benchmark_module.Topology(1, 1, 1), + lane=benchmark_module.EngineLane.SUBPROCESS, + mode=benchmark_module.ExecutionMode.SYNC, + runs=1, + warmup=0, + seed=11, + run_id="cancel-final-write", + scratch=tmp_path / "scratch", + socket_path=tmp_path / "scratch" / "tmux.sock", + output=report_path, + markdown_output=markdown_path, + guard_decision=decision, + original_guard_decision=decision, + policy=benchmark_module.ResourcePolicy(), + watchdog_s=30.0, + cleanup_grace_s=0.3, + ) + finally: + release_write.set() + interrupter.join(timeout=5.0) + + assert not interrupter.is_alive() + durable = benchmark_module.load_run_report(report_path) + assert isinstance(cancellation.value, KeyboardInterrupt) + assert durable.status == "cutoff" + assert durable.failed_phase == "cancellation" + assert durable.cleanup.complete + assert all( + not benchmark_module.process_identity_matches(row) for row in durable.processes + ) + assert "cutoff" in markdown_path.read_text(encoding="utf-8") + + +def test_supervisor_interrupt_before_popen_return_is_durable_and_reraised( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """A Popen interruption with no child must still publish then re-raise.""" + interruption = KeyboardInterrupt("popen interruption") + mask_calls: list[tuple[int, frozenset[signal.Signals]]] = [] + prior_mask = frozenset({signal.SIGUSR1}) + real_popen = benchmark_module.subprocess.Popen + interrupted = False + + def record_mask( + how: int, + signals: t.Iterable[signal.Signals], + ) -> frozenset[signal.Signals]: + mask_calls.append((how, frozenset(signals))) + return prior_mask + + def interrupt_popen(*args: t.Any, **kwargs: t.Any) -> t.Any: + nonlocal interrupted + if not interrupted: + interrupted = True + raise interruption + return real_popen(*args, **kwargs) + + monkeypatch.setattr(benchmark_module.signal, "pthread_sigmask", record_mask) + monkeypatch.setattr(benchmark_module.subprocess, "Popen", interrupt_popen) + output = tmp_path / "popen-interrupt.json" + markdown = tmp_path / "popen-interrupt.md" + decision = benchmark_module.GuardDecision( + True, + "ok", + None, + None, + None, + False, + benchmark_module.HostSnapshot(), + ) + + with pytest.raises(KeyboardInterrupt) as raised: + benchmark_module.supervise_worker( benchmark_module.Topology(1, 1, 1), lane=benchmark_module.EngineLane.SUBPROCESS, mode=benchmark_module.ExecutionMode.SYNC, runs=1, warmup=0, seed=11, - run_id="cancel-final-write", + run_id="popen-interrupt", scratch=tmp_path / "scratch", socket_path=tmp_path / "scratch" / "tmux.sock", - output=report_path, - markdown_output=markdown_path, + output=output, + markdown_output=markdown, guard_decision=decision, original_guard_decision=decision, policy=benchmark_module.ResourcePolicy(), watchdog_s=30.0, cleanup_grace_s=0.3, ) + + assert raised.value is interruption + assert mask_calls == [ + (signal.SIG_BLOCK, frozenset({signal.SIGINT, signal.SIGTERM})), + (signal.SIG_SETMASK, prior_mask), + ] + durable = benchmark_module.load_run_report(output) + assert durable.status == "cutoff" + assert durable.failed_phase == "cancellation" + assert durable.cleanup.complete + assert durable.processes == () + assert not (tmp_path / "scratch").exists() + + +def test_supervisor_interrupt_during_pidfd_handoff_recovers_exact_worker( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """Post-spawn interruption must retain a pidfd, recover, and restore the mask.""" + interruption = KeyboardInterrupt("handoff interruption") + real_popen = benchmark_module.subprocess.Popen + real_record = benchmark_module._record_process + spawned: list[subprocess.Popen[bytes]] = [] + record_calls = 0 + mask_calls: list[tuple[int, frozenset[signal.Signals]]] = [] + prior_mask = frozenset({signal.SIGUSR2}) + + def record_popen(*args: t.Any, **kwargs: t.Any) -> subprocess.Popen[bytes]: + process = t.cast("subprocess.Popen[bytes]", real_popen(*args, **kwargs)) + spawned.append(process) + return process + + def interrupt_record(role: str, pid: int) -> t.Any: + nonlocal record_calls + identity = real_record(role, pid) + record_calls += 1 + if role == "worker" and record_calls == 1: + raise interruption + return identity + + def record_mask( + how: int, + signals: t.Iterable[signal.Signals], + ) -> frozenset[signal.Signals]: + mask_calls.append((how, frozenset(signals))) + return prior_mask + + monkeypatch.setattr(benchmark_module.subprocess, "Popen", record_popen) + monkeypatch.setattr(benchmark_module, "_record_process", interrupt_record) + monkeypatch.setattr(benchmark_module.signal, "pthread_sigmask", record_mask) + output = tmp_path / "handoff-interrupt.json" + markdown = tmp_path / "handoff-interrupt.md" + decision = benchmark_module.GuardDecision( + True, + "ok", + None, + None, + None, + False, + benchmark_module.HostSnapshot(), + ) + worker_absent_before_test_cleanup = False + durable: t.Any = None + try: + with pytest.raises(KeyboardInterrupt) as raised: + benchmark_module.supervise_worker( + benchmark_module.Topology(1, 1, 1), + lane=benchmark_module.EngineLane.SUBPROCESS, + mode=benchmark_module.ExecutionMode.SYNC, + runs=1, + warmup=0, + seed=11, + run_id="handoff-interrupt", + scratch=tmp_path / "scratch", + socket_path=tmp_path / "scratch" / "tmux.sock", + output=output, + markdown_output=markdown, + guard_decision=decision, + original_guard_decision=decision, + policy=benchmark_module.ResourcePolicy(), + watchdog_s=30.0, + cleanup_grace_s=0.3, + ) + worker_absent_before_test_cleanup = all( + process.poll() is not None for process in spawned + ) + durable = benchmark_module.load_run_report(output) + finally: + for process in spawned: + if process.poll() is None: + process.kill() + process.wait(timeout=5.0) + + assert raised.value is interruption + assert worker_absent_before_test_cleanup + assert record_calls >= 2 + assert mask_calls == [ + (signal.SIG_BLOCK, frozenset({signal.SIGINT, signal.SIGTERM})), + (signal.SIG_SETMASK, prior_mask), + ] + assert durable is not None + assert durable.status == "cutoff" + assert durable.failed_phase == "cancellation" + assert durable.cleanup.complete + assert all( + not benchmark_module.process_identity_matches(row) for row in durable.processes + ) + assert not (tmp_path / "scratch").exists() + + +def test_supervisor_restores_exact_prior_signal_mask_after_real_handoff( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """A successful real worker handoff restores every preexisting mask bit.""" + capability_error = benchmark_module._pidfd_capability_error() + if capability_error is not None: + pytest.skip(capability_error) + original_mask = signal.pthread_sigmask(signal.SIG_BLOCK, {signal.SIGUSR1}) + expected_mask = frozenset((*original_mask, signal.SIGUSR1)) + output = tmp_path / "mask-restored.json" + try: + report = benchmark_module.run_scenario( + benchmark_module.Topology(1, 1, 1), + lane=benchmark_module.EngineLane.SUBPROCESS, + mode=benchmark_module.ExecutionMode.SYNC, + runs=1, + warmup=0, + output=output, + scratch_root=tmp_path / "scratch", + watchdog_s=30.0, + cleanup_grace_s=0.3, + ) + observed_mask = signal.pthread_sigmask(signal.SIG_BLOCK, ()) finally: - release_write.set() + signal.pthread_sigmask(signal.SIG_SETMASK, original_mask) + + assert report.status == "completed" + assert frozenset(observed_mask) == expected_mask + assert report.cleanup.complete + assert report.scratch_path is not None + assert not pathlib.Path(report.scratch_path).exists() + + +def test_supervisor_reraises_original_monitor_interrupt_after_recovery_interrupt( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """Repeated interruption during recovery cannot replace the monitor object.""" + interruption = KeyboardInterrupt("monitor interruption") + real_drain = benchmark_module._ProgressTracker.drain + real_recover = benchmark_module._recover_supervised_run + raised_monitor = False + recovery_started = threading.Event() + release_recovery = threading.Event() + + def interrupt_monitor(tracker: t.Any, *, terminal: bool = False) -> bool: + nonlocal raised_monitor + advanced = real_drain(tracker, terminal=terminal) + if not terminal and advanced and not raised_monitor: + raised_monitor = True + raise interruption + return t.cast(bool, advanced) + + def delayed_recovery(*args: t.Any, **kwargs: t.Any) -> t.Any: + recovery_started.set() + assert release_recovery.wait(timeout=20.0) + return real_recover(*args, **kwargs) + + monkeypatch.setattr(benchmark_module._ProgressTracker, "drain", interrupt_monitor) + monkeypatch.setattr(benchmark_module, "_recover_supervised_run", delayed_recovery) + + def interrupt_recovery() -> None: + assert recovery_started.wait(timeout=20.0) + os.kill(os.getpid(), signal.SIGINT) + time.sleep(0.05) + release_recovery.set() + + interrupter = threading.Thread(target=interrupt_recovery) + interrupter.start() + output = tmp_path / "monitor-interrupt.json" + markdown = tmp_path / "monitor-interrupt.md" + decision = benchmark_module.GuardDecision( + True, + "ok", + None, + None, + None, + False, + benchmark_module.HostSnapshot(), + ) + try: + with pytest.raises(KeyboardInterrupt) as raised: + benchmark_module.supervise_worker( + benchmark_module.Topology(1, 1, 1), + lane=benchmark_module.EngineLane.SUBPROCESS, + mode=benchmark_module.ExecutionMode.SYNC, + runs=1, + warmup=0, + seed=11, + run_id="monitor-interrupt", + scratch=tmp_path / "scratch", + socket_path=tmp_path / "scratch" / "tmux.sock", + output=output, + markdown_output=markdown, + guard_decision=decision, + original_guard_decision=decision, + policy=benchmark_module.ResourcePolicy(), + watchdog_s=30.0, + cleanup_grace_s=0.3, + _test_stall_after="worker.started", + ) + finally: + release_recovery.set() interrupter.join(timeout=5.0) assert not interrupter.is_alive() - assert report.status == "cutoff" - assert report.failed_phase == "cancellation" - assert report.cleanup.complete - durable = benchmark_module.load_run_report(report_path) - assert durable == report - assert "cutoff" in markdown_path.read_text(encoding="utf-8") + assert raised.value is interruption + durable = benchmark_module.load_run_report(output) + assert durable.status == "cutoff" + assert durable.failed_phase == "cancellation" + assert durable.cleanup.complete + assert all( + not benchmark_module.process_identity_matches(row) for row in durable.processes + ) + assert not (tmp_path / "scratch").exists() def test_cli_process_identity_mismatch_never_signals_unrelated_pid( @@ -5136,9 +5738,74 @@ def interrupt_aggregation() -> None: interrupter = threading.Thread(target=interrupt_aggregation) interrupter.start() - report = None try: - report = benchmark_module.run_ramp( + with pytest.raises(KeyboardInterrupt) as cancellation: + benchmark_module.run_ramp( + ( + benchmark_module.Topology(1, 1, 1), + benchmark_module.Topology(2, 1, 1), + ), + runs=1, + warmup=0, + output=output, + markdown_output=markdown, + ) + finally: + release_write.set() + interrupter.join(timeout=5.0) + + assert isinstance(cancellation.value, KeyboardInterrupt) + report = benchmark_module.load_run_report(output) + assert report.status == "cutoff" + assert report.ramp[-1].status == "cutoff" + assert report.cleanup.complete + assert "cutoff" in markdown.read_text(encoding="utf-8") + + +def test_ramp_reraises_same_child_cancellation_after_durable_aggregation( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """A child cancellation object survives ramp aggregation unchanged.""" + interruption = KeyboardInterrupt("child cancellation") + output = tmp_path / "ramp-child-cancel.json" + markdown = tmp_path / "ramp-child-cancel.md" + child_output: pathlib.Path | None = None + + def cancel_child(shape: t.Any, **kwargs: t.Any) -> t.NoReturn: + nonlocal child_output + child_output = kwargs["output"] + child = benchmark_module.RunReport( + shape, + status="cutoff", + cleanup=benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + run_id="child-cancelled", + lane="control", + mode="async", + warmup=0, + runs=1, + failed_phase="cancellation", + error="KeyboardInterrupt: child cancellation", + scratch_path="child-scratch", + socket_path="child-scratch/tmux.sock", + progress_path="child-progress.jsonl", + environment=benchmark_module.EnvironmentReport( + "3.10", None, 1, 11, ("run",), None + ), + ) + benchmark_module.write_json_atomic(child_output, child) + raise interruption + + monkeypatch.setattr(benchmark_module, "run_scenario", cancel_child) + + with pytest.raises(KeyboardInterrupt) as raised: + benchmark_module.run_ramp( ( benchmark_module.Topology(1, 1, 1), benchmark_module.Topology(2, 1, 1), @@ -5148,16 +5815,17 @@ def interrupt_aggregation() -> None: output=output, markdown_output=markdown, ) - finally: - release_write.set() - interrupter.join(timeout=5.0) - assert report is not None + assert raised.value is interruption + assert child_output is not None + report = benchmark_module.load_run_report(output) assert report.status == "cutoff" - assert report.ramp[-1].status == "cutoff" + assert tuple(step.status for step in report.ramp) == ( + "cutoff", + "not_attempted", + ) + assert report.ramp[0].reason == report.ramp[1].reason assert report.cleanup.complete - assert benchmark_module.load_run_report(output) == report - assert "cutoff" in markdown.read_text(encoding="utf-8") def test_cli_ramp_predictive_refusal_never_executes_tmux_binary( @@ -5276,15 +5944,15 @@ def test_cli_run_supports_all_four_engine_mode_lanes( completed = _run_cli( "run", "--shape", - "1x1x1", + "2x2x2", "--lane", lane, "--mode", mode, "--runs", - "1", + "3", "--warmup", - "0", + "1", "--output", str(report_path), "--scratch-root", @@ -5304,6 +5972,12 @@ def test_cli_run_supports_all_four_engine_mode_lanes( ) assert phases["wait.control-stream"]["status"] == expected_control assert phases["wait.capture-poll"]["status"] == "completed" + for phase_name in _RUNNER_REPEATABLE_PHASES: + assert len(phases[phase_name]["warmup_observations"]) == 1 + assert len(phases[phase_name]["samples"]) == 3 + if expected_control == "completed": + assert len(phases["wait.control-stream"]["warmup_observations"]) == 1 + assert len(phases["wait.control-stream"]["samples"]) == 3 _assert_terminal_cleanup(payload) From 019d45044efebacab11b58003ccad11f74c8dcf2 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 14:38:22 -0500 Subject: [PATCH 22/67] Bench(fix[runner]): Finalize exactly once why: Interrupts could retry a native thread launch, strand ramp bookkeeping, or publish phase rows for work that never started. what: - Block termination signals across the finalizer thread handoff - Shield every ramp aggregate write and preserve completed attempts - Create strategy rows lazily and validate aggregate cutoffs - Cover repeated interrupts and mid-strategy failures --- scripts/bench_orchestration.py | 388 +++++++++++------- tests/test_bench_orchestration_script.py | 492 ++++++++++++++++++++++- 2 files changed, 725 insertions(+), 155 deletions(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index db6eb37883..933980fbf4 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -2868,23 +2868,41 @@ def validate_report(report: RunReport) -> None: raise ValueError(message) if report.ramp_kind != "none" and report.status in terminal_statuses: terminals = [step for step in report.ramp if step.status in terminal_statuses] - if len(terminals) != 1 or terminals[0].status != report.status: + aggregate_cutoff = report.status == "cutoff" and not terminals + if aggregate_cutoff: + pending = False + if not report.error: + message = "aggregate cutoff requires an interruption reason" + raise ValueError(message) + for step in report.ramp: + if step.status == "completed" and not pending: + continue + if step.status == "not_attempted" and step.reason == report.error: + pending = True + continue + message = "aggregate cutoff requires a completed prefix" + raise ValueError(message) + elif len(terminals) != 1 or terminals[0].status != report.status: message = "terminal report requires exactly one matching terminal attempt" raise ValueError(message) - terminal_index = report.ramp.index(terminals[0]) - reason = terminals[0].reason - if ( - reason is None - or any(step.status != "completed" for step in report.ramp[:terminal_index]) - or any( - step.status != "not_attempted" or step.reason != reason - for step in report.ramp[terminal_index + 1 :] - ) - ): - message = ( - "invalid terminal ramp sequence: later attempts must be not_attempted" - ) - raise ValueError(message) + else: + terminal_index = report.ramp.index(terminals[0]) + reason = terminals[0].reason + if ( + reason is None + or any( + step.status != "completed" for step in report.ramp[:terminal_index] + ) + or any( + step.status != "not_attempted" or step.reason != reason + for step in report.ramp[terminal_index + 1 :] + ) + ): + message = ( + "invalid terminal ramp sequence: " + "later attempts must be not_attempted" + ) + raise ValueError(message) maximum = Topology(100, 100, 4) if report.maximum_completed and ( report.status != "completed" @@ -9208,7 +9226,7 @@ async def _run_worker_group( fail_after: str | None, active_phase: list[str], ) -> None: - """Run one deterministically interleaved family and checkpoint each call. + """Run one repeatable family lazily and checkpoint each accepted call. >>> async def empty_group(): ... try: @@ -9235,7 +9253,7 @@ async def _run_worker_group( runs : int Timed accepted calls per cell. seed : int - Deterministic family-order seed. + Deterministic per-cell measurement seed. policy : ResourcePolicy Runtime guard thresholds. latest : dict[str, PhaseMeasurement] @@ -9253,21 +9271,7 @@ async def _run_worker_group( if not strategies: message = "worker phase group requires strategies" raise ValueError(message) - active_phase[0] = next(iter(strategies)) topology = context.topology - for name in strategies: - recorder.report = _replace_phase( - recorder.report, - PhaseReport( - name=name, - requested_topology=topology, - observed_topology=topology, - status="in_progress", - warmup=warmup, - runs=runs, - ), - ) - recorder.checkpoint(f"{next(iter(strategies))}.started") async def on_progress( stage: str, @@ -9304,26 +9308,42 @@ async def on_progress( message = f"injected phase failure after {strategy}" raise RuntimeError(message) - result = await run_repeatable_phase( - strategies, - warmup=warmup, - runs=runs, - seed=seed, - snapshot_resources=lambda: probe_host(ProcessReader()), - live_postcondition=lambda measurement: _worker_live_postcondition( - context, measurement, policy - ), - progress_callback=on_progress, - boundary_callback=lambda strategy: active_phase.__setitem__(0, strategy), - ) - if result.failure is not None: - phases = {phase.name: phase for phase in recorder.report.phases} - failed = dataclasses.replace( - phases[result.failure.strategy], status="failed", summary=None + for strategy_index, (name, strategy) in enumerate(strategies.items()): + active_phase[0] = name + recorder.report = _replace_phase( + recorder.report, + PhaseReport( + name=name, + requested_topology=topology, + observed_topology=topology, + status="in_progress", + warmup=warmup, + runs=runs, + ), + ) + recorder.checkpoint(f"{name}.started") + result = await run_repeatable_phase( + {name: strategy}, + warmup=warmup, + runs=runs, + seed=seed + strategy_index, + snapshot_resources=lambda: probe_host(ProcessReader()), + live_postcondition=lambda measurement: _worker_live_postcondition( + context, measurement, policy + ), + progress_callback=on_progress, + boundary_callback=lambda strategy_name: active_phase.__setitem__( + 0, strategy_name + ), ) - recorder.report = _replace_phase(recorder.report, failed) - recorder.checkpoint(f"{result.failure.strategy}.failed") - raise PhaseExecutionError(result.failure.strategy, result.failure) + if result.failure is not None: + phases = {phase.name: phase for phase in recorder.report.phases} + failed = dataclasses.replace( + phases[result.failure.strategy], status="failed", summary=None + ) + recorder.report = _replace_phase(recorder.report, failed) + recorder.checkpoint(f"{result.failure.strategy}.failed") + raise PhaseExecutionError(result.failure.strategy, result.failure) def _position_target(ids: tuple[str, ...], position: str) -> str: @@ -10437,16 +10457,37 @@ def run() -> None: name="orchestration-finalization", ) interruption = initial_interrupt - started = False - while not started: + launch_failure: BaseException | None = None + blocked = frozenset({signal.SIGINT, signal.SIGTERM}) + previous_mask = signal.pthread_sigmask(signal.SIG_BLOCK, blocked) + try: try: thread.start() - except KeyboardInterrupt as error: # noqa: PERF203 + except KeyboardInterrupt as error: if interruption is None: interruption = error - started = thread.ident is not None - else: - started = True + except BaseException as error: # noqa: BLE001 + launch_failure = error + finally: + try: + signal.pthread_sigmask(signal.SIG_SETMASK, previous_mask) + except KeyboardInterrupt as error: + if interruption is None: + interruption = error + except BaseException as error: # noqa: BLE001 + if launch_failure is None: + launch_failure = error + try: + launched = thread.ident is not None or thread.is_alive() + except RuntimeError: + launched = thread.ident is not None + if not launched: + if interruption is not None: + raise interruption + if launch_failure is not None: + raise launch_failure + message = "finalization thread did not launch" + raise RuntimeError(message) while not completed.is_set(): try: completed.wait(timeout=0.02) @@ -10461,6 +10502,12 @@ def run() -> None: interruption = error except RuntimeError: pass + if launch_failure is not None: + if interruption is not None: + raise interruption from launch_failure + if failures: + raise launch_failure from failures[0] + raise launch_failure if failures: if interruption is not None: raise interruption from failures[0] @@ -11304,44 +11351,6 @@ def _finalize_ramp_aggregation( return terminal -def _publish_interrupted_ramp_report( - report: RunReport, - output: pathlib.Path, - markdown_output: pathlib.Path, -) -> RunReport: - """Publish cancellation that arrived while a completed ramp aggregated. - - >>> _publish_interrupted_ramp_report.__name__ - '_publish_interrupted_ramp_report' - - Returns - ------- - RunReport - Validated cutoff aggregate with the final attempt marked cutoff. - """ - reason = "KeyboardInterrupt: ramp interrupted during aggregation" - steps = list(report.ramp) - completed_indices = tuple( - index for index, step in enumerate(steps) if step.status == "completed" - ) - if not completed_indices: - return report - final_index = completed_indices[-1] - final = steps[final_index] - steps[final_index] = dataclasses.replace(final, status="cutoff", reason=reason) - interrupted = dataclasses.replace( - report, - status="cutoff", - maximum_completed=False, - ramp=tuple(steps), - error=reason, - ) - write_json_atomic(output, interrupted) - validate_report(interrupted) - render_markdown_summary(output, markdown_output) - return interrupted - - def run_ramp( shapes: t.Sequence[Topology], *, @@ -11398,17 +11407,58 @@ def run_ramp( include_tmux=False, ), ) - write_json_atomic(output, report) child_root = output.with_name(f"{output.stem}.runs") steps = list(attempts) last_observed: Topology | None = None terminal_status: t.Literal["refused", "failed", "cutoff"] | None = None terminal_reason: str | None = None interruption: KeyboardInterrupt | None = None - for index, shape in enumerate(declared): - child_output = child_root / f"{index:02d}-{shape}.json" - child_markdown = child_output.with_suffix(".md") - try: + active_index: int | None = None + active_child_output: pathlib.Path | None = None + final_report: RunReport | None = None + + def step_from_child( + index: int, + child: RunReport, + child_output: pathlib.Path, + ) -> RampStep: + return RampStep( + declared[index], + t.cast( + t.Literal["completed", "refused", "failed", "cutoff"], + child.status, + ), + child.error, + run_id=child.run_id, + report_path=str(child_output), + scratch_path=child.scratch_path, + socket_path=child.socket_path, + ) + + def write_checkpoint(candidate: RunReport) -> None: + _result, deferred = _drain_finalizer_thread( + lambda: write_json_atomic(output, candidate) + ) + if deferred is not None: + raise deferred + + def cancellation_reason(error: KeyboardInterrupt) -> str: + detail = str(error) + suffix = f": {detail}" if detail else "" + return f"KeyboardInterrupt: ramp bookkeeping interrupted{suffix}" + + def mark_pending(reason: str) -> None: + for pending_index, step in enumerate(steps): + if step.status == "not_attempted": + steps[pending_index] = dataclasses.replace(step, reason=reason) + + try: + write_checkpoint(report) + for index, shape in enumerate(declared): + active_index = index + child_output = child_root / f"{index:02d}-{shape}.json" + active_child_output = child_output + child_markdown = child_output.with_suffix(".md") child = run_scenario( shape, lane=lane, @@ -11428,63 +11478,93 @@ def run_ramp( _test_fail_after=_test_fail_after, _test_extra_identity=_test_extra_identity, ) - except KeyboardInterrupt as error: - interruption = error - child = load_run_report(child_output) - steps[index] = RampStep( - shape, - t.cast( - t.Literal["completed", "refused", "failed", "cutoff"], - child.status, - ), - child.error, - run_id=child.run_id, - report_path=str(child_output), - scratch_path=child.scratch_path, - socket_path=child.socket_path, - ) - if child.status == "completed": - last_observed = child.observed_topology - else: - terminal_status = t.cast( - t.Literal["refused", "failed", "cutoff"], child.status - ) - terminal_reason = child.error or child.status - for later in range(index + 1, len(declared)): - steps[later] = RampStep( - declared[later], "not_attempted", terminal_reason + steps[index] = step_from_child(index, child, child_output) + if child.status == "completed": + last_observed = child.observed_topology + else: + terminal_status = t.cast( + t.Literal["refused", "failed", "cutoff"], child.status ) - break - report = dataclasses.replace( - report, - observed_topology=last_observed, - ramp=tuple(steps), - ) - write_json_atomic(output, report) - report, deferred_interruption = _drain_finalizer_thread( - lambda: _finalize_ramp_aggregation( - report, - steps, - last_observed=last_observed, - terminal_status=terminal_status, - terminal_reason=terminal_reason, - output=output, - markdown_output=markdown_output, - ), - initial_interrupt=interruption, - ) - if deferred_interruption is not None and report.status == "completed": - report, deferred_interruption = _drain_finalizer_thread( - lambda: _publish_interrupted_ramp_report( + terminal_reason = child.error or child.status + mark_pending(terminal_reason) + break + report = dataclasses.replace( report, - output, - markdown_output, - ), - initial_interrupt=deferred_interruption, - ) - if deferred_interruption is not None: - raise deferred_interruption - return report + observed_topology=last_observed, + ramp=tuple(steps), + ) + write_checkpoint(report) + active_index = None + active_child_output = None + except KeyboardInterrupt as error: + if interruption is None: + interruption = error + terminal_status = "cutoff" + terminal_reason = cancellation_reason(interruption) + if ( + active_index is not None + and active_child_output is not None + and active_child_output.exists() + ): + try: + child = load_run_report(active_child_output) + except ValueError: + child = None + if child is not None and child.status in { + "completed", + "refused", + "failed", + "cutoff", + }: + steps[active_index] = step_from_child( + active_index, + child, + active_child_output, + ) + if child.status == "completed": + last_observed = child.observed_topology + elif child.status == "cutoff": + terminal_reason = child.error or terminal_reason + mark_pending(terminal_reason) + finally: + while final_report is None: + try: + finalize: t.Callable[[], RunReport] = functools.partial( + _finalize_ramp_aggregation, + report, + steps, + last_observed=last_observed, + terminal_status=terminal_status, + terminal_reason=terminal_reason, + output=output, + markdown_output=markdown_output, + ) + final_report, deferred_interruption = _drain_finalizer_thread( + finalize, + initial_interrupt=interruption, + ) + except KeyboardInterrupt as error: + if interruption is None: + interruption = error + terminal_status = "cutoff" + terminal_reason = terminal_reason or cancellation_reason(interruption) + mark_pending(terminal_reason) + continue + assert final_report is not None + if deferred_interruption is not None: + if interruption is None: + interruption = deferred_interruption + if final_report.status != "cutoff": + terminal_status = "cutoff" + terminal_reason = terminal_reason or cancellation_reason( + interruption + ) + mark_pending(terminal_reason) + final_report = None + assert final_report is not None + if interruption is not None: + raise interruption + return final_report def parse_topology(shape: str) -> Topology: diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index cbaf7df133..d8621e1984 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -69,6 +69,15 @@ "search.contents", ) +_RUNNER_PHASES = ( + "setup", + "stabilization", + "mutation.bulk", + "wait.capture-poll", + "wait.control-stream", + *_RUNNER_REPEATABLE_PHASES[2:], +) + def _benchmark_script() -> pathlib.Path: """Return the real standalone benchmark entry point.""" @@ -4860,6 +4869,25 @@ def test_validator_rejects_failed_phase_that_disagrees_with_active_row( benchmark_module.validate_report(report) +def test_validator_rejects_future_row_after_lazy_active_failure( + benchmark_module: types.ModuleType, +) -> None: + """A canonical future row is invalid even when it carries no evidence.""" + report = _failed_report_with_completed_mutation_prefix(benchmark_module) + future = benchmark_module.PhaseReport( + "wait.control-stream", + report.requested_topology, + report.requested_topology, + status="not_applicable", + warmup=report.warmup, + runs=report.runs, + ) + report = dataclasses.replace(report, phases=(*report.phases, future)) + + with pytest.raises(ValueError, match="phase status prefix"): + benchmark_module.validate_report(report) + + def test_cli_phase_failure_uses_supervisor_cleanup_contract( tmp_path: pathlib.Path, ) -> None: @@ -4895,6 +4923,72 @@ def test_cli_phase_failure_uses_supervisor_cleanup_contract( _assert_terminal_cleanup(payload) +@pytest.mark.parametrize( + "failed_phase", + ( + "enumeration.windows", + "capture.batched", + "search.snapshot.windows.middle", + ), +) +def test_cli_mid_strategy_failure_has_one_active_prefix_and_no_future_rows( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + failed_phase: str, +) -> None: + """A strategy failure cannot manufacture evidence for a later cell.""" + report_path = tmp_path / f"{failed_phase}.json" + markdown_path = report_path.with_suffix(".md") + + completed = _run_cli( + "run", + "--shape", + "1x1x1", + "--lane", + "subprocess", + "--mode", + "sync", + "--runs", + "1", + "--warmup", + "0", + "--output", + str(report_path), + "--markdown-output", + str(markdown_path), + "--scratch-root", + str(tmp_path / "scratch"), + "--watchdog-seconds", + "30", + "--_test-fail-after", + failed_phase, + cwd=tmp_path, + ) + + assert completed.returncode != 0 + report = benchmark_module.load_run_report(report_path) + phase_index = _RUNNER_PHASES.index(failed_phase) + assert report.status == "failed" + assert report.failed_phase == failed_phase + assert ( + tuple(phase.name for phase in report.phases) + == _RUNNER_PHASES[: phase_index + 1] + ) + assert all( + phase.status in {"completed", "not_applicable"} for phase in report.phases[:-1] + ) + assert report.phases[-1].status == "failed" + benchmark_module.validate_report(report) + assert failed_phase in benchmark_module.render_markdown_summary(report_path) + assert markdown_path.exists() + assert report.cleanup.complete + assert report.cleanup.errors == () + assert report.scratch_path is not None + assert report.socket_path is not None + assert not pathlib.Path(report.scratch_path).exists() + assert not pathlib.Path(report.socket_path).exists() + + def test_worker_reports_the_exact_active_stabilization_boundary( benchmark_module: types.ModuleType, monkeypatch: pytest.MonkeyPatch, @@ -5181,6 +5275,136 @@ def test_cli_cancellation_uses_supervisor_cleanup_contract( _assert_terminal_cleanup(payload) +@pytest.mark.parametrize( + "start_failure", (KeyboardInterrupt("start"), RuntimeError("start")) +) +def test_finalizer_start_after_native_launch_executes_once_and_restores_mask( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + start_failure: BaseException, +) -> None: + """A post-launch start failure must never cause a second native launch.""" + real_thread = threading.Thread + callback_started = threading.Event() + release_callback = threading.Event() + invocation_count = 0 + wrapper: t.Any = None + prior_mask = frozenset({signal.SIGUSR1}) + mask_calls: list[tuple[int, frozenset[signal.Signals]]] = [] + + def callback() -> str: + nonlocal invocation_count + invocation_count += 1 + callback_started.set() + assert release_callback.wait(timeout=10.0) + return "finished" + + def thread_factory(*, target: t.Callable[[], None], name: str) -> t.Any: + nonlocal wrapper + delegate = real_thread(target=target, name=name) + start_calls = 0 + + def start() -> None: + nonlocal start_calls + start_calls += 1 + if start_calls != 1: + pytest.fail("finalizer attempted a second native launch") + delegate.start() + raise start_failure + + wrapper = types.SimpleNamespace( + start=start, + join=delegate.join, + is_alive=delegate.is_alive, + ident=None, + start_calls=lambda: start_calls, + ) + return wrapper + + def record_mask( + how: int, + signals: t.Iterable[signal.Signals], + ) -> frozenset[signal.Signals]: + mask_calls.append((how, frozenset(signals))) + return prior_mask + + def release() -> None: + assert callback_started.wait(timeout=10.0) + release_callback.set() + + monkeypatch.setattr(benchmark_module.threading, "Thread", thread_factory) + monkeypatch.setattr(benchmark_module.signal, "pthread_sigmask", record_mask) + releaser = real_thread(target=release) + releaser.start() + try: + if isinstance(start_failure, KeyboardInterrupt): + result, interruption = benchmark_module._drain_finalizer_thread(callback) + assert result == "finished" + assert interruption is start_failure + else: + with pytest.raises(RuntimeError) as raised: + benchmark_module._drain_finalizer_thread(callback) + assert raised.value is start_failure + finally: + release_callback.set() + releaser.join(timeout=5.0) + + assert invocation_count == 1 + assert wrapper is not None + assert wrapper.start_calls() == 1 + assert not wrapper.is_alive() + assert mask_calls == [ + (signal.SIG_BLOCK, frozenset({signal.SIGINT, signal.SIGTERM})), + (signal.SIG_SETMASK, prior_mask), + ] + + +def test_finalizer_drains_repeated_sigint_once_and_restores_real_mask( + benchmark_module: types.ModuleType, +) -> None: + """Repeated SIGINT cannot duplicate work or leave a finalizer thread alive.""" + callback_started = threading.Event() + release_callback = threading.Event() + invocation_count = 0 + original_mask = signal.pthread_sigmask(signal.SIG_BLOCK, {signal.SIGUSR1}) + expected_mask = frozenset((*original_mask, signal.SIGUSR1)) + + def callback() -> int: + nonlocal invocation_count + invocation_count += 1 + callback_started.set() + assert release_callback.wait(timeout=10.0) + return 7 + + def interrupt_twice() -> None: + assert callback_started.wait(timeout=10.0) + os.kill(os.getpid(), signal.SIGINT) + time.sleep(0.05) + os.kill(os.getpid(), signal.SIGINT) + time.sleep(0.05) + release_callback.set() + + interrupter = threading.Thread(target=interrupt_twice) + interrupter.start() + try: + result, interruption = benchmark_module._drain_finalizer_thread(callback) + observed_mask = signal.pthread_sigmask(signal.SIG_BLOCK, ()) + finally: + release_callback.set() + interrupter.join(timeout=5.0) + signal.pthread_sigmask(signal.SIG_SETMASK, original_mask) + + assert result == 7 + assert interruption is not None + assert invocation_count == 1 + assert not interrupter.is_alive() + assert frozenset(observed_mask) == expected_mask + assert not any( + thread.name == "orchestration-finalization" and thread.is_alive() + for thread in threading.enumerate() + ) + + def test_supervisor_cancellation_during_final_report_write_is_durable( benchmark_module: types.ModuleType, monkeypatch: pytest.MonkeyPatch, @@ -5326,6 +5550,8 @@ def interrupt_popen(*args: t.Any, **kwargs: t.Any) -> t.Any: assert mask_calls == [ (signal.SIG_BLOCK, frozenset({signal.SIGINT, signal.SIGTERM})), (signal.SIG_SETMASK, prior_mask), + (signal.SIG_BLOCK, frozenset({signal.SIGINT, signal.SIGTERM})), + (signal.SIG_SETMASK, prior_mask), ] durable = benchmark_module.load_run_report(output) assert durable.status == "cutoff" @@ -5421,6 +5647,8 @@ def record_mask( assert mask_calls == [ (signal.SIG_BLOCK, frozenset({signal.SIGINT, signal.SIGTERM})), (signal.SIG_SETMASK, prior_mask), + (signal.SIG_BLOCK, frozenset({signal.SIGINT, signal.SIGTERM})), + (signal.SIG_SETMASK, prior_mask), ] assert durable is not None assert durable.status == "cutoff" @@ -5672,6 +5900,268 @@ def test_cli_ramp_refusal_marks_later_shapes_not_attempted( assert "not_attempted" in markdown_path.read_text(encoding="utf-8") +def _write_completed_ramp_child( + benchmark_module: types.ModuleType, + shape: t.Any, + output: pathlib.Path, + ordinal: int, + write: t.Callable[[pathlib.Path, t.Any], None], +) -> t.Any: + """Write one complete fake child artifact while keeping aggregation real.""" + child = benchmark_module.RunReport( + shape, + observed_topology=shape, + status="completed", + cleanup=benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + run_id=f"child-{ordinal}", + lane="control", + mode="async", + warmup=0, + runs=1, + scratch_path=f"scratch-{ordinal}", + socket_path=f"scratch-{ordinal}/tmux.sock", + progress_path=f"progress-{ordinal}.jsonl", + environment=benchmark_module.EnvironmentReport( + "3.10", None, 1, 11, ("run",), None + ), + ) + write(output, child) + return child + + +def test_ramp_interrupt_before_child_finalizes_unattempted_suffix( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """Pre-child interruption must still publish and validate a cutoff aggregate.""" + interruption = KeyboardInterrupt("before child") + output = tmp_path / "before-child.json" + markdown = output.with_suffix(".md") + + def interrupt_child(*_args: t.Any, **_kwargs: t.Any) -> t.NoReturn: + raise interruption + + monkeypatch.setattr(benchmark_module, "run_scenario", interrupt_child) + + with pytest.raises(KeyboardInterrupt) as raised: + benchmark_module.run_ramp( + ( + benchmark_module.Topology(1, 1, 1), + benchmark_module.Topology(2, 1, 1), + ), + runs=1, + warmup=0, + output=output, + markdown_output=markdown, + ) + + assert raised.value is interruption + report = benchmark_module.load_run_report(output) + assert report.status == "cutoff" + assert tuple(step.status for step in report.ramp) == ( + "not_attempted", + "not_attempted", + ) + assert report.error is not None + assert all(step.reason == report.error for step in report.ramp) + benchmark_module.validate_report(report) + assert report.cleanup.complete + assert markdown.exists() + + +def test_ramp_interrupt_between_child_return_and_step_mutation_preserves_child( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """A durable completed child survives interruption before step bookkeeping.""" + interruption = KeyboardInterrupt("after child") + output = tmp_path / "after-child.json" + markdown = output.with_suffix(".md") + real_write = benchmark_module.write_json_atomic + + class InterruptingChild: + def __init__(self, child: t.Any) -> None: + self.child = child + self.interrupted = False + + @property + def status(self) -> str: + if not self.interrupted: + self.interrupted = True + raise interruption + return t.cast(str, self.child.status) + + def __getattr__(self, name: str) -> t.Any: + return getattr(self.child, name) + + def complete_then_interrupt(shape: t.Any, **kwargs: t.Any) -> t.Any: + child = _write_completed_ramp_child( + benchmark_module, + shape, + kwargs["output"], + 1, + real_write, + ) + return InterruptingChild(child) + + monkeypatch.setattr(benchmark_module, "run_scenario", complete_then_interrupt) + + with pytest.raises(KeyboardInterrupt) as raised: + benchmark_module.run_ramp( + ( + benchmark_module.Topology(1, 1, 1), + benchmark_module.Topology(2, 1, 1), + ), + runs=1, + warmup=0, + output=output, + markdown_output=markdown, + ) + + assert raised.value is interruption + report = benchmark_module.load_run_report(output) + assert report.status == "cutoff" + assert tuple(step.status for step in report.ramp) == ( + "completed", + "not_attempted", + ) + assert report.ramp[1].reason == report.error + benchmark_module.validate_report(report) + assert report.cleanup.complete + assert markdown.exists() + + +def test_ramp_interrupt_during_child_checkpoint_write_preserves_completed_prefix( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """Every successful-child aggregate write is cancellation-shielded.""" + interruption = KeyboardInterrupt("child checkpoint write") + output = tmp_path / "child-checkpoint.json" + markdown = output.with_suffix(".md") + real_write = benchmark_module.write_json_atomic + child_ordinal = 0 + interrupted = False + + def complete_child(shape: t.Any, **kwargs: t.Any) -> t.Any: + nonlocal child_ordinal + child_ordinal += 1 + return _write_completed_ramp_child( + benchmark_module, + shape, + kwargs["output"], + child_ordinal, + real_write, + ) + + def interrupt_checkpoint(path: pathlib.Path, value: t.Any) -> None: + nonlocal interrupted + if ( + not interrupted + and path == output + and isinstance(value, benchmark_module.RunReport) + and value.status == "in_progress" + and any(step.status == "completed" for step in value.ramp) + ): + interrupted = True + raise interruption + real_write(path, value) + + monkeypatch.setattr(benchmark_module, "run_scenario", complete_child) + monkeypatch.setattr(benchmark_module, "write_json_atomic", interrupt_checkpoint) + + with pytest.raises(KeyboardInterrupt) as raised: + benchmark_module.run_ramp( + ( + benchmark_module.Topology(1, 1, 1), + benchmark_module.Topology(2, 1, 1), + ), + runs=1, + warmup=0, + output=output, + markdown_output=markdown, + ) + + assert raised.value is interruption + report = benchmark_module.load_run_report(output) + assert report.status == "cutoff" + assert tuple(step.status for step in report.ramp) == ( + "completed", + "not_attempted", + ) + assert report.ramp[1].reason == report.error + benchmark_module.validate_report(report) + assert report.cleanup.complete + assert markdown.exists() + + +def test_ramp_interrupt_after_last_child_preserves_every_completed_attempt( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """Final-render interruption cannot relabel a completed child as cutoff.""" + interruption = KeyboardInterrupt("before final render") + output = tmp_path / "before-render.json" + markdown = output.with_suffix(".md") + real_write = benchmark_module.write_json_atomic + real_render = benchmark_module.render_markdown_summary + child_ordinal = 0 + interrupted = False + + def complete_child(shape: t.Any, **kwargs: t.Any) -> t.Any: + nonlocal child_ordinal + child_ordinal += 1 + return _write_completed_ramp_child( + benchmark_module, + shape, + kwargs["output"], + child_ordinal, + real_write, + ) + + def interrupt_render( + report_path: pathlib.Path, + output_path: pathlib.Path | None = None, + ) -> str: + nonlocal interrupted + if not interrupted and report_path == output: + interrupted = True + raise interruption + return t.cast(str, real_render(report_path, output_path)) + + monkeypatch.setattr(benchmark_module, "run_scenario", complete_child) + monkeypatch.setattr(benchmark_module, "render_markdown_summary", interrupt_render) + + with pytest.raises(KeyboardInterrupt) as raised: + benchmark_module.run_ramp( + ( + benchmark_module.Topology(1, 1, 1), + benchmark_module.Topology(2, 1, 1), + ), + runs=1, + warmup=0, + output=output, + markdown_output=markdown, + ) + + assert raised.value is interruption + report = benchmark_module.load_run_report(output) + assert report.status == "cutoff" + assert tuple(step.status for step in report.ramp) == ("completed", "completed") + benchmark_module.validate_report(report) + assert report.cleanup.complete + assert markdown.exists() + + def test_ramp_cancellation_during_aggregation_is_durable( benchmark_module: types.ModuleType, monkeypatch: pytest.MonkeyPatch, @@ -5757,7 +6247,7 @@ def interrupt_aggregation() -> None: assert isinstance(cancellation.value, KeyboardInterrupt) report = benchmark_module.load_run_report(output) assert report.status == "cutoff" - assert report.ramp[-1].status == "cutoff" + assert tuple(step.status for step in report.ramp) == ("completed", "completed") assert report.cleanup.complete assert "cutoff" in markdown.read_text(encoding="utf-8") From 438f31e6ac368003f3be606cba473e2b011eba62 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 15:50:51 -0500 Subject: [PATCH 23/67] Bench(fix[runner]): Drain repeated interrupts why: Repeated signals and seeded phase families could escape terminal finalization, overwrite child failures, or publish biased partial evidence. what: - Defer repeated signals through one exactly-once terminal finalizer - Preserve ramp terminal outcomes and validate lazy interleaved prefixes - Bind exact tmux server identities during socket fallback cleanup - Add deterministic signal, scheduler, ramp, and orphan regressions --- scripts/bench_orchestration.py | 771 ++++++++++++++++------- tests/test_bench_orchestration_script.py | 708 ++++++++++++++++++++- 2 files changed, 1245 insertions(+), 234 deletions(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index 933980fbf4..7b9a8273c7 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -94,6 +94,23 @@ "wait.control-stream", *_RUNNER_REPEATABLE_PHASES[2:], ) +_RUNNER_PHASE_GROUPS = ( + ("setup",), + ("stabilization",), + ("mutation.bulk",), + ("wait.capture-poll", "wait.control-stream"), + tuple(f"enumeration.{kind}" for kind in _ENUMERATION_KINDS), + ("capture.serial", "capture.batched"), + ( + *( + f"search.{family}.{kind}.{position}" + for family in _SEARCH_FAMILIES + for kind in _ENUMERATION_KINDS + for position in _SEARCH_POSITIONS + ), + "search.contents", + ), +) def _is_terminal_safe_component(value: object) -> bool: @@ -3008,11 +3025,41 @@ def _validate_executable_report(report: RunReport) -> None: if not report.scratch_path or not report.socket_path or not report.progress_path: message = "attempted executable report requires owned resource paths" raise ValueError(message) - phase_names = tuple(phase.name for phase in report.phases) - if phase_names != _RUNNER_PHASES[: len(phase_names)]: - message = "attempted executable report phases must be a runner phase prefix" - raise ValueError(message) terminal_unsuccessful = report.status in {"failed", "cutoff"} + phase_names = tuple(phase.name for phase in report.phases) + canonical_prefix = phase_names == _RUNNER_PHASES[: len(phase_names)] + if not canonical_prefix: + active_names = tuple( + phase.name + for phase in report.phases + if phase.status in {"failed", "in_progress"} + ) + active_name = active_names[0] if len(active_names) == 1 else None + interleaved_terminal_prefix = False + if terminal_unsuccessful and len(active_names) <= 1: + for group_index, group in enumerate(_RUNNER_PHASE_GROUPS): + prior_names = tuple( + name + for prior_group in _RUNNER_PHASE_GROUPS[:group_index] + for name in prior_group + ) + active_suffix = phase_names[len(prior_names) :] + if ( + phase_names[: len(prior_names)] != prior_names + or not active_suffix + or len(set(active_suffix)) != len(active_suffix) + or not all(name in group for name in active_suffix) + ): + continue + if active_name is not None and active_suffix[-1] != active_name: + continue + if active_name is None and report.failed_phase != "cancellation": + continue + interleaved_terminal_prefix = True + break + if not interleaved_terminal_prefix: + message = "attempted executable report phases must be a runner phase prefix" + raise ValueError(message) if terminal_unsuccessful and (not report.failed_phase or not report.error): message = "terminal unsuccessful report requires phase and error" raise ValueError(message) @@ -3889,6 +3936,7 @@ def _kill_exact_tmux_socket( socket_path: pathlib.Path, *, timeout_s: float, + server_identity: ProcessIdentity | None = None, ) -> tuple[str, ...]: """Boundedly request ``kill-server`` on one exact isolated socket. @@ -3901,11 +3949,13 @@ def _kill_exact_tmux_socket( Exact isolated socket owned by one benchmark run. timeout_s : float Maximum wait for the helper and each stable-handle escalation. + server_identity : ProcessIdentity | None + Already recorded exact server identity, queried from the socket when absent. Returns ------- tuple[str, ...] - Empty only when the socket is absent after the bounded request. + Empty only when the exact server and socket are absent afterward. Raises ------ @@ -3921,6 +3971,51 @@ def _kill_exact_tmux_socket( environment.pop("TMUX", None) environment.pop("TMUX_PANE", None) errors: list[str] = [] + identity_was_supplied = server_identity is not None + if server_identity is None: + try: + server_pid_result = subprocess.run( + ( + "tmux", + "-S", + str(socket_path), + "display-message", + "-p", + "#{pid}", + ), + env=environment, + check=False, + capture_output=True, + text=True, + timeout=timeout_s, + ) + except subprocess.TimeoutExpired: + return ("exact socket server identity query timed out",) + except OSError as error: + return (f"exact socket server identity: {type(error).__name__}: {error}",) + if server_pid_result.returncode != 0: + detail = server_pid_result.stderr.strip() + suffix = f": {detail}" if detail else "" + exit_code = server_pid_result.returncode + return (f"exact socket server identity query exited {exit_code}{suffix}",) + try: + server_identity = _record_process( + "server", + int(server_pid_result.stdout.strip()), + ) + except (RuntimeError, ValueError) as error: + return (f"exact socket server identity: {type(error).__name__}: {error}",) + + if identity_was_supplied and not _wait_identity_absence( + (server_identity,), + timeout_s=timeout_s, + ): + return _remove_proven_stale_socket(socket_path, (server_identity,)) + + server_registry = _PidfdRegistry() + server_registry.retain(server_identity) + helper: subprocess.Popen[bytes] | None = None + helper_timed_out = False try: helper = subprocess.Popen( ("tmux", "-S", str(socket_path), "kill-server"), @@ -3930,42 +4025,61 @@ def _kill_exact_tmux_socket( stderr=subprocess.DEVNULL, ) except OSError as error: - return (f"exact socket kill-server: {type(error).__name__}: {error}",) - try: - identity = _record_process("tmux-kill-server", helper.pid) - except RuntimeError: + errors.append(f"exact socket kill-server: {type(error).__name__}: {error}") + if helper is not None: try: - helper.wait(timeout=timeout_s) - except subprocess.TimeoutExpired: - errors.append("exact socket kill-server identity unavailable") - else: - registry = _PidfdRegistry() - try: - registry.retain(identity) + helper_identity = _record_process("tmux-kill-server", helper.pid) + except RuntimeError: try: helper.wait(timeout=timeout_s) except subprocess.TimeoutExpired: - registry.signal(identity, signal.SIGTERM) + helper_timed_out = True + errors.append("exact socket kill-server identity unavailable") + else: + registry = _PidfdRegistry() + try: + registry.retain(helper_identity) try: helper.wait(timeout=timeout_s) except subprocess.TimeoutExpired: - registry.signal(identity, signal.SIGKILL) + helper_timed_out = True + registry.signal(helper_identity, signal.SIGTERM) try: helper.wait(timeout=timeout_s) except subprocess.TimeoutExpired: - errors.append("exact socket kill-server helper remains") - errors.extend(registry.errors) - finally: - registry.close() - if helper.returncode == 0 and socket_path.exists(): - try: - socket_status = socket_path.lstat() - if not stat.S_ISSOCK(socket_status.st_mode): - errors.append(f"configured socket was replaced: {socket_path}") - else: - socket_path.unlink() - except OSError as error: - errors.append(f"exact socket path removal: {type(error).__name__}: {error}") + registry.signal(helper_identity, signal.SIGKILL) + try: + helper.wait(timeout=timeout_s) + except subprocess.TimeoutExpired: + errors.append("exact socket kill-server helper remains") + errors.extend(registry.errors) + finally: + registry.close() + if helper_timed_out: + errors.append("exact socket kill-server timed out") + elif helper.returncode != 0: + errors.append(f"exact socket kill-server exited {helper.returncode}") + + try: + survivors = _wait_identity_absence((server_identity,), timeout_s=timeout_s) + for signal_number in (signal.SIGTERM, signal.SIGKILL): + if not survivors: + break + server_registry.signal(server_identity, signal_number) + survivors = _wait_identity_absence( + (server_identity,), + timeout_s=timeout_s, + ) + if survivors: + errors.append( + f"server pid {server_identity.pid} with start time " + f"{server_identity.start_time} remains" + ) + elif socket_path.exists(): + errors.extend(_remove_proven_stale_socket(socket_path, (server_identity,))) + errors.extend(server_registry.errors) + finally: + server_registry.close() return tuple(errors) @@ -8564,7 +8678,17 @@ async def cleanup_run( except Exception as error: # noqa: BLE001 errors.append(f"server kill: {type(error).__name__}: {error}") if context.socket_path.exists(): - errors.extend(_kill_exact_tmux_socket(context.socket_path, timeout_s=grace_s)) + server_identity = next( + (identity for identity in context.processes if identity.role == "server"), + None, + ) + errors.extend( + _kill_exact_tmux_socket( + context.socket_path, + timeout_s=grace_s, + server_identity=server_identity, + ) + ) survivors = await _wait_for_process_absence( context.processes, @@ -9253,7 +9377,7 @@ async def _run_worker_group( runs : int Timed accepted calls per cell. seed : int - Deterministic per-cell measurement seed. + Deterministic family-order seed. policy : ResourcePolicy Runtime guard thresholds. latest : dict[str, PhaseMeasurement] @@ -9273,6 +9397,37 @@ async def _run_worker_group( raise ValueError(message) topology = context.topology + strategy_names = tuple(strategies) + prior_phases = recorder.report.phases + phase_states: dict[str, PhaseReport] = {} + + def publish_active(strategy: str) -> None: + completed = tuple( + phase for phase in phase_states.values() if phase.status == "completed" + ) + active = phase_states[strategy] + active_suffix = () if active.status == "completed" else (active,) + recorder.report = dataclasses.replace( + recorder.report, + phases=(*prior_phases, *completed, *active_suffix), + ) + + async def on_boundary(strategy: str) -> None: + active_phase[0] = strategy + if strategy in phase_states: + publish_active(strategy) + return + phase_states[strategy] = PhaseReport( + name=strategy, + requested_topology=topology, + observed_topology=topology, + status="in_progress", + warmup=warmup, + runs=runs, + ) + publish_active(strategy) + recorder.checkpoint(f"{strategy}.started") + async def on_progress( stage: str, strategy: str, @@ -9281,8 +9436,7 @@ async def on_progress( sample: RawSample | None, ) -> None: latest[strategy] = measurement - phases = {phase.name: phase for phase in recorder.report.phases} - phase = phases[strategy] + phase = phase_states[strategy] observation = _observation_for_measurement(strategy, ordinal, measurement) if stage == "warmup": phase = dataclasses.replace( @@ -9297,53 +9451,66 @@ async def on_progress( ) if ordinal == runs - 1: phase = _completed_phase(phase) - recorder.report = _replace_phase(recorder.report, phase) + phase_states[strategy] = phase checkpoint = ( strategy if sample is not None and ordinal == runs - 1 else f"{strategy}.{stage}.{ordinal}" ) + will_fail = checkpoint == strategy and fail_after == strategy + if not will_fail and all( + phase_states.get(name) is not None + and phase_states[name].status == "completed" + for name in strategy_names + ): + recorder.report = dataclasses.replace( + recorder.report, + phases=( + *prior_phases, + *(phase_states[name] for name in strategy_names), + ), + ) + else: + publish_active(strategy) recorder.checkpoint(checkpoint) - if checkpoint == strategy and fail_after == strategy: + if will_fail: message = f"injected phase failure after {strategy}" raise RuntimeError(message) - for strategy_index, (name, strategy) in enumerate(strategies.items()): - active_phase[0] = name - recorder.report = _replace_phase( + def retain_terminal_group(strategy: str, *, failed: bool) -> None: + active = phase_states[strategy] + if failed: + active = dataclasses.replace(active, status="failed", summary=None) + completed = tuple( + phase + for name, phase in phase_states.items() + if name != strategy and phase.status == "completed" + ) + recorder.report = dataclasses.replace( recorder.report, - PhaseReport( - name=name, - requested_topology=topology, - observed_topology=topology, - status="in_progress", - warmup=warmup, - runs=runs, - ), + phases=(*prior_phases, *completed, active), ) - recorder.checkpoint(f"{name}.started") + + try: result = await run_repeatable_phase( - {name: strategy}, + strategies, warmup=warmup, runs=runs, - seed=seed + strategy_index, + seed=seed, snapshot_resources=lambda: probe_host(ProcessReader()), live_postcondition=lambda measurement: _worker_live_postcondition( context, measurement, policy ), progress_callback=on_progress, - boundary_callback=lambda strategy_name: active_phase.__setitem__( - 0, strategy_name - ), + boundary_callback=on_boundary, ) - if result.failure is not None: - phases = {phase.name: phase for phase in recorder.report.phases} - failed = dataclasses.replace( - phases[result.failure.strategy], status="failed", summary=None - ) - recorder.report = _replace_phase(recorder.report, failed) - recorder.checkpoint(f"{result.failure.strategy}.failed") - raise PhaseExecutionError(result.failure.strategy, result.failure) + except RuntimeCutoffError: + retain_terminal_group(active_phase[0], failed=False) + raise + if result.failure is not None: + retain_terminal_group(result.failure.strategy, failed=True) + recorder.checkpoint(f"{result.failure.strategy}.failed") + raise PhaseExecutionError(result.failure.strategy, result.failure) def _position_target(ids: tuple[str, ...], position: str) -> str: @@ -10203,7 +10370,17 @@ def wait_worker() -> bool: process_identity_matches(identity) for identity in owned ) if socket_path.exists(): - errors.extend(_kill_exact_tmux_socket(socket_path, timeout_s=grace_s)) + server_identity = next( + (identity for identity in owned if identity.role == "server"), + None, + ) + errors.extend( + _kill_exact_tmux_socket( + socket_path, + timeout_s=grace_s, + server_identity=server_identity, + ) + ) if processes_absent and socket_path.exists(): errors.extend(_remove_proven_stale_socket(socket_path, owned)) socket_absent = not socket_path.exists() @@ -10416,6 +10593,7 @@ def _drain_finalizer_thread( callback: cabc.Callable[[], _FinalizerResult], *, initial_interrupt: KeyboardInterrupt | None = None, + interrupt_state: list[KeyboardInterrupt | None] | None = None, ) -> tuple[_FinalizerResult, KeyboardInterrupt | None]: """Drain independent synchronous finalization through repeated interrupts. @@ -10429,6 +10607,8 @@ def _drain_finalizer_thread( Recovery, report, or ramp finalization that must run to completion. initial_interrupt : KeyboardInterrupt | None First interruption already translated into terminal report state. + interrupt_state : list[KeyboardInterrupt | None] | None + Optional single-item state shared with an interruption-aware callback. Returns ------- @@ -10440,12 +10620,21 @@ def _drain_finalizer_thread( BaseException Callback failure after its independently owned thread terminates. """ + if interrupt_state is None: + interrupt_state = [initial_interrupt] + elif len(interrupt_state) != 1: + message = "finalizer interrupt state must contain exactly one item" + raise ValueError(message) + elif interrupt_state[0] is None: + interrupt_state[0] = initial_interrupt results: list[_FinalizerResult] = [] failures: list[BaseException] = [] completed = threading.Event() + launch_released = threading.Event() def run() -> None: try: + launch_released.wait() results.append(callback()) except BaseException as error: # noqa: BLE001 failures.append(error) @@ -10456,31 +10645,129 @@ def run() -> None: target=run, name="orchestration-finalization", ) - interruption = initial_interrupt + + def record_interrupt(error: KeyboardInterrupt) -> None: + if interrupt_state[0] is None: + interrupt_state[0] = error + + interruption_failures: list[BaseException] = [] launch_failure: BaseException | None = None blocked = frozenset({signal.SIGINT, signal.SIGTERM}) + previous_handlers = { + signal_number: signal.getsignal(signal_number) for signal_number in blocked + } + + def record_signal( + signal_number: int, + frame: types.FrameType | None, + ) -> None: + previous = previous_handlers[signal.Signals(signal_number)] + if previous == signal.SIG_IGN: + return + if callable(previous): + try: + previous(signal_number, frame) + except KeyboardInterrupt as error: + record_interrupt(error) + except BaseException as error: # noqa: BLE001 + interruption_failures.append(error) + return + message = f"received {signal.Signals(signal_number).name} during finalization" + record_interrupt(KeyboardInterrupt(message)) + previous_mask = signal.pthread_sigmask(signal.SIG_BLOCK, blocked) + installed_handlers: list[signal.Signals] = [] try: + for signal_number in blocked: + signal.signal(signal_number, record_signal) + installed_handlers.append(signal_number) try: thread.start() except KeyboardInterrupt as error: - if interruption is None: - interruption = error + record_interrupt(error) except BaseException as error: # noqa: BLE001 launch_failure = error finally: + while True: + try: + signal.pthread_sigmask(signal.SIG_SETMASK, previous_mask) + break + except KeyboardInterrupt as error: + record_interrupt(error) + except BaseException as error: # noqa: BLE001 + if launch_failure is None: + launch_failure = error + break + + launched = False + while True: try: - signal.pthread_sigmask(signal.SIG_SETMASK, previous_mask) + identity = thread.ident + alive = thread.is_alive() + launched = identity is not None or alive + break except KeyboardInterrupt as error: - if interruption is None: - interruption = error - except BaseException as error: # noqa: BLE001 - if launch_failure is None: - launch_failure = error - try: - launched = thread.ident is not None or thread.is_alive() - except RuntimeError: - launched = thread.ident is not None + record_interrupt(error) + except RuntimeError: + launched = thread.ident is not None + break + if launched: + while True: + try: + launch_released.set() + break + except KeyboardInterrupt as error: + record_interrupt(error) + while True: + try: + if completed.is_set(): + break + completed.wait(timeout=0.02) + except KeyboardInterrupt as error: + record_interrupt(error) + while True: + try: + alive = thread.is_alive() + except KeyboardInterrupt as error: + record_interrupt(error) + continue + except RuntimeError: + alive = False + if not alive: + break + try: + thread.join(timeout=0.02) + except KeyboardInterrupt as error: + record_interrupt(error) + except RuntimeError: + pass + + final_result: _FinalizerResult | None = None + callback_failure: BaseException | None = None + while launched: + try: + if failures: + callback_failure = failures[0] + else: + final_result = results[0] + break + except KeyboardInterrupt as error: + record_interrupt(error) + + restore_failure: BaseException | None = None + for signal_number in reversed(installed_handlers): + while True: + try: + signal.signal(signal_number, previous_handlers[signal_number]) + break + except KeyboardInterrupt as error: + record_interrupt(error) + except BaseException as error: # noqa: BLE001 + if restore_failure is None: + restore_failure = error + break + + interruption = interrupt_state[0] if not launched: if interruption is not None: raise interruption @@ -10488,31 +10775,23 @@ def run() -> None: raise launch_failure message = "finalization thread did not launch" raise RuntimeError(message) - while not completed.is_set(): - try: - completed.wait(timeout=0.02) - except KeyboardInterrupt as error: # noqa: PERF203 - if interruption is None: - interruption = error - while thread.is_alive(): - try: - thread.join(timeout=0.02) - except KeyboardInterrupt as error: # noqa: PERF203 - if interruption is None: - interruption = error - except RuntimeError: - pass if launch_failure is not None: if interruption is not None: raise interruption from launch_failure - if failures: - raise launch_failure from failures[0] + if callback_failure is not None: + raise launch_failure from callback_failure raise launch_failure - if failures: + if callback_failure is not None: if interruption is not None: - raise interruption from failures[0] - raise failures[0] - return results[0], interruption + raise interruption from callback_failure + raise callback_failure + if interruption_failures: + raise interruption_failures[0] + if restore_failure is not None: + if interruption is not None: + raise interruption from restore_failure + raise restore_failure + return t.cast(_FinalizerResult, final_result), interruption def _finalize_supervised_worker( @@ -10934,35 +11213,34 @@ def request_termination( failed_phase = "supervisor" terminal_error = f"{type(error).__name__}: {error}" - try: - final_report, deferred_interruption = _drain_finalizer_thread( - lambda: _finalize_supervised_worker( - worker, - worker_identity, - progress, - topology=topology, - lane=lane, - mode=mode, - runs=runs, - warmup=warmup, - seed=seed, - run_id=run_id, - scratch=scratch, - socket_path=socket_path, - output=output, - markdown_output=markdown_output, - checkpoint_path=checkpoint_path, - admission_path=admission_path, - guard_decision=guard_decision, - original_guard_decision=original_guard_decision, - cleanup_grace_s=cleanup_grace_s, - supervisor_status=supervisor_status, - failed_phase=failed_phase, - terminal_error=terminal_error, - ), - initial_interrupt=interruption, + interrupt_state = [interruption] + + def finalize_once() -> RunReport: + final_report = _finalize_supervised_worker( + worker, + worker_identity, + progress, + topology=topology, + lane=lane, + mode=mode, + runs=runs, + warmup=warmup, + seed=seed, + run_id=run_id, + scratch=scratch, + socket_path=socket_path, + output=output, + markdown_output=markdown_output, + checkpoint_path=checkpoint_path, + admission_path=admission_path, + guard_decision=guard_decision, + original_guard_decision=original_guard_decision, + cleanup_grace_s=cleanup_grace_s, + supervisor_status=supervisor_status, + failed_phase=failed_phase, + terminal_error=terminal_error, ) - if deferred_interruption is not None and ( + if interrupt_state[0] is not None and ( final_report.cleanup.complete and not any(phase.status == "failed" for phase in final_report.phases) and ( @@ -10970,14 +11248,19 @@ def request_termination( or final_report.failed_phase != "cancellation" ) ): - final_report, deferred_interruption = _drain_finalizer_thread( - lambda: _publish_interrupted_supervisor_report( - final_report, - output, - markdown_output, - ), - initial_interrupt=deferred_interruption, + final_report = _publish_interrupted_supervisor_report( + final_report, + output, + markdown_output, ) + return final_report + + try: + final_report, deferred_interruption = _drain_finalizer_thread( + finalize_once, + initial_interrupt=interruption, + interrupt_state=interrupt_state, + ) if deferred_interruption is not None: raise deferred_interruption return final_report @@ -11312,6 +11595,9 @@ def _finalize_ramp_aggregation( terminal_reason: str | None, output: pathlib.Path, markdown_output: pathlib.Path, + active_index: int | None = None, + active_child_output: pathlib.Path | None = None, + interrupt_state: list[KeyboardInterrupt | None] | None = None, ) -> RunReport: """Validate child cleanup and durably publish one terminal ramp aggregate. @@ -11323,31 +11609,119 @@ def _finalize_ramp_aggregation( RunReport Validated aggregate retaining every declared step. """ - cleanup_complete = all( - step.status == "not_attempted" - or load_run_report(pathlib.Path(t.cast(str, step.report_path))).cleanup.complete - for step in steps - ) - cleanup = CleanupReport( - cleanup_complete, - processes_absent=cleanup_complete, - socket_absent=cleanup_complete, - scratch_absent=cleanup_complete, - ) - final_status: t.Literal["completed", "refused", "failed", "cutoff"] = ( - terminal_status or "completed" - ) - terminal = dataclasses.replace( - report, - status=final_status, - observed_topology=last_observed, - cleanup=cleanup, - ramp=tuple(steps), - error=terminal_reason, + state: list[KeyboardInterrupt | None] = ( + interrupt_state if interrupt_state is not None else [None] ) - write_json_atomic(output, terminal) - validate_report(terminal) - render_markdown_summary(output, markdown_output) + resolved_steps = list(steps) + resolved_observed = last_observed + + def record_interrupt(error: KeyboardInterrupt) -> None: + if state[0] is None: + state[0] = error + + def interruption_reason(error: KeyboardInterrupt) -> str: + detail = str(error) + suffix = f": {detail}" if detail else "" + return f"KeyboardInterrupt: ramp bookkeeping interrupted{suffix}" + + if ( + active_index is not None + and active_child_output is not None + and active_child_output.exists() + ): + try: + child = load_run_report(active_child_output) + except ValueError: + child = None + if child is not None and child.status in { + "completed", + "refused", + "failed", + "cutoff", + }: + resolved_steps[active_index] = RampStep( + resolved_steps[active_index].shape, + child.status, + child.error, + run_id=child.run_id, + report_path=str(active_child_output), + scratch_path=child.scratch_path, + socket_path=child.socket_path, + ) + if child.status == "completed": + resolved_observed = child.observed_topology + + def build_terminal() -> RunReport: + final_status: t.Literal["completed", "refused", "failed", "cutoff"] + existing_terminal = next( + ( + step + for step in resolved_steps + if step.status in {"refused", "failed", "cutoff"} + ), + None, + ) + if existing_terminal is not None: + final_status = t.cast( + t.Literal["refused", "failed", "cutoff"], + existing_terminal.status, + ) + final_reason = existing_terminal.reason or final_status + elif terminal_status is not None: + final_status = terminal_status + final_reason = terminal_reason or terminal_status + elif state[0] is not None: + final_status = "cutoff" + final_reason = interruption_reason(state[0]) + else: + final_status = "completed" + final_reason = None + if final_status != "completed": + for index, step in enumerate(resolved_steps): + if step.status == "not_attempted": + resolved_steps[index] = dataclasses.replace( + step, + reason=final_reason, + ) + cleanup_complete = all( + step.status == "not_attempted" + or load_run_report( + pathlib.Path(t.cast(str, step.report_path)) + ).cleanup.complete + for step in resolved_steps + ) + cleanup = CleanupReport( + cleanup_complete, + processes_absent=cleanup_complete, + socket_absent=cleanup_complete, + scratch_absent=cleanup_complete, + ) + return dataclasses.replace( + report, + status=final_status, + observed_topology=resolved_observed, + cleanup=cleanup, + ramp=tuple(resolved_steps), + error=final_reason, + ) + + def publish(terminal: RunReport) -> None: + write_json_atomic(output, terminal) + validate_report(terminal) + render_markdown_summary(output, markdown_output) + + observed_interrupt = state[0] + terminal = build_terminal() + try: + publish(terminal) + except KeyboardInterrupt as error: + record_interrupt(error) + terminal = build_terminal() + publish(terminal) + else: + if state[0] is not observed_interrupt: + terminal = build_terminal() + publish(terminal) return terminal @@ -11442,11 +11816,6 @@ def write_checkpoint(candidate: RunReport) -> None: if deferred is not None: raise deferred - def cancellation_reason(error: KeyboardInterrupt) -> str: - detail = str(error) - suffix = f": {detail}" if detail else "" - return f"KeyboardInterrupt: ramp bookkeeping interrupted{suffix}" - def mark_pending(reason: str) -> None: for pending_index, step in enumerate(steps): if step.status == "not_attempted": @@ -11499,68 +11868,28 @@ def mark_pending(reason: str) -> None: except KeyboardInterrupt as error: if interruption is None: interruption = error - terminal_status = "cutoff" - terminal_reason = cancellation_reason(interruption) - if ( - active_index is not None - and active_child_output is not None - and active_child_output.exists() - ): - try: - child = load_run_report(active_child_output) - except ValueError: - child = None - if child is not None and child.status in { - "completed", - "refused", - "failed", - "cutoff", - }: - steps[active_index] = step_from_child( - active_index, - child, - active_child_output, - ) - if child.status == "completed": - last_observed = child.observed_topology - elif child.status == "cutoff": - terminal_reason = child.error or terminal_reason - mark_pending(terminal_reason) finally: - while final_report is None: - try: - finalize: t.Callable[[], RunReport] = functools.partial( - _finalize_ramp_aggregation, - report, - steps, - last_observed=last_observed, - terminal_status=terminal_status, - terminal_reason=terminal_reason, - output=output, - markdown_output=markdown_output, - ) - final_report, deferred_interruption = _drain_finalizer_thread( - finalize, - initial_interrupt=interruption, - ) - except KeyboardInterrupt as error: - if interruption is None: - interruption = error - terminal_status = "cutoff" - terminal_reason = terminal_reason or cancellation_reason(interruption) - mark_pending(terminal_reason) - continue - assert final_report is not None - if deferred_interruption is not None: - if interruption is None: - interruption = deferred_interruption - if final_report.status != "cutoff": - terminal_status = "cutoff" - terminal_reason = terminal_reason or cancellation_reason( - interruption - ) - mark_pending(terminal_reason) - final_report = None + interrupt_state = [interruption] + finalize: t.Callable[[], RunReport] = functools.partial( + _finalize_ramp_aggregation, + report, + steps, + last_observed=last_observed, + terminal_status=terminal_status, + terminal_reason=terminal_reason, + output=output, + markdown_output=markdown_output, + active_index=active_index, + active_child_output=active_child_output, + interrupt_state=interrupt_state, + ) + final_report, deferred_interruption = _drain_finalizer_thread( + finalize, + initial_interrupt=interruption, + interrupt_state=interrupt_state, + ) + if interruption is None: + interruption = deferred_interruption assert final_report is not None if interruption is not None: raise interruption diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index d8621e1984..b2194b9081 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -1258,11 +1258,13 @@ def test_pidfd_registry_rejects_an_already_mismatched_identity( def test_exact_socket_fallback_kills_only_the_configured_tmux_server( benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, tmp_path: pathlib.Path, ) -> None: - """Fallback cleanup must address one socket and preserve another daemon.""" + """A failed helper must not hide a live exact server behind an unlinked socket.""" target = tmp_path / "target.sock" unrelated = tmp_path / "unrelated.sock" + identities: dict[pathlib.Path, t.Any] = {} for socket_path in (target, unrelated): created = subprocess.run( ( @@ -1279,11 +1281,42 @@ def test_exact_socket_fallback_kills_only_the_configured_tmux_server( text=True, ) assert created.returncode == 0, created.stderr + server_pid = subprocess.run( + ("tmux", "-S", str(socket_path), "display-message", "-p", "#{pid}"), + check=False, + capture_output=True, + text=True, + ) + assert server_pid.returncode == 0, server_pid.stderr + identities[socket_path] = benchmark_module._record_process( + "server", int(server_pid.stdout.strip()) + ) + + real_popen = subprocess.Popen + + def unlink_without_killing_server( + args: t.Any, + **kwargs: t.Any, + ) -> subprocess.Popen[bytes]: + if tuple(args) == ("tmux", "-S", str(target), "kill-server"): + args = ( + sys.executable, + "-c", + "import os, sys; os.unlink(sys.argv[1]); raise SystemExit(7)", + str(target), + ) + return t.cast("subprocess.Popen[bytes]", real_popen(args, **kwargs)) + + monkeypatch.setattr( + benchmark_module.subprocess, "Popen", unlink_without_killing_server + ) try: errors = benchmark_module._kill_exact_tmux_socket(target, timeout_s=1.0) - assert errors == () + assert errors == ("exact socket kill-server exited 7",) assert not target.exists() + assert not benchmark_module.process_identity_matches(identities[target]) + assert benchmark_module.process_identity_matches(identities[unrelated]) still_alive = subprocess.run( ("tmux", "-S", str(unrelated), "list-sessions"), check=False, @@ -1292,12 +1325,30 @@ def test_exact_socket_fallback_kills_only_the_configured_tmux_server( ) assert still_alive.returncode == 0, still_alive.stderr finally: - subprocess.run( - ("tmux", "-S", str(unrelated), "kill-server"), - check=False, - capture_output=True, - text=True, - ) + for socket_path in (target, unrelated): + real_popen( + ("tmux", "-S", str(socket_path), "kill-server"), + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ).wait(timeout=2.0) + registry = benchmark_module._PidfdRegistry() + try: + for identity in identities.values(): + if registry.retain(identity): + registry.signal(identity, signal.SIGKILL) + benchmark_module._wait_identity_absence(identities.values(), timeout_s=2.0) + finally: + registry.close() + for socket_path, identity in identities.items(): + benchmark_module._remove_proven_stale_socket(socket_path, (identity,)) + + assert all( + not benchmark_module.process_identity_matches(identity) + for identity in identities.values() + ) + assert not target.exists() + assert not unrelated.exists() def test_run_scenario_refuses_when_pidfds_are_unavailable( @@ -3849,6 +3900,275 @@ def fail_second() -> t.Any: assert result.failure.error == "RuntimeError: live postcondition lost" +def test_worker_group_preserves_seeded_round_robin_strategy_order( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Calling each strategy as a separate phase would restore order bias.""" + topology = benchmark_module.Topology(1, 1, 1) + calls: list[str] = [] + repeatable_calls = 0 + real_repeatable = benchmark_module.run_repeatable_phase + + class Recorder: + def __init__(self) -> None: + self.report = benchmark_module.RunReport(topology) + self.checkpoints: list[str] = [] + + def checkpoint(self, name: str) -> None: + self.checkpoints.append(name) + + def measured(name: str) -> t.Callable[[], t.Any]: + def run() -> t.Any: + calls.append(name) + return benchmark_module.SearchResult( + duration_ns=17, + family="snapshot", + kind="sessions", + scanned_count=1, + target="$1", + matched_ids=("$1",), + verified=True, + ) + + return run + + async def count_repeatable(*args: t.Any, **kwargs: t.Any) -> t.Any: + nonlocal repeatable_calls + repeatable_calls += 1 + return await real_repeatable(*args, **kwargs) + + async def live_postcondition(*_args: t.Any, **_kwargs: t.Any) -> bool: + return True + + monkeypatch.setattr(benchmark_module, "run_repeatable_phase", count_repeatable) + monkeypatch.setattr( + benchmark_module, "_worker_live_postcondition", live_postcondition + ) + monkeypatch.setattr( + benchmark_module, + "probe_host", + lambda _reader: benchmark_module.HostSnapshot(), + ) + recorder = Recorder() + latest: dict[str, t.Any] = {} + + asyncio.run( + benchmark_module._run_worker_group( + recorder, + types.SimpleNamespace(topology=topology), + { + "alpha": measured("alpha"), + "beta": measured("beta"), + "gamma": measured("gamma"), + }, + warmup=1, + runs=2, + seed=11, + policy=benchmark_module.ResourcePolicy(), + latest=latest, + fail_after=None, + active_phase=["setup"], + ) + ) + + assert repeatable_calls == 1 + assert calls == [ + "alpha", + "gamma", + "beta", + "gamma", + "beta", + "alpha", + "beta", + "alpha", + "gamma", + ] + assert tuple(phase.name for phase in recorder.report.phases) == ( + "alpha", + "beta", + "gamma", + ) + assert all(phase.status == "completed" for phase in recorder.report.phases) + assert all( + len(phase.warmup_observations) == 1 and len(phase.observations) == 2 + for phase in recorder.report.phases + ) + + +def test_worker_group_failure_retains_only_invoked_terminal_prefix( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """A failed interleaved strategy must not create any uninvoked phase row.""" + topology = benchmark_module.Topology(1, 1, 1) + calls: list[str] = [] + + class Recorder: + def __init__(self) -> None: + self.report = benchmark_module.RunReport(topology) + + def checkpoint(self, _name: str) -> None: + return None + + def measured(name: str, *, fail: bool = False) -> t.Callable[[], t.Any]: + def run() -> t.Any: + calls.append(name) + if fail: + message = f"{name} failed" + raise RuntimeError(message) + return benchmark_module.SearchResult( + duration_ns=19, + family="snapshot", + kind="sessions", + scanned_count=1, + target="$1", + matched_ids=("$1",), + verified=True, + ) + + return run + + async def live_postcondition(*_args: t.Any, **_kwargs: t.Any) -> bool: + return True + + monkeypatch.setattr( + benchmark_module, "_worker_live_postcondition", live_postcondition + ) + monkeypatch.setattr( + benchmark_module, + "probe_host", + lambda _reader: benchmark_module.HostSnapshot(), + ) + recorder = Recorder() + + with pytest.raises(benchmark_module.PhaseExecutionError): + asyncio.run( + benchmark_module._run_worker_group( + recorder, + types.SimpleNamespace(topology=topology), + { + "alpha": measured("alpha"), + "beta": measured("beta"), + "gamma": measured("gamma", fail=True), + }, + warmup=0, + runs=1, + seed=11, + policy=benchmark_module.ResourcePolicy(), + latest={}, + fail_after=None, + active_phase=["setup"], + ) + ) + + assert calls == ["alpha", "gamma"] + assert tuple(phase.name for phase in recorder.report.phases) == ( + "alpha", + "gamma", + ) + assert tuple(phase.status for phase in recorder.report.phases) == ( + "completed", + "failed", + ) + terminal = dataclasses.replace( + recorder.report, + status="failed", + cleanup=benchmark_module.CleanupReport(False), + failed_phase="gamma", + error="RuntimeError: gamma failed", + ) + report_path = tmp_path / "partial.json" + markdown_path = tmp_path / "partial.md" + benchmark_module.write_json_atomic(report_path, terminal) + benchmark_module.validate_report(terminal) + assert "gamma" in benchmark_module.render_markdown_summary( + report_path, markdown_path + ) + assert markdown_path.exists() + + +def test_worker_group_checkpoints_keep_only_one_active_interleaved_row( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Cancellation at any strategy checkpoint must leave a valid terminal view.""" + topology = benchmark_module.Topology(1, 1, 1) + + class ValidatingRecorder: + def __init__(self) -> None: + self.report = benchmark_module.RunReport(topology) + + def checkpoint(self, _name: str) -> None: + terminal = dataclasses.replace( + self.report, + status="cutoff", + cleanup=benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + failed_phase="cancellation", + error="CancelledError: checkpoint cancellation", + ) + benchmark_module.validate_report(terminal) + + def measured(name: str) -> t.Callable[[], t.Any]: + def run() -> t.Any: + return benchmark_module.SearchResult( + duration_ns=23, + family="snapshot", + kind="sessions", + scanned_count=1, + target=name, + matched_ids=(name,), + verified=True, + ) + + return run + + async def live_postcondition(*_args: t.Any, **_kwargs: t.Any) -> bool: + return True + + monkeypatch.setattr( + benchmark_module, "_worker_live_postcondition", live_postcondition + ) + monkeypatch.setattr( + benchmark_module, + "probe_host", + lambda _reader: benchmark_module.HostSnapshot(), + ) + recorder = ValidatingRecorder() + + asyncio.run( + benchmark_module._run_worker_group( + recorder, + types.SimpleNamespace(topology=topology), + { + "alpha": measured("alpha"), + "beta": measured("beta"), + "gamma": measured("gamma"), + }, + warmup=1, + runs=1, + seed=11, + policy=benchmark_module.ResourcePolicy(), + latest={}, + fail_after=None, + active_phase=["setup"], + ) + ) + + assert tuple(phase.name for phase in recorder.report.phases) == ( + "alpha", + "beta", + "gamma", + ) + assert all(phase.status == "completed" for phase in recorder.report.phases) + + def test_cli_run_executes_every_phase_and_writes_validated_artifacts( tmp_path: pathlib.Path, ) -> None: @@ -4856,6 +5176,72 @@ def test_validator_accepts_partial_active_failure_after_complete_prefix( ) +def test_validator_accepts_cancellation_between_interleaved_strategies( + benchmark_module: types.ModuleType, +) -> None: + """A completed interleaved subset remains valid before the next boundary.""" + base = _failed_report_with_completed_mutation_prefix(benchmark_module) + topology = base.requested_topology + + def completed_repeatable(name: str, *, row_count: int | None = None) -> t.Any: + samples = tuple( + benchmark_module.RawSample( + 10 + ordinal, + True, + verified=True, + strategy=name, + ordinal=ordinal, + ) + for ordinal in range(2) + ) + observations = tuple( + benchmark_module.PhaseObservation( + ordinal, + name, + 10 + ordinal, + row_count=row_count, + ) + for ordinal in range(2) + ) + return benchmark_module.PhaseReport( + name, + topology, + topology, + samples=samples, + summary=benchmark_module.summarize_ns((10, 11)), + status="completed", + warmup=2, + runs=2, + warmup_observations=tuple( + benchmark_module.PhaseObservation(ordinal, name, 2 + ordinal) + for ordinal in range(2) + ), + observations=observations, + ) + + report = dataclasses.replace( + base, + status="cutoff", + phases=( + *base.phases[:3], + completed_repeatable("wait.capture-poll"), + benchmark_module.PhaseReport( + "wait.control-stream", + topology, + topology, + status="not_applicable", + warmup=2, + runs=2, + ), + completed_repeatable("enumeration.panes", row_count=topology.panes), + ), + failed_phase="cancellation", + error="KeyboardInterrupt: cancelled between strategies", + ) + + benchmark_module.validate_report(report) + + def test_validator_rejects_failed_phase_that_disagrees_with_active_row( benchmark_module: types.ModuleType, ) -> None: @@ -4967,13 +5353,25 @@ def test_cli_mid_strategy_failure_has_one_active_prefix_and_no_future_rows( assert completed.returncode != 0 report = benchmark_module.load_run_report(report_path) - phase_index = _RUNNER_PHASES.index(failed_phase) + group_index, group = next( + (index, group) + for index, group in enumerate(benchmark_module._RUNNER_PHASE_GROUPS) + if failed_phase in group + ) + invocation_order = list(group) + benchmark_module.random.Random(11 + group_index - 2).shuffle(invocation_order) + failed_index = invocation_order.index(failed_phase) + expected_phases = ( + *( + name + for prior_group in benchmark_module._RUNNER_PHASE_GROUPS[:group_index] + for name in prior_group + ), + *invocation_order[: failed_index + 1], + ) assert report.status == "failed" assert report.failed_phase == failed_phase - assert ( - tuple(phase.name for phase in report.phases) - == _RUNNER_PHASES[: phase_index + 1] - ) + assert tuple(phase.name for phase in report.phases) == expected_phases assert all( phase.status in {"completed", "not_applicable"} for phase in report.phases[:-1] ) @@ -5405,6 +5803,176 @@ def interrupt_twice() -> None: ) +def test_finalizer_restores_mask_before_original_sigterm_handler( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """SIGTERM during handler restoration must be recorded, not delivered to DFL.""" + original_handler = signal.getsignal(signal.SIGTERM) + original_mask = frozenset(signal.pthread_sigmask(signal.SIG_BLOCK, ())) + interruption = KeyboardInterrupt("SIGTERM during handler restoration") + real_signal = signal.signal + raised_during_restore = False + restore_mask: frozenset[int | signal.Signals] | None = None + + def prior_sigterm( + _signal_number: int, + _frame: types.FrameType | None, + ) -> t.NoReturn: + raise interruption + + real_signal(signal.SIGTERM, prior_sigterm) + + def signal_with_pending_delivery( + signal_number: int, + handler: t.Any, + ) -> t.Any: + nonlocal raised_during_restore, restore_mask + if ( + signal_number == signal.SIGTERM + and handler is prior_sigterm + and not raised_during_restore + ): + raised_during_restore = True + restore_mask = frozenset(signal.pthread_sigmask(signal.SIG_BLOCK, ())) + signal.raise_signal(signal.SIGTERM) + return real_signal(signal_number, handler) + + monkeypatch.setattr(benchmark_module.signal, "signal", signal_with_pending_delivery) + try: + result, observed = benchmark_module._drain_finalizer_thread(lambda: 29) + restored_handler = signal.getsignal(signal.SIGTERM) + restored_mask = frozenset(signal.pthread_sigmask(signal.SIG_BLOCK, ())) + finally: + real_signal(signal.SIGTERM, original_handler) + signal.pthread_sigmask(signal.SIG_SETMASK, original_mask) + + assert result == 29 + assert observed is interruption + assert raised_during_restore + assert restore_mask == original_mask + assert restored_handler is prior_sigterm + assert restored_mask == original_mask + + +@pytest.mark.parametrize("boundary", ("state", "is_alive", "join")) +def test_finalizer_drains_injected_interrupt_at_every_post_launch_boundary( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, + boundary: str, +) -> None: + """Post-launch inspection cannot escape before one durable callback ends.""" + real_thread = threading.Thread + real_event = threading.Event + interruption = KeyboardInterrupt(f"{boundary} interruption") + callback_count = 0 + wrappers: list[t.Any] = [] + output = tmp_path / f"{boundary}.json" + expected_handlers = { + signal_number: signal.getsignal(signal_number) + for signal_number in (signal.SIGINT, signal.SIGTERM) + } + expected_mask = frozenset(signal.pthread_sigmask(signal.SIG_BLOCK, ())) + + class InterruptingEvent: + def __init__(self) -> None: + self.delegate = real_event() + self.interrupted = False + + def set(self) -> None: + self.delegate.set() + + def wait(self, timeout: float | None = None) -> bool: + return self.delegate.wait(timeout) + + def is_set(self) -> bool: + if boundary == "state" and not self.interrupted: + self.interrupted = True + raise interruption + return self.delegate.is_set() + + class InterruptingThread: + def __init__(self, *, target: t.Callable[[], None], name: str) -> None: + self.delegate = real_thread(target=target, name=name) + self.interrupted_is_alive = False + self.interrupted_join = False + self.joined = False + wrappers.append(self) + + @property + def ident(self) -> int | None: + return self.delegate.ident + + def start(self) -> None: + self.delegate.start() + + def is_alive(self) -> bool: + if boundary == "is_alive" and not self.interrupted_is_alive: + self.interrupted_is_alive = True + raise interruption + if not self.joined: + return True + return self.delegate.is_alive() + + def join(self, timeout: float | None = None) -> None: + if boundary == "join" and not self.interrupted_join: + self.interrupted_join = True + raise interruption + self.delegate.join(timeout) + if not self.delegate.is_alive(): + self.joined = True + + def callback() -> t.Any: + nonlocal callback_count + callback_count += 1 + report = benchmark_module.RunReport(benchmark_module.Topology(1, 1, 1)) + benchmark_module.write_json_atomic(output, report) + return report + + monkeypatch.setattr( + benchmark_module, + "threading", + types.SimpleNamespace( + Thread=InterruptingThread, + Event=InterruptingEvent if boundary == "state" else real_event, + ), + ) + result: t.Any = None + observed: KeyboardInterrupt | None = None + escaped: KeyboardInterrupt | None = None + try: + try: + result, observed = benchmark_module._drain_finalizer_thread(callback) + except KeyboardInterrupt as error: + escaped = error + observed_mask = frozenset(signal.pthread_sigmask(signal.SIG_BLOCK, ())) + observed_handlers = { + signal_number: signal.getsignal(signal_number) + for signal_number in (signal.SIGINT, signal.SIGTERM) + } + finally: + for signal_number, handler in expected_handlers.items(): + signal.signal(signal_number, handler) + signal.pthread_sigmask(signal.SIG_SETMASK, expected_mask) + for wrapper in wrappers: + if wrapper.delegate.ident is not None: + wrapper.delegate.join(timeout=5.0) + + assert escaped is None + assert observed is interruption + assert result == benchmark_module.load_run_report(output) + benchmark_module.validate_report(result) + assert callback_count == 1 + assert observed_mask == expected_mask + assert observed_handlers == expected_handlers + assert all(not wrapper.delegate.is_alive() for wrapper in wrappers) + assert not any( + thread.name == "orchestration-finalization" and thread.is_alive() + for thread in threading.enumerate() + ) + + def test_supervisor_cancellation_during_final_report_write_is_durable( benchmark_module: types.ModuleType, monkeypatch: pytest.MonkeyPatch, @@ -5416,6 +5984,13 @@ def test_supervisor_cancellation_during_final_report_write_is_durable( entered_write = threading.Event() release_write = threading.Event() real_write = benchmark_module.write_json_atomic + real_drain = benchmark_module._drain_finalizer_thread + drain_calls = 0 + + def counting_drain(*args: t.Any, **kwargs: t.Any) -> t.Any: + nonlocal drain_calls + drain_calls += 1 + return real_drain(*args, **kwargs) def delayed_terminal_write(path: pathlib.Path, value: t.Any) -> None: if ( @@ -5432,6 +6007,11 @@ def delayed_terminal_write(path: pathlib.Path, value: t.Any) -> None: "write_json_atomic", delayed_terminal_write, ) + monkeypatch.setattr( + benchmark_module, + "_drain_finalizer_thread", + counting_drain, + ) def interrupt_final_write() -> None: assert entered_write.wait(timeout=30.0) @@ -5475,6 +6055,7 @@ def interrupt_final_write() -> None: interrupter.join(timeout=5.0) assert not interrupter.is_alive() + assert drain_calls == 1 durable = benchmark_module.load_run_report(report_path) assert isinstance(cancellation.value, KeyboardInterrupt) assert durable.status == "cutoff" @@ -6318,6 +6899,107 @@ def cancel_child(shape: t.Any, **kwargs: t.Any) -> t.NoReturn: assert report.cleanup.complete +@pytest.mark.parametrize("child_status", ("refused", "failed", "cutoff")) +def test_ramp_late_interrupt_preserves_existing_child_terminal_state( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, + child_status: str, +) -> None: + """A late interrupt cannot replace an established child terminal reason.""" + real_thread = threading.Thread + real_finalize = benchmark_module._finalize_ramp_aggregation + interruption = KeyboardInterrupt("late ramp interruption") + output = tmp_path / f"late-{child_status}.json" + markdown = output.with_suffix(".md") + thread_count = 0 + finalizer_calls = 0 + + class InterruptingThread: + def __init__(self, *, target: t.Callable[[], None], name: str) -> None: + nonlocal thread_count + thread_count += 1 + self.ordinal = thread_count + self.delegate = real_thread(target=target, name=name) + self.interrupted = False + + @property + def ident(self) -> int | None: + return self.delegate.ident + + def start(self) -> None: + self.delegate.start() + + def join(self, timeout: float | None = None) -> None: + self.delegate.join(timeout) + + def is_alive(self) -> bool: + if self.ordinal == 2 and not self.interrupted: + self.interrupted = True + raise interruption + return self.delegate.is_alive() + + def terminal_child(shape: t.Any, **kwargs: t.Any) -> t.Any: + reason = f"existing {child_status} reason" + scratch_path = None if child_status == "refused" else f"{child_status}-scratch" + child = benchmark_module.RunReport( + shape, + status=child_status, + cleanup=benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + error=reason, + run_id=f"child-{child_status}", + scratch_path=scratch_path, + socket_path=(None if scratch_path is None else f"{scratch_path}/tmux.sock"), + ) + benchmark_module.write_json_atomic(kwargs["output"], child) + return child + + def count_finalize(*args: t.Any, **kwargs: t.Any) -> t.Any: + nonlocal finalizer_calls + finalizer_calls += 1 + return real_finalize(*args, **kwargs) + + monkeypatch.setattr( + benchmark_module, + "threading", + types.SimpleNamespace(Thread=InterruptingThread, Event=threading.Event), + ) + monkeypatch.setattr(benchmark_module, "run_scenario", terminal_child) + monkeypatch.setattr(benchmark_module, "_finalize_ramp_aggregation", count_finalize) + + with pytest.raises(KeyboardInterrupt) as raised: + benchmark_module.run_ramp( + ( + benchmark_module.Topology(1, 1, 1), + benchmark_module.Topology(2, 1, 1), + ), + runs=1, + warmup=0, + output=output, + markdown_output=markdown, + ) + + assert raised.value is interruption + report = benchmark_module.load_run_report(output) + assert report.status == child_status + assert tuple(step.status for step in report.ramp) == ( + child_status, + "not_attempted", + ) + assert report.error == f"existing {child_status} reason" + assert report.ramp[0].reason == report.error + assert report.ramp[1].reason == report.error + assert finalizer_calls == 1 + benchmark_module.validate_report(report) + assert child_status in benchmark_module.render_markdown_summary(output) + assert markdown.exists() + + def test_cli_ramp_predictive_refusal_never_executes_tmux_binary( tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch, From a84ea32f3d0ca004fe4424eb1651f6bc538620d6 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 17:11:54 -0500 Subject: [PATCH 24/67] Bench(fix[runner]): Bind socket ownership why: Cleanup could contact a replacement tmux server, signal restoration could leak caller state, and repeated boundaries lacked durable attribution. what: - Bind process identity to durable socket inode ownership - Restore exact caller signals inside protected finalization - Persist revisited boundaries and reject unreachable report snapshots --- scripts/bench_orchestration.py | 1058 +++++++++++++++----- tests/test_bench_orchestration_script.py | 1165 ++++++++++++++++++++-- 2 files changed, 1924 insertions(+), 299 deletions(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index 7b9a8273c7..fbad6339e7 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -1769,6 +1769,57 @@ class ProcessIdentity: start_time: int +@dataclasses.dataclass(frozen=True) +class SocketIdentity: + """Immutable filesystem identity of one owned Unix socket. + + Attributes + ---------- + st_dev : int + Device containing the socket node. + st_ino : int + Inode of the socket node. + st_uid : int + User that owns the socket node. + st_mode : int + Complete mode captured by ``lstat``; its file type must remain a socket. + + Examples + -------- + >>> identity = SocketIdentity(1, 2, os.getuid(), stat.S_IFSOCK | 0o600) + >>> stat.S_ISSOCK(identity.st_mode) + True + """ + + st_dev: int + st_ino: int + st_uid: int + st_mode: int + + +@dataclasses.dataclass(frozen=True) +class _SocketOwnership: + """Bind one tmux process identity to its original socket inode. + + Attributes + ---------- + process : ProcessIdentity + Exact tmux server PID and procfs start time. + socket : SocketIdentity + Socket fingerprint observed around the owner query. + + Examples + -------- + >>> owner = ProcessIdentity("server", 2, 3) + >>> socket_id = SocketIdentity(1, 2, os.getuid(), stat.S_IFSOCK | 0o600) + >>> _SocketOwnership(owner, socket_id).process is owner + True + """ + + process: ProcessIdentity + socket: SocketIdentity + + @dataclasses.dataclass(frozen=True) class ProgressEvent: """One append-only worker checkpoint observed by the supervisor. @@ -1785,6 +1836,8 @@ class ProgressEvent: Worker monotonic publication time. processes : tuple[ProcessIdentity, ...] Exact identities first learned at this checkpoint; never cumulative. + socket_ownership : _SocketOwnership | None + Exact server/socket capability first learned at this checkpoint. schema_version : int Progress stream schema version. @@ -1799,6 +1852,7 @@ class ProgressEvent: checkpoint: str monotonic_ns: int processes: tuple[ProcessIdentity, ...] = () + socket_ownership: _SocketOwnership | None = None schema_version: int = 1 @@ -2003,6 +2057,8 @@ class RunContext: Timed construction duration excluding activity stabilization. processes : tuple[ProcessIdentity, ...] Fuzzer, server, and pane-follower identities. + socket_ownership : _SocketOwnership | None + Immutable server/socket capability captured during bootstrap. session_ids : tuple[str, ...] Verified stable session identifiers. window_ids : tuple[str, ...] @@ -2061,6 +2117,7 @@ class RunContext: expected_window_parents: tuple[tuple[str, str], ...] setup_duration_ns: int processes: tuple[ProcessIdentity, ...] + socket_ownership: _SocketOwnership | None = None session_ids: tuple[str, ...] = () window_ids: tuple[str, ...] = () pane_ids: tuple[str, ...] = () @@ -2158,6 +2215,8 @@ class RunReport: Concise terminal reason retained by the supervisor. processes : tuple[ProcessIdentity, ...] Exact worker, fuzzer, server, and pane identities observed by progress. + socket_ownership : _SocketOwnership | None + Durable server/socket capability observed in worker progress. scratch_path : str | None Exact private run path whose absence cleanup verified. socket_path : str | None @@ -2192,6 +2251,7 @@ class RunReport: failed_phase: str | None = None error: str | None = None processes: tuple[ProcessIdentity, ...] = () + socket_ownership: _SocketOwnership | None = None scratch_path: str | None = None socket_path: str | None = None progress_path: str | None = None @@ -2537,6 +2597,80 @@ def _identity_from_json(value: object) -> ProcessIdentity: ) +def _socket_identity_from_json(value: object) -> SocketIdentity: + """Decode one exact socket fingerprint. + + >>> _socket_identity_from_json({ + ... "st_dev": 1, "st_ino": 2, "st_uid": 3, + ... "st_mode": stat.S_IFSOCK | 0o600, + ... }).st_ino + 2 + """ + row = _json_mapping(value, "socket identity") + return SocketIdentity( + st_dev=t.cast(int, row.get("st_dev")), + st_ino=t.cast(int, row.get("st_ino")), + st_uid=t.cast(int, row.get("st_uid")), + st_mode=t.cast(int, row.get("st_mode")), + ) + + +def _socket_ownership_from_json(value: object | None) -> _SocketOwnership | None: + """Decode an optional process/socket ownership capability. + + >>> _socket_ownership_from_json(None) is None + True + """ + if value is None: + return None + row = _json_mapping(value, "socket ownership") + ownership = _SocketOwnership( + process=_identity_from_json(row.get("process")), + socket=_socket_identity_from_json(row.get("socket")), + ) + _validate_socket_ownership(ownership) + return ownership + + +def _validate_socket_ownership(ownership: _SocketOwnership) -> None: + """Reject a capability that cannot identify one same-user tmux socket. + + Parameters + ---------- + ownership : _SocketOwnership + Process and filesystem identity decoded from durable evidence. + + Returns + ------- + None + After every identity component has its exact JSON-domain type and range. + + Raises + ------ + ValueError + If the process or socket fingerprint is malformed. + """ + process = ownership.process + socket_identity = ownership.socket + if ( + process.role != "server" + or type(process.pid) is not int + or process.pid <= 0 + or type(process.start_time) is not int + or process.start_time <= 0 + or type(socket_identity.st_dev) is not int + or socket_identity.st_dev < 0 + or type(socket_identity.st_ino) is not int + or socket_identity.st_ino <= 0 + or type(socket_identity.st_uid) is not int + or socket_identity.st_uid != os.getuid() + or type(socket_identity.st_mode) is not int + or not stat.S_ISSOCK(socket_identity.st_mode) + ): + message = "invalid socket ownership capability" + raise ValueError(message) + + def _environment_from_json(value: object | None) -> EnvironmentReport | None: """Decode optional descriptive environment evidence. @@ -2628,6 +2762,7 @@ def run_report_from_json(value: object) -> RunReport: failed_phase=row.get("failed_phase"), error=row.get("error"), processes=tuple(_identity_from_json(item) for item in row.get("processes", [])), + socket_ownership=_socket_ownership_from_json(row.get("socket_ownership")), scratch_path=row.get("scratch_path"), socket_path=row.get("socket_path"), progress_path=row.get("progress_path"), @@ -2934,6 +3069,128 @@ def validate_report(report: RunReport) -> None: _validate_executable_ramp(report) +def _reachable_group_signatures( + strategy_names: tuple[str, ...], + *, + warmup: int, + runs: int, + seed: int, +) -> frozenset[tuple[tuple[str, str, int, int], ...]]: + """Return report snapshots reachable at durable group boundaries. + + >>> signatures = _reachable_group_signatures( + ... ("a", "b"), warmup=1, runs=1, seed=1 + ... ) + >>> (("b", "in_progress", 0, 0),) in signatures + True + + Parameters + ---------- + strategy_names : tuple[str, ...] + Applicable strategies in production mapping order. + warmup : int + Untimed calls per strategy. + runs : int + Timed calls per strategy. + seed : int + Exact group schedule seed. + + Returns + ------- + frozenset[tuple[tuple[str, str, int, int], ...]] + Phase name, status, warmup count, and timed count for every snapshot + reachable before or after one scheduled invocation. Failed variants + represent an exception attributed to the current strategy. + """ + states: dict[str, tuple[str, int, int]] = {} + reachable: set[tuple[tuple[str, str, int, int], ...]] = set() + + def snapshot(strategy: str) -> tuple[tuple[str, str, int, int], ...]: + completed = tuple( + (name, status, warmup_count, timed_count) + for name, (status, warmup_count, timed_count) in states.items() + if status == "completed" + ) + status, warmup_count, timed_count = states[strategy] + active = ( + () + if status == "completed" + else ((strategy, status, warmup_count, timed_count),) + ) + if all(state[0] == "completed" for state in states.values()) and len( + states + ) == len(strategy_names): + return tuple((name, *states[name]) for name in strategy_names) + return (*completed, *active) + + def retain( + current: str, + observed: tuple[tuple[str, str, int, int], ...], + ) -> None: + reachable.add(observed) + completed = tuple( + (name, status, warmup_count, timed_count) + for name, (status, warmup_count, timed_count) in states.items() + if name != current and status == "completed" + ) + _status, warmup_count, timed_count = states[current] + reachable.add((*completed, (current, "failed", warmup_count, timed_count))) + + for stage, strategy, ordinal in _repeatable_schedule( + strategy_names, + warmup=warmup, + runs=runs, + seed=seed, + ): + states.setdefault(strategy, ("in_progress", 0, 0)) + retain(strategy, snapshot(strategy)) + status, warmup_count, timed_count = states[strategy] + if stage == "warmup": + warmup_count = ordinal + 1 + else: + timed_count = ordinal + 1 + if ordinal == runs - 1: + status = "completed" + states[strategy] = (status, warmup_count, timed_count) + retain(strategy, snapshot(strategy)) + return frozenset(reachable) + + +def _interleaved_suffix_is_reachable( + report: RunReport, + group_index: int, + suffix: tuple[PhaseReport, ...], +) -> bool: + """Return whether an executable suffix occurs in its exact seeded schedule.""" + assert report.environment is not None + assert report.warmup is not None + assert report.runs is not None + group = _RUNNER_PHASE_GROUPS[group_index] + strategy_names: tuple[str, ...] + if group == _RUNNER_PHASE_GROUPS[3] and not ( + report.lane == EngineLane.CONTROL.value + and report.mode == ExecutionMode.ASYNC.value + ): + strategy_names = ("wait.capture-poll",) + else: + strategy_names = group + observed = tuple( + ( + phase.name, + phase.status, + len(phase.warmup_observations), + len(phase.samples), + ) + for phase in suffix + ) + return observed in _reachable_group_signatures( + strategy_names, + warmup=report.warmup, + runs=report.runs, + seed=report.environment.seed + group_index - 2, + ) + + def _validate_executable_report(report: RunReport) -> None: """Validate the stronger contract for a supervisor-owned scenario. @@ -3002,6 +3259,16 @@ def _validate_executable_report(report: RunReport) -> None: ): message = "executable report has an invalid process identity" raise ValueError(message) + ownership = report.socket_ownership + if ownership is not None: + try: + _validate_socket_ownership(ownership) + except ValueError as error: + message = "executable report has an invalid socket ownership capability" + raise ValueError(message) from error + if ownership.process not in report.processes: + message = "executable report has an invalid socket ownership capability" + raise ValueError(message) if report.status == "refused": if ( report.failed_phase != "preflight" @@ -3009,6 +3276,7 @@ def _validate_executable_report(report: RunReport) -> None: or report.observed_topology is not None or report.phases or report.processes + or report.socket_ownership is not None or report.scratch_path is not None or report.socket_path is not None or report.progress_path is not None @@ -3022,44 +3290,63 @@ def _validate_executable_report(report: RunReport) -> None: if report.environment is None: message = "attempted executable report requires environment evidence" raise ValueError(message) + if type(report.environment.seed) is not int: + message = "attempted executable report requires an integer schedule seed" + raise ValueError(message) if not report.scratch_path or not report.socket_path or not report.progress_path: message = "attempted executable report requires owned resource paths" raise ValueError(message) + server_identities = tuple( + identity for identity in report.processes if identity.role == "server" + ) + if ( + report.status == "completed" or (report.cleanup.complete and server_identities) + ) and ( + ownership is None + or len(server_identities) != 1 + or ownership.process != server_identities[0] + ): + message = "attempted executable report lacks exact socket ownership" + raise ValueError(message) terminal_unsuccessful = report.status in {"failed", "cutoff"} phase_names = tuple(phase.name for phase in report.phases) canonical_prefix = phase_names == _RUNNER_PHASES[: len(phase_names)] - if not canonical_prefix: - active_names = tuple( - phase.name - for phase in report.phases - if phase.status in {"failed", "in_progress"} - ) - active_name = active_names[0] if len(active_names) == 1 else None - interleaved_terminal_prefix = False - if terminal_unsuccessful and len(active_names) <= 1: - for group_index, group in enumerate(_RUNNER_PHASE_GROUPS): - prior_names = tuple( - name - for prior_group in _RUNNER_PHASE_GROUPS[:group_index] - for name in prior_group - ) - active_suffix = phase_names[len(prior_names) :] - if ( - phase_names[: len(prior_names)] != prior_names - or not active_suffix - or len(set(active_suffix)) != len(active_suffix) - or not all(name in group for name in active_suffix) - ): - continue - if active_name is not None and active_suffix[-1] != active_name: - continue - if active_name is None and report.failed_phase != "cancellation": - continue - interleaved_terminal_prefix = True + interleaved_candidate = False + interleaved_reachable = False + if terminal_unsuccessful: + for group_index, group in enumerate(_RUNNER_PHASE_GROUPS): + if len(group) <= 1: + continue + prior_names = tuple( + name + for prior_group in _RUNNER_PHASE_GROUPS[:group_index] + for name in prior_group + ) + active_suffix = report.phases[len(prior_names) :] + active_names = tuple(phase.name for phase in active_suffix) + if ( + phase_names[: len(prior_names)] != prior_names + or not active_names + or len(set(active_names)) != len(active_names) + or not all(name in group for name in active_names) + ): + continue + interleaved_candidate = True + if _interleaved_suffix_is_reachable( + report, + group_index, + active_suffix, + ): + interleaved_reachable = True break - if not interleaved_terminal_prefix: - message = "attempted executable report phases must be a runner phase prefix" - raise ValueError(message) + if interleaved_candidate and not interleaved_reachable: + message = ( + "attempted executable report lacks a reachable seeded strategy boundary" + ) + raise ValueError(message) + if not canonical_prefix and not interleaved_reachable: + message = "attempted executable report phases must be a runner phase prefix" + raise ValueError(message) if terminal_unsuccessful and (not report.failed_phase or not report.error): message = "terminal unsuccessful report requires phase and error" raise ValueError(message) @@ -3932,30 +4219,171 @@ def _stop_owned_process( ) +def _socket_identity(socket_path: pathlib.Path) -> SocketIdentity: + """Return the exact same-user Unix-socket identity from ``lstat``. + + Parameters + ---------- + socket_path : pathlib.Path + Socket path inside the run's private directory. + + Returns + ------- + SocketIdentity + Device, inode, owner, and mode from one non-following stat. + + Raises + ------ + OSError + If the path is absent, replaced by another file type, or not same-user. + """ + status = socket_path.lstat() + if not stat.S_ISSOCK(status.st_mode) or status.st_uid != os.getuid(): + message = f"configured socket is not an owned Unix socket: {socket_path}" + raise OSError(message) + return SocketIdentity( + st_dev=status.st_dev, + st_ino=status.st_ino, + st_uid=status.st_uid, + st_mode=status.st_mode, + ) + + +def _query_exact_socket_owner( + socket_path: pathlib.Path, + *, + timeout_s: float, +) -> ProcessIdentity: + """Boundedly query one exact tmux socket for its current server identity. + + Parameters + ---------- + socket_path : pathlib.Path + Exact socket path to query without inherited tmux coordinates. + timeout_s : float + Maximum query duration. + + Returns + ------- + ProcessIdentity + Current tmux PID with a verified procfs start time. + + Raises + ------ + RuntimeError + If the query times out, fails, or does not identify a live process. + """ + environment = os.environ.copy() + environment.pop("TMUX", None) + environment.pop("TMUX_PANE", None) + try: + result = subprocess.run( + ( + "tmux", + "-S", + str(socket_path), + "display-message", + "-p", + "#{pid}", + ), + env=environment, + check=False, + capture_output=True, + text=True, + timeout=timeout_s, + ) + except subprocess.TimeoutExpired as error: + message = "exact socket server identity query timed out" + raise RuntimeError(message) from error + except OSError as error: + message = f"exact socket server identity query: {type(error).__name__}: {error}" + raise RuntimeError(message) from error + if result.returncode != 0: + detail = result.stderr.strip() + suffix = f": {detail}" if detail else "" + message = ( + f"exact socket server identity query exited {result.returncode}{suffix}" + ) + raise RuntimeError(message) + try: + identity = _record_process("server", int(result.stdout.strip())) + except (RuntimeError, ValueError) as error: + message = f"exact socket server identity query: {type(error).__name__}: {error}" + raise RuntimeError(message) from error + if not process_identity_matches(identity): + message = "exact socket server identity changed after query" + raise RuntimeError(message) + return identity + + +def _capture_socket_ownership( + socket_path: pathlib.Path, + *, + timeout_s: float, +) -> _SocketOwnership: + """Capture one immutable process/socket capability around an owner query. + + The socket lives in a mode-0700 run directory. Two ``lstat`` calls bind the + read-only tmux owner query to one inode before the capability is published. + + Parameters + ---------- + socket_path : pathlib.Path + Newly established exact tmux socket. + timeout_s : float + Maximum owner-query duration. + + Returns + ------- + _SocketOwnership + Process and socket identities that matched throughout capture. + + Raises + ------ + RuntimeError + If the path or process changes during capture. + OSError + If private-directory or socket ownership cannot be verified. + """ + _verify_private_directory_mode(socket_path.parent) + before = _socket_identity(socket_path) + process = _query_exact_socket_owner(socket_path, timeout_s=timeout_s) + after = _socket_identity(socket_path) + if before != after or not process_identity_matches(process): + message = "exact socket ownership changed during capture" + raise RuntimeError(message) + return _SocketOwnership(process=process, socket=before) + + def _kill_exact_tmux_socket( socket_path: pathlib.Path, *, timeout_s: float, - server_identity: ProcessIdentity | None = None, + socket_ownership: _SocketOwnership | None = None, ) -> tuple[str, ...]: - """Boundedly request ``kill-server`` on one exact isolated socket. + """Boundedly clean only the server/socket pair captured at establishment. >>> _kill_exact_tmux_socket(pathlib.Path("missing.sock"), timeout_s=0.01) () + An observed pathname mismatch fails closed. The original process is killed + only through its retained pidfd, while the replacement inode is preserved. + A same-user pathname race cannot be made atomic with ``unlink``; mode-0700 + parent ownership and repeated identity checks bound the supported contract. + Parameters ---------- socket_path : pathlib.Path Exact isolated socket owned by one benchmark run. timeout_s : float Maximum wait for the helper and each stable-handle escalation. - server_identity : ProcessIdentity | None - Already recorded exact server identity, queried from the socket when absent. + socket_ownership : _SocketOwnership | None + Capability captured while the owned tmux server established the socket. Returns ------- tuple[str, ...] - Empty only when the exact server and socket are absent afterward. + Empty only when the captured server and socket are absent afterward. Raises ------ @@ -3965,55 +4393,74 @@ def _kill_exact_tmux_socket( if timeout_s <= 0: message = "exact socket cleanup timeout must be positive" raise ValueError(message) - if not socket_path.exists(): - return () - environment = os.environ.copy() - environment.pop("TMUX", None) - environment.pop("TMUX_PANE", None) + if socket_ownership is None: + return ( + () + if not socket_path.exists() + else (f"exact socket ownership capability unavailable: {socket_path}",) + ) + owner = socket_ownership.process errors: list[str] = [] - identity_was_supplied = server_identity is not None - if server_identity is None: - try: - server_pid_result = subprocess.run( - ( - "tmux", - "-S", - str(socket_path), - "display-message", - "-p", - "#{pid}", - ), - env=environment, - check=False, - capture_output=True, - text=True, - timeout=timeout_s, + registry = _PidfdRegistry() + original_live = process_identity_matches(owner) + retained = registry.retain(owner) + + def finish() -> tuple[str, ...]: + errors.extend(registry.errors) + registry.close() + return tuple(errors) + + def kill_original_without_path() -> None: + if not process_identity_matches(owner): + return + if not retained: + errors.append( + f"server pid {owner.pid} stable handle unavailable; " + "pathname cleanup refused" ) - except subprocess.TimeoutExpired: - return ("exact socket server identity query timed out",) - except OSError as error: - return (f"exact socket server identity: {type(error).__name__}: {error}",) - if server_pid_result.returncode != 0: - detail = server_pid_result.stderr.strip() - suffix = f": {detail}" if detail else "" - exit_code = server_pid_result.returncode - return (f"exact socket server identity query exited {exit_code}{suffix}",) - try: - server_identity = _record_process( - "server", - int(server_pid_result.stdout.strip()), + return + registry.signal(owner, signal.SIGKILL) + survivors = _wait_identity_absence((owner,), timeout_s=timeout_s) + if survivors: + errors.append( + f"server pid {owner.pid} with start time {owner.start_time} remains" ) - except (RuntimeError, ValueError) as error: - return (f"exact socket server identity: {type(error).__name__}: {error}",) - if identity_was_supplied and not _wait_identity_absence( - (server_identity,), - timeout_s=timeout_s, - ): - return _remove_proven_stale_socket(socket_path, (server_identity,)) + if original_live and not retained: + errors.append( + f"server pid {owner.pid} stable handle unavailable; " + "pathname cleanup refused" + ) + return finish() + if not socket_path.exists(): + kill_original_without_path() + return finish() + try: + _verify_private_directory_mode(socket_path.parent) + before = _socket_identity(socket_path) + except OSError as error: + errors.append(f"exact socket ownership: {type(error).__name__}: {error}") + kill_original_without_path() + return finish() + if before != socket_ownership.socket: + errors.append(f"configured socket ownership changed: {socket_path}") + kill_original_without_path() + return finish() + try: + current_owner = _query_exact_socket_owner(socket_path, timeout_s=timeout_s) + after = _socket_identity(socket_path) + except (OSError, RuntimeError) as error: + errors.append(f"exact socket ownership query: {type(error).__name__}: {error}") + kill_original_without_path() + return finish() + if after != socket_ownership.socket or current_owner != owner: + errors.append(f"configured socket owner changed: {socket_path}") + kill_original_without_path() + return finish() - server_registry = _PidfdRegistry() - server_registry.retain(server_identity) + environment = os.environ.copy() + environment.pop("TMUX", None) + environment.pop("TMUX_PANE", None) helper: subprocess.Popen[bytes] | None = None helper_timed_out = False try: @@ -4036,83 +4483,94 @@ def _kill_exact_tmux_socket( helper_timed_out = True errors.append("exact socket kill-server identity unavailable") else: - registry = _PidfdRegistry() + helper_registry = _PidfdRegistry() try: - registry.retain(helper_identity) + helper_retained = helper_registry.retain(helper_identity) try: helper.wait(timeout=timeout_s) except subprocess.TimeoutExpired: helper_timed_out = True - registry.signal(helper_identity, signal.SIGTERM) - try: - helper.wait(timeout=timeout_s) - except subprocess.TimeoutExpired: - registry.signal(helper_identity, signal.SIGKILL) + if helper_retained: + helper_registry.signal(helper_identity, signal.SIGTERM) try: helper.wait(timeout=timeout_s) except subprocess.TimeoutExpired: - errors.append("exact socket kill-server helper remains") - errors.extend(registry.errors) + helper_registry.signal(helper_identity, signal.SIGKILL) + try: + helper.wait(timeout=timeout_s) + except subprocess.TimeoutExpired: + errors.append("exact socket kill-server helper remains") + else: + errors.append( + "exact socket kill-server helper stable handle unavailable" + ) + errors.extend(helper_registry.errors) finally: - registry.close() + helper_registry.close() if helper_timed_out: errors.append("exact socket kill-server timed out") elif helper.returncode != 0: errors.append(f"exact socket kill-server exited {helper.returncode}") try: - survivors = _wait_identity_absence((server_identity,), timeout_s=timeout_s) + survivors = _wait_identity_absence((owner,), timeout_s=timeout_s) for signal_number in (signal.SIGTERM, signal.SIGKILL): if not survivors: break - server_registry.signal(server_identity, signal_number) - survivors = _wait_identity_absence( - (server_identity,), - timeout_s=timeout_s, - ) + registry.signal(owner, signal_number) + survivors = _wait_identity_absence((owner,), timeout_s=timeout_s) if survivors: errors.append( - f"server pid {server_identity.pid} with start time " - f"{server_identity.start_time} remains" + f"server pid {owner.pid} with start time {owner.start_time} remains" ) elif socket_path.exists(): - errors.extend(_remove_proven_stale_socket(socket_path, (server_identity,))) - errors.extend(server_registry.errors) + errors.extend(_remove_proven_stale_socket(socket_path, socket_ownership)) + errors.extend(registry.errors) finally: - server_registry.close() + registry.close() return tuple(errors) def _remove_proven_stale_socket( socket_path: pathlib.Path, - identities: t.Iterable[ProcessIdentity], + socket_ownership: _SocketOwnership | None, ) -> tuple[str, ...]: - """Remove a stale socket only after its recorded server is absent. + """Remove only the captured inode after proving its server is absent. - >>> _remove_proven_stale_socket(pathlib.Path("missing.sock"), ()) + >>> _remove_proven_stale_socket(pathlib.Path("missing.sock"), None) () Parameters ---------- socket_path : pathlib.Path - Exact configured socket after a bounded ``kill-server`` attempt. - identities : collections.abc.Iterable[ProcessIdentity] - Recorded ownership evidence including the exact server identity. + Exact configured socket after a bounded server cleanup attempt. + socket_ownership : _SocketOwnership | None + Original process/socket capability. Returns ------- tuple[str, ...] - Empty only when the path is absent or safely removed. + Empty only when the path is absent or the captured stale inode was removed. """ if not socket_path.exists(): return () - servers = tuple(identity for identity in identities if identity.role == "server") - if not servers or any(process_identity_matches(identity) for identity in servers): - return (f"socket ownership is not proven absent: {socket_path}",) + if socket_ownership is None: + return (f"socket ownership capability unavailable: {socket_path}",) + try: + _verify_private_directory_mode(socket_path.parent) + before = _socket_identity(socket_path) + except OSError as error: + return (f"stale socket identity: {type(error).__name__}: {error}",) + if before != socket_ownership.socket: + return (f"configured socket ownership changed: {socket_path}",) + if process_identity_matches(socket_ownership.process): + return (f"socket owner is not proven absent: {socket_path}",) try: - status = socket_path.lstat() - if not stat.S_ISSOCK(status.st_mode) or status.st_uid != os.getuid(): - return (f"configured socket was replaced: {socket_path}",) + after = _socket_identity(socket_path) + if after != socket_ownership.socket or process_identity_matches( + socket_ownership.process + ): + return (f"configured socket ownership changed: {socket_path}",) socket_path.unlink() except OSError as error: return (f"stale socket removal: {type(error).__name__}: {error}",) @@ -4790,6 +5248,7 @@ def setup_sync( _process_identity_callback: ( cabc.Callable[[tuple[ProcessIdentity, ...]], None] | None ) = None, + _socket_ownership_callback: cabc.Callable[[_SocketOwnership], None] | None = None, ) -> RunContext: """Build and exactly verify one synchronous live topology. @@ -4824,6 +5283,8 @@ def setup_sync( Unique pane assigned the delayed stream. _process_identity_callback : collections.abc.Callable | None Private worker hook invoked as exact process identities become known. + _socket_ownership_callback : collections.abc.Callable | None + Private worker hook invoked when exact socket ownership is established. Returns ------- @@ -4843,7 +5304,6 @@ def setup_sync( from libtmux.experimental.ops import ( BatchingPlanner, BoundedPlanner, - DisplayMessage, KillSession, ListClients, ListSessions, @@ -4879,12 +5339,17 @@ def setup_sync( SetOption(server=True, option="exit-empty", value="off"), bootstrap, ).raise_for_status() - server_pid_result = run(DisplayMessage(message="#{pid}"), bootstrap) - server_pid_result.raise_for_status() - server_identity = _record_process("server", int(server_pid_result.text)) + socket_ownership = _capture_socket_ownership( + context.socket_path, + timeout_s=_WAIT_TIMEOUT_MAX_S, + ) + server_identity = socket_ownership.process + context.socket_ownership = socket_ownership context.processes = (*context.processes, server_identity) if context.process_identity_callback is not None: context.process_identity_callback((server_identity,)) + if _socket_ownership_callback is not None: + _socket_ownership_callback(socket_ownership) if lane is EngineLane.CONTROL: run(ListSessions(), engine).raise_for_status() @@ -4949,6 +5414,7 @@ async def setup_async( _process_identity_callback: ( cabc.Callable[[tuple[ProcessIdentity, ...]], None] | None ) = None, + _socket_ownership_callback: cabc.Callable[[_SocketOwnership], None] | None = None, ) -> RunContext: """Build and exactly verify one asynchronous live topology. @@ -4981,6 +5447,8 @@ async def setup_async( Unique pane assigned the delayed stream. _process_identity_callback : collections.abc.Callable | None Private worker hook invoked as exact process identities become known. + _socket_ownership_callback : collections.abc.Callable | None + Private worker hook invoked when exact socket ownership is established. Returns ------- @@ -5003,7 +5471,6 @@ async def setup_async( from libtmux.experimental.ops import ( BatchingPlanner, BoundedPlanner, - DisplayMessage, KillSession, ListClients, ListSessions, @@ -5043,12 +5510,17 @@ async def setup_async( bootstrap, ) ).raise_for_status() - server_pid_result = await arun(DisplayMessage(message="#{pid}"), bootstrap) - server_pid_result.raise_for_status() - server_identity = _record_process("server", int(server_pid_result.text)) + socket_ownership = _capture_socket_ownership( + context.socket_path, + timeout_s=_WAIT_TIMEOUT_MAX_S, + ) + server_identity = socket_ownership.process + context.socket_ownership = socket_ownership context.processes = (*context.processes, server_identity) if context.process_identity_callback is not None: context.process_identity_callback((server_identity,)) + if _socket_ownership_callback is not None: + _socket_ownership_callback(socket_ownership) if lane is EngineLane.CONTROL: control = t.cast(AsyncControlModeEngine, engine) await control.start() @@ -8416,6 +8888,46 @@ async def _await_if_needed(value: object) -> object: return value +def _repeatable_schedule( + strategy_names: cabc.Iterable[str], + *, + warmup: int, + runs: int, + seed: int, +) -> tuple[tuple[t.Literal["warmup", "timed"], str, int], ...]: + """Build the exact seeded, rotated repeatable-phase schedule. + + >>> _repeatable_schedule(("a", "b"), warmup=1, runs=1, seed=1) + (('warmup', 'b', 0), ('warmup', 'a', 0), ('timed', 'a', 0), ('timed', 'b', 0)) + + Parameters + ---------- + strategy_names : collections.abc.Iterable[str] + Strategy names in production mapping order. + warmup : int + Untimed invocation count per strategy. + runs : int + Timed invocation count per strategy. + seed : int + Seed used to shuffle the one base order. + + Returns + ------- + tuple[tuple[typing.Literal, str, int], ...] + Stage, strategy, and stage-local ordinal for every invocation. + """ + base_order = list(strategy_names) + random.Random(seed).shuffle(base_order) + schedule: list[tuple[t.Literal["warmup", "timed"], str, int]] = [] + for cycle in range(warmup + runs): + stage: t.Literal["warmup", "timed"] = "warmup" if cycle < warmup else "timed" + ordinal = cycle if stage == "warmup" else cycle - warmup + rotation = cycle % len(base_order) + cycle_order = (*base_order[rotation:], *base_order[:rotation]) + schedule.extend((stage, strategy, ordinal) for strategy in cycle_order) + return tuple(schedule) + + def _require_live_postcondition(value: object) -> None: """Require an exact true live postcondition result. @@ -8512,64 +9024,59 @@ async def run_repeatable_phase( message = "warmup must be nonnegative and runs must be positive" raise ValueError(message) sampler = snapshot_resources or (lambda: probe_host(ProcessReader())) - base_order = list(strategies) - random.Random(seed).shuffle(base_order) samples: list[RawSample] = [] order: list[str] = [] - total_cycles = warmup + runs - for cycle in range(total_cycles): - stage: t.Literal["warmup", "timed"] = "warmup" if cycle < warmup else "timed" - ordinal = cycle if stage == "warmup" else cycle - warmup - rotation = cycle % len(base_order) - cycle_order = (*base_order[rotation:], *base_order[:rotation]) - for strategy in cycle_order: - order.append(strategy) - try: - if boundary_callback is not None: - await _await_if_needed(boundary_callback(strategy)) - resources_before = sampler() - produced = strategies[strategy]() - measurement = _validated_phase_measurement( - await _await_if_needed(produced) + for stage, strategy, ordinal in _repeatable_schedule( + strategies, + warmup=warmup, + runs=runs, + seed=seed, + ): + order.append(strategy) + try: + if boundary_callback is not None: + await _await_if_needed(boundary_callback(strategy)) + resources_before = sampler() + produced = strategies[strategy]() + measurement = _validated_phase_measurement(await _await_if_needed(produced)) + postcondition = await _await_if_needed(live_postcondition(measurement)) + _require_live_postcondition(postcondition) + resources_after = sampler() + raw_sample: RawSample | None = None + if stage == "timed": + raw_sample = RawSample( + duration_ns=measurement.duration_ns, + accepted=True, + verified=True, + strategy=strategy, + ordinal=ordinal, + resources_before=resources_before, + resources_after=resources_after, ) - postcondition = await _await_if_needed(live_postcondition(measurement)) - _require_live_postcondition(postcondition) - resources_after = sampler() - raw_sample: RawSample | None = None - if stage == "timed": - raw_sample = RawSample( - duration_ns=measurement.duration_ns, - accepted=True, - verified=True, - strategy=strategy, - ordinal=ordinal, - resources_before=resources_before, - resources_after=resources_after, - ) - samples.append(raw_sample) - if progress_callback is not None: - await _await_if_needed( - progress_callback( - stage, - strategy, - ordinal, - measurement, - raw_sample, - ) + samples.append(raw_sample) + if progress_callback is not None: + await _await_if_needed( + progress_callback( + stage, + strategy, + ordinal, + measurement, + raw_sample, ) - except RuntimeCutoffError: - raise - except Exception as error: # noqa: BLE001 - return RepeatablePhaseResult( - samples=tuple(samples), - order=tuple(order), - failure=RepeatablePhaseFailure( - stage=stage, - strategy=strategy, - ordinal=ordinal, - error=f"{type(error).__name__}: {error}", - ), ) + except RuntimeCutoffError: + raise + except Exception as error: # noqa: BLE001 + return RepeatablePhaseResult( + samples=tuple(samples), + order=tuple(order), + failure=RepeatablePhaseFailure( + stage=stage, + strategy=strategy, + ordinal=ordinal, + error=f"{type(error).__name__}: {error}", + ), + ) return RepeatablePhaseResult(tuple(samples), tuple(order)) @@ -8673,22 +9180,13 @@ async def cleanup_run( except Exception as error: # noqa: BLE001 errors.append(f"engine close: {type(error).__name__}: {error}") - try: - context.server.kill() - except Exception as error: # noqa: BLE001 - errors.append(f"server kill: {type(error).__name__}: {error}") - if context.socket_path.exists(): - server_identity = next( - (identity for identity in context.processes if identity.role == "server"), - None, - ) - errors.extend( - _kill_exact_tmux_socket( - context.socket_path, - timeout_s=grace_s, - server_identity=server_identity, - ) + errors.extend( + _kill_exact_tmux_socket( + context.socket_path, + timeout_s=grace_s, + socket_ownership=context.socket_ownership, ) + ) survivors = await _wait_for_process_absence( context.processes, @@ -8699,6 +9197,8 @@ async def cleanup_run( if not survivors: break for identity in survivors: + if identity.role == "server": + continue registry.signal(identity, signal_number) survivors = await _wait_for_process_absence( context.processes, @@ -8717,10 +9217,6 @@ async def cleanup_run( processes_absent = all( not process_identity_matches(identity) for identity in context.processes ) - if processes_absent and context.socket_path.exists(): - errors.extend( - _remove_proven_stale_socket(context.socket_path, context.processes) - ) socket_absent = not context.socket_path.exists() if processes_absent and socket_absent: try: @@ -9113,6 +9609,7 @@ def _progress_event_from_json(value: object) -> ProgressEvent: checkpoint=t.cast(str, row.get("checkpoint")), monotonic_ns=t.cast(int, row.get("monotonic_ns")), processes=tuple(_identity_from_json(item) for item in row.get("processes", [])), + socket_ownership=_socket_ownership_from_json(row.get("socket_ownership")), schema_version=t.cast(int, row.get("schema_version", 1)), ) if ( @@ -9173,6 +9670,7 @@ def checkpoint( name: str, *, identity_delta: tuple[ProcessIdentity, ...] = (), + socket_ownership: _SocketOwnership | None = None, ) -> None: """Publish an increasing event before its matching report checkpoint. @@ -9199,6 +9697,7 @@ def checkpoint( checkpoint=name, monotonic_ns=time.monotonic_ns(), processes=identity_delta, + socket_ownership=socket_ownership, ), ) write_json_atomic(self.checkpoint_path, self.report) @@ -9263,6 +9762,34 @@ def record_identities( checkpoint = f"identity.{role}" self.checkpoint(checkpoint, identity_delta=tuple(delta)) + def record_socket_ownership(self, ownership: _SocketOwnership) -> None: + """Durably publish the immutable server/socket capability once. + + Parameters + ---------- + ownership : _SocketOwnership + Capability captured around the exact socket owner query. + + Raises + ------ + RuntimeError + If a different capability was already recorded for this run. + """ + existing = self.report.socket_ownership + if existing is not None and existing != ownership: + message = "worker socket ownership changed after establishment" + raise RuntimeError(message) + if existing is not None: + return + self.report = dataclasses.replace( + self.report, + socket_ownership=ownership, + ) + self.checkpoint( + "identity.server.socket", + socket_ownership=ownership, + ) + async def _worker_live_postcondition( context: RunContext, @@ -9416,16 +9943,16 @@ async def on_boundary(strategy: str) -> None: active_phase[0] = strategy if strategy in phase_states: publish_active(strategy) - return - phase_states[strategy] = PhaseReport( - name=strategy, - requested_topology=topology, - observed_topology=topology, - status="in_progress", - warmup=warmup, - runs=runs, - ) - publish_active(strategy) + else: + phase_states[strategy] = PhaseReport( + name=strategy, + requested_topology=topology, + observed_topology=topology, + status="in_progress", + warmup=warmup, + runs=runs, + ) + publish_active(strategy) recorder.checkpoint(f"{strategy}.started") async def on_progress( @@ -9753,6 +10280,7 @@ def fail_if_requested(phase: str) -> None: run_id=run_id, delayed_ordinal=delayed_ordinal, _process_identity_callback=recorder.record_identities, + _socket_ownership_callback=recorder.record_socket_ownership, ) else: context = await setup_async( @@ -9763,6 +10291,7 @@ def fail_if_requested(phase: str) -> None: run_id=run_id, delayed_ordinal=delayed_ordinal, _process_identity_callback=recorder.record_identities, + _socket_ownership_callback=recorder.record_socket_ownership, ) resources_after = probe_host(ProcessReader()) setup_duration = max(1, context.setup_duration_ns) @@ -10353,11 +10882,20 @@ def wait_worker() -> bool: unique[(identity.pid, identity.start_time)] = identity owned = tuple(unique.values()) progress.handles.retain_many(owned) + errors.extend( + _kill_exact_tmux_socket( + socket_path, + timeout_s=grace_s, + socket_ownership=progress.socket_ownership, + ) + ) survivors = _wait_identity_absence(owned, timeout_s=grace_s) for signal_number in (signal.SIGTERM, signal.SIGKILL): if not survivors: break for identity in survivors: + if identity.role == "server": + continue progress.handles.signal(identity, signal_number) survivors = _wait_identity_absence(owned, timeout_s=grace_s) errors.extend( @@ -10369,20 +10907,6 @@ def wait_worker() -> bool: processes_absent = worker_absent and not any( process_identity_matches(identity) for identity in owned ) - if socket_path.exists(): - server_identity = next( - (identity for identity in owned if identity.role == "server"), - None, - ) - errors.extend( - _kill_exact_tmux_socket( - socket_path, - timeout_s=grace_s, - server_identity=server_identity, - ) - ) - if processes_absent and socket_path.exists(): - errors.extend(_remove_proven_stale_socket(socket_path, owned)) socket_absent = not socket_path.exists() errors.extend(progress.handles.errors) if progress.journal_error is not None and not any( @@ -10500,6 +11024,8 @@ class _ProgressTracker: Registry that binds each accepted identity delta. identities : tuple[ProcessIdentity, ...] Cumulative identities retained once for the terminal report. + socket_ownership : _SocketOwnership | None + Immutable server/socket capability accepted from progress. highest_sequence : int Last strictly increasing accepted sequence. offset : int @@ -10524,6 +11050,7 @@ class _ProgressTracker: run_id: str handles: _PidfdRegistry identities: tuple[ProcessIdentity, ...] = () + socket_ownership: _SocketOwnership | None = None highest_sequence: int = -1 offset: int = 0 remainder: str = "" @@ -10559,7 +11086,32 @@ def drain(self, *, terminal: bool = False) -> bool: ValueError If terminal JSONL evidence is missing, torn, or malformed. """ + + def merge_socket_ownership( + events: tuple[ProgressEvent, ...], + *, + previous_highest: int, + highest: int, + identities: tuple[ProcessIdentity, ...], + ) -> _SocketOwnership | None: + ownership = self.socket_ownership + for event in events: + if not previous_highest < event.sequence <= highest: + continue + observed = event.socket_ownership + if observed is None: + continue + if observed.process not in identities: + message = "socket ownership process is absent from progress" + raise RuntimeError(message) + if ownership is not None and ownership != observed: + message = "socket ownership changed in progress" + raise RuntimeError(message) + ownership = observed + return ownership + try: + previous_highest = self.highest_sequence events, offset, remainder = _read_progress_chunk( self.path, self.offset, @@ -10572,6 +11124,12 @@ def drain(self, *, terminal: bool = False) -> bool: highest_sequence=self.highest_sequence, identities=self.identities, ) + socket_ownership = merge_socket_ownership( + events, + previous_highest=previous_highest, + highest=highest, + identities=identities, + ) except (RuntimeError, ValueError) as error: if self.journal_error is None: self.journal_error = f"{type(error).__name__}: {error}" @@ -10580,6 +11138,7 @@ def drain(self, *, terminal: bool = False) -> bool: self.remainder = remainder self.highest_sequence = highest self.identities = identities + self.socket_ownership = socket_ownership self.handles.retain_many( identity for event in events for identity in event.processes ) @@ -10594,6 +11153,7 @@ def _drain_finalizer_thread( *, initial_interrupt: KeyboardInterrupt | None = None, interrupt_state: list[KeyboardInterrupt | None] | None = None, + restore_handlers: cabc.Mapping[signal.Signals, t.Any] | None = None, ) -> tuple[_FinalizerResult, KeyboardInterrupt | None]: """Drain independent synchronous finalization through repeated interrupts. @@ -10609,6 +11169,9 @@ def _drain_finalizer_thread( First interruption already translated into terminal report state. interrupt_state : list[KeyboardInterrupt | None] | None Optional single-item state shared with an interruption-aware callback. + restore_handlers : collections.abc.Mapping | None + Exact caller handlers restored inside this protected ownership scope. + Unspecified signals restore the handlers observed on entry. Returns ------- @@ -10656,6 +11219,13 @@ def record_interrupt(error: KeyboardInterrupt) -> None: previous_handlers = { signal_number: signal.getsignal(signal_number) for signal_number in blocked } + target_handlers = dict(previous_handlers) + if restore_handlers is not None: + unknown = set(restore_handlers) - set(blocked) + if unknown: + message = "finalizer restore handlers contain an unsupported signal" + raise ValueError(message) + target_handlers.update(restore_handlers) def record_signal( signal_number: int, @@ -10755,10 +11325,19 @@ def record_signal( record_interrupt(error) restore_failure: BaseException | None = None + while True: + try: + signal.pthread_sigmask(signal.SIG_BLOCK, blocked) + break + except KeyboardInterrupt as error: + record_interrupt(error) + except BaseException as error: # noqa: BLE001 + restore_failure = error + break for signal_number in reversed(installed_handlers): while True: try: - signal.signal(signal_number, previous_handlers[signal_number]) + signal.signal(signal_number, target_handlers[signal_number]) break except KeyboardInterrupt as error: record_interrupt(error) @@ -10766,6 +11345,16 @@ def record_signal( if restore_failure is None: restore_failure = error break + while True: + try: + signal.pthread_sigmask(signal.SIG_SETMASK, previous_mask) + break + except KeyboardInterrupt as error: + record_interrupt(error) + except BaseException as error: # noqa: BLE001 + if restore_failure is None: + restore_failure = error + break interruption = interrupt_state[0] if not launched: @@ -10947,6 +11536,7 @@ def _finalize_supervised_worker( failed_phase=(None if final_status == "completed" else failed_phase), error=(None if final_status == "completed" else terminal_error), processes=progress.identities, + socket_ownership=progress.socket_ownership, progress_path=str(progress.path), progress_sequence=progress.highest_sequence, guard_decision=( @@ -11018,7 +11608,10 @@ def supervise_worker( """Launch and supervise the hidden worker with a sequence-based watchdog. The ``_test_*`` arguments are private CLI injection points used only by the - benchmark's subprocess recovery tests. + benchmark's subprocess recovery tests. Artifact ownership linearizes when + the sole finalizer has durably written and validated JSON and Markdown, + applied any already observed cutoff, and commits that result back to the + caller. A later signal is re-raised without rewriting committed evidence. >>> try: ... supervise_worker( @@ -11214,6 +11807,7 @@ def request_termination( terminal_error = f"{type(error).__name__}: {error}" interrupt_state = [interruption] + artifact_committed = [False] def finalize_once() -> RunReport: final_report = _finalize_supervised_worker( @@ -11253,19 +11847,21 @@ def finalize_once() -> RunReport: output, markdown_output, ) + artifact_committed[0] = True return final_report - try: - final_report, deferred_interruption = _drain_finalizer_thread( - finalize_once, - initial_interrupt=interruption, - interrupt_state=interrupt_state, - ) - if deferred_interruption is not None: - raise deferred_interruption - return final_report - finally: - signal.signal(signal.SIGTERM, previous_sigterm_handler) + final_report, deferred_interruption = _drain_finalizer_thread( + finalize_once, + initial_interrupt=interruption, + interrupt_state=interrupt_state, + restore_handlers={signal.SIGTERM: previous_sigterm_handler}, + ) + if not artifact_committed[0]: + message = "supervisor finalizer returned without committing artifacts" + raise RuntimeError(message) + if deferred_interruption is not None: + raise deferred_interruption + return final_report def _write_text_atomic(path: pathlib.Path, text: str) -> None: diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index b2194b9081..7497baaf3a 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -11,6 +11,7 @@ import os import pathlib import signal +import socket import stat import subprocess import sys @@ -1264,7 +1265,7 @@ def test_exact_socket_fallback_kills_only_the_configured_tmux_server( """A failed helper must not hide a live exact server behind an unlinked socket.""" target = tmp_path / "target.sock" unrelated = tmp_path / "unrelated.sock" - identities: dict[pathlib.Path, t.Any] = {} + ownerships: dict[pathlib.Path, t.Any] = {} for socket_path in (target, unrelated): created = subprocess.run( ( @@ -1288,8 +1289,9 @@ def test_exact_socket_fallback_kills_only_the_configured_tmux_server( text=True, ) assert server_pid.returncode == 0, server_pid.stderr - identities[socket_path] = benchmark_module._record_process( - "server", int(server_pid.stdout.strip()) + ownerships[socket_path] = benchmark_module._capture_socket_ownership( + socket_path, + timeout_s=1.0, ) real_popen = subprocess.Popen @@ -1311,12 +1313,16 @@ def unlink_without_killing_server( benchmark_module.subprocess, "Popen", unlink_without_killing_server ) try: - errors = benchmark_module._kill_exact_tmux_socket(target, timeout_s=1.0) + errors = benchmark_module._kill_exact_tmux_socket( + target, + timeout_s=1.0, + socket_ownership=ownerships[target], + ) assert errors == ("exact socket kill-server exited 7",) assert not target.exists() - assert not benchmark_module.process_identity_matches(identities[target]) - assert benchmark_module.process_identity_matches(identities[unrelated]) + assert not benchmark_module.process_identity_matches(ownerships[target].process) + assert benchmark_module.process_identity_matches(ownerships[unrelated].process) still_alive = subprocess.run( ("tmux", "-S", str(unrelated), "list-sessions"), check=False, @@ -1334,23 +1340,415 @@ def unlink_without_killing_server( ).wait(timeout=2.0) registry = benchmark_module._PidfdRegistry() try: - for identity in identities.values(): + for ownership in ownerships.values(): + identity = ownership.process if registry.retain(identity): registry.signal(identity, signal.SIGKILL) - benchmark_module._wait_identity_absence(identities.values(), timeout_s=2.0) + benchmark_module._wait_identity_absence( + (ownership.process for ownership in ownerships.values()), + timeout_s=2.0, + ) finally: registry.close() - for socket_path, identity in identities.items(): - benchmark_module._remove_proven_stale_socket(socket_path, (identity,)) + for socket_path, ownership in ownerships.items(): + benchmark_module._remove_proven_stale_socket(socket_path, ownership) assert all( - not benchmark_module.process_identity_matches(identity) - for identity in identities.values() + not benchmark_module.process_identity_matches(ownership.process) + for ownership in ownerships.values() ) assert not target.exists() assert not unrelated.exists() +def test_cleanup_preserves_live_replacement_after_original_server_exits( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """Cleanup must not contact or unlink a replacement at the recorded path.""" + scratch = tmp_path / "replacement-run" + context = benchmark_module.setup_sync( + benchmark_module.Topology(1, 1, 1), + benchmark_module.EngineLane.SUBPROCESS, + scratch, + socket_path=scratch / "tmux.sock", + run_id="replacement-live", + ) + original = next( + identity for identity in context.processes if identity.role == "server" + ) + replacement: t.Any = None + replacement_status: os.stat_result | None = None + try: + stopped = subprocess.run( + ("tmux", "-S", str(context.socket_path), "kill-server"), + check=False, + capture_output=True, + text=True, + timeout=2.0, + ) + assert stopped.returncode == 0, stopped.stderr + assert not benchmark_module._wait_identity_absence((original,), timeout_s=2.0) + + created = subprocess.run( + ( + "tmux", + "-S", + str(context.socket_path), + "new-session", + "-d", + "-s", + "replacement", + ), + check=False, + capture_output=True, + text=True, + timeout=2.0, + ) + assert created.returncode == 0, created.stderr + replacement_pid = subprocess.run( + ( + "tmux", + "-S", + str(context.socket_path), + "display-message", + "-p", + "#{pid}", + ), + check=False, + capture_output=True, + text=True, + timeout=2.0, + ) + assert replacement_pid.returncode == 0, replacement_pid.stderr + replacement = benchmark_module._record_process( + "replacement", int(replacement_pid.stdout.strip()) + ) + replacement_status = context.socket_path.lstat() + + cleanup = asyncio.run(benchmark_module.cleanup_run(context, grace_s=0.3)) + + assert not cleanup.complete + assert cleanup.errors + assert not benchmark_module.process_identity_matches(original) + assert benchmark_module.process_identity_matches(replacement) + observed = context.socket_path.lstat() + assert (observed.st_dev, observed.st_ino) == ( + replacement_status.st_dev, + replacement_status.st_ino, + ) + answering = subprocess.run( + ("tmux", "-S", str(context.socket_path), "list-sessions"), + check=False, + capture_output=True, + text=True, + timeout=2.0, + ) + assert answering.returncode == 0, answering.stderr + finally: + subprocess.run( + ("tmux", "-S", str(context.socket_path), "kill-server"), + check=False, + capture_output=True, + timeout=2.0, + ) + registry = benchmark_module._PidfdRegistry() + try: + for identity in ( + *context.processes, + *((replacement,) if replacement else ()), + ): + if registry.retain(identity): + registry.signal(identity, signal.SIGKILL) + benchmark_module._wait_identity_absence( + (*context.processes, *((replacement,) if replacement else ())), + timeout_s=2.0, + ) + finally: + registry.close() + context.socket_path.unlink(missing_ok=True) + benchmark_module._remove_supervised_scratch(scratch) + + +def test_cleanup_kills_live_original_without_touching_replacement_path( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """A replaced pathname must remain bound to its answering replacement.""" + scratch = tmp_path / "replaced-path-run" + context = benchmark_module.setup_sync( + benchmark_module.Topology(1, 1, 1), + benchmark_module.EngineLane.SUBPROCESS, + scratch, + socket_path=scratch / "tmux.sock", + run_id="replacement-original-live", + ) + original = next( + identity for identity in context.processes if identity.role == "server" + ) + original_socket = scratch / "original.sock" + replacement: t.Any = None + replacement_status: os.stat_result | None = None + try: + context.socket_path.rename(original_socket) + created = subprocess.run( + ( + "tmux", + "-S", + str(context.socket_path), + "new-session", + "-d", + "-s", + "replacement", + ), + check=False, + capture_output=True, + text=True, + timeout=2.0, + ) + assert created.returncode == 0, created.stderr + replacement_pid = subprocess.run( + ( + "tmux", + "-S", + str(context.socket_path), + "display-message", + "-p", + "#{pid}", + ), + check=False, + capture_output=True, + text=True, + timeout=2.0, + ) + assert replacement_pid.returncode == 0, replacement_pid.stderr + replacement = benchmark_module._record_process( + "replacement", int(replacement_pid.stdout.strip()) + ) + replacement_status = context.socket_path.lstat() + + cleanup = asyncio.run(benchmark_module.cleanup_run(context, grace_s=0.3)) + + assert not cleanup.complete + assert cleanup.errors + assert not benchmark_module.process_identity_matches(original) + assert benchmark_module.process_identity_matches(replacement) + observed = context.socket_path.lstat() + assert (observed.st_dev, observed.st_ino) == ( + replacement_status.st_dev, + replacement_status.st_ino, + ) + answering = subprocess.run( + ("tmux", "-S", str(context.socket_path), "list-sessions"), + check=False, + capture_output=True, + text=True, + timeout=2.0, + ) + assert answering.returncode == 0, answering.stderr + finally: + subprocess.run( + ("tmux", "-S", str(context.socket_path), "kill-server"), + check=False, + capture_output=True, + timeout=2.0, + ) + registry = benchmark_module._PidfdRegistry() + try: + for identity in ( + *context.processes, + *((replacement,) if replacement else ()), + ): + if registry.retain(identity): + registry.signal(identity, signal.SIGKILL) + benchmark_module._wait_identity_absence( + (*context.processes, *((replacement,) if replacement else ())), + timeout_s=2.0, + ) + finally: + registry.close() + context.socket_path.unlink(missing_ok=True) + original_socket.unlink(missing_ok=True) + benchmark_module._remove_supervised_scratch(scratch) + + +def test_cleanup_preserves_stale_same_user_replacement_socket( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """An unanswering replacement inode must remain as mismatch evidence.""" + scratch = tmp_path / "stale-replacement-run" + context = benchmark_module.setup_sync( + benchmark_module.Topology(1, 1, 1), + benchmark_module.EngineLane.SUBPROCESS, + scratch, + socket_path=scratch / "tmux.sock", + run_id="replacement-stale", + ) + original = next( + identity for identity in context.processes if identity.role == "server" + ) + stale_socket: socket.socket | None = None + stale_status: os.stat_result | None = None + try: + stopped = subprocess.run( + ("tmux", "-S", str(context.socket_path), "kill-server"), + check=False, + capture_output=True, + text=True, + timeout=2.0, + ) + assert stopped.returncode == 0, stopped.stderr + assert not benchmark_module._wait_identity_absence((original,), timeout_s=2.0) + context.socket_path.unlink(missing_ok=True) + stale_socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + stale_socket.bind(str(context.socket_path)) + stale_status = context.socket_path.lstat() + stale_socket.close() + stale_socket = None + + cleanup = asyncio.run(benchmark_module.cleanup_run(context, grace_s=0.3)) + + assert not cleanup.complete + assert cleanup.errors + assert not benchmark_module.process_identity_matches(original) + observed = context.socket_path.lstat() + assert (observed.st_dev, observed.st_ino) == ( + stale_status.st_dev, + stale_status.st_ino, + ) + finally: + if stale_socket is not None: + stale_socket.close() + registry = benchmark_module._PidfdRegistry() + try: + for identity in context.processes: + if registry.retain(identity): + registry.signal(identity, signal.SIGKILL) + benchmark_module._wait_identity_absence( + context.processes, + timeout_s=2.0, + ) + finally: + registry.close() + context.socket_path.unlink(missing_ok=True) + benchmark_module._remove_supervised_scratch(scratch) + + +def test_exact_socket_cleanup_without_capability_preserves_path( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """A pathname alone grants no authority to contact or unlink its socket.""" + private = tmp_path / "missing-capability" + private.mkdir(mode=0o700) + socket_path = private / "tmux.sock" + node = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + node.bind(str(socket_path)) + before = socket_path.lstat() + try: + errors = benchmark_module._kill_exact_tmux_socket( + socket_path, + timeout_s=0.1, + ) + + after = socket_path.lstat() + assert errors + assert "capability unavailable" in errors[0] + assert (after.st_dev, after.st_ino) == (before.st_dev, before.st_ino) + finally: + node.close() + socket_path.unlink(missing_ok=True) + + +def test_exact_socket_cleanup_retain_failure_never_uses_pathname( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """Failed stable-handle retention forbids pathname kill-server.""" + private = tmp_path / "retain-failure" + private.mkdir(mode=0o700) + socket_path = private / "tmux.sock" + created = subprocess.run( + ( + "tmux", + "-S", + str(socket_path), + "new-session", + "-d", + "-s", + "owned", + ), + check=False, + capture_output=True, + text=True, + timeout=2.0, + ) + assert created.returncode == 0, created.stderr + ownership = benchmark_module._capture_socket_ownership( + socket_path, + timeout_s=1.0, + ) + before = socket_path.lstat() + pathname_calls = 0 + + def refuse_owner_handle(current: t.Any, identity: t.Any) -> bool: + if identity == ownership.process: + current.errors.append("injected pidfd retain failure") + return False + pytest.fail("socket cleanup retained an unexpected identity") + + def reject_pathname(*_args: t.Any, **_kwargs: t.Any) -> t.NoReturn: + nonlocal pathname_calls + pathname_calls += 1 + pytest.fail("retain failure attempted pathname kill-server") + + monkeypatch.setattr( + benchmark_module._PidfdRegistry, + "retain", + refuse_owner_handle, + ) + monkeypatch.setattr( + benchmark_module, + "subprocess", + types.SimpleNamespace(Popen=reject_pathname), + ) + try: + errors = benchmark_module._kill_exact_tmux_socket( + socket_path, + timeout_s=0.1, + socket_ownership=ownership, + ) + + after = socket_path.lstat() + assert errors + assert any("stable handle unavailable" in error for error in errors) + assert pathname_calls == 0 + assert benchmark_module.process_identity_matches(ownership.process) + assert (after.st_dev, after.st_ino) == (before.st_dev, before.st_ino) + answering = subprocess.run( + ("tmux", "-S", str(socket_path), "list-sessions"), + check=False, + capture_output=True, + text=True, + timeout=2.0, + ) + assert answering.returncode == 0, answering.stderr + finally: + subprocess.run( + ("tmux", "-S", str(socket_path), "kill-server"), + check=False, + capture_output=True, + timeout=2.0, + ) + benchmark_module._wait_identity_absence( + (ownership.process,), + timeout_s=2.0, + ) + socket_path.unlink(missing_ok=True) + + def test_run_scenario_refuses_when_pidfds_are_unavailable( benchmark_module: types.ModuleType, monkeypatch: pytest.MonkeyPatch, @@ -4169,61 +4567,307 @@ async def live_postcondition(*_args: t.Any, **_kwargs: t.Any) -> bool: assert all(phase.status == "completed" for phase in recorder.report.phases) -def test_cli_run_executes_every_phase_and_writes_validated_artifacts( +def test_worker_group_checkpoints_revisited_boundary_before_progress( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, tmp_path: pathlib.Path, ) -> None: - """Skipping a phase or requested sample would publish incomplete evidence.""" - report_path = tmp_path / "run.json" - markdown_path = tmp_path / "run.md" - scratch_root = tmp_path / "scratch" - - completed = _run_cli( - "run", - "--shape", - "2x2x2", - "--runs", - "2", - "--warmup", - "1", - "--output", - str(report_path), - "--markdown-output", - str(markdown_path), - "--scratch-root", - str(scratch_root), - "--watchdog-seconds", - "30", - cwd=tmp_path, + """Cancellation after a repeated boundary must retain that exact strategy.""" + base = _failed_report_with_completed_mutation_prefix(benchmark_module) + initial = dataclasses.replace( + base, + status="in_progress", + phases=base.phases[:3], + cleanup=benchmark_module.CleanupReport(False), + failed_phase=None, + error=None, + lane="control", + mode="async", + ) + checkpoint_path = tmp_path / "boundary.json" + progress_path = tmp_path / "boundary.jsonl" + recorder = benchmark_module._WorkerRecorder( + initial, + checkpoint_path, + progress_path, ) + cancellation = asyncio.CancelledError("cancelled after repeated boundary") + real_checkpoint = benchmark_module._WorkerRecorder.checkpoint + boundary_count = 0 - assert completed.returncode == 0, completed.stderr - payload = json.loads(report_path.read_text(encoding="utf-8")) - assert payload["status"] == "completed" - assert payload["requested_topology"] == { - "sessions": 2, - "windows_per_session": 2, - "panes_per_window": 2, - } - assert payload["observed_topology"] == payload["requested_topology"] - assert payload["lane"] == "control" - assert payload["mode"] == "async" - phases = {phase["name"]: phase for phase in payload["phases"]} - assert tuple(phases) == ( - "setup", - "stabilization", - *_RUNNER_REPEATABLE_PHASES[:2], - "wait.control-stream", - *_RUNNER_REPEATABLE_PHASES[2:], + def interrupt_repeated_boundary( + current: t.Any, + name: str, + **kwargs: t.Any, + ) -> None: + nonlocal boundary_count + real_checkpoint(current, name, **kwargs) + if name == "wait.control-stream.started": + boundary_count += 1 + if boundary_count == 2: + raise cancellation + + def measured(name: str) -> t.Callable[[], t.Any]: + return lambda: benchmark_module.SearchResult( + 23, + "snapshot", + "sessions", + 1, + name, + (name,), + verified=True, + ) + + async def live_postcondition(*_args: t.Any, **_kwargs: t.Any) -> bool: + return True + + monkeypatch.setattr( + benchmark_module._WorkerRecorder, + "checkpoint", + interrupt_repeated_boundary, ) - assert phases["setup"]["summary"] is None - assert len(phases["setup"]["samples"]) == 1 - assert phases["stabilization"]["samples"] == [] - for phase_name in (*_RUNNER_REPEATABLE_PHASES, "wait.control-stream"): - phase = phases[phase_name] - assert phase["status"] == "completed" - assert phase["warmup"] == 1 - assert phase["runs"] == 2 - assert [row["ordinal"] for row in phase["warmup_observations"]] == [0] + monkeypatch.setattr( + benchmark_module, "_worker_live_postcondition", live_postcondition + ) + monkeypatch.setattr( + benchmark_module, + "probe_host", + lambda _reader: benchmark_module.HostSnapshot(), + ) + + with pytest.raises(asyncio.CancelledError) as raised: + asyncio.run( + benchmark_module._run_worker_group( + recorder, + types.SimpleNamespace(topology=base.requested_topology), + { + "wait.capture-poll": measured("wait.capture-poll"), + "wait.control-stream": measured("wait.control-stream"), + }, + warmup=2, + runs=2, + seed=12, + policy=benchmark_module.ResourcePolicy(), + latest={}, + fail_after=None, + active_phase=["mutation.bulk"], + ) + ) + + assert isinstance(raised.value, asyncio.CancelledError) + assert boundary_count == 2 + durable = benchmark_module.load_run_report(checkpoint_path) + assert tuple(phase.name for phase in durable.phases) == ( + "setup", + "stabilization", + "mutation.bulk", + "wait.control-stream", + ) + active = durable.phases[-1] + assert active.status == "in_progress" + assert len(active.warmup_observations) == 1 + assert active.samples == () + last_event = benchmark_module._progress_event_from_json( + json.loads(progress_path.read_text(encoding="utf-8").splitlines()[-1]) + ) + assert last_event.checkpoint == "wait.control-stream.started" + terminal = dataclasses.replace( + durable, + status="cutoff", + cleanup=benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + failed_phase="cancellation", + error="CancelledError: cancelled after repeated boundary", + ) + benchmark_module.validate_report(terminal) + + +def test_validator_accepts_actual_seeded_group_boundaries( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Real warmup, timed, cutoff, and failed snapshots remain executable.""" + base = _failed_report_with_completed_mutation_prefix(benchmark_module) + initial = dataclasses.replace( + base, + status="in_progress", + phases=base.phases[:3], + cleanup=benchmark_module.CleanupReport(False), + failed_phase=None, + error=None, + lane="control", + mode="async", + ) + checkpoints: list[str] = [] + + class ValidatingRecorder: + def __init__(self) -> None: + self.report = initial + + def checkpoint(self, name: str) -> None: + checkpoints.append(name) + cutoff = dataclasses.replace( + self.report, + status="cutoff", + cleanup=benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + failed_phase="cancellation", + error="CancelledError: exact boundary", + ) + benchmark_module.validate_report(cutoff) + current = name.removesuffix(".started") + for marker in (".warmup.", ".timed."): + if marker in current: + current = current.split(marker, maxsplit=1)[0] + if current not in {phase.name for phase in self.report.phases}: + return + prior = self.report.phases[:3] + group = self.report.phases[3:] + active = next(phase for phase in group if phase.name == current) + completed = tuple( + phase + for phase in group + if phase.name != current and phase.status == "completed" + ) + failed_phases = ( + *prior, + *completed, + dataclasses.replace(active, status="failed", summary=None), + ) + benchmark_module.validate_report( + dataclasses.replace( + cutoff, + status="failed", + phases=failed_phases, + failed_phase=current, + error=f"RuntimeError: {current} failed", + ) + ) + + def measured(name: str) -> t.Callable[[], t.Any]: + strategy = t.cast( + t.Literal["capture-poll", "control-stream"], + name.removeprefix("wait."), + ) + return lambda: benchmark_module.WaitResult( + strategy, + name, + "token", + "%1", + 5, + 7, + 12, + 14, + 17, + 2, + 3, + 1 if strategy == "capture-poll" else 0, + 1 if strategy == "control-stream" else 0, + 1.0, + False, + 0, + True, + ) + + async def live_postcondition(*_args: t.Any, **_kwargs: t.Any) -> bool: + return True + + monkeypatch.setattr( + benchmark_module, "_worker_live_postcondition", live_postcondition + ) + monkeypatch.setattr( + benchmark_module, + "probe_host", + lambda _reader: benchmark_module.HostSnapshot(), + ) + recorder = ValidatingRecorder() + + asyncio.run( + benchmark_module._run_worker_group( + recorder, + types.SimpleNamespace(topology=base.requested_topology), + { + "wait.capture-poll": measured("wait.capture-poll"), + "wait.control-stream": measured("wait.control-stream"), + }, + warmup=2, + runs=2, + seed=12, + policy=benchmark_module.ResourcePolicy(), + latest={}, + fail_after=None, + active_phase=["mutation.bulk"], + ) + ) + + assert any(name.endswith(".started") for name in checkpoints) + assert any(".warmup." in name for name in checkpoints) + assert any(".timed." in name for name in checkpoints) + + +def test_cli_run_executes_every_phase_and_writes_validated_artifacts( + tmp_path: pathlib.Path, +) -> None: + """Skipping a phase or requested sample would publish incomplete evidence.""" + report_path = tmp_path / "run.json" + markdown_path = tmp_path / "run.md" + scratch_root = tmp_path / "scratch" + + completed = _run_cli( + "run", + "--shape", + "2x2x2", + "--runs", + "2", + "--warmup", + "1", + "--output", + str(report_path), + "--markdown-output", + str(markdown_path), + "--scratch-root", + str(scratch_root), + "--watchdog-seconds", + "30", + cwd=tmp_path, + ) + + assert completed.returncode == 0, completed.stderr + payload = json.loads(report_path.read_text(encoding="utf-8")) + assert payload["status"] == "completed" + assert payload["requested_topology"] == { + "sessions": 2, + "windows_per_session": 2, + "panes_per_window": 2, + } + assert payload["observed_topology"] == payload["requested_topology"] + assert payload["lane"] == "control" + assert payload["mode"] == "async" + phases = {phase["name"]: phase for phase in payload["phases"]} + assert tuple(phases) == ( + "setup", + "stabilization", + *_RUNNER_REPEATABLE_PHASES[:2], + "wait.control-stream", + *_RUNNER_REPEATABLE_PHASES[2:], + ) + assert phases["setup"]["summary"] is None + assert len(phases["setup"]["samples"]) == 1 + assert phases["stabilization"]["samples"] == [] + for phase_name in (*_RUNNER_REPEATABLE_PHASES, "wait.control-stream"): + phase = phases[phase_name] + assert phase["status"] == "completed" + assert phase["warmup"] == 1 + assert phase["runs"] == 2 + assert [row["ordinal"] for row in phase["warmup_observations"]] == [0] assert len(phase["samples"]) == 2 assert phase["summary"]["count"] == 2 assert len(phase["observations"]) == 2 @@ -4472,6 +5116,122 @@ def test_worker_progress_precedes_matching_report_checkpoint( assert calls == ["progress", "checkpoint"] +def test_socket_ownership_round_trips_and_rejects_progress_mutation( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """Checkpoint and journal retain one immutable process/socket capability.""" + checkpoint = tmp_path / "ownership.json" + progress = tmp_path / "ownership.jsonl" + owner = benchmark_module._record_process("server", os.getpid()) + ownership = benchmark_module._SocketOwnership( + owner, + benchmark_module.SocketIdentity( + 7, + 11, + os.getuid(), + stat.S_IFSOCK | 0o600, + ), + ) + recorder = benchmark_module._WorkerRecorder( + benchmark_module.RunReport( + benchmark_module.Topology(1, 1, 1), + run_id="run-7", + ), + checkpoint, + progress, + ) + + recorder.record_identities((owner,), checkpoint="identity.server") + recorder.record_socket_ownership(ownership) + + loaded = benchmark_module.load_run_report(checkpoint) + payload = json.loads(checkpoint.read_text(encoding="utf-8")) + events = tuple( + benchmark_module._progress_event_from_json(json.loads(line)) + for line in progress.read_text(encoding="utf-8").splitlines() + ) + expected = { + "process": { + "role": "server", + "pid": owner.pid, + "start_time": owner.start_time, + }, + "socket": { + "st_dev": 7, + "st_ino": 11, + "st_uid": os.getuid(), + "st_mode": stat.S_IFSOCK | 0o600, + }, + } + assert loaded.socket_ownership == ownership + assert payload["socket_ownership"] == expected + assert events[-1].socket_ownership == ownership + + registry = benchmark_module._PidfdRegistry() + tracker = benchmark_module._ProgressTracker(progress, "run-7", registry) + try: + assert tracker.drain(terminal=True) + assert tracker.socket_ownership == ownership + changed = dataclasses.replace( + ownership, + socket=dataclasses.replace(ownership.socket, st_ino=12), + ) + benchmark_module.append_progress_event( + progress, + benchmark_module.ProgressEvent( + "run-7", + 2, + "identity.server.socket.changed", + time.monotonic_ns(), + socket_ownership=changed, + ), + ) + with pytest.raises(RuntimeError, match="socket ownership changed"): + tracker.drain(terminal=True) + finally: + registry.close() + + +@pytest.mark.parametrize( + ("field", "value"), + ( + ("st_dev", True), + ("st_ino", 0), + ("st_uid", -1), + ("st_mode", stat.S_IFREG | 0o600), + ), +) +def test_progress_decoder_rejects_invalid_socket_ownership_schema( + benchmark_module: types.ModuleType, + field: str, + value: object, +) -> None: + """Malformed durable socket capabilities cannot enter recovery state.""" + socket_row: dict[str, object] = { + "st_dev": 7, + "st_ino": 11, + "st_uid": os.getuid(), + "st_mode": stat.S_IFSOCK | 0o600, + } + socket_row[field] = value + row = { + "schema_version": 1, + "run_id": "run-7", + "sequence": 0, + "checkpoint": "identity.server.socket", + "monotonic_ns": 1, + "processes": [], + "socket_ownership": { + "process": {"role": "server", "pid": 2, "start_time": 3}, + "socket": socket_row, + }, + } + + with pytest.raises(ValueError, match="socket ownership"): + benchmark_module._progress_event_from_json(row) + + def test_progress_append_retries_short_writes_and_syncs_new_parent( benchmark_module: types.ModuleType, monkeypatch: pytest.MonkeyPatch, @@ -5176,10 +5936,37 @@ def test_validator_accepts_partial_active_failure_after_complete_prefix( ) -def test_validator_accepts_cancellation_between_interleaved_strategies( +def test_validator_requires_socket_capability_for_owned_server_cleanup( benchmark_module: types.ModuleType, ) -> None: - """A completed interleaved subset remains valid before the next boundary.""" + """Complete server cleanup claims require the durable paired capability.""" + owner = benchmark_module.ProcessIdentity("server", 200_001, 300_001) + report = dataclasses.replace( + _failed_report_with_completed_mutation_prefix(benchmark_module), + processes=(owner,), + ) + + with pytest.raises(ValueError, match="lacks exact socket ownership"): + benchmark_module.validate_report(report) + + ownership = benchmark_module._SocketOwnership( + owner, + benchmark_module.SocketIdentity( + 7, + 11, + os.getuid(), + stat.S_IFSOCK | 0o600, + ), + ) + benchmark_module.validate_report( + dataclasses.replace(report, socket_ownership=ownership) + ) + + +def test_validator_rejects_unreachable_completed_interleaved_subset( + benchmark_module: types.ModuleType, +) -> None: + """A unique subset is invalid when the seeded schedule cannot produce it.""" base = _failed_report_with_completed_mutation_prefix(benchmark_module) topology = base.requested_topology @@ -5239,7 +6026,8 @@ def completed_repeatable(name: str, *, row_count: int | None = None) -> t.Any: error="KeyboardInterrupt: cancelled between strategies", ) - benchmark_module.validate_report(report) + with pytest.raises(ValueError, match="reachable seeded strategy boundary"): + benchmark_module.validate_report(report) def test_validator_rejects_failed_phase_that_disagrees_with_active_row( @@ -5754,6 +6542,8 @@ def release() -> None: assert mask_calls == [ (signal.SIG_BLOCK, frozenset({signal.SIGINT, signal.SIGTERM})), (signal.SIG_SETMASK, prior_mask), + (signal.SIG_BLOCK, frozenset({signal.SIGINT, signal.SIGTERM})), + (signal.SIG_SETMASK, prior_mask), ] @@ -5807,7 +6597,7 @@ def test_finalizer_restores_mask_before_original_sigterm_handler( benchmark_module: types.ModuleType, monkeypatch: pytest.MonkeyPatch, ) -> None: - """SIGTERM during handler restoration must be recorded, not delivered to DFL.""" + """Handler restoration keeps managed signals blocked until the mask is exact.""" original_handler = signal.getsignal(signal.SIGTERM) original_mask = frozenset(signal.pthread_sigmask(signal.SIG_BLOCK, ())) interruption = KeyboardInterrupt("SIGTERM during handler restoration") @@ -5850,7 +6640,7 @@ def signal_with_pending_delivery( assert result == 29 assert observed is interruption assert raised_during_restore - assert restore_mask == original_mask + assert restore_mask == original_mask | {signal.SIGINT, signal.SIGTERM} assert restored_handler is prior_sigterm assert restored_mask == original_mask @@ -5985,7 +6775,21 @@ def test_supervisor_cancellation_during_final_report_write_is_durable( release_write = threading.Event() real_write = benchmark_module.write_json_atomic real_drain = benchmark_module._drain_finalizer_thread + real_signal = signal.signal + original_sigint = signal.getsignal(signal.SIGINT) + original_mask = frozenset(signal.pthread_sigmask(signal.SIG_BLOCK, ())) + first = KeyboardInterrupt("interrupt before artifact commit") drain_calls = 0 + observed_sigint: t.Any = None + observed_mask: frozenset[int | signal.Signals] | None = None + + def caller_sigint( + _signal_number: int, + _frame: types.FrameType | None, + ) -> t.NoReturn: + raise first + + real_signal(signal.SIGINT, caller_sigint) def counting_drain(*args: t.Any, **kwargs: t.Any) -> t.Any: nonlocal drain_calls @@ -6050,14 +6854,20 @@ def interrupt_final_write() -> None: watchdog_s=30.0, cleanup_grace_s=0.3, ) + observed_sigint = signal.getsignal(signal.SIGINT) + observed_mask = frozenset(signal.pthread_sigmask(signal.SIG_BLOCK, ())) finally: release_write.set() interrupter.join(timeout=5.0) + real_signal(signal.SIGINT, original_sigint) + signal.pthread_sigmask(signal.SIG_SETMASK, original_mask) assert not interrupter.is_alive() assert drain_calls == 1 durable = benchmark_module.load_run_report(report_path) - assert isinstance(cancellation.value, KeyboardInterrupt) + assert cancellation.value is first + assert observed_sigint is caller_sigint + assert observed_mask == original_mask assert durable.status == "cutoff" assert durable.failed_phase == "cancellation" assert durable.cleanup.complete @@ -6065,6 +6875,10 @@ def interrupt_final_write() -> None: not benchmark_module.process_identity_matches(row) for row in durable.processes ) assert "cutoff" in markdown_path.read_text(encoding="utf-8") + assert not any( + thread.name == "orchestration-finalization" and thread.is_alive() + for thread in threading.enumerate() + ) def test_supervisor_interrupt_before_popen_return_is_durable_and_reraised( @@ -6133,6 +6947,8 @@ def interrupt_popen(*args: t.Any, **kwargs: t.Any) -> t.Any: (signal.SIG_SETMASK, prior_mask), (signal.SIG_BLOCK, frozenset({signal.SIGINT, signal.SIGTERM})), (signal.SIG_SETMASK, prior_mask), + (signal.SIG_BLOCK, frozenset({signal.SIGINT, signal.SIGTERM})), + (signal.SIG_SETMASK, prior_mask), ] durable = benchmark_module.load_run_report(output) assert durable.status == "cutoff" @@ -6230,6 +7046,8 @@ def record_mask( (signal.SIG_SETMASK, prior_mask), (signal.SIG_BLOCK, frozenset({signal.SIGINT, signal.SIGTERM})), (signal.SIG_SETMASK, prior_mask), + (signal.SIG_BLOCK, frozenset({signal.SIGINT, signal.SIGTERM})), + (signal.SIG_SETMASK, prior_mask), ] assert durable is not None assert durable.status == "cutoff" @@ -6360,6 +7178,217 @@ def interrupt_recovery() -> None: assert not (tmp_path / "scratch").exists() +def test_supervisor_restores_caller_sigterm_under_repeated_interrupt( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """A restore-time interrupt cannot replace the first monitor interruption.""" + first = KeyboardInterrupt("first monitor interruption") + second = KeyboardInterrupt("SIGTERM restoration interruption") + original_handler = signal.getsignal(signal.SIGTERM) + original_mask = frozenset(signal.pthread_sigmask(signal.SIG_BLOCK, ())) + real_signal = signal.signal + real_drain_progress = benchmark_module._ProgressTracker.drain + real_drain_finalizer = benchmark_module._drain_finalizer_thread + monitor_interrupted = False + restore_interrupted = False + finalizer_calls = 0 + + def caller_sigterm( + _signal_number: int, + _frame: types.FrameType | None, + ) -> None: + return None + + real_signal(signal.SIGTERM, caller_sigterm) + + def interrupt_monitor(tracker: t.Any, *, terminal: bool = False) -> bool: + nonlocal monitor_interrupted + advanced = real_drain_progress(tracker, terminal=terminal) + if not terminal and advanced and not monitor_interrupted: + monitor_interrupted = True + raise first + return t.cast(bool, advanced) + + def interrupt_restore(signal_number: int, handler: t.Any) -> t.Any: + nonlocal restore_interrupted + if ( + signal_number == signal.SIGTERM + and handler is caller_sigterm + and not restore_interrupted + ): + restore_interrupted = True + raise second + return real_signal(signal_number, handler) + + def count_finalizer(*args: t.Any, **kwargs: t.Any) -> t.Any: + nonlocal finalizer_calls + finalizer_calls += 1 + return real_drain_finalizer(*args, **kwargs) + + monkeypatch.setattr(benchmark_module._ProgressTracker, "drain", interrupt_monitor) + monkeypatch.setattr(benchmark_module.signal, "signal", interrupt_restore) + monkeypatch.setattr(benchmark_module, "_drain_finalizer_thread", count_finalizer) + output = tmp_path / "restore-interrupt.json" + markdown = tmp_path / "restore-interrupt.md" + decision = benchmark_module.GuardDecision( + True, + "ok", + None, + None, + None, + False, + benchmark_module.HostSnapshot(), + ) + raised: KeyboardInterrupt | None = None + observed_handler: t.Any = None + observed_mask: frozenset[int | signal.Signals] | None = None + try: + with pytest.raises(KeyboardInterrupt) as cancellation: + benchmark_module.supervise_worker( + benchmark_module.Topology(1, 1, 1), + lane=benchmark_module.EngineLane.SUBPROCESS, + mode=benchmark_module.ExecutionMode.SYNC, + runs=1, + warmup=0, + seed=11, + run_id="restore-interrupt", + scratch=tmp_path / "scratch", + socket_path=tmp_path / "scratch" / "tmux.sock", + output=output, + markdown_output=markdown, + guard_decision=decision, + original_guard_decision=decision, + policy=benchmark_module.ResourcePolicy(), + watchdog_s=30.0, + cleanup_grace_s=0.3, + _test_stall_after="worker.started", + ) + raised = cancellation.value + observed_handler = signal.getsignal(signal.SIGTERM) + observed_mask = frozenset(signal.pthread_sigmask(signal.SIG_BLOCK, ())) + finally: + real_signal(signal.SIGTERM, original_handler) + signal.pthread_sigmask(signal.SIG_SETMASK, original_mask) + + durable = benchmark_module.load_run_report(output) + assert raised is first + assert restore_interrupted + assert finalizer_calls == 1 + assert observed_handler is caller_sigterm + assert observed_mask == original_mask + assert durable.status == "cutoff" + assert durable.failed_phase == "cancellation" + assert durable.cleanup.complete + assert not any( + thread.name == "orchestration-finalization" and thread.is_alive() + for thread in threading.enumerate() + ) + + +def test_supervisor_post_commit_interrupt_preserves_completed_artifact( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """An interrupt after artifact commit belongs to the restored caller.""" + interruption = KeyboardInterrupt("interrupt after artifact commit") + original_handler = signal.getsignal(signal.SIGTERM) + original_mask = frozenset(signal.pthread_sigmask(signal.SIG_BLOCK, ())) + real_signal = signal.signal + real_drain = benchmark_module._drain_finalizer_thread + output = tmp_path / "post-commit.json" + markdown = tmp_path / "post-commit.md" + restore_interrupted = False + finalizer_calls = 0 + committed_status: str | None = None + restore_mask: frozenset[int | signal.Signals] | None = None + + def caller_sigterm( + _signal_number: int, + _frame: types.FrameType | None, + ) -> None: + return None + + real_signal(signal.SIGTERM, caller_sigterm) + + def interrupt_restore(signal_number: int, handler: t.Any) -> t.Any: + nonlocal restore_interrupted, committed_status, restore_mask + if ( + signal_number == signal.SIGTERM + and handler is caller_sigterm + and not restore_interrupted + ): + restore_interrupted = True + committed_status = benchmark_module.load_run_report(output).status + assert markdown.exists() + restore_mask = frozenset(signal.pthread_sigmask(signal.SIG_BLOCK, ())) + raise interruption + return real_signal(signal_number, handler) + + def count_finalizer(*args: t.Any, **kwargs: t.Any) -> t.Any: + nonlocal finalizer_calls + finalizer_calls += 1 + return real_drain(*args, **kwargs) + + monkeypatch.setattr(benchmark_module.signal, "signal", interrupt_restore) + monkeypatch.setattr(benchmark_module, "_drain_finalizer_thread", count_finalizer) + decision = benchmark_module.GuardDecision( + True, + "ok", + None, + None, + None, + False, + benchmark_module.HostSnapshot(), + ) + raised: KeyboardInterrupt | None = None + observed_handler: t.Any = None + observed_mask: frozenset[int | signal.Signals] | None = None + try: + with pytest.raises(KeyboardInterrupt) as cancellation: + benchmark_module.supervise_worker( + benchmark_module.Topology(1, 1, 1), + lane=benchmark_module.EngineLane.SUBPROCESS, + mode=benchmark_module.ExecutionMode.SYNC, + runs=1, + warmup=0, + seed=11, + run_id="post-commit", + scratch=tmp_path / "scratch", + socket_path=tmp_path / "scratch" / "tmux.sock", + output=output, + markdown_output=markdown, + guard_decision=decision, + original_guard_decision=decision, + policy=benchmark_module.ResourcePolicy(), + watchdog_s=30.0, + cleanup_grace_s=0.3, + ) + raised = cancellation.value + observed_handler = signal.getsignal(signal.SIGTERM) + observed_mask = frozenset(signal.pthread_sigmask(signal.SIG_BLOCK, ())) + finally: + real_signal(signal.SIGTERM, original_handler) + signal.pthread_sigmask(signal.SIG_SETMASK, original_mask) + + durable = benchmark_module.load_run_report(output) + assert raised is interruption + assert committed_status == "completed" + assert restore_interrupted + assert restore_mask == original_mask | {signal.SIGINT, signal.SIGTERM} + assert finalizer_calls == 1 + assert observed_handler is caller_sigterm + assert observed_mask == original_mask + assert durable.status == "completed" + assert durable.cleanup.complete + assert not any( + thread.name == "orchestration-finalization" and thread.is_alive() + for thread in threading.enumerate() + ) + + def test_cli_process_identity_mismatch_never_signals_unrelated_pid( tmp_path: pathlib.Path, ) -> None: From 127ede6b5d3efba87eed7a0e35a1c6f992dc6eff Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 18:19:54 -0500 Subject: [PATCH 25/67] Bench(fix[runner]): Publish ownership atomically why: A durable server identity without its socket capability could leave watchdog recovery unable to clean exact resources safely. what: - Publish server identity and socket ownership in one version-2 event - Reuse retained pidfds and fail closed on replaced socket paths - Validate reachable wait boundaries and cover cleanup races --- scripts/bench_orchestration.py | 263 +++++++-- tests/test_bench_orchestration_script.py | 673 ++++++++++++++++++++++- 2 files changed, 875 insertions(+), 61 deletions(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index fbad6339e7..57e7449ab4 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -65,6 +65,8 @@ _SENTINEL_RECORD_MAX_BYTES = 422 _WAIT_TIMEOUT_MAX_S = 3.0 _WAIT_FRAME_RATE_MAX_HZ = 40.0 +_REPORT_SCHEMA_VERSION = 2 +_PROGRESS_SCHEMA_VERSION = 2 # At the maximum supported producer rate and wait duration, 5,000 joined # history rows retain 120 ordinary delayed frames plus the 422-byte sentinel # with ample wrapping headroom in the required active 1x2x2 topology. @@ -1783,10 +1785,12 @@ class SocketIdentity: User that owns the socket node. st_mode : int Complete mode captured by ``lstat``; its file type must remain a socket. + st_mtime_ns : int + Captured nanosecond modification timestamp used with device and inode. Examples -------- - >>> identity = SocketIdentity(1, 2, os.getuid(), stat.S_IFSOCK | 0o600) + >>> identity = SocketIdentity(1, 2, os.getuid(), stat.S_IFSOCK | 0o600, 3) >>> stat.S_ISSOCK(identity.st_mode) True """ @@ -1795,6 +1799,7 @@ class SocketIdentity: st_ino: int st_uid: int st_mode: int + st_mtime_ns: int @dataclasses.dataclass(frozen=True) @@ -1811,7 +1816,7 @@ class _SocketOwnership: Examples -------- >>> owner = ProcessIdentity("server", 2, 3) - >>> socket_id = SocketIdentity(1, 2, os.getuid(), stat.S_IFSOCK | 0o600) + >>> socket_id = SocketIdentity(1, 2, os.getuid(), stat.S_IFSOCK | 0o600, 4) >>> _SocketOwnership(owner, socket_id).process is owner True """ @@ -1853,7 +1858,7 @@ class ProgressEvent: monotonic_ns: int processes: tuple[ProcessIdentity, ...] = () socket_ownership: _SocketOwnership | None = None - schema_version: int = 1 + schema_version: int = _PROGRESS_SCHEMA_VERSION @dataclasses.dataclass(frozen=True) @@ -2242,7 +2247,7 @@ class RunReport: ramp_kind: t.Literal["none", "canonical", "custom"] = "none" guard_decision: GuardDecision | None = None original_guard_decision: GuardDecision | None = None - schema_version: int = 1 + schema_version: int = _REPORT_SCHEMA_VERSION run_id: str | None = None lane: t.Literal["subprocess", "control"] | None = None mode: t.Literal["sync", "async"] | None = None @@ -2603,6 +2608,7 @@ def _socket_identity_from_json(value: object) -> SocketIdentity: >>> _socket_identity_from_json({ ... "st_dev": 1, "st_ino": 2, "st_uid": 3, ... "st_mode": stat.S_IFSOCK | 0o600, + ... "st_mtime_ns": 4, ... }).st_ino 2 """ @@ -2612,6 +2618,7 @@ def _socket_identity_from_json(value: object) -> SocketIdentity: st_ino=t.cast(int, row.get("st_ino")), st_uid=t.cast(int, row.get("st_uid")), st_mode=t.cast(int, row.get("st_mode")), + st_mtime_ns=t.cast(int, row.get("st_mtime_ns")), ) @@ -2635,6 +2642,12 @@ def _socket_ownership_from_json(value: object | None) -> _SocketOwnership | None def _validate_socket_ownership(ownership: _SocketOwnership) -> None: """Reject a capability that cannot identify one same-user tmux socket. + Examples + -------- + >>> owner = ProcessIdentity("server", 2, 3) + >>> socket_id = SocketIdentity(1, 2, os.getuid(), stat.S_IFSOCK | 0o600, 4) + >>> _validate_socket_ownership(_SocketOwnership(owner, socket_id)) + Parameters ---------- ownership : _SocketOwnership @@ -2666,6 +2679,8 @@ def _validate_socket_ownership(ownership: _SocketOwnership) -> None: or socket_identity.st_uid != os.getuid() or type(socket_identity.st_mode) is not int or not stat.S_ISSOCK(socket_identity.st_mode) + or type(socket_identity.st_mtime_ns) is not int + or socket_identity.st_mtime_ns <= 0 ): message = "invalid socket ownership capability" raise ValueError(message) @@ -2694,6 +2709,7 @@ def run_report_from_json(value: object) -> RunReport: """Decode a JSON-native report into immutable typed evidence. >>> report = run_report_from_json({ + ... "schema_version": 2, ... "requested_topology": { ... "sessions": 1, "windows_per_session": 1, "panes_per_window": 1 ... }, @@ -2713,6 +2729,9 @@ def run_report_from_json(value: object) -> RunReport: Immutable artifact suitable for :func:`validate_report`. """ row = _json_mapping(value, "run report") + if row.get("schema_version") != _REPORT_SCHEMA_VERSION: + message = "unsupported report schema_version" + raise ValueError(message) observed = row.get("observed_topology") ramp_rows = [] for item in row.get("ramp", []): @@ -2753,7 +2772,7 @@ def run_report_from_json(value: object) -> RunReport: ramp_kind=row.get("ramp_kind", "none"), guard_decision=_guard_from_json(row.get("guard_decision")), original_guard_decision=_guard_from_json(row.get("original_guard_decision")), - schema_version=row.get("schema_version", 1), + schema_version=t.cast(int, row.get("schema_version")), run_id=row.get("run_id"), lane=row.get("lane"), mode=row.get("mode"), @@ -2804,7 +2823,7 @@ def validate_report(report: RunReport) -> None: ValueError If a discriminator, phase, cleanup, ramp, or maximum claim is inconsistent. """ - if report.schema_version != 1: + if report.schema_version != _REPORT_SCHEMA_VERSION: message = "unsupported report schema_version" raise ValueError(message) report_statuses = {"in_progress", "completed", "refused", "failed", "cutoff"} @@ -3161,7 +3180,29 @@ def _interleaved_suffix_is_reachable( group_index: int, suffix: tuple[PhaseReport, ...], ) -> bool: - """Return whether an executable suffix occurs in its exact seeded schedule.""" + """Return whether an executable suffix occurs in its exact seeded schedule. + + The non-async-control wait group ends with one synthetic disposition after + capture polling completes. + + >>> topology = Topology(1, 1, 1) + >>> capture = PhaseReport( + ... "wait.capture-poll", topology, topology, + ... samples=(RawSample(1, True),), status="completed", runs=1, + ... ) + >>> disposition = PhaseReport( + ... "wait.control-stream", topology, topology, + ... status="not_applicable", runs=1, + ... ) + >>> report = RunReport( + ... topology, lane="subprocess", mode="sync", warmup=0, runs=1, + ... environment=EnvironmentReport("3.10", None, 1, 11, ("run",), None), + ... ) + >>> _interleaved_suffix_is_reachable( + ... report, 3, (capture, disposition) + ... ) + True + """ assert report.environment is not None assert report.warmup is not None assert report.runs is not None @@ -3172,6 +3213,21 @@ def _interleaved_suffix_is_reachable( and report.mode == ExecutionMode.ASYNC.value ): strategy_names = ("wait.capture-poll",) + if suffix and suffix[-1].name == "wait.control-stream": + disposition = suffix[-1] + if ( + disposition.status != "not_applicable" + or disposition.requested_topology != report.requested_topology + or disposition.observed_topology != report.requested_topology + or disposition.warmup != report.warmup + or disposition.runs != report.runs + or disposition.samples + or disposition.summary is not None + or disposition.warmup_observations + or disposition.observations + ): + return False + suffix = suffix[:-1] else: strategy_names = group observed = tuple( @@ -4246,6 +4302,7 @@ def _socket_identity(socket_path: pathlib.Path) -> SocketIdentity: st_ino=status.st_ino, st_uid=status.st_uid, st_mode=status.st_mode, + st_mtime_ns=status.st_mtime_ns, ) @@ -4256,6 +4313,22 @@ def _query_exact_socket_owner( ) -> ProcessIdentity: """Boundedly query one exact tmux socket for its current server identity. + Examples + -------- + >>> with tempfile.TemporaryDirectory() as directory: + ... path = pathlib.Path(directory) / "tmux.sock" + ... _ = subprocess.run( + ... ("tmux", "-S", str(path), "new-session", "-d", "-s", "query"), + ... check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + ... ) + ... identity = _query_exact_socket_owner(path, timeout_s=1.0) + ... _ = subprocess.run( + ... ("tmux", "-S", str(path), "kill-server"), check=True, + ... stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + ... ) + ... identity.role + 'server' + Parameters ---------- socket_path : pathlib.Path @@ -4326,6 +4399,22 @@ def _capture_socket_ownership( The socket lives in a mode-0700 run directory. Two ``lstat`` calls bind the read-only tmux owner query to one inode before the capability is published. + Examples + -------- + >>> with tempfile.TemporaryDirectory() as directory: + ... path = pathlib.Path(directory) / "tmux.sock" + ... _ = subprocess.run( + ... ("tmux", "-S", str(path), "new-session", "-d", "-s", "capture"), + ... check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + ... ) + ... ownership = _capture_socket_ownership(path, timeout_s=1.0) + ... _ = subprocess.run( + ... ("tmux", "-S", str(path), "kill-server"), check=True, + ... stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + ... ) + ... ownership.process.role, stat.S_ISSOCK(ownership.socket.st_mode) + ('server', True) + Parameters ---------- socket_path : pathlib.Path @@ -4359,12 +4448,19 @@ def _kill_exact_tmux_socket( socket_path: pathlib.Path, *, timeout_s: float, + process_handles: _PidfdRegistry, + server_identity: ProcessIdentity | None = None, socket_ownership: _SocketOwnership | None = None, ) -> tuple[str, ...]: """Boundedly clean only the server/socket pair captured at establishment. - >>> _kill_exact_tmux_socket(pathlib.Path("missing.sock"), timeout_s=0.01) + >>> registry = _PidfdRegistry() + >>> _kill_exact_tmux_socket( + ... pathlib.Path("missing.sock"), timeout_s=0.01, + ... process_handles=registry, + ... ) () + >>> registry.close() An observed pathname mismatch fails closed. The original process is killed only through its retained pidfd, while the replacement inode is preserved. @@ -4377,6 +4473,10 @@ def _kill_exact_tmux_socket( Exact isolated socket owned by one benchmark run. timeout_s : float Maximum wait for the helper and each stable-handle escalation. + process_handles : _PidfdRegistry + Existing registry that retained the server when its identity was learned. + server_identity : ProcessIdentity | None + Exact original server identity learned from process progress. socket_ownership : _SocketOwnership | None Capability captured while the owned tmux server established the socket. @@ -4393,22 +4493,20 @@ def _kill_exact_tmux_socket( if timeout_s <= 0: message = "exact socket cleanup timeout must be positive" raise ValueError(message) - if socket_ownership is None: + owner = server_identity + if owner is None and socket_ownership is not None: + owner = socket_ownership.process + errors: list[str] = [] + if owner is None: return ( () if not socket_path.exists() else (f"exact socket ownership capability unavailable: {socket_path}",) ) - owner = socket_ownership.process - errors: list[str] = [] - registry = _PidfdRegistry() - original_live = process_identity_matches(owner) - retained = registry.retain(owner) - - def finish() -> tuple[str, ...]: - errors.extend(registry.errors) - registry.close() - return tuple(errors) + if socket_ownership is not None and socket_ownership.process != owner: + errors.append("socket ownership process differs from recorded server") + socket_ownership = None + retained = owner in process_handles.retained def kill_original_without_path() -> None: if not process_identity_matches(owner): @@ -4419,44 +4517,57 @@ def kill_original_without_path() -> None: "pathname cleanup refused" ) return - registry.signal(owner, signal.SIGKILL) + process_handles.signal(owner, signal.SIGTERM) survivors = _wait_identity_absence((owner,), timeout_s=timeout_s) + if survivors: + process_handles.signal(owner, signal.SIGKILL) + survivors = _wait_identity_absence((owner,), timeout_s=timeout_s) if survivors: errors.append( f"server pid {owner.pid} with start time {owner.start_time} remains" ) - if original_live and not retained: + if process_identity_matches(owner) and not retained: errors.append( f"server pid {owner.pid} stable handle unavailable; " "pathname cleanup refused" ) - return finish() + return tuple(errors) + if socket_ownership is None: + kill_original_without_path() + if socket_path.exists(): + errors.append( + f"exact socket ownership capability unavailable: {socket_path}" + ) + return tuple(errors) if not socket_path.exists(): kill_original_without_path() - return finish() + return tuple(errors) try: _verify_private_directory_mode(socket_path.parent) before = _socket_identity(socket_path) except OSError as error: errors.append(f"exact socket ownership: {type(error).__name__}: {error}") kill_original_without_path() - return finish() + return tuple(errors) if before != socket_ownership.socket: errors.append(f"configured socket ownership changed: {socket_path}") kill_original_without_path() - return finish() + return tuple(errors) + if not process_identity_matches(owner): + errors.extend(_remove_proven_stale_socket(socket_path, socket_ownership)) + return tuple(errors) try: current_owner = _query_exact_socket_owner(socket_path, timeout_s=timeout_s) after = _socket_identity(socket_path) except (OSError, RuntimeError) as error: errors.append(f"exact socket ownership query: {type(error).__name__}: {error}") kill_original_without_path() - return finish() + return tuple(errors) if after != socket_ownership.socket or current_owner != owner: errors.append(f"configured socket owner changed: {socket_path}") kill_original_without_path() - return finish() + return tuple(errors) environment = os.environ.copy() environment.pop("TMUX", None) @@ -4512,22 +4623,18 @@ def kill_original_without_path() -> None: elif helper.returncode != 0: errors.append(f"exact socket kill-server exited {helper.returncode}") - try: + survivors = _wait_identity_absence((owner,), timeout_s=timeout_s) + for signal_number in (signal.SIGTERM, signal.SIGKILL): + if not survivors: + break + process_handles.signal(owner, signal_number) survivors = _wait_identity_absence((owner,), timeout_s=timeout_s) - for signal_number in (signal.SIGTERM, signal.SIGKILL): - if not survivors: - break - registry.signal(owner, signal_number) - survivors = _wait_identity_absence((owner,), timeout_s=timeout_s) - if survivors: - errors.append( - f"server pid {owner.pid} with start time {owner.start_time} remains" - ) - elif socket_path.exists(): - errors.extend(_remove_proven_stale_socket(socket_path, socket_ownership)) - errors.extend(registry.errors) - finally: - registry.close() + if survivors: + errors.append( + f"server pid {owner.pid} with start time {owner.start_time} remains" + ) + elif socket_path.exists(): + errors.extend(_remove_proven_stale_socket(socket_path, socket_ownership)) return tuple(errors) @@ -4566,6 +4673,7 @@ def _remove_proven_stale_socket( if process_identity_matches(socket_ownership.process): return (f"socket owner is not proven absent: {socket_path}",) try: + _verify_private_directory_mode(socket_path.parent) after = _socket_identity(socket_path) if after != socket_ownership.socket or process_identity_matches( socket_ownership.process @@ -5346,10 +5454,12 @@ def setup_sync( server_identity = socket_ownership.process context.socket_ownership = socket_ownership context.processes = (*context.processes, server_identity) - if context.process_identity_callback is not None: - context.process_identity_callback((server_identity,)) if _socket_ownership_callback is not None: + assert context.process_handles is not None + context.process_handles.retain(server_identity) _socket_ownership_callback(socket_ownership) + elif context.process_identity_callback is not None: + context.process_identity_callback((server_identity,)) if lane is EngineLane.CONTROL: run(ListSessions(), engine).raise_for_status() @@ -5517,10 +5627,12 @@ async def setup_async( server_identity = socket_ownership.process context.socket_ownership = socket_ownership context.processes = (*context.processes, server_identity) - if context.process_identity_callback is not None: - context.process_identity_callback((server_identity,)) if _socket_ownership_callback is not None: + assert context.process_handles is not None + context.process_handles.retain(server_identity) _socket_ownership_callback(socket_ownership) + elif context.process_identity_callback is not None: + context.process_identity_callback((server_identity,)) if lane is EngineLane.CONTROL: control = t.cast(AsyncControlModeEngine, engine) await control.start() @@ -9184,6 +9296,15 @@ async def cleanup_run( _kill_exact_tmux_socket( context.socket_path, timeout_s=grace_s, + process_handles=registry, + server_identity=next( + ( + identity + for identity in context.processes + if identity.role == "server" + ), + None, + ), socket_ownership=context.socket_ownership, ) ) @@ -9597,7 +9718,7 @@ def _progress_event_from_json(value: object) -> ProgressEvent: """Decode and validate one complete progress line. >>> _progress_event_from_json({ - ... "schema_version": 1, "run_id": "run-7", "sequence": 0, + ... "schema_version": 2, "run_id": "run-7", "sequence": 0, ... "checkpoint": "start", "monotonic_ns": 1, "processes": [], ... }).checkpoint 'start' @@ -9610,10 +9731,10 @@ def _progress_event_from_json(value: object) -> ProgressEvent: monotonic_ns=t.cast(int, row.get("monotonic_ns")), processes=tuple(_identity_from_json(item) for item in row.get("processes", [])), socket_ownership=_socket_ownership_from_json(row.get("socket_ownership")), - schema_version=t.cast(int, row.get("schema_version", 1)), + schema_version=t.cast(int, row.get("schema_version")), ) if ( - event.schema_version != 1 + event.schema_version != _PROGRESS_SCHEMA_VERSION or not _is_terminal_safe_component(event.run_id) or type(event.sequence) is not int or event.sequence < 0 @@ -9623,6 +9744,15 @@ def _progress_event_from_json(value: object) -> ProgressEvent: ): message = "invalid progress event" raise ValueError(message) + server_delta = tuple( + identity for identity in event.processes if identity.role == "server" + ) + if (event.socket_ownership is None and server_delta) or ( + event.socket_ownership is not None + and server_delta != (event.socket_ownership.process,) + ): + message = "progress event lacks atomic server socket ownership" + raise ValueError(message) return event @@ -9765,6 +9895,24 @@ def record_identities( def record_socket_ownership(self, ownership: _SocketOwnership) -> None: """Durably publish the immutable server/socket capability once. + Examples + -------- + >>> with tempfile.TemporaryDirectory() as directory: + ... root = pathlib.Path(directory) + ... owner = _record_process("server", os.getpid()) + ... socket_id = SocketIdentity( + ... 1, 2, os.getuid(), stat.S_IFSOCK | 0o600, 4 + ... ) + ... recorder = _WorkerRecorder( + ... RunReport(Topology(1, 1, 1), run_id="run-7"), + ... root / "report.json", root / "progress.jsonl", + ... ) + ... recorder.record_socket_ownership( + ... _SocketOwnership(owner, socket_id) + ... ) + ... load_run_report(root / "report.json").processes == (owner,) + True + Parameters ---------- ownership : _SocketOwnership @@ -9775,18 +9923,30 @@ def record_socket_ownership(self, ownership: _SocketOwnership) -> None: RuntimeError If a different capability was already recorded for this run. """ + _validate_socket_ownership(ownership) existing = self.report.socket_ownership if existing is not None and existing != ownership: message = "worker socket ownership changed after establishment" raise RuntimeError(message) if existing is not None: return + owner = ownership.process + known_servers = tuple( + identity for identity in self.identities if identity.role == "server" + ) + if known_servers and known_servers != (owner,): + message = "worker server identity changed before socket establishment" + raise RuntimeError(message) + if owner not in self.identities: + self.identities.append(owner) self.report = dataclasses.replace( self.report, + processes=_merge_identities(self.report.processes, (owner,)), socket_ownership=ownership, ) self.checkpoint( "identity.server.socket", + identity_delta=(owner,), socket_ownership=ownership, ) @@ -10886,6 +11046,11 @@ def wait_worker() -> bool: _kill_exact_tmux_socket( socket_path, timeout_s=grace_s, + process_handles=progress.handles, + server_identity=next( + (identity for identity in owned if identity.role == "server"), + None, + ), socket_ownership=progress.socket_ownership, ) ) diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index 7497baaf3a..132a1403fa 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -391,7 +391,7 @@ def test_write_json_atomic_replaces_complete_report( benchmark_module.write_json_atomic(report_path, report) payload = json.loads(report_path.read_text(encoding="utf-8")) - assert payload["schema_version"] == 1 + assert payload["schema_version"] == 2 assert payload["status"] == "completed" assert payload["requested_topology"] == { "sessions": 100, @@ -400,6 +400,49 @@ def test_write_json_atomic_replaces_complete_report( } +@pytest.mark.parametrize("schema_version", (None, 1)) +def test_report_decoder_rejects_old_or_missing_ownership_schema( + benchmark_module: types.ModuleType, + schema_version: int | None, +) -> None: + """Evidence without report ownership semantics must not decode as current.""" + row = t.cast( + dict[str, object], + benchmark_module._json_value( + benchmark_module.RunReport(benchmark_module.Topology(1, 1, 1)) + ), + ) + if schema_version is None: + del row["schema_version"] + else: + row["schema_version"] = schema_version + + with pytest.raises(ValueError, match="report schema_version"): + benchmark_module.run_report_from_json(row) + + +@pytest.mark.parametrize("schema_version", (None, 1)) +def test_progress_decoder_rejects_old_or_missing_ownership_schema( + benchmark_module: types.ModuleType, + schema_version: int | None, +) -> None: + """Evidence without progress ownership semantics must not enter recovery.""" + row: dict[str, object] = { + "schema_version": schema_version, + "run_id": "run-7", + "sequence": 0, + "checkpoint": "worker.started", + "monotonic_ns": 1, + "processes": [], + "socket_ownership": None, + } + if schema_version is None: + del row["schema_version"] + + with pytest.raises(ValueError, match="progress event"): + benchmark_module._progress_event_from_json(row) + + def test_write_json_atomic_syncs_file_then_parent( benchmark_module: types.ModuleType, tmp_path: pathlib.Path ) -> None: @@ -1312,10 +1355,13 @@ def unlink_without_killing_server( monkeypatch.setattr( benchmark_module.subprocess, "Popen", unlink_without_killing_server ) + registry = benchmark_module._PidfdRegistry() + assert registry.retain(ownerships[target].process) try: errors = benchmark_module._kill_exact_tmux_socket( target, timeout_s=1.0, + process_handles=registry, socket_ownership=ownerships[target], ) @@ -1331,6 +1377,7 @@ def unlink_without_killing_server( ) assert still_alive.returncode == 0, still_alive.stderr finally: + registry.close() for socket_path in (target, unrelated): real_popen( ("tmux", "-S", str(socket_path), "kill-server"), @@ -1425,6 +1472,17 @@ def test_cleanup_preserves_live_replacement_after_original_server_exits( "replacement", int(replacement_pid.stdout.strip()) ) replacement_status = context.socket_path.lstat() + assert context.socket_ownership is not None + captured_socket = context.socket_ownership.socket + assert ( + replacement_status.st_dev, + replacement_status.st_ino, + replacement_status.st_mtime_ns, + ) != ( + captured_socket.st_dev, + captured_socket.st_ino, + captured_socket.st_mtime_ns, + ) cleanup = asyncio.run(benchmark_module.cleanup_run(context, grace_s=0.3)) @@ -1470,6 +1528,56 @@ def test_cleanup_preserves_live_replacement_after_original_server_exits( benchmark_module._remove_supervised_scratch(scratch) +def test_exact_socket_cleanup_preserves_reused_inode_number( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """A replacement socket may reuse the original inode number. + + The standard tmux fixtures cannot deterministically force filesystem inode + reuse, so this probe keeps device, inode, owner, and mode equal while + varying only the captured modification timestamp as inode-reuse evidence. + """ + private = tmp_path / "reused-inode-number" + private.mkdir(mode=0o700) + socket_path = private / "tmux.sock" + replacement = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + replacement.bind(str(socket_path)) + status = socket_path.lstat() + ownership = benchmark_module._SocketOwnership( + process=benchmark_module.ProcessIdentity("server", 2**31 - 1, 1), + socket=benchmark_module.SocketIdentity( + st_dev=status.st_dev, + st_ino=status.st_ino, + st_uid=status.st_uid, + st_mode=status.st_mode, + st_mtime_ns=status.st_mtime_ns - 1, + ), + ) + registry = benchmark_module._PidfdRegistry() + try: + errors = benchmark_module._kill_exact_tmux_socket( + socket_path, + timeout_s=0.1, + process_handles=registry, + server_identity=ownership.process, + socket_ownership=ownership, + ) + + observed = socket_path.lstat() + assert errors + assert any("ownership changed" in error for error in errors) + assert (observed.st_dev, observed.st_ino, observed.st_mtime_ns) == ( + status.st_dev, + status.st_ino, + status.st_mtime_ns, + ) + finally: + registry.close() + replacement.close() + socket_path.unlink(missing_ok=True) + + def test_cleanup_kills_live_original_without_touching_replacement_path( benchmark_module: types.ModuleType, tmp_path: pathlib.Path, @@ -1646,10 +1754,12 @@ def test_exact_socket_cleanup_without_capability_preserves_path( node = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) node.bind(str(socket_path)) before = socket_path.lstat() + registry = benchmark_module._PidfdRegistry() try: errors = benchmark_module._kill_exact_tmux_socket( socket_path, timeout_s=0.1, + process_handles=registry, ) after = socket_path.lstat() @@ -1657,6 +1767,7 @@ def test_exact_socket_cleanup_without_capability_preserves_path( assert "capability unavailable" in errors[0] assert (after.st_dev, after.st_ino) == (before.st_dev, before.st_ino) finally: + registry.close() node.close() socket_path.unlink(missing_ok=True) @@ -1693,9 +1804,9 @@ def test_exact_socket_cleanup_retain_failure_never_uses_pathname( before = socket_path.lstat() pathname_calls = 0 - def refuse_owner_handle(current: t.Any, identity: t.Any) -> bool: + def refuse_owner_handle(identity: t.Any) -> bool: if identity == ownership.process: - current.errors.append("injected pidfd retain failure") + registry.errors.append("injected pidfd retain failure") return False pytest.fail("socket cleanup retained an unexpected identity") @@ -1704,11 +1815,9 @@ def reject_pathname(*_args: t.Any, **_kwargs: t.Any) -> t.NoReturn: pathname_calls += 1 pytest.fail("retain failure attempted pathname kill-server") - monkeypatch.setattr( - benchmark_module._PidfdRegistry, - "retain", - refuse_owner_handle, - ) + registry = benchmark_module._PidfdRegistry() + monkeypatch.setattr(registry, "retain", refuse_owner_handle) + assert not registry.retain(ownership.process) monkeypatch.setattr( benchmark_module, "subprocess", @@ -1718,6 +1827,8 @@ def reject_pathname(*_args: t.Any, **_kwargs: t.Any) -> t.NoReturn: errors = benchmark_module._kill_exact_tmux_socket( socket_path, timeout_s=0.1, + process_handles=registry, + server_identity=ownership.process, socket_ownership=ownership, ) @@ -1736,6 +1847,7 @@ def reject_pathname(*_args: t.Any, **_kwargs: t.Any) -> t.NoReturn: ) assert answering.returncode == 0, answering.stderr finally: + registry.close() subprocess.run( ("tmux", "-S", str(socket_path), "kill-server"), check=False, @@ -1749,6 +1861,229 @@ def reject_pathname(*_args: t.Any, **_kwargs: t.Any) -> t.NoReturn: socket_path.unlink(missing_ok=True) +def test_exact_socket_cleanup_reuses_previously_retained_server_handle( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """Cleanup must not open a second pidfd after setup retained the server.""" + scratch = tmp_path / "retained-server-handle" + context = benchmark_module.setup_sync( + benchmark_module.Topology(1, 1, 1), + benchmark_module.EngineLane.SUBPROCESS, + scratch, + socket_path=scratch / "tmux.sock", + run_id="retained-server-handle", + ) + server_identity = next( + identity for identity in context.processes if identity.role == "server" + ) + assert server_identity in context.process_handles.retained + + loop = asyncio.new_event_loop() + real_pidfd_open = benchmark_module.os.pidfd_open + + def reject_new_server_handle(pid: int, flags: int) -> int: + if pid == server_identity.pid: + pytest.fail("cleanup attempted to acquire a second server pidfd") + return t.cast(int, real_pidfd_open(pid, flags)) + + try: + monkeypatch.setattr( + benchmark_module.os, + "pidfd_open", + reject_new_server_handle, + ) + cleanup = loop.run_until_complete( + benchmark_module.cleanup_run(context, grace_s=0.3) + ) + finally: + loop.close() + monkeypatch.setattr(benchmark_module.os, "pidfd_open", real_pidfd_open) + if scratch.exists(): + rescue = asyncio.run(benchmark_module.cleanup_run(context, grace_s=0.3)) + assert rescue.complete, rescue.errors + + assert cleanup.complete, cleanup.errors + assert not benchmark_module.process_identity_matches(server_identity) + + +def test_missing_socket_capability_escalates_retained_server_without_pathname( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """Missing socket authority still permits TERM/KILL through the retained pidfd.""" + private = tmp_path / "missing-socket-capability" + private.mkdir(mode=0o700) + socket_path = private / "tmux.sock" + node = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + node.bind(str(socket_path)) + child = subprocess.Popen( + ( + sys.executable, + "-c", + ( + "import signal,time; " + "signal.signal(signal.SIGTERM, signal.SIG_IGN); " + "print('ready', flush=True); time.sleep(60)" + ), + ), + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, + text=True, + ) + assert child.stdout is not None + assert child.stdout.readline() == "ready\n" + identity = benchmark_module._record_process("server", child.pid) + registry = benchmark_module._PidfdRegistry() + assert registry.retain(identity) + signals: list[signal.Signals] = [] + real_signal = registry.signal + + def tracked_signal(current: t.Any, number: signal.Signals) -> bool: + signals.append(number) + return t.cast(bool, real_signal(current, number)) + + monkeypatch.setattr(registry, "signal", tracked_signal) + + def reject_pathname(*_args: t.Any, **_kwargs: t.Any) -> t.NoReturn: + pytest.fail("missing capability contacted the socket pathname") + + monkeypatch.setattr(benchmark_module.subprocess, "Popen", reject_pathname) + reaper = threading.Thread(target=child.wait) + reaper.start() + try: + errors = benchmark_module._kill_exact_tmux_socket( + socket_path, + timeout_s=0.1, + server_identity=identity, + socket_ownership=None, + process_handles=registry, + ) + finally: + if child.poll() is None: + os.kill(child.pid, signal.SIGKILL) + reaper.join(timeout=2.0) + registry.close() + node.close() + socket_path.unlink(missing_ok=True) + + assert errors + assert any("capability unavailable" in error for error in errors) + assert signals == [signal.SIGTERM, signal.SIGKILL] + assert not reaper.is_alive() + + +def test_exact_stale_tmux_socket_is_removed_after_server_absence( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """The captured inode may be unlinked after its exact tmux owner is absent.""" + private = tmp_path / "stale-original" + private.mkdir(mode=0o700) + socket_path = private / "tmux.sock" + created = subprocess.run( + ("tmux", "-S", str(socket_path), "new-session", "-d", "-s", "owned"), + check=False, + capture_output=True, + text=True, + timeout=2.0, + ) + assert created.returncode == 0, created.stderr + ownership = benchmark_module._capture_socket_ownership(socket_path, timeout_s=1.0) + registry = benchmark_module._PidfdRegistry() + assert registry.retain(ownership.process) + try: + assert registry.signal(ownership.process, signal.SIGKILL) + assert not benchmark_module._wait_identity_absence( + (ownership.process,), timeout_s=2.0 + ) + assert socket_path.exists() + + errors = benchmark_module._kill_exact_tmux_socket( + socket_path, + timeout_s=0.2, + server_identity=ownership.process, + socket_ownership=ownership, + process_handles=registry, + ) + + assert errors == () + assert not socket_path.exists() + finally: + if benchmark_module.process_identity_matches(ownership.process): + registry.signal(ownership.process, signal.SIGKILL) + benchmark_module._wait_identity_absence((ownership.process,), timeout_s=2.0) + registry.close() + socket_path.unlink(missing_ok=True) + + +def test_stale_socket_removal_rechecks_directory_before_unlink( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """A deterministic replacement during the final recheck must be preserved.""" + private = tmp_path / "stale-recheck" + private.mkdir(mode=0o700) + socket_path = private / "tmux.sock" + replacement_source = private / "replacement.sock" + original = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + original.bind(str(socket_path)) + replacement = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + replacement.bind(str(replacement_source)) + status = socket_path.lstat() + replacement_status = replacement_source.lstat() + assert (replacement_status.st_dev, replacement_status.st_ino) != ( + status.st_dev, + status.st_ino, + ) + ownership = benchmark_module._SocketOwnership( + benchmark_module.ProcessIdentity("server", os.getpid(), 1), + benchmark_module.SocketIdentity( + status.st_dev, + status.st_ino, + status.st_uid, + status.st_mode, + status.st_mtime_ns, + ), + ) + checks = 0 + real_verify = benchmark_module._verify_private_directory_mode + + def replace_before_final_recheck(path: pathlib.Path) -> None: + nonlocal checks + checks += 1 + real_verify(path) + if checks == 2: + original.close() + socket_path.unlink() + os.link(replacement_source, socket_path) + + monkeypatch.setattr( + benchmark_module, + "_verify_private_directory_mode", + replace_before_final_recheck, + ) + try: + errors = benchmark_module._remove_proven_stale_socket( + socket_path, + ownership, + ) + + assert checks == 2 + assert errors + assert "ownership changed" in errors[0] + assert socket_path.exists() + finally: + original.close() + replacement.close() + socket_path.unlink(missing_ok=True) + replacement_source.unlink(missing_ok=True) + + def test_run_scenario_refuses_when_pidfds_are_unavailable( benchmark_module: types.ModuleType, monkeypatch: pytest.MonkeyPatch, @@ -2445,6 +2780,62 @@ async def exercise_async() -> None: assert os.environ["TMUX_PANE"] == "%ambient" +@pytest.mark.parametrize("mode_name", ("sync", "async")) +def test_control_socket_mtime_stays_captured_through_start_and_close( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, + mode_name: str, +) -> None: + """Control startup and close preserve the captured modification timestamp.""" + topology = benchmark_module.Topology(1, 1, 1) + scratch = tmp_path / f"control-mtime-{mode_name}" + socket_path = scratch / "tmux.sock" + context = None + cleanup = None + + async def exercise_async() -> None: + nonlocal context, cleanup + context = await benchmark_module.setup_async( + topology, + benchmark_module.EngineLane.CONTROL, + scratch, + socket_path=socket_path, + run_id=f"control-mtime-{mode_name}", + ) + try: + assert context.socket_ownership is not None + captured_mtime = context.socket_ownership.socket.st_mtime_ns + assert socket_path.lstat().st_mtime_ns == captured_mtime + await context.engine.aclose() + assert socket_path.lstat().st_mtime_ns == captured_mtime + finally: + cleanup = await benchmark_module.cleanup_run(context) + + if mode_name == "async": + asyncio.run(exercise_async()) + else: + context = benchmark_module.setup_sync( + topology, + benchmark_module.EngineLane.CONTROL, + scratch, + socket_path=socket_path, + run_id=f"control-mtime-{mode_name}", + ) + try: + assert context.socket_ownership is not None + captured_mtime = context.socket_ownership.socket.st_mtime_ns + assert socket_path.lstat().st_mtime_ns == captured_mtime + context.engine.close() + assert socket_path.lstat().st_mtime_ns == captured_mtime + finally: + cleanup = asyncio.run(benchmark_module.cleanup_run(context)) + + assert cleanup is not None + assert cleanup.complete, cleanup.errors + assert not socket_path.exists() + assert not scratch.exists() + + def _checksum_ids(ids: tuple[str, ...]) -> str: """Derive the test oracle without calling the benchmark helper.""" return hashlib.sha256("\0".join(ids).encode()).hexdigest() @@ -5055,6 +5446,58 @@ def test_cli_watchdog_recovers_stalled_worker_and_exact_resources( _assert_terminal_cleanup(payload) +def test_cli_watchdog_recovers_stall_after_atomic_server_ownership( + tmp_path: pathlib.Path, +) -> None: + """The first durable server identity must include its socket capability.""" + report_path = tmp_path / "atomic-server.json" + + completed = _run_cli( + "run", + "--shape", + "1x1x1", + "--runs", + "1", + "--warmup", + "0", + "--output", + str(report_path), + "--scratch-root", + str(tmp_path / "scratch"), + "--watchdog-seconds", + "0.3", + "--cleanup-grace-seconds", + "0.3", + "--_test-stall-after", + "identity.server.socket", + cwd=tmp_path, + ) + + assert completed.returncode != 0 + payload = json.loads(report_path.read_text(encoding="utf-8")) + events = tuple( + json.loads(line) + for line in pathlib.Path(payload["progress_path"]) + .read_text(encoding="utf-8") + .splitlines() + ) + server_events = tuple( + event + for event in events + if any(process["role"] == "server" for process in event["processes"]) + ) + assert len(server_events) == 1 + assert ( + server_events[0]["socket_ownership"]["process"] in server_events[0]["processes"] + ) + assert payload["socket_ownership"] == server_events[0]["socket_ownership"] + assert not any( + event["checkpoint"] == "identity.server" and event["socket_ownership"] is None + for event in events + ) + _assert_terminal_cleanup(payload) + + def test_watchdog_progress_requires_an_increasing_sequence( benchmark_module: types.ModuleType, ) -> None: @@ -5131,6 +5574,7 @@ def test_socket_ownership_round_trips_and_rejects_progress_mutation( 11, os.getuid(), stat.S_IFSOCK | 0o600, + 13, ), ) recorder = benchmark_module._WorkerRecorder( @@ -5142,7 +5586,6 @@ def test_socket_ownership_round_trips_and_rejects_progress_mutation( progress, ) - recorder.record_identities((owner,), checkpoint="identity.server") recorder.record_socket_ownership(ownership) loaded = benchmark_module.load_run_report(checkpoint) @@ -5162,11 +5605,15 @@ def test_socket_ownership_round_trips_and_rejects_progress_mutation( "st_ino": 11, "st_uid": os.getuid(), "st_mode": stat.S_IFSOCK | 0o600, + "st_mtime_ns": 13, }, } + assert loaded.processes == (owner,) assert loaded.socket_ownership == ownership assert payload["socket_ownership"] == expected - assert events[-1].socket_ownership == ownership + assert len(events) == 1 + assert events[0].processes == (owner,) + assert events[0].socket_ownership == ownership registry = benchmark_module._PidfdRegistry() tracker = benchmark_module._ProgressTracker(progress, "run-7", registry) @@ -5181,9 +5628,10 @@ def test_socket_ownership_round_trips_and_rejects_progress_mutation( progress, benchmark_module.ProgressEvent( "run-7", - 2, + 1, "identity.server.socket.changed", time.monotonic_ns(), + processes=(owner,), socket_ownership=changed, ), ) @@ -5200,6 +5648,7 @@ def test_socket_ownership_round_trips_and_rejects_progress_mutation( ("st_ino", 0), ("st_uid", -1), ("st_mode", stat.S_IFREG | 0o600), + ("st_mtime_ns", 0), ), ) def test_progress_decoder_rejects_invalid_socket_ownership_schema( @@ -5213,10 +5662,11 @@ def test_progress_decoder_rejects_invalid_socket_ownership_schema( "st_ino": 11, "st_uid": os.getuid(), "st_mode": stat.S_IFSOCK | 0o600, + "st_mtime_ns": 13, } socket_row[field] = value row = { - "schema_version": 1, + "schema_version": 2, "run_id": "run-7", "sequence": 0, "checkpoint": "identity.server.socket", @@ -5232,6 +5682,126 @@ def test_progress_decoder_rejects_invalid_socket_ownership_schema( benchmark_module._progress_event_from_json(row) +@pytest.mark.parametrize( + "case", + ("server-only", "ownership-only", "mismatched-server"), +) +def test_progress_decoder_requires_atomic_server_ownership_event( + benchmark_module: types.ModuleType, + case: str, +) -> None: + """One progress row must pair its server delta with its exact capability.""" + server = {"role": "server", "pid": 2, "start_time": 3} + other_server = {"role": "server", "pid": 5, "start_time": 7} + ownership = { + "process": server, + "socket": { + "st_dev": 11, + "st_ino": 13, + "st_uid": os.getuid(), + "st_mode": stat.S_IFSOCK | 0o600, + "st_mtime_ns": 17, + }, + } + processes: list[dict[str, object]] = [] + socket_ownership: dict[str, object] | None = None + if case == "server-only": + processes = [server] + elif case == "ownership-only": + socket_ownership = ownership + else: + processes = [other_server] + socket_ownership = ownership + row = { + "schema_version": 2, + "run_id": "run-7", + "sequence": 0, + "checkpoint": "identity.server.socket", + "monotonic_ns": 1, + "processes": processes, + "socket_ownership": socket_ownership, + } + + with pytest.raises(ValueError, match="atomic server socket ownership"): + benchmark_module._progress_event_from_json(row) + + +def test_progress_decoder_accepts_non_server_identity_without_socket_ownership( + benchmark_module: types.ModuleType, +) -> None: + """Non-server process deltas do not require a socket capability.""" + row = { + "schema_version": 2, + "run_id": "run-7", + "sequence": 0, + "checkpoint": "identity.fuzzer", + "monotonic_ns": 1, + "processes": [{"role": "fuzzer", "pid": 2, "start_time": 3}], + "socket_ownership": None, + } + + event = benchmark_module._progress_event_from_json(row) + + assert event.processes == (benchmark_module.ProcessIdentity("fuzzer", 2, 3),) + assert event.socket_ownership is None + + +def test_progress_tracker_rejects_split_server_ownership_before_merge( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """A server-only row cannot be repaired by a later ownership-only row.""" + progress = tmp_path / "split-progress.jsonl" + server = {"role": "server", "pid": 2, "start_time": 3} + rows = ( + { + "schema_version": 2, + "run_id": "run-7", + "sequence": 0, + "checkpoint": "identity.server", + "monotonic_ns": 1, + "processes": [server], + "socket_ownership": None, + }, + { + "schema_version": 2, + "run_id": "run-7", + "sequence": 1, + "checkpoint": "identity.server.socket", + "monotonic_ns": 2, + "processes": [], + "socket_ownership": { + "process": server, + "socket": { + "st_dev": 11, + "st_ino": 13, + "st_uid": os.getuid(), + "st_mode": stat.S_IFSOCK | 0o600, + "st_mtime_ns": 17, + }, + }, + }, + ) + progress.write_text( + "".join(f"{json.dumps(row)}\n" for row in rows), + encoding="utf-8", + ) + registry = benchmark_module._PidfdRegistry() + tracker = benchmark_module._ProgressTracker(progress, "run-7", registry) + try: + with pytest.raises( + ValueError, match="progress journal contains an invalid event" + ): + tracker.drain(terminal=True) + + assert tracker.highest_sequence == -1 + assert tracker.identities == () + assert tracker.socket_ownership is None + assert registry.retained == () + finally: + registry.close() + + def test_progress_append_retries_short_writes_and_syncs_new_parent( benchmark_module: types.ModuleType, monkeypatch: pytest.MonkeyPatch, @@ -5936,6 +6506,84 @@ def test_validator_accepts_partial_active_failure_after_complete_prefix( ) +def _non_control_wait_boundary_report(benchmark_module: types.ModuleType) -> t.Any: + """Build one terminal report at the production non-control wait boundary.""" + base = _failed_report_with_completed_mutation_prefix(benchmark_module) + topology = base.requested_topology + name = "wait.capture-poll" + samples = tuple( + benchmark_module.RawSample( + 10 + ordinal, + True, + verified=True, + strategy=name, + ordinal=ordinal, + ) + for ordinal in range(2) + ) + observations = tuple( + benchmark_module.PhaseObservation(ordinal, name, 10 + ordinal) + for ordinal in range(2) + ) + capture = benchmark_module.PhaseReport( + name, + topology, + topology, + samples=samples, + summary=benchmark_module.summarize_ns((10, 11)), + status="completed", + warmup=2, + runs=2, + warmup_observations=tuple( + benchmark_module.PhaseObservation(ordinal, name, 2 + ordinal) + for ordinal in range(2) + ), + observations=observations, + ) + disposition = benchmark_module.PhaseReport( + "wait.control-stream", + topology, + topology, + status="not_applicable", + warmup=2, + runs=2, + ) + return dataclasses.replace( + base, + status="cutoff", + phases=(*base.phases[:3], capture, disposition), + failed_phase="cancellation", + error="KeyboardInterrupt: cancelled after wait disposition", + ) + + +def test_validator_accepts_completed_capture_then_exact_not_applicable_control( + benchmark_module: types.ModuleType, +) -> None: + """A non-control wait group may stop after its synthetic disposition row.""" + report = _non_control_wait_boundary_report(benchmark_module) + + benchmark_module.validate_report(report) + + +@pytest.mark.parametrize("field", ("warmup", "runs")) +def test_validator_rejects_malformed_not_applicable_control_disposition( + benchmark_module: types.ModuleType, + field: str, +) -> None: + """Invented not-applicable counts cannot extend a reachable wait boundary.""" + report = _non_control_wait_boundary_report(benchmark_module) + disposition = report.phases[-1] + disposition = dataclasses.replace(disposition, **{field: 1}) + report = dataclasses.replace( + report, + phases=(*report.phases[:-1], disposition), + ) + + with pytest.raises(ValueError, match="reachable seeded strategy boundary"): + benchmark_module.validate_report(report) + + def test_validator_requires_socket_capability_for_owned_server_cleanup( benchmark_module: types.ModuleType, ) -> None: @@ -5956,6 +6604,7 @@ def test_validator_requires_socket_capability_for_owned_server_cleanup( 11, os.getuid(), stat.S_IFSOCK | 0o600, + 13, ), ) benchmark_module.validate_report( From de4a81b8196e8395cdb4859bed61955057ac656b Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Tue, 18 Aug 2026 19:42:16 -0500 Subject: [PATCH 26/67] Bench(feat[runner]): Add evidence CLI why: Make active-orchestration artifacts independently verifiable and portable before publication. what: - Add complete-tree validation and atomic Markdown rendering - Cross-check ramp identity, execution metadata, and cleanup evidence - Exercise malformed artifacts, clean PEP 723 use, and Python 3.10 --- scripts/bench_orchestration.py | 825 +++++++++- tests/test_bench_orchestration_script.py | 1882 ++++++++++++++++++++-- 2 files changed, 2492 insertions(+), 215 deletions(-) diff --git a/scripts/bench_orchestration.py b/scripts/bench_orchestration.py index 57e7449ab4..f0958e3ff6 100644 --- a/scripts/bench_orchestration.py +++ b/scripts/bench_orchestration.py @@ -3,7 +3,7 @@ # requires-python = ">=3.10" # dependencies = ["rich>=13"] # /// -"""Plan and report hermetic active-tmux orchestration benchmark runs. +"""Plan, run, validate, and render hermetic active-tmux benchmark evidence. The ``plan`` command intentionally depends only on host files. It does not import libtmux or start a tmux server. @@ -19,6 +19,7 @@ import enum import functools import hashlib +import html import inspect import json import math @@ -26,6 +27,7 @@ import pathlib import platform import random +import re import resource import shlex import shutil @@ -2579,9 +2581,13 @@ def _cleanup_from_json(value: object) -> CleanupReport: True """ row = _json_mapping(value, "cleanup") + errors = row.get("errors", []) + if not isinstance(errors, list): + message = "cleanup errors must be a JSON array" + raise ValueError(message) # noqa: TRY004 return CleanupReport( complete=t.cast(bool, row.get("complete")), - errors=tuple(t.cast(t.Iterable[str], row.get("errors", []))), + errors=tuple(t.cast(list[str], errors)), processes_absent=t.cast(bool | None, row.get("processes_absent")), socket_absent=t.cast(bool | None, row.get("socket_absent")), scratch_absent=t.cast(bool | None, row.get("scratch_absent")), @@ -2695,12 +2701,16 @@ def _environment_from_json(value: object | None) -> EnvironmentReport | None: if value is None: return None row = _json_mapping(value, "environment") + command_line = row.get("command_line", []) + if not isinstance(command_line, list): + message = "environment command_line must be a JSON array" + raise ValueError(message) # noqa: TRY004 return EnvironmentReport( python_version=t.cast(str, row.get("python_version")), tmux_version=t.cast(str | None, row.get("tmux_version")), cpu_count=t.cast(int | None, row.get("cpu_count")), seed=t.cast(int, row.get("seed")), - command_line=tuple(t.cast(t.Iterable[str], row.get("command_line", []))), + command_line=tuple(t.cast(list[str], command_line)), git_revision=t.cast(str | None, row.get("git_revision")), ) @@ -2801,10 +2811,14 @@ def load_run_report(path: pathlib.Path) -> RunReport: """ try: value = json.loads(path.read_text(encoding="utf-8")) - except (OSError, json.JSONDecodeError) as error: + except (OSError, json.JSONDecodeError, RecursionError) as error: message = f"report is not complete JSON: {path}" raise ValueError(message) from error - return run_report_from_json(value) + try: + return run_report_from_json(value) + except TypeError as error: + message = f"report has invalid JSON structure: {path}" + raise ValueError(message) from error def validate_report(report: RunReport) -> None: @@ -3037,6 +3051,13 @@ def validate_report(report: RunReport) -> None: ): message = "completed report may contain completed ramp attempts only" raise ValueError(message) + if ( + report.ramp_kind != "none" + and report.status == "completed" + and (report.error is not None or report.failed_phase is not None) + ): + message = "completed ramp cannot carry terminal failure metadata" + raise ValueError(message) if report.ramp_kind != "none" and report.status in terminal_statuses: terminals = [step for step in report.ramp if step.status in terminal_statuses] aggregate_cutoff = report.status == "cutoff" and not terminals @@ -3059,6 +3080,9 @@ def validate_report(report: RunReport) -> None: else: terminal_index = report.ramp.index(terminals[0]) reason = terminals[0].reason + if report.error != reason: + message = "aggregate terminal reason must match its terminal attempt" + raise ValueError(message) if ( reason is None or any( @@ -3088,6 +3112,375 @@ def validate_report(report: RunReport) -> None: _validate_executable_ramp(report) +def _validate_artifact_scalar_types(report: RunReport) -> None: + """Reject decoded scalar values outside their declared JSON domains. + + Pure in-memory validation remains independent from this path boundary. + + >>> invalid = RunReport( + ... Topology(1, 1, 1), status="failed", error=t.cast(str, {"x": 1}) + ... ) + >>> _validate_artifact_scalar_types(invalid) + Traceback (most recent call last): + ... + ValueError: report error must be a string or None + + Parameters + ---------- + report : RunReport + Report decoded from a JSON artifact. + + Returns + ------- + None + After every rendered string and cleanup scalar has an exact type. + + Raises + ------ + ValueError + If a decoded scalar is outside its declared domain. + """ + required_strings: list[tuple[str, object]] = [ + ("status", report.status), + ("ramp_kind", report.ramp_kind), + ] + optional_strings: list[tuple[str, object]] = [ + ("run_id", report.run_id), + ("lane", report.lane), + ("mode", report.mode), + ("failed_phase", report.failed_phase), + ("error", report.error), + ("scratch_path", report.scratch_path), + ("socket_path", report.socket_path), + ("progress_path", report.progress_path), + ] + for phase in report.phases: + required_strings.extend( + (("phase name", phase.name), ("phase status", phase.status)) + ) + optional_strings.extend( + ("sample error", sample.error) for sample in phase.samples + ) + required_strings.extend( + ("process role", identity.role) for identity in report.processes + ) + for step in report.ramp: + required_strings.append(("ramp step status", step.status)) + optional_strings.extend( + ( + ("ramp step reason", step.reason), + ("ramp step run_id", step.run_id), + ("ramp step report_path", step.report_path), + ("ramp step scratch_path", step.scratch_path), + ("ramp step socket_path", step.socket_path), + ) + ) + for decision in (report.guard_decision, report.original_guard_decision): + if decision is not None: + required_strings.append(("guard kind", decision.kind)) + optional_strings.append(("guard rule", decision.rule)) + for label, value in required_strings: + if not isinstance(value, str): + message = f"report {label} must be a string" + raise ValueError(message) # noqa: TRY004 + for label, value in optional_strings: + if value is not None and not isinstance(value, str): + message = f"report {label} must be a string or None" + raise ValueError(message) + + if type(report.maximum_completed) is not bool: + message = "report maximum_completed must be a bool" + raise ValueError(message) + for phase in report.phases: + for sample in phase.samples: + for label, value in ( + ("accepted", sample.accepted), + ("verified", sample.verified), + ): + if type(value) is not bool: + message = f"raw sample {label} must be a bool" + raise ValueError(message) + for label, value in ( + ("duration_ns", sample.duration_ns), + ("ordinal", sample.ordinal), + ): + if value is not None and type(value) is not int: + message = f"raw sample {label} must be an int or None" + raise ValueError(message) + for observation in (*phase.warmup_observations, *phase.observations): + if type(observation.verified) is not bool: + message = "phase observation verified must be a bool" + raise ValueError(message) + + cleanup = report.cleanup + if type(cleanup.complete) is not bool: + message = "cleanup complete must be a bool" + raise ValueError(message) + for label, value in ( + ("processes_absent", cleanup.processes_absent), + ("socket_absent", cleanup.socket_absent), + ("scratch_absent", cleanup.scratch_absent), + ): + if value is not None and type(value) is not bool: + message = f"cleanup {label} must be a bool or None" + raise ValueError(message) + if any(not isinstance(error, str) for error in cleanup.errors): + message = "cleanup errors must contain strings" + raise ValueError(message) + + environment = report.environment + if environment is not None: + if not isinstance(environment.python_version, str): + message = "environment python_version must be a string" + raise ValueError(message) + for label, value in ( + ("tmux_version", environment.tmux_version), + ("git_revision", environment.git_revision), + ): + if value is not None and not isinstance(value, str): + message = f"environment {label} must be a string or None" + raise ValueError(message) + if environment.cpu_count is not None and type(environment.cpu_count) is not int: + message = "environment cpu_count must be an int or None" + raise ValueError(message) + if type(environment.seed) is not int: + message = "environment seed must be an int" + raise ValueError(message) + if any(not isinstance(argument, str) for argument in environment.command_line): + message = "environment command_line must contain strings" + raise ValueError(message) + + +def _aggregate_ramp_cleanup( + children: t.Sequence[tuple[Topology, CleanupReport]], +) -> CleanupReport: + """Combine child cleanup evidence without collapsing independent facts. + + >>> cleanup = CleanupReport( + ... False, ("socket remains",), processes_absent=True, + ... socket_absent=False, scratch_absent=None, + ... ) + >>> combined = _aggregate_ramp_cleanup(((Topology(1, 1, 1), cleanup),)) + >>> combined.errors + ('1x1x1: socket remains',) + >>> combined.processes_absent, combined.socket_absent, combined.scratch_absent + (True, False, None) + + Parameters + ---------- + children : collections.abc.Sequence[tuple[Topology, CleanupReport]] + Attempted child shapes and their validated cleanup evidence. + + Returns + ------- + CleanupReport + Deterministic aggregate with shape-scoped errors and tri-state facts. + """ + + def aggregate_fact(attribute: str) -> bool | None: + values = tuple(getattr(cleanup, attribute) for _shape, cleanup in children) + if all(value is True for value in values): + return True + if any(value is False for value in values): + return False + return None + + return CleanupReport( + complete=all(cleanup.complete is True for _shape, cleanup in children), + errors=tuple( + f"{shape}: {error}" + for shape, cleanup in children + for error in cleanup.errors + ), + processes_absent=aggregate_fact("processes_absent"), + socket_absent=aggregate_fact("socket_absent"), + scratch_absent=aggregate_fact("scratch_absent"), + ) + + +def validate_report_artifact(path: pathlib.Path) -> RunReport: + """Load and validate one complete report artifact tree exactly once. + + Ramp children must use parent-relative paths beneath the aggregate's + sibling ``.runs`` directory. The pure :func:`validate_report` + contract remains usable for in-memory checkpoints and fixtures. + + >>> with tempfile.TemporaryDirectory() as directory: + ... path = pathlib.Path(directory) / "report.json" + ... report = RunReport( + ... Topology(1, 1, 1), status="refused", + ... cleanup=CleanupReport( + ... True, processes_absent=True, socket_absent=True, + ... scratch_absent=True, + ... ), + ... run_id="run-7", lane="control", mode="async", warmup=0, + ... runs=1, failed_phase="preflight", error="predictive refusal", + ... guard_decision=GuardDecision( + ... False, "predictive_refusal", "pid_reserve", 2, 1, True, + ... HostSnapshot(), + ... ), + ... ) + ... write_json_atomic(path, report) + ... validate_report_artifact(path).status + 'refused' + + Parameters + ---------- + path : pathlib.Path + Root JSON report to load and validate. + + Returns + ------- + RunReport + Validated root report. + + Raises + ------ + ValueError + If the root or any referenced child is missing, malformed, unsafe, or + inconsistent with the aggregate. + """ + try: + aggregate_path = path.resolve(strict=True) + except (OSError, RuntimeError) as error: + message = f"report is missing or inaccessible: {path}" + raise ValueError(message) from error + aggregate = load_run_report(aggregate_path) + _validate_artifact_scalar_types(aggregate) + validate_report(aggregate) + if aggregate.ramp_kind == "none": + if aggregate.status != "in_progress" and aggregate.run_id is None: + message = "terminal artifact requires executable run identity" + raise ValueError(message) + return aggregate + + if aggregate.lane not in {lane.value for lane in EngineLane}: + message = "ramp aggregate requires a valid lane" + raise ValueError(message) + if aggregate.mode not in {mode.value for mode in ExecutionMode}: + message = "ramp aggregate requires a valid mode" + raise ValueError(message) + if type(aggregate.runs) is not int or aggregate.runs <= 0: + message = "ramp aggregate requires positive runs" + raise ValueError(message) + if type(aggregate.warmup) is not int or aggregate.warmup < 0: + message = "ramp aggregate requires nonnegative warmup" + raise ValueError(message) + if aggregate.environment is None or type(aggregate.environment.seed) is not int: + message = "ramp aggregate requires an integer seed" + raise ValueError(message) + + for step in aggregate.ramp: + if step.status == "not_attempted" and any( + value is not None + for value in ( + step.run_id, + step.report_path, + step.scratch_path, + step.socket_path, + ) + ): + message = ( + "not_attempted ramp step cannot carry a child artifact or " + "resource identity" + ) + raise ValueError(message) + attempted_steps = tuple( + step for step in aggregate.ramp if step.status != "not_attempted" + ) + resolved_child_root: pathlib.Path | None = None + if attempted_steps: + child_root = aggregate_path.with_name(f"{aggregate_path.stem}.runs") + try: + resolved_child_root = child_root.resolve(strict=True) + except (OSError, RuntimeError) as error: + message = f"ramp child directory is missing: {child_root.name}" + raise ValueError(message) from error + if child_root.is_symlink() or not resolved_child_root.is_dir(): + message = "ramp child directory must be a real sibling directory" + raise ValueError(message) + + seen: set[pathlib.Path] = set() + loaded_children: list[tuple[RampStep, RunReport]] = [] + for step_index, step in enumerate(aggregate.ramp): + if step.status == "not_attempted": + continue + if not isinstance(step.report_path, str) or not step.report_path: + message = "attempted ramp step requires a string child report_path" + raise ValueError(message) + serialized = pathlib.Path(step.report_path) + if serialized.is_absolute() or ".." in serialized.parts: + message = "ramp child report path escapes the sibling ramp runs directory" + raise ValueError(message) + candidate = aggregate_path.parent / serialized + try: + child_path = candidate.resolve(strict=True) + except (OSError, RuntimeError) as error: + message = f"ramp child report is missing: {step.report_path}" + raise ValueError(message) from error + if child_path == aggregate_path: + message = "artifact cycle detected" + raise ValueError(message) + assert resolved_child_root is not None + if not child_path.is_relative_to(resolved_child_root): + message = ( + "ramp child report path is outside the sibling ramp runs directory" + ) + raise ValueError(message) + if child_path in seen: + message = "duplicate canonical ramp child report path" + raise ValueError(message) + seen.add(child_path) + child = load_run_report(child_path) + _validate_artifact_scalar_types(child) + validate_report(child) + if child.ramp_kind != "none": + message = "nested ramp child reports are not allowed" + raise ValueError(message) + if child.status not in {"completed", "refused", "failed", "cutoff"}: + message = "ramp child report must be terminal" + raise ValueError(message) + if child.requested_topology != step.shape: + message = "ramp child topology differs from its aggregate step" + raise ValueError(message) + if child.status != step.status: + message = "ramp child status differs from its aggregate step" + raise ValueError(message) + if child.error != step.reason: + message = "ramp child reason differs from its aggregate step" + raise ValueError(message) + for attribute in ("run_id", "scratch_path", "socket_path"): + if getattr(child, attribute) != getattr(step, attribute): + message = f"ramp child {attribute} differs from its aggregate step" + raise ValueError(message) + for attribute in ("lane", "mode", "runs", "warmup"): + if getattr(child, attribute) != getattr(aggregate, attribute): + message = f"ramp child {attribute} differs from its aggregate" + raise ValueError(message) + expected_seed = aggregate.environment.seed + step_index + if child.environment is None or child.environment.seed != expected_seed: + message = "ramp child seed differs from its aggregate step" + raise ValueError(message) + loaded_children.append((step, child)) + + completed_children = tuple( + child for step, child in loaded_children if step.status == "completed" + ) + expected_observed = ( + completed_children[-1].observed_topology if completed_children else None + ) + if aggregate.observed_topology != expected_observed: + message = "aggregate observed topology differs from completed child evidence" + raise ValueError(message) + expected_cleanup = _aggregate_ramp_cleanup( + tuple((step.shape, child.cleanup) for step, child in loaded_children) + ) + if aggregate.cleanup != expected_cleanup: + message = "aggregate cleanup differs from child cleanup evidence" + raise ValueError(message) + return aggregate + + def _reachable_group_signatures( strategy_names: tuple[str, ...], *, @@ -12074,6 +12467,100 @@ def _format_ns(value: float) -> str: return f"{float(value) / 1_000_000:.3f} ms" +def _format_cleanup_fact(value: bool | None) -> str: + """Render one cleanup fact without conflating unknown with false. + + >>> _format_cleanup_fact(True) + 'true' + >>> _format_cleanup_fact(None) + 'unknown' + + Parameters + ---------- + value : bool | None + Verified absence fact or unavailable evidence. + + Returns + ------- + str + Lowercase truth value or ``unknown``. + """ + return "unknown" if value is None else str(value).lower() + + +def _redact_owned_paths(value: str | None, report: RunReport) -> str: + """Remove known or explicit absolute local paths from summary prose. + + >>> report = RunReport(Topology(1, 1, 1), scratch_path="/tmp/private") + >>> _redact_owned_paths("failed at /tmp/private", report) + 'failed at [owned path]' + + Parameters + ---------- + value : str | None + Terminal or ramp-step reason intended for Markdown. + report : RunReport + Root report carrying known owned resource identities. + + Returns + ------- + str + Reason with owned and remaining absolute path tokens redacted. + """ + if value is None: + return "" + owned_paths = { + path + for path in (report.scratch_path, report.socket_path, report.progress_path) + if path + } + owned_paths.update( + path + for step in report.ramp + for path in (step.scratch_path, step.socket_path) + if path + ) + redacted = value + for owned_path in sorted(owned_paths, key=len, reverse=True): + redacted = redacted.replace(owned_path, "[owned path]") + redacted = re.sub( + r"(?P['\"])/[^'\"\r\n]+(?P=quote)", + lambda match: f"{match.group('quote')}[owned path]{match.group('quote')}", + redacted, + ) + redacted = re.sub(r"`/[^`\r\n]+`", "`[owned path]`", redacted) + return re.sub(r"(? str: + r"""Return redacted single-line text safe inside a Markdown table cell. + + >>> report = RunReport(Topology(1, 1, 1), scratch_path="/tmp/private path") + >>> _markdown_cell("bad | row\n`/tmp/private path`", report) + 'bad \\| row
`[owned path]`' + + Parameters + ---------- + value : str | None + Reason, error, or relative artifact reference intended for Markdown. + report : RunReport + Root report carrying modeled owned paths. + + Returns + ------- + str + Redacted text with row delimiters and newlines neutralized. + """ + redacted = html.escape(_redact_owned_paths(value, report), quote=False) + return ( + redacted.replace("\r\n", "\n") + .replace("\r", "\n") + .replace("\n", "
") + .replace("|", r"\|") + .replace("`", "`") + ) + + def render_markdown_summary( report_path: pathlib.Path, output_path: pathlib.Path | None = None, @@ -12116,11 +12603,28 @@ def render_markdown_summary( ValueError If JSON or its recomputed report contract is invalid. """ - report = load_run_report(report_path) - validate_report(report) + report = validate_report_artifact(report_path) if report.status == "in_progress": message = "cannot render an in-progress report" raise ValueError(message) + if output_path is not None: + try: + aggregate_path = report_path.resolve() + artifact_paths = {aggregate_path} + artifact_paths.update( + (aggregate_path.parent / step.report_path).resolve() + for step in report.ramp + if step.report_path is not None + ) + resolved_output = output_path.resolve() + except (OSError, RuntimeError) as error: + message = f"Markdown output is missing or inaccessible: {output_path}" + raise ValueError(message) from error + if resolved_output in artifact_paths: + message = ( + "Markdown output cannot replace the same file in the JSON artifact tree" + ) + raise ValueError(message) lines = [ "# Active orchestration benchmark", "", @@ -12138,19 +12642,41 @@ def render_markdown_summary( lines.extend((f"Observed topology: `{report.observed_topology}`", "")) if report.lane is not None and report.mode is not None: lines.extend((f"Lane: `{report.lane}/{report.mode}`", "")) + if report.runs is not None: + lines.extend((f"Runs: `{report.runs}`", "")) + if report.warmup is not None: + lines.extend((f"Warmup: `{report.warmup}`", "")) + if report.environment is not None: + python_version = _markdown_cell(report.environment.python_version, report) + tmux_version = _markdown_cell(report.environment.tmux_version or "n/a", report) + revision = _markdown_cell(report.environment.git_revision or "n/a", report) + lines.extend( + ( + f"Seed: `{report.environment.seed}`", + "", + f"Python: `{python_version}`", + "", + f"tmux: `{tmux_version}`", + "", + f"Revision: `{revision}`", + "", + ) + ) if report.error is not None: - lines.extend((f"Terminal reason: {report.error}", "")) + lines.extend((f"Terminal reason: {_markdown_cell(report.error, report)}", "")) if report.ramp: lines.extend( ( "## Ramp attempts", "", - "| Shape | Status | Reason |", - "| --- | --- | --- |", + "| Shape | Status | Reason | Child report |", + "| --- | --- | --- | --- |", ) ) lines.extend( - f"| `{step.shape}` | `{step.status}` | {step.reason or ''} |" + f"| `{step.shape}` | `{step.status}` | " + f"{_markdown_cell(step.reason, report)} | " + f"{_markdown_cell(step.report_path, report)} |" for step in report.ramp ) lines.append("") @@ -12159,39 +12685,76 @@ def render_markdown_summary( ( "## Phase timings", "", - "| Phase | Status | Samples | Median | p95 |", - "| --- | --- | ---: | ---: | ---: |", + ( + "| Phase | Status | Count | Min | Mean | Median | p90 | p95 | " + "p99 | Max |" + ), + "| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |", ) ) + setup_observations: tuple[str, ...] = () for phase in report.phases: + accepted = tuple( + t.cast(int, sample.duration_ns) + for sample in phase.samples + if sample.accepted + ) if phase.name == "setup": - values = ", ".join( + setup_observations = tuple( _format_ns(t.cast(int, sample.duration_ns)) for sample in phase.samples if sample.accepted ) lines.append( f"| `{phase.name}` | `{phase.status}` | " - f"{len(phase.samples)} individual: {values} | n/a | n/a |" + f"{len(accepted)} | n/a | n/a | n/a | " + "n/a | n/a | n/a | n/a |" ) - elif phase.summary is None: - lines.append(f"| `{phase.name}` | `{phase.status}` | 0 | n/a | n/a |") - else: + elif accepted: + summary = summarize_ns(accepted) lines.append( f"| `{phase.name}` | `{phase.status}` | " - f"{phase.summary['count']} | " - f"{_format_ns(phase.summary['median_ns'])} | " - f"{_format_ns(phase.summary['p95_ns'])} |" + f"{summary['count']} | " + f"{_format_ns(summary['min_ns'])} | " + f"{_format_ns(summary['mean_ns'])} | " + f"{_format_ns(summary['median_ns'])} | " + f"{_format_ns(summary['p90_ns'])} | " + f"{_format_ns(summary['p95_ns'])} | " + f"{_format_ns(summary['p99_ns'])} | " + f"{_format_ns(summary['max_ns'])} |" + ) + else: + lines.append( + f"| `{phase.name}` | `{phase.status}` | 0 | n/a | n/a | " + "n/a | n/a | n/a | n/a | n/a |" ) lines.append("") + if setup_observations: + values = ", ".join(f"`{value}`" for value in setup_observations) + lines.extend((f"Setup individual observations: {values}", "")) lines.extend( ( "## Cleanup", "", f"Verified complete: `{str(report.cleanup.complete).lower()}`", "", + ( + "Processes absent: " + f"`{_format_cleanup_fact(report.cleanup.processes_absent)}`" + ), + "", + f"Socket absent: `{_format_cleanup_fact(report.cleanup.socket_absent)}`", + "", + f"Scratch absent: `{_format_cleanup_fact(report.cleanup.scratch_absent)}`", + "", ) ) + if report.cleanup.errors: + lines.extend(("Cleanup errors:", "")) + lines.extend( + f"- {_markdown_cell(error, report)}" for error in report.cleanup.errors + ) + lines.append("") rendered = "\n".join(lines) if output_path is not None: _write_text_atomic(output_path, rendered) @@ -12316,6 +12879,14 @@ def run_scenario( if capability_error is not None and decision.rule == "pidfd_capability" else f"predictive refusal: {original.rule or 'unknown'}" ), + environment=EnvironmentReport( + python_version=platform.python_version(), + tmux_version=None, + cpu_count=os.cpu_count(), + seed=seed, + command_line=("run", "--shape", str(topology)), + git_revision=None, + ), ) write_json_atomic(output, report) validate_report(report) @@ -12405,7 +12976,7 @@ def interruption_reason(error: KeyboardInterrupt) -> str: child.status, child.error, run_id=child.run_id, - report_path=str(active_child_output), + report_path=active_child_output.relative_to(output.parent).as_posix(), scratch_path=child.scratch_path, socket_path=child.socket_path, ) @@ -12444,19 +13015,17 @@ def build_terminal() -> RunReport: step, reason=final_reason, ) - cleanup_complete = all( - step.status == "not_attempted" - or load_run_report( - pathlib.Path(t.cast(str, step.report_path)) - ).cleanup.complete + child_cleanups = tuple( + ( + step.shape, + load_run_report( + output.parent / pathlib.Path(t.cast(str, step.report_path)) + ).cleanup, + ) for step in resolved_steps + if step.status != "not_attempted" ) - cleanup = CleanupReport( - cleanup_complete, - processes_absent=cleanup_complete, - socket_absent=cleanup_complete, - scratch_absent=cleanup_complete, - ) + cleanup = _aggregate_ramp_cleanup(child_cleanups) return dataclasses.replace( report, status=final_status, @@ -12536,6 +13105,10 @@ def run_ramp( ramp=attempts, requested_shapes=declared, ramp_kind="canonical" if canonical else "custom", + lane=lane.value, + mode=mode.value, + warmup=warmup, + runs=runs, environment=_collect_environment( seed=seed, command_line=("ramp", "--shapes", ",".join(map(str, declared))), @@ -12565,7 +13138,7 @@ def step_from_child( ), child.error, run_id=child.run_id, - report_path=str(child_output), + report_path=child_output.relative_to(output.parent).as_posix(), scratch_path=child.scratch_path, socket_path=child.socket_path, ) @@ -12919,22 +13492,71 @@ def _add_execution_arguments(parser: argparse.ArgumentParser) -> None: 2 """ parser.add_argument( - "--lane", choices=tuple(lane.value for lane in EngineLane), default="control" + "--lane", + choices=tuple(lane.value for lane in EngineLane), + default="control", + help="transport lane; combine with --mode for one of four execution lanes", + ) + parser.add_argument( + "--mode", + choices=tuple(mode.value for mode in ExecutionMode), + default="async", + help="dispatch mode paired with --lane", + ) + parser.add_argument( + "--runs", + type=int, + default=100, + help="timed invocations retained per repeatable phase", + ) + parser.add_argument( + "--warmup", + type=int, + default=3, + help="untimed invocations before each repeatable phase", + ) + parser.add_argument("--seed", type=int, default=11, help="schedule seed") + parser.add_argument( + "--output", + type=pathlib.Path, + help="machine-readable JSON artifact destination", + ) + parser.add_argument( + "--markdown-output", + type=pathlib.Path, + help="local Markdown summary destination", + ) + parser.add_argument( + "--scratch-root", + type=pathlib.Path, + help="parent for private per-run state", + ) + parser.add_argument( + "--force-extreme", + action="store_true", + help=( + "relax predictive refusal only; never runtime cutoff, correctness, " + "or cleanup" + ), + ) + parser.add_argument("--pid-reserve", type=int, help="minimum free PID reserve") + parser.add_argument( + "--memory-floor-bytes", + type=int, + help="minimum free memory reserve", ) parser.add_argument( - "--mode", choices=tuple(mode.value for mode in ExecutionMode), default="async" - ) - parser.add_argument("--runs", type=int, default=100) - parser.add_argument("--warmup", type=int, default=3) - parser.add_argument("--seed", type=int, default=11) - parser.add_argument("--output", type=pathlib.Path) - parser.add_argument("--markdown-output", type=pathlib.Path) - parser.add_argument("--scratch-root", type=pathlib.Path) - parser.add_argument("--force-extreme", action="store_true") - parser.add_argument("--pid-reserve", type=int) - parser.add_argument("--memory-floor-bytes", type=int) - parser.add_argument("--watchdog-seconds", type=float, default=120.0) - parser.add_argument("--cleanup-grace-seconds", type=float, default=2.0) + "--watchdog-seconds", + type=float, + default=120.0, + help="maximum interval without progress", + ) + parser.add_argument( + "--cleanup-grace-seconds", + type=float, + default=2.0, + help="grace interval before exact process escalation", + ) parser.add_argument( "--_test-host-snapshot", dest="test_host_snapshot", @@ -13003,7 +13625,7 @@ def _hidden_worker_parser() -> argparse.ArgumentParser: def main(argv: t.Sequence[str] | None = None) -> int: - """Run planning, one supervised scenario, a ramp, or the hidden worker. + """Plan, run, validate, or render active orchestration evidence. >>> import contextlib, io >>> captured = io.StringIO() @@ -13035,21 +13657,114 @@ def main(argv: t.Sequence[str] | None = None) -> int: if raw_arguments[:1] == ("_worker",): worker_arguments = _hidden_worker_parser().parse_args(raw_arguments[1:]) return _run_hidden_worker(worker_arguments) - parser = argparse.ArgumentParser(description=__doc__) + parser = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=( + "Topology syntax: SxWxP means sessions x windows per session x " + "panes per window.\n" + "Ramp --shapes accepts comma-separated SxWxP values. --runs retains " + "timed samples; --warmup performs untimed calls.\n" + "Execution lanes: subprocess/sync, subprocess/async, control/sync, " + "and control/async.\n" + "Run and ramp write JSON evidence and optional Markdown summaries; " + "validate and render consume that evidence.\n" + "--force-extreme relaxes predictive refusal only, never runtime " + "cutoff, correctness, or cleanup." + ), + ) commands = parser.add_subparsers(dest="command", required=True) plan_parser = commands.add_parser("plan", help="inspect topology and host limits") - plan_parser.add_argument("--shape", required=True) - plan_parser.add_argument("--output", type=pathlib.Path) - plan_parser.add_argument("--force-extreme", action="store_true") + plan_parser.add_argument( + "--shape", + required=True, + help="topology in SxWxP notation", + ) + plan_parser.add_argument( + "--output", + type=pathlib.Path, + help="optional JSON plan destination", + ) + plan_parser.add_argument( + "--force-extreme", + action="store_true", + help="show predictive-refusal override without changing runtime safety", + ) run_parser = commands.add_parser("run", help="run one active topology") - run_parser.add_argument("--shape", required=True) + run_parser.add_argument( + "--shape", + required=True, + help="topology in SxWxP notation", + ) _add_execution_arguments(run_parser) ramp_parser = commands.add_parser("ramp", help="run fresh topologies in order") - ramp_parser.add_argument("--shapes") + ramp_parser.add_argument( + "--shapes", + help="comma-separated SxWxP shapes; default is the canonical ramp", + ) _add_execution_arguments(ramp_parser) + validate_parser = commands.add_parser( + "validate", + help="validate a complete JSON artifact tree", + description=( + "Read-only validation of a complete artifact tree without contacting tmux." + ), + epilog="Ramp child references are checked without modifying any artifact.", + ) + validate_parser.add_argument( + "--input", + required=True, + type=pathlib.Path, + help="complete JSON report artifact", + ) + render_parser = commands.add_parser( + "render", + help="render a complete JSON artifact tree as Markdown", + description="Validate a complete artifact tree before rendering Markdown.", + epilog=( + "With --output, Markdown is replaced atomically only after validation; " + "without it, Markdown is written to stdout." + ), + ) + render_parser.add_argument( + "--input", + required=True, + type=pathlib.Path, + help="complete JSON report artifact", + ) + render_parser.add_argument( + "--output", + type=pathlib.Path, + help="atomic Markdown destination; omit for stdout", + ) arguments = parser.parse_args(raw_arguments) if arguments.command == "plan": return run_plan(arguments.shape, arguments.output, arguments.force_extreme) + if arguments.command == "validate": + try: + report = validate_report_artifact(arguments.input) + except (OSError, TypeError, ValueError) as error: + print(f"invalid benchmark artifact: {error}", file=sys.stderr) + return 2 + if report.status == "in_progress": + print( + "invalid benchmark artifact: report is still in-progress", + file=sys.stderr, + ) + return 2 + print(f"Status: {report.status}") + print(f"Requested topology: {report.requested_topology}") + print(f"Observed topology: {report.observed_topology or 'n/a'}") + return 0 + if arguments.command == "render": + try: + rendered = render_markdown_summary(arguments.input, arguments.output) + except (OSError, TypeError, ValueError) as error: + print(f"invalid benchmark artifact: {error}", file=sys.stderr) + return 2 + if arguments.output is None: + print(rendered) + return 0 policy = ResourcePolicy( persistent_clients=(1 if arguments.lane == "control" else 0), pid_reserve=arguments.pid_reserve, diff --git a/tests/test_bench_orchestration_script.py b/tests/test_bench_orchestration_script.py index 132a1403fa..d6af3968b6 100644 --- a/tests/test_bench_orchestration_script.py +++ b/tests/test_bench_orchestration_script.py @@ -101,6 +101,46 @@ def _run_cli(*arguments: str, cwd: pathlib.Path) -> subprocess.CompletedProcess[ ) +def _run_pep723_script( + script: pathlib.Path, + *arguments: str, + cwd: pathlib.Path, +) -> subprocess.CompletedProcess[str]: + """Run one PEP 723 script without project or virtual-environment state.""" + environment = os.environ.copy() + for name in ( + "CONDA_PREFIX", + "PYTHONPATH", + "TMUX", + "TMUX_PANE", + "UV_CONFIG_FILE", + "UV_ENV_FILE", + "UV_PROJECT_ENVIRONMENT", + "UV_WORKSPACE", + "VIRTUAL_ENV", + ): + environment.pop(name, None) + return subprocess.run( + ( + "uv", + "run", + "--no-config", + "--no-env-file", + "--isolated", + "--no-project", + "--script", + str(script), + *arguments, + ), + cwd=cwd, + env=environment, + text=True, + capture_output=True, + check=False, + timeout=120, + ) + + def _start_cli(*arguments: str, cwd: pathlib.Path) -> subprocess.Popen[str]: """Start the real benchmark so a test can interrupt its supervisor.""" environment = os.environ.copy() @@ -275,110 +315,1638 @@ def complete_procfs_files() -> dict[str, str]: } -def test_probe_host_resolves_unified_cgroup_and_literal_limits( +def test_probe_host_resolves_unified_cgroup_and_literal_limits( + benchmark_module: types.ModuleType, +) -> None: + """A wrong cgroup join would hide the real container resource envelope.""" + snapshot = benchmark_module.probe_host(LiteralReader(complete_procfs_files())) + + assert snapshot.available_memory_bytes == 25_165_824 * 1024 + assert snapshot.physical_memory_bytes == 33_554_432 * 1024 + assert snapshot.pids_current == 472 + assert snapshot.pids_max == 45_343 + assert snapshot.memory_current_bytes == 8_589_934_592 + assert snapshot.memory_max_bytes == 34_359_738_368 + assert snapshot.nofile_soft_limit == 65_536 + assert snapshot.memory_pressure_some_avg10 == 0.0 + assert snapshot.source_errors == {} + + +def test_probe_host_preserves_missing_telemetry_as_unknown( + benchmark_module: types.ModuleType, +) -> None: + """Treating an unreadable cgroup value as zero would make false admissions.""" + files = complete_procfs_files() + del files[ + "/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/bench.scope/pids.current" + ] + + snapshot = benchmark_module.probe_host(LiteralReader(files)) + + assert snapshot.pids_current is None + assert "pids.current" in snapshot.source_errors + + +def test_predict_resources_refuses_when_projected_pids_break_reserve( + benchmark_module: types.ModuleType, +) -> None: + """Ignoring the PID reserve would let the 40,000-pane plan exhaust cgroups.""" + snapshot = benchmark_module.probe_host(LiteralReader(complete_procfs_files())) + + decision = benchmark_module.predict_resources( + benchmark_module.parse_topology("100x100x4"), snapshot + ) + + assert decision.allowed is False + assert decision.kind == "predictive_refusal" + assert decision.rule == "pid_reserve" + assert decision.observed == 40_475 + assert decision.limit == 38_541 + assert decision.forceable is True + + +def test_predict_resources_admits_small_shape_with_same_host_limits( + benchmark_module: types.ModuleType, +) -> None: + """A guard that over-refuses would prevent the canonical ramp from starting.""" + snapshot = benchmark_module.probe_host(LiteralReader(complete_procfs_files())) + + decision = benchmark_module.predict_resources( + benchmark_module.parse_topology("80x20x1"), snapshot + ) + + assert decision.allowed is True + assert decision.kind == "ok" + assert decision.rule is None + + +def completed_report(benchmark_module: types.ModuleType) -> t.Any: + """Build a literal completed artifact with one failed raw sample excluded.""" + topology = benchmark_module.parse_topology("100x100x4") + phase = benchmark_module.PhaseReport( + name="enumeration.sessions", + requested_topology=topology, + observed_topology=topology, + samples=( + benchmark_module.RawSample(duration_ns=10, accepted=True, verified=True), + benchmark_module.RawSample(duration_ns=30, accepted=True, verified=True), + benchmark_module.RawSample( + duration_ns=999, accepted=False, error="lost row count" + ), + ), + summary={ + "count": 2, + "min_ns": 10, + "mean_ns": 20, + "median_ns": 20.0, + "p90_ns": 30, + "p95_ns": 30, + "p99_ns": 30, + "max_ns": 30, + }, + ) + return benchmark_module.RunReport( + status="completed", + requested_topology=topology, + observed_topology=topology, + phases=(phase,), + cleanup=benchmark_module.CleanupReport( + complete=True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + maximum_completed=True, + ) + + +def _completed_executable_report( + benchmark_module: types.ModuleType, + shape: t.Any, + ordinal: int, + *, + seed: int = 11, + lane: str = "subprocess", + mode: str = "sync", +) -> t.Any: + """Build one fully schema-valid completed executable child artifact.""" + run_id = f"child-{ordinal}" + scratch_path = f"scratch-{ordinal}" + socket_path = f"{scratch_path}/tmux.sock" + owner = benchmark_module.ProcessIdentity( + "server", + 200_000 + ordinal, + 300_000 + ordinal, + ) + ownership = benchmark_module._SocketOwnership( + owner, + benchmark_module.SocketIdentity( + 7, + 10 + ordinal, + os.getuid(), + stat.S_IFSOCK | 0o600, + 20 + ordinal, + ), + ) + setup = benchmark_module.PhaseReport( + "setup", + shape, + shape, + samples=( + benchmark_module.RawSample( + 1_000_000, + True, + verified=True, + strategy="setup", + ordinal=0, + ), + ), + status="completed", + runs=1, + observations=(benchmark_module.PhaseObservation(0, "setup", 1_000_000),), + ) + stabilization = benchmark_module.PhaseReport( + "stabilization", + shape, + shape, + status="completed", + observations=( + benchmark_module.PhaseObservation( + 0, + "stabilization", + 2_000_000, + pane_count=shape.panes, + ), + ), + ) + + def completed_phase(name: str) -> t.Any: + observation_kwargs: dict[str, int] = {} + if name.startswith("enumeration."): + kind = name.removeprefix("enumeration.") + observation_kwargs["row_count"] = { + "sessions": shape.sessions, + "windows": shape.windows, + "panes": shape.panes, + }[kind] + elif name.startswith("capture."): + observation_kwargs.update( + pane_count=shape.panes, + line_count=shape.panes, + byte_count=shape.panes, + ) + elif name.startswith("search."): + observation_kwargs.update(scanned_count=1, matched_count=1) + return benchmark_module.PhaseReport( + name, + shape, + shape, + samples=( + benchmark_module.RawSample( + 3_000_000, + True, + verified=True, + strategy=name, + ordinal=0, + ), + ), + summary={ + "count": 1, + "min_ns": 3_000_000, + "mean_ns": 3_000_000, + "median_ns": 3_000_000, + "p90_ns": 3_000_000, + "p95_ns": 3_000_000, + "p99_ns": 3_000_000, + "max_ns": 3_000_000, + }, + status="completed", + warmup=0, + runs=1, + observations=( + benchmark_module.PhaseObservation( + 0, + name, + 3_000_000, + **observation_kwargs, + ), + ), + ) + + phases = tuple( + ( + setup + if name == "setup" + else stabilization + if name == "stabilization" + else benchmark_module.PhaseReport( + name, + shape, + shape, + status="not_applicable", + warmup=0, + runs=1, + ) + if name == "wait.control-stream" and (lane, mode) != ("control", "async") + else completed_phase(name) + ) + for name in _RUNNER_PHASES + ) + report = benchmark_module.RunReport( + shape, + observed_topology=shape, + status="completed", + phases=phases, + cleanup=benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + run_id=run_id, + lane=lane, + mode=mode, + warmup=0, + runs=1, + processes=(owner,), + socket_ownership=ownership, + scratch_path=scratch_path, + socket_path=socket_path, + progress_path=f"progress-{ordinal}.jsonl", + environment=benchmark_module.EnvironmentReport( + "3.10", None, 1, seed, ("run",), None + ), + ) + benchmark_module.validate_report(report) + return report + + +def _terminal_executable_report( + benchmark_module: types.ModuleType, + shape: t.Any, + ordinal: int, + status: str, + *, + reason: str | None = None, + lane: str = "subprocess", + mode: str = "sync", + warmup: int = 0, + runs: int = 1, + seed: int = 11, +) -> t.Any: + """Build one fully schema-valid unsuccessful executable child artifact.""" + reason = reason or f"{status} child {ordinal}" + cleanup = benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ) + if status == "refused": + report = benchmark_module.RunReport( + shape, + status="refused", + cleanup=cleanup, + guard_decision=benchmark_module.GuardDecision( + False, + "predictive_refusal", + "pid_reserve", + 2, + 1, + True, + benchmark_module.HostSnapshot(), + ), + run_id=f"child-{ordinal}", + lane=lane, + mode=mode, + warmup=warmup, + runs=runs, + failed_phase="preflight", + error=reason, + environment=benchmark_module.EnvironmentReport( + "3.10", None, 1, seed, ("run",), None + ), + ) + else: + report = benchmark_module.RunReport( + shape, + status=status, + cleanup=cleanup, + run_id=f"child-{ordinal}", + lane=lane, + mode=mode, + warmup=warmup, + runs=runs, + failed_phase="cancellation" if status == "cutoff" else "setup", + error=reason, + scratch_path=f"scratch-{ordinal}", + socket_path=f"scratch-{ordinal}/tmux.sock", + progress_path=f"progress-{ordinal}.jsonl", + environment=benchmark_module.EnvironmentReport( + "3.10", None, 1, seed, ("run",), None + ), + ) + benchmark_module.validate_report(report) + return report + + +def _write_completed_ramp_artifact( + benchmark_module: types.ModuleType, + output: pathlib.Path, + shapes: tuple[t.Any, ...], +) -> tuple[t.Any, tuple[pathlib.Path, ...]]: + """Write one completed aggregate and fully valid relative child reports.""" + child_root = output.with_name(f"{output.stem}.runs") + children: list[t.Any] = [] + child_paths: list[pathlib.Path] = [] + steps: list[t.Any] = [] + for ordinal, shape in enumerate(shapes, start=1): + child = _completed_executable_report( + benchmark_module, + shape, + ordinal, + seed=11 + ordinal - 1, + ) + child_path = child_root / f"{ordinal - 1:02d}-{shape}.json" + benchmark_module.write_json_atomic(child_path, child) + relative = child_path.relative_to(output.parent).as_posix() + children.append(child) + child_paths.append(child_path) + steps.append( + benchmark_module.RampStep( + shape, + "completed", + run_id=child.run_id, + report_path=relative, + scratch_path=child.scratch_path, + socket_path=child.socket_path, + ) + ) + aggregate = benchmark_module.RunReport( + shapes[-1], + observed_topology=children[-1].observed_topology, + status="completed", + cleanup=benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + ramp=tuple(steps), + requested_shapes=shapes, + ramp_kind="custom", + lane="subprocess", + mode="sync", + warmup=0, + runs=1, + environment=benchmark_module.EnvironmentReport( + "3.10", None, 1, 11, ("ramp",), None + ), + ) + benchmark_module.write_json_atomic(output, aggregate) + return aggregate, tuple(child_paths) + + +def test_validate_report_artifact_loads_complete_relative_child_tree_once( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, +) -> None: + """Skipping, rereading, or misresolving a child would weaken the evidence tree.""" + output = tmp_path / "ramp.json" + aggregate, child_paths = _write_completed_ramp_artifact( + benchmark_module, + output, + ( + benchmark_module.Topology(1, 1, 1), + benchmark_module.Topology(2, 1, 1), + ), + ) + reads: dict[pathlib.Path, int] = {} + real_read_text = pathlib.Path.read_text + + def count_read(path: pathlib.Path, *args: t.Any, **kwargs: t.Any) -> str: + canonical = path.resolve() + reads[canonical] = reads.get(canonical, 0) + 1 + return real_read_text(path, *args, **kwargs) + + monkeypatch.setattr(pathlib.Path, "read_text", count_read) + + loaded = benchmark_module.validate_report_artifact(output) + + assert loaded == aggregate + assert reads == { + output.resolve(): 1, + **{child.resolve(): 1 for child in child_paths}, + } + + +def test_validate_report_artifact_rejects_nonexecutable_terminal_report( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """Artifact validation must not publish a generic in-memory terminal value.""" + report = completed_report(benchmark_module) + output = tmp_path / "generic.json" + benchmark_module.validate_report(report) + benchmark_module.write_json_atomic(output, report) + + with pytest.raises(ValueError, match="executable"): + benchmark_module.validate_report_artifact(output) + + +def test_validate_report_artifact_rejects_traversal_and_invalid_child( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """A ramp reference cannot escape its sibling tree or admit malformed JSON.""" + root = tmp_path / "evidence" + output = root / "ramp.json" + aggregate, child_paths = _write_completed_ramp_artifact( + benchmark_module, + output, + (benchmark_module.Topology(1, 1, 1),), + ) + outside = tmp_path / "outside.json" + child_paths[0].replace(outside) + escaped = dataclasses.replace( + aggregate, + ramp=( + dataclasses.replace( + aggregate.ramp[0], + report_path="../outside.json", + ), + ), + ) + benchmark_module.write_json_atomic(output, escaped) + + with pytest.raises(ValueError, match=r"outside.*ramp.*runs|escape"): + benchmark_module.validate_report_artifact(output) + + invalid_child = output.with_name(f"{output.stem}.runs") / "bad.json" + invalid_child.parent.mkdir(parents=True, exist_ok=True) + invalid_child.write_text("{not-json\n", encoding="utf-8") + invalid = dataclasses.replace( + aggregate, + ramp=( + dataclasses.replace( + aggregate.ramp[0], + report_path=invalid_child.relative_to(output.parent).as_posix(), + ), + ), + ) + benchmark_module.write_json_atomic(output, invalid) + + with pytest.raises(ValueError, match="complete JSON"): + benchmark_module.validate_report_artifact(output) + + +@pytest.mark.parametrize("location", ("root", "child", "output")) +def test_validate_report_artifact_normalizes_python310_symlink_loop( + benchmark_module: types.ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: pathlib.Path, + location: str, +) -> None: + """A Python 3.10 resolve loop must become a stable artifact error.""" + output = tmp_path / "ramp.json" + _aggregate, child_paths = _write_completed_ramp_artifact( + benchmark_module, + output, + (benchmark_module.Topology(1, 1, 1),), + ) + markdown = tmp_path / "summary.md" + trigger = ( + output + if location == "root" + else child_paths[0] + if location == "child" + else markdown + ) + real_resolve = pathlib.Path.resolve + + def raise_python310_loop( + path: pathlib.Path, + strict: bool = False, + ) -> pathlib.Path: + if path == trigger: + message = "Symlink loop from '/evidence'" + raise RuntimeError(message) + return real_resolve(path, strict=strict) + + monkeypatch.setattr(pathlib.Path, "resolve", raise_python310_loop) + + with pytest.raises(ValueError, match=r"missing|inaccessible"): + if location == "output": + benchmark_module.render_markdown_summary(output, markdown) + else: + benchmark_module.validate_report_artifact(output) + + +def test_public_validate_and_render_fail_concisely_and_preserve_output( + tmp_path: pathlib.Path, +) -> None: + """Invalid public evidence must return two without traceback or replacement.""" + invalid = tmp_path / "invalid.json" + destination = tmp_path / "summary.md" + invalid.write_text("{not-json\n", encoding="utf-8") + destination.write_text("retained\n", encoding="utf-8") + + validated = _run_cli("validate", "--input", str(invalid), cwd=tmp_path) + rendered = _run_cli( + "render", + "--input", + str(invalid), + "--output", + str(destination), + cwd=tmp_path, + ) + + for completed in (validated, rendered): + assert completed.returncode == 2 + assert "invalid benchmark artifact" in completed.stderr + assert "Traceback" not in completed.stderr + assert destination.read_text(encoding="utf-8") == "retained\n" + + +def test_markdown_phase_table_has_full_statistics_and_partial_count( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """Dropping a statistic or accepted partial row would misstate local evidence.""" + shape = benchmark_module.Topology(1, 1, 1) + setup = benchmark_module.PhaseReport( + "setup", + shape, + shape, + samples=( + benchmark_module.RawSample( + 1_000_000, + True, + verified=True, + strategy="setup", + ordinal=0, + ), + ), + status="completed", + runs=1, + observations=(benchmark_module.PhaseObservation(0, "setup", 1_000_000),), + ) + stabilization = benchmark_module.PhaseReport( + "stabilization", + shape, + shape, + status="completed", + observations=( + benchmark_module.PhaseObservation( + 0, + "stabilization", + 1_500_000, + pane_count=shape.panes, + ), + ), + ) + partial = benchmark_module.PhaseReport( + "mutation.bulk", + shape, + shape, + samples=tuple( + benchmark_module.RawSample( + duration, + True, + verified=True, + strategy="mutation.bulk", + ordinal=ordinal, + ) + for ordinal, duration in enumerate((2_000_000, 4_000_000)) + ), + status="failed", + runs=3, + observations=tuple( + benchmark_module.PhaseObservation( + ordinal, + "mutation.bulk", + duration, + ) + for ordinal, duration in enumerate((2_000_000, 4_000_000)) + ), + ) + report = benchmark_module.RunReport( + shape, + observed_topology=shape, + status="failed", + phases=(setup, stabilization, partial), + cleanup=benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + run_id="partial-1", + lane="subprocess", + mode="sync", + warmup=0, + runs=3, + failed_phase="mutation.bulk", + error="injected partial phase", + scratch_path="scratch-partial-1", + socket_path="scratch-partial-1/tmux.sock", + progress_path="progress-partial-1.jsonl", + environment=benchmark_module.EnvironmentReport( + "3.10", None, 1, 11, ("run",), None + ), + ) + report_path = tmp_path / "partial.json" + benchmark_module.write_json_atomic(report_path, report) + + markdown = benchmark_module.render_markdown_summary(report_path) + + assert ( + "| Phase | Status | Count | Min | Mean | Median | p90 | p95 | p99 | Max |" + in markdown + ) + assert ( + "| `setup` | `completed` | 1 | n/a | n/a | n/a | n/a | n/a | n/a | n/a |" + ) in markdown + assert "Setup individual observations: `1.000 ms`" in markdown + assert ( + "| `mutation.bulk` | `failed` | 2 | 2.000 ms | 3.000 ms | 3.000 ms | " + "4.000 ms | 4.000 ms | 4.000 ms | 4.000 ms |" + ) in markdown + + +def test_validate_report_artifact_accepts_all_not_attempted_without_child_dir( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """A pre-child interruption has no child directory to resolve or require.""" + shape = benchmark_module.Topology(1, 1, 1) + reason = "KeyboardInterrupt: before first child" + report = benchmark_module.RunReport( + shape, + status="cutoff", + cleanup=benchmark_module.CleanupReport( + True, + processes_absent=True, + socket_absent=True, + scratch_absent=True, + ), + ramp=(benchmark_module.RampStep(shape, "not_attempted", reason),), + requested_shapes=(shape,), + ramp_kind="custom", + lane="subprocess", + mode="sync", + warmup=0, + runs=1, + error=reason, + environment=benchmark_module.EnvironmentReport( + "3.10", None, 1, 11, ("ramp",), None + ), + ) + output = tmp_path / "ramp.json" + benchmark_module.write_json_atomic(output, report) + + assert benchmark_module.validate_report_artifact(output) == report + assert not output.with_name("ramp.runs").exists() + + +def test_public_validate_rejects_non_string_child_path_without_traceback( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """A malformed JSON path type must remain a concise artifact error.""" + output = tmp_path / "ramp.json" + _aggregate, _children = _write_completed_ramp_artifact( + benchmark_module, + output, + (benchmark_module.Topology(1, 1, 1),), + ) + payload = json.loads(output.read_text(encoding="utf-8")) + payload["ramp"][0]["report_path"] = 42 + output.write_text(json.dumps(payload), encoding="utf-8") + + completed = _run_cli("validate", "--input", str(output), cwd=tmp_path) + + assert completed.returncode == 2 + assert "invalid benchmark artifact" in completed.stderr + assert "report_path" in completed.stderr + assert "Traceback" not in completed.stderr + + +def test_public_render_rejects_output_aliasing_input_without_overwrite( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """Markdown output must never atomically replace its source JSON evidence.""" + report_path = tmp_path / "report.json" + benchmark_module.write_json_atomic( + report_path, + _completed_executable_report( + benchmark_module, + benchmark_module.Topology(1, 1, 1), + 1, + ), + ) + original = report_path.read_bytes() + + completed = _run_cli( + "render", + "--input", + str(report_path), + "--output", + str(report_path), + cwd=tmp_path, + ) + + assert completed.returncode == 2 + assert "same file" in completed.stderr + assert "Traceback" not in completed.stderr + assert report_path.read_bytes() == original + + +def test_public_render_rejects_output_aliasing_child_json_without_overwrite( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """Markdown output cannot replace any child in the validated JSON tree.""" + report_path = tmp_path / "ramp.json" + _aggregate, child_paths = _write_completed_ramp_artifact( + benchmark_module, + report_path, + (benchmark_module.Topology(1, 1, 1),), + ) + child_path = child_paths[0] + original = child_path.read_bytes() + + completed = _run_cli( + "render", + "--input", + str(report_path), + "--output", + str(child_path), + cwd=tmp_path, + ) + + assert completed.returncode == 2 + assert "JSON artifact tree" in completed.stderr + assert "Traceback" not in completed.stderr + assert child_path.read_bytes() == original + + +def test_ramp_markdown_retains_relative_child_references_only( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """A ramp summary must stay auditable without leaking local absolute paths.""" + output = tmp_path / "ramp.json" + aggregate, _children = _write_completed_ramp_artifact( + benchmark_module, + output, + (benchmark_module.Topology(1, 1, 1),), + ) + + markdown = benchmark_module.render_markdown_summary(output) + + assert "| Shape | Status | Reason | Child report |" in markdown + assert t.cast(str, aggregate.ramp[0].report_path) in markdown + assert str(tmp_path) not in markdown + + +def test_markdown_includes_run_environment_and_exact_cleanup_without_owned_paths( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """A summary must identify the run and cleanup without leaking owned paths.""" + shape = benchmark_module.Topology(1, 1, 1) + report = dataclasses.replace( + _completed_executable_report(benchmark_module, shape, 1), + scratch_path=str(tmp_path / "owned-scratch"), + socket_path=str(tmp_path / "owned-scratch" / "tmux.sock"), + progress_path=str(tmp_path / "progress.jsonl"), + ) + report_path = tmp_path / "run.json" + benchmark_module.write_json_atomic(report_path, report) + + markdown = benchmark_module.render_markdown_summary(report_path) + + for expected in ( + "Runs: `1`", + "Warmup: `0`", + "Seed: `11`", + "Python: `3.10`", + "tmux: `n/a`", + "Revision: `n/a`", + "Processes absent: `true`", + "Socket absent: `true`", + "Scratch absent: `true`", + ): + assert expected in markdown + assert str(tmp_path / "owned-scratch") not in markdown + assert str(tmp_path / "progress.jsonl") not in markdown + + +def test_markdown_escapes_environment_metadata( + benchmark_module: types.ModuleType, + tmp_path: pathlib.Path, +) -> None: + """Environment strings cannot terminate inline code or inject raw HTML.""" + shape = benchmark_module.Topology(1, 1, 1) + hostile = "v`|\n