diff --git a/.claude/agents/t27c-steward.md b/.claude/agents/t27c-steward.md new file mode 100644 index 0000000000..440ea44693 --- /dev/null +++ b/.claude/agents/t27c-steward.md @@ -0,0 +1,45 @@ +--- +name: t27c-steward +description: t27c Steward - keeps the silent-misread stack (epic #6092) moving to zero and holding there; reads the Railway t27c lab, hands spec-source fixes to the Queen as tasks she can run, drives compiler fixes through one lane at a time, reports on the epic +color: "#22c55e" +--- + +# t27c Steward + +You own one outcome: every spec-shape pair `tri misread --list` reports is +either refused by `typecheck` or fixed in its spec, every line a backend emits +compiles, and the open pull requests of epic gHashTag/t27#6092 stay green +until the owner merges them. You do not merge, approve or review-approve +anything, and you do not build or run a suite on the workstation (load average +600-840 there on 2026-10-04; the same release build is 15 min 32 s there and +30 s on the lab). + +## Where things are + +- The epic, its stack in merge order and its follow-up list: gHashTag/t27#6092. Read it each round; do not restate it here. +- The lab: `https://t27c-lab-production.up.railway.app/latest.json` (heads, verdicts, queue, the running gate), `/runs/.json` (one run, every gate), `/runs//.log`. What it runs, how it picks heads and how to queue a commit by hand: the docstring of `infra/t27c-lab/lab.py`. Railway project `t27c-lab`, service `t27c-lab`. +- The Queen: `https://trios-agent-server-production.up.railway.app/queen/status` (`lastTick.skipSummary` says why an issue was skipped, `claimed` lists what bees hold) and her board at `https://app.t27.ai/game/kanban`. +- The shape a task needs before the Queen dispatches it: `python3 tools/queen/task_shape.py --issue N` (its docstring holds the rule). The first two tasks filed from this epic, #6095 and #6096, are the template. +- State: `~/.local/state/t27c-steward/ledger.md` (one row per round) and `~/.local/state/t27c-steward/claim.json` (`{"since": "", "pid": N}`). + +## One round + +1. **Claim.** `date -u`; read claim.json. If `since` is under 100 minutes old and `ps -p ` is alive, add a ledger row `| | alive, skipped |` and stop. Otherwise write a new claim. +2. **Observe.** The epic (`gh issue view 6092 --repo gHashTag/t27 --json body,comments`), each stack PR (`gh pr view N --json state,mergeable,headRefOid` and `gh pr checks N`), the lab's `latest.json`, the Queen's `status`, `uptime`. The `tri misread` numbers are the `misread` gate's `counts` in the lab run of master and of the stack's last head; that gate is published and left out of the verdict. +3. **Name a cause only from a run.** A red gate is read from its `/runs//.log` before anything is changed. A check red on the PR and red on the same master commit (`gh api repos/gHashTag/t27/commits//check-runs`) is master's, and goes in the epic's "Master, not this stack" line instead of being fixed on the branch. +4. **Advance**, at most one item of each kind per round: + - *Stack branch red, or behind its base.* Merge the base with `git fetch origin ` + `git merge FETCH_HEAD` in a worktree under `/tmp/t27c-` (never under `.claude/worktrees/`, never a rebase, never a force-push). Recompute `bootstrap/stage0/FROZEN_HASH` when `bootstrap/src/compiler.rs` changed. A seal the merge left stale is re-sealed on the lab (`railway ssh`, a worktree of `/data/src`, `PATH=/opt/zig:$PATH t27c seal --save && tri seals sync-twins`) and only the diff is brought back; without zig on PATH the seal's `tests` field becomes "zig not on PATH", which no committed seal says. `git fetch` the branch again right before pushing: a non-fast-forward means another session is on it, so drop yours and read theirs. Push. The lab picks the head up within one poll; its verdict, not a local build, decides the next step. + - *Spec-source fix* (a refused pair from `tri misread --list`, an orphan seal). File it for the Queen: `## Boundary` with the files, `## Defect`, `## User Scenarios & Testing` (Given/When/Then), `## Requirements` (FR-NNN, MUST), `## Success Criteria` (commands with the output a reference run produced), `Refs #6092`. Then `task_shape.py --issue N`; anything but `ready` is fixed before you move on. Keep at most three such tasks open and unclaimed at once. + - *Compiler fix* (needs cargo, so no bee can do it). One subagent lane at a time, in `/tmp/t27c-` on a branch named `claude/t27c-` or `claude/gen-`, which the lab watches. Its own issue (`Refs #6092`), a commit with `Closes #N`, before/after numbers from the lab in the PR body, a docs/now entry. The lane may run `cargo check` locally; builds, suites and seal checks are read from the lab. + - *A bee's pull request for a task from this epic.* Read its diff against the task's Success Criteria and the lab or CI run of its head. Say what holds and what does not in one comment on the task; do not approve. +5. **Report.** If anything changed, one comment on #6092: what moved (links), each stack PR's state and lab verdict (link the run), `tri misread` silent/refused counts, the ledger cap, open bee tasks and who claimed them, load average, next step. Nothing changed: a ledger row, no comment. +6. **Learn.** A slip this round (a wrong claim, a failed command, a misread check) becomes a ledger row the same round, and a line in this file through the next pull request that touches it. +7. **Done.** When the lab shows the epic's last PR head green and `tri misread --list` reports 0 silent pairs on master, post a final comment on #6092 with the run link and ask the owner whether to stop the scheduled task. + +## Never + +- Merge, approve, enable auto-merge, `gh pr merge --admin`, force-push, rebase a pushed branch, `git reset --hard`, bare `git stash`. +- Print, store or commit a secret. The lab holds none; its Railway variables are `T27_WATCH` and `LAB_POLL_S`, nothing else. Never `PUT /queen/registry`, never `POST /queen/report`. +- Compile or run a suite on the workstation; kill another session's process; switch branches in a worktree a suite is running in. +- Hand-edit files under `gen/` (L2), add a `*.sh` to the critical path (L7), or write non-ASCII into a source file (L3). +- Report a number no run produced. diff --git a/docs/now/2026-10-04-t27c-railway-lab.md b/docs/now/2026-10-04-t27c-railway-lab.md new file mode 100644 index 0000000000..42e7ff1c45 --- /dev/null +++ b/docs/now/2026-10-04-t27c-railway-lab.md @@ -0,0 +1,13 @@ +# NOW -- t27c lab on Railway and the t27c steward (2026-10-04) + +## The compiler's CI gates run on Railway for every watched head (Closes #6093) + +- `infra/t27c-lab/lab.py` (stdlib only) polls `git ls-remote --heads` every 180 s and runs FROZEN_HASH, build, suite `--ratchet`, the Lean completeness test, seal currency, seal coverage, specs-still-parse and specs-generate on each new head of a watched branch, one commit at a time. Results are GET-only JSON: `/latest.json`, `/runs/.json`, `/runs//.log`. +- No secret, no GitHub write, no request that changes anything; `refs/heads` only, so a fork's pull request is never built. A queued commit its branch moved past is dropped. +- Measured: the #5947 head f521a8a35, cold, on 24 vCPU: build 30 s, suite 95 s (RATCHET CLEAN, 112 / 112), all eight gates green in 3 min 29 s. The same build took 15 min 32 s on the workstation at load 780-840, and the same suite 1 h 47 min. +- The lab publishes `tri misread --list` as a `misread` gate with parsed `counts` (pairs / refused / silent), left out of the verdict because master still carries silent pairs; the steward reads its numbers there instead of running the command on the workstation. +- The lab image carries zig 0.16.0 (sha256-pinned) at `/opt/zig`, off the gates' PATH, so a re-seal done on the lab records a real zig compile in the seal's `tests` field. + +## A profile agent owns the loop (Closes #6094) + +- `.claude/agents/t27c-steward.md`: the map, one round (claim, observe, advance, report, learn) and the never-list for epic #6092. It points at `lab.py` and `tools/queen/task_shape.py` instead of copying them. diff --git a/infra/t27c-lab/Dockerfile b/infra/t27c-lab/Dockerfile new file mode 100644 index 0000000000..b5b1913896 --- /dev/null +++ b/infra/t27c-lab/Dockerfile @@ -0,0 +1,25 @@ +# t27c lab: the compiler's CI gates, run on Railway. See lab.py's docstring. +FROM rust:1-bookworm + +RUN apt-get update \ + && apt-get install -y --no-install-recommends python3 git ca-certificates xz-utils \ + && rm -rf /var/lib/apt/lists/* + +# zig for re-seals by hand only (`PATH=/opt/zig:$PATH t27c seal ... --save`), +# so a seal's `tests` field records a real compile. Not on PATH: the gates run +# without zig, as CI's do. 0.16.0 is the zig the seals were made with. +ARG ZIG_SHA256=70e49664a74374b48b51e6f3fdfbf437f6395d42509050588bd49abe52ba3d00 +RUN curl -fsSL -o /tmp/zig.tar.xz https://ziglang.org/download/0.16.0/zig-x86_64-linux-0.16.0.tar.xz \ + && echo "$ZIG_SHA256 /tmp/zig.tar.xz" | sha256sum -c - \ + && mkdir /opt/zig && tar -xJf /tmp/zig.tar.xz -C /opt/zig --strip-components=1 \ + && rm /tmp/zig.tar.xz && /opt/zig/zig version + +ENV CARGO_HOME=/data/cargo \ + LAB_DATA=/data \ + PYTHONUNBUFFERED=1 + +WORKDIR /app +COPY lab.py /app/lab.py +RUN python3 /app/lab.py --self-check + +CMD ["python3", "/app/lab.py"] diff --git a/infra/t27c-lab/lab.py b/infra/t27c-lab/lab.py new file mode 100644 index 0000000000..5f76669186 --- /dev/null +++ b/infra/t27c-lab/lab.py @@ -0,0 +1,656 @@ +#!/usr/bin/env python3 +"""t27c lab: the compiler's checks, run on Railway instead of the owner's Mac. + +Owner, 2026-10-04: run every experiment on Railway, the Mac is overloaded. +Measured that day on the Mac: load average 780-840, one `cargo build --release +-p t27c -p tri` took 15 min 32 s, and one `t27c suite --corpus-only --ratchet` +ran for more than an hour. + +What it does, and all it does: + + * every LAB_POLL_S seconds it runs `git ls-remote --heads` on T27_REPO and + keeps the branches that match T27_WATCH (comma-separated fnmatch globs); + * every head it has not run yet is checked out and run through GATES, one + commit at a time, in the order the globs are listed; + * what happened is published over plain HTTP GET: + + /health "ok" + /latest.json heads, verdicts, queue, the running gate + /runs/.json one run: each gate's command, exit code, + seconds and summary line + /runs//.log one gate's whole output + +The `misread` gate is published and left out of the verdict (REPORT_ONLY): +its "counts" are `tri misread`'s pairs / refused / silent totals, the numbers +epic #6092 drives to zero. + +A re-seal run here by hand should put the image's zig on PATH, so the seal's +`tests` field records a real compile instead of "zig not on PATH": + + PATH=/opt/zig:$PATH t27c seal --save && tri seals sync-twins + +The gates themselves run without zig, as CI's do. + +It holds no secret, accepts no request that changes anything, and never writes +to GitHub. Heads come from refs/heads only, so a pull request from a fork is +never built: only someone who can push to the repository can start a run. + +One more way in, for a pushed commit on a branch nobody watches: through +`railway ssh` (which needs the Railway login, not anything this service holds) + + python3 /app/lab.py enqueue + +drops a request file the poller picks up on its next pass. + +Self-check of the pure parts, no network: python3 lab.py --self-check +""" + +from __future__ import annotations + +import datetime as _dt +import fnmatch +import hashlib +import json +import os +import re +import shutil +import subprocess +import sys +import threading +import time +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path + +REPO = os.environ.get("T27_REPO", "https://github.com/gHashTag/t27.git") +WATCH = [g.strip() for g in os.environ.get("T27_WATCH", "master").split(",") if g.strip()] +DATA = Path(os.environ.get("LAB_DATA", "/data")) +POLL_S = int(os.environ.get("LAB_POLL_S", "180")) +PORT = int(os.environ.get("PORT", "8080")) +KEEP_RUNS = int(os.environ.get("LAB_KEEP_RUNS", "200")) + +SRC = DATA / "src" +TARGET = DATA / "target" +WWW = DATA / "www" +RUNS = WWW / "runs" +REQUESTS = DATA / "requests" + +SHA_RE = re.compile(r"^[0-9a-f]{40}$") +# A branch name as git allows it, minus anything that could leave the refspec. +BRANCH_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._/-]{0,199}$") + +# name, argv run from the checkout, timeout in seconds, the regex whose last +# matching line is the summary (None: the last non-empty line). "{run}" is the +# run's directory. The order is the CI order: nothing runs on a failed build. +GATES = [ + ("frozen-hash", None, 30, None), + ("build", ["cargo", "build", "--release", "-p", "t27c", "-p", "tri"], 3600, + r"Finished|^error"), + ("suite", ["./target/release/t27c", "suite", "--repo-root", ".", "--corpus-only", + "--ratchet", "--json", "{run}/suite.json"], 5400, r"RATCHET"), + ("lean", ["cargo", "test", "--release", "-p", "t27c", "--test", "icarus_lowerable", + "corpus_classifier_matches_lean_completeness"], 3600, r"test result:"), + ("seal-currency", ["python3", "tools/check_seal_currency.py"], 1800, None), + ("seal-coverage", ["python3", "tools/check_seal_coverage.py"], 1800, None), + ("specs-parse", ["python3", "tools/ci/check_specs_still_parse.py", "--base", + "origin/master"], 1800, None), + ("specs-generate", ["python3", "tools/check_specs_generate.py"], 1800, None), + ("misread", ["./target/release/tri", "misread", "--list"], 1800, + r"pair\(s\) are silent|^\s*Nothing found|CONTROL FAILED"), +] +NEEDS_BUILD = {"suite", "lean", "seal-currency", "seal-coverage", "specs-parse", + "specs-generate", "misread"} +# Measured and published, never part of the verdict: `tri misread` counts what +# the epic (#6092) is driving to zero, and master is not there yet. Its counts +# land in the gate as "counts"; a red here is the steward's to read, not a +# reason to call a commit broken that CI would pass. +REPORT_ONLY = {"misread"} + +_lock = threading.Lock() +_state: dict = {"running": None, "queue": [], "heads": {}, "error": None} + + +def now() -> str: + return _dt.datetime.now(_dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + +# ---------------------------------------------------------------- pure parts + +def watched(branch: str, globs: list[str]) -> int | None: + """Index of the first glob the branch matches, or None. Pure.""" + for i, g in enumerate(globs): + if fnmatch.fnmatchcase(branch, g): + return i + return None + + +def parse_heads(ls_remote: str) -> dict[str, str]: + """`git ls-remote --heads` output -> {branch: sha}. Pure.""" + heads = {} + for line in ls_remote.splitlines(): + parts = line.split("\t") + if len(parts) != 2 or not SHA_RE.match(parts[0]): + continue + ref = parts[1] + if ref.startswith("refs/heads/"): + heads[ref[len("refs/heads/"):]] = parts[0] + return heads + + +def plan(heads: dict[str, str], globs: list[str], done: set[str], + queued: set[str]) -> list[dict]: + """New work: one entry per sha not run and not queued, every branch that + points at it, in glob order then branch name. Pure.""" + by_sha: dict[str, dict] = {} + for branch, sha in heads.items(): + rank = watched(branch, globs) + if rank is None or sha in done or sha in queued: + continue + item = by_sha.setdefault(sha, {"sha": sha, "branches": [], "rank": rank}) + item["branches"].append(branch) + item["rank"] = min(item["rank"], rank) + out = sorted(by_sha.values(), key=lambda x: (x["rank"], sorted(x["branches"])[0])) + for item in out: + item["branches"].sort() + del item["rank"] + return out + + +def stale_dropped(queue: list[dict], heads: dict[str, str]) -> list[dict]: + """The queue minus commits no watched branch points at any more, unless + they were asked for by hand. Pure.""" + live = set(heads.values()) + return [q for q in queue if q.get("asked") or q["sha"] in live] + + +def summary(text: str, pattern: str | None) -> str: + """The line a person reads first: the last line matching `pattern`, else + the last non-empty line, cut to 300 characters. Pure.""" + lines = [ln.rstrip() for ln in text.splitlines() if ln.strip()] + if pattern: + rx = re.compile(pattern) + hits = [ln for ln in lines if rx.search(ln)] + if hits: + return hits[-1].strip()[:300] + return lines[-1].strip()[:300] if lines else "" + + +def misread_counts(text: str) -> dict | None: + """`tri misread` output -> {"pairs", "refused", "silent"}, or None when the + command did not reach its table (a failed control, a missing binary). + + Two formats are in the tree. Since #5947 the table has a header and three + columns, " ". Before it (master on + 2026-10-04) one column, " rust: ", under the tool's own + statement that each pair "parses and typechecks": every pair is silent and + refusals were not measured, so "refused" is None there, not 0. Pure.""" + if "CONTROL FAILED" in text: + return None + if "pairs refused silent" in text: + rows = re.findall(r"^\s+(\d+)\s+(\d+)\s+(\d+)\s+\S", text, re.M) + return {"pairs": sum(int(r[0]) for r in rows), + "refused": sum(int(r[1]) for r in rows), + "silent": sum(int(r[2]) for r in rows)} + if "parses and typechecks" in text or "Nothing found" in text: + n = sum(int(r) for r in re.findall(r"^\s+(\d+)\s+[a-z]+:\s", text, re.M)) + return {"pairs": n, "refused": None, "silent": n} + return None + + +def misread_line(counts: dict | None, fallback: str) -> str: + """The misread gate's summary: the counts in words, else the log's line.""" + if not counts: + return fallback + refused = "refusals not measured" if counts["refused"] is None else \ + f"{counts['refused']} refused by typecheck" + return f"{counts['silent']} of {counts['pairs']} pair(s) silent, {refused}" + + +def frozen_hash_ok(root: Path) -> tuple[bool, str]: + """bootstrap/stage0/FROZEN_HASH names the sha256 of bootstrap/src/compiler.rs.""" + src = root / "bootstrap" / "src" / "compiler.rs" + pin = root / "bootstrap" / "stage0" / "FROZEN_HASH" + if not src.exists() or not pin.exists(): + return False, "missing compiler.rs or FROZEN_HASH" + actual = hashlib.sha256(src.read_bytes()).hexdigest() + pinned = pin.read_text().split()[0] if pin.read_text().split() else "" + if actual == pinned: + return True, f"FROZEN_HASH matches compiler.rs ({actual[:12]})" + return False, f"FROZEN_HASH {pinned[:12]} != sha256(compiler.rs) {actual[:12]}" + + +def verdict(gates: list[dict]) -> str: + """green when every gate that ran exited 0 and none was skipped; a + REPORT_ONLY gate is left out. Pure.""" + gates = [g for g in gates if g.get("name") not in REPORT_ONLY] + if not gates: + return "red" + return "green" if all(g.get("exit") == 0 for g in gates) else "red" + + +def safe_path(path: str) -> str | None: + """The file under WWW a GET may read, or None. Pure.""" + path = path.split("?", 1)[0] + if path in ("/latest.json",): + return path.lstrip("/") + m = re.match(r"^/runs/([0-9a-f]{40})\.json$", path) + if m: + return f"runs/{m.group(1)}.json" + m = re.match(r"^/runs/([0-9a-f]{40})/([a-z-]{1,40})\.(log|json)$", path) + if m: + return f"runs/{m.group(1)}/{m.group(2)}.{m.group(3)}" + return None + + +def cpu_quota() -> int: + """CPUs this container may use (cgroup v2 cpu.max), else os.cpu_count().""" + try: + q, p = Path("/sys/fs/cgroup/cpu.max").read_text().split() + if q != "max": + return max(1, int(int(q) / int(p))) + except (OSError, ValueError): + pass + return os.cpu_count() or 1 + + +def mem_gb() -> float | None: + try: + v = Path("/sys/fs/cgroup/memory.max").read_text().strip() + if v != "max": + return round(int(v) / 1e9, 1) + except (OSError, ValueError): + pass + return None + + +# ------------------------------------------------------------- side effects + +def sh(argv: list[str], cwd: Path | None = None, timeout: int = 600, + log: Path | None = None, env: dict | None = None) -> tuple[int, str]: + """Run argv; tee into `log` when given. Returns (exit, output).""" + full_env = dict(os.environ) + full_env.update(env or {}) + start = time.time() + out_f = open(log, "w", encoding="utf-8", errors="replace") if log else None + try: + p = subprocess.Popen(argv, cwd=cwd, stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, env=full_env, text=True, + errors="replace") + chunks = [] + assert p.stdout is not None + for line in p.stdout: + chunks.append(line) + if out_f: + out_f.write(line) + out_f.flush() + if time.time() - start > timeout: + p.kill() + chunks.append(f"\n[lab] killed after {timeout} s\n") + if out_f: + out_f.write(chunks[-1]) + break + code = p.wait() + return (124 if time.time() - start > timeout else code), "".join(chunks) + except FileNotFoundError as e: + msg = f"[lab] {e}\n" + if out_f: + out_f.write(msg) + return 127, msg + finally: + if out_f: + out_f.close() + + +def write_json(path: Path, obj) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(path.suffix + ".tmp") + tmp.write_text(json.dumps(obj, indent=2, sort_keys=True) + "\n") + tmp.replace(path) + + +def done_shas() -> set[str]: + return {p.stem for p in RUNS.glob("*.json") if SHA_RE.match(p.stem)} + + +def publish_latest() -> None: + with _lock: + snap = json.loads(json.dumps(_state)) + recent = [] + for p in sorted(RUNS.glob("*.json"), key=lambda p: p.stat().st_mtime, reverse=True)[:20]: + try: + r = json.loads(p.read_text()) + except (OSError, ValueError): + continue + recent.append({k: r.get(k) for k in ("sha", "branches", "verdict", "red_gates", + "finished")}) + heads = {} + for branch, sha in sorted(snap["heads"].items()): + run = RUNS / f"{sha}.json" + if run.exists(): + try: + v = json.loads(run.read_text()).get("verdict") + except (OSError, ValueError): + v = "unreadable" + elif snap["running"] and snap["running"]["sha"] == sha: + v = "running" + elif any(q["sha"] == sha for q in snap["queue"]): + v = "queued" + else: + v = "not run" + heads[branch] = {"sha": sha, "verdict": v, "run": f"/runs/{sha}.json"} + write_json(WWW / "latest.json", { + "lab": "t27c-lab", + "repo": REPO, + "watch": WATCH, + "updated": now(), + "host": {"cpus": cpu_quota(), "mem_gb": mem_gb()}, + "running": snap["running"], + "queue": snap["queue"], + "heads": heads, + "recent": recent, + "error": snap["error"], + }) + + +def ensure_clone() -> None: + if (SRC / ".git").exists(): + return + SRC.parent.mkdir(parents=True, exist_ok=True) + code, out = sh(["git", "clone", "--filter=blob:none", "--no-checkout", REPO, str(SRC)], + timeout=3600) + if code != 0: + raise RuntimeError(f"clone failed: {summary(out, None)}") + + +def checkout(item: dict) -> None: + refspecs = ["+refs/heads/master:refs/remotes/origin/master"] + for b in item["branches"]: + if BRANCH_RE.match(b) and ".." not in b: + refspecs.append(f"+refs/heads/{b}:refs/remotes/origin/{b}") + code, out = sh(["git", "fetch", "--no-tags", "origin", *refspecs], cwd=SRC, timeout=1800) + if code != 0: + # a request by sha alone: fetch the commit itself + code, out = sh(["git", "fetch", "--no-tags", "origin", item["sha"], + "+refs/heads/master:refs/remotes/origin/master"], cwd=SRC, timeout=1800) + if code != 0: + raise RuntimeError(f"fetch failed: {summary(out, None)}") + for argv in (["git", "checkout", "--detach", "-f", item["sha"]], + ["git", "clean", "-fdx", "-e", "/target"]): + code, out = sh(argv, cwd=SRC, timeout=1800) + if code != 0: + raise RuntimeError(f"{argv[1]} failed: {summary(out, None)}") + link = SRC / "target" + if not link.is_symlink(): + if link.exists(): + shutil.rmtree(link) + TARGET.mkdir(parents=True, exist_ok=True) + link.symlink_to(TARGET) + + +def run_item(item: dict) -> dict: + sha = item["sha"] + run_dir = RUNS / sha + run_dir.mkdir(parents=True, exist_ok=True) + record = {"sha": sha, "branches": item["branches"], "repo": REPO, "started": now(), + "host": {"cpus": cpu_quota(), "mem_gb": mem_gb()}, "gates": []} + env = {"CARGO_TARGET_DIR": str(TARGET), "CARGO_INCREMENTAL": "0", + "CARGO_BUILD_JOBS": str(cpu_quota()), "CARGO_TERM_COLOR": "never"} + try: + checkout(item) + except RuntimeError as e: + record["gates"].append({"name": "checkout", "exit": 1, "seconds": 0, + "summary": str(e)[:300]}) + record.update(finished=now(), verdict="red", red_gates=["checkout"]) + write_json(RUNS / f"{sha}.json", record) + return record + built = True + for name, argv, timeout, pattern in GATES: + with _lock: + if _state["running"]: + _state["running"]["gate"] = name + publish_latest() + gate = {"name": name, "log": f"/runs/{sha}/{name}.log"} + log = run_dir / f"{name}.log" + if name in NEEDS_BUILD and not built: + gate.update(exit=None, seconds=0, summary="skipped: the build failed") + log.write_text("skipped: the build failed\n") + elif argv is None: + t0 = time.time() + ok, line = frozen_hash_ok(SRC) + log.write_text(line + "\n") + gate.update(cmd="sha256 bootstrap/src/compiler.rs vs bootstrap/stage0/FROZEN_HASH", + exit=0 if ok else 1, seconds=round(time.time() - t0, 1), summary=line) + else: + cmd = [a.replace("{run}", str(run_dir)) for a in argv] + t0 = time.time() + code, out = sh(cmd, cwd=SRC, timeout=timeout, log=log, env=env) + gate.update(cmd=" ".join(argv), exit=code, seconds=round(time.time() - t0, 1), + summary=summary(out, pattern)) + if name == "misread": + counts = misread_counts(out) + gate.update(counts=counts, report_only=True, + summary=misread_line(counts, gate["summary"])) + if name == "build" and code != 0: + built = False + record["gates"].append(gate) + write_json(RUNS / f"{sha}.json", dict(record, finished=None, verdict="running")) + red = [g["name"] for g in record["gates"] + if g.get("exit") != 0 and g["name"] not in REPORT_ONLY] + record.update(finished=now(), verdict=verdict(record["gates"]), red_gates=red) + write_json(RUNS / f"{sha}.json", record) + return record + + +def prune() -> None: + runs = sorted(RUNS.glob("*.json"), key=lambda p: p.stat().st_mtime, reverse=True) + with _lock: + keep = set(_state["heads"].values()) + for p in runs[KEEP_RUNS:]: + if p.stem in keep: + continue + p.unlink(missing_ok=True) + shutil.rmtree(RUNS / p.stem, ignore_errors=True) + + +def take_requests(heads: dict[str, str]) -> list[dict]: + """Request files dropped by `lab.py enqueue` -> work items.""" + items = [] + for req in sorted(REQUESTS.glob("*.req")): + ref = req.read_text().strip() + req.unlink(missing_ok=True) + if SHA_RE.match(ref): + items.append({"sha": ref, "branches": [b for b, s in heads.items() if s == ref], + "asked": True}) + elif BRANCH_RE.match(ref) and ref in heads: + items.append({"sha": heads[ref], "branches": [ref], "asked": True}) + return items + + +def poll_once() -> None: + code, out = sh(["git", "ls-remote", "--heads", REPO], timeout=300) + if code != 0: + with _lock: + _state["error"] = f"{now()} ls-remote failed: {summary(out, None)}" + return + heads = parse_heads(out) + mine = {b: s for b, s in heads.items() if watched(b, WATCH) is not None} + done = done_shas() + with _lock: + _state["error"] = None + _state["heads"] = mine + # Only a branch's newest head is worth a run: a queued commit a branch + # has moved past is dropped, so the queue never holds more than one + # entry per watched branch plus what was asked for by hand. + _state["queue"] = stale_dropped(_state["queue"], mine) + queued = {q["sha"] for q in _state["queue"]} + if _state["running"]: + queued.add(_state["running"]["sha"]) + asked = [i for i in take_requests(heads) if i["sha"] not in queued] + _state["queue"] = asked + _state["queue"] + plan(mine, WATCH, done, + queued | {i["sha"] for i in asked}) + + +def poller() -> None: + while True: + try: + poll_once() + prune() + publish_latest() + except Exception as e: # keep polling; the error is published + with _lock: + _state["error"] = f"{now()} poll: {e!r}"[:500] + time.sleep(POLL_S) + + +def worker() -> None: + while True: + item = None + with _lock: + if _state["queue"]: + item = _state["queue"].pop(0) + _state["running"] = {"sha": item["sha"], "branches": item["branches"], + "gate": None, "since": now()} + if item is None: + time.sleep(10) + continue + try: + ensure_clone() + run_item(item) + except Exception as e: + write_json(RUNS / f"{item['sha']}.json", { + "sha": item["sha"], "branches": item["branches"], "finished": now(), + "verdict": "red", "red_gates": ["lab"], + "gates": [{"name": "lab", "exit": 1, "seconds": 0, "summary": repr(e)[:300]}]}) + finally: + with _lock: + _state["running"] = None + publish_latest() + + +class Handler(BaseHTTPRequestHandler): + def do_GET(self): # noqa: N802 -- http.server's name + if self.path in ("/health", "/health/"): + return self._send(200, b"ok\n", "text/plain") + if self.path == "/": + body = ("t27c lab: GET /latest.json, /runs/.json, /runs//.log\n" + f"watching {', '.join(WATCH)} on {REPO}\n").encode() + return self._send(200, body, "text/plain") + rel = safe_path(self.path) + if rel is None: + return self._send(404, b"not found\n", "text/plain") + f = WWW / rel + if not f.is_file(): + return self._send(404, b"not found\n", "text/plain") + ctype = "application/json" if rel.endswith(".json") else "text/plain; charset=utf-8" + return self._send(200, f.read_bytes(), ctype) + + def _send(self, code: int, body: bytes, ctype: str): + self.send_response(code) + self.send_header("Content-Type", ctype) + self.send_header("Content-Length", str(len(body))) + self.send_header("Cache-Control", "no-store") + self.send_header("Access-Control-Allow-Origin", "*") + self.end_headers() + self.wfile.write(body) + + def log_message(self, fmt, *args): + pass + + +def serve() -> None: + for d in (RUNS, REQUESTS, TARGET): + d.mkdir(parents=True, exist_ok=True) + # A run cut short by a redeploy is run again from the start. + for p in RUNS.glob("*.json"): + try: + if json.loads(p.read_text()).get("verdict") == "running": + p.unlink() + except (OSError, ValueError): + p.unlink(missing_ok=True) + publish_latest() + threading.Thread(target=poller, daemon=True).start() + threading.Thread(target=worker, daemon=True).start() + ThreadingHTTPServer(("0.0.0.0", PORT), Handler).serve_forever() + + +def enqueue(ref: str) -> int: + if not (SHA_RE.match(ref) or (BRANCH_RE.match(ref) and ".." not in ref)): + print(f"not a branch name or a 40-hex sha: {ref!r}", file=sys.stderr) + return 2 + REQUESTS.mkdir(parents=True, exist_ok=True) + name = hashlib.sha256(ref.encode()).hexdigest()[:16] + (REQUESTS / f"{name}.req").write_text(ref + "\n") + print(f"queued {ref}; picked up within {POLL_S} s, see /latest.json") + return 0 + + +def self_check() -> int: + globs = ["master", "claude/t27c-*"] + assert watched("master", globs) == 0 + assert watched("claude/t27c-match-tail-expr", globs) == 1 + assert watched("claude/gen-zig-discard-call", globs) is None + assert watched("refs/pull/1/head", globs) is None + a, b = "a" * 40, "b" * 40 + heads = parse_heads(f"{a}\trefs/heads/master\n{b}\trefs/heads/claude/t27c-x\n" + f"{b}\trefs/pull/9/head\nnot-a-sha\trefs/heads/y\n") + assert heads == {"master": a, "claude/t27c-x": b}, heads + p = plan({"claude/t27c-x": b, "claude/t27c-y": b, "master": a}, globs, set(), set()) + assert [i["sha"] for i in p] == [a, b], p + assert p[1]["branches"] == ["claude/t27c-x", "claude/t27c-y"] + assert plan(heads, globs, {a}, {b}) == [] + c = "c" * 40 + q = [{"sha": a}, {"sha": c}, {"sha": c, "asked": True}] + assert stale_dropped(q, {"master": a}) == [{"sha": a}, {"sha": c, "asked": True}] + assert summary("x\nRATCHET CLEAN 113/113\ny\n", r"RATCHET") == "RATCHET CLEAN 113/113" + assert summary("x\n\nlast\n\n", None) == "last" + assert summary("", r"RATCHET") == "" + assert verdict([{"exit": 0}, {"exit": 0}]) == "green" + assert verdict([{"exit": 0}, {"exit": None}]) == "red" + assert verdict([]) == "red" + assert verdict([{"name": "build", "exit": 0}, {"name": "misread", "exit": 1}]) == "green" + assert verdict([{"name": "misread", "exit": 0}]) == "red" + table = (" pairs refused silent\n" + " 12 12 0 an empty type slot\n" + " 5 3 2 a colon inside a field type\n\n" + " 2 of 17 pair(s) are silent: the spec parses\n") + assert misread_counts(table) == {"pairs": 17, "refused": 15, "silent": 2} + assert misread_counts(" CONTROL FAILED -- these shapes did not fire\n") is None + assert misread_counts("error: target/release/t27c is not built\n") is None + old = (" generated for 1178 of 1190 spec(s)\n\n" + " 28 rust: `pub f: ,` field with no type\n" + " specs/tools/registry.t27\n" + " 1 c: `0 f;` literal in type position\n\n" + " Each of these parses and typechecks. The count above is a count of\n") + assert misread_counts(old) == {"pairs": 29, "refused": None, "silent": 29} + assert misread_line(misread_counts(old), "") == \ + "29 of 29 pair(s) silent, refusals not measured" + assert misread_line(misread_counts(table), "") == \ + "2 of 17 pair(s) silent, 15 refused by typecheck" + assert misread_counts(" pairs refused silent\n\n Nothing found\n") == \ + {"pairs": 0, "refused": 0, "silent": 0} + assert summary(table, GATES[-1][3]).startswith("2 of 17 pair(s) are silent") + assert safe_path("/latest.json") == "latest.json" + assert safe_path(f"/runs/{a}.json") == f"runs/{a}.json" + assert safe_path(f"/runs/{a}/suite.log") == f"runs/{a}/suite.log" + for bad in ("/../etc/passwd", f"/runs/{a}/../../x.log", "/runs/abc.json", + f"/runs/{a}/Suite.log", "/data/src/.git/config"): + assert safe_path(bad) is None, bad + assert BRANCH_RE.match("claude/t27c-match-tail-expr") + assert not BRANCH_RE.match("-x") and not BRANCH_RE.match("a b") + print("lab self-check: ok") + return 0 + + +def main(argv: list[str]) -> int: + if argv[1:2] == ["--self-check"]: + return self_check() + if argv[1:2] == ["enqueue"] and len(argv) == 3: + return enqueue(argv[2]) + if argv[1:] == []: + serve() + return 0 + print(__doc__) + return 2 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/infra/t27c-lab/railway.json b/infra/t27c-lab/railway.json new file mode 100644 index 0000000000..925019371c --- /dev/null +++ b/infra/t27c-lab/railway.json @@ -0,0 +1,12 @@ +{ + "$schema": "https://railway.com/railway.schema.json", + "build": { + "builder": "DOCKERFILE", + "dockerfilePath": "Dockerfile" + }, + "deploy": { + "healthcheckPath": "/health", + "healthcheckTimeout": 120, + "restartPolicyType": "ALWAYS" + } +}