Skip to content

feat(infra): t27c lab on Railway + t27c steward profile agent (Closes #6093, Closes #6094) - #6097

Merged
gHashTag merged 3 commits into
masterfrom
claude/t27c-railway-lab
Oct 4, 2026
Merged

gHashTag merged 3 commits into
masterfrom
claude/t27c-railway-lab

Conversation

@gHashTag

@gHashTag gHashTag commented Oct 4, 2026

Copy link
Copy Markdown
Owner

Closes #6093
Closes #6094

Refs #6092 (the epic: board card and report channel)

Why

On 2026-10-04 the workstation ran at load average 600-840. One cargo build --release -p t27c -p tri took 15 min 32 s there, and one t27c suite --corpus-only --ratchet took 1 h 47 min. The owner asked for every experiment to run on Railway, for the work to be improved and monitored through the Queen, and for a profile agent to own it.

What

  • infra/t27c-lab/: lab.py (stdlib only), Dockerfile (rust:1-bookworm + python3 + git) and railway.json. It is deployed as Railway project/service t27c-lab with a /data volume, at https://t27c-lab-production.up.railway.app.
    • Every 180 s it runs git ls-remote --heads and takes each new head of a branch matching T27_WATCH. It runs the eight gates CI runs, one commit at a time: FROZEN_HASH, build, suite --ratchet, the Lean completeness test, seal currency, seal coverage, specs-still-parse and specs-generate. If the build fails, the rest are skipped.
    • Results are served over GET only: /latest.json (heads, verdicts, queue, the running gate), /runs/<sha>.json, /runs/<sha>/<gate>.log. Paths are whitelisted by regex.
    • It holds no secret, accepts no request that changes anything, and never writes to GitHub. It builds refs/heads only, so a fork's pull request never runs. A queued commit that its branch has moved past is dropped. A run cut short by a redeploy is run again.
    • railway ssh ... python3 /app/lab.py enqueue <branch|sha> queues an unwatched commit. That needs the Railway login, not anything the service holds.
  • .claude/agents/t27c-steward.md: the profile agent for epic EPIC: t27c silent misreads to zero and kept there -- lab on Railway, steward, follow-ups #6092. It holds the map, one round (claim, observe, advance, report, learn) and the never-list. It points at lab.py and tools/queen/task_shape.py rather than copying them. A scheduled task runs it from this file.
  • docs/now/2026-10-04-t27c-railway-lab.md.

Measured on the lab

Run of the #5947 head f521a8a, cold volume, 24 vCPU / 24 GB:

gate exit seconds summary
frozen-hash 0 0.0 FROZEN_HASH matches compiler.rs
build 0 30.0 122 crates compiled
suite 0 94.8 RATCHET CLEAN, ledger 112 / 112
lean 0 29.0 1 passed
seal-currency 0 8.0
seal-coverage 0 29.8 1449 seals, 1325 hold, 124 known-broken
specs-parse 0 5.6 no spec that parsed at the base stopped parsing
specs-generate 0 6.4 1251 specs, 1245 generate

tri misread --list, run on the lab over railway ssh: 0 of 35 pairs silent.

Checks run

  • python3 infra/t27c-lab/lab.py --self-check: ok. It is also run during the image build, so a broken self-check fails the deploy.
  • Local smoke test with an empty watch list: /health returns ok, /latest.json is well formed, and both /../etc/passwd and /runs/abc.json return 404.
  • PR_BASE_SHA=$(git merge-base origin/master HEAD) python3 tools/check_now_entry_shape.py: OK.
  • ASCII: 0 non-ASCII bytes in the five files.
  • This branch matches claude/t27c-*, so the lab runs it too: /runs/<this head>.json.

Not here

  • Railway reports railway.json (config as code) as deprecated, still working until 2026-12-01. deployment/check-runner and the other services use it too; migrating them all is a separate change.

🤖 Generated with Claude Code

…teward agent (Closes #6093, Closes #6094)

The workstation ran at load average 600-840 on 2026-10-04: one release build of
t27c took 15 min 32 s and one ratchet suite 1 h 47 min. The lab runs the same
eight gates CI runs (FROZEN_HASH, build, suite --ratchet, Lean completeness,
seal currency, seal coverage, specs-still-parse, specs-generate) on every new
head of a watched branch, on Railway. Cold on 24 vCPU the #5947 head took
3 min 29 s for all eight, green.

lab.py is stdlib only, serves GET-only JSON, holds no secret and never writes
to GitHub; it builds refs/heads only, so a fork's pull request never runs.

.claude/agents/t27c-steward.md is the one home of the loop for epic #6092:
read the lab, hand spec-source fixes to the Queen as tasks that pass
tools/queen/task_shape.py, run compiler fixes one lane at a time, report on
the epic, never merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-04 14:18:48 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 41
PRs with All Checks Green 9
READY 8
FAILING 41
PENDING 0
NO CHECKS YET 0

These columns do not partition: 8 + 41 + 0 + 0 = 49, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=a47525693295 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

The steward reports `tri misread` silent/refused counts every round and may
not run anything on the workstation, but the lab did not produce them. It now
runs `tri misread --list` as a `misread` gate and parses the table into
`counts` (pairs / refused / silent). The gate is REPORT_ONLY: published,
left out of the verdict, because master still carries silent pairs and CI
does not run it. Self-check covers the parse, a failed control (None) and
the verdict exclusion.

A re-seal done on the lab without zig wrote "zig not on PATH" into the
seal's `tests` field, which no committed seal says (caught on the wp18
re-seal of #5964 today). The image now carries zig 0.16.0, sha256-pinned, at
/opt/zig and off the gates' PATH, so the gates still match CI. The steward's
file says how to re-seal there, and to re-fetch before pushing (#6003 had
been re-sealed by another session meanwhile).

Refs #6092, Refs #6093, Refs #6094

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-04 15:13:48 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 42
PRs with All Checks Green 8
READY 8
FAILING 42
PENDING 0
NO CHECKS YET 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=32e37152eb23 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

`tri misread --list` on master still prints the older report, one count per
shape and no refused column. The gate's counts parser only knew the new
three-column table, so master's run published counts: null. It now reads
both: the old format gives pairs = silent = the sum of the shape rows and
refused = null (not measured, not 0), and the summary says so. On the
log of run 9289aad (master cdc02ad): 41 of 41 pair(s) silent,
refusals not measured. Self-check covers both formats.

Refs #6092

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-04 15:29:57 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 41
PRs with All Checks Green 9
READY 6
FAILING 41
PENDING 0
NO CHECKS YET 0

These columns do not partition: 6 + 41 + 0 + 0 = 47, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=32e37152eb23 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant