Skip to content

feat(tri): tri t27b -- honest in-reference ratio, compile-only passes, per-spec delta - #6185

Merged
gHashTag merged 2 commits into
masterfrom
claude/tri-t27b-delta
Oct 4, 2026
Merged

gHashTag merged 2 commits into
masterfrom
claude/tri-t27b-delta

Conversation

@gHashTag

@gHashTag gHashTag commented Oct 4, 2026

Copy link
Copy Markdown
Owner

Closes #6184
Refs #6063

Why

tri t27b status (from #6114) printed t27b 70 / reference 648, and the steward posted that number to #6063. It divides every t27b pass by the reference's pass count, but 14 of those 70 are specs the reference itself fails. The honest figure is 56 of 648 (8.6%). The epic comment has been corrected.

What changes

  • status: quotes in-reference passes / reference passes. From the lab's per-spec results it splits them into 27 that ran a test or invariant and 29 compile-only (0 tests, 0 invariants). Out-of-reference passes and frontend rejects go on their own "not counted" line.
  • doctor: LAB-FRONTEND-DISAGREES. Live: specs/port/scripts/gen_w378.t27, where t27b's typecheck rejects I32 -> ?i32 and the reference passes. mismatch did not count it.
  • delta (new): tri t27b delta [--from SHA] [--to SHA], a per-spec ratchet over <lab>/runs/<sha>.json. REGRESSED, NEW-MISMATCH and CHECK-LOST exit 1; GAINED and REF-MOVED are reported. The default --from is the run that finished last before --to, chosen by finished and not by sha (the listing is sorted by sha).

Live, delta over f59fd76 (branch) -> dd3864d (master): 47/646 -> 56/648, 9 gained, 0 regressions, 1 reference move (specs/compiler/optimizer.t27, blocked -> pass).

Tests

scripts/ci/test_a_t27b_tick_reads_before_it_acts.py (already in loop-tools-gate) passes locally. New checks cover the honest card, the frontend code, five delta transitions (with an unchanged spec that must stay silent), default---from ordering, and an absent run (exit 2). Mutation control: with the numerator switched back to t27b_pass and REGRESSED disabled, 4 checks go red. No workflow step was added, so the census is unchanged.

🤖 Generated with Claude Code

…, per-spec delta

`tri t27b status` printed "t27b 70 / reference 648" and the steward
posted it; 14 of the 70 are passes where the reference fails. The card
now quotes in-reference passes (56 of 648), splits them into checked (27)
and compile-only (29), and lists out-of-reference passes and frontend
rejects separately. doctor gains LAB-FRONTEND-DISAGREES. New
`tri t27b delta` compares two lab runs per spec (REGRESSED, NEW-MISMATCH,
CHECK-LOST exit 1). No workflow step added; census unchanged.

Closes #6184

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-04 16:31:48 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 44
PRs with All Checks Green 6
READY 4
FAILING 44
PENDING 0
NO CHECKS YET 0

These columns do not partition: 4 + 44 + 0 + 0 = 48, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=4a3d2f43c9f5 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-04 16:49:54 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 44
PRs with All Checks Green 6
READY 4
FAILING 44
PENDING 0
NO CHECKS YET 0

These columns do not partition: 4 + 44 + 0 + 0 = 48, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=4a3d2f43c9f5 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@gHashTag
gHashTag merged commit cf4f3bb into master Oct 4, 2026
25 checks passed
gHashTag added a commit that referenced this pull request Oct 4, 2026
…894f

Conflict in scripts/tri_loop/t27b.py: both `delta` (#6185) and
`ratchet` are kept; `tri t27b` accepts status|doctor|delta|ratchet.

Ledger re-blessed from lab run e0f894f: 75 pass, 582 not pass
(max_not_pass 592 -> 582). faculty_board.t27 is pass -> frontend
because typecheck refuses its typeless field; recorded as
reference-bug (the reference path never typechecks).

Refs #6115 #6063

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tri t27b: honest in-reference ratio, compile-only passes, frontend disagreements, per-spec delta between lab runs

1 participant