Repository navigation
feat(t27b): per-test reference verdicts, reference_disagree apart from jit/interp mismatch - #6530
Merged
Merged
Conversation
…m jit/interp mismatch t27b's `mismatch` compares t27b's JIT with t27b's own interpreter. Both run lower.rs's IR, so a lowering defect they share agrees with itself. The reference comparison was per file: a file both sides failed counted as agreement even when different tests failed. - blockers.rs: parse_test_verdicts reads t27c test-report --verbose per test; t27b_verdicts names repeats like t27c's __dupN; disagreements; the reference cache rows carry a tests= field (old rows re-run). - corpus --reference: reference_disagree (files) and _tests, counted apart; JSON totals add jit_interp_mismatch, reference_disagree, reference_disagree_tests, reference_compared; records carry test_verdicts, reference_tests, reference_disagree. Exit 4 on either. - steward.t27: is_alarm_tests and lanes_stop with tests; gen/c steward.c regenerated (tri t27b gen-check on the t27c lab: SAME 904af7fc1f22). - tri t27b status/doctor: honest line and LAB-REF-DISAGREE; tri t27b diff. - steward CI test counts the spec's test blocks instead of pinning a number. - lab.py mirrors the per-test merge (takes effect on the next redeploy). Closes #6441 Refs #6063 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gHashTag
added
the
owner-approved-foreign
gHashTag
enabled auto-merge
October 5, 2026 16:03
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #6441
Refs #6063
Problem
t27b's
mismatchcompares t27b's JIT with t27b's own interpreter. Both run lower.rs's IR, so a lowering defect they share agrees with itself. The only comparison with the reference (t27c + zig) was per file: a file both sides failed counted as agreement even when different tests failed.Change
cli/t27b/src/blockers.rs:parse_test_verdictsreadst27c test-report --verboseper test.t27b_verdictsnames repeated tests the way t27c does (__dupN).disagreementslists the differences.tests=field. Old rows are re-run.cli/t27b/src/main.rs,corpus --reference:jit_interp_mismatch(same value asmismatch, kept for back-compat),reference_disagree(files),reference_disagree_testsandreference_compared.test_verdicts,reference_testsandreference_disagree.specs/tri/t27b/steward.t27:is_alarm_tests: a shared fail is no alarm only when the same tests fail.lanes_stop: a jit/interp mismatch or a reference disagreement stops the lanes.gen/c/tri/t27b/steward.cregenerated.tri t27b gen-checkon the t27c lab: SAME 904af7fc1f22.tri t27b:statusshows jit/interp mismatch and reference disagree on their own line.doctorraises LAB-REF-DISAGREE.tri t27b diff SPEC [--run R]prints both verdict lists side by side.contrib/railway/t27b-lab/lab.pydoes the same per-test merge. It takes effect after a lab redeploy.scripts/ci/test_the_t27b_steward_decides_in_t27.pyno longer pins "N tests passed". It counts the spec'stestblocks, so a spec-only PR that adds tests cannot break it.Measurement
Branch build on the Railway t27b lab, 2026-10-05 15:39-15:47Z:
qemu-aarch64 t27b corpus specs --blockers --reference t27c --jobs 12, exit 0.gen_vector2d.t27failsVec2_lengthon both.specs/ui/viewport.t27was not compared: the reference is blocked on it.No disagreement was found, so no root-cause issues were filed.
Owner exception
Hand-written Rust/Python allowed by the owner on 2026-10-05 (chat: fix it, do what is best). Debt to be replaced by t27-generated code under #6198.
specs/tri/t27b/steward.t27gen/c/tri/t27b/steward.c(t27c gen-c, from the t27c lab)cli/t27b/src/blockers.rs,cli/t27b/src/main.rs,cli/t27b/tests/blockers.rs,cli/t27b/tests/refcache.rscontrib/railway/t27b-lab/lab.py,scripts/tri_loop/t27b.py,scripts/tri_loop/t27b_rules.py,scripts/ci/test_the_t27b_steward_decides_in_t27.py,scripts/ci/test_a_t27b_tick_reads_before_it_acts.pydocs/now/2026-10-05-t27b-reference-disagree.mdVerification
blockers10/10 andrefcache5/5, run locally and on the lab under qemu.scripts/ci/test_*t27b*.pypass.check_now_entry_shapeOK.Not in this PR: issue item 4 (an interpreter Fuel/Depth stop reported as unknown).
🤖 Generated with Claude Code