Repository navigation
feat(t27c): test-report counts asserts and flags vacuous passes (Closes #6509) - #6788
Merged
Merged
Conversation
#6509) `t27c test-report` now builds the generated Zig with every runtime assert counted -- `assert` (both lowered shapes), the prelude `assert_eq`, and the `std.testing.expect*` checks specs write themselves -- and prints, per test, the number of asserts executed, then `vacuous passes N of M`. A pass with 0 is VACUOUS: the trivial backend passes it too (T730, epic #6488). Semantics mirror t27b `pass_vacuous` (#6115): callees counted, once per execution, all-constant asserts and comptime evaluation not counted. The count is a text rewrite of the generated source inside test_report.rs; codegen and compiler.rs are untouched, so FROZEN_HASH and seals do not move. If the counting build does not compile, the plain build runs exactly as before and the report says the count is unknown. The lines the t27b and lab parsers read (BLOCKED, tests N, FAIL N, pass/FAIL <name>) are unchanged; the new lines come after them. Tree mode adds checked / VACUOUS / uncounted pass totals and the number of specs whose every pass is vacuous. Fixture bootstrap/tests/fixtures/vacuity/vacuity.t27: empty test 0, folded constant assert 0, two asserts 2, a 4-iteration loop 4 -- vacuous 2 of 4. Foreign Rust: label owner-approved-foreign on #6509 (owner, 2026-10-06) approves these edits to existing Rust in bootstrap/ for this issue; the two paths are entered in tools/policy/foreign-exceptions.txt. Refs #6488 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gHashTag
added
the
owner-approved-foreign
gHashTag
enabled auto-merge (squash)
October 6, 2026 12:11
This was referenced Oct 6, 2026
Contributor
This was referenced Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #6509
Refs #6488
What
t27c test-reportnow reports, per test, how many runtime asserts the test executed, and a summary line counting vacuous passes -- passes that executed 0 asserts. Such a pass certifies nothing: the trivial backend that compiles every test toreturnpasses it too (sieve filter C10, theorem T730 inspecs/compiler/theory/toolchain.t27). Semantics mirror t27bpass_vacuous(#6115,cli/t27b/src/eval.rsInterp::asserts): every assert executed at run time counts, callees included, once per execution (a loop counts per iteration); an assert whose operands are all compile-time constants, and comptime evaluation (invariants), do not count.Single spec:
Tree (
--all) addschecked passes,VACUOUS passes,uncounted passesandspecs every pass vacuous(t27b's per-filepass_vacuous).How
bootstrap/src/test_report.rs(instrument): the two loweredassertshapes (if (!(c)) __t27_assert_fail(..)/@panic(..)), the preludeassert_eq, and thestd.testing.expect*checks specs write themselves. A tuple literal tells Zig which operands are comptime-known, so constant-folded asserts are not counted;@inComptime()excludes comptime evaluation. The counting runner printsasserts<TAB>nafter a passing test.compiler.rsare untouched, so FROZEN_HASH and the seals do not move (no reseal needed; CONTRIBUTING 'Build speed' does not apply). No overlap with the gen-* work in t27c gen emits code for specs typecheck refuses (35 misread pairs stay 'green') #6446.parse_test_report/parse_test_verdicts(cli/t27b/src/blockers.rs, contrib/railway/t27b-lab/lab.py),string_eq_zig.rsand the xilinx7 bench workflow read (BLOCKED,tests N,FAIL N,pass <name>) are unchanged; every new line comes after them and starts with a count,-,?orvacuous.seal --saverecords are unchanged.Test
bootstrap/tests/fixtures/vacuity/vacuity.t27: an empty test (0, VACUOUS), a constant-only assert (0, VACUOUS), two asserts (2), a 4-iteration loop (4) --vacuous passes 2 of 4. Unit testan_empty_test_is_a_vacuous_pass_and_a_checked_test_is_notruns it end to end.std.testing.allocator,expectErrorleft alone), the refusal on an unrecognisedassert_eq, the counting runner contract, and that no new line can be read aspass/FAIL/tests/BLOCKED.Corpus numbers (Railway t27c lab, zig 0.16.0)
t27c test-reportover every spec underspecs/(scratch excluded), run per spec:pass_vacuousanalogue)Largest:
automation/browser-pod-restart13 of 18,physics/gamma-conflict10 of 10,numeric/gfternary8 of 13,physics/e8_lqg_bridge8 of 8,physics/hslm_benchmark7 of 7.No measurement moved: the same 1318 specs were run with the base binary (master 5bad205) and this branch's binary, and every spec's BLOCKED status,
tests/pass/FAILtotals and per-test pass/FAIL verdicts are identical (0 differences). (Units differ from the t27b lab's 281 vacuous of 723: those are files, these are tests.)Lab
/runs/a6edf9e92da7f9e35b531aa364515eba7ecd2ac7.json): green -- frozen-hash matches (3c78f3c7ffb7, compiler.rs untouched), build, suiteRATCHET CLEAN, lean, seal-currency, seal-coverage, specs-parse, specs-generate, misread all exit 0.cargo test --release -p t27c --bin t27c test_report14 passed (5 new);--test string_eq_zig4 passed;--test seal_refuses_failing_tests4 passed.Foreign code approval
This PR edits existing Rust in
bootstrap/(bootstrap/src/test_report.rs,bootstrap/src/main.rs). The owner approved it on 2026-10-06 by putting the labelowner-approved-foreignon issue #6509, which approves edits to existing Rust in bootstrap/ (t27c) for this issue. The two paths are entered intools/policy/foreign-exceptions.txtwith that citation, and this PR carries the same label. The fixture is.t27.🤖 Generated with Claude Code