Map the element type of a sized array (seal moved, Architect-approved) - #4114
Merged
Merged
Conversation
`str` is not a Zig type. The mapper turns it into `[]const u8` everywhere it
looks: bare, optional `?str`, t27's slice `[str]`, Zig's slice `[]str`, and
t27's sized array `[str; N]`. It did not look inside a ZIG-shaped sized array,
`[0]str` / `[2]str` / `[SIZE]str`, which is what the specs actually write.
The slice case only matches a type that ENDS at its `]`, and the `[T; N]` case
only matches t27's own spelling, so `[0]str` fell past both and reached the
scalar mapper as one opaque string. Zig then answered
`use of undeclared identifier 'str'` — 285 occurrences, the largest remaining
failure class in the corpus by a factor of four.
The dimension is copied through untouched; it is a count or a named constant,
not a type. Only the element is mapped, so `[4]u8`, `[SIZE]u8` and `[]u8` are
unchanged.
## Why the seal moved
FROZEN.md §5 step 2, as an Architect-approved hotfix — authorised explicitly.
M1 cargo build --release green
M2 no .t27 spec is touched by this PR
M3 cargo test --release 2715 passed, 0 failed — identical to the
same suite on d9872f1 without the patch
M4 no tests/run_all.sh in the repo; the CI checks on this PR stand in
Old seal 8932aae0c5a95a0528e3e6b8dbbf4e77e6fbb75e55f7d6e7d0a2f92f8b2ff2b6
New seal 403499176a5dc5a7...
## Measured
Generating all 945 specs and running each under `zig test`, with a t27c built
from this branch and one from master:
specs that compile and pass their tests 280 -> 553 (+273)
specs that do not compile 655 -> 382
`undeclared identifier 'str'` 285 -> 0
`use of undeclared identifier` overall 420 -> 135
273 specs started passing. **None stopped.**
945 specs generate either way; none newly fails to generate.
The two literal fixes before this one moved the pass count by +0 and +1, and
were reported that way. This one moves it by 273 because it clears a whole
layer rather than exposing the next one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
This was referenced Sep 17, 2026
gHashTag
added a commit
that referenced
this pull request
Sep 17, 2026
…ailed generations as passes (#4235) * Regenerate the oracle baseline from master, and stop one hung test holding the run ## The baseline recorded a number that never existed tools/oracle/baseline.tsv said `pass 541`. That came from a local t27c built eight days earlier from uncommitted source. Regenerated here with a t27c built from the commit it judges (5719414) and zig 0.16.0: 278 pass 655 do not compile 13 fail or panic at runtime (of 946) A baseline of 541 would have failed the ratchet on every honest run — the nightly job carried continue-on-error precisely because of it. With a true floor, the ratchet can gate. The ratchet compares BEFORE it writes, so against the stale ledger it exited before it could record anything. The regeneration therefore removed the old file first; that is the only way to replace a baseline the script believes. Note: this is master WITHOUT the sized-array type fix (#4114). When that lands, compilation rises to ~553 and this floor should be ratcheted up by the nightly. A floor below the truth is safe; one above it fails every run. ## One test that never returns held the whole oracle open fpga/testbench/gf16_accel_tb compiles, then its test never returns. run.sh had no per-spec limit, so xargs waited on it for 24 minutes after every other spec had finished. In CI it would have burned the job's entire timeout. Now each spec is bounded (ORACLE_TIMEOUT, default 120s) and a hang is its own verdict, TIMEOUT, instead of being misfiled as NOCOMPILE — it compiled fine. The first timeout did not work, and the reason matters. The process that hangs is not zig: `zig test` runs the test binary as a CHILD. Killing zig orphaned the grandchild, which kept the output pipe open, and the command substitution waited on the pipe forever — a 20s limit stayed open for seven minutes. GNU `timeout` signals only its direct child and has the same flaw. The limit now forks the run into its own process group and kills the GROUP. perl, because it is on every CI runner and on macOS, where `timeout` is not. Verified on real specs, through the new wrapper: gf16_accel_tb (hangs) TIMEOUT in 22s, no orphaned test binary left 4 specs PASS here PASS a NOCOMPILE spec NOCOMPILE a TESTFAIL spec TESTFAIL (a panic is not a hang) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * A spec whose generation fails is NOGEN, not a tested fragment — baseline is 189 The baseline this branch first recorded, 278, was itself inflated. run.sh wrote `t27c gen` output into the tree whatever the exit code; the `&&` on that line only decided whether a counter moved. t27c emits what it managed before failing, so a TRUNCATED module stayed in the tree, and when that fragment happened to be valid Zig it ran under `zig test` and scored PASS. Checked directly: 89 of the 278 "passing" specs have a `t27c gen` that exits non-zero. specs/ar/ternary_logic.t27 carries 33 tests; its fragment carried none, compiled, and passed. Four of the specs used earlier to verify the timeout wrapper as "known PASS" were among them. A failed generation now removes the fragment and is recorded as its own verdict, NOGEN. It is appended to the results rather than dropped, because dropping it would shrink the denominator and flatter the pass rate. Scope, stated so it is not over-read: the review witness already refused a failed gen (`if gen && [ -s ]`), and the tree baked into the image already removed one (`gen || rm -f`). So bee verdicts were sound. What was inflated was every CORPUS number this script produced — the baseline, and each before/after count reported from it. Regenerated on master 5719414, t27c built from that commit, zig 0.16.0, 120s per spec: 189 PASS 654 NOCOMPILE 89 NOGEN 13 TESTFAIL 1 TIMEOUT (946) gf16_accel_tb resolved as TIMEOUT inside the run, not a 24-minute hang. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * An oracle run that measured nothing must fail, not report 0 / 0 pass Run from the wrong directory, `find specs` sees nothing, nothing generates, nothing is tested, and every count is zero. The script then printed "oracle: 0 / 0 pass" and exited 0: the ratchet read that as no regression, and any caller read it as a clean result. This happened, rather than being imagined. A copy of run.sh placed at the repository root instead of at tools/oracle/ resolved its root as `../..` from there, climbed out of the repo, and produced exactly that silent success. Zero generated specs now exits 2 with the directory it looked in. The same guard protects the nightly job: an empty checkout or a moved script fails loudly instead of going green. Verified: run against a directory with no specs/, exit code 2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #3960
stris not a Zig type. The mapper turns it into[]const u8everywhere it looks — bare, optional?str, t27's slice[str], Zig's slice[]str, t27's sized array[str; N]. It did not look inside a Zig-shaped sized array,[0]str/[2]str/[SIZE]str, which is what the specs actually write.The slice case only matches a type that ENDS at its
]; the[T; N]case only matches t27's own spelling. So[0]strfell past both, reached the scalar mapper as one opaque string, and Zig answereduse of undeclared identifier 'str'— 285 occurrences, the largest remaining failure class by a factor of four.The dimension is copied through untouched — it is a count or a named constant, not a type — so
[4]u8,[SIZE]u8and[]u8are unchanged.Freeze ceremony (FROZEN.md §5)
Architect-approved hotfix, authorised explicitly.
cargo build --releaset27c parseon touched specs.t27spec is touchedcargo test --released9872f137without the patchtests/run_all.shMeasured
All 945 specs generated and run under
zig test, with at27cbuilt from this branch and one frommaster:zig testexits 0)undeclared identifier 'str'use of undeclared identifieroverall273 specs started passing. None stopped. Zero newly fail to generate.
The two literal fixes before this (#3962, #3973) moved the pass count by +0 and +1 and were reported that way. This one moves it by 273 because it clears a layer outright rather than exposing the next.
On the red checks
emit-bitexact,spec-guardsand the ratchet fail on the same pre-existing set of specs that do not parse — red before this series began. Verified, not asserted: 945 specs generate with at27cfrom either side, with zero newly failing to generate.Correction to the numbers above
The first version of this description said 553 specs "compile and pass their tests". That overstated it.
zig testexits 0 for a file with no test in it, and 347 of the 553 contain no test at all — for those, passing means only that the generated Zig compiles.What this change does, stated precisely: it fixes a compile error, so 273 more specs now compile (280 → 553). The number that is actually verified by a test rose by 55 (151 → 206). The other 218 newly compiling specs have nothing to run.
Both are real. They are not the same claim, and only the first was the right thing to put in a headline.
Second correction — the absolute numbers were inflated by failed generations
The oracle script that produced every count above kept the fragment a failed
t27c genleft behind and tested it; when the fragment happened to be valid Zig it scored PASS. 89 specs on master are in that state (fixed in #4235, which records them as NOGEN instead).Re-measured with that fix, on master
5719414eewith and without this change, same compiler source, zig 0.16.0:t27c genfailed (NOGEN)The deltas stand: +273 compile, +55 tested, 273 started passing, none stopped. The 89 failed generations inflated both sides equally, so they cancelled in the difference. The absolute figures above — 280 and 553 — did not, and should be read as 189 and 462.