Regenerate the oracle baseline from master (189), and stop counting failed generations as passes - #4235
Merged
Conversation
…lding the run ## The baseline recorded a number that never existed tools/oracle/baseline.tsv said `pass 541`. That came from a local t27c built eight days earlier from uncommitted source. Regenerated here with a t27c built from the commit it judges (5719414) and zig 0.16.0: 278 pass 655 do not compile 13 fail or panic at runtime (of 946) A baseline of 541 would have failed the ratchet on every honest run — the nightly job carried continue-on-error precisely because of it. With a true floor, the ratchet can gate. The ratchet compares BEFORE it writes, so against the stale ledger it exited before it could record anything. The regeneration therefore removed the old file first; that is the only way to replace a baseline the script believes. Note: this is master WITHOUT the sized-array type fix (#4114). When that lands, compilation rises to ~553 and this floor should be ratcheted up by the nightly. A floor below the truth is safe; one above it fails every run. ## One test that never returns held the whole oracle open fpga/testbench/gf16_accel_tb compiles, then its test never returns. run.sh had no per-spec limit, so xargs waited on it for 24 minutes after every other spec had finished. In CI it would have burned the job's entire timeout. Now each spec is bounded (ORACLE_TIMEOUT, default 120s) and a hang is its own verdict, TIMEOUT, instead of being misfiled as NOCOMPILE — it compiled fine. The first timeout did not work, and the reason matters. The process that hangs is not zig: `zig test` runs the test binary as a CHILD. Killing zig orphaned the grandchild, which kept the output pipe open, and the command substitution waited on the pipe forever — a 20s limit stayed open for seven minutes. GNU `timeout` signals only its direct child and has the same flaw. The limit now forks the run into its own process group and kills the GROUP. perl, because it is on every CI runner and on macOS, where `timeout` is not. Verified on real specs, through the new wrapper: gf16_accel_tb (hangs) TIMEOUT in 22s, no orphaned test binary left 4 specs PASS here PASS a NOCOMPILE spec NOCOMPILE a TESTFAIL spec TESTFAIL (a panic is not a hang) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gHashTag
added a commit
that referenced
this pull request
Sep 17, 2026
tools/oracle/run.sh now bounds each spec and emits TIMEOUT for a test that never returns (#4235). Without a row for it the summary would silently drop those specs from every column. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gHashTag
enabled auto-merge (squash)
September 17, 2026 09:20
This was referenced Sep 17, 2026
…ine is 189 The baseline this branch first recorded, 278, was itself inflated. run.sh wrote `t27c gen` output into the tree whatever the exit code; the `&&` on that line only decided whether a counter moved. t27c emits what it managed before failing, so a TRUNCATED module stayed in the tree, and when that fragment happened to be valid Zig it ran under `zig test` and scored PASS. Checked directly: 89 of the 278 "passing" specs have a `t27c gen` that exits non-zero. specs/ar/ternary_logic.t27 carries 33 tests; its fragment carried none, compiled, and passed. Four of the specs used earlier to verify the timeout wrapper as "known PASS" were among them. A failed generation now removes the fragment and is recorded as its own verdict, NOGEN. It is appended to the results rather than dropped, because dropping it would shrink the denominator and flatter the pass rate. Scope, stated so it is not over-read: the review witness already refused a failed gen (`if gen && [ -s ]`), and the tree baked into the image already removed one (`gen || rm -f`). So bee verdicts were sound. What was inflated was every CORPUS number this script produced — the baseline, and each before/after count reported from it. Regenerated on master 5719414, t27c built from that commit, zig 0.16.0, 120s per spec: 189 PASS 654 NOCOMPILE 89 NOGEN 13 TESTFAIL 1 TIMEOUT (946) gf16_accel_tb resolved as TIMEOUT inside the run, not a 24-minute hang. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gHashTag
changed the title
gHashTag
added a commit
that referenced
this pull request
Sep 17, 2026
tools/oracle/run.sh now records a failed t27c gen as NOGEN instead of testing the fragment it left behind (#4235). Without a row the summary drops them, and the table no longer adds up to the corpus. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 17, 2026
Open
This was referenced Sep 17, 2026
This was referenced Sep 17, 2026
Run from the wrong directory, `find specs` sees nothing, nothing generates, nothing is tested, and every count is zero. The script then printed "oracle: 0 / 0 pass" and exited 0: the ratchet read that as no regression, and any caller read it as a clean result. This happened, rather than being imagined. A copy of run.sh placed at the repository root instead of at tools/oracle/ resolved its root as `../..` from there, climbed out of the repo, and produced exactly that silent success. Zero generated specs now exits 2 with the directory it looked in. The same guard protects the nightly job: an empty checkout or a moved script fails loudly instead of going green. Verified: run against a directory with no specs/, exit code 2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 17, 2026
This was referenced Sep 17, 2026
Merged
Contributor
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
1 similar comment
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
PR DashboardGenerated at: 2026-09-17 12:29:29 UTC
Summary
Seal Status
|
gHashTag
added a commit
that referenced
this pull request
Sep 17, 2026
…#4234) * Run the correctness oracle nightly in CI, and split what "pass" means The oracle had only ever run on a laptop. There it silently used a t27c built eight days earlier from uncommitted source and reported 541 passing specs; the same corpus measured with a t27c built from master was 279. Local measurement is also exactly what the operator ruled out for recurring swarm work: anything that should happen on a schedule runs in the cloud. This job builds t27c from the commit it judges and prints that commit beside every count, installs the zig the oracle's numbers were taken with, runs tools/oracle/run.sh, and writes the result to the job summary. Nightly rather than per pull request: a t27c build plus a zig run over ~950 specs is tens of minutes, and the per-PR queue was measured saturated today. The summary splits PASS in two, because `zig test` exits 0 for a file with no test in it. Measured on master today: 553 specs compile, but 347 of them carry no test, so only 206 are verified by one. Reporting 553 as "passing" would repeat an overstatement already made once. The run step is continue-on-error for now. tools/oracle/baseline.tsv still records `pass 541` from the stale compiler, and the ratchet inside run.sh would fail against a number that never existed. The baseline is regenerated from this job's output before the ratchet is allowed to gate. Report step verified locally against a real results.tsv; it reproduces 553 / 206 / 347 / 10 / 382 exactly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Report hung specs in the nightly summary tools/oracle/run.sh now bounds each spec and emits TIMEOUT for a test that never returns (#4235). Without a row for it the summary would silently drop those specs from every column. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Report specs that fail to generate in the nightly summary tools/oracle/run.sh now records a failed t27c gen as NOGEN instead of testing the fragment it left behind (#4235). Without a row the summary drops them, and the table no longer adds up to the corpus. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Contributor
PR DashboardGenerated at: 2026-09-17 12:34:16 UTC
Summary
Seal Status
|
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #3903
Two defects in the oracle that the nightly job (#4234) would have tripped over on its first run.
1. The baseline recorded a number that never existed
tools/oracle/baseline.tsvsaidpass 541. That came from a localt27cbuilt eight days earlier from uncommitted source. Regenerated with at27cbuilt from the commit it judges (5719414ee), zig 0.16.0:t27c genfailed outright (NOGEN)A baseline of 541 fails the ratchet on every honest run — which is why #4234 carries
continue-on-error. With a true floor, the ratchet can gate.This is master without #4114 (the sized-array type fix). When that lands, compilation rises to ~553 and the nightly should ratchet this floor up. A floor below the truth is safe; one above it fails every run.
2. One test that never returns held the whole oracle open
fpga/testbench/gf16_accel_tbcompiles, then its test never returns.run.shhad no per-spec limit, so it waited 24 minutes after every other spec finished. In CI that burns the job's entire timeout.Each spec is now bounded (
ORACLE_TIMEOUT, default 120s), and a hang is its own verdict,TIMEOUT— not misfiled asNOCOMPILE, since it compiled fine.The first attempt did not work, and why is worth keeping. The hung process is not zig:
zig testruns the test binary as a child. Killing zig orphaned that grandchild, which kept the output pipe open, and the command substitution waited forever — a 20s limit stayed open for seven minutes. GNUtimeoutsignals only its direct child and has the same flaw. The limit now forks the run into its own process group and kills the group, using perl because it exists on every CI runner and on macOS.Verified on real specs through the new wrapper:
gf16_accel_tb(hangs)TIMEOUTin 22s, no orphaned test binaryPASSNOCOMPILETESTFAIL— a panic is not a hangCorrection — the first baseline on this branch was also inflated
This PR first recorded 278. That was wrong too, and for a reason worth keeping.
run.shwrotet27c genoutput into the tree whatever the exit code — the&&on that line only moved a counter. t27c emits what it managed before failing, so a truncated module stayed, and when that fragment happened to be valid Zig it ran underzig testand scored PASS.89 of the 278 "passing" specs have a
t27c genthat exits non-zero.specs/ar/ternary_logic.t27carries 33 tests; its fragment carried none, compiled, and passed. Four of the specs used above to verify the timeout wrapper as "known PASS" were among them.A failed generation is now removed and recorded as its own verdict, NOGEN — appended to the results, not dropped, because dropping it would shrink the denominator and flatter the pass rate. The honest floor is 189.
Scope, so this is not over-read: the review witness already refused a failed gen, and the tree baked into the image already removed one. Bee verdicts were sound. What was inflated was every corpus number this script produced.
🤖 Generated with Claude Code