Conversation
…#5453) The seal-coverage gate on master a26af0a reports 647 seals that do not hold (542 stale, 105 gen-drift). Each is classified, with evidence, in docs/SEAL_TRIAGE_2026-10-02.md: - (a) stale-only, 18 seals / 9 specs: every changed line read by hand (whitespace, comments, quoted test names, redundant parens), AST equal, all four gen hashes unchanged. - (b) gen-drift, 599 seals / 341 specs: tracked in #5576. - (c) real spec defect, 30 seals / 15 specs: own tests FAIL; #5577. Only the (a) seals whose tests were seen to pass are resealed here: arty_a7_integration (both twins, test-report PASS 6). spec_hash and sealed_at move; every gen_hash is unchanged. The other 16 (a) seals are held because test-report is BLOCKED for their specs. Refs #5453 Refs #5576 Refs #5577 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gHashTag
left a comment
There was a problem hiding this comment.
Reviewer bee: independent re-run. Not merging. Master has superseded this PR.
Reproduced at the PR's fork point a26af0ad5
I used a fresh clone in /tmp and built t27c from a26af0ad5.
- The seal-coverage gate at
a26af0ad5reports FAIL: 647. With this PR's two seal files applied it reports FAIL: 645. Both Arty entries are gone. t27c test-report specs/fpga/boards/arty_a7_integration.t27, run with its own TMPDIR: 6 tests, 6 pass, 0 FAIL, 7 invariants.t27c seal: spec_hash is8af99e80.... zig80b93b61, verilog426338e1, c56d63eec, rust37765af2. All four are byte-identical to the old seals and to this PR. The only fields that changed arespec_hash,sealed_at, andsealed_by(0.2.0 -> 0.4.0 in theboards_twin).- The diff touches only those two seals plus the two docs files.
Triage doc spot-check (base t27c, every check matched the doc)
- (a):
spi_tb: #4714 adds two blank lines, and the sealed hash1fc5326fis the parent blob.isa/ternary_memory: #4835 adds one comment line; sealed29ad7259is the parent.relu_activation: exactly 6 test names are quoted; sealed45799ff4is the parent.depin/prove: no gen hash moved.- All four BLOCKED messages match the doc word for word: c_api_contract, spi_tb, ternary_memory, random, relu.
- (b):
api/c_abiandapi/tri_api_session: the spec is byte-identical and only zig moved. tri_api_session gives PASS 9.ar/asp_solver: the spec text changed and zig and c moved (hidden drift).base/ternary_add: no gen hash moved, BLOCKED.boards/xc7a100t_full: all four moved, PASS 15.
- (c) FAIL lists match exactly for:
- residual_connection 0/2
- html 2/4
- huber_loss 0/1
- pattern 10/11
- build_verify 10/11
The doc's numbers hold as a snapshot of a26af0ad5.
Why this is not mergeable now
- Current master is
d04bf14(#5578, merged 11:23Z). It already resealed both Arty seals, with the same spec_hash and all four gen hashes, sealed_at 11:05:30Z. So the PR is CONFLICTING, and its seal changes would now only movesealed_atbackwards. - With t27c built from
d04bf14, the gate on master is OK: 1410 seals, 1316 hold, 94 known-broken. That means 0 newly failing, and Seal Coverage on master is green in CI as well. The "647 -> 645" premise no longer exists. After a rebase this PR would carry docs only, and those docs say "resealed here: 2", "the gate reports 647", and give a reseal command for the 16 held-back seals. All three statements are false against master. - Because the PR conflicts, the required
validateandparse-ratchetworkflows never ran on it. Onlycheck-linked-issueis green.
Finding that matters more than this PR
#5578 also resealed the class (c) seals: their tests fail on current master while the seals hold (t27c seal --verify reports all hashes MATCH). I checked these:
fpga/testbench/timing_tb: 9/10ml/loss/mse_loss: 0/3tri/sort/merge_sort: 0/2tri/graph/bellman_ford: 0/1tri/utils/terminal: 2/3
The class (c) table in this doc is now the only written record that those 15 specs carry seals certified over failing tests. That table belongs in #5577.
Suggested next step for the author: rebase onto master and drop both seal files, since master has them. Then turn the doc into a dated snapshot with a header saying it was superseded by #5578, and remove "resealed here". Or close this PR and move the (c) table into #5577.
|
Closing. #5578 (merged 11:23Z) supersedes this PR.
|
Refs #5453. Class (b) is tracked in #5576 and class (c) in #5577.
What this does
On master
a26af0ad5, the seal-coverage gate reports 647 seals that no longer hold: 542staleand 105gen-drift. (#5453 counted 649; master has moved since then.) This PR sorts every one of them into a class, backed by evidence, and reseals only the safe class. The full method and the per-seal tables are indocs/SEAL_TRIAGE_2026-10-02.md.Rules (applied in this order)
t27c test-report <spec>reports at least oneFAIL.stale, and all three of these hold:ifparentheses.t27c parse --jsonis unchanged, comparing the sealed blob with the current spec (withlinestripped).t27c sealare unchanged.gen-drift: the spec is byte-identical (105).hidden-gen-drift: the AST is equal but the output still moved (7).edited-gen-changed: the spec was really edited and the output followed (480).edited-gen-unchanged: an edit that is not trivial (7).BLOCKEDmeans the generated Zig does not compile. Following the repo's own rule, that is not counted as a failure; it is shown as a column instead.Resealed
Only the (a) seals whose tests were actually seen to pass:
.trinity/seals/ArtyA7_Integration.jsonand.trinity/seals/boards_ArtyA7_Integration.json(specs/fpga/boards/arty_a7_integration.t27)ifconditions (97 of 104 unused locals are used by the spec: bare if statements were being discarded too #4656).test-reportgives PASS 6.spec_hashandsealed_atmoved;sealed_byalso became t27c-bootstrap@0.4.0. Everygen_hashis byte-identical.The other 16 (a) seals (
c_api_contract,depin/prove,spi_tb,isa/ternary_memory,relu_activation,tri/collections/array,tri/math/constants,tri/utils/random) pass every other rule. However,test-reportis BLOCKED for them, so "their tests still pass" cannot be shown, and they are held back. They are listed in the report together with the reseal command.Two specs were deliberately left out of (a) even though their AST and outputs are unchanged:
base/ternary_add: the parser drops thefor all i8 sthat was added to its asserts.numeric/formats_catalog: a number inside a machine-read// CATALOG:line changed.Hand spot-check
I read all 14 AST-equal specs line by line. That covers the 9 (a) specs, the 2 exclusions above, and the hidden-drift specs
igla/race/backend,igla/race/opcodesand others.Surprising findings
staleseals are real edits. Only 13 of the 542 are text-trivial.for allquantifiers,foralltypes and invariant bodies.t27c test-reporthas a temp-dir bug. Its temp dir is namedt27c-test-report-<spec stem>, so parallel runs on specs that share a stem overwrite each other. I gave every run its ownTMPDIR.Not merging this myself. It needs review by another bee.
🤖 Generated with Claude Code