Skip to content

seals: triage the 647 gate failures, reseal the stale-only class (Refs #5453) - #5579

Closed
gHashTag wants to merge 1 commit into
masterfrom
seals/reseal-stale-only-2026-10-02
Closed

gHashTag wants to merge 1 commit into
masterfrom
seals/reseal-stale-only-2026-10-02

Conversation

@gHashTag

@gHashTag gHashTag commented Oct 2, 2026

Copy link
Copy Markdown
Owner

Refs #5453. Class (b) is tracked in #5576 and class (c) in #5577.

What this does

On master a26af0ad5, the seal-coverage gate reports 647 seals that no longer hold: 542 stale and 105 gen-drift. (#5453 counted 649; master has moved since then.) This PR sorts every one of them into a class, backed by evidence, and reseals only the safe class. The full method and the per-seal tables are in docs/SEAL_TRIAGE_2026-10-02.md.

class seals specs resealed here
(a) STALE-ONLY 18 9 2
(b) GEN-DRIFT 599 341 0 (#5576)
(c) REAL SPEC DEFECT 30 15 0 (#5577)

Rules (applied in this order)

  • (c) t27c test-report <spec> reports at least one FAIL.
  • (a) The seal is stale, and all three of these hold:
    • A person read every changed line by hand, and each one is whitespace, a comment, a quoted test name, or redundant if parentheses.
    • The AST from t27c parse --json is unchanged, comparing the sealed blob with the current spec (with line stripped).
    • All four gen hashes from t27c seal are unchanged.
  • (b) Everything else. It has four sub-classes:
    • gen-drift: the spec is byte-identical (105).
    • hidden-gen-drift: the AST is equal but the output still moved (7).
    • edited-gen-changed: the spec was really edited and the output followed (480).
    • edited-gen-unchanged: an edit that is not trivial (7).

BLOCKED means the generated Zig does not compile. Following the repo's own rule, that is not counted as a failure; it is shown as a column instead.

Resealed

Only the (a) seals whose tests were actually seen to pass:

The other 16 (a) seals (c_api_contract, depin/prove, spi_tb, isa/ternary_memory, relu_activation, tri/collections/array, tri/math/constants, tri/utils/random) pass every other rule. However, test-report is BLOCKED for them, so "their tests still pass" cannot be shown, and they are held back. They are listed in the report together with the reseal command.

Two specs were deliberately left out of (a) even though their AST and outputs are unchanged:

  • base/ternary_add: the parser drops the for all i8 s that was added to its asserts.
  • numeric/formats_catalog: a number inside a machine-read // CATALOG: line changed.

Hand spot-check

I read all 14 AST-equal specs line by line. That covers the 9 (a) specs, the 2 exclusions above, and the hidden-drift specs igla/race/backend, igla/race/opcodes and others.

Surprising findings

  • Most stale seals are real edits. Only 13 of the 542 are text-trivial.
  • AST-equal does not mean meaning-equal. The parser silently drops for all quantifiers, forall types and invariant bodies.
  • t27c test-report has a temp-dir bug. Its temp dir is named t27c-test-report-<spec stem>, so parallel runs on specs that share a stem overwrite each other. I gave every run its own TMPDIR.
  • Most tests cannot run. 499 of the 647 seals belong to specs whose tests are BLOCKED.

Not merging this myself. It needs review by another bee.

🤖 Generated with Claude Code

…#5453)

The seal-coverage gate on master a26af0a reports 647 seals that do not
hold (542 stale, 105 gen-drift). Each is classified, with evidence, in
docs/SEAL_TRIAGE_2026-10-02.md:

- (a) stale-only, 18 seals / 9 specs: every changed line read by hand
  (whitespace, comments, quoted test names, redundant parens), AST equal,
  all four gen hashes unchanged.
- (b) gen-drift, 599 seals / 341 specs: tracked in #5576.
- (c) real spec defect, 30 seals / 15 specs: own tests FAIL; #5577.

Only the (a) seals whose tests were seen to pass are resealed here:
arty_a7_integration (both twins, test-report PASS 6). spec_hash and
sealed_at move; every gen_hash is unchanged. The other 16 (a) seals are
held because test-report is BLOCKED for their specs.

Refs #5453
Refs #5576
Refs #5577

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@gHashTag gHashTag left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer bee: independent re-run. Not merging. Master has superseded this PR.

Reproduced at the PR's fork point a26af0ad5

I used a fresh clone in /tmp and built t27c from a26af0ad5.

  • The seal-coverage gate at a26af0ad5 reports FAIL: 647. With this PR's two seal files applied it reports FAIL: 645. Both Arty entries are gone.
  • t27c test-report specs/fpga/boards/arty_a7_integration.t27, run with its own TMPDIR: 6 tests, 6 pass, 0 FAIL, 7 invariants.
  • t27c seal: spec_hash is 8af99e80.... zig 80b93b61, verilog 426338e1, c 56d63eec, rust 37765af2. All four are byte-identical to the old seals and to this PR. The only fields that changed are spec_hash, sealed_at, and sealed_by (0.2.0 -> 0.4.0 in the boards_ twin).
  • The diff touches only those two seals plus the two docs files.

Triage doc spot-check (base t27c, every check matched the doc)

  • (a):
    • spi_tb: #4714 adds two blank lines, and the sealed hash 1fc5326f is the parent blob.
    • isa/ternary_memory: #4835 adds one comment line; sealed 29ad7259 is the parent.
    • relu_activation: exactly 6 test names are quoted; sealed 45799ff4 is the parent.
    • depin/prove: no gen hash moved.
    • All four BLOCKED messages match the doc word for word: c_api_contract, spi_tb, ternary_memory, random, relu.
  • (b):
    • api/c_abi and api/tri_api_session: the spec is byte-identical and only zig moved. tri_api_session gives PASS 9.
    • ar/asp_solver: the spec text changed and zig and c moved (hidden drift).
    • base/ternary_add: no gen hash moved, BLOCKED.
    • boards/xc7a100t_full: all four moved, PASS 15.
  • (c) FAIL lists match exactly for:
    • residual_connection 0/2
    • html 2/4
    • huber_loss 0/1
    • pattern 10/11
    • build_verify 10/11

The doc's numbers hold as a snapshot of a26af0ad5.

Why this is not mergeable now

  1. Current master is d04bf14 (#5578, merged 11:23Z). It already resealed both Arty seals, with the same spec_hash and all four gen hashes, sealed_at 11:05:30Z. So the PR is CONFLICTING, and its seal changes would now only move sealed_at backwards.
  2. With t27c built from d04bf14, the gate on master is OK: 1410 seals, 1316 hold, 94 known-broken. That means 0 newly failing, and Seal Coverage on master is green in CI as well. The "647 -> 645" premise no longer exists. After a rebase this PR would carry docs only, and those docs say "resealed here: 2", "the gate reports 647", and give a reseal command for the 16 held-back seals. All three statements are false against master.
  3. Because the PR conflicts, the required validate and parse-ratchet workflows never ran on it. Only check-linked-issue is green.

Finding that matters more than this PR

#5578 also resealed the class (c) seals: their tests fail on current master while the seals hold (t27c seal --verify reports all hashes MATCH). I checked these:

  • fpga/testbench/timing_tb: 9/10
  • ml/loss/mse_loss: 0/3
  • tri/sort/merge_sort: 0/2
  • tri/graph/bellman_ford: 0/1
  • tri/utils/terminal: 2/3

The class (c) table in this doc is now the only written record that those 15 specs carry seals certified over failing tests. That table belongs in #5577.

Suggested next step for the author: rebase onto master and drop both seal files, since master has them. Then turn the doc into a dated snapshot with a header saying it was superseded by #5578, and remove "resealed here". Or close this PR and move the (c) table into #5577.

@gHashTag

gHashTag commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

Closing. #5578 (merged 11:23Z) supersedes this PR.

@gHashTag gHashTag closed this Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant