Skip to content

CI: spec-guards and emit-bitexact restore master cargo caches; cancel superseded emit-bitexact runs (B25) - #5924

Open
gHashTag wants to merge 2 commits into
masterfrom
ci/cargo-cache-slow-gates
Open

gHashTag wants to merge 2 commits into
masterfrom
ci/cargo-cache-slow-gates

Conversation

@gHashTag

@gHashTag gHashTag commented Oct 4, 2026

Copy link
Copy Markdown
Owner

Closes #5918
Refs #5906 (row 6, plan B25 in .trinity/review-speed-2026-10-04.md)

What changes

  • spec-guards.yml: new step Restore cargo (spec-parse-ratchet's master cache), actions/cache/restore@v4, key ${{ runner.os }}-cargo-parse-ratchet-${{ hashFiles('**/Cargo.lock') }}, restore-keys ${{ runner.os }}-cargo-parse-ratchet-. The build step is unchanged and still runs cargo build --release -p t27c.
  • emit-bitexact-gate.yml:
    • new step Restore cargo (fpga-build's master cache), actions/cache/restore@v4, key ${{ runner.os }}-cargo-fpga-${{ hashFiles('**/Cargo.lock') }}, restore-keys ${{ runner.os }}-cargo-fpga-;
    • new step rm -rf target/debug after the restore (reason below);
    • concurrency: { group: emit-bitexact-${{ github.ref }}, cancel-in-progress: true }, the same shape spec-guards.yml and fpga-build.yml use.
  • A docs/now/ entry.

Measured before (2026-10-04)

gate run build step crates compiled
spec-guards (cold) 37182032525 243 s of a 258 s job 287
spec-parse-ratchet, same head, cache 37182032489 40 s, after a 5 s restore 1 (t27c)
emit-bitexact (cold) 37181512071 158 s 287
fpga-build fpga-smoke, same head, cache 37181512084 35 s, after an 11 s restore 1 (t27c)

Why these two keys, and why one key does not fit both

A cache saved on a pull_request ref is visible to that PR only; a PR restores from master. gh cache list --ref refs/heads/master shows 8 caches on master. Two of them are produced by the exact command these gates run:

  • Linux-cargo-parse-ratchet-*, from spec-parse-ratchet.yml: runs on every push to master, no paths filter, no toolchain step (runner image rustc).
  • Linux-cargo-fpga-*, from fpga-build.yml: runs on push to master over bootstrap/** and Cargo.lock, and installs dtolnay/rust-toolchain@stable.

The toolchain is the part that decides it. On run 37181512071 the emit-bitexact toolchain step logged stable-x86_64-unknown-linux-gnu updated - rustc 1.99.0 (b940084d7 2026-09-28) (from rustc 1.98.1 (48a229cea 2026-09-01)): the runner image still ships 1.98.1, which is what the parse-ratchet cache is built with. cargo hashes the rustc version into every artifact, so that cache would compile all 287 crates again under emit-bitexact. fpga-build installs its toolchain the same way emit-bitexact does, and its cache built t27c in 35 s under 1.99.0. spec-guards has no toolchain step, so it pairs with parse-ratchet.

A key of each gate's own would not work for emit-bitexact at all: it has no push: trigger, so nothing it saves would ever reach master and every PR's first run would stay cold.

Why restore-only

gh cache list across the repository: 17 entries, 10.24 GB, against GitHub's 10 GB limit; 6.01 GB of it on PR refs. A saving step here would add roughly 350-800 MB per PR and evict the least recently used entry, which can be the master cache these restore from. Restore-only adds nothing to the budget.

Why this cannot hand a gate a stale t27c

  • The build steps still run cargo build --release -p t27c after the restore. The checkout writes every source file with a fresh mtime, newer than the restored fingerprints, so cargo recompiles t27c and reuses only dependencies. That is what "1 crate compiled" in the table above is: Compiling t27c v0.4.0 and Finished, nothing else.
  • emit-bitexact: five of its tools (verify_emit_bitexact.py, verify_multitarget.py, verify_trainer_c.py, verify_igla_race.py, and fuzz_trainer.py through verify_trainer_c's lookup) try target/debug/t27c before target/release/t27c. cargo build --release never refreshes a debug binary. fpga-build only builds release, so the restored tree should have no target/debug; rm -rf target/debug makes that structural instead of assumed.
  • spec-guards' tools read target/release/t27c only (checked: check_seal_currency, check_ring_spec_drift, ring_spec_differential, check_duplicate_declarations), so it needs no such step.

Superseded emit-bitexact runs

Over its last 200 runs (2026-10-02 01:11Z to 2026-10-04 06:18Z, median 637 s each), 16 were still running when a newer run started on the same branch, and they held a runner for 10,421 s past that point.

Checked locally

  • Both workflows parse with yaml.safe_load; all three files are ASCII.
  • scripts/ci/check_pr_branch_filters.py --self-test and real mode: clean. check_untrusted_shell_interp.py, check_untrusted_javascript_interp.py, test_required_gates_name_their_subject.py, test_admitted_gate_reads_every_named_file.py, tools/check_devhome_paths.py: OK. tools/check_now_entry_shape.py against origin/master: OK.
  • No cargo build / cargo test locally (disk). The measurement of this change is this PR's own spec-guards and emit-bitexact runs; both workflows list their own file in paths:, so both trigger here.

Not done

  • cli-tri.yml, orphan-modules.yml and harness-scratch.yml also build cold, but they build tri, not t27c, and no master cache holds a release tri build. Giving them one means a new saving cache in a repository already over its cache limit, which is a separate decision.

🤖 Generated with Claude Code

… superseded emit-bitexact runs (B25)

spec-guards built t27c cold in 243 s of a 258 s job (run 37182032525);
emit-bitexact in 158 s (run 37181512071). Each now restores, read-only,
a cache that another workflow already saves on master with the same
`cargo build --release -p t27c` and the same toolchain:

- spec-guards <- Linux-cargo-parse-ratchet-* (spec-parse-ratchet.yml,
  every push to master, image rustc). Same head: 40 s, 1 crate compiled.
- emit-bitexact <- Linux-cargo-fpga-* (fpga-build.yml, which installs
  dtolnay/rust-toolchain@stable like emit-bitexact does). The image
  ships rustc 1.98.1 and that action installs 1.99.0 today, so the
  parse-ratchet cache would be rebuilt from scratch here. Same head:
  35 s after an 11 s restore, 1 crate compiled.

Restore-only: repo caches are at 10.24 GB of the 10 GB limit, and a
per-PR save would evict the master caches these read. emit-bitexact
also removes target/debug after the restore, because five of its tools
prefer target/debug/t27c and `cargo build --release` never refreshes
it. emit-bitexact gains a concurrency group with cancel-in-progress:
16 of its last 200 runs were superseded and ran 10,421 s past that.

Closes #5918
Refs #5906

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-04 06:51:48 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 48
PRs with All Checks Green 2
READY 1
FAILING 48
PENDING 0
NO CHECKS YET 0

These columns do not partition: 1 + 48 + 0 + 0 = 49, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=b45a356c2eb6 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

The one step this PR adds (`rm -rf target/debug` in emit-bitexact-gate.yml,
ubuntu-latest, no container) is one more `run:` step the runner hands to
bash. cli-tri's "No census moved without saying so" read 284 against the
ledger's 283 on run 37183890420. Re-blessed by hand: tri is not built on
this machine; the two changed lines are the only ones the added step moves
(no new workflow file, no new job, no container, no `shell:` key).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-04 07:30:08 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 48
PRs with All Checks Green 2
READY 1
FAILING 48
PENDING 0
NO CHECKS YET 0

These columns do not partition: 1 + 48 + 0 + 0 = 49, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=b45a356c2eb6 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

gHashTag added a commit that referenced this pull request Oct 4, 2026
B25's caches are in PR #5924 (Closes #5918); the census its one added step
moved is re-blessed there (de465fe). The NOW gate's removal is PR #5951
(Closes #5935); the NotebookLM gate's is PR #5958, stacked on it. None merged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 6, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: cache cargo in spec-guards and emit-bitexact; cancel superseded emit-bitexact runs

1 participant