Skip to content

Send and check gate evidence before CI builds Rust - #742

Open
jserv wants to merge 13 commits into
mainfrom
pr/gate-evidence
Open

jserv wants to merge 13 commits into
mainfrom
pr/gate-evidence

Conversation

@jserv

@jserv jserv commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem and behavior

A pull request from a contributor with no Rust toolchain installed gets its first compile from CI, so the Rust lane and every mutation shard are spent learning that the build fails, and a held first-time run could be approved without anyone checking what the contributor ran locally. After this change a missing tool is reported before any gate runs, a passing ./scripts/test.sh run leaves a receipt that names the tree it saw, the pre-push hook refuses a push that touches Rust sources, tests or build files without a matching receipt and sends the receipt as a note on refs/notes/codetrial-gate only when the remote does not already hold that note, and both a first CI job and the maintainer's approval step read that note from the fork as data and either fail early with the install steps or ask the contributor for the toolchain. Mutation shards now wait for the gates and run under --locked, so a pull request that already fails the Rust lane does not pay for every shard to rediscover it. CONTRIBUTING.md states the local-first rule for anyone sending a change, a person or an automated agent, in a Before you push section that the contribute and verify skills point to.

Approach

scripts/env-check.sh is the one owner of what a run needs and scripts/test.sh stops on it before any gate, instead of each gate probing for its own tool. scripts/check-gate-evidence.py is the one scope function, used by the hook, the checker and the maintainer sweep, and its tests pin it to the paths the Rust sources read and to the texts that name them. scripts/fetch-gate-note.sh is the only code that fetches a fork's notes: it takes the owner, repository and sha as separate arguments, validates them before any git call, builds the URL itself, checks the note's size before reading it, and runs git under a cap on the size of any one file it writes so a fork's notes cannot fill the disk. The checker reads only data files and runs nothing, and the CI job checks out the base only and prints at most one sanitized line, so nothing from a fork can start a workflow command in the log. The note is evidence from the contributor's machine and can be forged: it makes the maintainer's decision mechanical and says nothing about whether the gate ran, and the CI job is cost control, not a security boundary, because a pull request runs its own merged workflow.

Known limits:

  • A forged receipt is possible; the approval step and maintainer judgment are what bind, and the pre-push hook can be bypassed.
  • A note keyed by tree says nothing about the merged result, an edit made and then reverted during a run is not detected, and a partial commit needs the gate re-run on what is committed.
  • Notes for abandoned heads accumulate on the contributor's own fork.
  • The CI job and the held-run sweep add latency, the sweep runs serially, and both fail closed on an API or fetch error.
  • A few files under web/ that Rust tests read are in scope, so a change to only those needs a receipt; the rest of web/ is not.
  • The pre-push hook leaves the local note alone when it already holds the blob and skips the nested notes upload when the remote already holds the same blob for every pushed head. That removes a needless second upload; it was not shown to prevent the rare rejected pushes I saw against GitHub while testing. I did not explain those, and a retry succeeded each time.
  • A change to tests/harness/harness-audit.test.js answers in scope, because everything under tests/ is in scope except the browser lane, the stubs and the top-level Python files; the file runs only under Node, so the answer asks for a Rust receipt that the change cannot affect, which errs toward asking.
  • The receipt's env field has no reader yet.
  • The rule is stated only in CONTRIBUTING.md, so an agent that never reads it is not reached; the pre-push hook and the CI job are what enforce it.

Validation

I ran ./scripts/test.sh portable and then ./scripts/test.sh rust, one after the other, on the final tip after the rebase onto the current main, and both exited 0 with desktop-fmt included; the portable run includes the new C and C++ harness audit from the current main (66 tests, none skipped), and it skipped prettier, eslint and script-requires because npm is not installed here, while the rust run skipped the browser tests, desktop clippy and tests, and cargo-deny. I also ran sh scripts/test-git-hooks.sh (501 checks), the twelve Python suites that cover the new scripts (620 tests), shellcheck, dash -n, actionlint, ruff check and ruff format --check, scripts/check-skill-refs.py, scripts/check-commit-log.sh and scripts/hashchain.py verify, and broke each new rule on purpose to check that a test fails, including the new test that the hook skips an unchanged note and the one that bounds the file size of a fork's fetch. Each commit passes the suites it adds when checked out alone, and the first run of tests/test_test_gate.py in a fresh worktree passes.

Not run: anything on GitHub (the preflight job, the held-run approval against a real fork, and gh api against a fork), the browser lane, desktop clippy and tests, cargo-deny, prettier and eslint, and macOS or Windows.

cubic-dev-ai[bot]

This comment was marked as resolved.

jserv added 13 commits November 17, 2026 07:48
A mutation shard rebuilds and runs the whole suite in an unmutated tree,
so when the Rust lane fails the shards fail the same way and each pays
the whole setup to learn it. The mutants job needed only the plan, so it
started beside the gates.

Make it need the gates as well, and start only when both the plan and
the gates succeeded and the plan found mutants. The plan result is
checked explicitly because the plan is skipped on a push or a merge
group, where its empty count would otherwise pass the zero guard. The
aggregate check job already accepts a skipped mutants job after a
successful plan, and a failed gates lane fails it first.

Also hold the runs to the lockfile. The Makefile mutants target passes
--cargo-arg=--locked so a mutant is never judged against a lockfile
cargo quietly updated, but the plan's --list and the shard's --list and
run did not, so CI could test against whatever lock cargo produced and
the local and CI answers could differ. Pass the flag on all three
commands.

A new suite pins the job's needs and condition, that each command
carries the flag, and that the plan and the shard list select with the
same flags so the shard count matches what runs.

The development guide says the shards wait for the gate lanes and are
skipped when a lane failed.

Change-Id: I02d001e8746051adf98b194ddab92f7a7a0fa123
The lane tests run scripts/test.sh with its gate payload replaced, but
the Python lookup above the gates is the real one. In a clone with no
instructor tool environment it builds one on the first run, several lane
tests running at once, so they see a failed task-tools gate or a
half-built environment, and pass only on the second run once it exists.
CI names its own environment and never sees it.

Give run_lane the running interpreter as PYTHON, when it is 3.11 or
later, with the packaging probe answered as ready, unless the test names
PYTHON or CODETRIAL_TASK_TOOLS itself. The tests of the lookup and of
the setup name one of them and keep the real behavior; the one that
named neither now names an empty environment, which the script treats as
unset.

Change-Id: Idace6b303f73ae9182e538c893838cf3dc98e8da
What a gate run needs (the pinned toolchain, Clang, node, optional
linters) was learned one failing gate at a time, because the gate script
only records a failure and carries on. A machine without the pinned
toolchain produced a wall of unrelated failures, and rustup installs the
pinned toolchain the first time any rustup command runs in the
repository, so even asking which toolchains exist could start a
download.

Add scripts/env-check.sh, which only looks and never installs. Each
profile lists what the matching lane needs, and a missing tool is named
with the command that installs it. It asks rustup for the installed
toolchains from outside the repository, so no pin applies, before it
runs cargo or rustc there, and it reports a broken cargo as a failure
instead of passing it over.

scripts/test.sh runs it before any gate and stops there when a required
tool is missing. The optional tools that are absent come from the same
report, so the per-gate probes for ruff, shellcheck, dash, actionlint,
cargo-audit and cargo-deny are gone and the skip lines keep their
wording. The suites use stub executables, and the system-tool PATH they
need is shared in tests/support.py. The development guide gains an
Environment check section.

The compiler check applies on Linux only, where webrtc-sys needs Clang
21 or later; another platform builds with its own compiler.

Change-Id: I77c2c1ce3d4818027dec8d1eb072c16acff452dc
A receipt carried only a fingerprint that changes on every rebase or
amend, so nothing could tell whether a later commit holds what the run
saw, and the next run on any branch overwrote the lane file.

Add worktree_tree, the tree id a commit of the same content gets. It is
computed in a scratch index, so the real index and partial staging are
never touched, and it is taken at the start and the end of the run and
kept only when both agree, so a tree edited during the run claims
nothing, though an edit made and then undone cannot be seen. The receipt
also records rustc, cargo and the environment report, and a copy is kept
per tree, newest 20, because the lane file is overwritten by the next
run, possibly on another branch. A tree folder written within the last
hour is never pruned, since another run in the clone may have just
written it and nothing locks the folder.

The development guide describes each receipt field. The extractor the
suites use to load a shell function is shared in tests/support.py, and
a closeout check test pins that the new fields and the per-tree
folder do not break reading older receipts.

Change-Id: I3351b78479dc59aca590898343e88765e1d4139f
Which paths the Rust gate covers is decided nowhere, so every caller
that needs to know whether a change requires a Rust receipt would
copy its own list and the lists would drift. check-gate-evidence.py
--scope-only reads paths from stdin (NUL-separated with -z, since a
path can hold a newline) and prints in or out. A .github/ path is never
in scope. Line mode refuses a path that ends in a carriage return, which
would otherwise stop an exact name such as Cargo.toml from matching.

The scope is src/, desktop/, vendor/, problem-bank/,
scripts/hook-check/, .cargo/, the root manifests and toolchain files,
the clippy, rustfmt and deny configs, and tests/ except the browser
lane, the stubs and the top-level Python suites. web/ stays out through
one named constant, apart from the few files a Rust source names.

A list written from the directory layout misses files that Rust
tests read from outside it. A test therefore scans the tracked Rust
sources for include macros, folder attributes, CARGO_MANIFEST_DIR
joins and string literals that are tracked files, and fails when the
scope omits one. It tokenizes left to right so a quote inside a
comment or a character literal cannot shift the literals after it,
and a failing git raises instead of passing as an empty listing.

Change-Id: I1343874fe42a09b12b6090bf03e9deede587b5e8
Approving a held run needed a person to read a contributor's claim that
the Rust gate had passed, and nothing compared it with what is pushed.
The evidence check turns that into a mechanical answer from data
files alone: a change with no Rust-scope path owes no receipt; otherwise
some receipt must be for lane rust or all, passed, name the head
commit's tree, and carry the rustc that the base rust-toolchain.toml
pins (the head's when the change edits that file). A bot author is
exempt, a .github/ path is never in scope, and a truncated changed-file
list or an unreadable pin is indeterminate rather than a pass.

The receipt is untrusted: it is size-checked before it is parsed, parsed
strictly (no duplicate keys, NaN, trailing data or bool-as-number), and
none of its text reaches a path, a command or the output. It is a hint a
dishonest author can forge, not proof the gate ran.

Change-Id: I2b86f020d9716447c6c117d9761bf5a17177f3c9
A contributor with no toolchain can open a pull request that only CI
compiles, and nothing on the push says whether the gate was run. The
pre-push hook now asks, for a topic branch that changes what the Rust
gate covers, for a passing rust or all receipt of the pushed tree,
and sends it as a note on the head before the branch. Scope comes
from check-gate-evidence.py, measured from the first main that
resolves, looked up once per push; with none the hook says so and
refuses nothing. A refusal prints the environment report, and when a
run changed a tracked file while it ran, which writes no by-tree
receipt, it says that too.

The note is built by scripts/gate-note.py: one JSON object, bounded
in size and in every field, that copies only the keys of the schema
from the receipt files and drops what is malformed. It is sent only
when the remote does not already hold it: the hook leaves a local note
that already holds the blob alone, and after fetching the remote notes
it drops every head whose note the remote holds with the same blob, so
an unchanged head causes no second upload. The review-record block of
the hook is left as it is and the new one sits beside it, because that
one is gated by a completed review and keeps its own state. The
receipt can be forged and a hook can be skipped, so CI and approval
are what bind. The development guide describes the check and the note,
and the hook tests cover them.

Change-Id: Ia17dd2dd4dfe9cae131d115bda6e1ac32cf708f3
Approving a held run on gate evidence needs the note a contributor
pushed to the fork, and that fetch reaches a repository nobody here
controls. Written ad hoc in each caller it would drift: one would build
the URL from PR text, another would skip the size check.

scripts/fetch-gate-note.sh takes the head owner, name and sha as
separate arguments, validates them, builds the URL itself and fetches
only refs/notes/codetrial-gate into a scratch bare repository:
anonymous, one commit deep, a 60 second limit, a cap on the size of any
one file it writes, and the note blob sized before it is read. Each
failure has its own exit status, so a caller can tell a note that is
absent from a fetch that did not answer.

Change-Id: I000fe438ae75773c1c2e224849e8f942a607f902
A sweep approves the runs GitHub holds for first-time contributors
without looking at what the contributor ran locally, so a PR from an
agent with no Rust toolchain still spends the full Rust lane to learn
whether the code compiles.

gate-evidence.sh gathers the head tree, the changed paths (renames name
both ends, a cut list is indeterminate), the toolchain pin from the base
branch and the note from the fork as data, and asks
check-gate-evidence.py. pr-sweep.sh reports the verdict beside each held
run. apply-pr-actions.sh reads it again for the head the sweep saw
immediately before the approval call, approves only on a pass, leaves an
ask on the confirmed list as the toolchain comment to draft, and skips
when the answer cannot be given: a moved head, a deleted fork, a failed
fetch.

pulls.md now says what the pull_request_target grep prints, which is
pr-description.yml alone and the one reviewed exception, and drops the
wording that read as approving every held run at once.

Change-Id: I041ce0f91479a3e11a60fdfd913d9e74f752d018
A pull request that changes Rust and carries no gate receipt spends a
full Rust build before anyone learns the receipt is missing.

Add a preflight job that runs the base branch's evidence check over the
head as data and fails first. The gates, the mutation plan and the
aggregate status wait for it, and a push or a merge group still runs the
gates because preflight is skipped there. It is cost control, not a
security boundary: a pull request runs its own workflow.

The reason the gatherer reports is built from data a pull request
writes, and jq decodes a JSON newline into a real one, so printing it
as it came could put a line starting with :: in the log: a forged
error or a mask. The step keeps the verdict out of the log, captures
the gatherer's stderr, and reduces the reason to one line of
printable ASCII behind the step's own error prefix, written with
printf because dash's echo turns a backslash and n into a line break.

The tests pin the job's shape, run its script and the aggregate
script over each outcome, and run the step under dash against hostile
output on both streams, requiring that the only :: line is the
step's own. The development guide describes the job.

Change-Id: Ib0a11991ce865d142596bb4ddd74fa6c2dbb01b1
A contributor with no Rust toolchain can push a change that only CI
compiles, and the first sign is a failed run. The rule was stated in no
place a contributor reads before a push.

Add a Before you push section to CONTRIBUTING.md with the order to work
in, for a person or an automated agent: run the environment check and
stop when it fails, run the gate and push only after it passes, never
bypass a hook, and report what was run and what was not. The same file
states which paths need a local test.sh pass (the directories Rust
reads, the root manifests, the toolchain and lint configs, and the few
web/ files a Rust test reads), what the pre-push hook measures the
branch from, how a receipt that names a tree matches a commit, and that
the receipt is a hint while the maintainer's approval and the first CI
job are what bind. The pull request template asks for the command that
was run and the end of its output, and the contribute and verify skills
each point to the new section.

Change-Id: I70001cb5b242f673baa936912d3210e56a1241a6
The note builder and the evidence checker each have a suite built from
hand-made inputs, so a field one side writes and the other no longer
reads would pass both. Run the receipt writer of scripts/test.sh in a
scratch repository, build the note from what it wrote, and judge it:
accepted for its tree, refused for another tree, for a lane that does
not build Rust, for a failed run and for a compiler other than the
pinned one. A malformed receipt is skipped by the builder, and the
checker refuses an entry whose field has the wrong type or size.

CONTRIBUTING.md and the scope function of scripts/check-gate-evidence.py
each list the Rust scope, so an edit to one left the other stale without
any check failing. Require the text to name every directory and file the
function lists, with the tests/ exclusions stated, and every path it
names to be classified the way it says. web/ stays out as a folder, but
the text must say that a few web/ files a Rust test reads are in.

The note's size, lanes, tree id shape and receipt count are written in
the builder, the checker and the fetcher, so a test requires the three
to agree.

Change-Id: I1c2f82fc951116ff17ab35b419a866e1d0cde485
A contributor asked to reproduce a failed mutation shard, or a
reviewer's request for a mutation run, had only make mutants, which
audits a fixed set of files whatever the branch changed, so it can
pass while the diff has a surviving mutant and fail on code the
branch never touched.

Document the diff-scoped cargo-mutants commands CI runs, with the
shard option and the note that --in-place mutates the working tree.
The commands add --cargo-arg=--locked, which the Makefile target
passes and which keeps a mutant from being judged against a lockfile
cargo quietly updated, and the text says it is diff-scoped like CI's
check rather than identical to it. The cost is explained by its
cause: a run costs the number of mutants times the suite's time, and
the mutant timeout follows the baseline's test time, which is why it
is for reproducing a failure and not a per-change requirement.

A test pins that both documented commands carry the flag and select
what the shard step selects.

Change-Id: I8d6327f833de7298c79a0d4e71179b8514ced610

@alanhc alanhc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I read the workflow, gate-evidence.sh, fetch-gate-note.sh, the check_rustc change and the pre-push receipt path:

  • preflight runs only the base branch's scripts with a read-only token, and its output is cut to printable ASCII under its own prefix, so a fork cannot emit a workflow command. gates passes on a skipped preflight, so push and merge group are unchanged.
  • gate-evidence.sh reads the head before and after the check and fails closed on any API error, a file count mismatch or a NUL in a path.
  • The fetch is anonymous, https-only, one commit of the notes ref, with the blob size checked before it is read; --batch-check now tells a missing blob (7) apart from a failed probe (5), and the ulimit -f kill lands on 5 as well.
  • The dedup in the pre-push hook is right in each case I traced: an unchanged note is neither rewritten nor uploaded, a first push with no remote notes ref still sends, and other heads' notes survive the -s ours merge. It relies on the note bytes being deterministic, which the new test covers.

One thing worth doing after merge: the description says preflight has not run against a real fork yet. repos/{base}/git/commits/{sha} should resolve a fork head through refs/pull/N/head, but a single fork PR touching Rust would confirm both that and the anonymous note fetch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants