Repository navigation
Conversation
jserv
force-pushed
the
pr/gate-evidence
branch
from
November 16, 2026 21:33
b542ba1 to
5e39b3c
Compare
A mutation shard rebuilds and runs the whole suite in an unmutated tree, so when the Rust lane fails the shards fail the same way and each pays the whole setup to learn it. The mutants job needed only the plan, so it started beside the gates. Make it need the gates as well, and start only when both the plan and the gates succeeded and the plan found mutants. The plan result is checked explicitly because the plan is skipped on a push or a merge group, where its empty count would otherwise pass the zero guard. The aggregate check job already accepts a skipped mutants job after a successful plan, and a failed gates lane fails it first. Also hold the runs to the lockfile. The Makefile mutants target passes --cargo-arg=--locked so a mutant is never judged against a lockfile cargo quietly updated, but the plan's --list and the shard's --list and run did not, so CI could test against whatever lock cargo produced and the local and CI answers could differ. Pass the flag on all three commands. A new suite pins the job's needs and condition, that each command carries the flag, and that the plan and the shard list select with the same flags so the shard count matches what runs. The development guide says the shards wait for the gate lanes and are skipped when a lane failed. Change-Id: I02d001e8746051adf98b194ddab92f7a7a0fa123
The lane tests run scripts/test.sh with its gate payload replaced, but the Python lookup above the gates is the real one. In a clone with no instructor tool environment it builds one on the first run, several lane tests running at once, so they see a failed task-tools gate or a half-built environment, and pass only on the second run once it exists. CI names its own environment and never sees it. Give run_lane the running interpreter as PYTHON, when it is 3.11 or later, with the packaging probe answered as ready, unless the test names PYTHON or CODETRIAL_TASK_TOOLS itself. The tests of the lookup and of the setup name one of them and keep the real behavior; the one that named neither now names an empty environment, which the script treats as unset. Change-Id: Idace6b303f73ae9182e538c893838cf3dc98e8da
What a gate run needs (the pinned toolchain, Clang, node, optional linters) was learned one failing gate at a time, because the gate script only records a failure and carries on. A machine without the pinned toolchain produced a wall of unrelated failures, and rustup installs the pinned toolchain the first time any rustup command runs in the repository, so even asking which toolchains exist could start a download. Add scripts/env-check.sh, which only looks and never installs. Each profile lists what the matching lane needs, and a missing tool is named with the command that installs it. It asks rustup for the installed toolchains from outside the repository, so no pin applies, before it runs cargo or rustc there, and it reports a broken cargo as a failure instead of passing it over. scripts/test.sh runs it before any gate and stops there when a required tool is missing. The optional tools that are absent come from the same report, so the per-gate probes for ruff, shellcheck, dash, actionlint, cargo-audit and cargo-deny are gone and the skip lines keep their wording. The suites use stub executables, and the system-tool PATH they need is shared in tests/support.py. The development guide gains an Environment check section. The compiler check applies on Linux only, where webrtc-sys needs Clang 21 or later; another platform builds with its own compiler. Change-Id: I77c2c1ce3d4818027dec8d1eb072c16acff452dc
A receipt carried only a fingerprint that changes on every rebase or amend, so nothing could tell whether a later commit holds what the run saw, and the next run on any branch overwrote the lane file. Add worktree_tree, the tree id a commit of the same content gets. It is computed in a scratch index, so the real index and partial staging are never touched, and it is taken at the start and the end of the run and kept only when both agree, so a tree edited during the run claims nothing, though an edit made and then undone cannot be seen. The receipt also records rustc, cargo and the environment report, and a copy is kept per tree, newest 20, because the lane file is overwritten by the next run, possibly on another branch. A tree folder written within the last hour is never pruned, since another run in the clone may have just written it and nothing locks the folder. The development guide describes each receipt field. The extractor the suites use to load a shell function is shared in tests/support.py, and a closeout check test pins that the new fields and the per-tree folder do not break reading older receipts. Change-Id: I3351b78479dc59aca590898343e88765e1d4139f
Which paths the Rust gate covers is decided nowhere, so every caller that needs to know whether a change requires a Rust receipt would copy its own list and the lists would drift. check-gate-evidence.py --scope-only reads paths from stdin (NUL-separated with -z, since a path can hold a newline) and prints in or out. A .github/ path is never in scope. Line mode refuses a path that ends in a carriage return, which would otherwise stop an exact name such as Cargo.toml from matching. The scope is src/, desktop/, vendor/, problem-bank/, scripts/hook-check/, .cargo/, the root manifests and toolchain files, the clippy, rustfmt and deny configs, and tests/ except the browser lane, the stubs and the top-level Python suites. web/ stays out through one named constant, apart from the few files a Rust source names. A list written from the directory layout misses files that Rust tests read from outside it. A test therefore scans the tracked Rust sources for include macros, folder attributes, CARGO_MANIFEST_DIR joins and string literals that are tracked files, and fails when the scope omits one. It tokenizes left to right so a quote inside a comment or a character literal cannot shift the literals after it, and a failing git raises instead of passing as an empty listing. Change-Id: I1343874fe42a09b12b6090bf03e9deede587b5e8
Approving a held run needed a person to read a contributor's claim that the Rust gate had passed, and nothing compared it with what is pushed. The evidence check turns that into a mechanical answer from data files alone: a change with no Rust-scope path owes no receipt; otherwise some receipt must be for lane rust or all, passed, name the head commit's tree, and carry the rustc that the base rust-toolchain.toml pins (the head's when the change edits that file). A bot author is exempt, a .github/ path is never in scope, and a truncated changed-file list or an unreadable pin is indeterminate rather than a pass. The receipt is untrusted: it is size-checked before it is parsed, parsed strictly (no duplicate keys, NaN, trailing data or bool-as-number), and none of its text reaches a path, a command or the output. It is a hint a dishonest author can forge, not proof the gate ran. Change-Id: I2b86f020d9716447c6c117d9761bf5a17177f3c9
A contributor with no toolchain can open a pull request that only CI compiles, and nothing on the push says whether the gate was run. The pre-push hook now asks, for a topic branch that changes what the Rust gate covers, for a passing rust or all receipt of the pushed tree, and sends it as a note on the head before the branch. Scope comes from check-gate-evidence.py, measured from the first main that resolves, looked up once per push; with none the hook says so and refuses nothing. A refusal prints the environment report, and when a run changed a tracked file while it ran, which writes no by-tree receipt, it says that too. The note is built by scripts/gate-note.py: one JSON object, bounded in size and in every field, that copies only the keys of the schema from the receipt files and drops what is malformed. It is sent only when the remote does not already hold it: the hook leaves a local note that already holds the blob alone, and after fetching the remote notes it drops every head whose note the remote holds with the same blob, so an unchanged head causes no second upload. The review-record block of the hook is left as it is and the new one sits beside it, because that one is gated by a completed review and keeps its own state. The receipt can be forged and a hook can be skipped, so CI and approval are what bind. The development guide describes the check and the note, and the hook tests cover them. Change-Id: Ia17dd2dd4dfe9cae131d115bda6e1ac32cf708f3
Approving a held run on gate evidence needs the note a contributor pushed to the fork, and that fetch reaches a repository nobody here controls. Written ad hoc in each caller it would drift: one would build the URL from PR text, another would skip the size check. scripts/fetch-gate-note.sh takes the head owner, name and sha as separate arguments, validates them, builds the URL itself and fetches only refs/notes/codetrial-gate into a scratch bare repository: anonymous, one commit deep, a 60 second limit, a cap on the size of any one file it writes, and the note blob sized before it is read. Each failure has its own exit status, so a caller can tell a note that is absent from a fetch that did not answer. Change-Id: I000fe438ae75773c1c2e224849e8f942a607f902
A sweep approves the runs GitHub holds for first-time contributors without looking at what the contributor ran locally, so a PR from an agent with no Rust toolchain still spends the full Rust lane to learn whether the code compiles. gate-evidence.sh gathers the head tree, the changed paths (renames name both ends, a cut list is indeterminate), the toolchain pin from the base branch and the note from the fork as data, and asks check-gate-evidence.py. pr-sweep.sh reports the verdict beside each held run. apply-pr-actions.sh reads it again for the head the sweep saw immediately before the approval call, approves only on a pass, leaves an ask on the confirmed list as the toolchain comment to draft, and skips when the answer cannot be given: a moved head, a deleted fork, a failed fetch. pulls.md now says what the pull_request_target grep prints, which is pr-description.yml alone and the one reviewed exception, and drops the wording that read as approving every held run at once. Change-Id: I041ce0f91479a3e11a60fdfd913d9e74f752d018
A pull request that changes Rust and carries no gate receipt spends a full Rust build before anyone learns the receipt is missing. Add a preflight job that runs the base branch's evidence check over the head as data and fails first. The gates, the mutation plan and the aggregate status wait for it, and a push or a merge group still runs the gates because preflight is skipped there. It is cost control, not a security boundary: a pull request runs its own workflow. The reason the gatherer reports is built from data a pull request writes, and jq decodes a JSON newline into a real one, so printing it as it came could put a line starting with :: in the log: a forged error or a mask. The step keeps the verdict out of the log, captures the gatherer's stderr, and reduces the reason to one line of printable ASCII behind the step's own error prefix, written with printf because dash's echo turns a backslash and n into a line break. The tests pin the job's shape, run its script and the aggregate script over each outcome, and run the step under dash against hostile output on both streams, requiring that the only :: line is the step's own. The development guide describes the job. Change-Id: Ib0a11991ce865d142596bb4ddd74fa6c2dbb01b1
A contributor with no Rust toolchain can push a change that only CI compiles, and the first sign is a failed run. The rule was stated in no place a contributor reads before a push. Add a Before you push section to CONTRIBUTING.md with the order to work in, for a person or an automated agent: run the environment check and stop when it fails, run the gate and push only after it passes, never bypass a hook, and report what was run and what was not. The same file states which paths need a local test.sh pass (the directories Rust reads, the root manifests, the toolchain and lint configs, and the few web/ files a Rust test reads), what the pre-push hook measures the branch from, how a receipt that names a tree matches a commit, and that the receipt is a hint while the maintainer's approval and the first CI job are what bind. The pull request template asks for the command that was run and the end of its output, and the contribute and verify skills each point to the new section. Change-Id: I70001cb5b242f673baa936912d3210e56a1241a6
The note builder and the evidence checker each have a suite built from hand-made inputs, so a field one side writes and the other no longer reads would pass both. Run the receipt writer of scripts/test.sh in a scratch repository, build the note from what it wrote, and judge it: accepted for its tree, refused for another tree, for a lane that does not build Rust, for a failed run and for a compiler other than the pinned one. A malformed receipt is skipped by the builder, and the checker refuses an entry whose field has the wrong type or size. CONTRIBUTING.md and the scope function of scripts/check-gate-evidence.py each list the Rust scope, so an edit to one left the other stale without any check failing. Require the text to name every directory and file the function lists, with the tests/ exclusions stated, and every path it names to be classified the way it says. web/ stays out as a folder, but the text must say that a few web/ files a Rust test reads are in. The note's size, lanes, tree id shape and receipt count are written in the builder, the checker and the fetcher, so a test requires the three to agree. Change-Id: I1c2f82fc951116ff17ab35b419a866e1d0cde485
A contributor asked to reproduce a failed mutation shard, or a reviewer's request for a mutation run, had only make mutants, which audits a fixed set of files whatever the branch changed, so it can pass while the diff has a surviving mutant and fail on code the branch never touched. Document the diff-scoped cargo-mutants commands CI runs, with the shard option and the note that --in-place mutates the working tree. The commands add --cargo-arg=--locked, which the Makefile target passes and which keeps a mutant from being judged against a lockfile cargo quietly updated, and the text says it is diff-scoped like CI's check rather than identical to it. The cost is explained by its cause: a run costs the number of mutants times the suite's time, and the mutant timeout follows the baseline's test time, which is why it is for reproducing a failure and not a per-change requirement. A test pins that both documented commands carry the flag and select what the shard step selects. Change-Id: I8d6327f833de7298c79a0d4e71179b8514ced610
jserv
force-pushed
the
pr/gate-evidence
branch
from
November 17, 2026 00:21
5e39b3c to
65af763
Compare
alanhc
approved these changes
Nov 17, 2026
alanhc
left a comment
Collaborator
There was a problem hiding this comment.
LGTM. I read the workflow, gate-evidence.sh, fetch-gate-note.sh, the check_rustc change and the pre-push receipt path:
preflightruns only the base branch's scripts with a read-only token, and its output is cut to printable ASCII under its own prefix, so a fork cannot emit a workflow command.gatespasses on a skipped preflight, so push and merge group are unchanged.gate-evidence.shreads the head before and after the check and fails closed on any API error, a file count mismatch or a NUL in a path.- The fetch is anonymous, https-only, one commit of the notes ref, with the blob size checked before it is read;
--batch-checknow tells a missing blob (7) apart from a failed probe (5), and theulimit -fkill lands on 5 as well. - The dedup in the pre-push hook is right in each case I traced: an unchanged note is neither rewritten nor uploaded, a first push with no remote notes ref still sends, and other heads' notes survive the
-s oursmerge. It relies on the note bytes being deterministic, which the new test covers.
One thing worth doing after merge: the description says preflight has not run against a real fork yet. repos/{base}/git/commits/{sha} should resolve a fork head through refs/pull/N/head, but a single fork PR touching Rust would confirm both that and the anonymous note fetch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and behavior
A pull request from a contributor with no Rust toolchain installed gets its first compile from CI, so the Rust lane and every mutation shard are spent learning that the build fails, and a held first-time run could be approved without anyone checking what the contributor ran locally. After this change a missing tool is reported before any gate runs, a passing
./scripts/test.shrun leaves a receipt that names the tree it saw, the pre-push hook refuses a push that touches Rust sources, tests or build files without a matching receipt and sends the receipt as a note onrefs/notes/codetrial-gateonly when the remote does not already hold that note, and both a first CI job and the maintainer's approval step read that note from the fork as data and either fail early with the install steps or ask the contributor for the toolchain. Mutation shards now wait for the gates and run under--locked, so a pull request that already fails the Rust lane does not pay for every shard to rediscover it.CONTRIBUTING.mdstates the local-first rule for anyone sending a change, a person or an automated agent, in a Before you push section that the contribute and verify skills point to.Approach
scripts/env-check.shis the one owner of what a run needs andscripts/test.shstops on it before any gate, instead of each gate probing for its own tool.scripts/check-gate-evidence.pyis the one scope function, used by the hook, the checker and the maintainer sweep, and its tests pin it to the paths the Rust sources read and to the texts that name them.scripts/fetch-gate-note.shis the only code that fetches a fork's notes: it takes the owner, repository and sha as separate arguments, validates them before any git call, builds the URL itself, checks the note's size before reading it, and runs git under a cap on the size of any one file it writes so a fork's notes cannot fill the disk. The checker reads only data files and runs nothing, and the CI job checks out the base only and prints at most one sanitized line, so nothing from a fork can start a workflow command in the log. The note is evidence from the contributor's machine and can be forged: it makes the maintainer's decision mechanical and says nothing about whether the gate ran, and the CI job is cost control, not a security boundary, because a pull request runs its own merged workflow.Known limits:
web/that Rust tests read are in scope, so a change to only those needs a receipt; the rest ofweb/is not.tests/harness/harness-audit.test.jsanswers in scope, because everything undertests/is in scope except the browser lane, the stubs and the top-level Python files; the file runs only under Node, so the answer asks for a Rust receipt that the change cannot affect, which errs toward asking.envfield has no reader yet.CONTRIBUTING.md, so an agent that never reads it is not reached; the pre-push hook and the CI job are what enforce it.Validation
I ran
./scripts/test.sh portableand then./scripts/test.sh rust, one after the other, on the final tip after the rebase onto the currentmain, and both exited 0 withdesktop-fmtincluded; the portable run includes the new C and C++ harness audit from the currentmain(66 tests, none skipped), and it skipped prettier, eslint and script-requires because npm is not installed here, while the rust run skipped the browser tests, desktop clippy and tests, andcargo-deny. I also ransh scripts/test-git-hooks.sh(501 checks), the twelve Python suites that cover the new scripts (620 tests),shellcheck,dash -n,actionlint,ruff checkandruff format --check,scripts/check-skill-refs.py,scripts/check-commit-log.shandscripts/hashchain.py verify, and broke each new rule on purpose to check that a test fails, including the new test that the hook skips an unchanged note and the one that bounds the file size of a fork's fetch. Each commit passes the suites it adds when checked out alone, and the first run oftests/test_test_gate.pyin a fresh worktree passes.Not run: anything on GitHub (the preflight job, the held-run approval against a real fork, and
gh apiagainst a fork), the browser lane, desktop clippy and tests,cargo-deny, prettier and eslint, and macOS or Windows.