[MaxGhenis]

CI's replay-engine matrix now comes from data/uk/replay_variants.json, not a list hand-kept in ci.yml. The results-PR bot therefore never edits a workflow, and its App token loses permission-workflows: write. This was finding 4/6 in both independent reviews of #193 (review_193.md and review_193_delta.md), left as a separate change because it alters step 3 of the original brief. It builds on #194 (merged as 9321a7b5) and was rebased onto main after #194 and #197 merged; the three commits' stable patch-ids are unchanged from the ones reviewed (a902c91c…, 2a3a711a…, 728cef58…).

Changes

1. The matrix is listed from the index (.github/workflows/ci.yml, new .github/scripts/uk_replay_engine_matrix.py).
   • A new job, certified replay variants (replay-variants), has contents: read and no persisted credential. Its checkout is sparse, holding only the script and the index; actions/checkout v4 fetches those with --filter=blob:none.
   • It runs the script with the runner's own python3 -I and the standard library, and emits the certified ids in index order as one JSON line.
   • replay-engine now needs it, with matrix.variant: ${{ fromJSON(needs.replay-variants.outputs.variants) }}.
   • Unchanged in replay-engine: the job name replay engine · <variant>, fail-fast: false, contents: read, and every step and condition. The dataset token is still only in the push-only fetch step, and tests/test_uk_replay_queue.py's token pins pass unchanged.
   • The lister fails the job, and so every leg, rather than list fewer variants. It refuses any of these:
     • a record that is not an object;
     • an id that is not plain: letters, digits, ., _ and -, starting with a letter or digit, no .., at most 120 characters. CI puts ids in job names, environment variables, shell commands and paths. The character rule is the one the builder already applied to a new variant. The length cap, now in both, keeps two ids joined by __ (a release diff's directory) within one 255-byte path component, and keeps the job output near 63 KB at most, under GitHub's 1 MB per job.
     • a status outside certified, candidate and retired;
     • a repeated id;
     • no certified variant;
     • more than GitHub's 256 jobs per matrix.
2. The builder no longer edits ci.yml (pipeline/uk_replay_results_pr.py).
   • Removed: add_matrix_variant, the MATRIX regex, CI and its entry in EDITED, and the only yaml import.
   • apply now needs only the standard library in both jobs, so --stdlib remains only on check.
   • scope now refuses any path under .github/ explicitly, before its other checks. This is defence in depth: check already pins every result tree under results/ and docs/.
   • The plain-id rule is now plain_variant_id, which the lister must agree with (a differential test checks this).
   • The commit message and the PR body say the variant record is what adds the variant to CI's matrix.
3. The token has no workflows permission (.github/workflows/uk-replay-results.yml). permission-workflows: write is removed and the header comment rewritten.
4. The build job's control environment loses pyyaml (.github/scripts/uk_replay_results_pr.sh).
   • Fix the non-blocking review findings on the UK replay results PR builder #194's transitive import scan no longer reaches yaml, so CONTROL is uv run --with rfc8785==0.1.4 --with jsonschema python.
   • The scan's pin and the uv-emulation probe are updated to match.
5. What a refused rebuild says (propose).
   • The bot's own commit never touches a workflow. But a branch rebuilt on a newer main also carries main's commits since the old tip.
   • GitHub documents the permission ("If your app specifically needs to access or edit Actions files in the .github/workflows directory, request the "Workflows" repository permission", docs). It does not say which commits a push is judged by.
   • If the force-push is refused and main changed .github/workflows since the old tip (at the tips, or in any commit between, even one later reverted or merged in from a side branch: the check reads git log --full-history, since default path-limited history can prune such a merge), the run now fails with an error that says so and gives the fix: close the pull request, delete the branch and run again. A new branch holds only the bot's commit. The branch and its pull request are left as they were.
   • I tried to settle GitHub's behaviour with a GITHUB_TOKEN probe on a private scratch repository. Actions could not start there: private-repo minutes on the account are blocked by billing.
6. Docs (docs/uk_replay/RELEASE_REPLAY.md): updated "Replay in CI", the propose row of the job table, the App-token paragraph, step 3 and step 5 of "The results pull request", the rerun paragraph, and a new Engine matrix invariant.

What dropping workflows buys, and what it does not

Without workflows, a leaked token can no longer push a workflow that reads a secret. It can still push to any branch, main included. Main has no classic protection (gh api repos/PolicyEngine/policyengine-scorecard/branches/main/protection returns 404 "Branch not protected") and no ruleset (rules/branches/main and rulesets?includes_parents=true both return []). The scripts and pipeline code that secret-holding jobs run live outside .github/workflows. The doc says this. Closing that route means protecting main, or limiting the App's pushes to uk-replay-results/*, which is a repository-settings decision; I've queued it separately.

Invariants (all in tests/)

• Engine matrix (Hypothesis). For any index the lister accepts, the matrix is exactly the certified ids in index order. One bad record anywhere refuses the whole matrix.
• Differential.
  • The lister's plain-id rule and the builder's plain_variant_id agree on any string.
  • On the ci.yml bytes Certify the policyengine.py 6.4.1 replay results from qualification attempt 3 #188 and Certify the policyengine.py 6.5.0 replay results, the first with the processed-value check #190 actually edited (the Fix the non-blocking review findings on the UK replay results PR builder #194 fixtures, pinned by blob id), each hand-kept matrix plus that PR's variant equals the lister's output on the index as the PR left it.
• Wiring. The listing step's own run: text, run with bash on a copy of the checkout, writes exactly variants=<certified ids>; a bad index fails the step with nothing written. The engine job's matrix is that output. No workflow names a variant id.
• Scope (Hypothesis). A results commit touching anything under .github/ is refused, even for a plan whose trees lie there.
• End to end.
  • The pushed commit leaves main:.github byte-identical, and the lister on the pushed index ends with the new variant.
  • A pre-receive hook simulates GitHub without workflows, under the strictest reading: it refuses a bot-branch update whose new commits, or whose old-to-new change, touch .github/workflows. Under it:
    • the first push passes;
    • a rebuild over a main that changed something else passes;
    • a rebuild over a main that changed ci.yml is refused with the fix in the error, and the branch is unchanged;
    • after the branch is deleted, the rerun pushes a fresh branch;
    • a workflow change that main then reverted still gets the hint (the tips match, but the range carries it), including a change and revert merged in from a side branch, which only --full-history shows.
  • The lease-race test now also checks that an unrelated refusal prints no workflows hint.

Reviews

• Hard-tier review of 6866605d: APPROVE, three non-blocking findings (~/PolicyEngine/_drafts/continuous-budget-ci/w6-lanes/reviews/review_matrix_from_index.md), answered in 34fba891.
• Hard-tier delta review of 34fba891: APPROVE, one non-blocking finding (review_matrix_from_index_delta.md), answered in b7b23575.

Verification

• Before the rebase, Fleet ops' hold allowed single targeted files (free disk 60–70 GB, above the 45 GB pause line). I ran each file alone, serially or at -n 4, with a scratch basetemp removed afterwards. After the rebase, disk pause App design: public scoreboard + mission control, four views #9 holds local pytest, so this PR's CI is the run on the rebased commits:
  • tests/test_uk_replay_engine_matrix.py: 15 passed
  • tests/test_uk_replay_results_pr.py: 82 passed
  • tests/test_uk_replay_variants.py: 87 passed, 1 skipped
  • tests/test_uk_replay_discover.py: 126 passed
  • tests/test_uk_replay_determinism.py: 12 passed
  • tests/test_uk_replay_queue.py: 22 passed
  • tests/test_ci_gates.py: 8 passed
• ruff format --check, bash -n and shellcheck are clean. actionlint reports only SC2129 on the existing "Install the variant's exact environment" step, which is also on main.
• Mutations: removing the range check from the hint fails the revert case, and dropping --full-history fails the merged-revert case. The hard-tier review also mutated the lister (dropping a variant, accepting a $(...) id, skipping the status check), scope's .github/ refusal and the hint; each was caught.
• CI runs the full suite.

After merge

I'll dispatch uk-replay-results.yml for request d83eaf590d3ebe846506d9ce9be399fe5b9b95dc58362d573d4e19fd501a649a (the committed 6.5.0 qualification). It must end with the skip notice: pe6.5.0_uk2.130.1_policyengine-uk-data-1.58.1 is already in data/uk/replay_variants.json.

axiom: n/a: CI wiring and the results-PR builder; no policy encoding.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policyengine-scorecard	Ready	Preview	Oct 19, 2026 3:37pm UTC

[MaxGhenis]

Merged on the gates: CI green at the reviewed head; MERGEABLE; no changes requested. Independent review: an Opus 5.5 reviewer, standing in for the GPT-6.1 Sol run, which is queued behind the closed Codex lanes. It found all four stack PRs (#199, #196, #201, #202) MERGE-SAFE, and the full suite passed with #206 on top (2313 passed). Review: ~/reviews/budget-coordinator/scorecard-results-stack/review-out-opus.md. The Sol run stays queued as a second opinion.