[MaxGhenis]

Automate the results PR for a certified UK replay qualification

Step 4 of the per-release flow was manual. Someone committed the receiver's certified namespace, record, CI matrix entry and test updates by hand: #188 for 6.4.1 and #190 for 6.5.0. With this PR, the receiver opens that PR itself. Merging stays on the normal gates: CI and an independent review. Nothing auto-merges.

How it works

• Trigger. When the record job (.github/scripts/uk_replay_record.sh) writes terminal status complete, it dispatches .github/workflows/uk-replay-results.yml with the request_id. If that dispatch fails, the job warns and the queue continues. The workflow can also be run by hand. Its only trigger is workflow_dispatch, so it never runs on pull_request. Its single job runs only on the default branch's code, and each request gets its own concurrency group.
• Builder. .github/scripts/uk_replay_results_pr.sh and pipeline/uk_replay_results_pr.py (check, apply, scope, body) take these steps in order:

  1. Read requests/<rid> from replay-state and pass it through the receiver's output guard again.

  2. Check eligibility. The request needs STATUS complete, a receipt that validates and matches STATUS, a REQUEST.json that hashes to the id, and a results/VARIANT.json that is certified, is the receipt's variant and was built from this request. Failed, rejected and candidate runs, and variants already committed, end with a notice.

  3. Refuse anything inconsistent:

     • a results tree holding anything beyond the namespace, its release diff and its freeze;
     • a namespace receipt that differs from the request's receipt, or a manifest or diff the receipt doesn't bind;
     • any path that main already holds;
     • results changed after the receipt commit;
     • a policyengine version older than the newest certified one (appended, it would become latest_certified);
     • a variant id that couldn't name a branch and paths.

     An exact re-qualification of a committed pair (W4's determinism check returns the committed record, note included) is skipped, not refused.

  4. Extract requests/<rid>/results/{results,docs} with git archive | tar -x --strip-components=3.

  5. Append VARIANT.json plus a note to the index with compute_uk_event.canonical_bytes.

  6. Add the matrix entry. After the edit, the matrix must list exactly the certified variants, and nothing else in ci.yml may change.

  7. Move NEWEST in the qualify and determinism tests and extend COMMITTED in the variants test.

  8. Add a RELEASE_REPLAY.md paragraph built only from receipt counts: integrity, grids and diff flags.

  9. Commit, and refuse the commit if it touches any other path.

  10. Check that every committed result file has the blob replay-state holds.

  11. Run verify-sealed (the new namespace, then all of them) and CI's uk_replay_regenerate.sh, then build the database twice with scorecard_db.build_db: the content hashes must match and no tracked file may drift.

  12. Mint the App token (next bullet), then push uk-replay-results/<variant_id>, force-pushing only over the builder's own commits for this request, and open the PR. It opens as a draft when integrity is not complete.

• Authority, in two jobs on two runners.
  • build runs steps 1–11. Its GITHUB_TOKEN is contents: read, the checkout persists no credential, and it holds no secret. It outputs the verified tree, the main and replay-state commits it used, and two check results.
  • propose runs on a fresh runner, with no uv or installed package and its actions pinned to commits. It assembles the same commit again with git and python3 -I -S (stdlib only) on the commits build used, which must be in main's and replay-state's history. It refuses unless its tree equals build's. Only then does it mint a token for the organisation's policyengine GitHub App, with actions/create-github-app-token (v2.2.2, pinned) and the org secrets APP_ID and APP_PRIVATE_KEY. The token is scoped to this repository with contents, pull-requests and workflows write, the same App policyengine-us's weekly lock PR and policyengine-api's bundle update use.
  • Two jobs, not two steps: steps in one job share a runner, so anything the checks' PyPI packages did there (a $GITHUB_PATH shim, say) would reach the step holding the token.
  • Neither job holds a dataset token, and neither installs, imports or runs candidate or release code.

Why an App token, not GITHUB_TOKEN

The first commit here used GITHUB_TOKEN and dispatched CI on the branch. That design could not work, for two reasons:

• Workflow files. The commit edits .github/workflows/ci.yml. A push that changes a workflow file needs workflows permission, which GITHUB_TOKEN can never hold.
• Checks. A PR opened with GITHUB_TOKEN starts no pull_request run (docs). Checks from a workflow_dispatch run don't count for a PR (docs), so gh pr checks would never have passed.

gh api /orgs/PolicyEngine/installations shows the policyengine App installed on all repositories with contents, pull_requests and workflows write. Its push and PR start ordinary pull_request CI.

Tests made editable, discovery made stable

• tests/test_uk_replay_variants.py: the committed list is now one COMMITTED tuple the builder appends to.
• tests/test_uk_replay_determinism.py: POST_EVIDENCE pins the first namespace sealed after Check processed replay values; revise AB2024 blended CGT for day-weighting engines #186, so NEWEST only drives the base_namespace check.
• tests/test_uk_replay_discover.py: the floor cases assumed 6.5.0 was the newest certified variant, and appending 6.6.0 would have broken eleven of them. They now read the index as of 6.5.0 (floor_document()). Their assertions are unchanged.

Invariants

Each has deterministic cases, and Hypothesis properties where it ranges over inputs (tests/test_uk_replay_results_pr.py):

• Append-only index. The new index is the old one plus one record, in canonical form.
• Matrix. After the edit, the matrix lists exactly the certified variants; nothing else in ci.yml changes.
• Paragraph. It states every nonzero flag count and no zero one.
• Scope. A commit touches the extracted results and the six edited files, nothing else.
• Bytes. Every committed result file has replay-state's blob. A sealed file under a path main's .gitignore ignores fails this check instead of being dropped silently.
• Idempotence. The same request on the same main pushes nothing the second time. A rebuild force-pushes, with a lease, only over exactly one commit that the bot both authored and committed, with this request's trailer, so it never overwrites a person's commit, amended ones included.
• Same tree in both jobs. The propose job pushes only a commit it built itself, and only when its tree is the one build verified.
• Order. Every guard runs before the commit or push it protects.

Validation

• Hand-built output reproduced. The 6.4.1 note comes out byte for byte. Appending the 6.4.1 and 6.5.0 records reproduces the committed index, and appending 6.5.0 to the matrix reproduces ci.yml. The 6.5.0 paragraph's integrity and grid sentence matches Certify the policyengine.py 6.5.0 replay results, the first with the processed-value check #190's hand-written one word for word.
• Real-data differential through both jobs, at 78e080c6. The job-1 build (strict, the real checks) and the fresh-clone job-2 stdlib rebuild produced the same tree, 803717e2. That tree differs from the hand-built Certify the policyengine.py 6.5.0 replay results, the first with the processed-value check #190 only in the note and the paragraph. All three phases took 84 s.
• Real-data differential (local, not committed), earlier heads. I removed Certify the policyengine.py 6.5.0 replay results, the first with the processed-value check #190's changes from a copy of this branch. The builder then rebuilt Certify the policyengine.py 6.5.0 replay results, the first with the processed-value check #190 from the real replay-state request d83eaf59… against a local bare remote, with the real verify-sealed, regeneration and build_db, running build then propose as the workflow does. It took 88 s. The bot's tree equals the hand-built one in all 249 changed paths except two generated texts: the record's note (Certify the policyengine.py 6.5.0 replay results, the first with the processed-value check #190's added the Check processed replay values; revise AB2024 blended CGT for day-weighting engines #186 context) and the RELEASE_REPLAY paragraph (counts only, no CGT interpretation). Same result at 71161e9, ed76b45 and 854081d.
• End to end on a local bare remote, with a stub gh:
  • one bot commit plus a PR;
  • an idempotent rerun;
  • a rebuild after main moves;
  • refusal to overwrite a person's commit;
  • each ineligible and inconsistent case;
  • each failed check stopping before any push;
  • a draft PR for incomplete integrity;
  • an exact re-qualification skipped;
  • a hostile variant id refused;
  • propose refusing a commit that moved after verification, or a missing build output or token;
  • the record job's dispatch, including when the dispatch fails.
• Mutation runs. Each of these safeguards, removed, fails a test: the byte-identity check, the guard, the receipt-commit check, the version order, the scope's missing-edit check and the main-collision check.
• Test runs. At 78e080c6, the targeted replay test files pass locally with -n 4: results-PR 62, and the other six files 335 passed, 1 skipped. At f7247f19, local runs were held for Fleet ops' disk pause, and CI found four failures, fixed in 78e080c6.
• Formatting. ruff format --check is clean.

Independent review

• Hard-tier review of 97c351b9: REQUEST_CHANGES. Report in w6-lanes/reviews/review_193.md. Two blocking findings, both fixed in f7247f19:
  1. The build/propose split was two steps in one job, so the build's PyPI packages could reach the token through the shared runner. Now there are two jobs, and the second rebuilds with the stdlib only.
  2. An amended bot commit kept its author and trailer, so a rerun could overwrite a reviewer's edit. Now the committer must be the bot too, and the branch must hold exactly one commit.
• Non-blocking items fixed:
  • an id main holds for another identity is refused instead of skipped;
  • the actions in the token job are pinned to commits;
  • new tests for the force-with-lease race, the gh credential helper and the recorder dispatching nothing for a failed request;
  • grammar in the PR-body table;
  • doc wording.
• Kept as is: reading the CI matrix from the index, which would drop workflows: write. The brief asks for the variant to be added to ci.yml's matrix, and the token no longer shares a runner with third-party code. That change is a separate follow-up.
• Delta review of f7247f19: REQUEST_CHANGES. Report in w6-lanes/reviews/review_193_delta.md. It confirmed the two-job split closes finding 1. Two blocking findings remained, both fixed in 78e080c6:
  1. CI had four failures. Three were test bugs: a prose match, a probe the lazy imports made vacuous, and a stub dir colliding with the fixture.
  2. The fourth was a regression: rerunning a merged request failed instead of skipping, because main's record carries the note. Records are now compared without it.
• Non-blocking items from that review, also fixed:
  • job 2 refuses a base older than the workflow's commit, and decides again on main's tip, so a rerun after the merge skips;
  • the credential test also runs the real gh;
  • a test checks the stdlib and strict matrix edits are byte-identical on the real ci.yml;
  • doc wording.
• Second delta review of 78e080c6: APPROVE (hard tier). Report in w6-lanes/reviews/review_193_delta2.md. It found no blocking issue and six non-blocking ones, which a follow-up PR handles: the tip check running before the guard, comparing records by identity, two tests that are weaker than they look, a positive GITHUB_SHA run, and a comment nit.

Not verified until it runs on GitHub

The App token path (minting, checkout with the token, the push, PR creation and the PR's pull_request checks) can only run once this workflow is on main. After merge, I will dispatch it for the 6.5.0 request. That request is already committed, so the run exercises checkout, fetch, guard and check, then ends with a notice; the token is minted only for a built commit, so minting, the push and the PR first run on the next certified release.

axiom: n/a: CI automation and tests; no policy encoding.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policyengine-scorecard	Ready	Preview	Oct 18, 2026 6:13pm UTC