[MaxGhenis]

No workflow that runs on pull requests names a secret any more. publish-db, the job that holds SUPABASE_SCORECARD_URL and SUPABASE_SCORECARD_KEY, moves out of ci.yml into a new publish-db.yml, triggered by workflow_run when CI completes on main.

Follow-up to #215, which did the same for the dataset token.

Problem

ci.yml runs on pull requests. Its python and replay-engine jobs install and run PyPI packages and each certified variant's pins, as a runner user with passwordless sudo. Its publish-db job was push-only, but it named the Supabase secrets.

A debug re-run of #215's pull-request CI (job 119587048984) evaluated a secret that its job named only in a skipped step. GitHub scrubs the secrets "not referenced in the workflow". So the pull-request runners in ci.yml may have held the key.

Nobody has shown that a job receives a secret named only by another job of the same workflow. Moving the job settles the question without having to prove it either way.

What changes

• New .github/workflows/publish-db.yml. It runs on: workflow_run: workflows: [CI], types: [completed], branches: [main], and has one job, publish db to supabase storage. The job:
  • Runs only for CI's own pushes to main. workflow_run runs main's copy of the file with the repository's secrets after any CI run, a fork's pull request included. The branches filter names a branch, not an event or a repository. So the job's if also requires the triggering run's event == 'push', head_branch == 'main', head_repository.full_name == github.repository and path == '.github/workflows/ci.yml'.
  • Interpolates nothing into a shell. The run id, attempt and commit reach the scripts as CI_RUN_ID, CI_ATTEMPT and CI_SHA; no script contains a ${{ }}. No script reads GITHUB_SHA: in a workflow_run run it is not the CI commit, and a re-run keeps its original value.
  • Publishes what CI built and tested. It reads the pytest job's conclusion in the CI attempt that fired it, as needs: python did, and lists the CI run's artifacts with the image's gh. Unless that job passed and scorecard-db-<sha> is there, it skips with a notice. Otherwise it downloads the artifact with download-artifact (run-id + github-token), now pinned to v4.3.0's commit, the one v4 names today.
  • Runs no package code beside the key. There is no checkout, no uv and no Python: only download-artifact and the runner image's gh, gzip and curl.
  • Holds actions: read and contents: read, and nothing else. Reading CI's jobs and artifact takes the first, and reading main's tip the second. There is no checkout.
  • Publishes one at a time, with queue: max. A job-level concurrency group stops a publish for an older tip from overwriting latest.db.gz behind a newer one. By default GitHub cancels a group's pending job when another enters, so an older run's completion could drop the publish for main's tip; queue: max keeps up to 100 waiting publishes. A workflow-level group would hold every run of the file, the ones the guard skips included.
• ci.yml loses the publish-db job. Its permissions comment now says where publishing happens.
• scorecard_db/README.md says where publishing runs, and that every publish since 6 October has failed.

Why workflow_run, not a push workflow that rebuilds the database

The task offered both. A rebuild (python -m scorecard_db.build_db) runs the project's PyPI packages. Run in the job that holds the key, they sit next to it. Run in a sibling job of the same workflow, they depend on exactly the boundary that is unproven. A rebuild would also publish a database whose test suite never ran: before, needs: python meant publishing waited for the format check, the determinism check, the no-drift gate, the suite and the replay regeneration.

workflow_run keeps that gate and those exact bytes, and puts the key where no package code runs. Its costs are two read scopes on one job and the guard above.

Behaviour

Kept, as in the old step:

• Unset secrets skip with ::notice::Supabase secrets not set — skipping publish.
• A failed upload stops at that object, posts ::warning title=Supabase publish failed::… and a step-summary line, and exits 0.
• curl runs -sf with its output discarded, so the host in the secret URL is never printed.
• A failed pytest job publishes nothing.

Changed:

• The notices and warnings now appear in the "publish db" run, not in CI's.
• Publishing happens when the whole CI run completes, not right after the pytest job.
• The failure message now points to the CI run that holds the artifact.
• Each upload has two minutes, and one that hits the limit counts as a failed upload, so a stall warns and lets the next publish start. The find and download steps have five minutes each, and a stall there fails the run. A real publish takes about ten seconds.
• latest.db.gz moves only for the commit at main's tip, read from the API as the job starts. <sha>.db.gz is uploaded either way, and a notice says when latest was left alone.
  • This stops a re-run of an older CI run, or of an older "publish db" run, from rolling latest back. Under workflow_run, any re-run of a CI run triggers a publish, and a re-run keeps its original GITHUB_SHA, so only the live tip tells them apart.
  • Edge case: a run that a newer push cancelled no longer updates latest on its way out. If the newer commit's pytest job then fails, latest stays one commit further back than it would have before.
• A cancelled CI run whose pytest job had already uploaded still publishes its <sha>.db.gz, as before. Run 39694074043 was cancelled and its publish-db still succeeded.

Invariants (tested)

1. No workflow that runs on pull requests names a secret. This covers pull_request, pull_request_target, pull_request_review, pull_request_review_comment, issue_comment and merge_group; the allowed set is empty (tests/test_ci_gates.py).

   • The scanner reads every ${{ }} as the runner's TemplateReader.ParseScalar does, so a }} inside a quoted string doesn't end it. It also reads every if in full and treats a job's secrets: as every secret.
   • It catches secrets.X, secrets['X'] and secrets["X"], toJSON(secrets), computed names, any case and spacing, secrets: inherit, and }} hidden in string literals, with or without an escaped quote.
   • It ignores inputs.secrets, the word in a script, and comments.

2. Only publish-db.yml names the Supabase secrets. They appear only in its upload step's environment, and it has no pull-request trigger. It is the only workflow_run workflow, so a new one gets the same guard and its own test.

3. The guard admits only CI's pushes to main here. It is evaluated over a push to main in this repository (admitted) and seven refused cases:

   • a pull request
   • a fork's pull request from a branch named main
   • pull_request_target
   • workflow_dispatch
   • another branch
   • a fork
   • another workflow file

   The evaluator compares strings case-insensitively, as GitHub does.

4. It downloads exactly what ci.yml's pytest job uploaded, and only when that job passed in the attempt that fired it. The name is scorecard-db-<head_sha> and the run is the triggering one. The find step asks for that same name, and for the conclusion of the job ci.yml names, so a rename fails a test. The upload is that job's last step, with no if and no continue-on-error in the job, so any artifact of that name passed its checks, whichever attempt uploaded it.

5. No package code and the narrowest token. The job's only action is the pinned download-artifact. No script calls uv, pip, Python, bun, npm, node, make, bash or sh. github.token appears only in the find and download steps, and the workflow and job environments name no secret.

6. Behaviour under bash -e, the shell the job log shows, with stub gh and curl:

   • Find reports found, or posts a notice when nothing was uploaded or the attempt's pytest job did not pass; any failed API call fails the step.
   • An unset secret posts a notice, never runs curl and writes no .gz.
   • Success uploads <sha>.db.gz and then latest.db.gz, with the old headers and a payload that gunzips back to the database.
   • When main has moved on, only <sha>.db.gz goes up, even with GITHUB_SHA equal to the CI commit, as on a re-run of an old publish.
   • A failed upload is one warning and one summary line, exits 0, and leaves the host and the key out of every output.
   • The curl stub prints the host whenever -s is dropped, so that check has something to catch.

7. latest.db.gz moves only to main's tip. It lags main while the tip's pytest job has not passed, or after a push that runs no CI. See 6; no script reads GITHUB_SHA, the job holds a queue: max concurrency group, and neither it nor the publish step has a time limit of its own, so no limit can end it inside an upload, before the warning. Each upload has its own limit, and the find and download steps five minutes each.

Each of these 49 mutations fails at least one test:

• triggers and guard: a pull_request trigger; dropping the event or repository clause; || in the guard
• secrets: a secret in the job env; the Supabase key or toJSON(secrets) back in ci.yml; renaming CI
• the upload script: GITHUB_SHA for the object name; curl without -s or without its time limit; a changed curl limit; a time limit on the job or the publish step, or none on find or download; exit 1 on a failed upload; the unset-secrets skip failing; latest always, never, or compared with GITHUB_SHA; the tip missing from the step's environment
• the find step: ignoring pytest's conclusion or the artifact count; another job's conclusion; the latest attempt in place of this one; tolerating a failed read of the jobs or the tip; dropping the tip output or CI_ATTEMPT
• the steps: a ${{ head_branch }} in a script; adding a checkout; adding a pip install; counting expired artifacts; downloading unconditionally; another run id
• the token: actions: write; contents: write; no contents scope
• the action pin and concurrency: unpinning the download; a workflow-level group; no job group; the default queue, queue: single or cancel-in-progress
• ci.yml: renaming CI or the pytest job; an if on the upload; a step after it; continue-on-error in the job

Checked live

• The find step's exact script, run with the real gh:
  • gives found=true on main's run 40394071052, and on run 39694074043, which a newer push cancelled after its pytest job passed;
  • gives the notice on attempt 1 of runs 37638883821 (pytest failed) and 39657036477 (pytest cancelled), although each run holds an artifact, and found=true on their attempt 2, where pytest passed.
• Run objects. Main's push runs carry event=push, head_branch=main, head_repository.full_name=PolicyEngine/policyengine-scorecard and path=.github/workflows/ci.yml, the fields the guard reads.
• Publish history. The last upload that worked was 28 September. Every main run from 6 October failed or was cancelled at publish, and the latest one warned curl exit 6.
• A skipped job gets no runner. The reviewer found the old publish-db job, skipped on pull-request runs, with runner_id: null and no steps. GitHub's docs don't say this.
• queue: max at job level. On a throwaway branch, now deleted, three pushes 14 seconds apart ran one after another and none was cancelled (runs 41138146103, 41138172124 and 41138195694). Under the default queue the third would have cancelled the second.
• actionlint 1.7.12 doesn't know queue, and reports only that for publish-db.yml. In ci.yml it reports only the existing SC2129 hint. CI doesn't run actionlint.

workflow_run workflows run only from the default branch, so this PR's own CI cannot run publish-db.yml. It first runs when CI completes on the merge commit; I'll watch that run and expect the job to run and end with the usual curl exit 6 warning.

Not addressed here

• Re-runs from before this merge still name the secrets. A pull_request run takes ci.yml from the merge commit with the base, not from the branch, so new runs of open pull requests left the job out as soon as this merged. Run 41141999014 on Certify the policyengine.py 6.11.0 replay results #235 lists no publish job, although that head's own ci.yml still has it; the branch's run 30 minutes earlier listed one. GitHub's docs say a re-run keeps its original commit, so a re-run of a run from before this merge still uses the old file.
  • The lasting fix is to move the Supabase secrets into a deployment environment limited to main, or to delete them, since the host stopped resolving on 6 October.
  • Either needs the secret values or a call on the Supabase project, so it is queued for Max as d1846.
  • An earlier version of this section said open branches kept naming the secrets until they merged main. That was wrong, and the first-publish check below found it.
• A write curl abandons may still land. If the storage server takes the whole body and curl then hits its limit, the server may still commit it. That depends on the storage service, which this workflow cannot see.
• The artifact's bytes rest on CI's other jobs.
  • upload-artifact v4 can overwrite an artifact by name, so another job of the same push run could replace it before CI completes.
  • The python job also runs unpinned PyPI packages before it uploads.
  • Neither is new: the old job downloaded the same artifact while replay legs were still running. Nothing in this repository reads the published copy.

Review

A GPT-6.1 Sol review of the first version requested changes on three points, each now fixed:

• A re-run of an old "publish db" run could move latest back, because it keeps its GITHUB_SHA. The job now reads main's tip.
• An older run's completion could cancel the pending publish for main's tip. The concurrency group is gone.
• An artifact did not prove the attempt that fired the publish had passed pytest. The job now reads that job's conclusion.

A second review (Opus 5.5, with no shell) found all three gone and requested two more changes, both now made:

• The comment's bound covered one upload, not the whole window after the tip read, so the job now has its own limit.
• "Never moves off main's tip" read as if latest always equals the tip. It lags while the tip's pytest job has not passed.

A third review (GPT-6.1 Sol) requested three more changes:

• The timing argument for overlapping publishes could not be a guarantee, since a cancelled CI run has no minimum duration. The job now publishes one at a time with queue: max.
• The job limit that argument needed could end a job before its warning. It is gone.
• On a re-run of an old publish, the download may take another attempt's artifact of the same name. This stays: every such artifact comes from the pytest job's last step on that commit, as the review itself notes, and a test now pins that.

A fourth review (GPT-6.1 Sol) confirmed the queue reads the tip inside its slot and requested two more changes, both now made:

• queue: max holds up to 100 pending publishes, not every one, so the comments now say that.
• With no job limit, a stalled gh call could hold the queue for GitHub's 360-minute default. The find and download steps now have five minutes each.

A fifth review (GPT-6.1 Sol) approved at 4b297fe. I scoped it to that commit and the whole workflow, with a bar of a broken requirement, a false statement or a line that does nothing, and it found nothing that met the bar. It ran read-only, so it probed the scripts directly and did not run pytest. Live GitHub scheduling, cancellation and timeouts remain unverified beyond the queue: max probe above.

Before those, an independent Opus 5.5 review approved, with no blockers (the Codex lanes were closed). This version addresses its findings:

• latest and re-runs
• the size note (~196 MB built, ~14 MB gzipped)
• }} inside string literals in the scanner
• the narrow event list
• workflow-level concurrency
• the case-sensitive guard model
• the unpinned download

"Not addressed here" covers the open-branches finding. I left the artifact-overwrite hardening out, as the review called it optional and nothing reads the copy.

A second pass by the same reviewer on the fix commit also approved:

• It ported TemplateReader.ParseScalar to Python, and the scanner's reader agreed with it on every short string and on 80,000 random longer ones.
• It confirmed the pin and the mutations.
• It found that nothing but CI-running pushes moves main, so the latest gate cannot freeze it.

Its two remaining nits, issue_comment and the concurrency comment, are in the last commit.

Locally, merged with main at 7591e3c, the publish, CI-gate, replay-queue, determinism, engine-matrix and results-PR test files give 281 passed. ruff format --check is clean, and ci.yml names no secret.

axiom: n/a: CI only

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policyengine-scorecard	Ready	Preview	Oct 30, 2026 10:48pm UTC

[MaxGhenis]

First publishes on main, both as designed:

• Run 41141691285 followed CI run 41140109657 for a740605, which this merge cancelled while its pytest job was running. The find step posted its notice and published nothing.
• Run 41143401145 followed CI run 41141568000 for 4e9bc76. It found and downloaded the artifact, warned curl exit 6 and stayed green, as every publish has since 6 October. Neither log contains the host.

Correction to this PR's body: a pull_request run takes ci.yml from the merge commit, so new runs of open PRs stopped naming the secrets at this merge, not when each branch merges main. Run 41141999014 on #235 shows it. Only a re-run of a run from before this merge still uses the old file. The body and d1846 now say so.