Skip to content

A Scorecard publish check: an action that sees a failed publish and runs Scorecard once more - #199

Merged
askalf merged 4 commits into
mainfrom
claude/nifty-bell-1lpk8e
Nov 10, 2026
Merged

askalf merged 4 commits into
mainfrom
claude/nifty-bell-1lpk8e

Conversation

@askalf

@askalf askalf commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

What changes, and why

scorecard-action reports a failed publish to scorecard.dev as a warning, and the run stays green. On 2026-11-09 the scheduled Scorecard run of askalf/pitlane had its publish answered 500 three times. scorecard.dev then served "not found" for the repository for about fifteen hours, the README badge read "invalid repo path", and no run was red. One clean publish cleared it.

This adds the check for that, as a fifth shared action:

  • scripts/scorecard-publish/check.mjs, run by actions/scorecard-publish-check (a JavaScript action on the runner's Node, like actions/fleet-status) on workflow_run of a repository's Scorecard workflow.
  • A run is checked when it succeeded on the default branch of a public repository. Any other run is left alone and the step passes.
  • For a checked run it reads https://api.scorecard.dev/projects/github.com/<owner>/<name>, the address the badge reads, up to four times, 15 s apart.
  • Published means the result served carries a time of day and that time is no earlier than a minute before the run's start; the minute allows for clock skew. The results this action publishes are dated to the second: the API served 2026-11-09T10:25:29Z and 2026-11-10T01:38:45Z for askalf/pitlane's publishes of those days, and the tests use those values. A date with no time of day is OpenSSF's own weekly scan, which no run of ours wrote, so it is never taken for a run's publish, whatever its day.
  • An unpublished schedule, push or branch_protection_rule run: the Scorecard workflow is dispatched on the default branch once, with a warning.
  • An unpublished workflow_dispatch run: the step fails. A dispatch is never answered with another, so one failed publish costs at most one extra run, and a publish that fails twice is a red run.
  • It cannot be a step of the Scorecard job: scorecard.dev accepts results only from a job whose steps it recognises.
  • The run's facts reach the script as inputs of the action, each defaulting to the workflow_run event's own field (github.event.workflow_run.id and so on), so a caller passes none and the script reads no file. A workflow_run that arrives with no run id fails the step. The workflow file must be a plain file name under .github/workflows; the default branch is used as GitHub names it and is sent only as a JSON string in the dispatch body.

pin.mjs lists the action in ACTIONS and redline-pin-bump.yml watches its two directories, so its pins move with the other actions'. The README has the caller workflow and a section on the check; scripts/scorecard-publish/README.md has the rules.

Two things this relies on and does not prove:

  • that the runner fills an action input's default from github.event.*, as it does from github.token. If it does not, the first run fails on the missing run id, which is why that is an error and not a skip; a caller can then pass the inputs itself.
  • that the run dispatched with the job token fires workflow_run for the check in turn. If it does not, the retry still happens and only the second check, the one that turns a twice-failed publish red, is missing.

How it is tested: node test-all.mjs passes (all 12; the new one is scripts/scorecard-publish/check.test.mjs, also added to the test job). It covers which runs are checked, a default branch with a + or a letter outside ASCII, the dating rule with the dates the API served and with date-only values, and main against a stubbed scorecard.dev and GitHub: a published run, a result that arrives on a later read, the 2026-11-09 case (404 on every read, one dispatch of scorecard.yml on main with the job token), an older result, only the weekly scan's result, no answer, an unpublished dispatch failing with nothing sent, a dry run, a refused dispatch, a missing run id, a default branch with a + and quotes sent as a JSON string, and a workflow path that is not a plain file name. The action's entry is run for real for the paths that make no request, and a test holds each input's default to its event field.

Not tested: the action on a runner against the real scorecard.dev. No repository calls it yet.

Callers: no caller runs anything new until it adds the workflow. pin.mjs and the pin bump's watch list change, so the next pin bump moves every caller's existing pins to this commit. The callers' own scorecard-publish.yml files come after this merges, pinned at the merged commit.

…uns Scorecard once more

scorecard-action reports a failed publish to scorecard.dev as a warning on
a green run. On 2026-11-09 a publish from askalf/pitlane was answered 500
three times, scorecard.dev served "not found" for the repository for
fifteen hours, and the README badge read "invalid repo path" with no run
red.

actions/scorecard-publish-check runs on workflow_run of a repository's
Scorecard workflow. For a run that succeeded on the default branch of a
public repository it reads what scorecard.dev serves, four times at most.
A result dated no earlier than the run's start is published. Otherwise it
dispatches the Scorecard workflow once; when the unpublished run was itself
a dispatch it fails the job, so there is no loop and a publish that fails
twice is a red run.

It is the fifth action the pin bump moves. No repository calls it yet: a
caller pins a commit of main that has it.
@askalf
askalf marked this pull request as ready for review November 10, 2026 02:35
…t carry a time

CodeQL flagged the event file's data going into the dispatch request. The
script no longer reads the event file: each fact is an input of the action
that defaults to the workflow_run event's own field, and a workflow_run
that arrives with no run id fails the step.

Redline's review: a date with no time of day cannot say a result is this
run's. The results this action publishes are dated to the second (the API
served 2026-11-09T10:25:29Z and 2026-11-10T01:38:45Z for askalf/pitlane),
and a date-only value is OpenSSF's weekly scan, so it is now never taken
for a run's publish. Tests use the dates the API served.
The plain-name pattern on the default branch read a valid name such as
release+stable as none, and a run on it was then skipped with no read of
scorecard.dev. The branch is kept as given: it goes into the dispatch as
a JSON string and nowhere else. The README states the one-minute skew
allowance the script applies.
The test built a pattern from each input's event field, escaping only its
dots, which CodeQL reads as incomplete escaping. Each input's block is now
found by splitting the action file and compared as a string.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: approve. The check bounds its polling, dispatches one retry for an unpublished run, and fails an unpublished dispatch without starting another. The tests cover freshness, skipped runs, request failures, dispatch handling and input wiring; the new action is also included in pin updates and the test job.

@askalf
askalf merged commit 50ee9df into main Nov 10, 2026
13 checks passed
@askalf
askalf deleted the claude/nifty-bell-1lpk8e branch November 10, 2026 03:00
askalf pushed a commit to askalf/pitlane that referenced this pull request Nov 10, 2026
Moves this repository's askalf/ci pins (its reusable workflow calls,
their redline-ref inputs and any action pins) to
askalf/ci@50ee9df, the main commit that
last changed the review, the fix or an action (main 2026-11-10,
askalf/ci#199). The scripts are fetched at that same commit. The pins
are the only change.

---------

Co-authored-by: askalf-pin-bump[bot] <352119504+askalf-pin-bump[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants