Skip to content

ci: report release performance runs that do not finish - #1536

Merged
kvlonge merged 1 commit into
mainfrom
ci/release-performance-unfinished
Nov 17, 2026
Merged

kvlonge merged 1 commit into
mainfrom
ci/release-performance-unfinished

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Why

A main Release Performance run hung in the comparison step and ran into the 90 minute job timeout. GitHub then cancelled the job: no log, no evidence upload, no issue. A timed-out comparison passed silently.

Changes

  • The comparison step has its own 65 minute limit (normal runs take 10-21 minutes, plus up to 30 while a new baseline reaches PyPI). The job limit rises to 130 minutes so the step limit fires first; evidence still uploads and the job fails.
  • Setup commands (venv creation, installs, release source extraction, version and library probes, the baseline generators) stop after 40 minutes with an error naming their phase. They normally take seconds.
  • Each phase is appended to the job summary with a UTC timestamp as it starts, so a run whose log is lost still shows where it stopped.
  • A push run whose comparison did not finish opens or updates one release-performance-unfinished issue. A failed compare job (step error, step timeout, lost runner) is reported. A cancelled one is reported only when its annotations carry GitHub's job-timeout message. Runs cancelled by a newer run or by hand carry other annotations and are not reported. The classifier uses only the standard library, so the report job skips the dependency install.

DuckShop: no case applies; this changes CI tooling only, not compiler, Rules or lifecycle behaviour.

Verification

  • Unit tests classify real annotations from this workflow's timed-out, superseded and manually cancelled runs, plus a step timeout, a lost runner, a regression and success; they run with -S to prove no third-party imports.
  • Hanging setup tests: a sleeping baseline generator and a sleeping sqb --version fail with the phase named.
  • The end-to-end comparison test checks the progress lines in the job summary.
  • actionlint and make check-ci pass; make test passes apart from the Postgres publication tests, which need Docker.

A main comparison that hung ran into the job timeout, lost its log and
evidence upload, and reported nothing. The comparison step now has its
own limit below the job's, setup commands stop with an error naming their
phase, each phase is listed in the job summary as it starts, and a main
run that errors, loses its runner or times out opens or updates one
release-performance-unfinished issue. Runs cancelled by concurrency or by
hand are not reported.
@kvlonge
kvlonge enabled auto-merge November 17, 2026 05:15
@kvlonge
kvlonge added this pull request to the merge queue Nov 17, 2026
@kvlonge
kvlonge removed this pull request from the merge queue due to a manual request Nov 17, 2026
@kvlonge
kvlonge added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit d3dda67 Nov 17, 2026
34 checks passed
@kvlonge
kvlonge deleted the ci/release-performance-unfinished branch November 17, 2026 07:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant