A burst of pull requests here starves the organization's runners and delayed a production deploy by an hour

[devantler]

🤖 Generated by the Agentic Engineer

Evidence

On Monday 2026-10-05 between about 08:20Z and 09:30Z, six pull requests were open in this repository at once (four Dependabot updates plus two agent drafts). Each pull request in this repository starts roughly 260–275 check jobs. Together they filled the organization's shared pool of hosted runners:

• 167 workflow runs were created in this repository after 08:00Z.
• In platform, the merge-queue run that deploys to production (run 37281958629, for platform#4506) was created at 08:10:10Z and finished at 09:14:36Z — 64 minutes. Its deploy job did not get a runner until 08:42Z, and its last two short jobs waited from 08:54Z until about 09:14Z. Three more pull requests sat queued behind it the whole time.
• Pull-request CI in platform, world-at-ruin and this repository showed jobs queued for 30–50 minutes with nothing running.

Nothing failed. Every job was simply waiting for a runner.

Problem and audience

Every repository in the organization shares one runner allowance, so a burst here delays everything else, including production deploys through the platform's merge queue, which has a time limit (platform#3097). Dependabot runs with no cooldown for our own actions, so each release of this repository opens several updates here at once and each one runs the full job set. The audience is the maintainer and both agent lanes, who wait an hour for feedback that normally takes minutes.

Hypothesis

Most of the ~270 jobs are self-tests of individual actions and workflows that a given pull request does not touch. Running only the tests affected by the changed paths (as platform and the monorepo already do), or giving merge-queue deploys priority over pull-request self-tests, would keep a burst from starving the rest of the organization.

Acceptance criteria

• Measure first: for the last 20 pull requests, how many jobs ran and how many tested something the pull request changed.
• A pull request that changes one action or one workflow runs the tests for that surface and the always-on gates, not the whole catalogue; the required check still reports and cannot be skipped by a path filter.
• A change to shared test machinery or to the CI workflow itself still runs everything.
• After the change, a burst of five simultaneous dependency updates here does not delay a platform merge-queue run by more than a few minutes (measured on a real burst).

Success signal

Median job count per pull request in this repository, and the wall-clock time of platform merge-queue runs on days with a dependency burst.

[devantler]

🤖 Generated by the Agentic Engineer

First measurement (criterion 1, part one). The last 20 merged pull requests here; 17 still have their CI run (read 2026-10-05, jobs of the latest attempt).

Pull request	Jobs defined	Jobs that ran	Runner-minutes	Files changed
#466	233	157	111	5
#462	233	157	111	3
#460	233	192	156	12
#458	233	157	110	2
#454	233	192	158	10
#451	233	157	111	5
#450	233	157	113	2
#448	233	157	111	8
#444	233	157	106	10
#443	233	192	155	30
#436	233	157	105	6
#434	233	192	157	19
#432	234	158	105	27
#431	243	202	170	17
#430	235	152	108	5
#426	235	152	103	8
#423	235	194	161	13

What it shows:

• Every pull request runs one of two fixed sets: about 157 jobs (roughly 110 runner-minutes) or about 192 jobs (roughly 157 runner-minutes). The size of the change does not matter: a 2-file change (fix: propagate trusted deploy guard read failures #458, fix: use canonical World at Ruin required workflow #462) costs the same as a 27-file one (feat(go)!: retire the default-branch scan input #432).
• So some path filtering exists (the larger set appears when .scripts/ or .github/scripts/ change), but the floor is 157 jobs for any change at all.
• A burst of five dependency updates therefore asks for about 800 jobs and 9 to 13 runner-hours at once.
• Today the effect lasted well past the burst: at 10:30Z single 10-second jobs created at 08:53Z were still waiting for a runner, and the platform merge queue was five deep.

Not measured yet: how many of the 157 always-on jobs test something the pull request changed. That needs the job list mapped to the paths each job covers, and is the next step before changing the workflow.

[devantler]

🤖 Generated by the Agentic Engineer

The deployment-only repair in #486 and the affected-action selection in #487 overlap on CI scheduling and its contract guards. Land #486 first for the urgent deployment queue problem; #487 then needs a fresh-base integration that preserves that reviewed deployment-only path before it can be promoted. Neither PR is evidence that the comparative latency and burst measurements in #475 have passed.

[devantler]

🤖 Generated by the Agentic Engineer

Fresh CI measurement and queue recovery, 2026-10-05:

• All 20 sampled merged PRs have complete, terminal native job observations; observed job counts match each API total. Executed jobs range from 153 to 192, with a median of 157. Median aggregate runner execution is 113.1 minutes per PR; the sample totals 39.5 runner-hours. This measures execution separately from the current provider outage.
• Replayed the actual trusted classifier and fix(ci): select affected catalogue jobs before allocating runners #487 selector at 44785d3cb2cb6e71f8c395af7d119e683e0a2d95 against every sampled immutable PR head and complete Git history, using current main 2f561837958e7a55d2436f5330c7ab03ff494ede. The action-only changes chore(deps): bump Homebrew/actions/setup-homebrew from ac399ee818eb2bb93b838aeccd2cbaa6272e2403 to dc7099b3e807f1e2ecc61f3ecabc840eedd5586a in /actions/setup-ksail-cli #481 and chore(deps): bump actions/upload-code-coverage from 1.4.2 to 1.4.3 in /actions/upload-coverage #472 select 1 and 31 of 89 gated test definitions. Deployment changes fix(rulesets): extend platform queue response window #478 and fix: use canonical World at Ruin required workflow #462 retain the independently required floor of 41. The other 16 changes retain all 89 because they include shared, workflow or unclassified paths. The complete 20-case replay takes 3.87 seconds locally.
• The 89/41/31/1 figures count gated job definitions, not hosted matrix/reusable job executions. They are selection evidence, not post-merge savings or completion of the real five-update burst criterion. The deployment-only behavior is already delivered in fix(ci): bound deployment-only catalogue runner demand #486; fix(ci): select affected catalogue jobs before allocating runners #487 adds the action selection.

An additional live bottleneck was reproduced: five superseded PR CI attempts had already cancelled their ordinary jobs but left a runnerless always-run aggregate pending. They held their current-head workflows in concurrency pending. Each old run was bound to its native PR and obsolete SHA, and every remaining job was verified queued with no runner before cancellation. Cancelling only those obsolete attempts released current-head CI to queued:

PR	Obsolete run	Current run released
#487	37358475260	37361249423
#502	37357687206	37364815268
#503	37359645420	37366541995
#504	37359235520	37364976980
#505	37365071408	37366695555

No current-head or production run was cancelled. Two provider-cancelled current-head security jobs for #487 were retried without restarting passed checks. The selector, selected-but-skipped required-check negatives, and composed classifier/selector/reducer tests pass at the unchanged head.

Blocker: GitHub still reports delayed hosted-runner assignment in its active Actions incident. #487 remains draft until exact-head hosted CI completes. Platform #4507 was removed when its final runnerless checks were cancelled; fresh complete readiness allowed normal queue re-admission, and #4536 remains normally queued. Actual merge and post-merge hosted selection/burst measurements remain required.

Next optimization evidence should distinguish isolated action changes from reusable-workflow and leaf-test changes: the latter dominate this sample and currently retain full coverage. Broadening selection requires explicit dependency coverage and failure controls; the present measurements do not authorize dropping tests.

[devantler]

🤖 Generated by the Agentic Engineer

Exact-head local selector measurements for delivery #487 at 6fed936cbac8f67f07d98c3c29fc2105349b38a3, using the actual binary, complete current workflow/inventory and separate immutable Git diffs for every action owner:

• Full scope requires 81 gated execution jobs plus three independently admitted cleanup callers. Eight intentionally skipped interface-only calls retain original admission; all their failures still reach the global reducer.
• Cleanup-only scope requires only those three native callers rather than 84 full-scope results: a 96.4% reduction in the selected execution set.
• Seventeen of 20 action families select 1–8 gated jobs. Two select zero gated jobs because their native reusable callers remain independently admitted. The shared coverage action selects 30 jobs because its reviewed transitive consumers must remain covered.
• Unknown/shared changes retain full scope. Deployment-only classification retains 33 gated jobs plus three native cleanup results and every shared guard.

These are observed selection counts, not hosted runner allocations or latency measurements. The PR-event result map passes the unchanged reducer; all 84 unexpected selected skips and each of the eight interface-call failures are rejected. Hosted CI run 37389680868 passed at this exact head (194 successful jobs and 41 intentional skips), and PR #487 merged as 2249783 on 2026-10-05T23:58:18Z; delivery issue #483 is closed.

Post-merge verification found a main-event regression: runs 37391448920 and 37391759492 failed the required reducer because two reusable callers legitimately exclude push events. Signed hotfix #509 restores their original admission without changing the reducer, callee guards, credentials or permissions. Its corrected full scope is 79 gated jobs plus three native cleanup callers; deployment-only scope is 31 plus three. The main-event regression was demonstrated RED then GREEN, the actual failed native result map passes the unchanged reducer with the repaired selection, all 82 unexpected selected skips and all ten preserved-caller failures remain rejected, and the affected contracts plus all 14 maintenance checks pass. Hosted hotfix CI run 37393395416 passed at cced1503031cd153a7cd83ffaeef74f4c581d76e with 159 successful jobs and 76 intentional skips. Current-head review 5422828570 is GREEN. PR #509 merged through the normal protected path as signed commit 726fb7edf7ecae9dff8589cbfa52d71378007597 at 2026-10-06T01:45:29Z, and issue #508 closed. All five changed files are byte-identical between reviewed source and merged main. Main push CI run 37400780658 passed at that exact merged commit: 164 successful jobs and 71 intentional skips across 235 native job records. The unchanged required aggregate (job 112073803113) passed, and the native selector log confirms 79 gated jobs plus three required native cleanup callers. Current main still equals that signed merged commit at the final readback. The main-event incident is now verified cleared.

Historical baseline, captured at 2026-10-06T00:28:43Z: the 20 newest PRs with a completed CI run at their captured current head and branch were measured through complete native job pages. All PR/head memberships and changed-file lists were rebound; closed unmerged PRs were checked through their direct commit inventories. Pending #509 is excluded. A representative is the latest non-cancelled exact-head run when available; #474 has only a cancelled run.

PR	Representative CI run	Result	Executed jobs	CI elapsed minutes
#507	37380667215	success	158	34.65
#505	37380036929	success	158	40.82
#504	37388167072	success	158	9.38
#503	37382125287	success	193	25.57
#502	37389978876	success	158	10.25
#501	37373002083	success	158	86.90
#500	37348452704	success	157	36.03
#489	37344643214	success	157	41.90
#487	37389680868	success	194	12.72
#486	37347412226	success	158	32.48
#485	37346103329	success	157	41.47
#481	37330599016	success	153	64.72
#480	37339233393	success	153	48.87
#478	37331184651	success	157	95.43
#474	37354584562	cancelled	5	8.42
#473	37303813269	success	157	142.72
#472	37288694159	success	153	187.98
#471	37283269493	failure	153	101.40
#470	37291856149	success	153	116.72
#469	37288688292	success	153	143.75

The 19 non-cancelled representatives have median 157 executed jobs and 41.90 minutes elapsed; the 18 successful representatives have median 157 jobs and 41.68 minutes. Across 27 exact-head completed runs, eight cancellations consumed 379 executed jobs. None was a native rerun (run_attempt=1); repeated same-head runs have different run IDs. Executed jobs have a non-skipped native step with a start timestamp, independently matching runner-assigned counts. CI elapsed is run updated_at minus created_at; it does not prove runner wait or causality. Other workflows and organization-required checks are excluded.

For action-only PRs #481, #472 and #471, the reviewed hotfix inventory estimates 1, 30 and 8 selectable callers before deployment classification. These counts are distinct from historical executed jobs, matrix expansion, always-on guards and independently admitted callers. Bounded historical test-to-change coverage is recorded below; complete changed-behavior coverage remains unresolved.

Bounded historical test-to-change audit of those same 20 representative runs: all 3,043 executed native jobs uniquely bind to historical top-level callers, and each recorded merge-source commit includes its captured PR head. Distinct matrix jobs and reusable leaves remain distinct. Native started, non-skipped steps plus historical local action references prove 24 jobs entered changed first-party actions: two in #481, nine in #472, eight in #471, and five in #502. This proves component entry rather than changed-line, assertion or complete behavior coverage.

Historical classification	Executed jobs
Entered a changed action, demonstrated	24
Always-on/shared guards	104
Unrelated scheduler fixtures, demonstrated	56
Change-specific coverage UNKNOWN	2,859

The 56 negative joins are three queue-slot sleeps and their observer in 14 PRs that did not change CI machinery; historical definitions perform no checkout and invoke no first-party surface. The same fixtures remain UNKNOWN on CI-changing PRs. Two #502 lint leaves executed the changed workflow but skipped the prepare-fixes steps affected by the repair, so they remain UNKNOWN. Other unknowns include changed shared machinery, unresolved reusable source/ref joins, and opaque scripts or transitive dependencies. These are conservative proof floors, not coverage percentages or a claim that every UNKNOWN job is unnecessary. Current selector ownership estimates did not determine historical positive coverage. Complete changed-behavior coverage remains unresolved.

A separately verified hotspot in run 37389680868 is the three native cleanup dry-run tests: their cleanup action steps took 377, 416 and 215 seconds, or 16.8 combined minutes, and the minimal-parameters log reported enumerating 2,150 orphan runs. This identifies execution cost without attributing the organization's queue delays to it.

The hotfix's completed PR CI run 37393395416 took 72.2 minutes by run metadata timestamps. Its three real scheduler probes occupied runners for 24, 24 and 25 seconds; the all-slots observer took 19 seconds and the required aggregate four seconds. Complete native pages contain all 235 job records. The intervals between each completed probe and the next probe's actual start were 12:58 and 12:14; the observer started 12:59 after the third probe, and the required aggregate started 10:03 after the observer, when every other non-skipped CI job had completed. Those four gaps total 48:14 in a 49:50 chain containing 96 seconds of job runtime. These observed dependency-to-start intervals are separate from executable test cost; they do not establish organizational capacity, billing or the cause of runner assignment delay. The deployed main selector in run 37400780658 also passed and its native log confirms 79 of 79 gated catalogue jobs plus three required native cleanup callers. That full-scope main run subsequently passed in 26.3 minutes, with all three scheduler probes and their observer successful. Its different event and timing do not constitute a matched narrow-change performance comparison or a five-update burst test.

Complete historical changed-behavior coverage, measured post-merge narrow-change savings and the real five-update burst criterion remain unfinished. This observational baseline does not establish an improvement.

Service-level caveat: GitHub's October 5 Actions incident reported runner-assignment delays from 19:11 UTC and was resolved at 22:49 UTC. This is a possible confound for overlapping historical runs. At the earlier status readback, Actions was operational; neither this incident nor the bounded native scheduling snapshot establishes the cause of the hotfix run queue delay.

[devantler]

🤖 Generated by the Agentic Engineer

Refreshed completed-run evidence for the CI runtime work. The following cohorts are different event populations, so these are baselines for selecting work, not a controlled comparison across products:

Repository	Cohort	Median elapsed CI	Median occupied runner time
Catalogue (.github)	20 successful PR runs bound to their PR current heads	41.67 min	119.99 min
KSail	3 successful main-push runs	27.35 min	113.08 min
Platform	3 successful PR runs bound to their PR current heads	23.13 min	38.20 min
Monorepo	3 successful PR runs bound to their PR current heads	9.98 min	21.53 min
World at Ruin	3 successful main-push runs	1.97 min	3.83 min

All sampled job inventories were completely paginated, deduplicated and terminal. Elapsed time is run creation to the last executed job completion; runner time sums executed-job durations. Successful samples do not establish overall reliability or include cancelled-attempt spend. The historical catalogue cohort includes the October 5 GitHub incident, so its duration is not an uncontaminated before/after baseline.

Delivery #512 / PR #513 removes the serial runner handoff between trusted classification and test selection. Its exact-head hosted preflight succeeded in 22 seconds; prior completed PR #511 spanned 60 seconds across those two jobs, including an 8-second handoff. That is one unmatched observation, not a claim about the full CI median.

The next measured bottleneck is retained Go compilation: the KSail test and race-coverage jobs restore the same immutable cache key, execute for 17–21 minutes and then decline to save their warmed cache. Captured separately for implementation and cold/warm validation.

The parent remains open: the real update-burst evaluation and complete changed-behavior coverage evidence still need their own readbacks.

[devantler]

🤖 Generated by the Agentic Engineer

Two CI improvements are merged:

• ci: remove one runner wait before catalogue tests #513 removes the separate catalogue-scope job and its extra runner handoff. The combined selector still runs the trusted-base checks before inspecting the candidate, preserves the full catalogue for shared machinery changes, and feeds the required reducer. Its PR preflight occupied 22 seconds; the preceding PR's two preflights spanned 60 seconds. These are separate, unmatched observations, not a controlled estimate of the median improvement. Post-merge CI passed at eff5d4a27e6747f21ffda9afff55d9c8be600fe1 (run).
• perf(go): reuse compilation caches for both test modes #517 stores separate compilation archives for ordinary tests and race coverage in the shared Go validator and its generated read-only variant. Exact keys include OS, architecture, Go version, working directory, test mode and dependency files; both original test commands still execute. Hosted fixtures demonstrated real cache saves and exact restores: ordinary cold/warm test steps took 11.53/3.24 seconds, and race coverage took 17.73/3.91 seconds. These calls ran on different runners, so they prove cache reuse rather than a controlled consumer-runtime reduction. A local regression also changed a non-Go test input and confirmed that both warm modes still fail on invalid input.

The cache change is published in v7.2.10. Post-merge CI passed at 2fa404276b0ce5c0527683b080e39045045c4e42 (run); the manifest publication run also passed its push-and-sign step. Main has saved both Go archives, allowing future PRs in this repository to restore them. Consumer repositories create their own repository-scoped archives when they execute the updated validator.

The final main fixture exposed a separate remaining bottleneck: the Go 1.25.14 allowlist job reported a cache hit at 05:18:35Z, a failed restore at 05:20:34Z, and did not start installing the scanner until 05:28:35Z. It ultimately passed. This transfer/fallback delay is observed evidence for a further investigation, not a demonstrated improvement from either merge above.

This parent remains open. A measured five-update burst with Platform merge-queue timing, complete changed-behavior coverage accounting for the historical PR cohort, and an overall KSail runtime comparison are still outstanding. These two merges do not establish those results.