diff --git a/.abcd/work/issues/open/iss-2610110941283143-a-push-to-an-open-pull-request-started.md b/.abcd/work/issues/open/iss-2610110941283143-a-push-to-an-open-pull-request-started.md index a1b112277..794e5a8e9 100644 --- a/.abcd/work/issues/open/iss-2610110941283143-a-push-to-an-open-pull-request-started.md +++ b/.abcd/work/issues/open/iss-2610110941283143-a-push-to-an-open-pull-request-started.md @@ -13,3 +13,5 @@ remedy: "have the loop and the landing steps check, after a push to an open pull --- A push to an open pull request started only some of its workflows: after b36b551ac was pushed to #904 (2026-10-10, about 22:50Z) only external-review ran, ci and attribution never triggered, and the pull request sat BLOCKED with eight of its nine required checks missing until it was closed and reopened. Nothing noticed: the checks were not failing, only absent. + +**Likely cause (2026-10-11, abcd-0b).** GitHub runs no `pull_request` workflow while it reports a pull request as mergeable CONFLICTING (DIRTY), because it cannot build the test merge commit; `pull_request_target` workflows such as external-review run on the base branch and still fire. #919 and #920 showed exactly this on 2026-10-11: a body edit and a close and reopen started only external-review. A local merge of main into #919 was clean, so the conflict is probably the forge ignoring the `.abcd/work/DECISIONS.md` union driver, as with #902 (iss-2610110908443341; root cause iss-2609100507439414). Whether #904 was conflicting at 22:50Z on 2026-10-10 is not yet checked. diff --git a/.abcd/work/issues/open/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md b/.abcd/work/issues/open/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md deleted file mode 100644 index bfaddaead..000000000 --- a/.abcd/work/issues/open/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -schema_version: 1 -id: "iss-2610110941289527" -slug: "the-merge-queue-s-check-timeout-was" -severity: "major" -category: "bug" -source: "agent-finding" -found_during: "merge-pipeline evidence from abcd-0b, 2026-10-11" -origin: researcher-authored -production_mode: hand-written -found_at: ".abcd/work/rulesets/main-protection.json" -remedy: "once the macOS leg runs only in the queue, measure a queue entry's macOS wait plus run time and set check_response_timeout_minutes above its p90, with the measurement on this record; meanwhile keep max_entries_to_build within the macOS slots" ---- - -The merge queue's check timeout was shorter than one macOS queue entry takes: over 33 merge-group ci runs (2026-10-10 18:00Z to 2026-10-11 09:40Z) a macOS job waited a median 40 minutes for a runner (p90 68, max 81) and then ran a median 40 (p90 44), so an entry needed about 80 minutes against a 45-minute timeout. Every entry that had to wait was removed: #903 and #906 twice each, #908, #910, #913 and #915 once (#913 at 04:46:54Z with every check green but macOS, run 38110208736). Nothing merged from 03:36Z to 09:38Z with 15 entries queued. The person has since set 2 entries and 90 minutes (ruleset 19969675, mirror PR #926); 90 is still tight against 80. diff --git a/.abcd/work/issues/open/iss-2610110613372021-stale-merge-group-runs-hold-runners.md b/.abcd/work/issues/resolved/iss-2610110613372021-stale-merge-group-runs-hold-runners.md similarity index 80% rename from .abcd/work/issues/open/iss-2610110613372021-stale-merge-group-runs-hold-runners.md rename to .abcd/work/issues/resolved/iss-2610110613372021-stale-merge-group-runs-hold-runners.md index 92b3f9303..23cca8388 100644 --- a/.abcd/work/issues/open/iss-2610110613372021-stale-merge-group-runs-hold-runners.md +++ b/.abcd/work/issues/resolved/iss-2610110613372021-stale-merge-group-runs-hold-runners.md @@ -10,6 +10,10 @@ origin: researcher-authored production_mode: hand-written found_at: ".github/workflows/ci.yml" remedy: "ci.yml keys merge_group runs on github.run_id with cancel-in-progress off, and each group's ref (gh-readonly-queue/main/pr-N-) is unique, so neither the run id nor github.ref can cancel a superseded run. Key merge_group runs on the pull request number parsed from the queue ref (pr-N), with cancel-in-progress on for merge_group, so a rebuilt group cancels the older run for the same PR; do the same for the attribution and external-review workflows; check the zizmor audit accepts it." +resolution: "queue-sweep.yml (#933) cancels, then force-cancels, the merge-group runs of ci, attribution and external-review whose gh-readonly-queue ref is gone, and never a live entry's; its first four live runs on 2026-10-11 left every live run alone" +impact: fix +resolved_by: + commit: "0e47fb78337f" --- When the merge queue rebuilds its groups (an entry dropped, re-queued or re-armed), the CI runs of the superseded groups keep running and holding runners, though their gh-readonly-queue refs are deleted and nothing reads their results. On 2026-10-11 nothing merged from 03:36 to past 07:00 BST. Seven stale merge-group ci runs (PRs 903, 906, 908, 911 and 915 at older shas) held the runners while every live group's run sat queued, until a session cancelled them by hand. diff --git a/.abcd/work/issues/resolved/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md b/.abcd/work/issues/resolved/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md new file mode 100644 index 000000000..2f19540c6 --- /dev/null +++ b/.abcd/work/issues/resolved/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md @@ -0,0 +1,25 @@ +--- +schema_version: 1 +id: "iss-2610110941289527" +slug: "the-merge-queue-s-check-timeout-was" +severity: "major" +category: "bug" +source: "agent-finding" +found_during: "merge-pipeline evidence from abcd-0b, 2026-10-11" +origin: researcher-authored +production_mode: hand-written +found_at: ".abcd/work/rulesets/main-protection.json" +remedy: "once the macOS leg runs only in the queue, measure a queue entry's macOS wait plus run time and set check_response_timeout_minutes above its p90, with the measurement on this record; meanwhile keep max_entries_to_build within the macOS slots" +resolution: "the macOS suite runs once, in the queue, in one pass (#933), and the queue builds 4 entries with a 90-minute check timeout (ruleset 19969675); over 24 queue runs the macOS wait plus run p90 is 35.6 minutes and no entry dropped on a timeout" +impact: fix +resolved_by: + commit: "5a915fd2c" +--- + +The merge queue's check timeout was shorter than one macOS queue entry takes: over 33 merge-group ci runs (2026-10-10 18:00Z to 2026-10-11 09:40Z) a macOS job waited a median 40 minutes for a runner (p90 68, max 81) and then ran a median 40 (p90 44), so an entry needed about 80 minutes against a 45-minute timeout. Every entry that had to wait was removed: #903 and #906 twice each, #908, #910, #913 and #915 once (#913 at 04:46:54Z with every check green but macOS, run 38110208736). Nothing merged from 03:36Z to 09:38Z with 15 entries queued. The person has since set 2 entries and 90 minutes (ruleset 19969675, mirror PR #926); 90 is still tight against 80. + +**A second way an entry drops (2026-10-11).** The macOS job's own `timeout-minutes: 45` removes entries too: merge-group run 38131330623 (pr-914) had `check (macos-latest)` start at 10:18:34Z and cancelled at 11:03:56Z, 45m22s later, inside "Test (race, internal)", and the queue removed #914 at 11:04:27Z, counting the cancelled check as a failure. + +**After #933 (one sample, 2026-10-11).** #933 moved the macOS suite to the merge queue only and made it one race-enabled pass. Its own queue run, 38134740530, ran `check (macos-latest)` in 24.7 minutes (the pass 23.6), against a median of 40 before, and waited 0.1 minutes for a runner, against a median of 40 (p90 68) before. With the person's ruling of 4 entries built at once and a 90-minute check timeout, one entry's wait plus run is now well inside the timeout. That is one run: the remedy's p90 over a day of queue runs is still owed before this record is resolved. + +**The remedy's measurement (2026-10-11, 24 merge-group runs since #933 merged, 11:40Z to 16:30Z).** `check (macos-latest)` waited for a runner a median of 0.1 minutes (p90 0.8, max 26.9) and ran a median of 30.1 minutes (p90 33.0, max 35.5); wait plus run had a median of 30.6 minutes and a p90 of 35.6 (max 57.9). The queue's check timeout is 90 minutes and the queue builds 4 entries at once (ruleset 19969675, mirrored in `.abcd/work/rulesets/main-protection.json` on main), so the timeout sits at two and a half times the p90 and the macOS job's own 45-minute ceiling is 12 minutes above its slowest run. No entry dropped on a timeout in the window: of the 24 runs 22 passed, and the 2 cancelled were superseded runs (pr-903, pr-923) that queue-sweep cancelled at 13:08Z because their queue refs were gone, leaving the live pr-922 runs alone.