From 23decddd98b6c3b7e3d2584283796e27ba887411 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sun, 11 Oct 2026 16:55:40 +0100 Subject: [PATCH 1/5] chore: defer the two CI majors past v0.13.4 and add their evidence The queue timeout (iss-2610110941289527) is mitigated by the person's queue ruling and #933 but owes a day of measurement; it gains the second way an entry drops and #933's first numbers. The loop's silence on a conflicting armed pull request (iss-2610110908443341) is next cycle's loop code. The missing-trigger record gains its likely cause. Refs: iss-2610110941289527 Refs: iss-2610110908443341 Refs: iss-2610110941283143 Assisted-by: Claude:claude-opus-5-5 --- ...908443341-the-implement-loop-arms-a-pull-request.md | 6 ++++++ ...941283143-a-push-to-an-open-pull-request-started.md | 2 ++ ...110941289527-the-merge-queue-s-check-timeout-was.md | 10 ++++++++++ 3 files changed, 18 insertions(+) diff --git a/.abcd/work/issues/open/iss-2610110908443341-the-implement-loop-arms-a-pull-request.md b/.abcd/work/issues/open/iss-2610110908443341-the-implement-loop-arms-a-pull-request.md index a3092ed54..e78d4f027 100644 --- a/.abcd/work/issues/open/iss-2610110908443341-the-implement-loop-arms-a-pull-request.md +++ b/.abcd/work/issues/open/iss-2610110908443341-the-implement-loop-arms-a-pull-request.md @@ -10,6 +10,12 @@ origin: researcher-authored production_mode: hand-written found_at: "internal/core/implement/loop/land.go" remedy: "at the arm and merged-check moves, read the pull request's mergeability from the forge and refuse naming a conflicting pull request and its paths, with the hand-merge or rebuild as the remedy, instead of repeating 'not merged yet'; the cause it meets most is iss-2609100507439414" +deferred_after: v0.13.4 +deferral_reason: "the fix is implement-loop code (read the forge's mergeability at the arm and merged-check moves and refuse naming a conflicting pull request), next cycle's work; meanwhile the hand merge the product thinker ruled for on 2026-10-11 is the remedy, and the root cause, appends to DECISIONS.md, is planned away by itd-2609151138388536" --- The implement loop arms a pull request that GitHub reports as conflicting and then waits on it in silence. #902, lane-1 of run-2610101724174688, had appended to .abcd/work/DECISIONS.md while later merges to main appended too. Local git merges that cleanly through the union driver; the forge ignores the driver, so #902 was armed but never entered the queue. For 13 hours every implement step said only 'pull request #902 is not merged yet', and the loop's rule that nothing is pushed after arming left it no way out. + +## Deferral 2026-10-11 + +Deferred past v0.13.4: the fix is implement-loop code (read the forge's mergeability at the arm and merged-check moves and refuse naming a conflicting pull request), next cycle's work; meanwhile the hand merge the product thinker ruled for on 2026-10-11 is the remedy, and the root cause, appends to DECISIONS.md, is planned away by itd-2609151138388536 diff --git a/.abcd/work/issues/open/iss-2610110941283143-a-push-to-an-open-pull-request-started.md b/.abcd/work/issues/open/iss-2610110941283143-a-push-to-an-open-pull-request-started.md index a1b112277..794e5a8e9 100644 --- a/.abcd/work/issues/open/iss-2610110941283143-a-push-to-an-open-pull-request-started.md +++ b/.abcd/work/issues/open/iss-2610110941283143-a-push-to-an-open-pull-request-started.md @@ -13,3 +13,5 @@ remedy: "have the loop and the landing steps check, after a push to an open pull --- A push to an open pull request started only some of its workflows: after b36b551ac was pushed to #904 (2026-10-10, about 22:50Z) only external-review ran, ci and attribution never triggered, and the pull request sat BLOCKED with eight of its nine required checks missing until it was closed and reopened. Nothing noticed: the checks were not failing, only absent. + +**Likely cause (2026-10-11, abcd-0b).** GitHub runs no `pull_request` workflow while it reports a pull request as mergeable CONFLICTING (DIRTY), because it cannot build the test merge commit; `pull_request_target` workflows such as external-review run on the base branch and still fire. #919 and #920 showed exactly this on 2026-10-11: a body edit and a close and reopen started only external-review. A local merge of main into #919 was clean, so the conflict is probably the forge ignoring the `.abcd/work/DECISIONS.md` union driver, as with #902 (iss-2610110908443341; root cause iss-2609100507439414). Whether #904 was conflicting at 22:50Z on 2026-10-10 is not yet checked. diff --git a/.abcd/work/issues/open/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md b/.abcd/work/issues/open/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md index bfaddaead..c75c73786 100644 --- a/.abcd/work/issues/open/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md +++ b/.abcd/work/issues/open/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md @@ -10,6 +10,16 @@ origin: researcher-authored production_mode: hand-written found_at: ".abcd/work/rulesets/main-protection.json" remedy: "once the macOS leg runs only in the queue, measure a queue entry's macOS wait plus run time and set check_response_timeout_minutes above its p90, with the measurement on this record; meanwhile keep max_entries_to_build within the macOS slots" +deferred_after: v0.13.4 +deferral_reason: "mitigated, not yet proven: the person's ruling (4 entries built, a 90-minute check timeout) and #933 (macOS suite once, in the queue, one pass) brought one queue entry to 24.7 minutes with a 0.1-minute runner wait; the remedy's p90 over a day of queue runs is owed before it is resolved" --- The merge queue's check timeout was shorter than one macOS queue entry takes: over 33 merge-group ci runs (2026-10-10 18:00Z to 2026-10-11 09:40Z) a macOS job waited a median 40 minutes for a runner (p90 68, max 81) and then ran a median 40 (p90 44), so an entry needed about 80 minutes against a 45-minute timeout. Every entry that had to wait was removed: #903 and #906 twice each, #908, #910, #913 and #915 once (#913 at 04:46:54Z with every check green but macOS, run 38110208736). Nothing merged from 03:36Z to 09:38Z with 15 entries queued. The person has since set 2 entries and 90 minutes (ruleset 19969675, mirror PR #926); 90 is still tight against 80. + +**A second way an entry drops (2026-10-11).** The macOS job's own `timeout-minutes: 45` removes entries too: merge-group run 38131330623 (pr-914) had `check (macos-latest)` start at 10:18:34Z and cancelled at 11:03:56Z, 45m22s later, inside "Test (race, internal)", and the queue removed #914 at 11:04:27Z, counting the cancelled check as a failure. + +**After #933 (one sample, 2026-10-11).** #933 moved the macOS suite to the merge queue only and made it one race-enabled pass. Its own queue run, 38134740530, ran `check (macos-latest)` in 24.7 minutes (the pass 23.6), against a median of 40 before, and waited 0.1 minutes for a runner, against a median of 40 (p90 68) before. With the person's ruling of 4 entries built at once and a 90-minute check timeout, one entry's wait plus run is now well inside the timeout. That is one run: the remedy's p90 over a day of queue runs is still owed before this record is resolved. + +## Deferral 2026-10-11 + +Deferred past v0.13.4: mitigated, not yet proven: the person's ruling (4 entries built, a 90-minute check timeout) and #933 (macOS suite once, in the queue, one pass) brought one queue entry to 24.7 minutes with a 0.1-minute runner wait; the remedy's p90 over a day of queue runs is owed before it is resolved From 251e6c86a4201739670baad570f6573652118a1c Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sun, 11 Oct 2026 17:19:07 +0100 Subject: [PATCH 2/5] chore: resolve the stale merge-group runs issue fixed by the queue sweep #933 added queue-sweep.yml (0e47fb783), which cancels the merge-group runs of ci, attribution and external-review whose queue ref is gone; its first live runs left every live entry's run alone. The record reached main with #925 after #933, so it is resolved here. Resolves: iss-2610110613372021 Assisted-by: Claude:claude-opus-5-5 --- ...ss-2610110613372021-stale-merge-group-runs-hold-runners.md | 4 ++++ 1 file changed, 4 insertions(+) rename .abcd/work/issues/{open => resolved}/iss-2610110613372021-stale-merge-group-runs-hold-runners.md (80%) diff --git a/.abcd/work/issues/open/iss-2610110613372021-stale-merge-group-runs-hold-runners.md b/.abcd/work/issues/resolved/iss-2610110613372021-stale-merge-group-runs-hold-runners.md similarity index 80% rename from .abcd/work/issues/open/iss-2610110613372021-stale-merge-group-runs-hold-runners.md rename to .abcd/work/issues/resolved/iss-2610110613372021-stale-merge-group-runs-hold-runners.md index 92b3f9303..23cca8388 100644 --- a/.abcd/work/issues/open/iss-2610110613372021-stale-merge-group-runs-hold-runners.md +++ b/.abcd/work/issues/resolved/iss-2610110613372021-stale-merge-group-runs-hold-runners.md @@ -10,6 +10,10 @@ origin: researcher-authored production_mode: hand-written found_at: ".github/workflows/ci.yml" remedy: "ci.yml keys merge_group runs on github.run_id with cancel-in-progress off, and each group's ref (gh-readonly-queue/main/pr-N-) is unique, so neither the run id nor github.ref can cancel a superseded run. Key merge_group runs on the pull request number parsed from the queue ref (pr-N), with cancel-in-progress on for merge_group, so a rebuilt group cancels the older run for the same PR; do the same for the attribution and external-review workflows; check the zizmor audit accepts it." +resolution: "queue-sweep.yml (#933) cancels, then force-cancels, the merge-group runs of ci, attribution and external-review whose gh-readonly-queue ref is gone, and never a live entry's; its first four live runs on 2026-10-11 left every live run alone" +impact: fix +resolved_by: + commit: "0e47fb78337f" --- When the merge queue rebuilds its groups (an entry dropped, re-queued or re-armed), the CI runs of the superseded groups keep running and holding runners, though their gh-readonly-queue refs are deleted and nothing reads their results. On 2026-10-11 nothing merged from 03:36 to past 07:00 BST. Seven stale merge-group ci runs (PRs 903, 906, 908, 911 and 915 at older shas) held the runners while every live group's run sat queued, until a session cancelled them by hand. From c7dadb1a0dc371cf631671b809ff93c9eeb2cde7 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sun, 11 Oct 2026 17:32:11 +0100 Subject: [PATCH 3/5] chore: withdraw the two CI deferrals; both are fixed this cycle The product thinker ruled against deferring: the queue timeout and the loop's silence on a conflicting armed pull request are fixed before the cut instead. Their evidence stays. Refs: iss-2610110941289527 Refs: iss-2610110908443341 Assisted-by: Claude:claude-opus-5-5 --- ...s-2610110908443341-the-implement-loop-arms-a-pull-request.md | 2 -- .../iss-2610110941289527-the-merge-queue-s-check-timeout-was.md | 2 -- 2 files changed, 4 deletions(-) diff --git a/.abcd/work/issues/open/iss-2610110908443341-the-implement-loop-arms-a-pull-request.md b/.abcd/work/issues/open/iss-2610110908443341-the-implement-loop-arms-a-pull-request.md index e78d4f027..5f4d72dd5 100644 --- a/.abcd/work/issues/open/iss-2610110908443341-the-implement-loop-arms-a-pull-request.md +++ b/.abcd/work/issues/open/iss-2610110908443341-the-implement-loop-arms-a-pull-request.md @@ -10,8 +10,6 @@ origin: researcher-authored production_mode: hand-written found_at: "internal/core/implement/loop/land.go" remedy: "at the arm and merged-check moves, read the pull request's mergeability from the forge and refuse naming a conflicting pull request and its paths, with the hand-merge or rebuild as the remedy, instead of repeating 'not merged yet'; the cause it meets most is iss-2609100507439414" -deferred_after: v0.13.4 -deferral_reason: "the fix is implement-loop code (read the forge's mergeability at the arm and merged-check moves and refuse naming a conflicting pull request), next cycle's work; meanwhile the hand merge the product thinker ruled for on 2026-10-11 is the remedy, and the root cause, appends to DECISIONS.md, is planned away by itd-2609151138388536" --- The implement loop arms a pull request that GitHub reports as conflicting and then waits on it in silence. #902, lane-1 of run-2610101724174688, had appended to .abcd/work/DECISIONS.md while later merges to main appended too. Local git merges that cleanly through the union driver; the forge ignores the driver, so #902 was armed but never entered the queue. For 13 hours every implement step said only 'pull request #902 is not merged yet', and the loop's rule that nothing is pushed after arming left it no way out. diff --git a/.abcd/work/issues/open/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md b/.abcd/work/issues/open/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md index c75c73786..7c4f01357 100644 --- a/.abcd/work/issues/open/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md +++ b/.abcd/work/issues/open/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md @@ -10,8 +10,6 @@ origin: researcher-authored production_mode: hand-written found_at: ".abcd/work/rulesets/main-protection.json" remedy: "once the macOS leg runs only in the queue, measure a queue entry's macOS wait plus run time and set check_response_timeout_minutes above its p90, with the measurement on this record; meanwhile keep max_entries_to_build within the macOS slots" -deferred_after: v0.13.4 -deferral_reason: "mitigated, not yet proven: the person's ruling (4 entries built, a 90-minute check timeout) and #933 (macOS suite once, in the queue, one pass) brought one queue entry to 24.7 minutes with a 0.1-minute runner wait; the remedy's p90 over a day of queue runs is owed before it is resolved" --- The merge queue's check timeout was shorter than one macOS queue entry takes: over 33 merge-group ci runs (2026-10-10 18:00Z to 2026-10-11 09:40Z) a macOS job waited a median 40 minutes for a runner (p90 68, max 81) and then ran a median 40 (p90 44), so an entry needed about 80 minutes against a 45-minute timeout. Every entry that had to wait was removed: #903 and #906 twice each, #908, #910, #913 and #915 once (#913 at 04:46:54Z with every check green but macOS, run 38110208736). Nothing merged from 03:36Z to 09:38Z with 15 entries queued. The person has since set 2 entries and 90 minutes (ruleset 19969675, mirror PR #926); 90 is still tight against 80. From 66705c982767d834516b2905bce51b8c29597298 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sun, 11 Oct 2026 17:33:44 +0100 Subject: [PATCH 4/5] chore: resolve the queue timeout issue on the measurement it asked for Over the 24 merge-group runs since #933 the macOS wait plus run has a p90 of 35.6 minutes against the 90-minute check timeout, and no entry dropped on a timeout; the two cancelled runs were superseded ones the queue sweep cancelled. Resolves: iss-2610110941289527 Assisted-by: Claude:claude-opus-5-5 --- ...-2610110941289527-the-merge-queue-s-check-timeout-was.md | 6 ++++++ 1 file changed, 6 insertions(+) rename .abcd/work/issues/{open => resolved}/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md (68%) diff --git a/.abcd/work/issues/open/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md b/.abcd/work/issues/resolved/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md similarity index 68% rename from .abcd/work/issues/open/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md rename to .abcd/work/issues/resolved/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md index 7c4f01357..17c8ddd97 100644 --- a/.abcd/work/issues/open/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md +++ b/.abcd/work/issues/resolved/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md @@ -10,6 +10,10 @@ origin: researcher-authored production_mode: hand-written found_at: ".abcd/work/rulesets/main-protection.json" remedy: "once the macOS leg runs only in the queue, measure a queue entry's macOS wait plus run time and set check_response_timeout_minutes above its p90, with the measurement on this record; meanwhile keep max_entries_to_build within the macOS slots" +resolution: "the macOS suite runs once, in the queue, in one pass (#933), and the queue builds 4 entries with a 90-minute check timeout (ruleset 19969675); over 24 queue runs the macOS wait plus run p90 is 35.6 minutes and no entry dropped on a timeout" +impact: fix +resolved_by: + commit: "5a915fd2c" --- The merge queue's check timeout was shorter than one macOS queue entry takes: over 33 merge-group ci runs (2026-10-10 18:00Z to 2026-10-11 09:40Z) a macOS job waited a median 40 minutes for a runner (p90 68, max 81) and then ran a median 40 (p90 44), so an entry needed about 80 minutes against a 45-minute timeout. Every entry that had to wait was removed: #903 and #906 twice each, #908, #910, #913 and #915 once (#913 at 04:46:54Z with every check green but macOS, run 38110208736). Nothing merged from 03:36Z to 09:38Z with 15 entries queued. The person has since set 2 entries and 90 minutes (ruleset 19969675, mirror PR #926); 90 is still tight against 80. @@ -21,3 +25,5 @@ The merge queue's check timeout was shorter than one macOS queue entry takes: ov ## Deferral 2026-10-11 Deferred past v0.13.4: mitigated, not yet proven: the person's ruling (4 entries built, a 90-minute check timeout) and #933 (macOS suite once, in the queue, one pass) brought one queue entry to 24.7 minutes with a 0.1-minute runner wait; the remedy's p90 over a day of queue runs is owed before it is resolved + +**The remedy's measurement (2026-10-11, 24 merge-group runs since #933 merged, 11:40Z to 16:30Z).** `check (macos-latest)` waited for a runner a median of 0.1 minutes (p90 0.8, max 26.9) and ran a median of 30.1 minutes (p90 33.0, max 35.5); wait plus run had a median of 30.6 minutes and a p90 of 35.6 (max 57.9). The queue's check timeout is 90 minutes and the queue builds 4 entries at once (ruleset 19969675, mirrored in `.abcd/work/rulesets/main-protection.json` on main), so the timeout sits at two and a half times the p90 and the macOS job's own 45-minute ceiling is 12 minutes above its slowest run. No entry dropped on a timeout in the window: of the 24 runs 22 passed, and the 2 cancelled were superseded runs (pr-903, pr-923) that queue-sweep cancelled at 13:08Z because their queue refs were gone, leaving the live pr-922 runs alone. From c1101fb343c78e6b17a9d60db043d2ec47d666e5 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sun, 11 Oct 2026 17:34:32 +0100 Subject: [PATCH 5/5] chore: remove the deferral notes the withdrawn deferrals left in the records `capture defer` also writes a dated Deferral section into the record's body; withdrawing the deferral fields left both sections behind, saying the records were deferred when they are being fixed. Refs: iss-2610110908443341 Refs: iss-2610110941289527 Assisted-by: Claude:claude-opus-5-5 --- ...2610110908443341-the-implement-loop-arms-a-pull-request.md | 4 ---- ...ss-2610110941289527-the-merge-queue-s-check-timeout-was.md | 4 ---- 2 files changed, 8 deletions(-) diff --git a/.abcd/work/issues/open/iss-2610110908443341-the-implement-loop-arms-a-pull-request.md b/.abcd/work/issues/open/iss-2610110908443341-the-implement-loop-arms-a-pull-request.md index 5f4d72dd5..a3092ed54 100644 --- a/.abcd/work/issues/open/iss-2610110908443341-the-implement-loop-arms-a-pull-request.md +++ b/.abcd/work/issues/open/iss-2610110908443341-the-implement-loop-arms-a-pull-request.md @@ -13,7 +13,3 @@ remedy: "at the arm and merged-check moves, read the pull request's mergeability --- The implement loop arms a pull request that GitHub reports as conflicting and then waits on it in silence. #902, lane-1 of run-2610101724174688, had appended to .abcd/work/DECISIONS.md while later merges to main appended too. Local git merges that cleanly through the union driver; the forge ignores the driver, so #902 was armed but never entered the queue. For 13 hours every implement step said only 'pull request #902 is not merged yet', and the loop's rule that nothing is pushed after arming left it no way out. - -## Deferral 2026-10-11 - -Deferred past v0.13.4: the fix is implement-loop code (read the forge's mergeability at the arm and merged-check moves and refuse naming a conflicting pull request), next cycle's work; meanwhile the hand merge the product thinker ruled for on 2026-10-11 is the remedy, and the root cause, appends to DECISIONS.md, is planned away by itd-2609151138388536 diff --git a/.abcd/work/issues/resolved/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md b/.abcd/work/issues/resolved/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md index 17c8ddd97..2f19540c6 100644 --- a/.abcd/work/issues/resolved/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md +++ b/.abcd/work/issues/resolved/iss-2610110941289527-the-merge-queue-s-check-timeout-was.md @@ -22,8 +22,4 @@ The merge queue's check timeout was shorter than one macOS queue entry takes: ov **After #933 (one sample, 2026-10-11).** #933 moved the macOS suite to the merge queue only and made it one race-enabled pass. Its own queue run, 38134740530, ran `check (macos-latest)` in 24.7 minutes (the pass 23.6), against a median of 40 before, and waited 0.1 minutes for a runner, against a median of 40 (p90 68) before. With the person's ruling of 4 entries built at once and a 90-minute check timeout, one entry's wait plus run is now well inside the timeout. That is one run: the remedy's p90 over a day of queue runs is still owed before this record is resolved. -## Deferral 2026-10-11 - -Deferred past v0.13.4: mitigated, not yet proven: the person's ruling (4 entries built, a 90-minute check timeout) and #933 (macOS suite once, in the queue, one pass) brought one queue entry to 24.7 minutes with a 0.1-minute runner wait; the remedy's p90 over a day of queue runs is owed before it is resolved - **The remedy's measurement (2026-10-11, 24 merge-group runs since #933 merged, 11:40Z to 16:30Z).** `check (macos-latest)` waited for a runner a median of 0.1 minutes (p90 0.8, max 26.9) and ran a median of 30.1 minutes (p90 33.0, max 35.5); wait plus run had a median of 30.6 minutes and a p90 of 35.6 (max 57.9). The queue's check timeout is 90 minutes and the queue builds 4 entries at once (ruleset 19969675, mirrored in `.abcd/work/rulesets/main-protection.json` on main), so the timeout sits at two and a half times the p90 and the macOS job's own 45-minute ceiling is 12 minutes above its slowest run. No entry dropped on a timeout in the window: of the 24 runs 22 passed, and the 2 cancelled were superseded runs (pr-903, pr-923) that queue-sweep cancelled at 13:08Z because their queue refs were gone, leaving the live pr-922 runs alone.