Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 44 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -222,8 +222,48 @@ jobs:
# green in 775s against a 780s cap -- 5s of margin -- and the step was killed anyway, reported as
# a failure on a run whose last line was "9598 passed, 855 skipped". A watchdog that cannot
# separate "deadlocked" from "slow today" stops being a watchdog and becomes a coin flip, so the
# ubuntu budget is raised here. Re-check the margin when the suite grows; the Windows legs are
# unchanged because 26 min against the same suite is still ~2x headroom.
# ubuntu budget is raised here. Re-check the margin when the suite grows.
#
# It then happened again on WINDOWS, 2026-08-01, because the sentence that used to close this
# paragraph asserted the Windows legs held "~2x headroom" and nobody re-derived it. PR #119 was
# killed at 26:07 against the 26:00 cap with ZERO tests failing. What moved was the suite, not the
# code under test: #74 landed tests/test_worktree_prune_merged.py (1,506 lines) and windows-2025
# went 19:35 -> 26:07 on the same branch. The claim was already false when it was written.
#
# Measured over the 11 PASSING windows-2025 runs on 2026-08-01, timing the `Tests (pytest)` STEP,
# which is what step_timeout gates. NOT the job: the job is ~3 min longer and capped separately,
# and two sessions misread job durations as step durations while triaging this (c53f752b's JOB ran
# 28:41 and PASSED, because job cap 30 vs step cap 26).
#
# leg max passing step old cap old margin
# ubuntu-latest 12:27 19:00 1.53x
# windows-2022 18:39 26:00 1.39x
# windows-2025 24:35 26:00 1.06x
#
# windows-2025 had already PASSED at 24:35 -- 85 seconds of margin -- before #119 died. The "2x"
# figure matched no leg. State the MEASURED value and its DATE, never a round multiple: a bare
# multiple gives the next reader no way to tell when it has rotted, which is how this one survived.
#
# PROOF IT WAS THE CAP, NOT THE BRANCH. #119's windows-2025 leg was RE-RUN on the SAME commit
# against the SAME 26:00 cap: attempt 1 was killed at the cap, attempt 2 concluded SUCCESS. Same
# code, same config, same ceiling, two outcomes. The leg was not failing -- it was coin-flipping
# against the cap, exactly the state the ubuntu note above names, and the reason "re-run it and
# see" is not a diagnosis here. A green re-run at 26:00 does not mean the suite fits; it means
# that runner was fast enough that time.
#
# 36:00 is 1.46x over that 24:35 maximum -- the margin ubuntu already runs with. Both Windows legs
# take the same number: windows-2022 is the faster of the two, so sizing on windows-2025 only
# leaves it more room, and one value is one thing to re-derive. job_timeout moves 30 -> 40 to keep
# the nesting invariant above: the step must still expire strictly BEFORE the job.
#
# This cap is NOT what catches a hung test; pytest_timeout (120s) is, per test. The step cap only
# catches a whole-process deadlock both in-process watchdogs miss, which is why it can sit well
# above a healthy run. Sizing it tight buys no detection and costs false failures on green suites,
# twice now.
#
# The remaining margin is a SHARED budget across every PR that lands and nothing accounts for it:
# three PRs each adding a minute of Windows time reproduce #119's death, individually blameless.
# A mechanical guard for that is BACKLOG #341; the underlying slowness is #320.
- name: Tests (pytest)
if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch'
timeout-minutes: ${{ matrix.step_timeout }}
Expand Down Expand Up @@ -382,8 +422,8 @@ jobs:
# expression interpolation into the run body), so it is zizmor-safe and cannot be misparsed as
# an Actions expression the way a literal double-brace token in a run: block would be.
U='{"os":"ubuntu-latest","python-version":"3.14","hosted":["ubuntu-latest"],"job_timeout":22,"step_timeout":19,"pytest_timeout":60,"fault_timeout":90}'
W22='{"os":"windows-2022","python-version":"3.14","hosted":["windows-2022"],"job_timeout":30,"step_timeout":26,"pytest_timeout":120,"fault_timeout":150}'
W25='{"os":"windows-2025","python-version":"3.14","hosted":["windows-2025"],"job_timeout":30,"step_timeout":26,"pytest_timeout":120,"fault_timeout":150}'
W22='{"os":"windows-2022","python-version":"3.14","hosted":["windows-2022"],"job_timeout":40,"step_timeout":36,"pytest_timeout":120,"fault_timeout":150}'
W25='{"os":"windows-2025","python-version":"3.14","hosted":["windows-2025"],"job_timeout":40,"step_timeout":36,"pytest_timeout":120,"fault_timeout":150}'
if [ "${GITHUB_REPOSITORY:-}" = "MEFORORG/MessageFoundry" ]; then
echo "matrix={\"include\":[$U,$W22,$W25]}" >> "$GITHUB_OUTPUT"
else
Expand Down
57 changes: 57 additions & 0 deletions .github/workflows/stalled-prs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
name: Stalled PRs

# A pull request can be finished, green, armed to merge -- and unable to merge, forever, silently.
#
# THE DEFECT THIS EXISTS FOR — measured on this repo, 2026-08-01: NINE open pull requests with zero
# failing checks and zero pending checks, not one of which could merge. Six had auto-merge ARMED, which
# will never fire. PR #74 had been in that state since 2026-07-30 and was found only because somebody
# went looking for "stuck CI" by hand.
#
# THE MECHANISM. Branch protection sets `required_status_checks.strict = true` and there is no merge
# queue, so a PR that goes green must finish while `main` holds still. When anything else lands first it
# flips to BEHIND and stops. Armed auto-merge does NOT update a BEHIND branch — it waits on checks that
# already passed. Nothing re-syncs it and nothing reports it.
#
# WHY NO EXISTING SIGNAL CATCHES IT. Every other signal is a check OUTCOME, and no check has failed —
# that is the whole problem. `nightly-notice.yml` watches CI runs and there is no failing run to watch.
# The author's last signal was a full pass, so they have no reason to look. A green dashboard and a
# wedged repository are indistinguishable unless something asks "can this still merge at all?".
#
# SCHEDULED, not per-PR, deliberately: the stall arrives when a DIFFERENT pull request merges, so the
# affected PR has no run in flight to hang a check on. It becomes true while the repo is idle.
#
# ADVISORY BY PLACEMENT — this must never become a required context. It reports on OTHER pull requests,
# so a stall on #71 would block #128, which is both wrong and a way to wedge the repo with the very
# tool meant to unwedge it. See .github/required-contexts.txt for the required-but-absent trap.
#
# This does NOT fix the race; only a merge queue does. It converts a SILENT failure into a LOUD one,
# which is the part that let #74 sit for days. If a merge queue is enabled, this goes quiet on its own.
on:
schedule:
# 07:05 UTC — just after `Required workflow state`, so the two merge-health checks report together.
- cron: "5 7 * * *"
workflow_dispatch:

permissions:
contents: read

jobs:
stalled:
name: open PRs can still reach a merge
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: read # list PRs + their merge state; no write scope anywhere
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
# No pip install: the script is stdlib-only and shells out to `gh`, which is preinstalled on the
# runner. Nothing here parses the repo, so there is no PyYAML dependency to pin.
- name: Report pull requests that are green but cannot merge
env:
GH_TOKEN: ${{ github.token }}
run: python scripts/ci/check_stalled_prs.py --repo "$GITHUB_REPOSITORY"
66 changes: 66 additions & 0 deletions docs/BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8242,3 +8242,69 @@ Separately: the docstring argues *"the return value is the point"*, but all six
**Source:** public-repo disclosure audit, 2026-08-01. Re-verified and re-measured at HEAD on the same date.

---

## 340. Enable a GitHub merge queue: strict + no queue makes every merge a race that fails silently

> 🔢 **Filed 2026-08-01 — not started.** Value **8/10** · Difficulty **3/10** · _fill-in_. `strict = true` with no merge queue and a ~20-minute suite means a PR must go green *and* have `main` hold still, or it flips to `BEHIND` and stops. Measured this date: **9 open PRs green (0 failing, 0 pending) and none able to merge**, 6 with auto-merge armed that will never fire.

**Cluster:** Developer Experience & CI. **Priority:** P2. **Verdict:** build. **Severity:** medium.

**What:** branch protection on `main` sets `required_status_checks.strict = true` — a PR must be up to date with the base to merge — and the repo has **no merge queue** (`repository.mergeQueue` is null; `allow_auto_merge` is true). The slowest required leg runs ~20–25 minutes. Those three facts compose into a race: a PR is mergeable only in the window between its checks going green and the next thing landing on `main`.

Losing that race is **silent**. Armed auto-merge does *not* update a `BEHIND` branch — it waits on checks that already passed — so the PR sits armed and stalled with no failing check, no notification, and no run in flight. Measured 2026-08-01:

```
PR mergeState failing pending auto-merge
#128 BEHIND 0 0 ARMED
#125 BEHIND 0 0 ARMED
#107 BEHIND 0 0 -
#106 BEHIND 0 0 ARMED
#101 BEHIND 0 0 ARMED
#96 BEHIND 0 0 ARMED
#71 BEHIND 0 0 ARMED
#60 BEHIND 0 0 -
```

(Nine at the hand survey; eight when [`check_stalled_prs.py`](../scripts/ci/check_stalled_prs.py) ran ~20 minutes later, because #120 had been re-synced in between. The set moves — the condition does not.)

Two worked instances the same day. **#74** went green on 2026-07-30 and sat unmergeable until 2026-08-01, found only by someone hunting "stuck CI" by hand; it took three merges from `main` to land. **#119** was green with 25 passing checks and armed, stalled, was re-synced, and lost the window again. A hand-coordinated merge freeze across five sessions *did* hold `main` still for a full window — and #119 still failed, on an unrelated timeout (#344) — which is the evidence that hand coordination is not the fix.

**Why:** the cost is finished work sitting undelivered while everyone believes it is landing. This is the repo's recurring defect shape — a signal accurate about what it looks at and silent about what it does not ([`Secure_Development_Standards`](Secure_Development_Standards.md) §3) — but the worst variant, because every other instance has *someone waiting on a result*. Here the author already had their full pass and has no reason to look again. It also scales the wrong way: the more sessions working in parallel, the more often `main` moves, so the race gets harder to win exactly as throughput rises.

**Proposed:**
1. Enable a merge queue on `main` (branch protection → *Require merge queue*), squash method to match the existing history.
2. Reconcile the required set against it: a queue runs checks on a `gh-readonly-queue/**` ref, so any workflow that must gate the queue needs a `merge_group:` trigger. Every context in [`.github/required-contexts.txt`](../.github/required-contexts.txt) lacking one will never report there — that file's own required-but-absent trap, in a new place.
3. Decide the interaction with `strict = true`. A queue makes it largely redundant; leaving both on is safe but keeps the re-sync burden for anything bypassing the queue.
4. Once landed, [`check_stalled_prs.py`](../scripts/ci/check_stalled_prs.py) goes quiet on its own. Keep it — it is the detector for this class returning.

**Related:** [`scripts/ci/check_stalled_prs.py`](../scripts/ci/check_stalled_prs.py) + [`.github/workflows/stalled-prs.yml`](../.github/workflows/stalled-prs.yml) (built alongside this filing — it reports the condition, it does not remove it); [`.github/required-contexts.txt`](../.github/required-contexts.txt); [`scripts/ci/check_required_workflow_state.py`](../scripts/ci/check_required_workflow_state.py) (sibling: "can this context ever report?" to this one's "can this PR ever merge?"); #344 (the wall-clock bounds that actually killed #119); #320.

**Source:** stuck-CI triage, 2026-08-01. Measured live against `MEFORORG/MessageFoundry` branch protection and the open-PR set that date; independently reached by three parallel sessions from separate evidence.

---

## 344. Fixed wall-clock bounds have drifted out of proportion to the work they bound

> 🔢 **Filed 2026-08-01 — not started.** Value **6/10** · Difficulty **4/10** · _fill-in_. Hardcoded real-time budgets — a CI `step_timeout`, a test helper's poll deadline — were sized when the work was smaller or the machine faster, and nothing re-derives them. Two confirmed instances; each presents as an unrelated flake.

**Cluster:** Developer Experience & CI. **Priority:** P2. **Verdict:** build. **Severity:** medium.

**What:** a fixed wall-clock bound with no relationship to the work it bounds fails the day the work grows, and it fails as a **timeout with zero assertion failures** — which reads as a broken branch when nothing is broken.

*Instance 1 (fixed 2026-08-01, this filing).* `ci.yml`'s Windows `step_timeout: 26`. windows-2025's max PASSING `Tests (pytest)` step was **24:35** over 11 runs — 1.06x margin — and PR #119 was killed at 26:07 with zero test failures after #74 added a 1,506-line test file. The comment beside the cap asserted "~2x headroom", a figure that matched no leg. Raised to 36:00 (1.46x) with the measurement and its date recorded in place of the multiple.

*Instance 2 (open).* `tests/test_stage_dispatcher.py`'s `_wait_until` (:356) polls `loop.time()` against a hardcoded **8.0s** budget while the dispatcher under test is driven by an injected `ManualClock`. On PR #129 — whose diff is provably AST-identical to main, docstrings only — `test_adr0070_9_content_retry_is_not_an_infra_fault[sqlserver]` failed on that bound alone against a Dockerised SQL Server; every logic assertion in the same loop passed. A real-time deadline gating a virtual-clock system has no principled value.

**Why:** these fail *individually blameless*. The remaining CI margin is a **shared budget nobody accounts for** — three PRs each adding a minute of Windows time reproduce #119's death, with no single PR at fault. And this repo has twice mislabelled such a failure: the two famous "flakes" turned out to be a livelock and a test that was right. A timeout with no failing assertion is the exact signature that invites the wrong diagnosis.

**Proposed:**
1. **A mechanical margin check** (suggested by the ASVS-scorecard session, whose framing this is): compare each leg's actual `Tests (pytest)` step duration against its configured `step_timeout` and fail below ~1.3x. Computable from data CI already emits. It would have flagged windows-2025 *before* #119 died — it was already at 1.06x and nothing said a word — and unlike the "re-check the margin when the suite grows" instruction in `ci.yml`, it does not depend on anyone remembering. Two traps for whoever builds it: time the **STEP**, not the job (the job is ~3 min longer with its own cap — `c53f752b`'s job ran 28:41 and passed against job cap 30 / step cap 26, and two sessions misread job for step while triaging this); and size against the **max passing** run, not the mean (windows-2025's mean ~21 min looks comfortable, its max passing 24:35 is what bites).
2. Size the remaining bounds: `grep` hardcoded `timeout=` / deadline floats under `tests/` and judge each against the work it bounds.
3. Where a virtual clock drives the system under test, the poll deadline should follow that clock, not `loop.time()` — instance 2 is the worked example.
4. Prefer bounds expressed as a measured ratio with a date over round multiples, per instance 1's post-mortem.

**Related:** [`.github/workflows/ci.yml`](../.github/workflows/ci.yml) §*Tests (pytest)* (instance 1 and its measurement table); `tests/test_stage_dispatcher.py`:356 (instance 2); #320 (windows-2025 slowness — the capacity fact that shrinks every Windows margin); #340 (the other half of this triage); [`Secure_Development_Standards`](Secure_Development_Standards.md) §3 (prose asserting a margin the numbers do not support — five instances found on 2026-08-01 alone).

**Source:** stuck-CI triage, 2026-08-01. Instance 1 measured across 11 windows-2025 runs; instance 2 reported and diagnosed by the HA-construct-recheck session from PR #129's sqlserver leg.

---
Loading
Loading