ci: the margin table was wrong twice — re-measure the pool, and size the job cap from maxima - #138
Merged
Merged
Conversation
…had already fired #131 replaced a "~2x headroom" claim with a measured table. The table was itself wrong -- every row, each in the safe-looking direction -- and the job cap it left in place was already negative. MEASURED over all 101 CI runs created 2026-08-01 (`gh api --paginate`), timing each leg's `Tests (pytest)` STEP and filtering on the STEP's own conclusion: leg claimed true (n) old cap true margin ubuntu-latest 12:27 12:31 (n=57) 19:00 1.518x windows-2022 18:39 21:34 (n=52) 26:00 1.206x windows-2025 24:35 25:51 (n=49) 26:00 1.006x Nine seconds, not 85. Two mechanical causes, both cheap to repeat: * THE POOL WAS A PAGE, NOT A QUESTION. It came from `gh run list --limit 20` -- a default-sized listing reported as though the sample had been chosen. 20 of 101. * FILTERING ON *JOB* CONCLUSION DROPS THE TIGHTEST STEPS BY CONSTRUCTION. A step near step_timeout is the one most likely to push its job into job_timeout, so the job is cancelled while the step concluded success. Five such rows exist that day and they include the maximum. Only the windows-2025 row was ever a maximum; ubuntu's and windows-2022's figures were that same run's other two legs. AND THE TWO-GATED-STEPS HAZARD IS NOT LATENT -- IT FIRED. Both `Tests (pytest)` and `Web console tests (pytest)` carry `step_timeout`, so a job can hold 2x step_timeout of gated work that step_timeout cannot bound. Run 30724385719 (main @ 8f01cef): Tests (pytest) 25:51 SUCCESS (9s under the 26:00 cap) Web console tests (pytest) CANCELLED JOB 30:13 CANCELLED <- job_timeout 30 fired A green first step, then an unattributed job-level kill during the second -- precisely what ci.yml:218's nesting note exists to prevent, by the path it does not consider. It cannot happen when a step is KILLED (that ends the job and skips what follows), only when the first step PASSES near its budget. Sizing job_timeout to hold both gated steps plus setup, rather than step_timeout plus a constant: leg step + web(max) + overhead old job new job ubuntu 19:00 + 1:58 + 0:41 = 21:39 22:00 +21s 26:00 +4:21 (1.20x) W22/W25 36:00 + 3:27 + 0:41 = 40:08 40:00 -8s 46:00 +5:52 (1.15x) All three legs sat inside a minute of their job cap and Windows was already negative: the +4 convention was carried through two cap changes without anyone summing what it had to cover. The 36:00 step decision is unchanged and remains correct -- 1.393x over the true 25:51 maximum, still comfortably above the 5:26 observed spread. Only its justification moves. Found by the ADR 0158 verification pass (eight agents re-deriving every claimed number against the API; 50 claims checked, 6 refuted), reported by the intersession-communication-hooks session, and re-derived here before acting. BACKLOG #344 still restates the superseded figures; that edit is blocked on three live sessions holding docs/BACKLOG.md and follows separately.
wshallwshall
enabled auto-merge (squash)
August 2, 2026 03:40
Written at the owner's stop-work instruction (usage cap). Chat does not survive; a claim that lives only in a transcript reaches nobody. Carries: #138's state and its SQL Server blocker (NOT called a flake -- unproven), the corrected margin table, four retractions of my own findings with their corrected forms, and seven traps stated as fact-plus-measurement. The load-bearing line: the cap raise in #131 is correct and unchanged; its justification was wrong in every row, and #138 fixes it.
wshallwshall
disabled auto-merge
August 2, 2026 04:11
…om maxima The correction in this PR fixed #131's maxima but repeated #131's own failure mode in a new place: it reported a pool and an n that its stated pool cannot produce. Two independent re-derivations agree the day holds 70 ci.yml runs, not 101, and per-leg n = 42 / 39 / 36, not 57 / 52 / 49. The three MAXIMA (12:31 / 21:34 / 25:51) re-derive to the second and are unchanged, so the 36:00 step / 46:00 job decision stands on the same numbers it always did. What actually changed: * POOL AND n. 70 runs created 2026-08-01 UTC, cross-checked four ways; n = 42 / 39 / 36. A table whose own point is "state your pool and your n" has to carry an n the next reader can recompute. * THE POOL IS RIGHT-CENSORED, and nothing said so. Every run in it predates #131 (28d186b, 2026-08-02T00:35:28Z), so each observation survived a 26:00 cap. 25:51 is the largest step that FIT, not the largest the suite wants, and a multiple of a censored maximum under-provisions by construction. This is why 1.06x read as survivable right up to the moment #119 died. * THE JOB-CAP ADDENDS WERE TYPICAL VALUES, NOT MAXIMA. 0:41 is the median setup; the web-console figures 1:58 / 3:27 are each only third-highest on their leg. Measured maxima are 2:00 / 2:33 / 3:33 (web) and 1:20 / 1:09 / 1:04 (setup). Re-summed, ubuntu was -0:20 and Windows -0:37 against the old caps: BOTH were already negative, not just Windows. * THE NESTING INVARIANT IS NOT RESTORED, and now says so. It holds for `Tests (pytest)` on every leg and for `Web console tests (pytest)` on none: reaching that step already spends setup plus `Tests`, so its own cap can never fire first. A hang there still surfaces as an unattributed job kill. Stating otherwise would rest a compensating control on a false premise. * The "5:26 spread ... identical code" claim is withdrawn; neither endpoint re-derives. The true in-pool spread is 9:55, which is wider and supports the decision more strongly than the figure it replaces. Also corrects three pre-existing claims in this file that match no configuration -- the same defect class this note is about: - the pytest step's cap described as `timeout-minutes: 13` (it is matrix.step_timeout, 19 or 36) - "the 15-min job cap" (it is matrix.job_timeout, 26 or 46) - the mechanical margin guard cited as BACKLOG #341, which does not exist on main; it is #344 item 1 Drops HANDOFF-ci-margin-correction.md: session ephemera, and no HANDOFF file has ever been tracked at the repo root on main.
Run 30724385719 ran under the retired 26/30 pair and would have passed under #131's 40:00. It demonstrates that two steps sharing one step_timeout lets the job cap fire behind a green step; it is not itself evidence that 40:00 is too tight. That case rests on the arithmetic, which is arithmetic -- no job has been observed hitting 40:00.
wshallwshall
changed the title
#344 is the item about bounds stated independently of the work they bound. Its instance 1 still carried the retracted measurement -- 24:35 over "11 runs", 1.06x, raised to "1.46x" -- every figure of which was superseded, and the edit had been outstanding since it was filed. Instance 1 now reads 25:51 / 1.006x / 1.393x, and carries its pool (70 ci.yml runs created 2026-08-01 UTC; per-leg n = 42 / 39 / 36) so the next reader can recompute it rather than trust it. Adds two things the re-measurement turned up: * INSTANCE 3 -- the CI job cap. Two steps in that job carry step_timeout, so job_timeout must cover their sum, and the +4 convention that sized it was never summed against what it had to hold. Recomputed from measured maxima, ubuntu was -0:20 and windows-2025 -0:37: both already underwater. It presents as a GREEN first step followed by an unattributed job-level kill, which is a signature instance 1's own proposed margin check would not catch, because the step it measures passed. * THE CENSORING TRAP. A max-passing figure is a LOWER BOUND: the pool is censored by whatever cap was in force when it was collected, so the runs that would have exceeded it were killed and are missing from exactly the tail being measured. This is why 1.006x looked survivable until #119 died. Proposal 5 records the structural fix instance 3 does not make: give the web-console step its own cap, because until then ci.yml's nesting invariant is unenforceable for the second gated step on every leg. Banner moves from "not started" to PARTIAL -- instances 1 and 3 are fixed, instance 2 is not. One banner, still OPEN. Also records, once, that instance 1's figures have now been published wrong twice; the maxima survived both passes and the pools did not.
Nine agents re-derived this block; two re-derived the table from scratch under instructions to refute it. They confirmed every figure in the step table and refuted six things written around it. Fixing my own text, since the whole point of this change is not to ship a third unchecked table. * 26:07 EXISTS, and I said it did not. It is in this very pool -- run 30717229521 attempt 1, sha 8c407fb, step conclusion FAILURE, killed at the 26:00 cap -- hidden because the jobs endpoint defaults to `filter=latest`, which returns only the passing attempt-2 re-run. Same filter hid seven same-commit pairs, so the "no identical-code spread is computable" claim was also wrong. `?filter=all` shows both. * THE SPREAD RULE IS NOT MET, and I claimed it was. The first day after the raise produced 26:23 TWICE, both concluding SUCCESS -- uncensored evidence that the population exceeds the old 26:00 cap. Against 26:23 the headroom is 9:37 and the spread 10:27, so "headroom must exceed observed spread" FAILS at 36:00; it would need ~37:00. 36:00 is kept and the reason is now stated plainly -- this cap catches a deadlock, not slowness, and 1.365x over the worst observed run is ample for that -- rather than the rule being asserted as satisfied. Re-derive if a windows-2025 step is ever seen above 28:00. * WINDOWS setup(max) IS 1:20, NOT 1:04. 1:04 came from restricting to rows where BOTH gated steps succeeded, which drops the exhibit run printed 20 lines above (its web-console step was cancelled) -- the same censoring mistake as filtering the step table by job conclusion, made again. W25 is 40:53, so the old cap was -0:53, not -0:37. windows-2022 now gets its own row with its own addends instead of an unrecheckable 39:42. * "the +4 convention" was Windows-only. ubuntu went 15/13 -> 22/19, so +2 then +3, never +4. * "2 x step_timeout ... which job_timeout must cover" asserted a requirement the shipped caps do not meet (38 > 26, 72 > 46). Now says what they are actually sized against. * The nesting invariant at the top of the block asserted a guarantee it does not provide for the SECOND gated step on any leg. Amended there, where a reader meets it, not only in a caveat 100 lines below. Also: pytest_timeout / fault_timeout were quoted as flat 60s / 90s in three places; they are matrix values (60/120 and 90/150) passed explicitly on the command line, so each figure was false on two of three legs. And records that `test` is the only one of this file's ten jobs with any cap at all -- the other nine run on GitHub's 6h default.
…ing now has evidence The windows-2025 job-cap shortfall is -0:53, not -0:37: the 1:04 setup addend it was computed from excluded the exhibit run itself, because that run's web-console step was cancelled. Same censoring mistake, one layer down. Instance 1 now cites the uncensored observation rather than only arguing the maximum must be a lower bound: the day after the raise, windows-2025 produced 26:23 twice, both passing -- runs the old 26:00 cap would have killed. Records that the jobs endpoint hides a killed attempt behind its passing re-run unless asked for ?filter=all, which is why nobody had seen them.
Reported by the announce-hook session, verified here: gh pr view 119 gives MERGED, mergedAt 2026-08-02T01:45:00Z. What the 26:00 cap killed was a RUN, not the PR. "#119 died" reads as never-landed, and that reading had already propagated -- docs/WORKTREES.md asserted #119 "never merged (it died on an unrelated CI timeout)", which that session corrected against the API. This block was one of the places the phrasing came from, so it is fixed at the source rather than only downstream.
Caught by running an exact-arithmetic assertion over every figure in the block rather than re-reading it: 2160/1583 = 1.3644978, which rounds to 1.364. I had rounded it up by hand in both places. Trivial in size and not in kind -- this is a change whose entire subject is numbers published without being recomputed, so it does not get to ship one. Every ratio, sum and delta in the block is now covered by that assertion and all fifteen are exact.
…agnosed Found by investigating a sqlserver failure on this very PR, which turned out to be instance 2 recurring on a different test. PROPOSAL 3 IS WITHDRAWN. It said a poll deadline over a virtual-clock system should follow that clock rather than loop.time(). Implementing that would have HUNG the suite: _wait_until waits on real store I/O, never on virtual time, and ManualClock.now advances only inside advance(), which nothing calls from the poll loop (tests/test_stage_dispatcher.py:182-204). A mc.now-based deadline is never reached, so a bounded `assert False` becomes an unbounded hang stopped only by pytest_timeout or the job cap -- manufacturing the exact signature instances 1 and 3 are about. A virtual clock can only bound work it drives. INSTANCE 2 IS RE-DIAGNOSED, and the original reading -- "the 8.0s bound is too small" -- is refuted by the recurrence's own timings. The failing test took 8.185s, so _wait_until burned its full 8.000s and setup+teardown cost 0.185s: the store was FAST when it failed. In the same process against the same container the sibling retry_forever[sqlserver] drove seven identical fault cycles in 0.364s, and the [sqlite] variant of the failing test passed in 0.144s. A cycle costs ~30-45ms against an 8000ms bound. The lane is not slow to transition; it never transitions. Raising the number would not fix it and would bury it -- which is precisely the mislabelling this item's own Why warns about. The leading mechanism is recorded as evidenced-but-unconfirmed rather than asserted: a sanctioned EMPTY claim drops the lane to IDLE, and these tests deliberately disable the sweep that recovers it in production. What is settled is the negative: not latency, and not a bigger number. PROPOSAL 1 now carries the evidence for preferring a computed gate to a written instruction. Seven claims were retracted across this triage cluster and none was caught by re-reading; every one fell to a mechanism that could return "no". The one an author caught themselves was caught by running exact arithmetic over all fifteen ratios in the block. As a method, re-reading is 0-for-7 here.
…ervability first A second, larger pass measured what instance 2 actually costs, and the numbers change what should be done about it. RATE AND MARGIN. The two affected tests fail 2 times in ~479 observations (~0.4%), zero on postgres (119) and zero across 1,105 sqlite executions. Over a sample of green sqlserver jobs the failing test passes in min 0.185s / median 0.196s / max 0.204s -- the 8.0s bound is ~39x its worst passing run, ~14x over a wider 21-day pool. Both failures sit ~7x beyond the whole passing distribution: a gap, not a tail. SO THIS INSTANCE INVERTS THE ITEM'S OWN GENERIC REMEDY. "Size the bound against the work" would derive ~1-2s here -- TIGHTER than the 8.0s already in place. There is no larger number to justify, and raising it would only convert a 0.4% visible failure into a 0.4% invisible 30-60s pause. An item about bounds that have drifted too LOOSE has to be able to say when the answer is not a bigger number, and this is that case. MECHANISM IS EXPLICITLY UNRESOLVED. Two independent passes disagreed -- one proposes a sanctioned EMPTY claim dropping the lane to a terminal IDLE (these tests disable the sweep that recovers it in production), the other returned NOT PROVEN and is right that the evidence cannot separate that from a genuine stall. Recorded as unresolved rather than picking the more satisfying story. NEW PROPOSAL 6: make the expiry diagnostic before tuning it. `assert await _wait_until(...)` prints `assert False` and nothing else -- no phase, no park deadline, no streak, no task state -- which is why this was read as latency for a day. That is the prerequisite for judging any other proposal here, and unlike them it cannot itself be wrong about the cause. NOT changed: the note that a killed attempt hides behind its passing re-run unless you pass ?filter=all. A reviewing pass claimed filter=all does not return prior attempts; checked directly against both cited runs and it does (attempts [1,2] on each), so the existing text stands.
…eady ships Both adversarial passes converged on the same verdict (raising the bound would mask, not fix) but neither proposed the cheap discriminator, and it turns out not to need building. StageDispatcher.empty_claims (stage_dispatcher.py:1230) already returns (total, wake_fanout, idle_poll) and is fed by _record_empty, whose ONLY call site is the EMPTY branch of _claim_and_dispatch (:686). Under these tests' topology -- lane_provider=set(), sweep_interval=3600, one seeded row -- a clean run must read (0,0,0). So at the moment of failure `empty_claims[0] > 0` proves a spurious EMPTY dropped the lane to a terminal IDLE (T12 sets phase=IDLE and arms no timer), and `== 0` proves the claim never returned at all. One assertion separates the two hypotheses this item currently records as unresolved. A second signature costs nothing and is already in the captured log: a healthy run emits FOUR `re-pending head with backoff` records (1001.000 / 1003.500 / 1008.000 / 1016.500, the ManualClock base plus the infra backoff ladder); the failing run emitted ONE. The lane never took a second fault. Verified against the source before citing it -- accessor, call site and the T12 branch all read directly, not taken from the analysis.
wshallwshall
enabled auto-merge (squash)
August 2, 2026 14:13
… signature that does not generalise Both reported by the session that settled instance 2 on a live SQL Server, and both are defects in text I had already committed. POSTGRES'S ZERO EXONERATES NOTHING. I wrote "zero on postgres (119 observations)" alongside sqlite's zero, which reads as two backends clearing the mechanism. It is not: at ~0.4% a 119-observation sample expects ~0.5 events, so zero is the expected outcome whether or not Postgres is affected -- and Postgres claims via FOR UPDATE SKIP LOCKED, the SAME head-of-line skip. Only SQLite's 0 in 1,105 is structural, because its global lock totally orders producers and claimers. Citing an underpowered sample as evidence of absence is the same error as the censored maximum three paragraphs above it. THE LOG SIGNATURE HOLDS FOR ONE TEST, NOT BOTH. I wrote that "a healthy run" emits four `re-pending head with backoff` records against the failing run's one. True of test_adr0070_1_* only. test_adr0070_9_* takes the content path, which uses mark_failed and never emits that line at all, so zero there is EXPECTED and is not evidence of a second mechanism. Scoped, with the reason, and pointed at the counter rather than the log. Proposal 6 itself is vindicated: empty_claims settled it in one assertion, read (1,0,1) forced deterministically against a live SQL Server. The mechanism write- up belongs to that session; I have corrected only my own two claims and left instance 2's resolution to them.
wshallwshall
added a commit
that referenced
this pull request
Aug 2, 2026
…strings The document's own rule, applied to itself: a quoted string survives a file edit, a line number does not. Ten citations replaced. WHY NOW. All three ci.yml citations (:229, :233, :254) resolve to unrelated text the moment #138 lands, and six docs/BACKLOG.md citations had ALREADY rotted on main before that -- +14 to +40 lines of drift from #345/#346/#347 being appended, with every cited claim surviving verbatim at a new address. Measured fresh against origin/main and against #138's branch, not reused from the report that found them. TENSE, not just addresses. Two of the quoted strings do not survive #138 -- "Measured over the 11 PASSING windows-2025 runs" and "1.46x" are both deleted by it, because #138 ADOPTS this ADR's retractions 1-3 wholesale (12:31, 21:34, 25:51, 1.006x, 1.206x, pools 42/39/36). Left in the present tense those two sentences would ship knowingly false the hour #138 merges, so they now say what ci.yml stated when this was written. The retractions themselves are unchanged and are vindicated by #138, not contradicted. ANCHORS ARE SINGLE-LINE ON PURPOSE. A first pass rewrapped two quotes across a newline, which makes them ungreppable and would have swapped one rot for another. Every anchor is now verified to grep as one line AND to resolve in the tree it points at -- "ZERO tests failing" resolves in ci.yml both on main and after #138. pyproject.toml:266 was simply wrong: the zizmor pin is at :271, in the group opening at :268. Replaced with the group name, which is what the sentence needed and cannot rot. The residual it reports -- that the pin's home is outside zizmor's paths filter -- is verified TRUE and unchanged. OUT OF SCOPE, deliberately: line numbers into less volatile files remain (test_stage_dispatcher.py, claim.ps1, zizmor.yml, install-coordination.ps1, freethread-smoke.yml, collision_gate.ps1). So the ADR does not yet "state no line numbers" outright -- see the handoff note.
wshallwshall
added a commit
that referenced
this pull request
Aug 2, 2026
…ready pointed at (#145) * feat(coord): announce yourself to the other sessions in this repo Every coordination control in this repo is PULL-based: a new session discovers its peers from the SessionStart banner and the peers learn nothing until someone trips the collision gate. That is too late for the collision that costs the most -- two sessions building the same THING in different files, where nothing file-shaped can catch it. This closes the push direction. It ASKS, it cannot send. Hooks are shell commands and session messaging is MCP, so the hook prints the instruction, the live peer roster and the id-resolution rule at the first prompt that has intent to report; the model does the sending. UserPromptSubmit, not SessionStart: at SessionStart a session knows it exists and nothing else, so it can only say hello -- the interrupt without the information. THE ID RULE IS THE PAYLOAD, and it is counter-intuitive enough that the text states it with its evidence. The registry id in this repo's banners is NOT the MCP session id; measured, a registry id and an MCP id for one session shared no characters. Branch does not join them either -- the two rosters reported different branches for the same checkout in 2 of 6 cases. Only cwd joins, and it must be matched EXACTLY: every worktree cwd is an extension of the primary's, so a prefix match resolves a peer in the primary to an arbitrary worktree session. A registry id passed to send_message fails SILENTLY, which reads as the peer ignoring you. EVERY DECISION LEAVES A RECEIPT, because the bug being fixed was a hook that was wired, fired, resolved nothing and exited 0 for weeks -- byte-identical to a healthy hook with no peers. For the same reason the shim carries its OWN missing-script notice: every receipt the hook writes lives INSIDE the script, strictly downstream of the resolution failure that IS the bug, so the shim is the one surface that still reports when the script does not resolve. It is gated on presence.ps1 so the entry stays silent in every unrelated repo on the machine. It always exits 0 -- a UserPromptSubmit hook that fails can block the user's prompt. It consumes presence.ps1 and therefore the single liveness fence; it does not invent a second notion of live. A separate 'mefor-announce' marker keeps it outside install-coordination's mefor-coord strip and outside the website repo's mefor-web-announce entry in the same settings file, so no installer can delete another's hook, and -Only UserPromptSubmit -Uninstall removes announce alone without disarming the collision gate. * test(coord): pin the announce hook, and the anti-no-op wiring class Most tests for a hook like this assert an ABSENCE, and a hook that does nothing at all satisfies every one of them -- which is precisely the production failure being fixed. So the silence assertions are paired with a positive arm: two tests run the SAME runner against fixtures differing only in whether a peer exists, and if the silence tests ever start passing for the wrong reason the positive one goes red first. test_announce_wiring.py is the class the repo had no test for AT ALL: does the thing that gets INSTALLED reach a script that EXISTS, and does it say so when it does not? Its absence is exactly how a wired-but-inert shim survived for weeks. test_every_wired_script_exists_in_this_checkout was written FIRST and watched fail, naming the missing script and printing all three paths it scanned; a green gate is only evidence if it was shown it can see the failure. Also pinned, each because it was got wrong somewhere first: - The foreign UserPromptSubmit entries -- another repo's shim and an unmarked waiting-flag cleanup -- survive install AND uninstall byte-identical. That is the only thing standing between a one-line wiring edit and deleting a hook this repo does not own. - A peer with no StartedAt ranks LAST, not first. ConvertFrom-Json coerces ISO-8601 to DateTime while the '' fallback stays String; Sort-Object over that mixed column raises ZERO errors and puts the empty string FIRST, so without an explicit projected key the least-trustworthy row silently takes the top of a capped target list. - NO_SESSION_ID and DISABLED write their receipt with NO injected -StateDir. An earlier draft resolved the state dir after those branches, so the receipt was unwritable in production while a test that always injected one went green. - Self is excluded by BOTH nets independently: a roster that cannot tell you from a sibling makes the session message itself. - Hostile peer text cannot escape the peer-data block or emit a non-ASCII byte, a hostile session id cannot escape the state dir, and two ids that sanitise identically get two markers. - Two concurrent runs announce exactly once. session-context.ps1 is registered twice on this box today, so double firing is a live pattern, not a hypothetical. * docs(coord): document announcing yourself, and correct a false claim about .claude WORKTREES.md gains the "Announcing yourself" section that the hook's own emitted text and the shim's missing-script notice both cite by name, so the pointer has to land on main in the same merge. It states the id rule ONCE, as the source of record: registry id is not the MCP id, cwd is the only join key and must be matched exactly rather than by prefix, a usable id starts with local_, and a wrong one fails silently. It also states what the change does NOT do. There is no receive-side hook, so the rule that an announcement is peer DATA -- not an operator instruction, and not something to reply to -- lives in the prose and in the fixed message shape and nowhere else. Reachability is given honestly: presence.ps1 is authoritative for who EXISTS, list_sessions only for who can be MESSAGED, and measured, they disagreed 6-to-1. Cost is stated rather than left to be discovered. CORRECTION, and it is why this doc change is in scope rather than deferred: the same chapter claimed ".claude/settings.json is tracked (shared across worktrees)". It is not. /.claude/ is git-ignored, and git ls-files .claude/ returns nothing -- so a worktree's copy is a creation-time snapshot nothing refreshes and several siblings have none at all. That sentence sat at the exact point a reader decides where to install a hook, and it argues for the wrong answer; the new section directly contradicted it. SESSION-DRIFT-CONTROLS.md records announce as the only PUSH control in the D4 layer, plus the two new guarantees worth tracking separately: that wiring reaches a script that exists, and that a resolution failure is now reported by the shim. * fix(coord): stop the collision gate blocking files a peer committed and finished Reported by another session with a repro: it committed a file, went clean, said in writing it was done and handed the file over -- and the peer it handed off to was still refused the edit. overlap.ps1's `Files` is the UNION of what a branch COMMITTED-and-not-yet-landed with what is dirty in its tree. The gate denied on any live row in that set, so "this branch authored it" was treated as "someone is typing in it right now". Those are different claims. The first stays true for the branch's whole life; only the second is what the gate exists to detect. It self-clears on merge -- overlap already intersects three-dot with two-dot so a LANDED branch stops claiming its files. But nothing clears it before landing, and with PRs currently unable to merge, "until it lands" is indefinite: the blocked set grows monotonically and is never released. Two sessions that coordinated correctly and explicitly still cannot hand a file over. That is precisely the failure this gate's own docstring names -- a gate that cries wolf gets uninstalled. overlap.ps1 already told callers to treat its signals differently ("block on live, mention dormant"), but no caller COULD: the row unioned the two signals away. So the row now carries `Dirty`, and the single-file query sets `MatchedDirty` saying which signal actually matched. The gate now DENIES only on an uncommitted edit in a live worktree, and REPORTS committed-and-clean as context instead -- the peer may already have done what you are about to do, which is worth knowing and not worth refusing over. Fails SAFE across the upgrade: a cached row predating `MatchedDirty` has no such property and is treated as dirty, so the gate degrades to its previous over-blocking rather than silently permitting a real collision. Also, while in the file: `git status` now runs with --no-optional-locks. A plain status REWRITES the index of the repo it inspects, and this walks every peer worktree -- so merely asking "what is in flight" was mutating other sessions' checkouts. Verified against the live repro and both directions: the reported file now allows with context; a file with uncommitted changes in a live worktree still denies; an untouched file stays silent. * feat(coord): lead the announce roster with the claim note, not the worktree name Reported by the session it happened to: its worktree is named inter-session-communication-*, auto-generated at creation from a task that session has never worked on -- it has been doing ASVS scorecard work for its entire life. The directory name is the most visible identifier in presence.ps1, overlap.ps1 and this hook's output, and it had already misled TWO sessions (including this one) into guessing that session was building the announce hook. A worktree name is a creation-time label, not a statement of current work, and nothing keeps the two in sync. The claim note is the only field written DELIBERATELY to say what a session is doing, so the roster now prints it, and the legend tells the reader to prefer it over the name. Joined on the claim's `worktree` path, normalised the same way as every other cwd key here. Fail-open throughout: no claims directory, an unreadable claim, or a peer with no claim all just mean the name is the only thing we have -- which is exactly the status quo, never an error. Same session also flagged that the branch I read for it from list_sessions was stale (a spent, merged branch). The announce text already refuses to join on branch and says why; this is a second, independent reason not to trust it. * docs(coord): name the silent-control defect class in the drift inventory A control that cannot distinguish 'ran and resolved' from 'ran and found nothing' is not installed, however it looks. The announce shim outlived every other silent-control defect found the same day BECAUSE it printed a status message -- which is more convincing than silence. The structural cause is the reusable part: every receipt that hook would have written lived inside the script the shim failed to find, so every check sat strictly downstream of the failure it existed to detect. Looking was not neglected, it was impossible. The question to ask of a new control is which surface still reports when the control itself fails to load. Formulation owed to a peer session that hit four instances of this class in one day and named it more sharply than I had. * docs(coord): record the broadcast constraints six sessions learned the hard way Announce-on-join introduces a session; it does not let an established one push an operational notice. That increment is deferred, and on 2026-08-01 six sessions rehearsed it by hand for four hours. Three constraints fell out, recorded so the next attempt does not rediscover them: - A broadcast needs an EXPIRY or a predicate the RECIPIENT can evaluate, never a promise from the sender. A merge freeze shipped with 'lift when #119 merges'; #119 died on an unrelated CI timeout, so five sessions held on a condition that could not arrive and a second round was needed to retract it. - 'Don't do X' is the wrong primitive when automation already has X armed. The freeze asked for restraint while six PRs had auto-merge ARMED and would have landed with nobody clicking anything. The right ask was an action: disarm. - Coordination a tool cannot read does not count. Two sessions agreed IN WRITING to hand over a file and the gate still refused, because the agreement was prose and the gate reads git. Field data from the sessions that lived it, not speculation. * test(coord): pin overlap's dirty-vs-committed signals against real git Nothing drove overlap.ps1's row computation against a real repository, so the question "does MatchedDirty hold when a file is dirty AND committed at once" was unanswerable by the suite. Raised by the session that spent an evening in exactly that state. THAT CASE IS THE ONE THAT FAILS SILENT, which is why it gets a real fixture rather than a stub row. A peer with uncommitted edits in one region and landed work in another is a genuine collision. Had MatchedDirty been derived from the committed diff instead of the working tree it would read FALSE there, the gate would allow, and two sessions would write one file with nothing reported. The over-block this replaced was loud and annoying; that would be quiet and cost someone their work. Verified the tests can SEE it rather than assuming: sabotaged the row to publish an empty Dirty set -- the precise mis-implementation warned about -- and both MatchedDirty assertions went red; restored, all five green. A test written after the code, never observed failing, is a test of nothing. Also pins that overlap does not rewrite a peer worktree's git index, by comparing the index mtime across two queries. An observer must not perturb what it observes, and this one was doing so on every PreToolUse before f55d6c6. Stub rows would only have asserted that the plumbing carries a value someone else computed; the whole question here is what git actually reports. * test(coord): assert a wired coordination hook resolves to a script that exists Raised by the session that traced the shim: the coordination hooks are not installed copies, they are inline commands that locate their script in a working tree at every invocation. If neither base yields the file, Test-Path fails, the loop ends, nothing runs, and the tool call proceeds with no hook and no signal. "The hook is uninstalled" and "the hook ran and permitted this" are indistinguishable from outside, and nothing was watching. Not hypothetical: a foreign UserPromptSubmit entry sat in this same settings file for weeks probing a script that exists only in another repo. The risk composes badly for collision_gate.ps1 specifically, which now (a) fails OPEN on any error, (b) denies less by design after the dirty-vs-committed split, and (c) silently no-ops when unresolvable. Individually defensible; together the realistic bad day is "the gate was never running and nobody noticed". This closes (c) -- the observation is not mine, and it is a good one. Found immediately on writing it: FIVE user settings files across account directories, not the one I knew about. The informational test also prints the original defect as output rather than leaving it invisible: FOREIGN UserPromptSubmit [mefor-web-announce] -> scripts/hooks/announce.ps1: RESOLVES NOTHING HERE It is another repo's entry, so this reports it and does not touch it. Carries a NEGATIVE CONTROL, because the assertion passed on the first run and a green that has never been shown to fail is not evidence. The real hooks cannot be unwired to prove the predicate works -- the primary checkout is shared with live sessions -- so it is exercised against a path known not to exist. Local-machine only: CI has no user settings and these skip there, which means CI does NOT guard this property. Said plainly, and every test prints what it scanned BEFORE it can skip, per test_gate_installed_parity.py -- the pytest config has no -rs, so a skip would otherwise render as a bare dot with no reason. * docs(adr): ADR 0158 silent controls, plus a session handoff Session ended on an owner stop-work instruction at 96% weekly account usage, so this lands the two things that would otherwise have existed only in a transcript. ADR 0158 records a defect class that recurred at least a dozen times across independent surfaces in one working day, in at least two sub-classes: a bound stated independently of the thing it bounds, and a control that cannot observe or act on its own failure. Its spine is that a signal carrying too little information to act on makes every reader re-derive significance by hand until one of them derives it wrong -- so a correct-but-useless RED costs what a silent green costs. EVERY FIGURE IN IT WAS RE-DERIVED BY SOMEONE WHO DID NOT PRODUCE IT, against the repository and the GitHub API. That pass refuted six claims, including four CI numbers that were already merged, and including corrections this session had itself issued hours earlier. Seven retractions are recorded INSIDE the document, each carrying a found-by tag -- because the central empirical finding is that no retraction was made by the author of the claim it retracts, and that is invisible if attribution is smoothed into one voice. Shape over detection is reported as a ratio rather than flattered: three fixes are covered by tests in required CI legs, two by tests that always skip in CI, one by a workflow change with a live residual, and the rest are corrected prose or still open. The Decision separates ENFORCED rules, each naming its gate, from CONVENTION that is knowingly re-breakable. The handoff records what is pushed, what is filed-not-built, and the traps -- a linked worktree's .git being a FILE, a Windows Python unable to read MSYS paths, a raw hasher giving a false mismatch against a git blob on CRLF, and claim.ps1 silently discarding a note refresh. Each is stated as a fact plus its measurement. It also records, first, the five claims this session got wrong -- including retracting a CORRECT estimate on the strength of an incorrect measurement, and sending that false claim to four sessions and the correction to only three. One more arrived while committing this: the leak gate rejected the handoff for a branch slug, on a line a standalone run of the same scanner had passed. The hook scans STAGED files; the standalone run scanned tracked ones. Two scopes, one tool, and only the fail-closed gate could see it. Recorded in the handoff. No engine behaviour changes. * docs(adr): land ADR 0158 -- silent controls, green signals that mean nothing ADR 0158 was authored and committed in 994bfb1 on claude/intersession-communication-hooks-a52335, a trailing commit pushed about an hour and a half AFTER that branch's PR (#133) had already squash-merged. It therefore never reached main and no PR carried it, while the coordination ledger had already allocated the number: docs/adr/README.md stopped at 0156 and 0158 was taken, so the index pointed at a document that did not exist. That gap had a cost. At least four sessions cited this silent-controls taxonomy as "ADR 0157" -- an unrelated HA demotion-safety document allocated to another worktree and still in flight on PR #139. The document that settles the citation was the one sitting unmerged. This branch is cut from 994bfb1 itself, so the original commit stays in history and authorship is exact. The prose, voice and ASCII-only convention are its author's. This commit drops the session handoff and makes three factual corrections where main moved underneath the branch after it was written, each tagged inline in the ADR's own update convention rather than silently rewritten: * 0fdc326 is unreachable from main (this repo squash-merges). It is now given as "merged as 851c849 (#130)", matching the mapping the ADR already uses for 7ebb2ff/2a6649fb. * transports/email.py and transports/direct.py were cited as carrying the same bare starttls() call. 093db33 (#132) gave both an explicit verifying context; pipeline/alert_sinks.py:384 is now the only remaining instance. * The "the false sentence is still there" claim (five sites, one of them numbered Decision rule 13) is closed out: on main the clause survives only inside its own CORRECTED block at :5270 and as a quotation at :7476. The interval is recorded; the rule it produced is unchanged. HANDOFF-announce-hook.md from 994bfb1 is deliberately not landed: it is session state rather than project documentation, no root HANDOFF-*.md has ever existed on main, and it would publish local shim mechanics into a public repo. It stays on its own branch. Verified: exactly one commit in the repository ever added a 0158 ADR and exactly one 0158 filename exists across all refs, so nothing competes for the number. The index row is unchanged from 994bfb1 and appears exactly once. No engine behaviour changes. * docs(adr): make the 0158 TLS update non-perishable The correction I added said 093db33 (#132) left alert_sinks.py as "the only remaining instance on main". That is a checklist-shaped claim with an expiry date: BACKLOG #323 layer 3 (PR #142) closes the alerts call site, and the sentence goes false the moment it lands. Dating the observation does not help a reader who greps for it in a month and finds nothing. Restated as what happened rather than what is currently true -- #132 closed the two connectors, the alerts call site is tracked as #323 layer 3 -- so it holds whether or not #142 merges, and it says outright that the current state must be grepped rather than cited from here. Deliberately does NOT assert that #142 closed the cell: #142 is open at time of writing, and asserting a merge that has not happened is the same defect pointing the other way. found by: the repo-security-review session, which owns #142 and re-derived all three call sites against origin/main before raising it. * docs(adr-0158): replace rotting line-number citations with greppable strings The document's own rule, applied to itself: a quoted string survives a file edit, a line number does not. Ten citations replaced. WHY NOW. All three ci.yml citations (:229, :233, :254) resolve to unrelated text the moment #138 lands, and six docs/BACKLOG.md citations had ALREADY rotted on main before that -- +14 to +40 lines of drift from #345/#346/#347 being appended, with every cited claim surviving verbatim at a new address. Measured fresh against origin/main and against #138's branch, not reused from the report that found them. TENSE, not just addresses. Two of the quoted strings do not survive #138 -- "Measured over the 11 PASSING windows-2025 runs" and "1.46x" are both deleted by it, because #138 ADOPTS this ADR's retractions 1-3 wholesale (12:31, 21:34, 25:51, 1.006x, 1.206x, pools 42/39/36). Left in the present tense those two sentences would ship knowingly false the hour #138 merges, so they now say what ci.yml stated when this was written. The retractions themselves are unchanged and are vindicated by #138, not contradicted. ANCHORS ARE SINGLE-LINE ON PURPOSE. A first pass rewrapped two quotes across a newline, which makes them ungreppable and would have swapped one rot for another. Every anchor is now verified to grep as one line AND to resolve in the tree it points at -- "ZERO tests failing" resolves in ci.yml both on main and after #138. pyproject.toml:266 was simply wrong: the zizmor pin is at :271, in the group opening at :268. Replaced with the group name, which is what the sentence needed and cannot rot. The residual it reports -- that the pin's home is outside zizmor's paths filter -- is verified TRUE and unchanged. OUT OF SCOPE, deliberately: line numbers into less volatile files remain (test_stage_dispatcher.py, claim.ps1, zizmor.yml, install-coordination.ps1, freethread-smoke.yml, collision_gate.ps1). So the ADR does not yet "state no line numbers" outright -- see the handoff note.
wshallwshall
added a commit
that referenced
this pull request
Aug 2, 2026
…lane in terminal IDLE (BACKLOG #344 instance 2) (#153) * fix(tests): a swallowed SQL Server lock-timeout stranded an ADR 0070 lane in terminal IDLE (BACKLOG #344 instance 2) The intermittent sqlserver-only `assert False` in test_stage_dispatcher.py was never the 8.0s bound. Proven, not argued. MECHANISM. claim_fifo_heads runs under SET LOCK_TIMEOUT 0, so a momentarily contended head raises native error 1222 instantly -- and the store CATCHES 1222 and returns a normal EMPTY result (store/sqlserver.py, the _is_lock_timeout branch; deliberate, the "never-block" yield, logged only at DEBUG). The dispatcher's EMPTY branch then takes T12: phase=IDLE with NO timer armed. Production's periodic sweep re-readies exactly such a lane; the ADR 0070 tests set sweep_interval=3600 with an empty lane_provider ON PURPOSE, so IDLE is TERMINAL there and the lane can never be re-claimed. No budget would have helped: the poll cannot succeed at any size. PROOF. Holding a conflicting row lock across the post-unpark re-claim on a live SQL Server reproduces it deterministically -- empty_claims (0,0,0) -> (1,0,1), lane IDLE, park_until=None, no armed timer, head still `pending` and DUE -- and releasing the lock does NOT recover it. The same stranding then appeared UNFORCED on a local full-file run (test_adr0070_1_retry_forever_never_stops_alerts_stuck[sqlserver]). So this is a test-rig gap, not an engine defect. SQLite's immunity is structural (in-process, no lock-timeout translation); Postgres's 119 observations are too few to exonerate it, so read "SQL Server-only" as "server-backed store". FIX, two halves. 1. _wait_until now RAISES on expiry (BACKLOG #344 proposal 6) with the lane's phase, park deadline, streak, task liveness/exception, the store row's status and next_attempt_at, the clock, and empty_claims -- plus a one-line VERDICT naming which hypothesis the reading proves, so no future occurrence is re-diagnosed from scratch. All 60 bare-assert call sites convert for free (the two that legitimately tolerate a timeout pass required=False; the one that carried a message passes note=). 2. The ADR 0070 phase waits go through _wait_lane, which stands in for the sweep those tests disable -- re-readying a lane found in TERMINAL IDLE, but COUNTED and capped at one, and warned about, so a genuine contention regression still fails loudly rather than being absorbed. The stand-in is disabled when IDLE is itself the target phase. FALSIFIED. With the stand-in budget forced to 0, the same injected EMPTY reproduces the original failure -- now carrying the full diagnostic instead of a bare `assert False`. A guard that passes against unpatched code would have measured nothing. Verified: ruff format + check, mypy strict, and the full file green on all three backends (sqlite + a live SQL Server 2022 container + a live Postgres 16), 76 passed per server leg. NOTE for the merge: PR #138 (claude/ci-margin-correction) also edits #344 instance 2 -- it ADDS a "RE-DIAGNOSED" block recording the mechanism as unresolved. That block is superseded by this one and the two will need a one-time reconciliation. * fix(tests): three corrections from the adversarial review of the #344 fix Adversarial verification of the mechanism (30 agents, 23 candidates, 20 refuted) converged on the same diagnosis, but surfaced three things the first pass had wrong or incomplete. All three verified against the source before being written down. 1. empty_claims is NOT a complete discriminator on its own. `> 0` is biconditional with the T12 stranding only because the other EMPTY consumer (T11, EMPTY-with-dirty) is unreachable under this topology; and `== 0` covers at least three distinct stuck states -- CLAIMING (stalled claim), PROCESSING (hung serializer) and READY (claimer not picking it up) -- which only the PHASE separates. _verdict now reads phase alongside empty_claims and names what each combination means, including the previously unhandled "EMPTY recorded but the lane is not IDLE" case. 2. There is a SECOND, quieter route to the same spurious EMPTY on SQL Server. The batch claim's lock probe runs WITH (UPDLOCK, ROWLOCK, READPAST), so a contended head is SKIPPED, and the head-pinned contiguity step then drops the whole lane ("rn=1 missing drops the whole lane => EMPTY") -- with NO log line at all, not even the DEBUG one. Same observable, strictly harder to see. The fix already covers it (identical T12 stranding); the ledger and the verdict now name it. 3. Postgres is NOT structurally immune, and saying "too few observations" understated it. Its claim uses FOR UPDATE SKIP LOCKED (store/postgres.py) -- the same head-of-line skip. Only SQLite's 0/1,105 is structural. The ledger now says so instead of leaning on the sample size. Also fixes a real shadowing bug introduced by the first commit: the store-row loop used `for lane in ...`, rebinding the `lane` PARAMETER, so with two live dispatchers the second one's verdict would have named a stale lane. Measured while here: 12 consecutive full-file runs against a live SQL Server produced ZERO sweep stand-ins (the one failing run was a concurrently-running experiment of mine stealing DB time -- 56s vs the usual 27s), so the cap of 1 has comfortable margin. Re-verified: ruff format + check, mypy strict, the full file green on sqlite + live SQL Server + live Postgres, and both falsification arms still hold. * backlog(#344): the banner is a claim about the body, so it lands with the body #138's session declined my merged-banner proposal, and the argument is better than mine was. A banner asserting "instance 2 resolved" while main still carries "what is NOT settled is the mechanism" is a green signal ahead of its evidence -- the exact failure this item is about, and backlog_status_check.py cannot catch it because it counts banners rather than reading them. Their banner is accurate against the document that will be on main after #138; mine is accurate against the document that is on main after THIS branch. So the banner belongs in the same landing as the RESOLVED body, and that is here. I did NOT take their offered wording. They supplied instance 3's text so I could "name all three" -- but instance 3 and proposal 5 exist only on their branch, so naming them would recreate the mirror of what they just declined: a banner describing a body that is not there if this PR lands first. So the count is gone entirely, per CLAUDE.md §11 / Secure_Development_Standards §3 -- a completeness claim is a liability, and with concurrent PRs adding and closing instances an enumeration goes stale on whichever lands first. The banner now states only what this branch delivers (instance 2 resolved, with the mechanism named) and characterises the rest without counting, so it is true in either merge order. The reason is recorded in the banner itself so the next editor does not helpfully restore a tally. Also merges origin/main (dc0da09, #148's #345/#346 work). Both directions verified by CONTENT, not by the item count: my #344 text (RESOLVED, READPAST, "FOR UPDATE SKIP LOCKED", "not structurally immune") and main's new ##345/##346 all resolve in the merged file, and the merge touched no test files. Gate green at 270. KNOWN, and deliberately left: this still conflicts with #138 on exactly this one line (git merge-tree --write-tree reports a single hunk at :8328, banner only). That is the agreed division -- when #138 lands, take this banner and keep their two corrected sentences in the RE-DIAGNOSED paragraph. * docs(handoff): stopping note for #344 instance 2 — this work exists in exactly one place Paused on the owner's instruction via the coordinator session. Nothing is half-done; the four preceding commits are coherent and independently verified. Why this file exists: every other session's work is pushed to a remote branch. This is LOCAL ONLY -- no PR, no remote branch at all (`git rev-parse --verify origin/<branch>` fails outright). A resumer checking GitHub finds nothing and would reasonably conclude #344 instance 2 was never fixed. Records what is NOT recoverable from the diff: - The mechanism, which was expensive to prove: SET LOCK_TIMEOUT 0 -> native 1222 -> the store CATCHES it and returns a normal EMPTY (deliberate, DEBUG-logged only) -> T12 parks the lane in a TERMINAL IDLE because these tests disable the sweep on purpose. Plus the second, silent READPAST route, and why Postgres is not immune. - The deterministic reproduction recipe. The scratch harnesses were deleted and this is the part nobody should re-derive: hold a conflicting row lock across the post-unpark re-claim, watch empty_claims go (0,0,0) -> (1,0,1) with the head still pending AND due. Includes the signature to match and the [3] clause -- the lane does NOT recover when the lock is released -- which is what proves TERMINAL rather than slow. - That an ISOLATED repro loop cannot reproduce this (800/800 green): running the two tests alone is the one configuration that cannot generate contention. Force it. - Verification already done, including the falsification arm, so nobody redoes it blind. - How to rebuild the DB rigs, with the traps: the native MSSQLSERVER service owning 1434, the single shared MEFOR_STORE_* config that forbids both backends in one pytest invocation, and the silent skip that makes a green local run meaningless. - The #344 banner agreement with PR #138, which until now lived only in two sessions' heads and both are pausing -- including why the agreed order is preferable but NOT load-bearing, so a resumer neither panics about ordering nor restores an instance tally (instance 3 / proposal 5 exist only on #138's branch). - That the ledger claim on 344 is HELD ON PURPOSE and scoped to tests/ only. NOTE on what is deliberately absent: the branch name and worktree path. The leak guard blocked the first version of this file, correctly -- it matches worktree/branch slugs BY SHAPE, on the reasoning that a slug is whatever the task happened to be called and can therefore name a customer engagement, and this repo is public. Not a false positive and NOT allowlisted. Those two strings live in .git/mefor-coord/HANDOFF-344-instance-2.md, which is never published; the committed note points at it and is fully usable without it, since a reader is already on the branch. Docs-only: no code, no test, no ledger banner touched. * docs(handoff): a THIRD source of the 1222, and name the fix commit not the tip Two additions from the #348 / ADR 0159 session, which reached the same 1222 from the opposite end and sent it over rather than sitting on it. Still paused -- this finishes the stopping note, it does not resume work. 1. A CANCELLED store call is a third route to the same EMPTY, and it refines how the fix commit words the cause. CancelledError derives from BaseException, so the store's `except Exception: await conn.rollback()` idiom never runs its rollback on cancellation, and aioodbc's Pool.release() returns the connection to the free deque with no rollback or reset. THEIR measurement (live container): cancelling release_claimed left 7 KEY X locks on `queue`, vs 0 for a claim_fifo_heads control, after which a real claim returned EMPTY-all and a raw writer got 1222. VERIFIED HERE before recording it, rather than relayed: store/sqlserver.py has 90 rollback sites guarded by `except Exception` and ZERO using `except BaseException` or a `finally`. So the gap is real in our source, not just in principle. Why it matters: 58f8f13's message reasons from MOMENTARY contention -- a live producer holding a head lock in flight. A connection poisoned this way holds its locks as long as it sits unclaimed in the pool, so the 1222 can repeat across successive sweep ticks instead of clearing on the next. The chain into T12 and the terminal IDLE is identical; the duration and origin of the contention are not. It does NOT change this branch's conclusion -- production still recovers via the sweep, so "test-rig gap, not an engine defect" stands, and that session explicitly declined to escalate severity on it. Recorded as: if you see a 1222 you cannot attribute to a live producer, look at #348 first. 2. Name the fix commit, not the tip. HEAD was circulated between sessions as "the fix"; it is a banner edit plus this note and touches no code. THE FIX IS 58f8f13. The handoff now says so in a callout above the commit table, because the mis-relay already happened once and a tip SHA moves every time this note is amended. Cross-referenced by LEDGER NUMBER (#348 / ADR 0159), never by SHA -- that work is also unpushed and may be rebased. Docs-only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#131 replaced a "~2x headroom" claim with a measured table. That table was wrong in every row. The first correction on this branch fixed the maxima but repeated #131's own failure mode in a new place — it reported a pool and an
nthat its stated pool cannot produce. This revision fixes that, and sizesjob_timeoutfrom measured maxima instead of typical values.Two independent re-derivations agree on every figure below.
The measurement
Pool: every run of
ci.ymlcreated 2026-08-01 UTC — 70 runs, enumerated over a deliberately wider window and filtered locally oncreated_at, then cross-checked against the narrow?created=query (same 70 ids, symmetric difference 0). Rows are each leg'sTests (pytest)step, kept when that step concluded success.The maxima re-derive to the second. The pool size (
101) and everyn(57/52/49) do not reproduce under any pool definition tried.These maxima are lower bounds — the pool is right-censored
Every run in it predates #131 (
28d186b5, landed2026-08-02T00:35:28Z), so Windows ran under step 26:00 / job 30:00. A run that wanted longer was killed at the cap and then — correctly, by the filter above — dropped for not concluding success. 25:51 is the largest step that FIT in 26:00, not the largest the suite wants. Sizing a cap as a multiple of a censored maximum under-provisions by construction. That is exactly how "1.06x" read as survivable right up until #119 died.Three mechanisms, all cheap to repeat
gh run list --limit 20— a default page size, reported as though chosen.n" has to carry annthe next reader can recompute — that column is how they would tell it had rotted.The job cap
Both
Tests (pytest)andWeb console tests (pytest)carrystep_timeout, so a job can hold 2 ×step_timeoutof gated work thatstep_timeoutcannot bound. Run30724385719(main @8f01cef8), windows-2025 — all six timestamps confirmed:Read that exhibit for its mechanism, not as a verdict on the cap it replaces: it ran under the retired 26/30 pair and would have passed under #131's 40:00. What condemns 40:00 is the arithmetic below — and that is arithmetic; no job has been observed hitting 40:00.
Every addend is now that addend's measured maximum. The previous revision used
0:41for setup, which is the median, and web-console figures (1:58/3:27) that are each only the third-highest on their leg:ubuntu and windows-2025 were already negative — ubuntu as well, which an earlier revision had the wrong way round (it showed ubuntu at +21s). windows-2022 was the one row genuinely in the black.
The Windows setup addend was itself wrong twice:
0:41(a median), then1:04— which came from restricting to rows where both gated steps succeeded, silently dropping the exhibit run whose web-console step was cancelled. Same censoring mistake as filtering the step table by job conclusion, made a second time. The true figure,1:20, is visible in the exhibit printed directly above it.What this does not fix
The nesting invariant
ci.ymlasserts — "the step must expire BEFORE the job" — holds forTests (pytest)on every leg and forWeb console tests (pytest)on none of them: reaching that step already spends setup plusTests, so its own cap can never fire first. Satisfying it would needjob_timeoutpast 39:24 on ubuntu and 73:21 on Windows. A hang there still surfaces as an unattributed job-level kill. The file now says so instead of restating the claim — a compensating control must not rest on a false premise. The structural fix (stop the two steps sharing one budget) is BACKLOG #344.The spread rule is not met, and the file now says so
An earlier revision claimed 36:00 "comfortably clears" the observed spread. Against real values it does not. The first day after the raise produced windows-2025
Testssteps of 26:23 twice, both passing — uncensored evidence that the population exceeds the old 26:00 cap. Against 26:23: headroom 9:37 vs spread 10:27. The rule would need ~37:00.36:00 is kept, with the reason stated rather than dressed up: this cap exists to catch a whole-process deadlock, not slowness, so 1.365x over the worst observed run is ample for its actual job, and #131 already set this value. Re-derive if a windows-2025 step is ever seen above 28:00.
Also withdrawn — both were wrong: "
26:07does not exist" and "no identical-code spread is computable". Artifacts of the jobs endpoint defaulting tofilter=latest, which hides a step killed at the cap behind its passing re-run.26:07is in this very pool (run 30717229521 attempt 1, killed at the cap), and seven same-commit pairs exist under?filter=all.Also corrected
Three pre-existing claims in this same file that match no configuration — the very defect class this note is about:
timeout-minutes: 13(it ismatrix.step_timeout, 19 or 36)matrix.job_timeout, 26 or 46)main; it is #344 item 1Drops
HANDOFF-ci-margin-correction.md: session ephemera, and no HANDOFF file has ever been tracked at the repo root onmain.The 36:00 step / 46:00 job decision is unchanged and remains correct. Only its justification moves — for the second time, which is itself the finding.
Verification
Nine agents; two re-derived the table from scratch under instructions to refute it. They confirmed every figure in the step table and refuted six things written around it — all six fixed in
5be4d1fe.The sqlserver
HYT00isPRE_EXISTING_INTERMITTENT, not caused by this PR. Adversarially checked and upheld:adr0154-sync-reply@88f99b0c, ~10h before this PR was opened, carrying the pre-ci: the margin table was wrong twice — re-measure the pool, and size the job cap from maxima #138ci.ymlsqlserver-storedeclares notimeout-minutesand never readsneeds.changes.outputs.matrix(the only output this PR changes); and every leg runs on GitHub-hosted runners perci.yml:49("the self-hosted NucBox Windows runners are RETIRED"), so there is no shared pool for a longertestjob to starveNot claimed: that it is harmless.
HYT00is a query timeout — a fixed wall-clock bound, the same shape as #344.