fix: give the macOS leg a 70-minute ceiling, warning at 60

.abcd/work/decisions/dcn-2610181756328249-the-check-job-s-macos-leg-takes-a.md

@@ -0,0 +1,6 @@
+---
+id: dcn-2610181756328249
+slug: the-check-job-s-macos-leg-takes-a
+date: 2026-10-18
+---
+2026-10-18 — The check job's macOS leg takes a 70-minute ceiling, with its warning at 60, raised from 60 and 50 by the coordinating session on delegation from the technical facilitator, who is away until 2026-10-23 (iss-2610181711260494). Across ten merge-group runs on 2026-10-18 the leg took 35.1 to 49.3 minutes, and one came within 0.7 minutes of its 50-minute warning. Its one race-enabled pass keeps a steady 2.80 to 2.85 of the runner's three test slots busy, so the leg is set by its total work rather than by one package: the slowest, internal/core/implement/loop, starts 8 to 12 minutes in and ends 2.5 to 4.2 minutes before the pass does. Its time varies with the runner, since the same tree's package times summed to between 5,738 and 8,050 seconds. The new ceiling stays under the merge queue's 90-minute check response timeout, and the warning keeps its 10-minute lead. The per-package -timeout stays at 40 minutes: the slowest package under race took at most 1,901 seconds, 31.7 minutes, a margin of 8.3. release.yml's verify job is not touched, since it runs the ubuntu race lane, about 21 minutes, under its 55-minute ceiling, not this pass. Starting the loop package first was not taken (at most 2 to 3 minutes, against a list pinned in three files), nor was running it without the race detector on macOS (the macOS-only flakes seen so far came from the race leg). Cutting the pass's total work is the longer lever, captured on its own.

.abcd/work/issues/open/iss-2610181757540736-the-check-job-s-race-passes-are-set-by.md

@@ -0,0 +1,15 @@
+---
+schema_version: 1
+id: "iss-2610181757540736"
+slug: "the-check-job-s-race-passes-are-set-by"
+severity: "minor"
+category: "tech-debt"
+source: "agent-finding"
+found_during: "manual-capture"
+origin: researcher-authored
+production_mode: hand-written
+found_at: "internal/core/implement/loop"
+remedy: "Measure git processes per package across the whole loop, cli and claim suites, not one test, find the call sites that dominate there, and cut those, each change judged by the package-wide count and then by merge-group race-pass time over several runs"
+---
+
+The check job's race passes are set by their total work: the macOS pass keeps a steady 2.80 to 2.85 of its runner's three test slots busy and the ubuntu one about 3.4 of four, so no reordering shortens them, and their slowest packages (internal/core/implement/loop, internal/surface/cli, internal/core/claim, about 46% of the macOS pass) spend most of it in git processes. A first cut, a default-branch name per claim operation and a batched view (#1099, #1100), took one drain test from 1,107 git processes to 830 but the whole of the loop and claim packages only from 126,334 to 119,327 (5.5%), which CI could not tell from the difference between runners; a cut measured on one test overstates the package.

.abcd/work/issues/resolved/iss-2610180545134593-the-implement-loop-package-s-tests-are.md

@@ -22,4 +22,4 @@ The implement loop package's tests are bound by git subprocesses: the slowest, T
 
 ## Outcome
 
-The grounds were shown wrong on the first two merge-group runs after both changes (#1099, #1100, 2026-10-18). The ubuntu leg's race step took 20.8 and 20.6 minutes, at the top of the 14 to 20.6 minutes it had taken before, not a minute shorter. Its package times summed to 4,293 and 4,243 CPU-seconds, against 4,254 at the slowest run before. The loop package took 635 and 621 seconds under race, inside its earlier 470 to 650. The cut in git processes stands where it was measured (one drain test, 1,107 to 830), but on CI it is below the difference between runners. The leg's remaining time is tracked in iss-2610181711260494.
+The grounds were shown wrong on the first two merge-group runs after both changes (#1099, #1100, 2026-10-18). The ubuntu leg's race step took 20.8 and 20.6 minutes, at the top of the 14 to 20.6 minutes it had taken before, not a minute shorter. Its package times summed to 4,293 and 4,243 CPU-seconds, against 4,254 at the slowest run before. The loop package took 635 and 621 seconds under race, inside its earlier 470 to 650. The cut in git processes stands where it was measured (one drain test, 1,107 to 830), but that one test overstated it: across the whole of the loop and claim packages a plain run made 126,334 git calls before and 119,327 after, 5.5% fewer, which on CI is below the difference between runners. The longer lever is iss-2610181757540736. The leg's remaining time is tracked in iss-2610181711260494.

.abcd/work/issues/open/iss-2610181711260494-the-check-job-s-macos-leg-has-little.md → .abcd/work/issues/resolved/iss-2610181711260494-the-check-job-s-macos-leg-has-little.md

@@ -10,6 +10,12 @@ origin: researcher-authored
 production_mode: hand-written
 found_at: ".github/workflows/ci.yml"
 remedy: "Shorten the macOS race pass below about 40 minutes on a slow runner, starting from its slowest packages (internal/core/implement/loop, internal/surface/cli, internal/core/claim), or ask the technical facilitator to raise the macOS ceiling and warning with the spread as the evidence"
+resolution: "The macOS leg's ceiling is 70 minutes and its warning 60 (dcn-2610181756328249), with ten runs of 35.1 to 49.3 minutes as the evidence; a faster race pass is the longer lever, iss-2610181757540736"
+impact: internal
 ---
 
 The check job's macOS leg has little headroom before its 50-minute warning: across nine merge-group runs on 2026-10-18 it took 35.2, 36.5, 38.9, 42.3, 42.4, 43.0, 43.9, 49.4 and 44.8 minutes, and the run of #1099 came within 0.6 minutes of the warning. Nearly all of it is the one race-enabled pass over the whole module, 41.9 to 47.4 minutes in the last three; the plain-only guard step after it takes under a minute. The warning fires at 50 and the job ceiling is 60.
+
+## Grounds
+
+- pursued: a merge-group macOS leg stays below 60 minutes, so the warning fires only on a run slower than any seen; one past 60 would show the leg needs a faster pass, not another raise

.github/workflows/ci.yml

@@ -275,9 +275,14 @@ jobs:
     # runs of 23 to 30 minutes (iss-2609281514435020). The technical
     # facilitator raised it to 60 on 2026-10-15, when merge-queue macOS runs had
     # grown from 31-35 to 34-40 minutes (iss-2610142326511964), so on this leg
-    # the job's own ceiling, not the queue's, is the binding one. Raising it
-    # again is the technical facilitator's call, and the remedy for a leg near
-    # the cap is a faster suite. The ubuntu leg's ceiling is 30. The drain
+    # the job's own ceiling, not the queue's, is the binding one. The
+    # coordinating session raised it to 70 on 2026-10-18, on delegation from the
+    # technical facilitator (dcn-2610181756328249): ten merge-group runs took
+    # 35.1 to 49.3 minutes, one within a minute of the warning, on a leg whose
+    # race pass keeps its three test slots full, so its time is its total work
+    # and varies with the runner. Raising it again is the technical
+    # facilitator's call, and the remedy for a leg near the cap is a faster
+    # suite (iss-2610181711260494). The ubuntu leg's ceiling is 30. The drain
     # bundle (spc-2610161044461448) grew the loop package's race run until the
     # leg reached 31 minutes, and it spent from 2026-10-17 at 45
     # (dcn-2610171803056931). It came back to 30 when the leg's plain pass
@@ -292,7 +297,7 @@ jobs:
     # TestRaceLaneBudgetIsDeclaredAndFitsItsJob holds every leg at or below the
     # cap, and TestCheckJobWarnsBeforeTheQueueCap holds the warning step at the
     # end of the job below it.
-    timeout-minutes: ${{ matrix.os == 'macos-latest' && 60 || 30 }}
+    timeout-minutes: ${{ matrix.os == 'macos-latest' && 70 || 30 }}
     strategy:
       fail-fast: false
       matrix:
@@ -478,8 +483,8 @@ jobs:
 
       # The plain-only guards on the macOS leg (iss-2610180542030806): the same
       # command as the ubuntu leg's "Test (plain-only guards)", after the race
-      # pass. About a minute, on a leg that ran 36.5 of its 60 in the last
-      # merge-group run. TestEveryLegRunsEveryPlainOnlyGuard holds both
+      # pass, about a minute when it was added on a leg that ran 36.5 minutes.
+      # TestEveryLegRunsEveryPlainOnlyGuard holds both
       # steps to the list it derives from the test files.
       - name: Test (plain-only guards, macOS)
         if: matrix.os == 'macos-latest' && github.event_name != 'pull_request'
@@ -569,24 +574,26 @@ jobs:
 
       # Warn before the cap cancels (ruling Z's cancel policy, 2026-09-28). A
       # macOS check that runs past WARN_AFTER_MINUTES is ten minutes from the
-      # leg's 60-minute ceiling, and a cancellation there carries nothing but the
+      # leg's 70-minute ceiling, and a cancellation there carries nothing but the
       # cancellation. The rule for a cancelled check is to rerun it once; a
       # second cancellation for the same reason stops that pull request and
       # opens a speed lane, never another raise of the timeout. This warning
       # moves that signal ahead of the first cancellation: it is emitted at the
       # job's end, for a run that finished past the line, so a check that
-      # passed between 50 and 60 minutes says so before one is cancelled at
+      # passed between 60 and 70 minutes says so before one is cancelled at
       # the cap. It never fails the
       # job: `always()` runs it however the steps before it ended, and
       # `continue-on-error` holds even a broken script to a warning.
       # TestCheckJobWarnsBeforeTheQueueCap pins the threshold ten minutes below
-      # the leg's ceiling.
+      # the leg's ceiling, and CEILING_MINUTES, the cap the message names, to
+      # that ceiling.
       - name: Warn when the check nears the queue's cap
         if: always() && matrix.os == 'macos-latest' && github.event_name != 'pull_request'
         continue-on-error: true
         env:
           STARTED_AT: ${{ steps.started.outputs.epoch }}
-          WARN_AFTER_MINUTES: 50
+          WARN_AFTER_MINUTES: 60
+          CEILING_MINUTES: 70
         run: |
           case "${STARTED_AT:-}" in
             '' | *[!0-9]*)
@@ -598,7 +605,7 @@ jobs:
           minutes=$((elapsed / 60))
           echo "check (macos-latest) has run ${minutes}m $((elapsed % 60))s; the warning line is ${WARN_AFTER_MINUTES}m"
           if [ "$elapsed" -gt $((WARN_AFTER_MINUTES * 60)) ]; then
-            msg="check (macos-latest) ran ${minutes} minutes, past the ${WARN_AFTER_MINUTES}-minute warning line and close to the job's 60-minute cap. If it is cancelled at the cap, rerun it once; a second cancellation for the same reason stops the pull request: open a speed lane to make the suite faster, and do not raise the timeout."
+            msg="check (macos-latest) ran ${minutes} minutes, past the ${WARN_AFTER_MINUTES}-minute warning line and close to the job's ${CEILING_MINUTES}-minute cap. If it is cancelled at the cap, rerun it once; a second cancellation for the same reason stops the pull request: open a speed lane to make the suite faster, and do not raise the timeout."
             echo "::warning title=macOS check near the queue's cap::${msg}"
             printf '### macOS check near the merge queue cap\n\n%s\n' "$msg" >>"$GITHUB_STEP_SUMMARY"
           fi

internal/core/lint/racelanebudget_test.go

@@ -35,7 +35,7 @@ import (
 // records. Ruling Z (2026-09-28, the technical facilitator) raised the macOS
 // leg, and every other leg keeps 30, after a spell at 45 from 2026-10-17 (dcn-2610171803056931,
 // iss-2610171803053186); the job declares that as one matrix expression, which checkLegTimeouts reads leg by leg. The queue's cap is 90
-// minutes since 2026-10-11 and the macOS leg's is 60 since 2026-10-15, below
+// minutes since 2026-10-11 and the macOS leg's is 70 since 2026-10-18, below
 // it, so on that leg the job's own ceiling is the binding one. The step's -timeout still earns its place under that cap: it lifts the
 // ten-minute per-package default, so a slow package fails on the job's clock
 // rather than at ten minutes. Raising the queue's cap is an admin act on the
@@ -125,7 +125,7 @@ func TestRaceLaneBudgetIsDeclaredAndFitsItsJob(t *testing.T) {
 			}
 			if leg.os == "macos-latest" && leg.minutes != checkMacOSCeilingMinutes*time.Minute {
 				t.Errorf("%s: the macos-latest leg's timeout-minutes is %s; want %dm, the ceiling "+
-					"the technical facilitator set on 2026-10-15", where, leg.minutes, checkMacOSCeilingMinutes)
+					"set on 2026-10-18 (dcn-2610181756328249)", where, leg.minutes, checkMacOSCeilingMinutes)
 			}
 			// Ruling Z raised the macOS leg; every other leg keeps 30, after
 			// a spell at 45 from 2026-10-17 (dcn-2610171803056931).
@@ -153,10 +153,12 @@ func TestRaceLaneBudgetIsDeclaredAndFitsItsJob(t *testing.T) {
 }
 
 // checkMacOSCeilingMinutes is the macOS leg's timeout-minutes in ci.yml's check
-// job: 45 under ruling Z (2026-09-28), 60 since 2026-10-15, when merge-queue
-// macOS runs had grown from 31-35 to 34-40 minutes. It stays below the merge
+// job: 45 under ruling Z (2026-09-28); 60 from 2026-10-15, when merge-queue
+// macOS runs had grown from 31-35 to 34-40 minutes; 70 since 2026-10-18, when
+// ten merge-group runs took 35.1 to 49.3 minutes on a leg whose race pass keeps
+// its three test slots full (dcn-2610181756328249). It stays below the merge
 // queue's check response timeout, which the budget test reads from the mirror.
-const checkMacOSCeilingMinutes = 60
+const checkMacOSCeilingMinutes = 70
 
 // checkUbuntuCeilingMinutes is the check job's ceiling on every leg but
 // macOS: 30 under ruling Z; 45 from 2026-10-17, when the drain bundle
@@ -207,6 +209,17 @@ func TestCheckJobWarnsBeforeTheQueueCap(t *testing.T) {
 			"%d-minute ceiling (the cancel policy of ruling Z)",
 			where, checkWarnStep, warn, checkWarnAfterMinutes, checkWarnLeadMinutes, checkMacOSCeilingMinutes)
 	}
+	// The cap the message names comes from the step's own env, held to the
+	// leg's ceiling here, so the message cannot name a cap the job no longer
+	// has (dcn-2610181756328249).
+	c := regexp.MustCompile(`(?m)^\s+CEILING_MINUTES: "?(\d+)"?\s*$`).FindStringSubmatch(step)
+	if c == nil {
+		t.Errorf("%s step %q: no CEILING_MINUTES in its env; the message would name the cap from a "+
+			"number no test holds to the leg's ceiling", where, checkWarnStep)
+	} else if ceiling, _ := strconv.Atoi(c[1]); ceiling != checkMacOSCeilingMinutes {
+		t.Errorf("%s step %q: CEILING_MINUTES is %d; want %d, the macos-latest leg's ceiling",
+			where, checkWarnStep, ceiling, checkMacOSCeilingMinutes)
+	}
 	warnAfter := time.Duration(warn) * time.Minute
 	for _, leg := range checkLegTimeouts(t, where, job) {
 		if leg.os == "macos-latest" && warnAfter >= leg.minutes {
@@ -237,6 +250,10 @@ func TestCheckJobWarnsBeforeTheQueueCap(t *testing.T) {
 		t.Fatalf("%s step %q has no run: script", where, checkWarnStep)
 	}
 	run := step[at:]
+	if !strings.Contains(run, "${CEILING_MINUTES}-minute cap") {
+		t.Errorf("%s step %q: its script does not name the cap as ${CEILING_MINUTES}-minute cap; a "+
+			"cap written any other way can drift from the ceiling", where, checkWarnStep)
+	}
 	for _, want := range []string{"::warning", "GITHUB_STEP_SUMMARY", "rerun", "speed lane"} {
 		if !strings.Contains(run, want) {
 			t.Errorf("%s step %q: its script never mentions %q; the warning names the rerun-once "+