ci: cut the race steps back under their job limits with test-only seams

.abcd/work/issues/open/iss-2610150500162972-ubuntu-check-leg-cancelled-at-its-30.md

@@ -0,0 +1,15 @@
+---
+schema_version: 1
+id: "iss-2610150500162972"
+slug: "ubuntu-check-leg-cancelled-at-its-30"
+severity: "major"
+category: "process"
+source: "agent-observation"
+found_during: "the cancel policy's speed lane for the ubuntu check leg, 2026-10-15"
+origin: researcher-authored
+production_mode: hand-written
+found_at: ".github/workflows/ci.yml"
+remedy: "Cut the ubuntu check job's wall time by at least five minutes without raising its timeout or changing a required check or the step split: name the slowest packages first in both test steps, share the live-repository fixtures the heaviest race tests rebuild, and make the run lock's retry delay injectable for the take-back test, then confirm from the job's step times on the merge queue."
+---
+
+The ubuntu leg of ci.yml's check job (timeout-minutes 30) now runs at its limit for every pull request, and the cancel policy of 2026-09-28 has fired: a check cancelled at the cap twice for the same reason stops that pull request and opens a speed lane. PR #1003 (feat/memory-recall-notes) was cancelled at 30 minutes twice: run 38676280619 attempt 2 spent 03:58:37-04:06:22 in Test (about 8 minutes) and was still in Test (race, internal) from 04:06:22 when it was cancelled at 04:27:48. PR #1008 passed in 25m40 only on a rerun after its first attempt was cancelled at 30m05, and the merge-queue entry for PR #976 (run 38680018438) was cancelled at 30 minutes too, with internal/core/implement/loop still running. On passing runs the race step took 16-18 minutes and the plain step 6-7; the race step's package times sum to about 3,500-3,750 seconds on the four-core runner, so it is bound by total CPU rather than by one package, and its heaviest packages are internal/core/lifeboat (578-594s), internal/core/reading (547-557s), internal/core/launch (357-365s) and internal/surface/cli (333-367s). iss-2610142326511964 covers the macOS leg's slowdown; this is the ubuntu leg. <!-- record-lint: forward-looking -->

.github/workflows/ci.yml

@@ -375,11 +375,18 @@ jobs:
       # ones, and the plain pass reports a failure minutes before the race pass
       # would. The macOS leg runs one race-enabled pass over the whole module
       # instead (the last step of this group).
+      #
+      # The two packages named ahead of the pattern are the slowest in this
+      # pass (126-187s and 110-147s on the four-core runner, 2026-10-15). In
+      # import-path order internal/surface/cli started about three and a half
+      # minutes into the step and finished it alone; named first, both start
+      # with the step. The set is still exactly ./... (go test runs a package
+      # named twice once), which TestPlainTestStepRunsTheWholeModule pins.
       - name: Test
         if: matrix.os == 'ubuntu-latest'
         env:
           ABCD_TEST_SKIP_FLUSH: "1"
-        run: go test ./...
+        run: go test ./internal/core/implement/loop ./internal/surface/cli ./...
 
       # Only a source change can introduce a data race, and the plain lane above
       # has already run the same tests over the record.
@@ -402,11 +409,15 @@ jobs:
       # in import-path order internal/surface/cli started among the last and ran
       # alone as the step's tail. go test runs a package named twice once, so the
       # set is still exactly the pattern's; the same test pins that with go list.
+      # internal/core/implement/loop and internal/core/interview joined the list
+      # on 2026-10-15: 172-244s and 160s under -race on the ubuntu runner, they
+      # started last in import-path order and were still running when two
+      # checks were cancelled at the leg's 30-minute ceiling.
       - name: Test (race, internal)
         if: matrix.os == 'ubuntu-latest' && needs.changes.outputs.inert != 'true'
         env:
           ABCD_TEST_SKIP_FLUSH: "1"
-        run: go test -race -timeout 20m ./internal/surface/cli ./internal/core/reading ./internal/core/launch ./internal/core/lifeboat ./internal/core/lint ./internal/adapter/scanner ./internal/core/capture ./internal/core/ahoy ./internal/core/site ./internal/...
+        run: go test -race -timeout 20m ./internal/surface/cli ./internal/core/reading ./internal/core/launch ./internal/core/lifeboat ./internal/core/lint ./internal/adapter/scanner ./internal/core/implement/loop ./internal/core/interview ./internal/core/capture ./internal/core/ahoy ./internal/core/site ./internal/...
 
       # The macOS leg's one test pass (iss-2610110938293273): race-enabled, over
       # the whole module, with the same slowest-first order and the same
@@ -422,7 +433,7 @@ jobs:
         if: matrix.os == 'macos-latest' && github.event_name != 'pull_request'
         env:
           ABCD_TEST_SKIP_FLUSH: "1"
-        run: go test -race -timeout 20m ./internal/surface/cli ./internal/core/reading ./internal/core/launch ./internal/core/lifeboat ./internal/core/lint ./internal/adapter/scanner ./internal/core/capture ./internal/core/ahoy ./internal/core/site ./...
+        run: go test -race -timeout 20m ./internal/surface/cli ./internal/core/reading ./internal/core/launch ./internal/core/lifeboat ./internal/core/lint ./internal/adapter/scanner ./internal/core/implement/loop ./internal/core/interview ./internal/core/capture ./internal/core/ahoy ./internal/core/site ./...
 
       # Drift gate for the .abcd/development design record. Blocking: a
       # blocker-severity finding exits non-zero and fails the job; warn-level

.github/workflows/release.yml

@@ -139,7 +139,7 @@ jobs:
         run: go test ./...
 
       - name: Test (race, internal)
-        run: go test -race -timeout 20m ./internal/surface/cli ./internal/core/reading ./internal/core/launch ./internal/core/lifeboat ./internal/core/lint ./internal/adapter/scanner ./internal/core/capture ./internal/core/ahoy ./internal/core/site ./internal/...
+        run: go test -race -timeout 20m ./internal/surface/cli ./internal/core/reading ./internal/core/launch ./internal/core/lifeboat ./internal/core/lint ./internal/adapter/scanner ./internal/core/implement/loop ./internal/core/interview ./internal/core/capture ./internal/core/ahoy ./internal/core/site ./internal/...
 
       - name: Record-lint (design-record drift gate)
         run: go run ./cmd/record-lint

Makefile

@@ -344,7 +344,7 @@ preflight: load-check fmt-check lint-reviews lint-issues lint-decisions record-l
 	p=$$(go run ./cmd/test-parallelism) && \
 	go test -p "$$p" ./...
 	p=$$(go run ./cmd/test-parallelism) && \
-	go test -race -p "$$p" -timeout 20m ./internal/surface/cli ./internal/core/reading ./internal/core/launch ./internal/core/lifeboat ./internal/core/lint ./internal/adapter/scanner ./internal/core/capture ./internal/core/ahoy ./internal/core/site ./internal/...
+	go test -race -p "$$p" -timeout 20m ./internal/surface/cli ./internal/core/reading ./internal/core/launch ./internal/core/lifeboat ./internal/core/lint ./internal/adapter/scanner ./internal/core/implement/loop ./internal/core/interview ./internal/core/capture ./internal/core/ahoy ./internal/core/site ./internal/...
 	@scripts/preflight-receipt.sh mint "$(PREFLIGHT_BEGAN)"
 
 # The push receipt (iss-2608290810036869, iss-2608210738378295). The pre-push hook

internal/core/implement/loop/locktimeout_test.go

@@ -0,0 +1,18 @@
+package loop
+
+import (
+	"testing"
+	"time"
+)
+
+// shortLockTimeout shortens how long a mutation waits for the run's lock
+// before it reports contention, for the rest of t. A test that holds a lock
+// to see the waiter back off proves the backoff, not the length of the wait,
+// and at the shipped three seconds a waiter that retries spent most of its
+// test asleep.
+func shortLockTimeout(t *testing.T) {
+	t.Helper()
+	was := lockTimeout
+	lockTimeout = 100 * time.Millisecond
+	t.Cleanup(func() { lockTimeout = was })
+}

internal/core/implement/loop/nextseries_test.go

@@ -363,6 +363,7 @@ func TestARemovedPickIsPickedPast(t *testing.T) {
 // next.lock within the lock timeout is refused as contention (exit 3 at the
 // front door), at the pick, and starts nothing.
 func TestASeriesMoveBacksOffWhileAnotherHoldsItsLock(t *testing.T) {
+	shortLockTimeout(t)
 	root := seriesRepo(t)
 	first, err := Next(root, Options{}, NextOptions{Max: 2})
 	if err != nil {

internal/core/implement/loop/state.go

@@ -211,8 +211,10 @@ const (
 const maxStateBytes = 4 << 20
 
 // lockTimeout bounds how long a mutation waits for another invocation's
-// mutation to finish before it reports contention.
-const lockTimeout = 3 * time.Second
+// mutation to finish before it reports contention. It is a variable so a test
+// that holds the lock to see a waiter back off can shorten the wait
+// (shortLockTimeout); nothing else assigns it.
+var lockTimeout = 3 * time.Second
 
 // runIDRe is the shape of a run id, checked before any path is built from one.
 var runIDRe = regexp.MustCompile(`^run-[0-9]{16}$`)

internal/core/interview/main_test.go

@@ -22,10 +22,12 @@ import (
 	"errors"
 	"fmt"
 	"os"
+	"os/exec"
 	"path/filepath"
 	"slices"
 	"strconv"
 	"strings"
+	"sync"
 	"testing"
 	"time"
 )
@@ -49,7 +51,65 @@ func TestMain(m *testing.M) {
 	if dir := os.Getenv(stubScriptEnv); dir != "" && slices.Contains(os.Args, "--print") {
 		os.Exit(stubRunner(dir))
 	}
-	os.Exit(m.Run())
+	plainStub.env = os.Environ()
+	plainStub.goCmd, _ = exec.LookPath("go")
+	code := m.Run()
+	if plainStub.dir != "" {
+		_ = os.RemoveAll(plainStub.dir)
+	}
+	os.Exit(code)
+}
+
+// plainStub is this package's test binary built without the race detector,
+// made once per run, for stubBinary under -race; TestMain removes it. It is
+// built in the environment the run started with, never one a test has set
+// (a sandboxed HOME would point the go command at an empty cache).
+var plainStub struct {
+	once  sync.Once
+	env   []string
+	goCmd string
+	dir   string
+	path  string
+	err   error
+}
+
+// stubBinary returns the binary that plays the harness: this test binary, or
+// under -race the same package's test binary built without the detector. Both
+// run the same TestMain, so the stub's behaviour is the same; only its start
+// is cheaper, which every turn of every stubbed interview pays. A build that
+// fails leaves the instrumented binary in its place, which costs time and
+// changes nothing else.
+func stubBinary(t *testing.T) string {
+	t.Helper()
+	self, err := os.Executable()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !raceEnabled {
+		return self
+	}
+	plainStub.once.Do(func() {
+		if plainStub.goCmd == "" {
+			plainStub.err = errors.New("no go command on the PATH the run started with")
+			return
+		}
+		if plainStub.dir, plainStub.err = os.MkdirTemp("", "abcd-interview-stub-"); plainStub.err != nil {
+			return
+		}
+		out := filepath.Join(plainStub.dir, "interview.test")
+		cmd := exec.Command(plainStub.goCmd, "test", "-c", "-o", out, ".")
+		cmd.Env = plainStub.env
+		if b, err := cmd.CombinedOutput(); err != nil {
+			plainStub.err = fmt.Errorf("go test -c: %v: %s", err, b)
+			return
+		}
+		plainStub.path = out
+	})
+	if plainStub.err != nil {
+		t.Logf("the stub runs instrumented: %v", plainStub.err)
+		return self
+	}
+	return plainStub.path
 }
 
 func stubRunner(dir string) int {
@@ -132,10 +192,7 @@ func stubRunner(dir string) int {
 // script's directory.
 func stubOnPath(t *testing.T, receipts ...string) string {
 	t.Helper()
-	self, err := os.Executable()
-	if err != nil {
-		t.Fatal(err)
-	}
+	self := stubBinary(t)
 	bin, script := t.TempDir(), t.TempDir()
 	if err := os.Symlink(self, filepath.Join(bin, "claude")); err != nil {
 		t.Fatal(err)

internal/core/interview/race_off_test.go

@@ -0,0 +1,10 @@
+//go:build !race
+
+package interview
+
+// raceEnabled reports whether the race detector is instrumenting this build.
+// stubBinary reads it: under -race the stub runner is this package's test
+// binary built without the detector, since every start of an instrumented
+// binary pays the detector's set-up (about a second on a developer Mac) and
+// the stub is the harness's stand-in, not the code under test.
+const raceEnabled = false

internal/core/interview/race_on_test.go

@@ -0,0 +1,7 @@
+//go:build race
+
+package interview
+
+// raceEnabled reports whether the race detector is instrumenting this build.
+// See the sibling file for what reads it.
+const raceEnabled = true

internal/core/launch/dryrun.go

@@ -205,7 +205,9 @@ func DryRun(req DryRunRequest) (DryRunReport, error) {
 }
 
 // scanBundle adapts the bundle's Included files to the scanner and runs it.
-func scanBundle(repoRoot string, bundle Bundle) scanner.ScanResult {
+// It is a variable so a test can keep the scan a render of this repository
+// made, rather than scan the same payload a second time.
+var scanBundle = func(repoRoot string, bundle Bundle) scanner.ScanResult {
 	sc, err := scanner.New(repoRoot)
 	if err != nil {
 		return scanner.ScanResult{Unavailable: true, UnavailableReason: err.Error()}

internal/core/launch/installsurface_test.go

@@ -1,10 +1,8 @@
 package launch
 
 import (
-	"path/filepath"
 	"reflect"
 	"testing"
-	"time"
 )
 
 // bundleTreeFor resolves root's payload the way a ship would and returns the
@@ -263,13 +261,7 @@ func TestResolveInstallSurfaceCommittedPayload(t *testing.T) {
 // prevent.
 func TestPayloadTreeImplementationsResolveIdentically(t *testing.T) {
 	root := repoRootForTest(t)
-	dest := filepath.Join(t.TempDir(), "payload")
-	if _, err := RenderPayload(PayloadRenderRequest{
-		RepoRoot: root, Dest: dest, Version: "9.9.9",
-		Entry: ChangelogEntry{Tier: "patch", Reason: "seam parity", Date: time.Now(), SourceSHA: "deadbeef"}, Dirty: DirtySkip,
-	}); err != nil {
-		t.Fatalf("render the payload: %v", err)
-	}
+	dest, _ := renderThisRepo(t)
 
 	fromBundle, err := ResolveInstallSurface(bundleTreeFor(t, root))
 	if err != nil {

internal/core/launch/render_test.go

@@ -93,24 +93,9 @@ func TestRenderPayloadSatisfiesPublicLockstep(t *testing.T) {
 // working tree would pass every other test here and silently break the premise.
 func TestRenderPayloadLeavesSourceTreeUnversioned(t *testing.T) {
 	root := repoRootForTest(t)
-	manifests := []string{".claude-plugin/plugin.json", ".claude-plugin/marketplace.json"}
-	before := make(map[string][]byte, len(manifests))
-	for _, rel := range manifests {
-		data, err := os.ReadFile(filepath.Join(root, rel))
-		if err != nil {
-			t.Fatalf("read %s: %v", rel, err)
-		}
-		before[rel] = data
-	}
-
-	dest := filepath.Join(t.TempDir(), "payload")
-	if _, err := RenderPayload(PayloadRenderRequest{
-		RepoRoot: root, Dest: dest, Version: "9.9.9", Entry: sampleEntry(), Dirty: DirtySkip,
-	}); err != nil {
-		t.Fatalf("render this repository's payload: %v", err)
-	}
+	_, before := renderThisRepo(t)
 
-	for _, rel := range manifests {
+	for _, rel := range thisRepoManifests {
 		after, err := os.ReadFile(filepath.Join(root, rel))
 		if err != nil {
 			t.Fatalf("re-read %s: %v", rel, err)

internal/core/launch/scaffold/templates/release.yml.tmpl

@@ -230,7 +230,7 @@ jobs:
 <%- if .Abcd %>
 
       - name: Test (race, internal)
-        run: go test -race -timeout 20m ./internal/surface/cli ./internal/core/reading ./internal/core/launch ./internal/core/lifeboat ./internal/core/lint ./internal/adapter/scanner ./internal/core/capture ./internal/core/ahoy ./internal/core/site ./internal/...
+        run: go test -race -timeout 20m ./internal/surface/cli ./internal/core/reading ./internal/core/launch ./internal/core/lifeboat ./internal/core/lint ./internal/adapter/scanner ./internal/core/implement/loop ./internal/core/interview ./internal/core/capture ./internal/core/ahoy ./internal/core/site ./internal/...
 <%- else %>
 
       # A generic module may have no internal tree, so the race leg runs over the

internal/core/launch/scan_coverage_test.go

@@ -139,12 +139,9 @@ func TestBinaryPayloadSecretRefuses(t *testing.T) {
 // deliberately not asserted here — an environmental identity match on a text
 // file (iss-2608291444328326) is a different rule and a different file class.
 func TestRepoPayloadBinariesScanClean(t *testing.T) {
-	root := repoRootForTest(t)
-	bundle, err := ResolveBundle(root, nil)
-	if err != nil {
-		t.Fatalf("resolve the payload bundle: %v", err)
-	}
-	scan := scanBundle(root, bundle)
+	// The scan the render of this repository's payload made: the same scanner
+	// over the same bundle (ResolveBundle(root, nil)) a direct call reads.
+	scan := scanThisRepo(t)
 	if scan.Unavailable {
 		t.Fatalf("scanner unavailable on the repo's own payload: %s", scan.UnavailableReason)
 	}