Repository navigation
perf(test): cut the local affected-test gate's shell suites to a budget #8757
Description
Refs: #3716
Problem
The local gate run on #8714 took about 15 minutes: 137 *.test.sh files plus the planning surface Python tests. Workers and ready run this gate on the developer's machine, so every local verify pays its wall clock.
#9017 did the Python half in CI. #10534, #10756 and #10773 did the shell runner half:
scripts/check-affected.sh --testsnow runs the selected suites in parallel.- The run has a 120 s budget.
- A suite over its share fails by name.
Parallelism alone cannot meet the budget. From 8 jobs up, a selection's wall time is its longest suite's: #10534's 48-suite selection took 274 s at 8 jobs on the dev host, bounded by surface-journey.test.sh. The slowest planning suites have these medians in scripts/suite-seconds.txt:
| Suite | Median |
|---|---|
surface-journey.test.sh |
290 s |
surface.test.sh |
195 s |
surface-ui-c.test.sh |
153 s |
Fix
Cut those suites' wall clock: drop sleeps and real-server or browser startups that a fake can replace, or split a suite so that its parts run in parallel.
Done when
-
scripts/check-affected.sh --testspasses a default--jobs(one less than the core count, at most 8, env override) toscripts/affected-tests.sh. - The affected-only shell-suite run has a configurable total budget, 120 s by default.
- A suite that exceeds its share (suite-seconds.txt median times a margin, or a flat default) fails with its name; a test proves it.
-
scripts/run-plugin-tests.shprints each suite's wall seconds to its log in every run, not only under--budget(today it writes them only to a scratch.secsfile,run-plugin-tests.sh:158). -
plugins/planning/surface/surface-journey.test.sh,surface.test.shandsurface-ui-c.test.sheach finish in under 120 s in the pull request's CI run, as those printed seconds show. No test case is dropped to get there.
Check after merge, on a host run (not a lane criterion): scripts/check-affected.sh --tests on a planning-only change finishes under the 120 s budget on the dev WSL host.
Related
#3716, #3932, #8714, #9017 (CI half), #9070 (CI leg re-measure), #10534, #10756, #10773
Activity
github-actions commentedon Oct 21, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
#8757 (comment)
This decision settles the open questions. What remains is one measurement of the journey suite's median from main's CI logs, which is host work. No new questions.
Automated triage, applied by the lanes App.
[-]perf(test): cut the local affected-test gate to a budget; planning surface Python suite takes 489 s[/-][+]perf(test): cut the local affected-test gate's shell suites to a budget[/+]kyle-sexton commentedon Oct 22, 2026
Re-scoped (owner decision, attended session 2026-10-22): #9017 delivered the CI half (pytest-xdist for planning/surface). The local gate runs no Python, so the open work is the serial
*.test.shsuites and the budget. Body updated; issue stays open.kyle-sexton commentedon Oct 25, 2026
Released for re-triage under the owner's 2026-10-24 rule: an issue stops for a person only for an intent or vision question, a security call, a host or attended run, or spend outside prepaid credit. The hold label came off and intake triage runs again now; it re-applies a hold, with a numbered question, when one of those applies.
To hold this issue again, add
needs-human(orstatus: needs-decision) and say why in a comment.72 remaining items
kyle-sexton commentedon Oct 31, 2026
kyle-sexton commentedon Oct 31, 2026
Host run. Measured
surface-journey.test.shfrom main's whole-suite runs after #11523, then cut the slow phase it showed.Median after #11523: 141 s. The eight whole-suite
pr-require-checksdispatch runs on main between 11:59Z and 14:49Z (41199079392, 41200659367, 41201359701, 41202468872, 41204767979, 41206938075, 41209305383, 41211887017) printedTIME:137, 139, 140, 141, 141, 141, 142 and 144 s.Next cut: in each run's per-phase lines, phase 13 took 34 s, and every other phase took 2 to 12 s. Phase 13's two best-effort
page.click("#flyClose").catch(() => {})calls (tests/ui_journey.js:679and:683) had no timeout, and the script sets no default. A click on a closed flyout therefore waited Playwright's 30 s action default before the catch swallowed it.#11592 (merged,
c40d2c8) gives both clicks{timeout: 2000}. It also records the 141 s median inscripts/suite-seconds.txtand removes the suite's 480 s row fromscripts/suite-budget-exceptions.txt, since the median is under 150 s. In that PR's own CI run (test-bash job 124579564746), phase 13 took 5 s and the suite took 82 s. That is one run, not a median.scripts/suite-seconds.txtcarries a median under 120 s: not yet. The recorded median is 141 s, from runs before the cut. The re-measure needs main's whole-suite runs afterc40d2c8. If their median is under 120 s, it goes intosuite-seconds.txtand this issue closes.suite-budget-exceptions.txtrow is removed (median under 150 s).Target labels: add none; remove none.
kyle-sexton commentedon Nov 1, 2026
Re-measure after the phase 13 cut (#11592, merged 15:44:36Z). Eight whole-suite
pr-require-checksdispatch runs on main after it printedsurface-journey.test.shTIME:values of 112, 131, 131, 134, 135, 136, 137 and 142 s, a median of 134.5 s. The first eight successful runs listed after the merge are in.workon the laptop.The cut held in the page phases. The suite total barely moved. In run 41241452557 (142 s), phase 13 took 7 s, down from 34 s, and the 20 page phases sum to about 98 s, down from about 118 s. The rest of
TIMEgrew from about 24 s to about 44 s. That remainder is the shell's work between phases (server start,round.shapply and handle calls, the restart in phases 18 to 20) plus waiting for one ofrun-plugin-tests.sh's three parallel slots, whichTIMEcounts from the moment the suite could start (scripts/suite-seconds.txtheader). The PR's own CI leg ran on a quieter runner and printed 82 s, so one leg is not a median.Next cut. Measure the shell's share before cutting it. The page script already prints its own seconds per phase. The shell steps between phases (
surface-checks.shjourney block,jrun,japply,jhandle, and thesleep 1before phase 17) print nothing, so the 44 s cannot be divided yet. Add one wall-seconds line per shell step in that block, then cut the largest. The fixedsleep 1, and any wait for a state frame the page already confirms throughsettle, are the first candidates. Separately,scripts/suite-seconds.txtkeeps 141 s until a sub-120 s median replaces it: 134.5 s is within the noise of the leg balance it feeds.Target labels: add none; remove none.
fix(test): read the journey clock unset-safe so bash 4 falls back to ?
kyle-sexton commentedon Nov 1, 2026
Lane 5 handing this to the desktop HITL lane, as the hub directed. The per-shell-step timing PR #11768 merged as 312a9f2. I'm releasing my lease on this issue.
Next step: read the
ui_journey.shell:and phase lines from main's whole-suitepr-require-checksdispatch runs after 312a9f2, cut the step with the largest share (thesleep 1before phase 17 is the first candidate), then re-measure the median. Done when: main's whole-suite median forsurface-journey.test.shis under 120 s. The last median was 134.5 s, measured after #11592.kyle-sexton commentedon Nov 1, 2026
kyle-sexton commentedon Nov 1, 2026
Claimed by host-run Lane 5 (melo-lap-001 WSL), 2026-11-01T04:36Z: no newer claim followed my 03:09Z hand-off, so I'm taking the next cut. Next: read the
ui_journey.shell:lines from main's whole-suitepr-require-checksruns since 312a9f2, cut the largest share, and re-measure until the median is under 120 s.fix(planning): keep the journey's one-second fallback on bash without…
kyle-sexton commentedon Nov 1, 2026
Lane 5 re-measure after #11847 (merged 6b9b267, 06:49Z). The 4 whole-suite
pr-require-checksruns on main since then:surface-journey.test.sh127, 126, 101 and 129 s, median 126.5 s. Before #11847: 129, 111, 99 and 135 s, median 120.0 s.#11847 saved about 6 to 10 s per run locally, but the runs swing by up to 36 s, so 4 runs can't show the saving either way. Per-phase CI lines after the merge look much like before (phase 1 still 11-12 s). The cut didn't show above the noise, and the issue stays open.
Next:
playwright-cli run-codeprocess, about 1.3 s each, about 27 s in all (measured locally in the perf(planning): cut the journey suite's shell and page waits #11847 work). Fewer phases, with the shell's writes moved between fewer runs, is the next change.Lease held by Lane 5.
kyle-sexton commentedon Nov 1, 2026
Done check met:
surface-journey.test.shmedian 113.5 s over 8 main whole-suitepr-require-checksdispatch runs created after #11943 merged (bbdfcce, 14:46:19Z), target < 120 s. Noui_journeyFAIL ornot okline in any run.Before #11847 and #11943 the median was 126.5 s over 4 runs. The spread is still 85 to 120 s, so a single run near 120 s is not a regression. Closing as completed.