You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
iOS smoke lane: preflight 'prepare ios-runner' fails job with daemon_startup_failed on PR lanes (untracked third class) #3342
ios.ymlSmoke Tests fails at the step "Preflight iOS runner through public CLI" (pnpm clean:daemon + node src/bin.ts prepare ios-runner …) before any scenario runs, with the daemon client's typed details.kind: "daemon_startup_failed":
This is a third red class in the iOS smoke lane beyond the two in #3337, and it is exactly the "red that means nothing" problem #2491 / #3336 are about: the step fails the whole job while the diff cannot have caused it.
Evidence (PR-lane only; no failing main run observed)
main shows no failing ios.yml runs on 2026-10-08, so this is currently PR-lane-only evidence.
Both payloads show stateDir/lockPath/infoPath inside the workspace .tmp/agent-device-state, startupTimeoutMs: 15000, startupAttempts: 1, and the client hint pointing at stale lock/daemon metadata from a previous owner in that state directory.
Scope
Find the root owner: whether the preflight's clean:daemon can leave (or collide with) lock/daemon.json state that makes the next start unrecoverable within the 15 s one-attempt budget on shared CI hosts, or whether the startup probe itself is under-budgeted under host load. Note the failing runs overlapped PR #3330, which is actively changing daemon startup on Windows (#3291) — rule that interaction in or out before assuming pure environment flake. Not requested here: re-running lanes to mask it — the re-run of 37839500279 is recorded on #3336.
Update: the class did not reproduce on the fixed head. Job 113540191797 @ 98321a279 ran all 24 steps with Preflight iOS runner through public CLI = SUCCESS, followed by both replay steps SUCCESS; with the three other Smoke lanes SUCCESS on that head, that is four independent preflight passes on one head. The class therefore stands on its three cross-branch instances, all confirmed from attempt-1 job data:
Same ~1 h window, three unrelated branches, none touching daemon startup. A re-run of #3336's failed job at 98321a279 is queued to strengthen the non-reproduction sample; outcome will be appended here.
Re-run outcome: 37843633972 @ 98321a2 passed fully — the preflight step is now green on two consecutive full jobs on that head (the failing-attempt-1 job and this re-run), alongside the three cross-branch instances from the 19:38–20:32Z window. Non-reproduction sample strengthened; class remains open until its root owner is found.
Frequency table across branches tonight (coordinator consolidation, 2026-10-08)
Four instances of this class on four unrelated diffs inside 1h46m, all on macos-26 where ios.yml groups concurrency only per-ref (:57-65), so two iOS jobs can share a runner. Recorded so no future PR re-derives it:
job 113543477378, same step, same typed daemon_startup_failed
None of these four branches touch the daemon-startup code path this issue is about — #3331's diff is Android-only, #3340's is selector/docs, #3336's is seven test/ files with zero product code — and the simulator UDID these jobs used (5FEB61C5-...) is not pinned anywhere in the repo, i.e. host-assigned.
Two cancelled jobs that are NOT instances
Counting Smoke Tests (cancelled) wakes would double this table. Both have zero steps and a sub-minute lifetime, i.e. supersede/cancel housekeeping, not a failed run:
No assertion change was involved in any of those greens; the step simply got a healthy daemon. Four failures across four unrelated diffs, with the same step green on re-run, reads as runner/host contention on the shared macOS pool rather than a startup regression. Fourth instance (#3340) is new relative to the #3336 thread's three-instance record. Curated by #3336's worker; consolidated here by the coordinating thread. No action asked of anyone.
job 113537548137, step Preflight iOS runner through public CLI, details.kind: daemon_startup_failed
So: five instances, five unrelated diffs, 19:38Z → 21:24Z (1h46m).
The new detail is host identity. 5FEB61C5-E3F5-471D-AE23-8A07438D5E92 appears in the failed-step log of this fifth job, and re-reading the older jobs confirms the same UDID in job 113543477378 (row 4) and job 113524981450 (row 3). That is three of five confirmed on one simulator identity; rows 1 and 2 are listed as instances from their step/kind evidence only, because their job logs are no longer retrievable through the API path I used — I am not extending the UDID claim to them. Even 3-of-5 is the corroborating signal: unrelated diffs, one host-assigned device, consistent with contention or stale device state on a runner rather than a product regression.
Diff-scope argument for this fifth one, recorded so it does not have to be re-derived: #3341's ten files are all under packages/maestro/, and the daemon reaches Maestro only lazily — src/daemon/replay-device-selection.ts:52 and src/commands/replay/script-source-bundle.ts:24 both use await import('@agent-device/maestro'). There is no static edge, so nothing in that PR is evaluated when the daemon boots, which is the phase this step exercises. Its Size lane independently reports CLI startup medians faster than base (--version −0.6 ms, --help −1.5 ms).
One housekeeping note for whoever triages this: a GitHub Checks wake can re-fire for an already-recorded failure when the PR head moves, without any new run existing. run 37843216989 is still run_attempt: 1, completed/failure, updated_at: 21:11:22Z — the "new" failure signal at ~21:53Z was that same long-settled job. Reading run_attempt and updated_at separates a genuine re-failure from a re-announce, which matters here because the action for a first instance and a repeat at the same head are different decisions.
That is the cleanest causal evidence available for this class, and it is now on two heads (#3336 at 98321a279 earlier, #3341 at 29641150c here): same commit, same runner assignment, same host-assigned simulator, opposite outcome, with a mergeable diff sitting unchanged between the two runs. No property of the diff can produce that.
For the triage decision this implies: nothing on any of these branches needs a code change, and a re-run is the correct first response, not a signal to widen a retry policy or gate a lane. Both affected PR authors recorded their re-run on the PR rather than redesigning, which is the outcome worth preserving.
Cross-row tally from the coordinator tonight: the class has hit five unrelated PRs; two went green on re-run at an identical head (#3336 at 98321a2, artifact-verified zero absorbed re-issues — see #3336 (comment) — and #3341 at 2964115). Non-reproduction-on-rerun is now the class signature, which strengthens the environment/daemon-state-dir root-cause hypothesis over any branch-specific cause.
Correction to my UDID claim: 5FEB61C5 is not corroborating evidence
I posted earlier that five failures across five branches sharing simulator UDID 5FEB61C5-E3F5-471D-AE23-8A07438D5E92 was "consistent with contention or stale device state." I checked that claim tonight and it does not hold, so I am withdrawing it rather than leaving it in a table whose purpose is to stop people re-deriving this.
The same UDID appears in the log of a fully green iOS job on an unrelated branch: job 113564812060 (run 37851061077, fix/macos-background-activation-3254 @ 74f3899f0), all steps successful. 5FEB61C5-… is therefore simply the simulator that this runner image uses for every iOS job, success or failure. It identifies the runner, not a degraded device, and it cannot discriminate a flaky run from a healthy one. My 3-of-5 wording inflated a constant into a signal.
What survives the check, and why I still consider this a lane condition rather than five product bugs:
Cross-branch frequency with no shared code path: five instances, five unrelated diffs (Windows host-kit, Android IME, seven test/ files, selector/docs, Maestro-only), all dying at the same preflight step before any scenario runs.
The step is a daemon-boot probe, and three of those diffs provably cannot reach daemon startup (lazily-loaded Maestro, test-only files, selector/docs).
That is still enough to justify "re-run first, don't redesign," which is the operational point of this issue. It is not enough to say anything about which device or why the boot probe times out, and this issue should not be cited for host identity. Whoever root-causes daemon_startup_failed still needs the runner-side evidence: ios.yml groups concurrency only per-ref (:57-65), so two iOS jobs can share one macos-26 runner, and the UDID is unpinned in the repo.
One more lane data point, explicitly a different class so it is not folded into this table: job 113585608927 (run 37857656021, fix/android-shutdown-ime-flush-window @ fcf968b22, 23:07:54Z → 23:17:59Z) failed at Run fixture-backed iOS simulator E2E smoke, not at the preflight step: smoke:automation-input, step "wait for the deep-link destination (5/5)", typed reason: wait_capture_stalled, retriable: true. That is the transport/observation-wait class already attributed under #2491, whose re-issue policy is exactly what #3336 adds — and that diff is Android-only (7 files: packages/platform-android/**, one contracts type, one application-tools file), so it cannot reach an iOS deep-link wait. Noted here only because it is the same shared host in the same hour, not because the mechanisms are related.
job 113596105913, step Preflight iOS runner through public CLI, details.kind: daemon_startup_failed
Worth recording because this instance is the one where the "cannot be the diff" argument needed real work rather than a file-list glance. That PR now touches src/platform-runtime-android-application-tools.ts, which is statically imported by src/platform-runtime-operation-host.ts:35 — i.e. it sits on a daemon-reachable path, unlike the other five diffs. The ruling still holds, but only after checking the specific mechanism that would make it false: every import line added anywhere in that diff lives in a .test.ts file or under packages/platform-android/**, and the src/ change adds no new static import, so the daemon's eager closure is unchanged. Repo Guards green; the eager-closure budget lives in the Coverage lane.
Operationally for triage: six instances, five unrelated diffs, 19:38Z → 23:42Z (4h04m) on one step that runs before any scenario. Two of them went green on re-run at an identical head. The one property I have not been able to establish is host identity — see my correction above: the simulator UDID is constant across green and red jobs and identifies the runner, not a degraded device. So whoever owns #3342 still needs runner-side evidence, and ios.yml grouping concurrency only per-ref (:57-65) remains the leading suspect for two iOS jobs sharing one macos-26 runner.
Failed step: Preflight iOS runner through public CLI (job 113616892498, run 37867200258 attempt 1; started 00:56:28Z, failed 01:01:28Z, 1 of 23 steps). Payload is this issue's canonical signature verbatim:
"code": "COMMAND_FAILED"
"message": "Failed to start daemon"
Nothing new mechanically — same seam, same message, seventh occurrence tonight across seven unrelated diffs (Android IME flush window ×3, iOS fill normalization, is-ambiguity, macOS background activation, iOS smoke-lane reliability). Two things this instance adds that are worth having on the record:
cancelled in a wake is not a second failure. The GitHub checks API reported this lane as cancelled for attempt 2 with 0 failed steps in 23, because attempt 2 was superseded while attempt 3 was being queued. Anyone reading commits/{sha}/check-runs without run_attempt will see "Smoke Tests: cancelled" and conclude either a new class or a repeat failure. It is neither: attempt 1 failed with the payload above, attempt 2 was housekeeping. Per-head failure counts have to be taken from runs/{id}/attempts/N, not from the check-run roll-up — which is also why I count one failure at this head, not two.
The diff cannot be the cause and this row's maintainer has been told not to chase it.fix(android): keep close --shutdown's IME restore out of the settings flush window #3331 is packages/platform-android/** plus doc comments; it touches no daemon-boot or Apple path, and main has been green on iOS all night. I have explicitly instructed that agent to re-run rather than widen any timeout or add a retry, since a per-PR widening to paper over this is suppression, not a fix. Six prior instances all went green on re-run.
Cost accrual, for whoever owns this: tonight this class has consumed at least seven smoke-lane jobs on PRs that were otherwise green, and it is currently the single reason my supervised PRs show red boards. Three distinct preflight/prepare signatures now hit that same first step (daemon_startup_failed, the 15 s xcrun --show-sdk-version probe timeout recorded under #2491, and this one), which is why I keep the classes separate rather than filing them together.
Instance 8 — 2026-10-09T03:23:40Z, head a7821ef58 of #3330 (native Windows daemon startup), iOS Smoke lane, step Preflight iOS runner through public CLI:
Attribution pending — deliberately not counted in the flake tally yet. Unlike the seven instances above, which rode unrelated diffs, this PR is not a bystander to daemon startup: it owns src/daemon-registration-owner.ts, and its newest commit rewrote parseProcessTableRows, the shared row walk the macOS POSIX ps path also uses, plus the owner-liveness rules the startup owner probe consumes on posix. Lane history on that PR: green at bad805c21 and 16ff5d83b, first red at a7821ef58.
A re-run was triggered at 03:25Z. Green ⇒ counts as instance 8 of this class. Red at the same head with the same signature ⇒ it is a #3330 defect, not this flake, and it will be removed from this tally. Will update either way.
Update on instance 8 — not counted in this class's tally; attribution held against it.
Re-ran the failed job at the same head. It failed again, attempt 2 (job/113655150013, 03:31:52Z), identical signature. Two same-head failures is my threshold for stopping calling something a flake, so #3330's iOS Smoke is currently excluded from this issue's baseline rather than added to it.
Two things worth recording here for whoever triages this issue:
1. The shape of these two failures is a readiness timeout, not an owner-liveness miss. Both attempts report daemonProcess spawned ~startupTimeoutMs before the failure (pid 29551 @ 03:31:37, failed 03:31:52), cleanupResults: retired / exited / graceful, metadataState: hasInfo:false, hasLock:false, and zeroregistration busy / not-running lines in the entire log. That is "the process came up and never published a reachable daemon.json inside 15 s", which may be a different sub-class than the instances where registration contention is visible.
2. My earlier bisect evidence was weaker than I stated. I wrote that #3330 was "green at bad805c21 and 16ff5d83b". Its 16ff5d83b iOS run was cancelled, never completed — only bad805c21 actually has a completed green iOS run. So "first red at this head" is not established, and I re-ran the iOS workflow at 16ff5d83b (run 37878336230) to settle it. Green there + red at a7821ef58 would make this a #3330 defect and remove it from this tally permanently; red there too clears the PR and makes this instance 8 with a readiness-timeout sub-shape.
Counter-evidence against a #3330 cause, for context: Linux CI, macOS, Android and Coverage are all green at a7821ef58, and the POSIX ps identity path is shared with Linux. Only the iOS lane fails.
Will post the resolution either way and correct this comment's tally claim.
Instance 8 resolved: counted as this class. Not a #3330 defect.
Attempt 4 at the same head a7821ef58 came back green (job 113660776342, 04:01:10Z, run conclusion success, 18/18 non-skipped). Three identical failures followed by a success at the same commit settles it: the failures were not deterministic, which no code defect at a fixed head can be. Three independent lines agree:
Non-determinism at a fixed head. attempts 1-3 failed at 03:23/03:30/03:40, attempt 4 passed at 04:01 — same SHA, same image, same step.
Startup latency measured equivalent, head vs base, on the worker's own macOS host in the preflight's exact shape (fresh state dir, spawn, reuse, stop): medians 2.67 s head vs 2.66 s base, 6 head cycles + 3 base cycles, all publishing a reachable daemon.json, all reusing the same pid, all stopping gracefully, nowhere near the 15 s budget. No added probe exists to fix. Full table in #3330.
The code surface cannot reach the failure. Non-test changes at that head are the process-table row-walk factoring (reachable only from parseHostProcessList / readWindowsProcessRows) plus branch-for-branch-equivalent liveness-rule renames. The POSIX snapshot loop that daemon startup actually consumes has its own inline regex and was untouched; the rules net fewerkill(0) probes than base.
Tally: 8 instances of this class. The sub-shape observation from my previous comment stands and is worth separating when someone triages this: these two show daemonProcess spawned ~startupTimeoutMs before the failure, retired / exited / graceful, hasInfo:false, hasLock:false, and zeroregistration busy / not-running lines — i.e. "came up, never published a reachable daemon.json", which may be a different sub-class from the instances where registration contention is visible in the log.
Two corrections to what I wrote an hour ago, so this issue's record is not misleading:
I proposed cross-head re-run bisection as the discriminator. It is structurally unusable on this branch: the workflow's cancel-in-progress concurrency group cancelled every re-run attempt (mine twice, at 03:50:33Z and again later). Do not recommend re-run bisection on a ref that is still receiving pushes; use a local reproduction instead.
Seen again 2026-10-12T00:49Z on PR #3459, run 38189161525 (head 942f3d2): identical step (Preflight iOS runner through public CLI), identical signature (kind: "daemon_startup_failed", startupTimeoutMs: 15000, startupAttempts: 1, workspace .tmp/agent-device-state, cleanupResults: [retired/graceful pid 4169], metadataState: {hasInfo: false, hasLock: false}). The PR diff is test files plus one closure in packages/platform-apple/src/core/simulator.ts — no daemon, host-kit, or startup-path change — which keeps this in environment/flake territory. Notable in this payload: the pre-cleanup daemon retired gracefully and metadata showed no stale info/lock, yet the fresh start still missed the 15 s one-attempt window.
All three failed Preflight iOS runner through public CLI with daemon_startup_failed, a 15,000 ms startup budget, one attempt, graceful cleanup, and absent info/lock metadata. The #3714 and #3707 saved daemon logs are empty. Native build and simulator boot occurred before preflight; smoke scenarios and their XCTest assertions did not execute.
Correction to the initial attribution: these observations do not establish an infrastructure cause or exclude interactions with either PR. The historical 9c7d2602c diff against merge-base 59fdf008b contains TypeScript command, contract and MCP changes; its branch name does not establish a text-only control. #3707 failed at 17:55:53, not 17:19.
The logs contain the same simulator UDID, but the job API reports different runner IDs: 1000281901, 1000281934, and 1000282057, respectively. A matching UDID and per-ref concurrency setting do not prove a shared host, concurrent simulator access, or contention. The first two job intervals do not overlap (17:22:25–17:28:42 and 17:29:47–17:35:29).
The #3714 worker's committed-source probe found 373 eager module paths at both cac1247ac and merge-base 4db1b7d67, with identical path sets. Its other probe reported 4,288 versus 4,283 top-level statement weights. Equal module sets are not equal source, execution cost, or proof that startup cannot regress.
Local devices calls used one isolated state directory: the first took 3.30 s and the following calls 2.01/1.93 s. These are whole-command wall times, including discovery, not three independent cold-start measurements or a controlled base/head comparison. The worker's startup-focused invocation reported 31 passing tests; it used npx vitest with a tail pipeline, so this is not a directly verified pnpm command exit or CI reproduction. Future commands must use pnpm.
The root owner and failure mechanism remain unestablished. No CI retry, timeout increase, suppression, or production change was made for these observations. The #3714 exact-commit gate and independent review remain separate evidence; its iOS check remains failed. This is a diagnostic reference to existing #3342, not an expansion of campaign implementation scope.
One more preflight failure for the table: #3719 at 293c63e61 (docs and help text only), job 117014864094, 14:23 UTC on 2026-10-16. prepare ios-runner returned daemon_startup_failed (startupTimeoutMs: 15000, startupAttempts: 1), and a re-run of the job passed.
Fix contract for this class: #3782. Cause on origin/main: startLocalDaemon kills a live but CPU-starved daemon at a fixed 15 s deadline, and a timeout result never retries, so every report shows startupAttempts: 1. Newest instance: #3760, run 39019020372 (2026-10-17). In that run, daemon.log was empty and pnpm clean:daemon took 70 s, which shows that the host was overloaded.
Activity
thymikee commentedon Oct 8, 2026
Update: the class did not reproduce on the fixed head. Job 113540191797 @
98321a279ran all 24 steps withPreflight iOS runner through public CLI= SUCCESS, followed by both replay steps SUCCESS; with the three other Smoke lanes SUCCESS on that head, that is four independent preflight passes on one head. The class therefore stands on its three cross-branch instances, all confirmed from attempt-1 job data:daemon_startup_failed)Preflight iOS runner through public CLI(~19:44Z; the latest attempt shows cancelled after later pushes)daemon_startup_failed)Same ~1 h window, three unrelated branches, none touching daemon startup. A re-run of #3336's failed job at
98321a279is queued to strengthen the non-reproduction sample; outcome will be appended here.thymikee commentedon Oct 8, 2026
Re-run outcome: 37843633972 @ 98321a2 passed fully — the preflight step is now green on two consecutive full jobs on that head (the failing-attempt-1 job and this re-run), alongside the three cross-branch instances from the 19:38–20:32Z window. Non-reproduction sample strengthened; class remains open until its root owner is found.
thymikee commentedon Oct 8, 2026
Frequency table across branches tonight (coordinator consolidation, 2026-10-08)
Four instances of this class on four unrelated diffs inside 1h46m, all on
macos-26whereios.ymlgroups concurrency only per-ref (:57-65), so two iOS jobs can share a runner. Recorded so no future PR re-derives it:fix/windows-daemon-start-3291(#3330)0743ddba0Preflight iOS runner through public CLI,details.kind: daemon_startup_failedfix/android-shutdown-ime-flush-window(#3331)209600938Smoke Testsfailed at that same step (its latest attempt showscancelledafter later pushes)fix/ios-smoke-lane-reliability-2491(#3336)ffd575ae7fix/is-ambiguous-selector-2870(#3340)810b923a8daemon_startup_failedNone of these four branches touch the daemon-startup code path this issue is about — #3331's diff is Android-only, #3340's is selector/docs, #3336's is seven
test/files with zero product code — and the simulator UDID these jobs used (5FEB61C5-...) is not pinned anywhere in the repo, i.e. host-assigned.Two
cancelledjobs that are NOT instancesCounting
Smoke Tests (cancelled)wakes would double this table. Both have zero steps and a sub-minute lifetime, i.e. supersede/cancel housekeeping, not a failed run:21:00:16Z -> 21:00:17Z, 0 steps21:34:26Z -> 21:34:57Z, 0 stepsEvery real instance above arrived as
failurewith a readable log tail. Step count is what separates the two.Green controls on the same step
98321a279: preflight green on two consecutive full jobs; as of 21:39Z all 19 checks pass on that head.1869aed46, fix(ios): observe fill once without destructive repair #3334cb24ef6d7, fix(is): report ambiguous selector matches as ambiguity, not absence (#2870) #3340810b923a8: all fourSmoke Testslanes passed on their current heads.No assertion change was involved in any of those greens; the step simply got a healthy daemon. Four failures across four unrelated diffs, with the same step green on re-run, reads as runner/host contention on the shared macOS pool rather than a startup regression. Fourth instance (#3340) is new relative to the #3336 thread's three-instance record. Curated by #3336's worker; consolidated here by the coordinating thread. No action asked of anyone.
thymikee commentedon Oct 8, 2026
Fifth instance, and the simulator UDID matches on every log I could re-read
Adding a row the table above is missing (it was written before I triaged this PR, and I did not want a silent count drift):
fix/maestro-test-system-sheet-2560(#3341)29641150cPreflight iOS runner through public CLI,details.kind: daemon_startup_failedSo: five instances, five unrelated diffs, 19:38Z → 21:24Z (1h46m).
The new detail is host identity.
5FEB61C5-E3F5-471D-AE23-8A07438D5E92appears in the failed-step log of this fifth job, and re-reading the older jobs confirms the same UDID in job113543477378(row 4) and job113524981450(row 3). That is three of five confirmed on one simulator identity; rows 1 and 2 are listed as instances from their step/kind evidence only, because their job logs are no longer retrievable through the API path I used — I am not extending the UDID claim to them. Even 3-of-5 is the corroborating signal: unrelated diffs, one host-assigned device, consistent with contention or stale device state on a runner rather than a product regression.Diff-scope argument for this fifth one, recorded so it does not have to be re-derived: #3341's ten files are all under
packages/maestro/, and the daemon reaches Maestro only lazily —src/daemon/replay-device-selection.ts:52andsrc/commands/replay/script-source-bundle.ts:24both useawait import('@agent-device/maestro'). There is no static edge, so nothing in that PR is evaluated when the daemon boots, which is the phase this step exercises. ItsSizelane independently reports CLI startup medians faster than base (--version−0.6 ms,--help−1.5 ms).One housekeeping note for whoever triages this: a GitHub Checks wake can re-fire for an already-recorded failure when the PR head moves, without any new run existing.
run 37843216989is stillrun_attempt: 1,completed/failure,updated_at: 21:11:22Z— the "new" failure signal at ~21:53Z was that same long-settled job. Readingrun_attemptandupdated_atseparates a genuine re-failure from a re-announce, which matters here because the action for a first instance and a repeat at the same head are different decisions.thymikee commentedon Oct 8, 2026
Update on instance 5: green on re-run at the identical head
Row 5 in the table above is now stale in the direction that matters, so correcting my own entry: re-running that same job passed.
completed/failure, 21:05:19Z → 21:11:21Z, stepPreflight iOS runner through public CLI,details.kind: daemon_startup_failedcompleted/success, 21:59:44Z → 22:09:14Z,head_shaunchanged at29641150cThat is the cleanest causal evidence available for this class, and it is now on two heads (#3336 at
98321a279earlier, #3341 at29641150chere): same commit, same runner assignment, same host-assigned simulator, opposite outcome, with a mergeable diff sitting unchanged between the two runs. No property of the diff can produce that.For the triage decision this implies: nothing on any of these branches needs a code change, and a re-run is the correct first response, not a signal to widen a retry policy or gate a lane. Both affected PR authors recorded their re-run on the PR rather than redesigning, which is the outcome worth preserving.
thymikee commentedon Oct 8, 2026
Cross-row tally from the coordinator tonight: the class has hit five unrelated PRs; two went green on re-run at an identical head (#3336 at 98321a2, artifact-verified zero absorbed re-issues — see #3336 (comment) — and #3341 at 2964115). Non-reproduction-on-rerun is now the class signature, which strengthens the environment/daemon-state-dir root-cause hypothesis over any branch-specific cause.
thymikee commentedon Oct 8, 2026
Correction to my UDID claim:
5FEB61C5is not corroborating evidenceI posted earlier that five failures across five branches sharing simulator UDID
5FEB61C5-E3F5-471D-AE23-8A07438D5E92was "consistent with contention or stale device state." I checked that claim tonight and it does not hold, so I am withdrawing it rather than leaving it in a table whose purpose is to stop people re-deriving this.The same UDID appears in the log of a fully green iOS job on an unrelated branch: job
113564812060(run 37851061077,fix/macos-background-activation-3254@74f3899f0), all steps successful.5FEB61C5-…is therefore simply the simulator that this runner image uses for every iOS job, success or failure. It identifies the runner, not a degraded device, and it cannot discriminate a flaky run from a healthy one. My 3-of-5 wording inflated a constant into a signal.What survives the check, and why I still consider this a lane condition rather than five product bugs:
98321a279and fix(maestro): typed failure reasons for target-resolution misses #3341 at29641150c. Same commit, same runner, opposite outcome, unchanged diff between them.test/files, selector/docs, Maestro-only), all dying at the same preflight step before any scenario runs.That is still enough to justify "re-run first, don't redesign," which is the operational point of this issue. It is not enough to say anything about which device or why the boot probe times out, and this issue should not be cited for host identity. Whoever root-causes
daemon_startup_failedstill needs the runner-side evidence:ios.ymlgroups concurrency only per-ref (:57-65), so two iOS jobs can share onemacos-26runner, and the UDID is unpinned in the repo.One more lane data point, explicitly a different class so it is not folded into this table: job
113585608927(run 37857656021,fix/android-shutdown-ime-flush-window@fcf968b22, 23:07:54Z → 23:17:59Z) failed atRun fixture-backed iOS simulator E2E smoke, not at the preflight step:smoke:automation-input, step "wait for the deep-link destination (5/5)", typedreason: wait_capture_stalled,retriable: true. That is the transport/observation-wait class already attributed under #2491, whose re-issue policy is exactly what #3336 adds — and that diff is Android-only (7 files:packages/platform-android/**, one contracts type, one application-tools file), so it cannot reach an iOS deep-link wait. Noted here only because it is the same shared host in the same hour, not because the mechanisms are related.thymikee commentedon Oct 8, 2026
Sixth instance, 23:42:30Z — same class, and a case where the diff looked like it could reach daemon boot
fix/android-shutdown-ime-flush-window(#3331)83e78a7b8Preflight iOS runner through public CLI,details.kind: daemon_startup_failedWorth recording because this instance is the one where the "cannot be the diff" argument needed real work rather than a file-list glance. That PR now touches
src/platform-runtime-android-application-tools.ts, which is statically imported bysrc/platform-runtime-operation-host.ts:35— i.e. it sits on a daemon-reachable path, unlike the other five diffs. The ruling still holds, but only after checking the specific mechanism that would make it false: every import line added anywhere in that diff lives in a.test.tsfile or underpackages/platform-android/**, and thesrc/change adds no new static import, so the daemon's eager closure is unchanged.Repo Guardsgreen; the eager-closure budget lives in theCoveragelane.Operationally for triage: six instances, five unrelated diffs, 19:38Z → 23:42Z (4h04m) on one step that runs before any scenario. Two of them went green on re-run at an identical head. The one property I have not been able to establish is host identity — see my correction above: the simulator UDID is constant across green and red jobs and identifies the runner, not a degraded device. So whoever owns #3342 still needs runner-side evidence, and
ios.ymlgrouping concurrency only per-ref (:57-65) remains the leading suspect for two iOS jobs sharing onemacos-26runner.thymikee commentedon Oct 9, 2026
Seventh instance, 01:03:40Z — same payload, same step, seventh distinct diff
fix/android-shutdown-ime-flush-window(#3331)b0516feb8Failed step:
Preflight iOS runner through public CLI(job113616892498, run37867200258attempt 1; started 00:56:28Z, failed 01:01:28Z, 1 of 23 steps). Payload is this issue's canonical signature verbatim:Nothing new mechanically — same seam, same message, seventh occurrence tonight across seven unrelated diffs (Android IME flush window ×3, iOS fill normalization,
is-ambiguity, macOS background activation, iOS smoke-lane reliability). Two things this instance adds that are worth having on the record:cancelledin a wake is not a second failure. The GitHub checks API reported this lane ascancelledfor attempt 2 with 0 failed steps in 23, because attempt 2 was superseded while attempt 3 was being queued. Anyone readingcommits/{sha}/check-runswithoutrun_attemptwill see "Smoke Tests: cancelled" and conclude either a new class or a repeat failure. It is neither: attempt 1 failed with the payload above, attempt 2 was housekeeping. Per-head failure counts have to be taken fromruns/{id}/attempts/N, not from the check-run roll-up — which is also why I count one failure at this head, not two.packages/platform-android/**plus doc comments; it touches no daemon-boot or Apple path, andmainhas been green on iOS all night. I have explicitly instructed that agent to re-run rather than widen any timeout or add a retry, since a per-PR widening to paper over this is suppression, not a fix. Six prior instances all went green on re-run.Cost accrual, for whoever owns this: tonight this class has consumed at least seven smoke-lane jobs on PRs that were otherwise green, and it is currently the single reason my supervised PRs show red boards. Three distinct preflight/prepare signatures now hit that same first step (
daemon_startup_failed, the 15 sxcrun --show-sdk-versionprobe timeout recorded under #2491, and this one), which is why I keep the classes separate rather than filing them together.thymikee commentedon Oct 9, 2026
Instance 8 — 2026-10-09T03:23:40Z, head
a7821ef58of #3330 (native Windows daemon startup), iOS Smoke lane, stepPreflight iOS runner through public CLI:Attribution pending — deliberately not counted in the flake tally yet. Unlike the seven instances above, which rode unrelated diffs, this PR is not a bystander to daemon startup: it owns
src/daemon-registration-owner.ts, and its newest commit rewroteparseProcessTableRows, the shared row walk the macOS POSIXpspath also uses, plus the owner-liveness rules the startup owner probe consumes on posix. Lane history on that PR: green atbad805c21and16ff5d83b, first red ata7821ef58.A re-run was triggered at 03:25Z. Green ⇒ counts as instance 8 of this class. Red at the same head with the same signature ⇒ it is a #3330 defect, not this flake, and it will be removed from this tally. Will update either way.
thymikee commentedon Oct 9, 2026
Update on instance 8 — not counted in this class's tally; attribution held against it.
Re-ran the failed job at the same head. It failed again, attempt 2 (
job/113655150013, 03:31:52Z), identical signature. Two same-head failures is my threshold for stopping calling something a flake, so #3330's iOS Smoke is currently excluded from this issue's baseline rather than added to it.Two things worth recording here for whoever triages this issue:
1. The shape of these two failures is a readiness timeout, not an owner-liveness miss. Both attempts report
daemonProcessspawned ~startupTimeoutMsbefore the failure (pid 29551 @ 03:31:37, failed 03:31:52),cleanupResults: retired / exited / graceful,metadataState: hasInfo:false, hasLock:false, and zeroregistration busy/not-runninglines in the entire log. That is "the process came up and never published a reachabledaemon.jsoninside 15 s", which may be a different sub-class than the instances where registration contention is visible.2. My earlier bisect evidence was weaker than I stated. I wrote that #3330 was "green at
bad805c21and16ff5d83b". Its16ff5d83biOS run was cancelled, never completed — onlybad805c21actually has a completed green iOS run. So "first red at this head" is not established, and I re-ran the iOS workflow at16ff5d83b(run37878336230) to settle it. Green there + red ata7821ef58would make this a #3330 defect and remove it from this tally permanently; red there too clears the PR and makes this instance 8 with a readiness-timeout sub-shape.Counter-evidence against a #3330 cause, for context: Linux CI, macOS, Android and Coverage are all green at
a7821ef58, and the POSIXpsidentity path is shared with Linux. Only the iOS lane fails.Will post the resolution either way and correct this comment's tally claim.
thymikee commentedon Oct 9, 2026
Instance 8 resolved: counted as this class. Not a #3330 defect.
Attempt 4 at the same head
a7821ef58came back green (job113660776342, 04:01:10Z, run conclusion success, 18/18 non-skipped). Three identical failures followed by a success at the same commit settles it: the failures were not deterministic, which no code defect at a fixed head can be. Three independent lines agree:daemon.json, all reusing the same pid, all stopping gracefully, nowhere near the 15 s budget. No added probe exists to fix. Full table in #3330.parseHostProcessList/readWindowsProcessRows) plus branch-for-branch-equivalent liveness-rule renames. The POSIX snapshot loop that daemon startup actually consumes has its own inline regex and was untouched; the rules net fewerkill(0)probes than base.Tally: 8 instances of this class. The sub-shape observation from my previous comment stands and is worth separating when someone triages this: these two show
daemonProcessspawned ~startupTimeoutMsbefore the failure,retired / exited / graceful,hasInfo:false, hasLock:false, and zeroregistration busy/not-runninglines — i.e. "came up, never published a reachabledaemon.json", which may be a different sub-class from the instances where registration contention is visible in the log.Two corrections to what I wrote an hour ago, so this issue's record is not misleading:
bad805c21and16ff5d83b". Its16ff5d83biOS run was cancelled, never completed — that head has never been observed green.cancel-in-progressconcurrency group cancelled every re-run attempt (mine twice, at 03:50:33Z and again later). Do not recommend re-run bisection on a ref that is still receiving pushes; use a local reproduction instead.thymikee commentedon Oct 12, 2026
Seen again 2026-10-12T00:49Z on PR #3459, run 38189161525 (head 942f3d2): identical step (
Preflight iOS runner through public CLI), identical signature (kind: "daemon_startup_failed",startupTimeoutMs: 15000,startupAttempts: 1, workspace.tmp/agent-device-state,cleanupResults: [retired/graceful pid 4169],metadataState: {hasInfo: false, hasLock: false}). The PR diff is test files plus one closure inpackages/platform-apple/src/core/simulator.ts— no daemon, host-kit, or startup-path change — which keeps this in environment/flake territory. Notable in this payload: the pre-cleanup daemon retired gracefully and metadata showed no stale info/lock, yet the fresh start still missed the 15 s one-attempt window.thymikee commentedon Oct 16, 2026
Additional iOS preflight failures, 2026-10-16 — cause remains unknown
9c7d2602ccac1247acee2a12ef5All three failed
Preflight iOS runner through public CLIwithdaemon_startup_failed, a 15,000 ms startup budget, one attempt, graceful cleanup, and absent info/lock metadata. The #3714 and #3707 saved daemon logs are empty. Native build and simulator boot occurred before preflight; smoke scenarios and their XCTest assertions did not execute.Correction to the initial attribution: these observations do not establish an infrastructure cause or exclude interactions with either PR. The historical
9c7d2602cdiff against merge-base59fdf008bcontains TypeScript command, contract and MCP changes; its branch name does not establish a text-only control. #3707 failed at 17:55:53, not 17:19.The logs contain the same simulator UDID, but the job API reports different runner IDs:
1000281901,1000281934, and1000282057, respectively. A matching UDID and per-ref concurrency setting do not prove a shared host, concurrent simulator access, or contention. The first two job intervals do not overlap (17:22:25–17:28:42 and 17:29:47–17:35:29).The #3714 worker's committed-source probe found 373 eager module paths at both
cac1247acand merge-base4db1b7d67, with identical path sets. Its other probe reported 4,288 versus 4,283 top-level statement weights. Equal module sets are not equal source, execution cost, or proof that startup cannot regress.Local
devicescalls used one isolated state directory: the first took 3.30 s and the following calls 2.01/1.93 s. These are whole-command wall times, including discovery, not three independent cold-start measurements or a controlled base/head comparison. The worker's startup-focused invocation reported 31 passing tests; it usednpx vitestwith a tail pipeline, so this is not a directly verified pnpm command exit or CI reproduction. Future commands must use pnpm.The root owner and failure mechanism remain unestablished. No CI retry, timeout increase, suppression, or production change was made for these observations. The #3714 exact-commit gate and independent review remain separate evidence; its iOS check remains failed. This is a diagnostic reference to existing #3342, not an expansion of campaign implementation scope.
thymikee commentedon Oct 17, 2026
One more preflight failure for the table: #3719 at
293c63e61(docs and help text only), job 117014864094, 14:23 UTC on 2026-10-16.prepare ios-runnerreturneddaemon_startup_failed(startupTimeoutMs: 15000,startupAttempts: 1), and a re-run of the job passed.thymikee commentedon Oct 17, 2026
Fix contract for this class: #3782. Cause on origin/main:
startLocalDaemonkills a live but CPU-starved daemon at a fixed 15 s deadline, and atimeoutresult never retries, so every report showsstartupAttempts: 1. Newest instance: #3760, run 39019020372 (2026-10-17). In that run,daemon.logwas empty andpnpm clean:daemontook 70 s, which shows that the host was overloaded.