Flaky e2e: gateway-fast-fail times out on a lease of the busy-worker tests #395

Activity

1. V3RON

   added

   bug:newReported. Nobody has looked yet.Reported. Nobody has looked yet.

   flaky-testNames a test that fails without a code change. Delivery skips its base-commit check.Names a test that fails without a code change. Delivery skips its base-commit check.

   bug:triageAn agent may reproduce it and write the triage report.An agent may reproduce it and write the triage report.

   and removed

   bug:newReported. Nobody has looked yet.Reported. Nobody has looked yet.

   on Oct 6, 2026
2. V3RON

   self-assigned this

   on Oct 6, 2026
3. V3RON commented on Oct 6, 2026

   

   V3RON

   on Oct 6, 2026

   ContributorAuthor

   More actions

   Reproduction

   This is a defect. It reproduces on origin/main 4ad4791: the lease timed out in 4 of 52 runs of the file, which also fails in other ways under load. The call that hangs is always the first lease that should be granted after a worker joins, in whichever test runs it. #396 is the same failure.

   The test on bug/395-repro holds the worker's socket claim, so the race always goes the bad way. It fails in 60 ms:

   pnpm exec vitest run --project unit src/daemon/gateway-fleet.e2e.test.ts
   × reports a worker running and grants a no-wait lease on it, when the worker's socket claim is slow
   AssertionError: expected [ 'starting' ] to deeply equal [ 'running' ]
    ❯ src/daemon/gateway-fleet.e2e.test.ts:365
   

   With that line removed, the lease fails with NO_CAPACITY. Evidence from a failing e2e run, read 20 s into the hang:

   gateway worker.list: connection "connected", catalogReadAt set, health "starting"
   worker status.get:   health "running"
   worker daemon.log:   "Uplink to gateway established" logged before "Claimed daemon socket"
   gateway events:      lease.requested, lease.queued, and no request.dispatched
   

   Root cause

   src/daemon/main.ts:515 starts the gateway uplink before daemon.start() (main.ts:559) runs, so it starts before #readyPromise is set (server.ts:464). Until then, #awaitReady (server.ts:596) parks nothing. The gateway's events.subscribe and first refresh run during convergence, and its view stores health starting. Nothing refreshes that view afterwards, because daemon.started does not trigger a refresh (changesCapacityOrLeases, src/gateway/worker-link.ts). The healthy routing stage drops the worker, and the request waits for the 30 s refresh tick. The CLI's 20 s deadline fires first. The comment at main.ts:511 says the uplink waits for the claim. It does not.

   Simplest fix

   Start the uplink from onSocketClaimed, which start() calls right after setting #readyPromise. Wire that callback whether or not HTTP is on, and correct the comment. I tried this locally. The new test passes, and so do main.test.ts, gateway-uplink.test.ts and the three gateway e2e files.

   Alternatives rejected

   • Refresh the view on daemon.started: this fixes health, but the gateway would still read the catalog and config during convergence.
   • Make #awaitReady park before start(): a larger change, and it alters every test that builds a DaemonServer without starting it.
   • Make the e2e wait for health running: this hides the defect from users.

   Risk

   • The uplink must still start with HTTP off. Today the callback is wired only under config.http.enabled.
   • A failed socket claim must leave the uplink undialled.
   • The gateway's first status.get still answers starting because it never parks. Only the refresh after it must see running.

   Side findings

   • Flaky e2e: gateway-fast-fail's 'token create --role worker' on a fresh gateway prints no JSON #404

   Written by an agent.
4. V3RON

   removed their assignment

   on Oct 6, 2026
5. V3RON

   added 2 commits that reference this issue on Oct 6, 2026

   feat(core): a reserve of running slots, and a request waits for a dev…

   ...

   dfd48a8

   feat(core): warm targets boot or create devices ahead of demand (#408)

   ...

   37188b5
6. V3RON

   added

   bug:readyAn agent may fix it.An agent may fix it.

   and removed

   bug:triageAn agent may reproduce it and write the triage report.An agent may reproduce it and write the triage report.

   on Oct 6, 2026
7. V3RON

   self-assigned this

   on Oct 6, 2026
8. V3RON

   added 3 commits that reference this issue on Oct 6, 2026

   test: pin that the claim callback starts no HTTP gateway with HTTP off (

   ...

   f175498

   test: count a refused HTTP bind as a started frontend too (#395)

   ...

   0459840

   docs: correct onSocketClaimed uplink comment (#395)

   ...

   9b49b7a
9. V3RON

   closed this as completedin #427on Oct 7, 2026