Skip to content

fix: retry read-only host RPC timeouts - #2369

Merged
ymichael merged 5 commits into
get-bb:mainfrom
jshph:fix/retry-timeout-host-rpcs
Aug 25, 2026
Merged

fix: retry read-only host RPC timeouts#2369
ymichael merged 5 commits into
get-bb:mainfrom
jshph:fix/retry-timeout-host-rpcs

Conversation

@jshph

@jshph jshph commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Context

Thread provisioning intermittently failed with thread_provisioning_failed after waiting the full 30-second host RPC timeout for a retryable inspection command. The daemon connection remained usable, so repeating the inspection was safer and more useful than failing the whole provisioning attempt.

This is transport resilience, not an OpenRouter/quota change.

What changed

  • Retry a host RPC once when its first response times out, but only through callHostRetryableOnlineRpc.
  • Preserve single-attempt behavior for ordinary/non-retryable host RPCs.
  • Keep the existing reconnect wait for websocket-unavailable failures.
  • Split the caller's existing timeout budget across two response-timeout attempts, so enabling the retry does not double an established command deadline.
  • Increment HOST_DAEMON_PROTOCOL_VERSION to 167 because the server can now send a second host-rpc.request after a response timeout.
  • Add deterministic coverage where the first response is dropped and the second request succeeds, plus deadline, non-retry, fresh-request-ID, and late-response handling.

Safety boundary

A timeout is ambiguous: the host may have executed the first request and lost only its response. Retrying can therefore execute the command twice. The retry helper accepts only HostDaemonRetryableOnlineRpcCommand, derived from command-registry entries explicitly marked retryable: true; commands not explicitly safe to repeat cannot enter this path through the typed API.

Each attempt uses a fresh request ID. The first waiter's timeout removes it from the hub, so a late response from the first attempt is treated as stale and cannot resolve the second attempt. Tests also keep the existing reconnect retry and ordinary-call single-attempt behavior covered.

Verification

  • pnpm exec turbo run test --filter=@bb/server -- test/hosts/online-rpc.test.ts test/system/execution-options.test.ts test/public/public-provider-installations.test.ts — 50/50 passed
  • pnpm exec turbo run test --filter=@bb/host-daemon-contract -- test/contract.test.ts — 37/37 passed
  • pnpm exec turbo run typecheck --filter=@bb/host-daemon-contract --filter=@bb/host-daemon --filter=@bb/server — 6/6 Turbo tasks successful
  • git diff --check origin/main...HEAD — passed

The deterministic regression drops the first response while leaving the daemon socket registered. Before this change the call rejected with 504 command_timeout; after the change the second request succeeds inside the original command budget. Fake timers pin the attempt boundary, and the test verifies that the first late response is stale and the two requests have distinct IDs.

@ymichael ymichael self-assigned this Aug 25, 2026
@ymichael
ymichael force-pushed the fix/retry-timeout-host-rpcs branch from 718e78e to 3c8532b Compare August 25, 2026 18:45
@ymichael
ymichael merged commit 69596f1 into get-bb:main Aug 25, 2026
22 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants