Skip to content

fix(server): recover opted-in threads after machine restarts - #9803

Merged
maria-rcks merged 4 commits into
pingdotgg:mainfrom
maria-rcks:t3code/reliable-restart-continuation
Sep 4, 2026
Merged

fix(server): recover opted-in threads after machine restarts#9803
maria-rcks merged 4 commits into
pingdotgg:mainfrom
maria-rcks:t3code/reliable-restart-continuation

Conversation

@maria-rcks

@maria-rcks maria-rcks commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

The existing continuation setting now covers normal close/reopen, server crashes, and machine reboots once T3 starts again. It remains off by default and is stored per environment, so existing client-local opt-ins must be enabled once on each environment.

Recovery requires matching unfinished thread/provider state and saved provider resume state; preparation persists intent before clearing the old turn, and send failures clear it without blindly retrying the provider. Superseded recovery markers cannot replay an older turn. Prepared recovery survives provider reconnection without replaying work that finished after update marking.

Verified with 207 focused tests plus the added bulk-read failure regression (24 recovery tests), server/web typechecks, targeted lint (existing settings-panel warnings), and a production web build. In the real client, a Codex thread resumed and finished after SIGKILL/relaunch and another resumed after clean SIGTERM/relaunch; a completed thread stayed idle. The opt-in persisted across server restart and browser reload, and the settings row was checked in dark/light desktop and narrow web layouts. The OS itself and Electron's native quit UI were not restarted.

Environment opt-in for restart continuation

Real Codex thread completed automatically after clean server close and reopen

Browser recordings were captured on the remote desktop, but this environment cannot retrieve those files for inspection/upload; video evidence remains unavailable.

Two independent gpt-6-astra reviewers at xhigh recommend merging after the recovery fixes.

Implemented with gpt-5.6-sol in the Codex harness.

Note

Recover opted-in provider threads after server restarts

  • Moves continueThreadsAfterServerUpdate from ClientSettingsSchema to ServerSettingsSchema (default false), making the preference per-environment rather than per-client. ServerSettingsPatch can now modify it.
  • On shutdown, ProviderService.runStopAll records the active turn as continueAfterServerUpdate for running bindings when the server setting is enabled; stopSession and new turns clear both continuation markers.
  • On startup, reconcileProviderSessions scans persisted bindings for prepared continuations, recovers eligible interrupted sessions when projection, binding, active turn, and resume cursor all match, and settles invalid or mismatched sessions instead.
  • UI relabels the setting as "Continue threads after restarts" and reads it via useEnvironmentSettings in ServerUpdateAction.tsx.
  • Behavioral Change: continueThreadsAfterServerUpdate removed from ClientSettingsSchema and ClientSettingsPatchSchema in settings.ts; previously enabled client-side preferences are no longer read and must be re-enabled per environment.

Macroscope summarized da3d799.

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Sep 4, 2026
Comment thread apps/server/src/serverRuntimeStartup.ts Outdated
@macroscopeapp

macroscopeapp Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds a cross-cutting, durable provider-session recovery workflow for crashes and machine restarts, including new startup reconciliation and provider lifecycle state transitions. It also moves the opt-in to environment-scoped server settings, so the breadth of runtime and settings behavior warrants human review.

You can add or adjust custom eligibility rules. Learn more.

@github-actions github-actions Bot added size:L 100-499 changed lines (additions + deletions). and removed size:M 30-99 changed lines (additions + deletions). labels Sep 4, 2026
Comment thread apps/server/src/serverRuntimeStartup.ts Outdated
@maria-rcks
maria-rcks merged commit b906ce2 into pingdotgg:main Sep 4, 2026
27 checks passed
patroza pushed a commit to patroza/t3code that referenced this pull request Sep 4, 2026
Recover opted-in threads after machine restarts (pingdotgg#9803). Keep fork
interactionMode and graceful-stop continuation markers beside the
per-environment restart opt-in.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L 100-499 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant