Skip to content

feat(agent-org): add durable pause and resume handoff - #881

Draft
ShiboSheng wants to merge 1 commit into
codex/issue-759-pr4-task-fsmfrom
codex/issue-760-pr5-pause-resume
Draft

feat(agent-org): add durable pause and resume handoff#881
ShiboSheng wants to merge 1 commit into
codex/issue-759-pr4-task-fsmfrom
codex/issue-760-pr5-pause-resume

Conversation

@ShiboSheng

Copy link
Copy Markdown
Collaborator

Problem

Fixes #760.

The PR4 baseline can change a Team status and cancel whole Sessions, but it cannot durably describe which formal Turns were paused, prove which runtime instance may be released, or resume only work that remains legal after a pause/restart boundary. A late Turn can therefore cross lifecycle generations, consume formal Inbox work, or clear a replacement runtime; the UI can also remain stuck on draining after a transient push gap.

Solution

Add a durable Pause/Resume handoff protocol for Agent Org formal work:

  • Commit the Paused fence, generation bump, Pause episode, and all captured Coordinator/TaskExecution child receipts in one immediate SQLite transaction.
  • Give each Session runtime a lease and bind teardown to the exact Session, lease, dialog Turn generation, and durable Turn intent. Ten-second timeout evidence never rolls back the Paused fence or starts a replacement runtime.
  • Consume an episode once on Resume, re-check Task ownership/status/materialization, allocate Task continuations through the existing Member FIFO, and dispatch only after the captured runtime is released or proven absent.
  • Fence Inbox materialization/acknowledgement, Provider admission, Task mutation, and Group Chat submission against lifecycle generation; remove the old automatic/synthetic resume path.
  • Persist restart-safe continuation evidence and reconcile it before generic in-flight cleanup.
  • Expose Pause/Resume outcomes and a read-only draining phase through Tauri, update Overview/Group Chat behavior, and keep Paused/Idle Teams free of fallback polling. Add all 13 locale strings.
  • Make task_update expose role-aware strict-provider schemas while preserving the typed parser as the mutation authority.

The rollout gate remains off. Archive/Delete, UserDirectedWork, interventions, multi-Member mentions, and the other later delivery-stack features are unchanged.

Potential risks

  • Strict schema compatibility: the canonical runtime manifest grows from 15 to 17 tables with no PR4-to-PR5 migration. PR4 and PR5 binaries must use their matching database or a new isolated ORGII_HOME. Rollback uses the database paired with the target commit; an older binary must not open the new manifest and guess a downgrade.
  • Concurrency: Pause teardown and Resume dispatch are asynchronous. Durable receipts, unique constraints, generation checks, and runtime leases make retries idempotent, but a Provider that does not yield within ten seconds remains recorded as timed out until its real completion callback releases it.
  • Restart: continuation dispatch is intentionally at-most-once by receipt. Startup reconciliation and duplicate-request tests cover the crash windows, but this is a new persistence/recovery path and should remain behind the rollout gate until the stacked series completes.
  • Stack dependency: this PR is based on PR feat(agent-org): enforce the formal Task FSM and owner permissions #869 and requires its formal Task FSM, owner authority, Turn context, and Member FIFO contracts.
  • UI transport: Paused Teams do not poll. A missed push is repaired once on WebSocket reconnect or visibility restoration; there is deliberately no Paused interval fallback.
  • Review size: 81 files / 6,693 review lines exceed the original file-count control line because runtime leases touch the existing Session runtime call surface and language parity requires 13 locale files. The issue owner approved this Impact Spike; review lines remain below the 15,500 P90.

Scope and invariants

  • Team lifecycle status, Task status, Turn-intent status, and drain status remain separate concepts.
  • Runtime leases stay inside the general Session runtime owner; Agent Org episode/continuation policy stays in coordination/lifecycle.
  • Pause/Resume never creates, deletes, reassigns, completes, fails, or cancels a Task.
  • UserDirectedWork is excluded from PR5 child selection and yield.
  • Ordinary SDE Send, Queue, Force Send, Stop, Resume, and Compaction retain their existing behavior.

Verification

Automated checks:

  • cargo test -p agent_core — 3,175 passed, 0 failed, 2 ignored; doc tests completed with 11 ignored.
  • cargo clippy -p agent_core --all-targets -- -D warnings — passed; only the existing future-incompatibility notice for block v0.1.6 was emitted.
  • Focused schema regression after the final naming cleanup — 1 passed.
  • pnpm vitest run — 1,116 files and 8,767 tests passed.
  • pnpm typecheck — passed.
  • ESLint for every changed TS/TSX file — passed.
  • Commit hooks — lint-staged, staged TypeScript, scoped Clippy for agent_core and org2, commitlint, and commit stats passed (Total eslint: 5, total circular: 0).
  • Changed E2E scripts passed node --check; git diff --check passed.
  • pnpm run check:circular did not reach graph analysis because Madge cannot resolve two existing ?raw React artifact imports in reactArtifactDocument.ts; that file is outside this diff. The staged-import hook reported zero circular dependencies in changed files.

Rendered and packaged-app verification:

  • BuildFast from exact commit 49886324af912783f871a353a70aa51d02595957 passed. Verified artifact: /private/tmp/ORG2-PR5-49886324.app; binary SHA-256 6c20b63aefb6d498195573aace4503727c6fa738f9bab6973a7bba2834c60675.
  • Final exact-commit core WebDriver smoke used the real Pause/Resume buttons with ten formal Turns: Pause fence 74 ms, all ten runtimes drained in 838 ms, and the complete pause/chat-block/resume scenario passed.
  • Twenty sequential isolated packaged-app samples on the frozen final tree all passed: Pause fence P50 123 ms / P90 157 ms / max 170 ms (budget 250 ms); ten-runtime drain P50 882 ms / P90 989 ms / max 1,041 ms (budget 10 seconds).
  • Rendered restart, coordinator/member A→B→A history switching, durable Paused Overview, duplicate Load More, cancelled/reassigned continuation skip, duplicate requests, and old-lease/new-runtime scenarios passed.
  • Computer Use verified real product clicks, Paused/draining UI, readable history, disabled Paused Group Chat, quit/relaunch persistence, and Resume. SQLite reads confirmed one episode, exact child/continuation receipts, generation bumps, unchanged Tasks, and no duplicate Inbox work.
  • A real Provider smoke used the configured codexharry account with GPT-5.4 Mini. Formal Task create/start/read/complete succeeded with zero task_update errors; Pause, app restart, history, and Resume completed with generation 1 -> 2 -> 3, one consumed episode, the completed Task unchanged, and zero unread Inbox rows. No credential or token is included in this diff.
  • Command+5 resource checks, after legitimate startup recovery settled, showed no Agent Org polling, formal Wake, or new Provider in two 75-second foreground windows and a 60-second hidden window. CPU was 0.0%; hidden-window RSS changed by approximately 16 KiB and durable counts stayed unchanged.

Architecture audit

All ten layers were checked: compilation/types, dead and duplicate paths, naming, state semantics, default branches, cross-domain boundaries, understandability, wire parity, initialization/restart consistency, and Session-to-Run resolver symmetry. The old direct status mutators and automatic resume path were removed; strict 17-table initialization and restart continuation dispatch are covered by regression tests.

UI evidence

The packaged app was exercised through Computer Use and WebDriver using real product controls. No static screenshot is attached because the acceptance is a lifecycle transition (Paused -> draining -> Paused -> Resume) plus persisted SQLite/runtime evidence rather than a stable single frame; the PR remains Draft for stacked review.

Persist Pause episodes and per-Turn handoff receipts so a Team can stop formal work immediately, drain runtimes safely, and resume only work that remains legal after restart or concurrent state changes.

Add runtime leases, generation-fenced Inbox and Task boundaries, one-shot continuation dispatch, paused Group Chat enforcement, push-driven draining UI, strict-provider task_update schemas, and rendered recovery coverage.

Verification:
- cargo test -p agent_core (3175 passed, 2 ignored)
- cargo clippy -p agent_core --all-targets -- -D warnings
- pnpm vitest run (1116 files, 8767 tests)
- pnpm typecheck and changed-file ESLint
- BuildFast packaged app with 20/20 ten-runtime Pause/Resume samples
- codexharry GPT-5.4 Mini packaged-app Pause/restart/Resume smoke

Pre-commit hook ran. Total eslint: 5, total circular: 0
@ShiboSheng

Copy link
Copy Markdown
Collaborator Author

orgii://cloud/session/ref?v=1&org=bfa7b134-2486-45fa-81ad-a369441fafb4&owner=776dbd69-ac1d-4f72-a0d4-69cb4f2667dd&session=codexapp-rollout-2026-08-22T18-01-45-01a028eb-4fec-74b2-bbb7-528b8361c483

@ShiboSheng
ShiboSheng requested a review from Neonforge98 August 22, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant