Skip to content

[v2] Interactive waits survive a v2-server restart - #393

Open
ObsisMc wants to merge 3 commits into
ora-space:mainfrom
ObsisMc:feat/1228-interactive-wait-restart
Open

ObsisMc wants to merge 3 commits into
ora-space:mainfrom
ObsisMc:feat/1228-interactive-wait-restart

Conversation

@ObsisMc

@ObsisMc ObsisMc commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Interactive Agent nodes parked awaiting input now survive a v2-server restart, following desktop's selective recovery (desktop docs/workflows/workflow-runs.md, "Recovery after restart"; crates/backend/src/workflow/run/engine.rs is_awaiting_input). Before this PR, a restart failed every parked node.

Refs ora-space/circus#1228

v2, does not involve specs.

Read this first

  • The recovered Agent is a new provider session. The pinned Node (c5265072) cannot reattach the original ACP session. Its protocol has no session/load entry, and it deletes the Agent's own session store when the session ends. So after a restart the node's next turn starts a new Agent on the same checkout and sends it a handoff of the recorded conversation first. This is what desktop does with rebuild_provider_session when session/load fails.
  • Commit 3 changes the existing failure semantics and is meant to be reviewed on its own. It covers a Node that dies (process exit or broken connection) while v2-server keeps running. Before, the parked interactive session ended with the Node, and the node could only complete with its last normal answer (session_ended: true). Now the session stays open and takes the same rebuild path as after a restart. Details below.

Commits

Each commit builds and passes its packages' tests on its own.

  1. Execution layer: start a new Agent on the recorded checkout for a kept held session.
    • store.RecoverExecutions keeps the held session of an interactive node parked in a running run (pending row, interactive summary, no unfinished turn). It still ends every other session.
    • Each workspace records the Node clone execution behind its checkout (v2_workspaces.clone_execution).
    • On startup, Driver.Reattach hands the run's recorded checkout to the driver. Every later session of the run then starts on the same directory, uncommitted edits included, instead of a fresh clone. Node resolves the checkout from its own ledger across incarnations; this was verified on a real Node with both graceful and SIGKILL restarts before implementation.
    • A recovered session's next turn sends the handoff as its own first text block. The handoff and the prompt together stay within Node's 64 KB turn limit, dropping the earliest turns first.
    • The turn records only the typed prompt.
    • The published history joins the earlier settled records with the new Agent's records, offset past them. The handoff record is never published, so the page shows one continuous conversation.
    • If the new Agent cannot start (NotStarted), the turn fails and the session keeps waiting for a retry. A checkout that no longer exists ends the session with workspace_missing.
  2. Workflow host: keep a valid interactive wait across a restart.
    • The restart sweep keeps a valid wait: a pending root interactive Agent attempt with a session and a parked summary, in a running run with no executing root Agent.
    • A run with an executing root Agent still fails as a whole, parked nodes included. This includes the parked node's own follow-up executing at the restart. Nothing is replayed.
    • The host rebuilds the node's process-local state from the row and the recorded turns: turn count, last and accepted answers, follow-up numbering.
    • A node whose executor no longer resolves, or whose session is gone, fails and its session is ended.
    • Sessions kept for nodes the sweep failed are ended before any run resumes.
    • A node whose checkout is gone fails as interactive_session.
  3. Rebuild a parked node's Agent after its Node is lost (changed failure semantics).
    • A held interactive session survives agent_transport_closed, and the lost Node's process tree is retired through its supervisor at once.
    • A message that finds the Node already gone before reaching the Agent (execution.ErrNodeLost) runs at once on a new Agent with the handoff.
    • A turn executing, or being sent, when the Node died fails and is not replayed. The next message rebuilds.
    • The run's checkouts outlive the lost incarnation, so later nodes keep the directory. This also applies to automatic nodes, which used to get a fresh clone after a Node loss.
    • Unchanged: a first turn lost with its Node still fails as interactive_session, and an Agent process crash with its Node alive still ends the session. Conversations and task runs still end. The one visible difference for them is that a turn finding its Node gone now reports agent_transport_closed instead of agent_session_unavailable / agent_dispatch_failed.

Out of scope

  • Conversations and task runs still end on a restart (circus#1148 scenario 4).
  • The Agent completion signal and confirmation gates (circus#1166) are separate. When they land, a rebuilt session will need its completion tool and a new attempt token, as on desktop.

Tests

  • Unit and PostgreSQL tests:
    • store: kept sessions, ended-session workspaces, unparked sessions per run.
    • Execution layer: recovery on the recorded checkout, unstarted retry, missing checkout, a Node lost before or during a turn, a first turn lost with its Node, handoff rendering and budget.
    • Driver: history join without the handoff.
    • Workflow host: wait survives and continues with follow-up numbering; an executing follow-up fails; a parked node beside an executing Agent fails; missing workspace; unstarted recovery; executor or session gone.
  • Real Node and OpenCode with the fixture model, TEST_DATABASE_URL plus V2_NODE_ASSETS_DIR:
    • TestRealInteractiveWaitSurvivesARestart
    • TestRealSuccessorAfterARestartRunsInTheRunsCheckout
    • TestRealInteractiveNodeOutlivesItsNode
  • GOMAXPROCS=2 go test -race -p 1 over store, workflowhost, executionruntime, agentruntime, boot and execution passes on the head. Every commit was also tested on its own.
  • TestRetryWaitAbandonedWhenAContainerFailsTheRun fails intermittently, on upstream main 5416335a as well (2 of 8 runs there), unrelated to this change.
  • Three rounds of independent review; the last round found no High or Medium.

Screenshots

Real v2-server, real Node and OpenCode with the fixture model, frontend-v2. v2-server was stopped with SIGTERM and started again between 1 and 2.

1. Parked before the restart 1
2. After the restart: still awaiting input 2
3. Follow-up after the restart: one conversation, no handoff shown 3
4. Completed after the restart: the successor ran, run succeeded 4

…t held session

On startup the execution service keeps the held session of an interactive
workflow node parked awaiting input, instead of ending it with the rest. Its
Agent went with the old process, and the pinned Node can neither load a
provider session nor keep the Agent's own session store, so the session's
next turn starts a new Agent on the same checkout, as desktop's
rebuild_provider_session does after a failed session/load:

- Each workspace records the Node clone execution behind its checkout; Node
  resolves the checkout from it in its own ledger, also on a new incarnation.
  The kept run's workspace is reattached on startup (Driver.Reattach), so
  every later session of the run, not only the recovered one, starts on the
  same directory with its uncommitted edits instead of a fresh clone.
- The new Agent is first sent a handoff of the recorded conversation (user
  messages, answers, tool calls by name and outcome only) as its own block;
  the turn records only the typed prompt.
- The page keeps one conversation: the earlier settled history comes first,
  the new Agent's records follow it, and the handoff block is never published.
- A new Agent that could not start, Node's checkout refusals included, fails
  the turn and leaves the session waiting for another; a checkout that no
  longer exists ends the session with workspace_missing.
Desktop keeps a node parked awaiting input across a restart
(is_awaiting_input) and fails the rest; v2 failed every parked node. Now the
restart sweep keeps a pending root interactive Agent attempt with a session
and a parked summary, and the host rebuilds its process-local state (turns,
last and accepted answers, follow-up numbering) from the row and the
session's recorded turns, so the starter can talk to it, stop it or complete
it again. Nothing is replayed: a run with an executing root Agent, this
node's own follow-up included, still fails as a whole.

A parked node whose executor no longer resolves or whose session is gone
fails (its row moves back to running so the engine can fail it) and its
session is ended. The sessions execution recovery kept for nodes the sweep
failed are ended before any run is resumed, so sessions the resumed runs
start are never touched. A node whose checkout is gone fails as
interactive_session.

The README says what the recovered Agent does not have: the handoff carries
no tool inputs or outputs, and the first turn after a restart has no prompt
cache.
A Node that exits or loses its connection while the server keeps running
used to end every held interactive session with it, leaving the node only
completable with its last normal answer. It now takes the restart's rebuild
path: the held session stays open (the lost Node's process tree is retired
through its supervisor at once), the run's checkouts outlive the lost
incarnation, and the session's next turn starts a new Agent on the original
checkout with the recorded conversation.

A turn that finds its Node already gone before reaching the Agent
(execution.ErrNodeLost) is handed to the replacement Agent at once; a turn
executing or being sent when the Node died fails and is not replayed, and
the following one rebuilds. A first turn lost with its Node has nothing to
hand over and fails as before, and an Agent process crash with its Node
alive still ends the session. Conversations and task runs keep ending; a
turn finding its Node gone now reports agent_transport_closed for them too.
The README records the changed failure semantics.

@zhangxinhao zhangxinhao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM(38f5b89)

@ObsisMc 我看了这个 PR,理解它是要让合法的交互等待跨 v2-server 重启保留,并在下一轮用原 checkout 和历史交接重建 Agent;Node 丢失时也采用同样的恢复路径。可以合并,建议补齐两处恢复边界:Node 退役后其他等待会话无法重建,以及 checkout 丢失后直接完成节点会让后继静默重新 clone。定向 race 测试通过,数据库恢复测试因环境缺少配置而跳过。

  • 🔧 1 处小改已写成建议(P2-1):见行内 suggestion,可以一键采纳,也可以不改或留到以后。
  • 💬 其他建议:P2-2 目录丢失时直接完成会静默重新克隆
  • 测试:go test -race -timeout=5m -run 'Test(Handoff|RecoveredHistory)' ./internal/executionruntime ./internal/agentruntime 通过;go test -race -timeout=5m -run 'TestDriverIPC(Disconnect|EndDisconnected|Rebuild|WorkflowSessions|FullTurn)' ./internal/agentruntime 通过;go test -race -timeout=5m -v -run 'Test(RecoveryKeepsTheSessionOfAParkedInteractiveNode|WorkspaceKeepsItsCloneExecution|RecoveredSession|RunOfANodeParkedOnAnEndedSession|HeldSessionContinuesOnANewAgent|HeldSessionOutlivesANode|FirstTurnLostWithItsNode|ParkedNodeSurvivesARestart|RestartFailsA|MissingWorkspaceFailsTheRecoveredNode|UnstartedRecoveryKeepsTheNodeWaiting|ParkedNodeThatCannotContinue)' ./internal/store ./internal/executionruntime ./internal/workflowhost 没跑(已尝试运行,相关用例均因未设置 TEST_DATABASE_URL 而跳过,不能视为恢复行为验证通过。);没跑全量和 e2e。

涉及数据迁移、并发核心,建议 owner 再看一眼。

以上都是建议,怎么处理由你决定。

自动检视 · 基于 38f5b89 · 第 1 轮(全量)

Comment on lines 269 to +277
if a.ended || a.ending {
return nil, false, execution.ErrSessionEnded
}
// Checked before broken: a Node's death also marks its sessions broken.
select {
case <-a.project.dead:
return nil, false, execution.ErrNodeLost
default:
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2-1(建议) 这里先检查 ended/ending,会漏掉已经退役的 Node。同一项目有两个等待节点时,第一个节点重建或某个执行中的会话触发 stopAgent,closeResources 会对所有旧会话调用 requestEnd 和 markStopped;另一个节点随后追问就返回 ErrSessionEnded,运行时将其永久关闭,无法走新加的重建路径。建议先判断 Node 是否丢失,再判断物理会话是否结束,并补一个多个等待会话的回归测试。

Suggested change
if a.ended || a.ending {
return nil, false, execution.ErrSessionEnded
}
// Checked before broken: a Node's death also marks its sessions broken.
select {
case <-a.project.dead:
return nil, false, execution.ErrNodeLost
default:
}
// Check Node loss first: retirement also marks its sessions ending or ended.
select {
case <-a.project.dead:
return nil, false, execution.ErrNodeLost
default:
}
if a.ended || a.ending {
return nil, false, execution.ErrSessionEnded
}

可直接采纳,也可以不改。

Comment on lines +177 to +181
// A checkout that is gone is left out, and the recovered session's turn reports it; any
// other doubt reattaches, so the recovered Agent never lands in a fresh clone.
if _, err := os.Stat(at.Path); !errors.Is(err, fs.ErrNotExist) {
s.agent.Reattach(r.Session.WorkspaceID, at.CloneExecution, at.Path)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2-2(建议) checkout 在重启前被删除时,这里跳过 Reattach,但仍保留运行的 workspace 映射。如果用户重启后直接完成节点,CompleteNode 只调用 End,不会经过 recovery 的目录检查;后继进入 prepareCheckout 时找不到已记录的 checkout,便重新 clone,在缺失此前修改的目录上继续运行。建议让缺失 checkout 的状态同时约束直接完成和后继启动路径,按约定失败关闭,并补一个“不追问、直接完成”的缺目录测试。

@EricWvi EricWvi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

owner 验收第 1 轮(38f5b897):0H/2M/5L。请修 F1(Node 不认识记下的 checkout 时按已不在处理),补 F2 的 driver 测试,F3、F4 改 README,rebase 到 cloud#391 和 #394 之上,并逐条处理 8 条交叉点。详情见 ora-space/circus#1228。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants