Repository navigation
Conversation
…t held session On startup the execution service keeps the held session of an interactive workflow node parked awaiting input, instead of ending it with the rest. Its Agent went with the old process, and the pinned Node can neither load a provider session nor keep the Agent's own session store, so the session's next turn starts a new Agent on the same checkout, as desktop's rebuild_provider_session does after a failed session/load: - Each workspace records the Node clone execution behind its checkout; Node resolves the checkout from it in its own ledger, also on a new incarnation. The kept run's workspace is reattached on startup (Driver.Reattach), so every later session of the run, not only the recovered one, starts on the same directory with its uncommitted edits instead of a fresh clone. - The new Agent is first sent a handoff of the recorded conversation (user messages, answers, tool calls by name and outcome only) as its own block; the turn records only the typed prompt. - The page keeps one conversation: the earlier settled history comes first, the new Agent's records follow it, and the handoff block is never published. - A new Agent that could not start, Node's checkout refusals included, fails the turn and leaves the session waiting for another; a checkout that no longer exists ends the session with workspace_missing.
Desktop keeps a node parked awaiting input across a restart (is_awaiting_input) and fails the rest; v2 failed every parked node. Now the restart sweep keeps a pending root interactive Agent attempt with a session and a parked summary, and the host rebuilds its process-local state (turns, last and accepted answers, follow-up numbering) from the row and the session's recorded turns, so the starter can talk to it, stop it or complete it again. Nothing is replayed: a run with an executing root Agent, this node's own follow-up included, still fails as a whole. A parked node whose executor no longer resolves or whose session is gone fails (its row moves back to running so the engine can fail it) and its session is ended. The sessions execution recovery kept for nodes the sweep failed are ended before any run is resumed, so sessions the resumed runs start are never touched. A node whose checkout is gone fails as interactive_session. The README says what the recovered Agent does not have: the handoff carries no tool inputs or outputs, and the first turn after a restart has no prompt cache.
A Node that exits or loses its connection while the server keeps running used to end every held interactive session with it, leaving the node only completable with its last normal answer. It now takes the restart's rebuild path: the held session stays open (the lost Node's process tree is retired through its supervisor at once), the run's checkouts outlive the lost incarnation, and the session's next turn starts a new Agent on the original checkout with the recorded conversation. A turn that finds its Node already gone before reaching the Agent (execution.ErrNodeLost) is handed to the replacement Agent at once; a turn executing or being sent when the Node died fails and is not replayed, and the following one rebuilds. A first turn lost with its Node has nothing to hand over and fails as before, and an Agent process crash with its Node alive still ends the session. Conversations and task runs keep ending; a turn finding its Node gone now reports agent_transport_closed for them too. The README records the changed failure semantics.
521ba4d to
38f5b89
Compare
zhangxinhao
left a comment
There was a problem hiding this comment.
LGTM(38f5b89)
@ObsisMc 我看了这个 PR,理解它是要让合法的交互等待跨 v2-server 重启保留,并在下一轮用原 checkout 和历史交接重建 Agent;Node 丢失时也采用同样的恢复路径。可以合并,建议补齐两处恢复边界:Node 退役后其他等待会话无法重建,以及 checkout 丢失后直接完成节点会让后继静默重新 clone。定向 race 测试通过,数据库恢复测试因环境缺少配置而跳过。
- 🔧 1 处小改已写成建议(P2-1):见行内 suggestion,可以一键采纳,也可以不改或留到以后。
- 💬 其他建议:P2-2 目录丢失时直接完成会静默重新克隆
- 测试:
go test -race -timeout=5m -run 'Test(Handoff|RecoveredHistory)' ./internal/executionruntime ./internal/agentruntime通过;go test -race -timeout=5m -run 'TestDriverIPC(Disconnect|EndDisconnected|Rebuild|WorkflowSessions|FullTurn)' ./internal/agentruntime通过;go test -race -timeout=5m -v -run 'Test(RecoveryKeepsTheSessionOfAParkedInteractiveNode|WorkspaceKeepsItsCloneExecution|RecoveredSession|RunOfANodeParkedOnAnEndedSession|HeldSessionContinuesOnANewAgent|HeldSessionOutlivesANode|FirstTurnLostWithItsNode|ParkedNodeSurvivesARestart|RestartFailsA|MissingWorkspaceFailsTheRecoveredNode|UnstartedRecoveryKeepsTheNodeWaiting|ParkedNodeThatCannotContinue)' ./internal/store ./internal/executionruntime ./internal/workflowhost没跑(已尝试运行,相关用例均因未设置 TEST_DATABASE_URL 而跳过,不能视为恢复行为验证通过。);没跑全量和 e2e。
涉及数据迁移、并发核心,建议 owner 再看一眼。
以上都是建议,怎么处理由你决定。
自动检视 · 基于 38f5b89 · 第 1 轮(全量)
| if a.ended || a.ending { | ||
| return nil, false, execution.ErrSessionEnded | ||
| } | ||
| // Checked before broken: a Node's death also marks its sessions broken. | ||
| select { | ||
| case <-a.project.dead: | ||
| return nil, false, execution.ErrNodeLost | ||
| default: | ||
| } |
There was a problem hiding this comment.
P2-1(建议) 这里先检查 ended/ending,会漏掉已经退役的 Node。同一项目有两个等待节点时,第一个节点重建或某个执行中的会话触发 stopAgent,closeResources 会对所有旧会话调用 requestEnd 和 markStopped;另一个节点随后追问就返回 ErrSessionEnded,运行时将其永久关闭,无法走新加的重建路径。建议先判断 Node 是否丢失,再判断物理会话是否结束,并补一个多个等待会话的回归测试。
| if a.ended || a.ending { | |
| return nil, false, execution.ErrSessionEnded | |
| } | |
| // Checked before broken: a Node's death also marks its sessions broken. | |
| select { | |
| case <-a.project.dead: | |
| return nil, false, execution.ErrNodeLost | |
| default: | |
| } | |
| // Check Node loss first: retirement also marks its sessions ending or ended. | |
| select { | |
| case <-a.project.dead: | |
| return nil, false, execution.ErrNodeLost | |
| default: | |
| } | |
| if a.ended || a.ending { | |
| return nil, false, execution.ErrSessionEnded | |
| } |
可直接采纳,也可以不改。
| // A checkout that is gone is left out, and the recovered session's turn reports it; any | ||
| // other doubt reattaches, so the recovered Agent never lands in a fresh clone. | ||
| if _, err := os.Stat(at.Path); !errors.Is(err, fs.ErrNotExist) { | ||
| s.agent.Reattach(r.Session.WorkspaceID, at.CloneExecution, at.Path) | ||
| } |
There was a problem hiding this comment.
P2-2(建议) checkout 在重启前被删除时,这里跳过 Reattach,但仍保留运行的 workspace 映射。如果用户重启后直接完成节点,CompleteNode 只调用 End,不会经过 recovery 的目录检查;后继进入 prepareCheckout 时找不到已记录的 checkout,便重新 clone,在缺失此前修改的目录上继续运行。建议让缺失 checkout 的状态同时约束直接完成和后继启动路径,按约定失败关闭,并补一个“不追问、直接完成”的缺目录测试。
EricWvi
left a comment
There was a problem hiding this comment.
owner 验收第 1 轮(38f5b897):0H/2M/5L。请修 F1(Node 不认识记下的 checkout 时按已不在处理),补 F2 的 driver 测试,F3、F4 改 README,rebase 到 cloud#391 和 #394 之上,并逐条处理 8 条交叉点。详情见 ora-space/circus#1228。
Interactive Agent nodes parked awaiting input now survive a v2-server restart, following desktop's selective recovery (desktop
docs/workflows/workflow-runs.md, "Recovery after restart";crates/backend/src/workflow/run/engine.rsis_awaiting_input). Before this PR, a restart failed every parked node.Refs ora-space/circus#1228
v2, does not involve specs.
Read this first
c5265072) cannot reattach the original ACP session. Its protocol has nosession/loadentry, and it deletes the Agent's own session store when the session ends. So after a restart the node's next turn starts a new Agent on the same checkout and sends it a handoff of the recorded conversation first. This is what desktop does withrebuild_provider_sessionwhensession/loadfails.render_handoff.session/load) is tracked separately in Node 重启后恢复 Agent 会话:Node 协议支持接回已有 ACP 会话(对话、任务运行、工作流) circus#1230.session_ended: true). Now the session stays open and takes the same rebuild path as after a restart. Details below.Commits
Each commit builds and passes its packages' tests on its own.
store.RecoverExecutionskeeps the held session of an interactive node parked in a running run (pending row,interactivesummary, no unfinished turn). It still ends every other session.v2_workspaces.clone_execution).Driver.Reattachhands the run's recorded checkout to the driver. Every later session of the run then starts on the same directory, uncommitted edits included, instead of a fresh clone. Node resolves the checkout from its own ledger across incarnations; this was verified on a real Node with both graceful and SIGKILL restarts before implementation.NotStarted), the turn fails and the session keeps waiting for a retry. A checkout that no longer exists ends the session withworkspace_missing.interactive_session.agent_transport_closed, and the lost Node's process tree is retired through its supervisor at once.execution.ErrNodeLost) runs at once on a new Agent with the handoff.interactive_session, and an Agent process crash with its Node alive still ends the session. Conversations and task runs still end. The one visible difference for them is that a turn finding its Node gone now reportsagent_transport_closedinstead ofagent_session_unavailable/agent_dispatch_failed.Out of scope
Tests
TEST_DATABASE_URLplusV2_NODE_ASSETS_DIR:TestRealInteractiveWaitSurvivesARestartTestRealSuccessorAfterARestartRunsInTheRunsCheckoutTestRealInteractiveNodeOutlivesItsNodeGOMAXPROCS=2 go test -race -p 1over store, workflowhost, executionruntime, agentruntime, boot and execution passes on the head. Every commit was also tested on its own.TestRetryWaitAbandonedWhenAContainerFailsTheRunfails intermittently, on upstream main5416335aas well (2 of 8 runs there), unrelated to this change.Screenshots
Real v2-server, real Node and OpenCode with the fixture model, frontend-v2. v2-server was stopped with SIGTERM and started again between 1 and 2.