Repository navigation
[v2] Close the loop on parallel branches: slots until End is confirmed, failed rounds stop, Skill name clashes - #394
Conversation
Three paths gave a run's session slot back before Node confirmed the session ended: an automatic attempt whose End failed, an interactive turn that ended the session, and releaseOrphans, which dropped the parked node before ending it. A resume admitted right after a sibling failed the run could then ask Node for more sessions than the checkout admits and be refused with checkout_session_limit. A session nobody tracks any more whose End was not confirmed now lingers in the run's state, counts against the limit and is ended again in the background with backoff; only a confirmed End frees its slot. An interactive node keeps its slot until End is confirmed, and releaseOrphans ends the session before it gives the slot back. The run's state is kept while a session lingers.
When a node inside a Loop round fails, the store cancels the round's other attempts and fails the Loop and the run, but their executing turns ran on to their end: they kept changing the run's directory after the run failed and held session slots a resume needed, for results that are discarded. FailWorkflowEngineNode now reports the scope it closed, and the engine hands it to the host through FailedScopeDispatcher. The host fences the scope and stops its turns in the background, as on LoopExit, without waiting for confirmation. Desktop lets those turns finish; this deviation is documented in the workflow README, which also drops the stale claim that the host runs a run's Agents one at a time.
…er order TestRetryWaitAbandonedWhenAContainerFailsTheRun assumed the root Agent starts before the Iteration's worker. Both are in one wave and have executed concurrently since sibling Agents run in parallel, so the test failed about once in fifteen runs. Take both starts by node.
Dropping a tracked interactive node changes no slot count: a gone session gave its slot back, and woke the waiters, when its End was confirmed, and an unconfirmed one moves to the lingering sessions.
Cover what mutation testing showed unpinned in the parallel scheduling: two siblings completing at once join exactly once; Close and a cancellation whose stop is unconfirmed both drop a sibling queued behind a parked session; LoopExit drops a round member waiting for a slot; a run forgets its state only after a lingering End is confirmed; a failed Loop round leaves root siblings running and keeps their results; and a Start refused after a cancellation gave up waiting for it closes the run as cancelled.
A run's sessions share one checkout, and its Skill projection is the union of what the live sessions selected. When a starting session selected a different Skill under the name of one a live sibling projects, the starting session's version replaced the file the sibling's Agent may be reading. The same package version or the same document under one name is now shared; anything else refuses the starting session with workflow_skill_name_conflict and leaves the sibling's file as it was. Siblings that clash only with each other are still left out without refusing an unrelated start.
Two interactive nodes park side by side; the run view offers both in its parallel carousel, and opening each one's session shows only its own task and replies. Needs prepared Node assets, like the other interactive specs.
Sessions of one wave are admitted to the checkout before they project their Skills. Counting every admitted sibling meant two sessions selecting different Skills under one name could both be refused: the first against a sibling that had put nothing on the checkout yet, the second against the first, which was still registered when it released the projection lock. A session now counts as a live sibling only once its projection succeeded, so of two clashing selections only the one projecting second is refused.
…ground A stopped interactive turn whose End was not confirmed kept the node's slot until a completion ended the session again, but a stopped turn may leave nothing to complete, so siblings queued for the slot waited until the run was cancelled. Such an End is now retried in the background like a lingering session's, and the node gives its slot back once it is confirmed. An End that reports the session unknown or already ended now counts as confirmed, so it frees the slot at once instead of being retried forever, and a retry that wakes as the host shuts down no longer calls End and holds Close up for a stop timeout.
ObsisMc
left a comment
There was a problem hiding this comment.
Reviewed 239bc9f1 (on main f4560815). No High or Medium findings; good to merge from my side. Three Low items and two notes below. The first Low is a README sentence I would fix before merging.
1. Slots until End is confirmed (L1)
I traced every place a session stops being tracked and found no leak, no double release and no slot that is never given back.
- How the count works.
sessionsInUseis derived from state (attempts, tracked nodes,unended), not kept as a counter, so nothing can be released twice. A session moves from an attempt or a node tounendedinside one critical section (endSessionTurn,dropInteractive), so the count never dips in between. - Automatic attempts. Both End sites in
runAgentgo throughHost.endTurn, which starts the retry.runState.endTurnis only ever called withnil, so no path lingers a session without a retry. - Interactive nodes. A node's own retry stops once the node is dropped or its session is confirmed gone, and a dropped node hands the session to
linger. If the node's retry confirms just as the node is dropped, the lingering entry ends the session once more and clears on that call. That is a brief over-count, never an under-count. - Cancel paths still ignore the result of
h.end(settleInteractiveTurnand two sites inrunFollowUp). The node stays tracked there, soreleaseOrphanspicks it up fromforgetFinishedand applies the new rule. Fine as it is. forgetFinishedand resume. The state is kept whileunendedis not empty, andregister/admitResumereuse the existing state, so a retry never works on a replaced one.admitResumedoes not look atunended, which is right: the resume is admitted and the old session just counts.- Shutdown. The retry waits on
h.ctxand checks it again before calling End. An End already in flight is bounded bystopTimeout, as inrunAgent. - cloud#376
concludeand cloud#380held. No conflict.concludemarks a session ended only after its physical stop is proven, soEnd == nildoes prove Node let go. For a session whose stop was not proven, the runtime leaves it to "an End that obtains stop proof", and the retry is that End.heldis cleared inconcludeand plays no part in the count.
2. A failed Loop round stops its executing siblings
The code does what the description says.
- Not waiting for the stop is fine. A resume is fenced by the tracked attempts (
admitResume,settling), not by the stop: until a stopped turn has really ended, the resume is refused withrun_settling. If a stop is not confirmed, the turn runs to its end, which is desktop's behaviour, and keeps the resume refused until then. - The stopped sibling's late failure changes nothing. The run is no longer running, so no retry is scheduled, and the row is already cancelled, so
FailNodeis a no-op. - Scope ids are not reused. A resume restarts a cleared Loop from round 1 with new scopes, so
closedScopesnever fences a live round. - Interactive nodes. Interactive mode applies at the root only (a Loop member runs as an automatic node), so a failed round never holds one. A root interactive node parked beside the Loop is released by
releaseOrphansas before, now ending its session before giving the slot back. - Root siblings are not selected (
a.scope == scopeID) and their results are kept.TestAFailedRoundLeavesRootSiblingsRunningfails when the selector is widened. - "Nothing is executing" on the container path holds: containers cannot nest, so a round closed by
applyContainerhas no running member.
My view on the behaviour change. I agree with stopping. The results are discarded either way, the turns stop changing the shared directory sooner, and a resume no longer waits a whole turn behind run_settling. The cost: a stopped turn can leave half-written files in the directory, where desktop would let the turn finish its edits. A resume keeps files and restarts the Loop from round 1, so those files are what the rerun starts from. It is a deliberate deviation from desktop, so I suggest @EricWvi confirms it at acceptance.
3. Skills under one name
Consistent with the union from cloud#382. Live selections are still read inside the skillsMu section that projects them, and projected only narrows them to siblings whose files are on the checkout. A refusal returns before any file is touched, so the sibling's file and the state file stay as they were. A refused session stays in owners until it is ended, but never counts, because it is not in projected.
4. Mutation checks (all restored)
17 mutants, each run with GOMAXPROCS=2 go test -p 1 -race:
| Mutant | Result |
|---|---|
sessionsInUse ignores unended |
killed |
forgetFinished ignores unended |
killed |
endConfirmed accepts only nil |
killed |
| automatic attempt does not linger | killed |
| dropped node does not linger | killed |
| parked node's End is not retried | killed |
| confirmed retry does not wake waiters | killed |
releaseOrphans gives the slot back before End |
killed |
StopFailedScope stops nothing |
killed |
StopFailedScope stops every turn |
killed |
wakeup in CancelRun removed |
killed |
wakeup in closeScope removed |
killed |
| M13: slot given back before End | killed |
| M15: slot waiter ignores shutdown | killed (by the 10-minute test timeout) |
| Skill clash not refused | killed |
| siblings still preparing count as live | killed |
| End retry ignores shutdown while it waits | survived (see Low 2) |
TestRetryWaitAbandonedWhenAContainerFailsTheRun: the assertions are not weaker. It now takes both starts by node instead of assuming their order; everything after is unchanged.
Not checked: M17, and the claim that the other survivors of your sweep are equivalent.
Low
v2-server/README.md:782now contradicts the code. The bullet is titled "与 desktop 相同" and ends with "Loop 轮次因子节点失败关闭时,已在执行的同轮兄弟收为cancelled,它的会话执行完本轮后结束". After this PR the turn is stopped. The same bullet says a waiting attempt starts when "有会话结束", which now means its End was confirmed.internal/workflow/README.mdis correct; this one was missed.- No test pins that the End retry stops at shutdown. With
case <-h.ctx.Done()removed fromendUntilConfirmed, the wholeworkflowhostpackage still passes, because every test that makes a session linger uses a 20 ms retry. In productionClosewould then wait up to 30 s. A test withendRetry = time.Hourand a lingering session catches it: it passes 5 of 5 on this head and fails on the mutant. Sketch:f.refuseEndOf(t, func() string { return ended.Load().(string) }) f.host.endRetry = time.Hour // run twoAgents to success with the last session's End refused, awaitIdle, then: closed := make(chan struct{}) go func() { f.host.Close(); close(closed) }() select { case <-closed: case <-time.After(3 * time.Second): t.Fatal("Close waited for the retry of a lingering End") }
endParkedandCompleteNodestill testerr == nil, notendConfirmed. The real service returnsnilfor an ended session, so onlyErrNotFounddiffers: a parked node whose session is unknown can be neither cancelled nor completed. This was so before the PR. I mention it because the PR now defines "confirmed" in one place.
Notes, not requests
- The retry also brings the Node rebuild forward. When the runtime's 30 s stop attempt fails, it marks the project's Node for rebuild, and the project's next session or a later End retires the process tree, which stops the project's other running sessions. The background retry is that later End, about a second after the first one fails. Before this PR the rebuild waited for whatever came next in the project. The Node was already marked for rebuild, so I think this is acceptable, but a slow stop of one session now ends its running siblings sooner.
- No cap on the retry is the right call, for the reason you give. One consequence to know: if eight sessions linger, new attempts of the run wait and read "waiting for a session slot" until the run is cancelled.
Tests and CI
go vetclean;internal/workflowhost,internal/workflow,internal/agentruntimeandinternal/storepass with-raceon this head.- CI on
239bc9f1is green: server, frontend, e2e. - The two screenshots match the description: both interactive siblings parked side by side, and the session of 规划 after switching.
- Not run: real Node, e2e, the full Go module.
…it for a confirmed End
…leting or cancelling
|
Thanks. All three Lows are addressed in |
EricWvi
left a comment
There was a problem hiding this comment.
owner 验收第 1 轮(3510d861):0H/1M/4L。请让后台 End 重试只在 run 运行中进行(M1),README 补上代价和卡住时的表现(L1、L2),补 L3 的测试。详情见 ora-space/circus#1226。
…efused park, and a failed round's reach
Refs ora-space/circus#1226
v2,不涉及 specs.
What it does
Closes the loop on parallel branches after #382 (sibling Agents run concurrently in the shared checkout), and takes the Lows the owner left for ora-space/circus#1178.
releaseOrphans, which dropped the parked node before ending it. A resume admitted right after a sibling failed the run could then be refused withcheckout_session_limit. Now a session nobody tracks any more whose End was not confirmed lingers in the run's state: it counts against the limit and, while the run runs, End is retried in the background with backoff (1 s, doubling up to 30 s). Once the run has failed, been cancelled or finished, the slot is given back without another End (owner's decision at acceptance): every End of a session whose stop was not proven lets the execution service rebuild the project's Node, which also ends the project's other sessions, and a finished run has no use for the slot. A parked interactive node keeps its slot and also retries in the background, since a stopped turn may leave nothing to complete.releaseOrphansends the session before giving the slot back. An End that reports the session unknown or already ended counts as confirmed. The run's state is kept while a session lingers.FailWorkflowEngineNodenow reports the scope it closed; the engine passes it through a new optionalFailedScopeDispatcher, and the host fences the scope and stops its turns in the background, as LoopExit does, without waiting for confirmation. This deviates from desktop, which lets those turns finish (scoped_node.rs:188-222atb373be51). The deviation is documented in the workflow README. Root-scope siblings are unaffected: they keep running and their results are kept (desktop D2).workflow_skill_name_conflictand leaves the sibling's file alone. Only siblings whose projection succeeded count, so of two clashing sessions in one wave only the one projecting second is refused.Already in place, now pinned by tests: a join waits for both predecessors and starts once when they complete together; an inactive branch does not block (engine projection); a root failure keeps a running sibling's result; cancel stops every turn; LoopExit stops and waits for its round.
Choices for review
Tests
unended_test.go: automatic attempt keeps its slot until End returns (M13) and until a retry confirms an unconfirmed End; a stopped interactive turn keeps its slot, and a background retry frees it with no completion needed; an orphan with an unconfirmed End keeps its slot across an immediate resume; a gone session frees the slot at once.parallel_paths_test.go: two siblings completing together join once (M17);Closedrops a sibling queued behind a parked session (M15); an unconfirmed cancel still drops a queued sibling (the first wakeup inCancelRun); LoopExit drops a round member waiting for a slot (theCleanupLoopExitwakeup); the run forgets its state only after a lingering End is confirmed; a failed round leaves root siblings running with their results; a Start refused after a cancellation gave up closes the run as cancelled.TestLoopChildFailureAndCancellationCloseScopenow expects the failed round's sibling to be stopped.agentruntime: a different Skill under a live sibling's name is refused and the sibling's file is untouched; two clashing sessions of one wave refuse only one (20 rounds); the shared-Skill case.TestRetryWaitAbandonedWhenAContainerFailsTheRunassumed the root Agent starts before the Iteration worker. They have run concurrently since [v2] Run a workflow run's sibling Agents concurrently in the shared checkout #382, and the test failed 2 of 30 runs on main. It now takes both starts by node (60/60).Mutation: the owner listed M13, M15, M17 and the two wakeups; each new test fails when its branch is mutated. M16 and M22–24 were not spelled out, so I swept 20 mutants over the same code; every survivor is equivalent (a check repeated just before, or a wakeup an earlier one already did). One wakeup in
dropInteractiveturned out never to change a count and was removed. M17 alone survives removing the engine gate, because the store's unique attempt row also blocks a second join.Verification
upstream/mainatf4560815:gofmt -l .clean;go vet ./...;go test -race -count=1 ./...with PostgreSQL: pass.c5265072, OpenCode, fixture model):TestRealWorkflowSiblingAgentsShareTheCheckout,TestRealWorkflowMCPLifecycleGate,TestRealAgentCloneInstallToolAndEnd,TestRealAgentLongHistoryPostgresAfterRestart: pass.npm run checkpasses. With real Node assets, the e2e specs this touches pass (workflow-interactive,workflow-resume,workflow-retry,task-workflow-stop: 11 passed); the full suite is left to CI.Not covered
applyContainerwith a failed scope: round limit, cleanup errors) does not stop turns. Nothing is executing at those points.