Skip to content

[v2] Close the loop on parallel branches: slots until End is confirmed, failed rounds stop, Skill name clashes - #394

Open
zhangxinhao wants to merge 17 commits into
ora-space:mainfrom
zhangxinhao:feat/1226-parallel-join-cancel
Open

zhangxinhao wants to merge 17 commits into
ora-space:mainfrom
zhangxinhao:feat/1226-parallel-join-cancel

Conversation

@zhangxinhao

@zhangxinhao zhangxinhao commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Refs ora-space/circus#1226

v2,不涉及 specs.

What it does

Closes the loop on parallel branches after #382 (sibling Agents run concurrently in the shared checkout), and takes the Lows the owner left for ora-space/circus#1178.

  • Session slots are released only once End is confirmed (owner's L1). Three paths freed a run's session slot before Node confirmed the session ended: an automatic attempt whose End failed, an interactive turn that ended its session, and releaseOrphans, which dropped the parked node before ending it. A resume admitted right after a sibling failed the run could then be refused with checkout_session_limit. Now a session nobody tracks any more whose End was not confirmed lingers in the run's state: it counts against the limit and, while the run runs, End is retried in the background with backoff (1 s, doubling up to 30 s). Once the run has failed, been cancelled or finished, the slot is given back without another End (owner's decision at acceptance): every End of a session whose stop was not proven lets the execution service rebuild the project's Node, which also ends the project's other sessions, and a finished run has no use for the slot. A parked interactive node keeps its slot and also retries in the background, since a stopped turn may leave nothing to complete. releaseOrphans ends the session before giving the slot back. An End that reports the session unknown or already ended counts as confirmed. The run's state is kept while a session lingers.
  • A failed Loop round stops its executing siblings. When a node in a round fails, the store already cancels the round's other attempts and fails the Loop and the run. Their turns used to run to completion: they kept changing the run's directory after the failure and held slots a resume needed, for results that are discarded. FailWorkflowEngineNode now reports the scope it closed; the engine passes it through a new optional FailedScopeDispatcher, and the host fences the scope and stops its turns in the background, as LoopExit does, without waiting for confirmation. This deviates from desktop, which lets those turns finish (scoped_node.rs:188-222 at b373be51). The deviation is documented in the workflow README. Root-scope siblings are unaffected: they keep running and their results are kept (desktop D2).
  • Skills under one name. The projection is the union of the live sessions' Skills. A starting session that selected a different Skill under a name a live sibling projects used to replace the file that sibling's Agent may be reading. Now the same package version or the same document is shared; anything else refuses the starting session with workflow_skill_name_conflict and leaves the sibling's file alone. Only siblings whose projection succeeded count, so of two clashing sessions in one wave only the one projecting second is refused.
  • Tests for the surviving mutants (owner's L2), plus a fix for an order-dependent test (below).
  • Rebased on [v2] Show nodes waiting for a session slot instead of running #389 (the "waiting for a session slot" state). A session whose End is still being retried counts against the limit, so a sibling queued behind it reads as waiting, not running.

Already in place, now pinned by tests: a join waits for both predecessors and starts once when they complete together; an inactive branch does not block (engine projection); a root failure keeps a running sibling's result; cancel stops every turn; LoopExit stops and waits for its round.

Choices for review

  • Stopping a failed round's turns instead of letting them finish (desktop's behaviour). This is not a long-term contract, so it is easy to revert if you prefer parity.
  • A lingering End is retried, without a cap, only while its run runs. Giving up earlier would either leak the slot or free it while Node may still hold the session. Not-found and already-ended count as confirmed, so a session that really is gone does not loop. An End that can never be confirmed holds its slot until the run is cancelled or the process restarts; with eight of them, new attempts keep showing "waiting for a session slot" (documented in the README).
  • Refusing the starting session on a Skill name clash. Overwriting breaks a running sibling, and desktop freezes one Skill set per run, where a clash cannot happen silently.

Tests

  • unended_test.go: automatic attempt keeps its slot until End returns (M13) and until a retry confirms an unconfirmed End; a stopped interactive turn keeps its slot, and a background retry frees it with no completion needed; an orphan with an unconfirmed End keeps its slot across an immediate resume; a gone session frees the slot at once.
  • parallel_paths_test.go: two siblings completing together join once (M17); Close drops a sibling queued behind a parked session (M15); an unconfirmed cancel still drops a queued sibling (the first wakeup in CancelRun); LoopExit drops a round member waiting for a slot (the CleanupLoopExit wakeup); the run forgets its state only after a lingering End is confirmed; a failed round leaves root siblings running with their results; a Start refused after a cancellation gave up closes the run as cancelled.
  • TestLoopChildFailureAndCancellationCloseScope now expects the failed round's sibling to be stopped.
  • agentruntime: a different Skill under a live sibling's name is refused and the sibling's file is untouched; two clashing sessions of one wave refuse only one (20 rounds); the shared-Skill case.
  • frontend-v2 e2e (real Node, skipped without assets like the other interactive specs): two interactive siblings park side by side; the parallel carousel switches between them, and each session shows only its own task and replies.
  • TestRetryWaitAbandonedWhenAContainerFailsTheRun assumed the root Agent starts before the Iteration worker. They have run concurrently since [v2] Run a workflow run's sibling Agents concurrently in the shared checkout #382, and the test failed 2 of 30 runs on main. It now takes both starts by node (60/60).

Mutation: the owner listed M13, M15, M17 and the two wakeups; each new test fails when its branch is mutated. M16 and M22–24 were not spelled out, so I swept 20 mutants over the same code; every survivor is equivalent (a check repeated just before, or a wakeup an earlier one already did). One wakeup in dropInteractive turned out never to change a count and was removed. M17 alone survives removing the engine gate, because the store's unique attempt row also blocks a second join.

Verification

  • On upstream/main at f4560815: gofmt -l . clean; go vet ./...; go test -race -count=1 ./... with PostgreSQL: pass.
  • Real Node (desktop c5265072, OpenCode, fixture model): TestRealWorkflowSiblingAgentsShareTheCheckout, TestRealWorkflowMCPLifecycleGate, TestRealAgentCloneInstallToolAndEnd, TestRealAgentLongHistoryPostgresAfterRestart: pass.
  • frontend-v2: npm run check passes. With real Node assets, the e2e specs this touches pass (workflow-interactive, workflow-resume, workflow-retry, task-workflow-stop: 11 passed); the full suite is left to CI.

Not covered

  • A round closed by the container path itself (applyContainer with a failed scope: round limit, cleanup errors) does not stop turns. Nothing is executing at those points.

Three paths gave a run's session slot back before Node confirmed the session
ended: an automatic attempt whose End failed, an interactive turn that ended
the session, and releaseOrphans, which dropped the parked node before ending
it. A resume admitted right after a sibling failed the run could then ask
Node for more sessions than the checkout admits and be refused with
checkout_session_limit.

A session nobody tracks any more whose End was not confirmed now lingers in
the run's state, counts against the limit and is ended again in the
background with backoff; only a confirmed End frees its slot. An interactive
node keeps its slot until End is confirmed, and releaseOrphans ends the
session before it gives the slot back. The run's state is kept while a
session lingers.
When a node inside a Loop round fails, the store cancels the round's other
attempts and fails the Loop and the run, but their executing turns ran on
to their end: they kept changing the run's directory after the run failed
and held session slots a resume needed, for results that are discarded.

FailWorkflowEngineNode now reports the scope it closed, and the engine
hands it to the host through FailedScopeDispatcher. The host fences the
scope and stops its turns in the background, as on LoopExit, without
waiting for confirmation. Desktop lets those turns finish; this deviation
is documented in the workflow README, which also drops the stale claim that
the host runs a run's Agents one at a time.
…er order

TestRetryWaitAbandonedWhenAContainerFailsTheRun assumed the root Agent starts
before the Iteration's worker. Both are in one wave and have executed
concurrently since sibling Agents run in parallel, so the test failed about
once in fifteen runs. Take both starts by node.
Dropping a tracked interactive node changes no slot count: a gone session
gave its slot back, and woke the waiters, when its End was confirmed, and an
unconfirmed one moves to the lingering sessions.
Cover what mutation testing showed unpinned in the parallel scheduling:
two siblings completing at once join exactly once; Close and a cancellation
whose stop is unconfirmed both drop a sibling queued behind a parked
session; LoopExit drops a round member waiting for a slot; a run forgets its
state only after a lingering End is confirmed; a failed Loop round leaves
root siblings running and keeps their results; and a Start refused after a
cancellation gave up waiting for it closes the run as cancelled.
A run's sessions share one checkout, and its Skill projection is the union of
what the live sessions selected. When a starting session selected a
different Skill under the name of one a live sibling projects, the starting
session's version replaced the file the sibling's Agent may be reading.

The same package version or the same document under one name is now shared;
anything else refuses the starting session with workflow_skill_name_conflict
and leaves the sibling's file as it was. Siblings that clash only with each
other are still left out without refusing an unrelated start.
Two interactive nodes park side by side; the run view offers both in its
parallel carousel, and opening each one's session shows only its own task
and replies. Needs prepared Node assets, like the other interactive specs.
Sessions of one wave are admitted to the checkout before they project their
Skills. Counting every admitted sibling meant two sessions selecting
different Skills under one name could both be refused: the first against a
sibling that had put nothing on the checkout yet, the second against the
first, which was still registered when it released the projection lock.

A session now counts as a live sibling only once its projection succeeded,
so of two clashing selections only the one projecting second is refused.
…ground

A stopped interactive turn whose End was not confirmed kept the node's slot
until a completion ended the session again, but a stopped turn may leave
nothing to complete, so siblings queued for the slot waited until the run
was cancelled. Such an End is now retried in the background like a
lingering session's, and the node gives its slot back once it is confirmed.

An End that reports the session unknown or already ended now counts as
confirmed, so it frees the slot at once instead of being retried forever,
and a retry that wakes as the host shuts down no longer calls End and holds
Close up for a stop timeout.

@ObsisMc ObsisMc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 239bc9f1 (on main f4560815). No High or Medium findings; good to merge from my side. Three Low items and two notes below. The first Low is a README sentence I would fix before merging.

1. Slots until End is confirmed (L1)

I traced every place a session stops being tracked and found no leak, no double release and no slot that is never given back.

  • How the count works. sessionsInUse is derived from state (attempts, tracked nodes, unended), not kept as a counter, so nothing can be released twice. A session moves from an attempt or a node to unended inside one critical section (endSessionTurn, dropInteractive), so the count never dips in between.
  • Automatic attempts. Both End sites in runAgent go through Host.endTurn, which starts the retry. runState.endTurn is only ever called with nil, so no path lingers a session without a retry.
  • Interactive nodes. A node's own retry stops once the node is dropped or its session is confirmed gone, and a dropped node hands the session to linger. If the node's retry confirms just as the node is dropped, the lingering entry ends the session once more and clears on that call. That is a brief over-count, never an under-count.
  • Cancel paths still ignore the result of h.end (settleInteractiveTurn and two sites in runFollowUp). The node stays tracked there, so releaseOrphans picks it up from forgetFinished and applies the new rule. Fine as it is.
  • forgetFinished and resume. The state is kept while unended is not empty, and register / admitResume reuse the existing state, so a retry never works on a replaced one. admitResume does not look at unended, which is right: the resume is admitted and the old session just counts.
  • Shutdown. The retry waits on h.ctx and checks it again before calling End. An End already in flight is bounded by stopTimeout, as in runAgent.
  • cloud#376 conclude and cloud#380 held. No conflict. conclude marks a session ended only after its physical stop is proven, so End == nil does prove Node let go. For a session whose stop was not proven, the runtime leaves it to "an End that obtains stop proof", and the retry is that End. held is cleared in conclude and plays no part in the count.

2. A failed Loop round stops its executing siblings

The code does what the description says.

  • Not waiting for the stop is fine. A resume is fenced by the tracked attempts (admitResume, settling), not by the stop: until a stopped turn has really ended, the resume is refused with run_settling. If a stop is not confirmed, the turn runs to its end, which is desktop's behaviour, and keeps the resume refused until then.
  • The stopped sibling's late failure changes nothing. The run is no longer running, so no retry is scheduled, and the row is already cancelled, so FailNode is a no-op.
  • Scope ids are not reused. A resume restarts a cleared Loop from round 1 with new scopes, so closedScopes never fences a live round.
  • Interactive nodes. Interactive mode applies at the root only (a Loop member runs as an automatic node), so a failed round never holds one. A root interactive node parked beside the Loop is released by releaseOrphans as before, now ending its session before giving the slot back.
  • Root siblings are not selected (a.scope == scopeID) and their results are kept. TestAFailedRoundLeavesRootSiblingsRunning fails when the selector is widened.
  • "Nothing is executing" on the container path holds: containers cannot nest, so a round closed by applyContainer has no running member.

My view on the behaviour change. I agree with stopping. The results are discarded either way, the turns stop changing the shared directory sooner, and a resume no longer waits a whole turn behind run_settling. The cost: a stopped turn can leave half-written files in the directory, where desktop would let the turn finish its edits. A resume keeps files and restarts the Loop from round 1, so those files are what the rerun starts from. It is a deliberate deviation from desktop, so I suggest @EricWvi confirms it at acceptance.

3. Skills under one name

Consistent with the union from cloud#382. Live selections are still read inside the skillsMu section that projects them, and projected only narrows them to siblings whose files are on the checkout. A refusal returns before any file is touched, so the sibling's file and the state file stay as they were. A refused session stays in owners until it is ended, but never counts, because it is not in projected.

4. Mutation checks (all restored)

17 mutants, each run with GOMAXPROCS=2 go test -p 1 -race:

Mutant Result
sessionsInUse ignores unended killed
forgetFinished ignores unended killed
endConfirmed accepts only nil killed
automatic attempt does not linger killed
dropped node does not linger killed
parked node's End is not retried killed
confirmed retry does not wake waiters killed
releaseOrphans gives the slot back before End killed
StopFailedScope stops nothing killed
StopFailedScope stops every turn killed
wakeup in CancelRun removed killed
wakeup in closeScope removed killed
M13: slot given back before End killed
M15: slot waiter ignores shutdown killed (by the 10-minute test timeout)
Skill clash not refused killed
siblings still preparing count as live killed
End retry ignores shutdown while it waits survived (see Low 2)

TestRetryWaitAbandonedWhenAContainerFailsTheRun: the assertions are not weaker. It now takes both starts by node instead of assuming their order; everything after is unchanged.

Not checked: M17, and the claim that the other survivors of your sweep are equivalent.

Low

  1. v2-server/README.md:782 now contradicts the code. The bullet is titled "与 desktop 相同" and ends with "Loop 轮次因子节点失败关闭时,已在执行的同轮兄弟收为 cancelled,它的会话执行完本轮后结束". After this PR the turn is stopped. The same bullet says a waiting attempt starts when "有会话结束", which now means its End was confirmed. internal/workflow/README.md is correct; this one was missed.
  2. No test pins that the End retry stops at shutdown. With case <-h.ctx.Done() removed from endUntilConfirmed, the whole workflowhost package still passes, because every test that makes a session linger uses a 20 ms retry. In production Close would then wait up to 30 s. A test with endRetry = time.Hour and a lingering session catches it: it passes 5 of 5 on this head and fails on the mutant. Sketch:
    f.refuseEndOf(t, func() string { return ended.Load().(string) })
    f.host.endRetry = time.Hour
    // run twoAgents to success with the last session's End refused, awaitIdle, then:
    closed := make(chan struct{})
    go func() { f.host.Close(); close(closed) }()
    select {
    case <-closed:
    case <-time.After(3 * time.Second):
        t.Fatal("Close waited for the retry of a lingering End")
    }
  3. endParked and CompleteNode still test err == nil, not endConfirmed. The real service returns nil for an ended session, so only ErrNotFound differs: a parked node whose session is unknown can be neither cancelled nor completed. This was so before the PR. I mention it because the PR now defines "confirmed" in one place.

Notes, not requests

  • The retry also brings the Node rebuild forward. When the runtime's 30 s stop attempt fails, it marks the project's Node for rebuild, and the project's next session or a later End retires the process tree, which stops the project's other running sessions. The background retry is that later End, about a second after the first one fails. Before this PR the rebuild waited for whatever came next in the project. The Node was already marked for rebuild, so I think this is acceptable, but a slow stop of one session now ends its running siblings sooner.
  • No cap on the retry is the right call, for the reason you give. One consequence to know: if eight sessions linger, new attempts of the run wait and read "waiting for a session slot" until the run is cancelled.

Tests and CI

  • go vet clean; internal/workflowhost, internal/workflow, internal/agentruntime and internal/store pass with -race on this head.
  • CI on 239bc9f1 is green: server, frontend, e2e.
  • The two screenshots match the description: both interactive siblings parked side by side, and the session of 规划 after switching.
  • Not run: real Node, e2e, the full Go module.

@zhangxinhao

Copy link
Copy Markdown
Contributor Author

Thanks. All three Lows are addressed in 239bc9f1..3510d861: README fixed (17a00064), your shutdown test added (398802f9, fails without case <-h.ctx.Done()), and CompleteNode/endParked now use endConfirmed, with a test where End reports the session unknown (3510d861, both sites fail when reverted to err == nil). Details on ora-space/circus#1226.

@EricWvi EricWvi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

owner 验收第 1 轮(3510d861):0H/1M/4L。请让后台 End 重试只在 run 运行中进行(M1),README 补上代价和卡住时的表现(L1、L2),补 L3 的测试。详情见 ora-space/circus#1226。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants