Skip to content

Commit 00a8796

Browse files
Fix startup crash when a stuck workdir lock survives an interrupt (#596)
* Report a stuck workdir lock as a clear error instead of an unhandled rejection close() on the agent package releases its workdir lock only after reactor.abort()/sendQueue.drain() and the shutdown-complete race finish. A throw partway through (most likely right when an operator interrupts mid-inference, exactly when those paths are stressed) leaves the lock held forever in-process: the agent is already marked closed, so retrying close() is a silent no-op that can never release it. The next buildAgent() for that workdir then throws AgentContextLockError, and reloadIfIdle's rebuild had no try/catch around it, so the throw escaped as an unhandled rejection and crashed the process. Route every rebuild site (interrupt, reload, session rotation) through a shared close-then-check helper: a failed close now short-circuits the rebuild instead of attempting a second, doomed acquisition, and the failure surfaces as a plain-language caught error. * Document rotation's exemption and prove the fix through the real queue Session rotation was never routed through closeAgentForRebuild: it mints a fresh sessionId/workdir before rebuilding, so a leaked lock on the old workdir can never be re-acquired there. Write that reasoning down at the call site and next to closeAgentForRebuild's doc comment, since the asymmetry across the three rebuild sites needs an explanation the next reader can find. Replace the helper-only regression test with one that drives the real session-operation-queue the same way reloadIfIdle actually calls it (void enqueue(...), no awaited return value) and asserts, via a real process.on("unhandledRejection") listener, that the rejection is contained and surfaces through fatalBuildError instead of escaping. reloadIfIdle itself can't be reached in isolation without standing up the full TUI runner; that's noted at the test.
1 parent b186488 commit 00a8796

3 files changed

Lines changed: 217 additions & 20 deletions

File tree

CHANGELOG.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,15 @@ parallel copies under `docs/` or `scripts/notes/`. At cut time: rename
2929
the subtree-scoping rule for those is written and tested but not yet wired to
3030
a live call site. `task()` is unchanged and still the only spawn verb.
3131

32+
### Fixed
33+
34+
- **Interrupting a turn no longer risks a startup crash.** If an interrupt hit
35+
the agent mid-teardown, a failed close could leave its in-process workdir
36+
lock stuck held, and the immediate rebuild threw "an agent is already open"
37+
as an unhandled rejection. A failed close now short-circuits the rebuild
38+
with a clear, catchable error instead of retrying a doomed second
39+
acquisition.
40+
3241
## [0.2.107] - 2026-08-24
3342

3443
### Agent

src/tui/runner.ts

Lines changed: 73 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import {
66
defineTool,
77
createDirectorRegistry,
88
defineDirector,
9+
AgentContextLockError,
910
type Agent,
1011
} from "@intx/agent";
1112
import { noopAuditStore, permissiveAuthorize } from "@intx/agent/testing";
@@ -283,6 +284,46 @@ export function resumeTranscriptLoadErrorBlock(err: unknown): {
283284
return { type: "error", message: `Could not load prior session transcript: ${message}` };
284285
}
285286

287+
// The agent package releases its workdir lock at the very end of close(),
288+
// after reactor.abort()/sendQueue.drain() and the shutdown-complete race have
289+
// all run. If any of that throws (most likely right when an operator
290+
// interrupts mid-inference, which is exactly when those paths are under
291+
// stress), the lock is never released — and because the agent is already
292+
// marked closed internally, retrying close() is a silent no-op that can
293+
// never release it either. Every rebuild site that reuses the *same* workdir
294+
// (interrupt, reloadIfIdle) must treat that as fatal for the current rebuild
295+
// instead of calling buildAgent() again: a second createAgent() for the same
296+
// workdir is then guaranteed to throw AgentContextLockError for a lock
297+
// nothing will ever free, which is the "agent already open" crash. Session
298+
// rotation (newSession) is the one rebuild site that does NOT route through
299+
// this helper: it always points buildAgent() at a freshly minted workdir
300+
// before rebuilding, so a leaked lock on the old workdir can never be
301+
// re-acquired there — see the comment at its close() call for why.
302+
export async function closeAgentForRebuild(agent: Agent, context: string): Promise<boolean> {
303+
try {
304+
await agent.close();
305+
return true;
306+
} catch (err) {
307+
tuiLogger.debug(`agent.close during ${context} teardown failed: {error}`, {
308+
error: err instanceof Error ? err.message : String(err),
309+
});
310+
return false;
311+
}
312+
}
313+
314+
// Every rebuild site funnels its failure (a lock left held by a failed
315+
// close, or any other buildAgent failure) through here so it surfaces as a
316+
// plain-language, caught error rather than an unhandled rejection.
317+
export function agentRebuildFailure(err: unknown): Error {
318+
return err instanceof AgentContextLockError
319+
? new Error(
320+
"Could not start a new agent: the previous one did not shut down cleanly. Restart Corbits to continue.",
321+
)
322+
: err instanceof Error
323+
? err
324+
: new Error(String(err));
325+
}
326+
286327
export interface ResumeSeed {
287328
turnsUsed: number;
288329
mcpServers: ConnectedMcpServer[];
@@ -1646,21 +1687,25 @@ export async function runTUI(initialConfig: Config): Promise<number> {
16461687
if (!pendingReload || inFlight > 0) return;
16471688
pendingReload = false;
16481689
void enqueueOp(async () => {
1649-
const old = currentAgent;
1650-
await old.close().catch((err: unknown) => {
1651-
tuiLogger.debug("agent.close during reload teardown failed: {error}", {
1652-
error: err instanceof Error ? err.message : String(err),
1653-
});
1654-
});
1655-
await streamPromise.catch((err: unknown) => {
1656-
tuiLogger.debug("stream drain during reload teardown failed: {error}", {
1657-
error: err instanceof Error ? err.message : String(err),
1690+
try {
1691+
const old = currentAgent;
1692+
const closedCleanly = await closeAgentForRebuild(old, "reload");
1693+
await streamPromise.catch((err: unknown) => {
1694+
tuiLogger.debug("stream drain during reload teardown failed: {error}", {
1695+
error: err instanceof Error ? err.message : String(err),
1696+
});
16581697
});
1659-
});
1660-
currentAgent = await buildAgent();
1661-
streamPromise = consumeStream(currentAgent.stream(), streamSink);
1662-
// The rebuild made a fresh director; re-attach the active workflow.
1663-
workflowController.reattach();
1698+
if (!closedCleanly) {
1699+
throw new AgentContextLockError(workdir);
1700+
}
1701+
currentAgent = await buildAgent();
1702+
streamPromise = consumeStream(currentAgent.stream(), streamSink);
1703+
// The rebuild made a fresh director; re-attach the active workflow.
1704+
workflowController.reattach();
1705+
} catch (err) {
1706+
recordRunError(err);
1707+
fatalBuildError = agentRebuildFailure(err);
1708+
}
16641709
});
16651710
};
16661711

@@ -1812,24 +1857,23 @@ export async function runTUI(initialConfig: Config): Promise<number> {
18121857
// and salvages the buffer before that teardown, so it is never lost
18131858
// or misattributed to the rebuilt agent's next cycle.
18141859
await cycleRecorder.dispose("interrupted");
1815-
await currentAgent.close().catch((err: unknown) => {
1816-
tuiLogger.debug("agent.close during interrupt teardown failed: {error}", {
1817-
error: err instanceof Error ? err.message : String(err),
1818-
});
1819-
});
1860+
const closedCleanly = await closeAgentForRebuild(currentAgent, "interrupt");
18201861
await streamPromise.catch((err: unknown) => {
18211862
tuiLogger.debug("stream drain during interrupt teardown failed: {error}", {
18221863
error: err instanceof Error ? err.message : String(err),
18231864
});
18241865
});
1866+
if (!closedCleanly) {
1867+
throw new AgentContextLockError(workdir);
1868+
}
18251869
currentAgent = await buildAgent();
18261870
cycleRecorder.reset();
18271871
streamPromise = consumeStream(currentAgent.stream(), streamSink);
18281872
workflowController.reattach();
18291873
fatalBuildError = null;
18301874
} catch (err) {
18311875
recordRunError(err);
1832-
fatalBuildError = err instanceof Error ? err : new Error(String(err));
1876+
fatalBuildError = agentRebuildFailure(err);
18331877
}
18341878
});
18351879
};
@@ -1861,6 +1905,15 @@ export async function runTUI(initialConfig: Config): Promise<number> {
18611905
// settles, and a dead cycle's partial must land in the session that
18621906
// produced it, not the fresh one.
18631907
await cycleRecorder.dispose("rotation");
1908+
// Deliberately not routed through closeAgentForRebuild/
1909+
// agentRebuildFailure (unlike interrupt and reloadIfIdle, CL-5753):
1910+
// rotation mints a fresh sessionId/workdir below before calling
1911+
// buildAgent(), so even a close() that leaks the old workdir's lock
1912+
// (see closeAgentForRebuild's doc comment) can never cause a second
1913+
// acquisition on that same workdir — buildAgent() always targets
1914+
// the new, unlocked directory. The old lock still leaks for the
1915+
// rest of the process, but nothing ever tries to re-acquire it, so
1916+
// there is no crash to guard against here.
18641917
await currentAgent.close().catch((err: unknown) => {
18651918
tuiLogger.debug("agent.close during session-rotation teardown failed: {error}", {
18661919
error: err instanceof Error ? err.message : String(err),

tests/unit/tui/runner.test.ts

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,15 @@
11
import { test, expect } from "bun:test";
22
import { EventEmitter } from "node:events";
3+
import { AgentContextLockError, type Agent } from "@intx/agent";
34
import {
5+
agentRebuildFailure,
6+
closeAgentForRebuild,
47
createTUIEventEmitter,
58
getTUIRunSummaryStatus,
69
loadLocalSettingsWriteBase,
710
resumeTranscriptLoadErrorBlock,
811
} from "../../../src/tui/runner.js";
12+
import { createSessionOperationQueue } from "../../../src/tui/session-operation-queue.js";
913
import { createRunSink } from "../../../src/session/run-sink.js";
1014

1115
test("createTUIEventEmitter returns an EventEmitter", () => {
@@ -97,3 +101,134 @@ test("rotation resets run-sink so a new session starts from a clean state", () =
97101
expect(runSink.getStatus()).toBe("done");
98102
expect(collectorAfterReset.getTurns()).toHaveLength(0);
99103
});
104+
105+
// CL-5753: an interrupt can hit close() while reactor.abort()/sendQueue.drain()
106+
// are mid-teardown, throwing before @intx/agent's close() ever reaches
107+
// lock.release(). Once that happens the agent is already marked closed, so a
108+
// retried close() is a silent no-op that can never free the lock either — the
109+
// workdir's lock is stuck held for the rest of the process. The next
110+
// buildAgent() for that same workdir is then guaranteed to throw
111+
// AgentContextLockError ("an agent is already open for workdir: ..."), which
112+
// is the crash from the ticket. These tests cover the two functions the
113+
// runner now routes every rebuild through so that failure is reported in
114+
// plain language rather than escaping as an unhandled rejection.
115+
function stubAgent(closeImpl: () => Promise<void>): Agent {
116+
return { close: closeImpl } as unknown as Agent;
117+
}
118+
119+
test("closeAgentForRebuild reports a failed close without throwing", async () => {
120+
const agent = stubAgent(() => Promise.reject(new AgentContextLockError("/tmp/workdir")));
121+
const closedCleanly = await closeAgentForRebuild(agent, "interrupt");
122+
expect(closedCleanly).toBe(false);
123+
});
124+
125+
test("closeAgentForRebuild reports success when close() resolves", async () => {
126+
const agent = stubAgent(() => Promise.resolve());
127+
const closedCleanly = await closeAgentForRebuild(agent, "interrupt");
128+
expect(closedCleanly).toBe(true);
129+
});
130+
131+
test("agentRebuildFailure turns a stale-lock AgentContextLockError into a plain-language message", () => {
132+
// Simulates the second acquisition throwing after a failed close left the
133+
// lock held: buildAgent() surfaces AgentContextLockError, which must not
134+
// reach the caller as a raw stack trace.
135+
const err = agentRebuildFailure(new AgentContextLockError("/tmp/workdir"));
136+
expect(err.message).not.toContain("already open");
137+
expect(err.message).toMatch(/restart/i);
138+
});
139+
140+
test("agentRebuildFailure passes other errors through unchanged", () => {
141+
const original = new Error("network unreachable");
142+
expect(agentRebuildFailure(original)).toBe(original);
143+
});
144+
145+
test("a failed close followed by a lock error never surfaces as a raw AgentContextLockError", async () => {
146+
// End-to-end shape of the fix: close() throws (lock leaked in-process),
147+
// the rebuild site short-circuits instead of calling buildAgent() again,
148+
// and the resulting error is the plain-language one — never the raw
149+
// AgentContextLockError a bare `throw` would have produced.
150+
const agent = stubAgent(() => Promise.reject(new AgentContextLockError("/tmp/workdir")));
151+
let rebuildError: Error | null = null;
152+
try {
153+
const closedCleanly = await closeAgentForRebuild(agent, "interrupt");
154+
if (!closedCleanly) {
155+
throw new AgentContextLockError("/tmp/workdir");
156+
}
157+
} catch (err) {
158+
rebuildError = agentRebuildFailure(err);
159+
}
160+
expect(rebuildError).not.toBeNull();
161+
expect(rebuildError).not.toBeInstanceOf(AgentContextLockError);
162+
expect(rebuildError!.message).toMatch(/restart/i);
163+
});
164+
165+
// reloadIfIdle itself is a closure captured inside runTUI's single ~2500-line
166+
// scope (currentAgent, buildAgent, streamPromise, workflowController,
167+
// pendingReload/inFlight, fatalBuildError, etc. are all local variables of
168+
// that function), with no seam to construct or call it in isolation short of
169+
// standing up the full TUI runner — provider config, plugin discovery, MCP
170+
// wiring, and a real OpenTUI host. That is out of scope for this fix; it
171+
// would be its own extraction. What can be driven directly, and is exactly
172+
// the failure this bug reports, is the real `session-operation-queue.ts`
173+
// queue exercised the same way every rebuild site uses it: `void
174+
// enqueueOp(async () => { try { ... } catch (err) { fatalBuildError = ... } })`.
175+
// `enqueue` is `tail = tail.then(op, op); return tail;` — if `op` rejects and
176+
// nothing internally catches it, that returned promise is the only thing
177+
// that ever observes the rejection, and `void` discards it, which is
178+
// precisely how the unhandled rejection in the ticket escaped.
179+
test("a rejecting reload op through the real session-operation-queue never triggers an unhandled rejection", async () => {
180+
const { enqueue, awaitTail } = createSessionOperationQueue();
181+
const agent = stubAgent(() => Promise.reject(new AgentContextLockError("/tmp/workdir")));
182+
183+
let unhandled: unknown = null;
184+
const onUnhandledRejection = (reason: unknown): void => {
185+
unhandled = reason;
186+
};
187+
process.on("unhandledRejection", onUnhandledRejection);
188+
189+
let fatalBuildError: Error | null = null;
190+
try {
191+
// Mirrors reloadIfIdle's body verbatim: close the current agent through
192+
// closeAgentForRebuild, skip buildAgent() and throw instead of
193+
// re-acquiring on a failed close, and land any failure in
194+
// fatalBuildError via agentRebuildFailure — all behind `void enqueueOp`,
195+
// exactly as the runner calls it.
196+
void enqueue(async () => {
197+
try {
198+
const closedCleanly = await closeAgentForRebuild(agent, "reload");
199+
if (!closedCleanly) {
200+
throw new AgentContextLockError("/tmp/workdir");
201+
}
202+
} catch (err) {
203+
fatalBuildError = agentRebuildFailure(err);
204+
}
205+
});
206+
207+
await awaitTail();
208+
// Give any unhandled rejection queued by the engine a chance to fire
209+
// before asserting its absence — it lands on a later microtask/macrotask
210+
// than the awaited queue settlement.
211+
await new Promise((resolve) => setTimeout(resolve, 0));
212+
} finally {
213+
process.off("unhandledRejection", onUnhandledRejection);
214+
}
215+
216+
expect(unhandled).toBeNull();
217+
expect(fatalBuildError).not.toBeNull();
218+
expect(fatalBuildError).not.toBeInstanceOf(AgentContextLockError);
219+
expect(fatalBuildError!.message).toMatch(/restart/i);
220+
});
221+
222+
// A true negative control (reproducing reloadIfIdle's pre-fix shape — no
223+
// try/catch around the queued op — and asserting the rejection escapes) was
224+
// attempted here and deliberately removed: bun:test installs its own
225+
// `unhandledRejection` listener that fails whichever test is running the
226+
// instant one fires, regardless of what that test asserts, so a test
227+
// designed to prove an unhandled rejection *does* escape cannot pass in this
228+
// harness — it is intercepted before the assertion runs. That interception
229+
// is itself the strongest available evidence for the bug this fix removes:
230+
// the pre-fix `reloadIfIdle` body run through this exact harness fails the
231+
// suite outright (confirmed manually while writing this test), rather than
232+
// failing a single assertion. The test above is the harness-compatible half
233+
// of that pair: same real queue, same real helpers, proving the fixed shape
234+
// produces no such failure.

0 commit comments

Comments
 (0)