Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,12 @@ parallel copies under `docs/` or `scripts/notes/`. At cut time: rename

## [Unreleased]

### Fixed

- A stalled worker now gets a full `stallTimeoutMs` grace after the first
continuation nudge before salvage — stall pings queued inside that window
wait instead of counting toward escalation.

### Removed

- `--force` is no longer accepted. It had no runtime effect; resume and the
Expand Down
2 changes: 1 addition & 1 deletion docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ The ChatDirector counts consecutive assistant turns that contain tool calls and

#### Sub-agent stall management

`SubAgentDirector` tracks `lastActivityAt`, updated on every real `inference.done` and `tool.done`. Directors are pure `decide(event, ...)` functions with no timer of their own and the reactor has no proactive "idle" event, so a genuinely silent worker (e.g. parked on a long-running background command with nothing else to do) produces no event for the director to react to. `runSubAgent` (`src/subagent/index.ts`) arms an external interval, at `subAgentStallTimeoutMs`, that pings the same content-less continuation channel the compaction governor uses to re-enter an idle reactor (`requestContinuation`). The director only acts on a ping if the elapsed time since `lastActivityAt` has crossed the timeout — a ping delivered while a tool call is still executing simply queues until that cycle finishes, so "no pending harness-tracked work" falls out of when the check can run at all rather than needing separate bookkeeping. The first stall past the timeout gets one continuation nudge (asking the worker to check on the background work or report status); a second **consecutive** stall (no activity since that nudge) escalates to the existing salvage path, returning a `stalled` `forcedStopReport` with the same structured shape (summary/findings/blockers) as `turn-budget` and `cancelled`. Any real activity between pings resets the streak, so a worker that is genuinely working through a slow single turn is never penalized. After the worker has already replied with a terminal report (complete envelope or salvage), further empty continuations — idle-compact meter sync or stall pings — return `wait` instead of falling through to `DefaultDirector.infer`; only a non-empty parent message (`resume_agent` / `send_input`) re-opens the brief.
`SubAgentDirector` tracks `lastActivityAt`, updated on every real `inference.done` and `tool.done`. Directors are pure `decide(event, ...)` functions with no timer of their own and the reactor has no proactive "idle" event, so a genuinely silent worker (e.g. parked on a long-running background command with nothing else to do) produces no event for the director to react to. `runSubAgent` (`src/subagent/index.ts`) arms an external interval, at `subAgentStallTimeoutMs`, that pings the same content-less continuation channel the compaction governor uses to re-enter an idle reactor (`requestContinuation`). The director only acts on a ping if the elapsed time since `lastActivityAt` has crossed the timeout — a ping delivered while a tool call is still executing simply queues until that cycle finishes, so "no pending harness-tracked work" falls out of when the check can run at all rather than needing separate bookkeeping. The first stall past the timeout records `stallNudgeAt` and issues one continuation nudge (asking the worker to check on the background work or report status). Later empty pings inside `subAgentStallTimeoutMs` of that instant wait without stopping or treating the ping as activity; salvage fires only once a ping arrives after that grace with still no `tool.done` / turn-boundary reset. That grace is what keeps two queued interval ticks from salvaging hundreds of milliseconds after the nudge. Any real activity clears `stallNudgeAt`, so a worker that is genuinely working through a slow single turn is never penalized. After the worker has already replied with a terminal report (complete envelope or salvage), further empty continuations — idle-compact meter sync or stall pings — return `wait` instead of falling through to `DefaultDirector.infer`; only a non-empty parent message (`resume_agent` / `send_input`) re-opens the brief.

**Intervention log**: every stop and nudge is appended as one JSONL record to `interventions.jsonl` in the firing worker's trace dir (`src/subagent/intervention-log.ts`), carrying the trigger's measured value beside the threshold it crossed, the provider/model/family it fired on, and the run state at that moment (turns used vs budget, tool calls, read/edit counts). A refused parent re-dispatch is recorded on the parent side, where no worker run exists to record it. The parent also appends one `outcome` record per completed dispatch — the salvage kind `classifyBriefSalvage` assigned, or a clean-complete marker, plus the dispatch count — so the log carries dispatch outcomes as well as interventions, and a stop record can later be read alongside what the dispatch it touched actually produced. Writes are fire-and-forget and swallow their own errors — a diagnostic must not be able to fail a run. `scripts/intervention-forensics.ts` aggregates these across local sessions: per-intervention counts by model family, the measured-value distribution against the threshold, two context columns (stops that fired on runs which had already edited files; stops that fired before half the turn budget was spent — neither is a measured false-positive rate, since either is equally consistent with a correct stop or a wrong one), and outcome counts by kind. This exists because every threshold in this tree was set by judgment and four of those judgments were later reverted — a threshold change is expected to cite this data (CL-6938).

Expand Down
2 changes: 1 addition & 1 deletion docs/PRODUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ Corbits Code fans work out to short-lived **fleet agents** — workers with thei
- **Tasks** are checklist items owned by one agent via `manage_tasks`.
- **Fleet agents** are spawned with `spawn_agent`. On the TUI primary, mailbox mail arrives as inbound when a worker finishes or fails — do not poll `wait_agents`. Nested orchestrators still collect with `wait_agents`. Workers ask the parent with `ask_director`. That parks a question while the worker stays `running`. Nested `wait_agents` returns `awaiting_director` with a question payload — that is not terminal. The parent answers with `send_input` (`target` = the worker's session id). When the parent TUI is not blocked in `wait_agents`, a parked question arrives as a synthetic idle-send wake. Escalate to the human only with `ask_operator`.

Dispatch uses a structured brief (context / goal / optional goals seed) and returns a structured report. The TUI Agents strip and fleet board show who is running; live tool progress updates the status bar without dumping the child transcript into the parent chat. There is no turn budget. A tool-less final turn completes only with the four-heading report envelope. Printed `<tool_call>` markup in assistant text gets one corrective nudge to issue a real tool call and does not count as the wrap-up; without the envelope, one incomplete-report nudge is given and a second tool-less turn without the envelope salvages as `incomplete-report-stop`. A silent worker (no activity for `stallTimeoutMs`, opt-in) gets one continuation nudge, then salvages as `stalled` if a second consecutive check finds no activity. An opt-in `deadlineMs`, or an operator cancel, can also end a run early. Each of these returns a salvage report so a runaway or idle child cannot quietly burn a large token budget or look done after prose alone.
Dispatch uses a structured brief (context / goal / optional goals seed) and returns a structured report. The TUI Agents strip and fleet board show who is running; live tool progress updates the status bar without dumping the child transcript into the parent chat. There is no turn budget. A tool-less final turn completes only with the four-heading report envelope. Printed `<tool_call>` markup in assistant text gets one corrective nudge to issue a real tool call and does not count as the wrap-up; without the envelope, one incomplete-report nudge is given and a second tool-less turn without the envelope salvages as `incomplete-report-stop`. A silent worker (no activity for `stallTimeoutMs`, opt-in) gets one continuation nudge, then salvages as `stalled` only after a full `stallTimeoutMs` grace with still no activity — queued checks inside that window wait, they do not salvage. An opt-in `deadlineMs`, or an operator cancel, can also end a run early. Each of these returns a salvage report so a runaway or idle child cannot quietly burn a large token budget or look done after prose alone.

## Roadmap (planned, not yet shipped)

Expand Down
111 changes: 111 additions & 0 deletions src/subagent/nudge-director.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1014,6 +1014,117 @@ describe("SubAgentDirector post-complete terminalization (CL-7068)", () => {
});
});

describe("SubAgentDirector stall nudge grace", () => {
test("two queued empty pings in the same tick nudge then wait, not stop", async () => {
let now = 3_000_000;
const director = new SubAgentDirector(
"system",
[],
undefined,
1_000,
() => now,
);
const caps = capabilities();

await director.decide(inferenceDoneText("working"), state, caps);

now += 1_000;
const first = actions(
await director.decide(messageReceived(""), state, caps),
);
expect(first).toContainEqual({
type: "checkpoint",
message: "subagent-stall-nudge",
});
expect(first.some((action) => action.type === "reply")).toBe(false);

const second = actions(
await director.decide(messageReceived(""), state, caps),
);
expect(second).toEqual([{ type: "wait" }]);
});

test("queued pings inside grace wait; stop only after grace with no activity", async () => {
let now = 1_000_000;
const director = new SubAgentDirector(
"system",
[],
undefined,
1_000,
() => now,
);
const caps = capabilities();

await director.decide(inferenceDoneText("working"), state, caps);

now += 1_000;
const first = actions(
await director.decide(messageReceived(""), state, caps),
);
expect(first).toContainEqual({
type: "checkpoint",
message: "subagent-stall-nudge",
});

now += 200;
const midGrace = actions(
await director.decide(messageReceived(""), state, caps),
);
expect(midGrace).toEqual([{ type: "wait" }]);

now += 200;
const stillGrace = actions(
await director.decide(messageReceived(""), state, caps),
);
expect(stillGrace).toEqual([{ type: "wait" }]);

now += 600;
const stopped = actions(
await director.decide(messageReceived(""), state, caps),
);
expect(stopped).toContainEqual({
type: "checkpoint",
message: "subagent-stalled",
});
expect(stopped.some((action) => action.type === "reply")).toBe(true);
});

test("tool.done during grace clears stallNudgeAt so a later silence nudges again", async () => {
let now = 2_000_000;
const director = new SubAgentDirector(
"system",
[],
undefined,
1_000,
() => now,
);
const caps = capabilities();

await director.decide(inferenceDone(["read-1"]), state, caps);
now += 1_000;
const first = actions(
await director.decide(messageReceived(""), state, caps),
);
expect(first).toContainEqual({
type: "checkpoint",
message: "subagent-stall-nudge",
});

now += 100;
await director.decide(toolDone("read-1"), state, caps);

now += 1_000;
const afterActivity = actions(
await director.decide(messageReceived(""), state, caps),
);
expect(afterActivity).toContainEqual({
type: "checkpoint",
message: "subagent-stall-nudge",
});
expect(afterActivity.some((action) => action.type === "reply")).toBe(false);
});
});

function stubAdmission(
notes: { provider: string; until: number }[],
): AdmissionQueue {
Expand Down
34 changes: 23 additions & 11 deletions src/subagent/nudge-director.ts
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,11 @@ export class SubAgentDirector extends DefaultDirector {
private readonly stallTimeoutMs: number | undefined;
private readonly now: () => number;
private lastActivityAt: number;
private consecutiveStalls = 0;
// Wall clock when the first stall nudge was issued. Later empty pings inside
// stallTimeoutMs of this instant wait without stopping or restarting grace;
// stop only after the grace elapses with no activity. Cleared on real
// tool.done / turn-boundary activity.
private stallNudgeAt: number | undefined;
private lastAssistantText = "";
// Every stop and nudge is recorded with its measured value beside its
// threshold, so a later threshold change can cite data instead of judgment
Expand Down Expand Up @@ -303,7 +307,7 @@ export class SubAgentDirector extends DefaultDirector {
if (onTurnBoundary(event)) {
this.lastConsumedNudgeText = null;
this.lastActivityAt = this.now();
this.consecutiveStalls = 0;
this.stallNudgeAt = undefined;
this.compaction.noteInferenceDone(event, state.turns);
this.turnsCompleted++;
const content = event.turn.content as readonly {
Expand Down Expand Up @@ -410,7 +414,7 @@ export class SubAgentDirector extends DefaultDirector {
}
if (event.type === "tool.done") {
this.lastActivityAt = this.now();
this.consecutiveStalls = 0;
this.stallNudgeAt = undefined;
if (event.result.isError === true) {
// Failed-tool recovery guidance. Arm once; coalesce consecutive failure
// audits until applyPendingNudge flushes a single counted record.
Expand Down Expand Up @@ -438,12 +442,12 @@ export class SubAgentDirector extends DefaultDirector {
* "no pending harness-tracked work" falls out of when this method can run
* at all rather than needing separate bookkeeping.
*
* First stall past the timeout: one continuation nudge, asking the leaf to
* report status or keep going. A second consecutive stall (no activity
* since the nudge) escalates to the existing salvage path, same shape as
* the turn-boundary checks above. Returns null when this event is not
* a stall check the director should act on (let it fall through as an
* ordinary continuation).
* First silence past the timeout: one continuation nudge, and record
* stallNudgeAt. Queued pings that arrive inside the stallTimeoutMs grace
* after that nudge neither stop nor restart the grace (and do not count as
* activity). Stop only when a ping arrives after the grace with still no
* activity. Returns null when this event is not a stall check the director
* should act on (let it fall through as an ordinary continuation).
*/
private checkStallPing(
event: ReactorInboundEvent,
Expand All @@ -456,8 +460,8 @@ export class SubAgentDirector extends DefaultDirector {
const elapsed = this.now() - this.lastActivityAt;
if (elapsed < this.stallTimeoutMs) return null;

this.consecutiveStalls++;
if (this.consecutiveStalls === 1) {
if (this.stallNudgeAt === undefined) {
this.stallNudgeAt = this.now();
this.interventions({
id: "stall-nudge",
class: "nudge",
Expand All @@ -473,6 +477,14 @@ export class SubAgentDirector extends DefaultDirector {
inferWithSubAgentNudge(capabilities, SUBAGENT_STALL_NUDGE),
];
}

const sinceNudge = this.now() - this.stallNudgeAt;
if (sinceNudge < this.stallTimeoutMs) {
// Still inside the post-nudge grace. Wait without faking activity or
// restarting the grace clock.
return [capabilities.wait()];
}

this.interventions({
id: "stalled",
class: "stop",
Expand Down
Loading