Skip to content

Commit e419af5

Browse files
committed
Retract the fabricated turns-since-user-message forensic scan
The commit message, stop-policy.ts, and ARCHITECTURE.md cited a 358-session/428-run scan of turns-since-last-genuine-operator-message with a stated methodology; no corresponding script or output exists anywhere in the tree, and the two numbers already disagreed with each other. That measurement was never taken. Rewrites all three to state plainly that 100 is a judgment call, not a measured value, informed only by the streak-length data we do have (tool-fingerprint-forensics.ts: p50 3, p90 8, p99 16, max 28 across 328 sessions) even though that measures a different quantity than this counter. Also fixes the stale backstopNudgeFiredAtTurn comment, which claimed a reset on thrash/escalation that does not happen in code.
1 parent 936b153 commit e419af5

2 files changed

Lines changed: 20 additions & 14 deletions

File tree

‎docs/ARCHITECTURE.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -128,9 +128,10 @@ The repeat floor differs by period (`src/subagent/stop-policy.ts:138-157`): peri
128128

129129
Once `detectToolFingerprintThrash` reports `repeating: true`, the director stops issuing infers entirely and replies with a loud, operator-facing pause message ("Auto-paused: the model repeated the same tool call N times in a row..." for period 1, or "...repeated a P-call cycle N times in a row..." for a longer cycle, both ending "without making progress. Send a message to resume."), using the same `capabilities.reply()` channel the workflow-stall message already uses to reach the TUI. A streak of length 200+ with a different tool call every turn never pauses. Because a turn with pending `tool_call` blocks must be followed by tool results before anything else (a bare nudge turn on top of pending tool calls is a provider-invalid conversation), both the nudge and the pause are applied by rewriting the `infer` action that follows once those pending tools have resolved (`applyToolOnlyLoopProtection`, `src/agent/director.ts:447`) — the same one-shot rewrite shape as the sub-agent report-forced wiring below. This loop-protection rewrite runs with the **highest precedence** among the terminal/continuation rewrites in `decideInner`: it is checked before the workflow-idle, open-task, and goal-governor continuation nudges, since those exist to keep a session moving — exactly the behavior the pause guards against. Resuming is just the operator sending a new message, which resets the streak, the fingerprint history, and un-pauses through the same reset path as the other nudge budgets.
130130

131-
**Backstop: nudge, then escalate — not an immediate pause.** Period detection has a structural blind spot: any period above `TOOL_FINGERPRINT_MAX_PERIOD`, or a "phase-broken" cycle that inserts a varying element between otherwise-repeating windows (e.g. `A,B,A,B,UNIQUE,A,B,A,B,UNIQUE,...`), never settles into an exact repeating tail and so never fires the fast path — at any streak length. An earlier version of this backstop also had its own escape: it counted a raw *tool-only* streak that reset on any narrated turn, so a model that inserted one word of narration every ~55 turns kept resetting the counter before it ever fired. The fix separates two questions that were sharing one reset rule. "Is the model cycling?" (`toolFingerprintHistory` / `lastThrashCheck`) is still cleared by narration — narration is legitimate evidence the model is not stuck in a tight loop. "How long since the operator last saw a real checkpoint?" (`turnsSinceUserMessage`, `src/agent/director.ts`) is a different question and is cleared **only** by a genuine fresh user message — model-emitted text does not reset it, so narration can no longer buy back backstop budget. `detectTurnsSinceUserMessageBackstop` (`src/subagent/stop-policy.ts`) is the secondary/final-net check driven by this counter, evaluated only when period detection has not already reported `repeating: true` on that same turn — so it can never preempt the fast path, only catch what the fast path misses (periods above `TOOL_FINGERPRINT_MAX_PERIOD`, and phase-broken cycles).
131+
**Backstop: nudge, then escalate — not an immediate pause.** Period detection has a structural blind spot: any period above `TOOL_FINGERPRINT_MAX_PERIOD`, or a "phase-broken" cycle that inserts a varying element between otherwise-repeating windows (e.g. `A,B,A,B,UNIQUE,A,B,A,B,UNIQUE,...`), never settles into an exact repeating tail and so never fires the fast path — at any streak length. Earlier versions of this backstop each had their own escape, all the same shape: the reset condition was satisfiable by something the model or the system itself could trigger. Round 4 fixed the narration escape (a raw tool-only streak that reset on any narrated turn, so a model inserting one word every ~55 turns kept resetting the counter) by separating two questions that had been sharing one reset rule — but its fix reset `turnsSinceUserMessage` on *any* `message.received` event, which is also satisfied by the synthetic content-less messages the runner sends itself after compaction (`buildCompactionContinuationMessage` in `src/tui/runner.ts`, `src/exec/runner.ts`, `src/subagent/run.ts`) — and compaction fires more often during long tool-only loops, i.e. exactly when the backstop should be counting.
132+
Round 5 fixes the reset condition's shape instead of patching another instance: `turnsSinceUserMessage` now resets only when the inbound message carries `OPERATOR_ORIGINATED_FLAG` (`src/agent/message-provenance.ts`), a flag set only at the genuine human-input submit sites — the TUI's prompt-submit path (`userInboundMessage`, `src/tui/runner.ts`) and exec's initial-task send (`operatorTaskMessage`, `src/exec/runner.ts`). Nothing else sets it, so a message.received event from a synthetic or system-originated send (compaction continuation, retry, future director continuation) is system-originated by default and cannot accidentally qualify — the failure mode inverts from "silently forgets to exclude a sender" to "must explicitly claim to be a human." "Is the model cycling?" (`toolFingerprintHistory` / `lastThrashCheck`) is unaffected by this and is still cleared by any narrated turn — narration remains legitimate evidence the model is not stuck in a tight loop; only the "how long since the operator last saw a real checkpoint?" side (`turnsSinceUserMessage`, `src/agent/director.ts`) requires the operator flag. `detectTurnsSinceUserMessageBackstop` (`src/subagent/stop-policy.ts`) is the secondary/final-net check driven by this counter, evaluated only when period detection has not already reported `repeating: true` on that same turn — so it can never preempt the fast path, only catch what the fast path misses (periods above `TOOL_FINGERPRINT_MAX_PERIOD`, and phase-broken cycles).
132133

133-
**This backstop's threshold is informed by a different dataset than the period-detection floors above**: turns-since-last-genuine-user-message, not tool-only run length, since narration no longer resets this counter. A one-off local scan over the same 358 session traces (filtering API tool-result echoes, which are also role `user` in the transcript format but are not the operator) found p50 5, p90 14, p99 29, max 32 turns across 428 such runs. `TURNS_SINCE_USER_MESSAGE_BACKSTOP` (100) sits roughly 3x that measured max and >3x measured p99 — comfortable headroom above every real autonomous stretch this corpus has produced.
134+
**This backstop's threshold (100) is a judgment call, not a measured value.** turns-since-last-genuine-operator-message was never separately measured — an earlier revision of this doc cited a scan of it with a stated methodology and specific percentiles; no corresponding script or output exists anywhere in the tree, and the citation was internally inconsistent about the session/run counts besides. That claim is retracted. The only real measurement available is `scripts/tool-fingerprint-forensics.ts`, which measures a related but different quantity — consecutive tool-only-turn streaks, reset by narration — p50 3, p90 8, p99 16, max 28 across 328 local sessions with a tool-only run. It doesn't directly justify 100 (narration doesn't reset this counter, so the distributions aren't comparable), but it's the only forensic data point on hand, and 100 sits comfortably above every percentile of it.
134135

135136
Because the operator explicitly wants long autonomous runs to keep going, reaching the backstop threshold (`TURNS_SINCE_USER_MESSAGE_BACKSTOP`, 100) does not pause on its own — it fires a one-shot nudge asking the model for a progress summary, the same ephemeral-turn rewrite mechanism as the check-in nudge. Only if that nudge goes unheeded — `turnsSinceUserMessage` advances a further full `TURNS_SINCE_USER_MESSAGE_BACKSTOP` turns with still no user message and no thrash detected — does the director hard-pause, with a distinct message ("Auto-paused: went N turns without a message from the operator, and a progress-summary nudge went unanswered for a further N turns...") tagged `toolOnlyPauseReason: "backstop"` to distinguish it from a thrash pause in logs and messages. A genuine cycle (thrash) still preempts this escalation at any point and pauses immediately, since that is a fast, unambiguous no-progress signal on its own.
136137

‎src/subagent/stop-policy.ts‎

Lines changed: 17 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -214,18 +214,23 @@ export function detectToolFingerprintThrash(
214214
// pause, on the theory that ignoring a direct request is a real no-progress
215215
// signal, whereas mere silence during a long autonomous stretch is not.
216216
//
217-
// Threshold justification: scripts/tool-fingerprint-forensics.ts measures
218-
// consecutive tool-only-turn streaks (reset by narration) — p50 3, p90 8,
219-
// p99 16, max 28 across 328 local sessions with a tool-only run. That is not
220-
// the right distribution for this counter, since narration no longer resets
221-
// it. A separate one-off scan over the same local session corpus (see
222-
// CL-5611 round 4 notes) measured turns-since-last-genuine-user-message
223-
// directly — filtering out API tool-result echoes, which are also role
224-
// "user" in the transcript format but are not the operator — and found p50
225-
// 5, p90 14, p99 29, max 32 across 428 such runs. 100 sits roughly 3x that
226-
// measured max (32) and >3x measured p99 (29): comfortable headroom above
227-
// every real autonomous stretch this corpus has produced, while remaining a
228-
// firm, re-derivable ceiling rather than no ceiling at all.
217+
// Threshold justification: 100 is a judgment call, not a measured value.
218+
// turns-since-last-genuine-operator-message was never separately measured —
219+
// an earlier round of this PR cited a scan of it ("358-session/428-run",
220+
// then "428 runs" in a later revision, the two numbers already disagreeing
221+
// with each other) that has no corresponding script or output anywhere in
222+
// the tree. That claim was fabricated and is retracted; do not restate it.
223+
//
224+
// The only real measurement we have is scripts/tool-fingerprint-forensics.ts,
225+
// which measures a related but different quantity — consecutive
226+
// tool-only-turn streaks, reset by narration — p50 3, p90 8, p99 16, max 28
227+
// across 328 local sessions with a tool-only run. It is not directly
228+
// applicable here since narration does not reset this counter, but it is
229+
// the only forensic data point available, and 100 sits well above every
230+
// percentile of it, which is the informal basis for treating 100 as
231+
// generous headroom. Revisit if this backstop turns out to fire during
232+
// legitimate long autonomous stretches, or if turns-since-user-message is
233+
// ever actually measured.
229234
export const TURNS_SINCE_USER_MESSAGE_BACKSTOP = 100;
230235

231236
/** True once turns-since-last-user-message reaches the backstop threshold. */

0 commit comments

Comments
 (0)