Optimize Coder/Lead orchestration LLM usage: measure prompt-section cost, stop cache-busting dynamic system prompts, and avoid redundant decider calls

[serge-ivo]

Problem

Coder/Lead orchestration spends the owner's Anthropic BYOK key on cloud brain calls even when the local coding engine is subscription/unattributed. Live MCP usage reads on 2026-09-01 show the largest token/value number is engine work with payer unknown, not charged API-key spend; the charged API-key bucket is much smaller but is led by chat/coding orchestration calls.

30d snapshot from MCP usage_summary:

• engine: $10,838.14 notional value, 739 calls, 84 sessions, $0 charged, mostly unattributed Claude Code/Opus work.
• chat: $59.20 notional, $40.99 charged, 2,107 calls.
• coding: $11.32 notional, $10.37 charged, 958 calls.
• copilot: $1.92 notional, 85 calls.
• overseer: $1.70 notional, 57 calls.

This is not #551. #551 is payer attribution for engine work. This ticket is about reducing the orchestration calls that actually spend BYOK/API-key tokens.

Suspected sinks

• workers/api/src/agent-think.ts builds a large dynamic chat system prompt every turn: memory, tasks, directions, recent work, repo/session state, terminal tails, connector/tool descriptions.
• workers/api/src/lib/user-ai.ts prompt-caches the whole system prompt as one block. Dynamic clock/repo/session/recent-work sections likely cause avoidable cache writes or misses.
• workers/api/src/workflows/agent-loop.ts pays for the target agent chat, then workers/api/src/lib/loop-orchestrator.ts pays a second Sonnet call to decide continue/done/escalate every iteration.
• workers/api/src/lib/coding-loop.ts Pilot decisions include the full step log plus a ~6k terminal tail every step, up to 30 steps, with PILOT_MAX_TOKENS = 2048.
• workers/api/src/lib/coding-copilot.ts substantive Co-pilot questions can run up to three tool rounds plus a final answer, with timeline memory and terminal context.

MVP slice

1. Add usage instrumentation that records prompt-section byte/token estimates per kind (chat, coding, overseer, copilot) into trace/debug events, without storing prompt text.
2. Split stable cached system prompt from dynamic per-turn context for agent-think and Pilot decisions.
3. Add a deterministic short-circuit before runLoopDecide when the target agent reply/tool outcome already gives a terminal verdict.
4. Cap Pilot Steps so far to recent N plus a compact summary.

Acceptance

• Usage/trace can show top prompt sections by estimated input tokens for one run.
• Chat cache-write tokens drop on repeated turns with unchanged stable instructions.
• A loop iteration with an obvious terminal outcome avoids the second decider call.
• Existing safety, authority and grounding prompt claims remain covered by tests.

Non-goals

• Do not map machine-login to subscription; that is 99.62% of the account's AI value is "Payer not established" — one stored claude-code token would resolve it, and nothing on any surface says so #551 and would be false attribution.
• Do not cut safety/authority text blindly.
• Do not treat total notional engine value as API-key spend; cache reads and unknown payer rows distort the picture.

Provenance

Filed from a BA + live MCP usage investigation on 2026-09-01. MCP tools used: usage_summary, keys_status. Anthropic is the currently active BYOK key; OpenAI has not been used since 2026-08-12.

[serge-ivo]

Shipped first code slice in d5701824.

What landed:

• Added llm.prompt_sections debug trace events for BYOK LLM calls, recording section labels, byte counts, rough token estimates, model/kind/phase, and trace scoping only. Prompt text is not stored.
• Instrumented chat, loop-decider, Pilot coding, Co-pilot, and Overseer paths so a single run trace can show whether system prompt, history, terminal tail, tools, or repo context dominates estimated input tokens.
• Threaded trace ids into loop-decider and Pilot usage contexts.
• Capped Pilot Steps so far prompt growth to the latest 8 concrete step lines plus a compact omitted-count summary.

Verification:

• pnpm vitest run workers/api/src/lib/prompt-section-estimates.test.ts workers/api/src/lib/coding-loop.test.ts workers/api/src/lib/user-ai.test.ts -> 73 passed.
• pnpm vitest run workers/api/src/lib/agent-loop.test.ts workers/api/src/lib/loop-decide.test.ts workers/api/src/lib/autonomous-budget.test.ts -> 68 passed.
• pnpm vitest run workers/api/src/routes/coding.test.ts workers/api/src/routes/coding-diagnostics.test.ts workers/api/src/routes/coding.contract.test.ts -> 83 passed.
• pnpm exec biome check <touched files> passed.
• git diff --check passed.

Known unrelated blocker remains: pnpm --filter proagentstore-api typecheck still fails in workers/api/src/lib/pdf-form.ts due to the existing pdf-lib typing/module issue.

Left open intentionally: this is the observability + Pilot step-log cap slice. The cache-splitting and deterministic decider short-circuit work should follow with trace data rather than guessing at which prompt sections are actually expensive.