Preserve role-specific context and recoverable compaction evidence

Author: TheGreatAxios

docs/ARCHITECTURE.md

@@ -165,15 +165,20 @@ The agent maintains an optional **`manage_tasks`** list (create/update via the h
 
 #### Context compaction (the compaction governor)
 
-When a cycle's input tokens cross a threshold, the director compacts the inference-facing history (the full run is always retained in the context store). The threshold is **model-aware** — roughly 60% of the active model's real context window — so small-window models compact early enough to avoid provider context-overflow while large-window models do not compact prematurely. The compacted prefix is **append-only across passes**: the existing compacted user turn stays byte-identical; new folds become later summary turns with a harness-inserted assistant spacer (identified by reserved `model: "harness"`, plus a visible sentinel; persisted `[compaction]` tokens without a producer still freeze) between them so the prompt head can remain in the provider KV cache. Model-emitted copies of the spacer are not frozen. Spacer-only model replies are incomplete: ChatDirector nudges, then falls through loop-protection, workflow-idle, and open-task rails rather than empty-settling with work still open. The governor covers three cases:
+When a cycle's input tokens cross a threshold, the director compacts the inference-facing history (the full run is always retained in the context store). The threshold is **model-aware** — roughly 60% of the active model's real context window — so small-window models compact early enough to avoid provider context-overflow while large-window models do not compact prematurely. Each successful pass produces **one refreshed handoff**, folding the previous handoff together with newly compacted history rather than accumulating frozen summary turns. Recent turns and selected anchors remain alongside it. Spacer-only model replies are incomplete: ChatDirector nudges, then falls through loop-protection, workflow-idle, and open-task rails rather than empty-settling with work still open. The governor covers three cases:
 
 - **Threshold at a tool pause** — Once over threshold, the follow-up `infer` after a tool batch is swapped for a `compact` cycle, and inference resumes via a host continuation message. After a compact that remains over the high watermark, the governor uses **growth hysteresis** (wait for usage to grow by ~10% of the window) instead of re-arming on every cycle; dropping under 60% is not required.
 - **Idle (end-of-turn)** — An interactive turn can end with a reply and then sit idle with no tool batch to intercept; the governor requests a continuation at that pause and compacts when it arrives. An operator message that races the continuation still compacts first, then re-enters inference to answer it.
 - **Overflow recovery** — A `context_overflow` inference error would otherwise become a terminal error reply; the governor compacts and retries instead, bounded so a history the compactor cannot shrink does not loop forever. Overflow ignores hysteresis for the compact itself.
 
 The compaction control flow is shaped by a reactor invariant: a `compact` action runs in its own cycle (it cannot be paired with `infer`), and **the reactor delivers no event after a compact cycle**. A director that simply emitted `compact` in place of the follow-up `infer` would leave the loop idle forever — the cause of an earlier stall. Instead the governor, after emitting `compact`, self-delivers a content-less inbound message (a host-supplied `requestContinuation` callback). That message adds no turn (`createInboundTurn` returns `null` for empty content) but re-enters the loop, where the director issues the follow-up `infer` against the freshly truncated history.
 
-Compaction replaces older turns with a structured, workflow-aware summary rather than a stats blob: sections for **What Happened / What We're Doing / Relevant Links / Action Items / Next Steps**, with the active workflow and step woven in so compacting mid-`/build` or mid-`/plan` preserves the contract. The summary is produced by a one-shot model call; on any failure it falls back to a deterministic summary so a compaction cycle never breaks the session.
+Two explicit role policies share the pruning engine, model summarizer, and governor:
+
+- **Orchestrator** — A broad, workflow-aware model handoff covers the objective, workflow progress, delegated work, decisions, outstanding obligations, and cross-task dependencies. It uses **What Happened / What We're Doing / Relevant Links / Action Items / Next Steps** sections and retains evidence references. The primary-only evidence archive preserves authorized, post-policy evidence separately from the working summary; existing scoped `read_file`, `grep`, and `search_files` access it through `archive:///` references, not direct sidecar paths.
+- **Worker** — The delivered task contract (including supplied plan, context, scope, required outcomes, constraints, and report requirements) and accepted parent follow-ups remain verbatim outside the model-generated execution summary. They are retained in acceptance order, with later conflicting instructions taking precedence; a successful `ask_director` answer remains associated with its question. Queued or rejected messages are not accepted contract entries. The worker summary covers substantive findings, evidence, failures, changes, and next actions within that scope. Workers do not receive an evidence archive.
+
+The summary input prioritizes the current fold and previous handoff, not an archive-wide sample of old history. Oversized evidence is processed in bounded, substantive chronological chunks before a final combined handoff; binary attachments contribute metadata and references rather than decoded text. If the model call fails, returns empty text, or mandatory evidence exceeds the input budget, compaction retains the original usable context. An unshrinkable worker contract likewise retains the original context. Production compaction has no statistics-only fallback; the governor's existing bounded overflow recovery stops retries when compaction cannot make room.
 
 ### Web Tools and Providers (`src/web/`)
 

docs/IMPLEMENTATION.md

@@ -439,6 +439,10 @@ session; that tree re-write is inherent to git and left as residual cost.
 
 `index.ts` installs `uncaughtException` and `unhandledRejection` handlers (and catches a rejected `main`). Each calls `writeCrashReport` (`crash/report.ts`) to write a best-effort report to `~/.corbits/projects/<project-key>/errors/<timestamp>.txt`, using the same `projectKeyFor`/`projectSessionsRoot` (`session/project-key.ts`) that keys that project's session directories, so a crash report lands next to the session's `run.json` and transcript rather than under a separately computed slug. The file records the failure kind, an ISO timestamp, the cwd, and the stack. `projectSessionsRoot` shells out to git with no timeout, so the handler never calls it directly — `primeCrashReporting` resolves and caches the directory once at startup (right after config load), and `writeCrashReport` only ever reads that cached value; if priming never ran or failed, it falls back to an `unresolved` bucket rather than touching git mid-crash. `writeCrashReport` swallows its own errors and returns `null` without logging; the handler in `index.ts` is what prints the one-line failure notice to stderr when that happens, then exits non-zero.
 
+### Binary compaction evidence
+
+The primary evidence archive stores authorized image and PDF payloads as raw bytes, with `encoding: "raw"`, MIME `contentType`, byte length, and a SHA-256 `contentHash`. Binary payloads are not decoded as UTF-8. The existing scoped `read_file` path for an `archive:///` occurrence verifies the hash and length, then exposes a metadata header with `encoding: "base64"` and `decodedByteLength`, followed by base64 wrapped at 76 characters. Reads use the existing line `offset` and `limit`, capped at 64 lines per binary page. Archive search matches binary metadata only, not the encoded payload. This representation uses the existing archive access tools; workers have no archive access.
+
 ### Event Stream
 
 `agent.stream()` emits `ReactorEmittedEvent` objects. `docs/ARCHITECTURE.md`'s

src/plugins/evidence-archive-binary-pages.test.ts

@@ -0,0 +1,90 @@
+import { expect, test } from "bun:test";
+import { mkdtemp } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { createCompactionArchive, hashAuthorizedBytes } from "../session/compaction-archive.js";
+import { evidenceArchiveSearchPlugin } from "./evidence-archive-search-plugin.js";
+
+test.each(["image/png", "application/pdf", "legacy-image"])(
+  "archive %s pages recover every byte after reopening, including beyond the scan ceiling",
+  async (format) => {
+    const blobs = new Map<string, Uint8Array>();
+    const opts = {
+      sessionId: "paging",
+      contextDir: await mkdtemp(join(tmpdir(), "archive-paging-")),
+      writeBlob: async (key: string, bytes: Uint8Array) => {
+        blobs.set(key, bytes);
+      },
+      readBlob: async (key: string) => {
+        const bytes = blobs.get(key);
+        if (bytes === undefined) throw new Error("missing");
+        return bytes;
+      },
+    };
+    const archive = createCompactionArchive(opts);
+    const bytes = Uint8Array.from(
+      { length: format === "legacy-image" ? 6000 : 7 * 1024 * 1024 },
+      (_, index) => index % 251,
+    );
+    const legacy = new TextEncoder().encode(Buffer.from(bytes).toString("base64"));
+    blobs.set("img-legacy", legacy);
+    const occurrence =
+      format === "legacy-image"
+        ? await archive.recordExistingBlobReference({
+            kind: "attachment",
+            blobKey: "img-legacy",
+            contentHash: hashAuthorizedBytes(legacy),
+            provenance: "persistBlobs:aged-image",
+          })
+        : await archive.recordAuthorizedPayload({
+            kind: "attachment",
+            payload: bytes,
+            binary: {
+              encoding: "raw",
+              name: "evidence",
+              contentType: format,
+              byteLength: bytes.length,
+            },
+          });
+    const reopened = createCompactionArchive(opts);
+    const plugin = evidenceArchiveSearchPlugin(() => reopened);
+    const handler = plugin.middleware!(async () => {
+      throw new Error("unexpected passthrough");
+    });
+    const read = async (offset: number) =>
+      String(
+        (
+          await handler(
+            {
+              id: "read",
+              name: "read_file",
+              arguments: { path: `archive:///${occurrence.occurrenceId}`, offset, limit: 64 },
+            },
+            new AbortController().signal,
+          )
+        ).content,
+      );
+    if (format !== "legacy-image") {
+      const deepPage = await read(110000);
+      expect(deepPage).not.toContain("scan limit");
+      expect(deepPage).toMatch(/110001\t/);
+    }
+    const lines: string[] = [];
+    let offset = 0;
+    for (let page = 0; page < 3000; page += 1) {
+      const text = await read(offset);
+      expect(text.length).toBeLessThan(10_000);
+      for (const match of text.matchAll(/^\s*(\d+)\t(.*)$/gm)) {
+        expect(Number(match[1])).toBe(lines.length + 1);
+        lines.push(match[2]!);
+      }
+      const next = /Use offset=(\d+) to continue/.exec(text);
+      if (next === null) break;
+      expect(Number(next[1])).toBeGreaterThan(offset);
+      offset = Number(next[1]);
+    }
+    expect(lines[0]).toContain('"encoding":"base64"');
+    expect(Buffer.from(lines.slice(1).join(""), "base64").equals(Buffer.from(bytes))).toBe(true);
+  },
+  120_000,
+);

src/plugins/evidence-archive-search-plugin.test.ts

@@ -76,6 +76,51 @@ describe("advertiseArchiveSurface", () => {
 });
 
 describe("evidenceArchiveSearchPlugin", () => {
+  test.each(["image/png", "application/pdf"])(
+    "binary %s reads paginate byte-identically without exposing base64 to search",
+    async (contentType) => {
+      const archive = memoryArchive("binary-pages");
+      const bytes = Uint8Array.from({ length: 20_000 }, (_, i) => i % 256);
+      const occ = await archive.recordAuthorizedPayload({
+        kind: "attachment",
+        payload: bytes,
+        binary: { encoding: "raw", contentType, name: "evidence-binary", byteLength: bytes.length },
+      });
+      const plugin = evidenceArchiveSearchPlugin(() => archive);
+      const handler = plugin.middleware ? plugin.middleware(nextHandler) : nextHandler;
+      const lines: string[] = [];
+      let offset = 0;
+      for (let page = 0; page < 20; page += 1) {
+        const result = await handler(
+          makeCall("read_file", { path: formatArchiveRef(occ.occurrenceId), offset, limit: 2000 }),
+          new AbortController().signal,
+        );
+        const text = String(result.content);
+        expect(text.length).toBeLessThan(10_000);
+        for (const match of text.matchAll(/^\s*\d+\t(.*)$/gm)) lines.push(match[1]!);
+        const continuation = /Use offset=(\d+) to continue/.exec(text);
+        if (continuation === null) break;
+        expect(Number(continuation[1])).toBeGreaterThan(offset);
+        offset = Number(continuation[1]);
+      }
+      expect(lines[0]).toContain("base64");
+      expect(lines[0]).toContain(occ.contentHash);
+      expect(Buffer.from(lines.slice(1).join(""), "base64").equals(Buffer.from(bytes))).toBe(true);
+      const result = await handler(
+        makeCall("grep", {
+          pattern: Buffer.from(bytes).toString("base64").slice(0, 60),
+          path: "archive:///",
+        }),
+        new AbortController().signal,
+      );
+      expect(String(result.content)).not.toContain(formatArchiveRef(occ.occurrenceId));
+      const metadata = await handler(
+        makeCall("grep", { pattern: "evidence-binary", path: "archive:///" }),
+        new AbortController().signal,
+      );
+      expect(String(metadata.content)).toContain(formatArchiveRef(occ.occurrenceId));
+    },
+  );
   test("search_files lists archive:/// refs and read_file returns the payload", async () => {
     const archive = memoryArchive("sess-primary");
     const occ = await archive.recordAuthorizedPayload({

src/plugins/evidence-archive-search-plugin.ts

@@ -229,12 +229,16 @@ async function readArchiveOccurrence(
   if (occurrenceId === undefined) {
     throw new Error("read_file archive path must be archive:///{occurrenceId}");
   }
-  const text = await archive.readAuthorizedPayload(occurrenceId);
   const offsetArg = num(args.offset);
   const offset = offsetArg !== undefined && offsetArg > 0 ? Math.floor(offsetArg) : 0;
   const limitArg = num(args.limit);
   const limit =
     limitArg !== undefined && limitArg > 0 ? Math.floor(limitArg) : READ_FILE_DEFAULT_MAX_LINES;
+  signal.throwIfAborted();
+  const page = await archive.readBinaryPage(occurrenceId, offset, limit);
+  signal.throwIfAborted();
+  if (page !== undefined) return page;
+  const text = await archive.readAuthorizedPayload(occurrenceId);
   const result = await readBytesBounded(
     new TextEncoder().encode(text),
     offset,
@@ -344,7 +348,10 @@ async function grepArchive(
     if (occ.gap === true) continue;
     let payload: string;
     try {
-      payload = await archive.readAuthorizedPayload(occ.occurrenceId);
+      payload =
+        occ.binary !== undefined
+          ? JSON.stringify({ ...occ.binary, contentHash: occ.contentHash })
+          : await archive.readAuthorizedPayload(occ.occurrenceId);
     } catch (err) {
       if (signal.aborted) throw err;
       continue;

src/session/assemble-runtime.ts

@@ -60,6 +60,7 @@ import {
   type CompactionArchive,
 } from "./compaction-archive.js";
 import path from "node:path";
+import { captureSyntheticToolResult } from "./synthetic-tool-result.js";
 import {
   loadProjectTrust,
   isPluginTrusted,
@@ -415,6 +416,12 @@ export function assembleChatAgent(wiring: ChatAgentWiring): AssembledChatAgent {
           getLiveFleetCount: wiring.getLiveFleetCount,
         },
       );
+      const archive = wiring.evidenceArchiveHolder?.current;
+      const decide = d.decide.bind(d);
+      d.decide = async (event, ...rest) => {
+        await captureSyntheticToolResult(archive, event);
+        return decide(event, ...rest);
+      };
       directorHolder.instance = d;
       return d;
     },

src/session/compaction-archive-schema.ts

@@ -19,6 +19,12 @@ export const ArchiveOccurrence = type({
   "lifecycle?": ToolRecordingLifecycle,
   "provenance?": "string",
   "gap?": "boolean",
+  "binary?": {
+    encoding: "'raw'",
+    contentType: "string",
+    name: "string",
+    byteLength: "number.integer >= 0",
+  },
 });
 export type ArchiveOccurrence = typeof ArchiveOccurrence.infer;
 

src/session/compaction-archive.test.ts

@@ -996,10 +996,13 @@ describe("wrapCompactorWithCompletenessGate", () => {
 
     await archive.recordAuthorizedPayload({
       kind: "attachment",
-      payload: JSON.stringify({
-        contentHash: hashAuthorizedBytes(pdfBytes),
-        mimeType: "application/pdf",
-      }),
+      payload: pdfBytes,
+      binary: {
+        encoding: "raw",
+        contentType: "application/pdf",
+        name: "document.pdf",
+        byteLength: pdfBytes.byteLength,
+      },
     });
     const allowed = await wrapped.apply(turns, ctx);
     expect(allowed.record.reason).toBe("compact");
@@ -1041,17 +1044,23 @@ describe("wrapCompactorWithCompletenessGate", () => {
 
     await archive.recordAuthorizedPayload({
       kind: "attachment",
-      payload: JSON.stringify({
-        contentHash: hashAuthorizedBytes(bytes),
-        mimeType: "application/octet-stream",
-      }),
+      payload: bytes,
+      binary: {
+        encoding: "raw",
+        contentType: "application/octet-stream",
+        name: "media",
+        byteLength: bytes.byteLength,
+      },
     });
     await archive.recordAuthorizedPayload({
       kind: "attachment",
-      payload: JSON.stringify({
-        contentHash: hashAuthorizedBytes(bytes),
-        mimeType: "application/octet-stream",
-      }),
+      payload: bytes,
+      binary: {
+        encoding: "raw",
+        contentType: "application/octet-stream",
+        name: "media",
+        byteLength: bytes.byteLength,
+      },
     });
     for (const type of ["video", "audio"] as const) {
       const turns: import("@intx/types/runtime").ConversationTurn[] = [