Merge pull request #842 from corbitsdev/cl-7521-preserve-compaction-evidence-with-fresh-working-context-and

Preserve evidence across primary agent compaction

Author: TheGreatAxios

docs/ARCHITECTURE.md

@@ -364,7 +364,8 @@ tool call
 **Rejection behavior:** Any plugin can short-circuit by returning a `ToolResult` with `isError: true`; the error propagates to the agent and downstream plugins/execution are skipped.
 
 - **Result truncation / leisure materialization** (`result-truncation-plugin.ts`, `tool-result-materialize.ts`) — Caps model-facing tool results at 10,000 chars (aligned with the reactor size-cap). Over the gate, any non-error result is leisure-materialized first (minified JSON → pretty `application/json`; NDJSON preserved; else `text/plain`), then the formatted bytes are spilled to the session blob store under `{callId}:full` and truncated inline with a `tool-output:///` URI plus absolute `contextDir/tool-output/…` path when plumbed. Under-gate results are unchanged (no pretty, no spill). Posix tools go through the middleware in `buildCorePosixToolPlugins` (Codex `posixTools.run` included). Fleet AgentTools (`wait_agents`, `search_agents`, …) skip that posix chain, so the same helper wraps them at mount in `createAgentToolset` and nested `runSubAgent`. MCP tools apply the same scrub-then-truncate path via `mcpClientToAgentTools` since they skip both.
-- **Path Escape** (`path-escape-plugin.ts`) — Canonicalizes path-like arguments against `cwd` and blocks `..` escapes, except into a root the permission layer's worktree-roots provider allowlists (e.g. a sibling git worktree of the same repo). Runs first so later plugins see resolved paths.
+- **Path Escape** (`path-escape-plugin.ts`) — Canonicalizes path-like arguments against `cwd` and blocks `..` escapes, except into a root the permission layer's worktree-roots provider allowlists (e.g. a sibling git worktree of the same repo). `tool-output://` and `archive:///` refs pass through unresolved. Runs first so later plugins see resolved paths.
+- **Evidence archive** (`evidence-archive-search-plugin.ts`, `evidence-archive-path-guard.ts`) — Primary-session compaction evidence is a first-class search/read surface on `search_files` / `read_file` / `grep` via `archive:///` refs. Dump paths (`evidence-archive/`, `tool-output/archive-*`) stay blocked so the on-disk sidecar is not the retrieval API. Blob keys reject `/` so they cannot nest under `tool-output`.
 - **Tool-output URI** (`tool-output-uri-plugin.ts`) — Normalizes mistaken `read_file` blob URIs to `tool-output:///id` (corbits-only; interchange stays unpatched).
 - **Secret Guard** (`secret-guard-plugin.ts`) — Hard-denies path-keyed tool calls (`read_file`, `write_file`, …) that would put a sensitive file into (or write it from) the model context. Runs before the permission plugin, so the path-arg deny holds even under `--dangerously-skip-permissions`. Shell commands that _reference_ a sensitive path (tokenized so `cat .env`, `bun --env-file=.env run …`, and quote/env-assignment forms are detected) are not hard-denied here: they require operator approval via the permission gate, and auto mode forces an ask through the auto-shell policy (`sensitive-path` rule). Once the operator approves, the command runs. Shell detection is best-effort: token matching defeats quoting and env-assignment/redirection forms but not dynamic path construction (variable indirection, `printf` assembly). Tool-result secret scrub still redacts credential-shaped output.
 - **Authorization** (`run-shell-authz.ts`, wired by `authz-plugin.ts`) — Denies catastrophic shell command patterns by regex, and hard-blocks shell `find`, head-position `rg`, and recursive `grep -r` (they can walk huge trees and OOM the host). Bounded `grep`/`search_files` tools remain practical alternatives (timeout + output caps); the patterns match those three command shapes only — an `ls -R`, `fd`, or scripted `os.walk` is just as unbounded and is not caught, so the block message tells the model not to substitute one. The permission gate’s shell auto-allow path consults the same policy so it never pre-approves a command authz would reject.

docs/IMPLEMENTATION.md

@@ -79,7 +79,10 @@ src/
     index.ts              Session lifecycle
     state.ts              RunState JSON save/load
     compactor.ts          Context compactor
-    summarizer.ts         Model-backed structured compaction summary (+ deterministic fallback)
+    summarizer.ts         Model-backed structured compaction summary (fails closed)
+    summary-excerpt.ts    Token-budgeted archive excerpt for the summary call
+    compaction-archive.ts Primary-only authorized evidence archive (post-policy capture)
+    compaction-archive-schema.ts Archive occurrence / completeness certificate schemas
     run-sink.ts           Run-level event sink
     stream-consumer.ts    Async stream consumer with error handling
     hooks.ts              Lifecycle hooks: discovery, turn collector, run summary
@@ -114,6 +117,8 @@ src/
     data-only-agent.ts         Markdown-only agent plugins (agents/*.md)
     loader.ts                  Plugin discovery + loadPluginEntry
     path-escape-plugin.ts      Path sandboxing (first)
+    evidence-archive-search-plugin.ts  archive:/// search/read via posix tools
+    evidence-archive-path-guard.ts     Block dump-path reads of the archive sidecar
     tool-output-uri-plugin.ts  Normalize read_file tool-output URIs
     secret-guard-plugin.ts     Hard-deny path-keyed secret files
     authz-plugin.ts            Catastrophic command blocking (thin wrapper)

evals/compaction/README.md

@@ -0,0 +1,151 @@
+# Primary compaction mechanics baseline
+
+This is the frozen, offline component baseline for the primary compaction
+replacement. It is **not** a complete TUI/exec, permission-isolation, attachment
+security, or live model-quality evaluation. The bounded integration-harness
+scope has Greybeard approval; production `src/` remains unchanged.
+
+## Reproduce
+
+```bash
+bun test ./evals/compaction/metrics.test.ts ./tests/integration/compaction-baseline.test.ts
+```
+
+The serial integration test uses the existing `openIntegrationSession`,
+`runUntilDone`, and `closeIntegrationSession`. Optional test-only wiring registers
+`createSessionPruningCompactor` with `createModelSummarizer`, and supplies the
+normal `buildCompactionContinuationMessage()` delivery callback. The actual
+primary director, governor, reactor, toolset, and optimized git-backed store run.
+No test calls the compactor directly, rewrites history, or uses a substitute
+compaction implementation. Existing harness callers remain unchanged.
+
+The harness defaults to `permissiveAuthorize`; this fixture also bypasses tool
+permission prompts in its temporary workspace. **It proves no authorization,
+approval-resume, or permission-isolation property.** No production permission or
+repeat guard is changed. There is no production hook, second host, network
+server, or paid provider request.
+
+## Frozen protocol
+
+- Product revision: `6ea596945657f3a0d3af5bfb0277b94af19e1589`, package `0.3.18`.
+- Research revision `b92dad53` is not the baseline. No version bump is included.
+- Protocol/model script: `primary-component-mechanics-v1` in the integration test.
+- External inference: `@intx/inference-testing` `0.3.0`, Anthropic wire format,
+  source/model `anthropic:claude-integration` / `claude-integration`.
+- Vendored Interchange base: `0205b07b64d03f0fec2e4be3593c764070a9ba8a`, with
+  repository-local patches recorded in `docs/VENDORING.md` and the patch ledger.
+- Runtime of the captured sample: Bun `1.3.14`, Darwin arm64.
+- Production policy: six recent turns; production anchor and no-op rules;
+  model summary limit 4,000 characters and deterministic factory limit 2,500.
+- Trigger schedule: primary inference calls **19, 31, 43** report **synthetic**
+  input 200,000; other explicitly scripted setup/growth calls report synthetic
+  input 100 and output 1. Zero cache/thinking fields are synthetic wire fields,
+  not measurements. Evidence-response wire frames use harness defaults, not
+  measured provider usage. Production thresholds and hysteresis are unchanged.
+- Each phase adds ten distinct user/assistant audit-item exchanges, then a
+  distinct real `read_file` call. The governor intercepts the post-tool infer and
+  resumes through the same agent's contentless inbound channel.
+- Time bounds: 30 seconds per send; 120 seconds for the positive fixture.
+
+Git blob identities freeze the uncommitted harness additions without inventing
+a commit revision. Recompute with `git hash-object` on these paths:
+
+- `evals/compaction/fixtures.ts`: `67f1fdfb45272464efc62111c1c7525ed067264b`
+- `evals/compaction/metrics.ts`: `3cc4efadb5dad1f8078b6db412287b0ab5c25e8f`
+- `tests/integration/compaction-baseline.test.ts`: `abed36b077cb16e30bc5015852144bb6bb7fb5b2`
+- Original captured-run evaluator: `80a627549e0e009ca7c3079e26875baf3407e8e9`
+- `tests/integration/harness.ts`: `4567ac03433b70f1eed4f3238e2a3b5e1f5b4557`
+
+The fixture module deterministically generates the exact input bytes: an early
+constraint, a later corrected decision, a failing `bun diagnose.ts` with decisive
+output after 250 preamble lines, and an oversized diagnostic with the decisive
+value after 1,500 lines. A full read and a targeted middle-line read exercise
+real tools. Before growth, the test verifies that all four facts reached
+persisted history. Generated workspace files contain no grader expectations.
+
+## Evidence and scoring
+
+The summarizer responder extracts only evidence markers in the **actual excerpt
+received from the production summarizer**. It never reads the original fixture
+or discarded turns. The primary response matcher selects an answer only when
+its exact set of source/value/id triples is present in the actual wire request,
+independent of their order. A real-agent reversed-order regression recovers all
+four facts without weakening source/value matching.
+All 16 subsets include an explicit all-missing response. A separate real-agent
+negative test supplies no evidence and verifies that fixture answers do not
+appear. These controlled responders measure transport/loss, not model judgment.
+
+`metrics.ts` scores exact source and value, separately from artifact completion.
+Its tests reject altered artifacts, wrong sources/answers, absent evidence,
+repeated work, requested-only folds, no-ops, and missing continuation. Denominators
+remain four required facts per observation; the failed recovery task is retained.
+
+At complete `runUntilDone` boundaries the fixture reads and validates the small
+`turns.jsonl` directly, without an in-flight `store.load()` or recovery read. A
+qualifying fold requires changed persisted SHA-256 bytes, fewer persisted turns,
+an additional production compacted-context marker, a new summarizer invocation,
+and primary continuation inference. Requests alone cannot qualify. This proves
+persisted replacement in a completed run, not crash atomicity or restart recovery.
+
+The work counters derive from actual tool start/done events. Failed shell calls
+include the production `exit code <nonzero>\n` content prefix, even without
+`isError`. A regression executes `exit 7` twice through real tools and observes
+two failures and one repeated failed attempt. The three fixed
+phase-end reads are labelled verification by their frozen call IDs, not by a
+model-provided excuse. Other repeated reads/searches, repeated failed attempts,
+and duplicated edits are distinct metrics. No search or edit is prescribed here;
+zero repetition is not evidence of capable live problem-solving.
+
+## Captured outcome
+
+`results/baseline.json` retains one successful mechanics run, including all three
+observations, persisted hashes, phase latencies, and the failed recovery result.
+
+- Mechanics task qualification: **1/1**; persisted folds **3/3**.
+- Persisted turn counts: **36 → 8**, **28 → 10**, **30 → 12**.
+- Continuation primary calls: **20, 32, 44**.
+- Required-fact recovery after each fold: **1/4**; full-recovery tasks **0/1**.
+- Only the initial constraint survives. The corrected decision, failed-command
+  evidence, and decisive oversized-output fact are lost from the primary reply.
+- Three actual summarizer calls; six tool calls; three verification reads;
+  zero observed repeated reads/searches, repeated failed attempts, or duplicated edits.
+- Captured phase latencies: approximately **757, 796, 801 ms**. They include the
+  tool call, folding/persistence, continuation and reply, not compaction alone.
+- Positive fixture duration: approximately **12.12 seconds**, including setup and growth.
+
+Primary/summarizer token totals, real cache reads/writes, monetary cost,
+compaction-only latency, and live completion quality are **unavailable**, not
+zero. Persisted hashes include runtime timestamps and legitimately vary between
+runs; the frozen source hashes identify the repeatable protocol.
+
+The test characterizes the observed baseline loss; passing tests do not mean
+factual recovery passes. Replacement comparison must reuse these fixture bytes,
+trigger schedule, budgets, and exact-source grader. Keep this captured result
+unchanged and report improved recovery separately rather than weakening the
+grade or excluding the baseline failure.
+
+## Remaining scope
+
+Real TUI/exec host continuity, workflow/controller state, approvals, worker/task
+ownership, attachments, concurrent incoming messages, recovery, and finalization
+belong to Unit 6 and the Unit 8 cross-surface matrix. Archive exactness and
+security belong to Units 2–4. Live quality and spend-approved token/cache/cost
+comparison belong to Unit 8. These requirements moved; they were not removed.
+
+## Verification
+
+The focused command above passes (9 tests, 47 assertions with the evaluator
+regressions; the original captured run has 7 tests and 44 assertions). Results
+retain the original sample and record corrected-evaluator verification separately;
+fixture bytes, trigger schedule, and the observed 1/4 baseline recovery are unchanged.
+Required regression and repository gates:
+
+```bash
+bun test ./src/agent/compaction.test.ts ./src/context-compactor.test.ts ./src/session/runtime-assembly.test.ts ./src/session/optimized-context-store.test.ts ./tests/unit/compactor-pairing.test.ts
+bun run typecheck
+bun run build
+bun run test
+bun run check
+```
+
+No commit or release action is part of this fixture.

evals/compaction/fixtures.ts

@@ -0,0 +1,48 @@
+import type { Evidence } from "./metrics.js";
+
+export const BASELINE = {
+  productRevision: "6ea596945657f3a0d3af5bfb0277b94af19e1589",
+  packageVersion: "0.3.18",
+  protocol: "primary-component-mechanics-v1",
+  model: "claude-integration",
+  provider: "anthropic",
+  folds: 3,
+  growthTurnsPerFold: 10,
+  syntheticLowInput: 100,
+  syntheticTriggerInput: 200000,
+  outputTokens: 1,
+  keepRecentTurns: 6,
+  summaryMaxChars: 4000,
+  wallTimeoutMs: 30000,
+} as const;
+
+export const REQUIRED_EVIDENCE: readonly Evidence[] = [
+  { id: "constraint", source: "operator:initial", value: "no-schema-change" },
+  { id: "decision", source: "operator:correction", value: "west-not-east" },
+  { id: "failure", source: "command:diagnose", value: "unsupported-format-7" },
+  {
+    id: "decisive",
+    source: "file:diagnostic.log:middle",
+    value: "route-cobalt",
+  },
+];
+
+export function evidenceText(facts: readonly Evidence[]): string {
+  return facts
+    .map((fact) => `[[evidence:${fact.id}|${fact.source}|${fact.value}]]`)
+    .join("\n");
+}
+
+export const INITIAL =
+  "Audit the deployment. Preserve this constraint: " +
+  evidenceText(REQUIRED_EVIDENCE.slice(0, 1));
+export const CORRECTION =
+  "Correction: target west instead of east. " +
+  evidenceText(REQUIRED_EVIDENCE.slice(1, 2));
+export const FAILED_OUTPUT =
+  "Diagnostic preamble.\n".repeat(250) +
+  evidenceText(REQUIRED_EVIDENCE.slice(2, 3));
+export const OVERSIZED_OUTPUT =
+  "Unrelated diagnostic row.\n".repeat(1500) +
+  evidenceText(REQUIRED_EVIDENCE.slice(3)) +
+  "\nUnrelated trailing row.".repeat(1500);

evals/compaction/metrics.test.ts

@@ -0,0 +1,123 @@
+import { describe, expect, test } from "bun:test";
+import { type } from "arktype";
+import {
+  grade,
+  Measurement,
+  qualifyingFold,
+  recoverEvidence,
+  repeatedWork,
+  type Fold,
+} from "./metrics.js";
+
+const fact = { id: "region", source: "operator:correction", value: "west" };
+const folds: Fold[] = [10, 20, 30].map((call) => ({
+  requestedAtCall: call,
+  beforeHash: `before-${call}`,
+  afterHash: `after-${call}`,
+  beforeTurns: 20,
+  afterTurns: 10,
+  persisted: true,
+  continuedAtCall: call + 1,
+}));
+const baseline = {
+  expected: [fact],
+  recovered: [fact],
+  expectedArtifact: "west\n",
+  artifact: "west\n",
+  folds,
+  trace: [],
+};
+
+describe("compaction baseline grading", () => {
+  test("requires evidence actually visible to the scripted responder", () => {
+    expect(
+      recoverEvidence("[[evidence:region|operator:correction|west]]"),
+    ).toEqual([fact]);
+    expect(recoverEvidence("The region was mentioned earlier.")).toEqual([]);
+    expect(
+      grade({ ...baseline, recovered: recoverEvidence("evidence removed") })
+        .factualRecovery,
+    ).toBe(false);
+  });
+
+  test("rejects wrong values, sources, and altered artifacts independently", () => {
+    expect(grade(baseline).passed).toBe(true);
+    expect(
+      grade({ ...baseline, recovered: [{ ...fact, value: "east" }] }).passed,
+    ).toBe(false);
+    expect(
+      grade({ ...baseline, recovered: [{ ...fact, source: "invented" }] })
+        .passed,
+    ).toBe(false);
+    const altered = grade({ ...baseline, artifact: "east\n" });
+    expect(altered.completion).toBe(false);
+    expect(altered.factualRecovery).toBe(true);
+  });
+
+  test("keeps failed fold denominators and rejects requests, no-ops, and missing continuation", () => {
+    expect(grade({ ...baseline, folds: [] }).qualifying).toBe(false);
+    for (const fold of folds) {
+      expect(qualifyingFold({ ...fold, persisted: false })).toBe(false);
+      expect(qualifyingFold({ ...fold, afterHash: fold.beforeHash })).toBe(
+        false,
+      );
+      expect(qualifyingFold({ ...fold, continuedAtCall: null })).toBe(false);
+      expect(qualifyingFold({ ...fold, afterTurns: fold.beforeTurns })).toBe(
+        false,
+      );
+    }
+    expect(grade({ ...baseline, recovered: [] }).requiredFacts).toBe(1);
+  });
+
+  test("counts repeated work separately from legitimate scheduled verification", () => {
+    const read = {
+      name: "read_file",
+      argumentsKey: "a",
+      outcome: "success",
+      purpose: "action",
+    } as const;
+    const search = { ...read, name: "grep" };
+    const failure = { ...read, name: "run_shell", outcome: "failure" } as const;
+    const edit = { ...read, name: "edit_file" };
+    const trace = [
+      read,
+      read,
+      search,
+      search,
+      failure,
+      failure,
+      edit,
+      edit,
+      { ...read, purpose: "verification" } as const,
+    ];
+    expect(repeatedWork(trace)).toEqual({
+      repeatedReads: 1,
+      repeatedSearches: 1,
+      repeatedFailedAttempts: 1,
+      duplicatedEdits: 1,
+      verificationCalls: 1,
+    });
+    expect(grade({ ...baseline, trace }).passed).toBe(false);
+  });
+
+  test("missing usage is unavailable, not zero or an unlabelled estimate", () => {
+    expect(
+      Measurement({ status: "unavailable", reason: "offline" }) instanceof
+        type.errors,
+    ).toBe(false);
+    expect(
+      Measurement({ status: "reported", value: -1, unit: "tokens" }) instanceof
+        type.errors,
+    ).toBe(true);
+    expect(
+      Measurement({ value: 0, unit: "tokens" }) instanceof type.errors,
+    ).toBe(true);
+    expect(
+      Measurement({
+        status: "synthetic",
+        value: 200000,
+        unit: "trigger tokens",
+      }) instanceof type.errors,
+    ).toBe(false);
+  });
+});