startup: prewarmed boot, budgeted companion join, detached cold-catalog fetch

build.zig

@@ -156,6 +156,7 @@ pub fn build(b: *std.Build) void {
     unit_tests.root_module.addAnonymousImport("spec_prompt_cache", .{ .root_source_file = b.path("spec/kernels/prompt_cache.json") });
     unit_tests.root_module.addAnonymousImport("spec_prompt_prefix", .{ .root_source_file = b.path("spec/kernels/prompt_prefix.json") });
     unit_tests.root_module.addAnonymousImport("spec_prompt_stable", .{ .root_source_file = b.path("spec/kernels/prompt_stable.json") });
+    unit_tests.root_module.addAnonymousImport("spec_acp_early_init", .{ .root_source_file = b.path("spec/kernels/acp_early_init.json") });
     const run_tests = b.addRunArtifact(unit_tests);
     const test_step = b.step("test", "Run unit and subprocess integration tests");
     test_step.dependOn(&run_tests.step);
@@ -178,6 +179,9 @@ pub fn build(b: *std.Build) void {
     const acp_preauth_test = b.addSystemCommand(&.{ "python3", "scripts/test-acp-preauth.py" });
     acp_preauth_test.addArtifactArg(exe);
     test_step.dependOn(&acp_preauth_test.step);
+    const acp_early_init_test = b.addSystemCommand(&.{ "python3", "scripts/test-acp-early-init.py" });
+    acp_early_init_test.addArtifactArg(exe);
+    test_step.dependOn(&acp_early_init_test.step);
     const issue_acp_test = b.addSystemCommand(&.{ "python3", "scripts/test-issue-acp.py" });
     issue_acp_test.addArtifactArg(exe);
     test_step.dependOn(&issue_acp_test.step);

docs/adr/0336-the-repl-update-notice-reads-a-cache.md

@@ -0,0 +1,53 @@
+# 0336. The REPL update notice reads a cache; the network never holds the banner
+
+Status: accepted 2026-11-12
+
+## Context
+
+The interactive REPL banner called `version_status.checkLatest`, a blocking
+GET of GitHub's latest-release endpoint, on every launch. There was no cache
+and no time bound beyond the HTTP client's own. With the check on, the banner
+phase took about 320–350 ms on a normal network (25–28 ms when the request
+failed fast), the largest REPL startup cost. `-p`, ACP and `--json` already
+skipped it; `GRAFF_NO_UPDATE_CHECK=1` disabled it.
+
+## Decision
+
+The banner notice is stale-while-revalidate (`src/update_check_cache.zig`),
+the same shape as model catalogs ([0306](0306-model-catalogs-never-hold-startup.md)):
+
+- The banner reads `~/.graff/update-check.json`
+  (`{"v":1,"tag":"<release tag>","checked_ms":<unix ms>,"tried_ms":<unix ms>}`,
+  `tried_ms` optional) and renders the notice from it. Boot makes no
+  network call.
+- When that file is missing, unreadable, a different `v`, holds an
+  unparseable tag, or is 24 h old or more (or dated in the future), one
+  detached thread refetches the tag and atomically rewrites the file. It has
+  a private `Io` and allocator, nothing waits for it, and exit never joins
+  it.
+- The refresh first records `tried_ms`, keeping the cached tag, and only
+  then sends the request. A success rewrites the file without `tried_ms`. A
+  refresh that fails, hangs, or is cut off by exit therefore holds the
+  following launches off for 30 minutes, not just until the next launch.
+  When the file cannot be written at all, a process starts at most one
+  refresh. The 24 h TTL still applies to successes.
+- The file is untrusted input. Numbers outside i64, or that are not
+  integers, fail the parse and the file reads as missing. Ages are computed
+  with overflow checks: any i64 timestamp, including the extremes, reads as
+  stale or as no back-off, never as a crash. A tag that is not a plain
+  release tag (letters, digits, `.-+`, at most 64 bytes, three or four u32
+  parts) never reaches the terminal.
+- `GRAFF_NO_UPDATE_CHECK` reads no file and starts no thread.
+- The notice wording and the version comparison (`version_status.compare`)
+  are unchanged; a cached tag not newer than the running build shows nothing.
+- `graff update` and `/version` still fetch live, and write the tag they see
+  to the same file.
+
+## Consequences
+
+A new release shows its notice one launch late, and on the first launch
+ever there is no notice. A machine that cannot reach the release endpoint
+makes at most one request per 30 minutes. If HOME cannot be written, it
+makes one per launch, the same as the old per-launch check. Revisit if the
+notice must be same-launch accurate; that needs a bounded wait, not a
+return to the blocking fetch.

docs/adr/0337-a-one-shot-prewarms-the-companion-handshake.md

@@ -0,0 +1,108 @@
+# 0337. A one-shot prewarms the companion handshake; teardown ends retry pauses
+
+Status: accepted 2026-11-12
+
+## Context
+
+`-p` connects the companion (`codedb-pro --mcp`) on the boot thread:
+ADR 0230 defers it only for interactive and ACP sessions
+(`companion_boot.deferCompanion`), so the first request's catalog already
+holds the companion. Its spawn and handshake ran in series after
+credentials, the plugin scan and the MCP merge. With no tool cache, a
+hermetic one-shot with a stub companion sent its first request at a median
+of 24.2 ms, against 14.0 ms with no companion on PATH.
+
+Teardown had the opposite problem. `Registry.deinit` joins every pending
+start, and a start that failed transiently slept 1.5 s before its one retry,
+with no way to wake it. A REPL whose companion failed at once was ready in
+about 50 ms but took about 1.54 s to exit.
+
+## Decision
+
+- **Overlap, not defer.** A one-shot starts the companion connect on its own
+  task (`boot_prewarm.startCompanion`) once credentials have resolved and
+  the banner has settled the cwd. `main` defers `boot_prewarm.discard`
+  right after the start. The boot thread joins the task in
+  `connectCompanion`, exactly where `addServer` ran.
+  The handshake is not deferred past the first request, so the first
+  request's tools, prompt and cache key are byte-identical to the serial
+  path.
+- **Only when the outcome is known in advance.** The prewarm starts only
+  when the registry will be the empty one and this companion is the one
+  `connectCompanion` would try first: lean `-p`, no positionals, no `-w`, no
+  project `.mcp.json`, none of `GRAFF_MCP_PROBE`, `GRAFF_MCP_EAGER`,
+  `GRAFF_MCP_HANDSHAKE_SECS` or `GRAFF_MCP_PROBE_MS`, not opted out in
+  `.harness/settings.json`, and on `PATH`. Every other session keeps the
+  serial path.
+- **Join at the consumer, or discard.** `take` adopts the result only if the
+  cwd, home and registry settings still match, awaits it, and merges it
+  under the registry mutex (ADR 0197). Discard stops the attempt's child,
+  joins the task and tears the outcome down. It applies to a mismatch, or to
+  a prewarm nobody adopted, at three points: in `connectCompanion`, in
+  `Registry.deinitAtExit`, and on any return between the start and the
+  registry (`main`'s defer). The task is never cancelled: a cancel delivered
+  inside the bounded handshake's `Io.Select` returns before the select's
+  read task stops. Instead, the worker's task-local registry publishes each
+  child it spawns (`Registry.spawn_watch`, `boot_prewarm.Watch`). Discard
+  kills that child, so the handshake read ends at once. The worker forgets
+  its child before it reaps it, under the same lock, so a pid that may
+  already be reused is never signalled.
+- **One attempt.** An attempt that ran to a result is the attempt the serial
+  path made. Its error is returned as `addServer`'s would have been, with the
+  same (empty) diagnostics; the companion is not started again. Only when no
+  attempt ran to a result (none started, or discarded on a mismatch) does the
+  consumer call `addServer` inline.
+- **Teardown ends retry pauses.** Each registry owns a heap event,
+  `retry_abort`. `Registry.deinit` sets it before joining, and the pause in
+  `mcp_start` waits on it. A start waiting to retry stops waiting and
+  records its first failure. The registry still joins every task, so no
+  thread outlives `io` (#325).
+- **Formal evidence.** `formal/BootPrewarm.tla` models the fatal exits, the
+  early error return and both discards. Each of these invariants has a
+  negative control: `ExitClean` (no live companion and no unjoined worker
+  once the process exits), `NoExitStall`, `NoRepeatAfterResult`,
+  `JoinedBeforeUse`, `NoStaleAdoption` and `NoRetryAfterAbort`. The model
+  also checks `NoUseAfterTeardown`, but the runtime's join rules that out by
+  construction (`formal/boot-prewarm.md`).
+
+## Consequences
+
+- The gain applies to a cold MCP tool cache only: the first `-p` per HOME,
+  or the first after the companion binary changes. There, `-p` reaches its
+  first request in about `max(companion, rest of boot)` instead of their
+  sum. With a warm cache the companion is restored dormant in both paths:
+  the prewarm task spawns no process, and gains and costs nothing
+  measurable. The pre-series binary spawned none either.
+- Measured on macOS, hermetic `-p hi --model lmstudio`, fresh HOME with 15
+  plugins, a stub companion that answers after 6 ms. Medians of the first
+  POST, 20 runs each, alternating with the pre-series binary:
+
+  | Companion | Pre-series | Now | Companion spawns per run |
+  | --- | --- | --- | --- |
+  | answers, cold tool cache | 22.1 ms | 18.6 ms | 1 / 1 |
+  | answers, warm tool cache | 12.6 ms | 12.4 ms | 0 / 0 |
+  | fails (exits 1) | 24.5 ms | 20.2 ms | 2 / 2 |
+  | not on PATH | 11.6 ms | 11.7 ms | 0 / 0 |
+
+  A failing companion is spawned twice by both, because the stdio probe
+  respawns a process that closed. The first revision of this change retried
+  the failed attempt inline: 4 spawns and 32.2 ms.
+- The companion's connect line on stderr can print earlier than other boot
+  diagnostics.
+- A run that fails on credentials or in the banner spawns no companion, as
+  before. Those paths exit through `std.process.fatal`, which runs no defer.
+  Credential resolution measures about 0 ms for a one-shot with an
+  environment key, so starting after it costs no overlap. With a
+  Keychain-only credential the lookups no longer overlap the handshake.
+- On Windows, `Watch` has no signal, so a discard waits for the attempt,
+  which the handshake timeout bounds.
+- An interrupted retry is not attempted again in that process; the next
+  session starts the server again.
+- The plugin scan is not prewarmed. `plugins.applyEnv` stores
+  `plugins.disabled` on the boot thread while a scan worker would read it in
+  `discoverFresh`. With the companion prewarmed, the scan already overlaps
+  the longer handshake. Revisit it if `applyEnv` stops storing an unchanged
+  value, or if the scan becomes the longest boot phase.
+- Moving the one-shot companion to the background entirely, as ADR 0230
+  does for interactive sessions, would make the first request's catalog
+  depend on timing. That stays an owner decision.

docs/adr/0338-acp-answers-initialize-during-boot.md

@@ -0,0 +1,85 @@
+# 0338. ACP answers `initialize` during boot
+
+Status: accepted 2026-11-12
+
+## Context
+
+An ACP client spawns `graff acp` and sends `initialize` first. The dispatch
+loop started only after the root Agent existed, so the answer waited for the
+whole boot: worktree identity, MCP registry, settings, companion, approvals,
+system prompt and root construction. The response itself reads none of that.
+It depends on the request, the `GRAFF_ACP_V2` gate (ADR 0206), the version
+string and whether the dispatch can load sessions. The credential-free path
+already answers it without an Agent.
+
+Answering earlier has three ways to go wrong. The request could be answered
+twice, or not at all. A stdin byte could be lost or reordered between an
+early reader and the loop's reader. Or the live `Dispatch` and the process
+latches that `initialize` sets could differ from the late path's.
+
+## Decision
+
+- Each boot phase mark is also a check, and so is the point where stdin and
+  stdout are created. The check runs on the main thread. It reads stdin only
+  while `poll` reports it readable, so boot never waits for the client. It
+  never reads past a full buffer, and it never reads outside ACP.
+- There is one stdin reader. The ACP inbox pump later reads the same reader,
+  so the bytes after the first line stay where they were.
+- For a complete first line that is an `initialize` request with an id, the
+  engine's own `handleLine` renders the answer on a fresh dispatch whose
+  `can_load` matches the live one. The answer is written only to a clean
+  protocol stdout. Only after the whole line was written and flushed is the
+  request taken off the reader and kept. If rendering or the write fails,
+  nothing stays buffered, the line stays on the reader and the checks stop.
+  The loop then answers it, or fails the way it would have anyway.
+  (Bytes the OS accepted before a write error cannot be taken back. The
+  transport is broken by then, and the loop's write fails too.)
+- Any other first line (no `initialize`, a notification, invalid JSON, or a
+  line larger than the buffer) is left in place. So is an incomplete one.
+  After a non-`initialize` first line the checks stop.
+- When the loop starts, the kept line is dispatched through the normal loop
+  body first, and its output is discarded. The live dispatch fields and the
+  latches are rebuilt by the code the late path runs. The latches are
+  last-writer-wins: the v2 message-id seed is the live dispatch's.
+- One pure `step` makes the decision. It is modelled in
+  `lean-proofs/Graff/AcpEarlyInit.lean`, and the Zig code calls it. Over
+  every boot trace, including ones where the early answer fails, the model
+  proves line bookkeeping only:
+  - the first `initialize` is answered exactly once;
+  - a line is taken only after its answer was sent;
+  - the replay writes nothing;
+  - a non-`initialize` first line reaches the loop unchanged;
+  - stdout and the dispatch history equal the late path's.
+  It assumes that the response is a function of the request alone, and that
+  equal dispatch history means equal dispatch state. It says nothing about
+  bytes, the poll, the reader or timing.
+- The byte-equality claim rests on two tests, not on the Lean kernel:
+  - the Zig conformance test, which runs the real engine early and late on
+    each of the 360 cells of the `initialize` cube and compares bytes and
+    latches;
+  - `scripts/test-acp-early-init.py`, which drives the built binary at
+    random write offsets during boot and compares stdout with a client that
+    writes after boot.
+- A build without ACP (`-Dacp=false`, `-Dminimal`) declines at comptime, so
+  the engine is not compiled in.
+- No thread is added, so no TLA+ model is needed (`formal/README.md`).
+
+## Consequences
+
+`initialize` arrives within one boot phase of the request instead of after
+root construction. `session/new` and everything after it still wait for the
+loop. The no-credential pre-auth path is unchanged.
+
+Differences from the late path:
+
+- If boot fails after the early answer, the client has seen `initialize`
+  succeed and then sees EOF. Before this change it saw EOF alone.
+- The handshake log's `startup` phase still measures time to the loop. It
+  times the replay, not when the client was answered.
+- A request that arrives during root construction is answered at the
+  `root agent` mark, just before the loop would have answered it. With
+  worktree identity resolved in-process that phase takes about 2 ms, so
+  `buildRootAgent` has no checks of its own. If it grows long again, it
+  needs a check between its steps; none exists today.
+- Windows has no `poll` on stdin. There, only bytes already buffered count,
+  which in practice means the late path.