Repository navigation
startup: prewarmed boot, budgeted companion join, detached cold-catalog fetch - #1896
Merged
Merged
Conversation
bench-startup.py times spawn to --version/--help exit, REPL loop reached, -p first request at a loopback mock, and ACP initialize response, against a C-hello floor. Spawn counts come from a PATH-shim pass. Budgets gate on median/floor ratios and exact git/security spawn counts. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
currentIdentity spawned two git rev-parse per call and ran three times per boot; maybeAutoIsolate spawned rev-parse --is-inside-work-tree before its lean check; the boot reap ran worktree prune + list whenever .graff/worktrees existed. REPL/ACP in a repo paid 7-9 git spawns before ready, -p paid 1. - git_dirs: answer --git-dir/--git-common-dir from the filesystem for a .git directory or gitdir file, mirroring git's discovery (filesystem boundary, HEAD/objects/refs, ownership, repo format). Discovery env vars, bare/inside-git-dir, symlinks, extensions/includes and anything unreadable fall back to git. - worktree_lease: the fallback is one rev-parse for both dirs, memoized per getcwd and invalidated on chdir (task_workspace.enter, presence.rebind). identityAt uses the same path. - maybeAutoIsolate: return before any git work for lean/isolated runs and derive is_git from the identity. - worktree_reap.orphansAtBoot: skip the git spawns when no registration is prunable and no listed tree is a dead session's. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
The banner did a blocking GitHub latest-release GET on every interactive launch (~300 ms). It now reads ~/.graff/update-check.json and refreshes a missing or >24h-old cache on a detached thread with a private Io that neither boot nor exit waits on. graff update and /version still fetch live and write the tag they see to the same cache. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
…pauses -p connected the companion (codedb-pro --mcp) on the boot thread after the plugin scan and MCP merge. boot_prewarm starts that connect after runSubcommand when the registry is known to be the empty one, and connectCompanion joins and merges it under the registry mutex. A cwd or settings mismatch discards it; a quick failure falls back to addServer. Registry.deinit joined a start sleeping 1.5s before its retry. The pause now waits on a per-registry event that deinit sets, so teardown returns promptly and still joins every task. Adds formal/BootPrewarm with negative controls and a draft ADR. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
Each boot phase mark polls stdin without blocking. A complete first line that is an initialize request is taken off the one shared stdin reader and answered through the engine's handleLine. The loop dispatches it first with its output discarded, so Dispatch and the latches match the late path. Any other first line is left for the loop. The decision is acp_early.step, modelled by the AcpEarlyInit Lean kernel: initialize is answered once, the replay writes nothing, and stdout and dispatch history equal the late path's on every boot trace. A conformance fixture runs the real step and engine on every cell. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
Assign 0336 (update-check cache), 0337 (boot prewarm) and 0338 (ACP early initialize) with index rows, list BootPrewarm in formal/README.md and its invariants in formal/change-evidence.md, and reuse catalog_bounded.privateThreaded from update_check_cache instead of a copy. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
REPL and ACP boot no longer spawn git, so their git gates go to 0 in both variants; the REPL-ready and ACP-initialize ratio gates drop to about 2.5x the measured ratios, which also hold with every core busy. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
Moves the process, fixture and report helpers into a sibling module so the benchmark script stays under the 600-line limit. No behaviour change: saved results load and compare byte-identically. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
…p, not cancel - A prewarm attempt that ran to a result is the one attempt: take() returns its error instead of starting the companion again inline (a failing companion was spawned 4 times instead of 2). - main starts the prewarm after resolveKeys and setupWorktreeAndBanner and defers its discard, so a fatal credential or banner exit spawns nothing and an error return before the registry exists discards it. - discard no longer cancels the worker: a cancel delivered inside the bounded handshake's Io.Select left the select's read task running past its frame. The task-local registry publishes each spawned child (Registry.spawn_watch) and discard kills it, then joins. - BootPrewarm.tla models the fatal exits, the early return and stop-then-join, with ExitClean, NoExitStall and NoRepeatAfterResult and four new negative controls; ADR 0337 scopes the gain to a cold tool cache. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
- Ages use checked subtraction: any i64 in the cache file reads as stale or as no back-off; a checked_ms of minInt(i64) crashed every REPL launch. - A refresh records tried_ms (keeping the cached tag) before its request, and a stale cache waits 30 minutes after an attempt before the next; a process starts at most one refresh, so an unwritable HOME does not retry in-process. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
- d_acp_session_usable: spawn to the session/new response written right after initialize, with a ratio budget; d_acp_initialize is documented as the handshake answered. - The lmstudio provider has no URL override, so the mock stays on 1234: each sample sends its own LMSTUDIO_API_KEY token, a POST with another bearer is a foreign client and the sample is retried; binding waits for the port. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
…ion when git cannot answer git_dirs now parses every config file rev-parse reads (system, global with includes, the repository's, GIT_CONFIG_COUNT pairs) the way git's config parser does, and checks the keys rev-parse interprets at startup. Anything it cannot prove git accepts falls back to git, so a corrupt config is again not a checkout. With a peer present, maybeAutoIsolate asks git once and skips when git cannot answer (git < 2.31, a config it rejects), as before, instead of failing boot. autoIsolationRoot reads the common dir in-process. currentIdentity caches only spawns that found a repository, reads the memo only while git_dirs defers, and names a non-git cwd by getcwd rather than a realpath of ".". Tests compare config and layout rows against git, cover a peer plus a git without --path-format, a re-ask after git init and five more reap-gate states; missing git fails under CI. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
…only once answered - boundary() declines at comptime without build_options.acp, so -Dacp=false and -Dminimal no longer link acp_engine. - The early answer is rendered aside, then written and flushed whole. Only then is the line taken off stdin (new step state `answering`, event `sent`, action `take`). On any failure nothing stays buffered and the loop handles the line as on the late path. - Lean kernel, Python ref (new mutant), fixture (415 cells) and conformance test follow the new step table; count checks become examples, and the header and KERNELS.md row state the model's assumptions and limits. - Remove the unused acp_early.check(). - scripts/test-acp-early-init.py drives the built binary at random write offsets during boot and compares stdout with a late client; wired into zig build test. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
0336 gained the refresh back-off and total age math, 0337 now starts after credentials, makes one attempt and stops rather than cancels, and 0338's Lean kernel covers line bookkeeping only. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
…utlasts it Under a loaded test step a debug build can boot slower than the fixed 1 s wait, so the reference itself was answered during boot and the check failed. Double the wait up to four times before failing. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
The startup series and its fixes added 45 unit tests (3921 passed, 6 skipped), putting the floor past the slack; the ratchet warned that it no longer guarded the suite. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
A `-p` one-shot that prewarms the companion (ADR 0337) joined that attempt on the boot thread with no bound of its own, so the attempt's ceilings were the whole wait: the stdio probe's 5 s plus `initialize`'s 15 s. A companion that spawns and then never answers held the first request for 20.1 s (median of 5 hermetic runs) on a run whose answer otherwise takes 23 ms, with nothing on screen naming the wait. A healthy companion answers in 6-50 ms and a missing one is not started at all, so the unbounded join only ever bought patience for the pathological case. The documented knobs could not shorten it either: `applyHandshakeTimeoutEnv` runs in `mcp_boot.initWithMerged`, which `oneshotSkipsImportedMcp` skips on this path, and `wantsCompanion` refuses to prewarm at all when those variables are set. The budget is the right lever because the join is a latency decision, not a correctness one. The attempt is independent of the boot thread; all an unbounded await adds is a catalog that a companion silent for 250 ms is unlikely to deliver inside the first request's lifetime. So `take` waits `default_budget_ms` (250 ms, the ACP `attach_budget_ms` precedent; 1500 ms on Windows) on an `Io.Event` the worker sets as its last write — a poll cost ~5 ms of oversleep per 2 ms step and turned 250 ms into 355 — and then returns without cancelling anything: the watch carries the live registry (`late_merge`, written under the same lock as `done`), so whichever side arrives first consumes the outcome exactly once. The worker merges late through `mcp_boot.adoptStarted`, the one merge point, which drops a latecomer whose server name is already connected instead of advertising every tool twice (#505). `discard` keeps meaning "stop, join, tear down" and now leaves a handed-off attempt alone; `Registry.deinitAtExit` calls the new `stopAtExit`, which stops and joins it before the registry goes away. `GRAFF_MCP_COMPANION_BOOT_MS` is the knob: read where the prewarm starts, capped at 60 s, unset/0/garbage meaning the default, and deliberately not one of the variables that disable the prewarm. Measured on macOS, hermetic `-p hi --model lmstudio`, a stub `codedb-pro` first on PATH, medians of wall clock over the same tree built with and without this change: silent stub 20 134.0 ms (n=5) -> 266.1 ms (n=5); a 600 ms stub 624.6 ms (n=5) -> 265.9 ms (n=5); a 6 ms stub 23.3 ms (n=8) -> 23.0 ms (n=8), with the same `codedbpro (search, read)` on the first request. Trade-off: a cold-and-slow companion can miss the one-shot catalog. It keeps connecting and merges when it lands — the catalog change `/mcp add` makes, which busts the prompt cache from that turn (ADR 0221) — but one slower than the budget can land after the last request. The silent case loses nothing: it produced no companion tools before either, after 20 s. Interactive and ACP sessions are untouched; they queue the companion on `pending_starts` (ADR 0230) and never create a prewarm slot, so the shared `Registry.deinitAtExit` line stays a no-op for them. The side finding that framed this work — a keyless run spawning the companion before its credential fatal — does not reproduce on this tree: `main` starts the prewarm after `resolveKeysOrRunAcpPreAuth`'s `orelse return`, so `-p hi` with no key and a stub on PATH spawns nothing in ~66 ms. A test pins that ordering. ADR 0339. The fixtures and tests move to `boot_prewarm_tests.zig` (this module was near the 600-line ceiling) with new cases for the budget, the late merge and its duplicate guard, and the knob. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
… hides Review of the boot-budget change (ADR 0339) found three defects in it and three nits; this fixes all six on the same branch. Teardown was bounded only on POSIX. `Watch.signal` returned early on Windows, so `stopAtExit` still awaited the attempt there, and the 20.1 s this change claimed to remove moved after stdout is flushed, where nothing sees it, bounded only by the stdio probe's 5 s plus `initialize`'s 15 s. It now ends the child on both platforms — `SIGKILL` on POSIX, and on Windows `NtTerminateProcess` on the process handle `Child.Id` already is, which is what `Child.kill` issues before it reaps — without waiting or freeing the `Child`, which the worker still owns. With the handshake cap lowered to 1200 ms and a silent child, `stopAtExit` on macOS measures 0-1 ms (n=3) with the kill in place and 1025-1037 ms (n=3) with it suppressed, which is the attempt's own handshake timeout. The Windows path cross-compiles and links (`zig build test-bin -Dtarget=x86_64-windows`) but is not run here: ADR 0339 now states the Windows path as unmeasured instead of asserting it works, and the index row no longer claims more than the measurement. A companion that failed after the budget was reported as a successful connect, with its error dropped and, on `-p`, nothing printed at all. `take` and `connect` now return `Adopted`: `.connected` (the catalog is in the registry), `.late` (still running, this registry is its merge target), or the attempt's error. A failure that lands after the budget has no request left to return to, so the worker records it (`lateOutcome`, `lateError`) and prints one line to stderr, and `connectCompanion` says the same for `.late` on every path — the engine sink when the session has one, stderr on `-p`, whose stdout is the answer. The pre-budget contract is untouched: `take` returns the attempt's own error, `error.McpClosed` included. The two late-merge tests depended on each other and on a fixture landing first, so deleting the hand-off failed the second one on a duplicate connect it never meant to test, and deleting the merge leaked the slot and crashed the next test inside a merge into freed memory. Each test now defers `stopAtExit` before its first assertion, states its own premise (`.late`, empty catalog) instead of inheriting one, and names its property through the worker's published outcome (`.merged`, `.duplicate`, `.failed`) rather than through "nothing visibly went wrong", which a merge that never happened also satisfies. Mutation-checked against the whole filtered suite: dropping `and !s.watch.handOff(reg)` fails the three budget tests in `waitForLate` with the other 120 passing; dropping `w.late_merge = reg` fails the same three the same way, and the tests that follow still pass with no leaked allocations, which is the case that used to segfault the next test; and dropping `done_event.set(io)` fails only the new join-wake test, at `expect(ms < 500)`, which nothing covered before. Nits: `handOff` asserts `late_merge` was null, since the write-once is what "exactly once" rests on; the one-shot's own `std.process.exit` paths call `stopAtExit` first, because `exit` unwinds no `defer` and they used to abandon a live worker and, on Windows, orphan the companion process; and the budget knob clamps above its 60 s cap to the cap instead of silently falling back to the default, with a one-line notice when the companion misses the budget so the degraded catalog is not invisible. Re-verified: `zig build test` 3935 tests, 3929 passed, 6 skipped, 0 failed (3932 before); `scripts/eval-tier1.sh --only fmt,reach,sdk` green; hermetic `-p hi --model lmstudio`, stub `codedb-pro` first on PATH, medians of wall clock: a silent stub 263.5-265.0 ms (n=5, three runs, one spawn per run), a 6 ms stub 20.3 ms (n=8) with `codedbpro (search, read)` on the first request in 8/8, and a keyless `-p` still spawns nothing and exits 1. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
Review of the boot-budget change found one user-visible defect and two false claims in ADR 0339; this fixes all three on the same branch. The exit path ends the attempt it can no longer wait for, and the child's death arrives as `McpClosed` like any other failure, so the worker reported it as one: a healthy companion printed "companion: codedbpro failed to connect after the boot budget (McpClosed); native tools only" on a normal `-p` exit, after the run had already answered. `Watch.settle` now reads the stop intent in the same critical section as the merge target — `stop` sets it before it kills, so an attempt whose read returned `McpClosed` because teardown killed it sees it — and such an attempt publishes `.stopped` instead: recorded, never printed, `lateError` stays null, and `.failed` remains the only state that carries an error. `stopAtExit` returns that outcome, because the slot and its watch are freed at exit and `lateOutcome` has nothing left to read; its three call sites (`Registry.deinitAtExit`, the one-shot's two `std.process.exit` paths) ignore it with `_ =`. A genuine failure that lands after the budget while the run is still going reports exactly as it did before. Measured on the binary, hermetic `-p hi --model lmstudio` against a loopback mock with the stub `codedb-pro` first on PATH: - a stub that never answers, default 250 ms budget — the row the ADR's table measures: the false line 3/3 before, 0/3 now, and the wall clock it measures did not move: 266.7-274.3 ms (n=12, two invocations, medians 268.3 and 268.1) against the 263.5-265.0 ms recorded here before; - a stub that answers in 200 ms on `GRAFF_MCP_COMPANION_BOOT_MS=1`, with the run finishing in ~18 ms so teardown kills the attempt mid-handshake: the false line 3/3 before, 0/3 now, `.stopped` recorded, nothing merged; - a stub that exits 1 at 300 ms while the run waits on a 1 s model response and is NOT ending: that line exactly once, 3/3 before and 3/3 after, on stderr only; - a healthy 5 ms companion: catalog present and no failure line, 3/3 before and after, at the 250 ms default and at a 1 ms budget alike. The test is new and mutation-checked. `boot prewarm: a teardown stop ends the attempt quietly, not as a connect failure` hands the healthy fixture off on a 20 ms budget, calls `stopAtExit` while it is still handshaking, and insists on `.stopped` — not `.failed` — with nothing landed. With the stop/failure distinction deleted in a copy of the tree it is the only test that fails (123 pass, 1 fail), at `expectEqual(.stopped, ended)`; the worktree's filtered suite is green. ADR 0339 named a command that does not run. `zig build test-bin -Dtarget=x86_64-windows` does not link — three undefined symbols, `fchdir`, `setenv` and `unsetenv`, from `src/git_dirs_tests.zig`; the command that passes is `zig build -Dtarget=x86_64-windows`, which cross-compiles the whole binary for that target (`zig-out/bin/graff.exe`). The ADR now names it, says what it proves (the branch builds and type-checks) and what it does not prove (nothing about whether a terminated child's pipe reads return promptly, so nothing about the join's cost on that platform), and the index row matches. ADR 0339 also claimed the late-failure notice goes to "the engine sink when the session has one". That branch is unreachable: `speak` needs `GRAFF_REPL_DEBUG` and an interactive session, and every interactive session is `background` (`deferMcpJoin` is false only for `--json`), so the paths that reach the notice are `-p` and `--json`, where `speak` is already false. Both lines go to stderr; the ADR, the module doc and `companion_boot.zig`'s comment now say so. The ADR also folds in the stop distinction and a plain list of what is still not fixed: a companion killed at exit loses its catalog quietly, `.stopped` and `.failed` cannot be told apart when they coincide, the Windows join is unrun, a late merge can miss the run entirely, and the budget clamps at 60 s. Re-verified: `zig build test` green at 3936 tests (3935 before, one new; the count comes off the artifact via `scripts/eval/tier1_test_binary.py`); `scripts/eval-tier1.sh --only fmt,reach,sdk` green in 3 s; `scripts/check-zig-lines.sh` clean with `boot_prewarm.zig` at 573 lines; and the silent-stub proof re-measured at 266.7-274.3 ms (n=12) as above. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
Boot fetched every keyed dynamic model catalog that had no disk cache and waited for it in catalog_bounded.Batch.wait, up to startup_budget. A cached catalog, fresh or stale, never waited. So the wait was paid exactly when nothing was on disk: every first-ever boot, every fresh HOME, and every boot after a rejected key, since a 401 writes no cache. With an Anthropic key that is one TLS round trip to the models endpoint before the prompt accepts input, for a list only the model pickers read. A cold cache now behaves like a stale one. startSpec (moved with ensureForStartup into router_catalog_boot.zig, router_catalog.zig being at the line ceiling) marks the provider in catalog_landing and detaches the fetch; Kimi's account catalog does the same. The worker only reports through static atomics; the main thread activates the cache it wrote at the next ensureForQuery beat (activateLanded), once. A stale cache's refresh is not marked and still lands for the next process only. Kept: a --model naming a row the baked table lacks still joins the bounded batch, because startup resolves --model before the prompt and would refuse a model the provider serves. Rejected: a small blocking budget on a cold cache, which costs every first keystroke and almost never covers a real round trip. Trade-off: a first-ever boot's model picker shows baked rows until the fetch lands. A failed fetch activates nothing, prints nothing at boot (the listing commands still name it), and is not retried in-process. Exit: the worker writes only its own task, the cache file and static atomics. The workspace router's spec strings were in the session arena main frees on return, which a detached worker could still read; router_config.load now copies them out (outliveSession), covering the paths that already detached. ADR 0340. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
ADR 0340 left one parity gap: a first-ever ACP session whose hydrate ran before a cold boot's detached catalog fetch landed kept the baked rows until a /model command or a new session, while the REPL picker activates landed rows on open. acp_model_option.hydrate now runs router_catalog_boot.activateLanded on every beat — one atomic load per slot when nothing landed, never a fetch — the same beat shape as the REPL's ensureForQuery. Measured with a fresh HOME, a loopback /v1/models behind a dummy Z.AI key (GRAFF_ZAI_URL) answering after 1.2 s, and `graff acp` initialized as t3-code: session/new before the landing served the 7 baked zai rows; after the fetch landed, the same session's model options (re-read through a session/set_config_option response) and the graff/models view listed the landed row with no /model issued. The pre-fix binary kept the 7 baked rows on both re-reads. The new acp_model_option test drives hydrate across a simulated landing; removing the activateLanded call fails it at the activation assertion (mutation-verified in a tree copy). ADR 0340's 'read, not measured' note is replaced with the measured closure.
justrach
added a commit
that referenced
this pull request
Nov 12, 2026
…n, detached cold catalog) and ACP output-length explanation (#1893) Release notes and what's-new text for v0.302.74.65, and bump test_count_baseline to 3945: the startup stack (#1896) grew the suite from 3927 to 3944 and the #1893 fix (#1894) added one more; measured on the merged main (cbeccc7) with scripts/eval/tier1_test_binary.py scan, suite green via zig build test. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Three boot-time waits held the first interaction hostage, in paths a user hits on every launch:
git rev-parseup to 7–9 times per start, and the boot-time worktree reap rangit worktree prune/listwhenever.graff/worktreesexisted. The interactive update banner did a blocking GitHub latest-release GET (~300 ms) on every launch, and an ACP client'sinitializewaited for the entire boot before being answered.-pone-shot prewarms its companion handshake, but the boot thread joined that attempt with no bound of its own — the attempt's ceilings (5 s stdio probe + 15 s initialize) were the whole wait. A companion that spawns and then never answers held the first request for 20,134 ms (median of 5 hermetic runs) on a run whose answer otherwise takes 23 ms, with nothing on screen naming the wait.What changed
-pcompanion handshake prewarms after credential resolution — one attempt, discarded on cwd/settings mismatch (ADR 0337) — and an ACPinitializereceived during boot is answered immediately, modeled by the AcpEarlyInit Lean kernel so stdout and dispatch history equal the late path on every boot trace (ADR 0338).GRAFF_MCP_COMPANION_BOOT_MS, capped at 60 s) on an event the worker sets as its last write, then returns without cancelling: the attempt merges late through the single merge point (which drops a latecomer whose server name is already connected, Duplicate MCP tool definitions reject every turn on the Responses wire (xai/codex): 'Duplicate function definition provided: mcp__codedbpro__read' #505). Teardown stops the attempt on both POSIX and Windows and reports it as stopped, not as a connect failure (ADR 0339).session/set_config_optionalike). A--modelnaming a row the baked table lacks still joins the bounded batch, so startup never accepts a model the provider does not serve (ADR 0340).Measured
Hermetic benchmarks (
scripts/bench-startup.py, loopback mocks, stub companion first on PATH), medians of wall clock over the same tree built with and without the change:-pwith a companion that never answers-pwith a 600 ms companion-pwith a healthy 6 ms companionTrade-offs (recorded in ADRs 0336–0340)
/mcp addmakes (ADR 0221) — but one slower than the budget can land after the last request. The silent case loses nothing: it produced no companion tools before either, after 20 s.pending_starts(ADR 0230) and never create a prewarm slot.Verification
zig build test: green at the branch tip (3944 tests, 6 skipped, 0 failed).scripts/eval-tier1.shfull suite green.scripts/bench-startup.py; the new join-budget and late-merge tests are mutation-checked (dropping the hand-off, the merge target, or the join wake fails exactly the tests that name them).