Skip to content

fix(server): dev DB seeding no longer writes gigabytes per worktree - #269

Merged
jayleaton merged 1 commit into
mainfrom
fix/dev-seed-writes-20261117
Nov 17, 2026
Merged

jayleaton merged 1 commit into
mainfrom
fix/dev-seed-writes-20261117

Conversation

@jayleaton

Copy link
Copy Markdown
Owner

Seeding a worktree's dev database (vp run migrate-dev-db, or the first vp run dev) wrote 7.7 GB every time. The 75-minute SSD audit found this was the largest writer we could attribute (~258 GB/day). Each run copied every visible thread out of the 5.5 GB live statev2.sqlite, pruned most of it, rebuilt the search index over the full copy, and then ran VACUUM. The result is 450 MB.

Fix

  • Copy only what is kept. The prune's selection (recent projects, stopped thread families, live-thread exclusions) now runs first, on a read-only connection to the source. The copy then writes only those threads and projects. The snapshot prune still re-runs the selection after migrating, so a thread that started running mid-copy is still dropped. This path is only used when the source has no migrations pending for this checkout. Otherwise it falls back to the previous copy of all visible threads.
  • Fewer rewrites. The scratch snapshot runs without a journal or fsync (it is deleted on any failure). The search index is built after the prune, over kept messages only. VACUUM INTO runs only when the prune left the file sparse (more than 25% free pages), which in practice means the fallback path. The seed is fsynced before it is published.
  • Build once, clone per worktree. The seed lives in ~/.t3/dev/seed-cache for 24 h. It is keyed on the source file's identity (dev/inode/birthtime), --projects/--threads-per-project, and this script's text. A worktree uses a cached seed only if every migration recorded in it is one this checkout knows; it then runs its own newer migrations on the clone. Each worktree gets a copy-on-write clone: clonefile on macOS through the existing CheckoutCloner, copyFile + FICLONE on Linux, and a plain copy otherwise. --fresh forces a rebuild. Entries older than a day, and scratch files left by dead PIDs, are removed on the next build.

All safety properties are unchanged. The source is only ever opened read-only (ATTACH ?mode=ro and a readonly client). Scheduled tasks, outbox, auth sessions and peer links are cleared exactly as before. Nothing writes under ~/.t3/userdata. --no-seed / T3CODE_DEV_SEED=0 and the dev-runner's 500 MiB auto-seed limit are untouched.

Bytes written

Measured with proc_pid_rusage diskio_byteswritten on the node apps/server/scripts/migrate-dev-db.ts process (read after exit via waitid(WNOWAIT)), against the real 5.5 GB source (default 5 projects × 10 threads):

Run main this PR
First seed (cold cache) 7.70 GB, 47 s 0.74 GB, 3 s
Second worktree seed 7.70 GB, 47 s 0.000 GB, <1 s
Daily cost, N worktrees 7.7 GB × N 0.74 GB + 0 × (N−1)

Breakdown on main: about 4.8 GB copying visible threads, about 1.2 GB building the search index, about 1.2 GB pruning, and 0.47 GB for VACUUM INTO.

The data is unchanged. A .dump of the seed built by the old path and one built by the new path differ only in one project row's order and in the live event sequence, which had advanced between the two runs. Both give 86 threads, 66,965 events, 16,619 turn items and 1,296 indexed messages. integrity_check returns ok, the header is in WAL mode, and auth/task/link/outbox tables are empty.

Tests

A new case in migrate-dev-db.test.ts checks that a second seed reuses the first one's snapshot, even after the source changes, and that a seed a day older is rebuilt and leaves one cache entry. It fails on main ("the second seed reuses the first's snapshot: expected true to equal false"). The existing cases cover both copy paths: 11 use the source selection and 3 older-schema sources use the fallback. The destination-overlap case now targets statev2.sqlite itself, since the snapshot no longer lives next to it.

vp run check:changed passes.

Follow-up worth considering, not in this PR: the 500 MiB auto-seed limit exists because building used to cost several GB. With the selected copy, a 5.5 GB source builds with 0.74 GB of writes, so the dev runner could auto-seed larger sources.

Model: Claude Opus 5.5, Claude Code harness in Ada.

🤖 Generated with Claude Code

Every migrate-dev-db run copied all visible threads from the live
database, pruned most of them, then vacuumed: 7.7 GB written per worktree
for a 450 MB result.

The prune's selection now runs on a read-only connection to the source,
so the copy writes only kept rows, and the already-compact snapshot skips
VACUUM. The seed is cached in ~/.t3/dev/seed-cache for a day, keyed on the
source's identity, the limits and this script, and each worktree gets a
copy-on-write clone of it. --fresh rebuilds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted Author is trusted by repo permissions or the VOUCHED list. size:L labels Nov 17, 2026
@github-actions

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 21.5 KiB 21.5 KiB 0 B (0.0%) 24.4 KiB ✅
Codex Live turn messages 1 1 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.9 KiB 21.9 KiB 0 B (0.0%) 24.4 KiB ✅
Claude Live turn messages 1 1 0 (0.0%) 8 ✅

Other scenarios

All 64 scenario measurements match main.

All 64 scenario measurements
Provider Scenario Metric Main baseline This PR Impact PR ceiling
Codex short turn resume WebSocket decoded 10.5 KiB 10.5 KiB 0 B (0.0%) 12.7 KiB ✅
Codex short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn running, cold open HTTP decoded 281.7 KiB 281.7 KiB 0 B (0.0%) 302.7 KiB ✅
Codex long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Codex long turn running, history page HTTP decoded 92.4 KiB 92.4 KiB 0 B (0.0%) 107.4 KiB ✅
Codex long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Codex long turn finished, cold open HTTP decoded 381.5 KiB 381.5 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, socket cold open WebSocket decoded 381.6 KiB 381.6 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, work page HTTP decoded 186.2 KiB 186.2 KiB 0 B (0.0%) 200.2 KiB ✅
Codex long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Codex idle thread resume WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell snapshot HTTP decoded 42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅
Codex fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Codex fleet shell subscribe WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Codex fleet shell resume check rows 0 0 0 0 ✅
Codex stale shell reload HTTP decoded 42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅
Codex stale shell reload rows 27 27 0 (0.0%) 27 ✅
Codex stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Codex stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex stale todo list cold open HTTP decoded 72.4 KiB 72.4 KiB 0 B (0.0%) 83.0 KiB ✅
Codex stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Codex stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅
Claude short turn resume WebSocket decoded 10.6 KiB 10.6 KiB 0 B (0.0%) 12.7 KiB ✅
Claude short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn running, cold open HTTP decoded 283.3 KiB 283.3 KiB 0 B (0.0%) 302.7 KiB ✅
Claude long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Claude long turn running, history page HTTP decoded 93.4 KiB 93.4 KiB 0 B (0.0%) 107.4 KiB ✅
Claude long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Claude long turn finished, cold open HTTP decoded 384.2 KiB 384.2 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, socket cold open WebSocket decoded 384.3 KiB 384.3 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, work page HTTP decoded 188.1 KiB 188.1 KiB 0 B (0.0%) 200.2 KiB ✅
Claude long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Claude idle thread resume WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell snapshot HTTP decoded 43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅
Claude fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Claude fleet shell subscribe WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Claude fleet shell resume check rows 0 0 0 0 ✅
Claude stale shell reload HTTP decoded 43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅
Claude stale shell reload rows 27 27 0 (0.0%) 27 ✅
Claude stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Claude stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude stale todo list cold open HTTP decoded 73.4 KiB 73.4 KiB 0 B (0.0%) 83.0 KiB ✅
Claude stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Claude stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅

Baseline: 42a9300 · PR result: f63a3bb · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@jayleaton
jayleaton merged commit 4f61e5f into main Nov 17, 2026
32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant