Repository navigation
fix(server): dev DB seeding no longer writes gigabytes per worktree - #269
Merged
Merged
Conversation
Every migrate-dev-db run copied all visible threads from the live database, pruned most of them, then vacuumed: 7.7 GB written per worktree for a 450 MB result. The prune's selection now runs on a read-only connection to the source, so the copy writes only kept rows, and the already-compact snapshot skips VACUUM. The seed is cached in ~/.t3/dev/seed-cache for a day, keyed on the source's identity, the limits and this script, and each worktree gets a copy-on-write clone of it. --fresh rebuilds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Thread transfer impact✅ Thread transfer remains within every enforced ceiling.
Other scenariosAll 64 scenario measurements match All 64 scenario measurements
Baseline: Scenario and decoded snapshot size10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Seeding a worktree's dev database (
vp run migrate-dev-db, or the firstvp run dev) wrote 7.7 GB every time. The 75-minute SSD audit found this was the largest writer we could attribute (~258 GB/day). Each run copied every visible thread out of the 5.5 GB livestatev2.sqlite, pruned most of it, rebuilt the search index over the full copy, and then ran VACUUM. The result is 450 MB.Fix
~/.t3/dev/seed-cachefor 24 h. It is keyed on the source file's identity (dev/inode/birthtime),--projects/--threads-per-project, and this script's text. A worktree uses a cached seed only if every migration recorded in it is one this checkout knows; it then runs its own newer migrations on the clone. Each worktree gets a copy-on-write clone:clonefileon macOS through the existingCheckoutCloner,copyFile+ FICLONE on Linux, and a plain copy otherwise.--freshforces a rebuild. Entries older than a day, and scratch files left by dead PIDs, are removed on the next build.All safety properties are unchanged. The source is only ever opened read-only (
ATTACH ?mode=roand areadonlyclient). Scheduled tasks, outbox, auth sessions and peer links are cleared exactly as before. Nothing writes under~/.t3/userdata.--no-seed/T3CODE_DEV_SEED=0and the dev-runner's 500 MiB auto-seed limit are untouched.Bytes written
Measured with
proc_pid_rusagediskio_byteswrittenon thenode apps/server/scripts/migrate-dev-db.tsprocess (read after exit viawaitid(WNOWAIT)), against the real 5.5 GB source (default 5 projects × 10 threads):Breakdown on main: about 4.8 GB copying visible threads, about 1.2 GB building the search index, about 1.2 GB pruning, and 0.47 GB for VACUUM INTO.
The data is unchanged. A
.dumpof the seed built by the old path and one built by the new path differ only in one project row's order and in the live event sequence, which had advanced between the two runs. Both give 86 threads, 66,965 events, 16,619 turn items and 1,296 indexed messages.integrity_checkreturns ok, the header is in WAL mode, and auth/task/link/outbox tables are empty.Tests
A new case in
migrate-dev-db.test.tschecks that a second seed reuses the first one's snapshot, even after the source changes, and that a seed a day older is rebuilt and leaves one cache entry. It fails on main ("the second seed reuses the first's snapshot: expected true to equal false"). The existing cases cover both copy paths: 11 use the source selection and 3 older-schema sources use the fallback. The destination-overlap case now targetsstatev2.sqliteitself, since the snapshot no longer lives next to it.vp run check:changedpasses.Follow-up worth considering, not in this PR: the 500 MiB auto-seed limit exists because building used to cost several GB. With the selected copy, a 5.5 GB source builds with 0.74 GB of writes, so the dev runner could auto-seed larger sources.
Model: Claude Opus 5.5, Claude Code harness in Ada.
🤖 Generated with Claude Code