PostgreSQL round 2: fewer connections and round trips in create, mount, Commit and reads (layerfs-project #61) by yifanxuaaa · Pull Request #597 · Ephemeral-AI-Lab/layerfs

yifanxuaaa · 2026-11-06T12:20:12Z

PostgreSQL Store performance, round 2 (Ephemeral-AI-Lab/layerfs-project#61): fewer connections and round trips in Sandbox create, mount, Commit and sequential committed reads. Closes the product side of #64, #95, #96, #97 and #66; measurement in Ephemeral-AI-Lab/layerfs-benchmark#10.

Changes (each merged and gated on its own)

Issue	Commits	What
#64 mount	`7af41a4eb7`, `9962df8d29`, `ed797caaf7`, `f773d9b8c1`	A CLI mount opens no Store connection: the daemon resolves the Branch by name (option B, owner-approved 2026-11-06). One idle reader opens after the daemon reports ready, never delaying readiness, within the pool cap of 4 (owner-approved).
#95 Commit	`617ee1f166`, `ad6afd11a9`, `61748eb674`, `723b9573f5`	validate reads through the Workspace's reader (no control reader); one-statement Commit/Layer reads; batched page reads in the difference walk; a request's pooled-leaf values looked up together. Every Commit keeps the held lane (owner rejected the unheld-lane variant).
#96 create	`2399fdb3c3`, `3d0e82acc3`	No TLS connector built for `sslmode=disable` (a new daemon's first connection 55 → 8 ms); the worker finds its project in one statement.
#66	`3a676c8f65`	Provider CLI calls see the command's exit at once instead of at the next 20 ms step.
#97 reads	`b9093b61d2`, `910f686c5a`	8 MiB sequential read-ahead, at most 2 windows (16 MiB) per reader, owner-approved; pack cohorts; followed streams kept in the Store's shared cache state (the daemon makes a client per kernel request, so round 1's read-ahead never worked through FUSE).
#94	`eba818e303`	Phase-attribution count test and Commit phase marks (instrumentation only).

Production LOC: 139,048 → 139,821 (+773) at 76a32a9a10: core +253, engine +468, connect +2, api +50. Each commit records its own. The last commit merges the current layerfs_v2_0_0 (8b701f42f0); its +357 are upstream changes. One conflict, in store/open.rs (both sides added functions), was resolved by keeping both.

Evidence

Counts (statements/connections, Docker PostgreSQL, round 1 → round 2): create 6/2 → 4/3 (the idle reader moved from mount to after readiness); mount 30/3 → 14/0; Commit 2 → 1 connection, 10,000-file Commit about 690 → 251; 64 MiB sequential read 847 → 68; stat case 39 → 24.

Docker subset gate (control b4fa173e40 vs each integration head, two paired rounds, fresh homes and workers): gates 01–05 all PASS on counts, with no aggregate timing flag. Final four-machine run at 76a32a9a10: PostgreSQL − SQLite gap, round 1 → round 2: Docker create +53 → −7 ms, mount +29 → +8, Commit +48 → +36; Lima, Apple Container and native similar (benchmark PR has the tables). SQLite didn't regress (paired checks).

Checks on the merged head 048d604dc1: Clippy -D warnings (persistence, store-service, daemon, control, storage, content, workspace, history, bridge with postgresql/machine) and rustfmt are clean. All 219 test binaries of storage, content, workspace, persistence and store-service ran against PostgreSQL 17.11: 218 pass (1,020 tests). filesystem_attributes fails 1/13 only because its fixture path is relative to the crate directory (the same at base). Count tests postgresql_round_trips, postgresql_open_cost, postgresql_commit_round_trips, postgresql_open_check and postgresql_phase_attribution pass. Durable: NOT_RUN (disabled by owner).

Not done (recorded on the issues): statement pipelining (the sync postgres client has no pipeline API); per-transaction framing of the begin, file_digest and binding_hint writes; status still opens one caller connection.

🤖 Generated with Claude Code

… session (layerfs-project #64) A forwarded Sandbox's workspace.mount and workspace.status (Docker, Lima: `sandbox/direct.rs`) found the Branch through `store_call`, which opened a whole Store service in the caller's process: a fully checked writer and, at the first read, its control reader. A CLI process is new per call, so every mount paid both connections. These lookups now run on the process's one held read session (`store::reader`, #65), as the native and worker routes already did: one light-checked connection, no writer, no full check, no overlay template. A process that already holds the Store's service open reads there and opens nothing. SQLite is unchanged. Counts (store-service `postgresql_open_cost`, wire proxy, PostgreSQL 17): caller side of a mount 2 connections / 16 round trips -> 1 / 12. With measure's statement split (#94): caller 2 / 15 statements -> 1 / 9; mount 3 / 30 -> 2 / 24. The daemon side is unchanged: a fresh daemon's first mount opens its Workspace reader (1 / 4); later mounts reuse an idle one (0 / 1). - store/call.rs: `held_branch`, the PostgreSQL route of the lookup. - store/reader.rs: `branch` (id and head) on the held session; shared `branch_on` with `branch_id`. - store/open.rs: `holds`, whether this process holds a Store open. - sandbox/direct.rs (create agent's file, 3 lines): `branch_of` asks `held_branch` first. - Test: postgresql_open_cost gains the before and after host-mount rows. - Docs: workspace.mount, routing table, Store service (first mount). No tests deleted. Production LOC: 139048 -> 139101 (delta +53); api 20764 -> 20817 (+53), core, engine and connect unchanged Scope: tools/production_loc.py combined (core, engine, connect, api); parent tree vs staged tree; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ts held session Production LOC: 139048 -> 139101 (delta +53), first-parent comparison (pg-r2 b4fa173 -> merged tree), counted with tools/production_loc.py; all in api. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…mand (layerfs-project #97) A sequential base read of committed chunked content now fetches the rest of its own range and 8 MiB past it (was 2 MiB) before it reads: the mapping pages a level at a time (at most 32 per level), then the chunks the cache lacks in one upstream demand (at most 32 MiB / largest chunk ids), instead of one 32-object read wave at a time. Read-ahead runs only when the cache allowance holds two windows (16 MiB, owner cap; was 4 x 2 MiB = 8 MiB) and only after the reader has passed the previous window, so it holds at most two windows per reader. Detection is unchanged: one stream per client, a read must start where the previous one ended; random reads, whole-file objects and memory-only clients never read ahead. Storage serves a demand one cohort of packs at a time, each fitting the 2 MiB pack cache by located length, so a pack a large demand acquires stays cached until its objects are resolved instead of being acquired again by reconstruction. A demand within the bound is one cohort. Counts (store-service postgresql_round_trips, cold reader, 64 MiB cache, 128 KiB reads), before -> after: - 64 MiB non-repeating sequential: PostgreSQL 301 -> 69 statements (locate 152 -> 21, whole-pack 104 -> 40, ranged 45 -> 8); SQLite 505 -> 314 statements - 64 MiB zeros (C09-like): 6 -> 6 on both - stat 1,000 committed files (C18-like): 48 -> 48 on both - resident peak during the 64 MiB read: PostgreSQL 161,372 -> 157,816 KiB, SQLite 163,708 -> 160,360 KiB Tests: new workspace read_ahead (3); round_trips gains a 64 MiB zeros Commit and read and RESIDENT lines; read_pack_locality now asserts each pack is acquired once (was at most twice). No tests deleted. Production LOC: 139048 -> 139233 (delta +185) Scope and method: tools/production_loc.py --root on the parent tree (git archive HEAD) and the staged tree (git checkout-index); core 38677 -> 38706 (+29), engine 63539 -> 63695 (+156), api 20764 and connect 16068 unchanged; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…cohorts Production LOC: 139101 -> 139286 (delta +185), first-parent comparison (pg-r2 eefa936 -> merged tree), counted with tools/production_loc.py; core +29, engine +156. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rk per step (layerfs-project #94) Instrumentation only; no behaviour change. - telemetry: `mark_phases` tells a caller each phase's name at its start and end while the thread records, outside the phase's SQL probe reads, so a measurement can read server-side counters per Commit phase. Untraced work and every result are unchanged. - New count test `postgresql_phase_attribution`: one fs-bench-quick case per fresh daemon Store (C01, C13, C14, C16), step by step as each process of the Docker CLI route makes it: the worker's reach check, the daemon's Store open, the CLI's mount-time Store open and Branch lookup, the daemon's bind and serial window, then each Commit with every phase's product counters and pg_stat_statements calls. It reproduces #71's live counts (create 6/2, mount 30/3, C01 Commit 91/2, C14 1/0). `postgresql_first_open` times a process's first session open against later ones (the process-wide TLS connector, built whatever the sslmode): about 40-70 ms more on Linux. Tests: postgresql_phase_attribution 2/2, postgresql_round_trips 2/2 and postgresql_open_cost 1/1 pass against PostgreSQL 17.11; Clippy clean for layerfs-store-service with postgresql, all targets. No tests deleted. Production LOC: 139048 -> 139067 (delta +19) Scope: tools/production_loc.py combined (api, connect, core, engine; reference absent (retired R9c; last counted 65,417 at tree 498dd19)), parent tree vs staged tree via git archive / checkout-index. Subtotal: engine 63539 -> 63558 (layerfs-store-service telemetry.rs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Production LOC: 139286 -> 139305 (delta +19), first-parent comparison (pg-r2 499078b -> merged tree), counted with tools/production_loc.py; engine (store-service telemetry.rs) +19. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… (layerfs-project #95) A Commit's `validate` (checked_base, prepare_base_install) read the candidate through the Store's control reader. On a fresh daemon that reader is not open yet, so the first Commit opened it only for validate: one connection, the light check and three statement prepares. The bound Workspace's own reader, opened at bind and kept until close, serves the same immutable reads (bind already checks the base through it), so the Commit now opens no control reader. Scope: store-service `store/commit.rs`; doc `store-calls/service.md`. No tests deleted. Production LOC: 139305 -> 139305 (delta 0) (tools/production_loc.py --root on the exported parent and staged trees; combined core+engine+connect+api; reference absent (retired R9c; last counted 65,417 at tree 498dd19)) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…s-project #96) Every new process built the process-wide TLS connector at its first PostgreSQL connection, whatever the URL's sslmode. On Linux that loads the system trust store through OpenSSL and costs about 40-70 ms, and every Sandbox create starts a new daemon process, so each create paid it. `Connector::new` now reads the parsed sslmode: `disable` connects with `NoTls` and never calls `tls()`; `prefer` and `require` build and share the connector exactly as before. The driver returns the raw stream for `disable` before it touches the TLS connector, so such a connection behaves the same. One error case changes: a process whose connector cannot be built used to refuse even a `disable` connection (`Integrity`); it now connects. Connections and statements are unchanged (create 6/2). `postgresql_first_open` (sslmode=disable, one fresh process, after one plain driver connect; first, second, third open): - build image: 46.8 / 5.7 / 6.0 ms -> 9.5 / 6.4 / 6.8 ms (VM load1 10.8 -> 4.6) - runtime (daemon) image: 55.3 / 9.2 / 9.0 ms -> 7.6 / 7.3 / 7.5 ms (VM load1 10.1 -> 4.6) Docs: engine/store/postgresql.md says when the connector is built. Tests (Linux, PostgreSQL 17.11): persistence postgresql_open_check 3/3, postgresql_commit_round_trips 1/1, postgresql_acquisition 4/4, postgresql_identical_ids 3/3, layerstack_removal 2/2; store-service postgresql_round_trips 2/2, postgresql_open_cost 1/1, postgresql_open 1/1, postgresql_phase_attribution 2/2; project init_postgresql 1/1; api postgres_connect 2/2, postgres_refusals 4/4, postgres_sandbox_reach 1/1 (with layerfs-api/machine). Clippy -D warnings for layerfs-persistence with postgresql, all targets (Linux) and lib (macOS). This edits the commit agent's (#95) connection.rs by grant: only `tls()`'s comment and the point where `Connector` chooses the connector. No tests deleted. Production LOC: 139305 -> 139310 (delta +5) Scope: tools/production_loc.py (pinned) on the parent tree and the staged tree, combined core/engine/connect/api production source; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…0 ms step (layerfs-project #66, #96) `tool::run` (every provider CLI call: `container`, `limactl`, `docker`, `ssh`, `sh`, `ps`) and `tool::wait` polled `try_wait` every 20 ms, so a call was answered at the next step after its command ended. #94/#66 measured one `container inspect` at 43.6 ms median as the product runs it against 26.3 ms reaped by a blocking wait: about 17 ms per call of polling. `wait_for` now sleeps an eighth of the command's run so far, at least 1 ms and at most 20 ms: a command is seen within about 12% of its run. Its deadline, the kill of the child or its process group, the reap and the `Unknown` answer are unchanged; nothing is retried. A command that runs for seconds is still checked every 20 ms after its first 160 ms. Where it shows on create: a Lima Sandbox's create opens its daemon forward with two such calls (`sh` to detach the ssh forward, `ps` to check it), and an Apple Container machine record from before 2f531cb still runs one `container inspect` per create. Docker creates run no provider CLI call (the worker uses the Engine API and a followed log stream). New test `a_provider_call_ends_when_its_command_does` (machine_probe): a stand-in `msb` that exits at once must be answered in under 20 ms (median of 9 probes, one call each); it also prints a 26 ms stand-in's median. macOS host, one run each, under other agents' builds: - quick stand-in: 22.78 -> 9.55 ms (load1 40.9 -> 35.4) - 26 ms stand-in: 70.70 -> 47.21 ms The test fails on the parent (22.78 ms) and passes here. Docs: connect/machines/README.md gives the exit-check cadence; api/sdk/sandbox/create.md says what a PostgreSQL create opens (the daemon's own light-checked session; 6 statements on 2 connections, 5 on 1 when the launching side holds its session) and that sslmode=disable builds no TLS connector (2399fdb). `tool.rs` is a shared helper outside the create agent's files (platforms/apple_container/); the edit is confined to `wait_for` and its constants. Tests (macOS host): every layerfs-machine test binary passes (conformance 13, disabled_platforms 2, machine_adapter 2, machine_forward 5 + 2 ignored, machine_lima 10, machine_platform 10, machine_probe 7, machine_providers 10 + 3 ignored, machine_reach 8, machine_spec 5, machine_ssh 4 + 3 ignored, support 4). Clippy -D warnings for layerfs-machine, all targets. No tests deleted. Production LOC: 139310 -> 139312 (delta +2) Scope: tools/production_loc.py (pinned) on the parent tree and the staged tree, combined core/engine/connect/api production source; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…vider-call exit wait Production LOC: 139305 -> 139312 (delta +7), first-parent comparison (pg-r2 435f0a9 -> merged tree), counted with tools/production_loc.py; engine (persistence connection.rs) +5, connect (machine setup/tool.rs) +2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…tement (layerfs-project #95) A Commit's `logical.known` reads the head node's logical size: one `commit` or `layer` row by id. On PostgreSQL that was a full read transaction (BEGIN REPEATABLE READ with SET LOCAL, the SELECT, COMMIT): 3 round trips warm, 4 cold. Both reads are one statement, so they now run as `Mode::Statement` (one autocommit statement, one snapshot) with schema-qualified texts: 1 round trip warm, 2 cold. SQLite is unchanged (`read_statement` is its one read transaction). Scope: persistence `history/catalog.rs` (commit, layer), `backend/postgresql/history/statements.rs` and two new qualified statements under `sql/postgresql/queries/history/`; doc `store/postgresql.md`. No tests deleted. Production LOC: 139305 -> 139313 (delta +8) (tools/production_loc.py --root on the exported parent and staged trees; combined core+engine+connect+api; reference absent (retired R9c; last counted 65,417 at tree 498dd19)) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…r, one-statement logical.known Merges 617ee1f and ad6afd1 only; 5ddaf6d673 (unmoved Commit without a held lane) waits for an owner decision. Production LOC: 139312 -> 139320 (delta +8), first-parent comparison (pg-r2 90bad45 -> merged tree), counted with tools/production_loc.py; engine +8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…equest client (layerfs-project #97) The daemon gives each kernel request its own operation client (FilesystemPort::request -> BoundWorkspace::operation -> StorePorts::client), so the read-ahead slot that lived on CanonicalClient was new for every FUSE READ and no sequential read was ever detected on a mount. This is why gate-01 showed C17 Exec at 847 statements on control and treatment alike, and why round 1's 2 MiB read-ahead never engaged live either. The followed streams now live in the shared cache state: up to allowance / 16 MiB streams (4 for the daemon's 64 MiB), least recently read replaced first. Detection is unchanged (a read must start where an earlier read of the same content ended); the 16 MiB per-reader cap and the 8 MiB window are unchanged. Live path, new daemon mount test mounted_sequential_read (cat of a committed 64 MiB non-repeating file through FUSE, daemon 64 MiB cache), before -> after: - PostgreSQL: 848 -> 69 statements, 541 -> 20 Store demands - SQLite: 1,138 -> 314 statements, 541 -> 20 Store demands - resident growth during the read (VmHWM - VmRSS at start, daemon in process): PostgreSQL 72,820 -> 74,468 KiB, SQLite 72,532 -> 74,976 KiB The Store-service count test is unchanged (69 / 314 statements). Tests: new daemon mounted_sequential_read (2, FUSE); new workspace read_ahead test a_client_per_request_still_reads_ahead. No tests deleted. Production LOC: 139305 -> 139315 (delta +10) Scope and method: tools/production_loc.py --root on the parent tree (git archive HEAD) and the staged tree (git checkout-index); engine 63714 -> 63724 (+10); core 38706, api 20817 and connect 16068 unchanged; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e's shared cache state Production LOC: 139320 -> 139330 (delta +10), first-parent comparison (pg-r2 7832f36 -> merged tree), counted with tools/production_loc.py; engine +10. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rfs-project #64 option B) Owner decision 2026-11-06: mount option B approved. A CLI mount opened a PostgreSQL connection in the caller to find the Branch's id (option A: one light-checked held session, 9 statements). The caller now sends the daemon the project's name and the Branch's (`Request::MountNamed`, wire tag 15), and the daemon finds the Branch with its snapshot in one statement on the Workspace's own Store reader, the read it already made by id. The caller reads no Store on any route (worker, local, forwarded). - core history: `NamedSnapshot` and `HistoryCatalog::branch_snapshot_named`, a default method (stack, Branch, snapshot reads) providers may answer in one statement. - persistence: `branch_snapshot_named` statement (SQLite and PostgreSQL; PostgreSQL runs it alone, no BEGIN/COMMIT) and the provider override. A new query; no schema or format change. - store-service: `Store::bind_named` (`NamedBindRequest`, `BindIntent`; `BindError::NoSuchProject` / `NoSuchBranch`), `ReadSession::snapshot_named`; `lookup::no_project` / `no_branch` are the one source of the refusals. - bridge: `Request::MountNamed`. - daemon: serves it; a missing project or Branch is `Reply::SdkRefused` with the lookup's code and message, after the admission is withdrawn (an uncertain cleanup keeps the original failure). - control: `WorkspaceApi::mount_named`; `serve::mount` sends names and passes the daemon's refusal through unchanged; the caller's Branch-id lookups (`store::service::branch_id`, `store::reader::branch_id`) are removed. `sandbox/direct.rs` (create agent's file): mount calls `serve::mount`. Status still reads its head on the caller's held session (option A). - Exact refusals kept: invalid project name, missing project -> NoSuchProject; missing Branch, `main`, invalid Branch name -> NoSuchBranch, same messages. Tests: daemon `named_mount` (new: success, the six refusals, withdrawn admission); bridge `control_records` gains MountNamed; store-service `postgresql_open_cost` gains the by-name snapshot rows; `postgresql_phase_attribution` models the new mount (no caller steps; `mount.caller_open` and `mount.caller_lookup` removed with the route). No tests deleted. Counts (PostgreSQL 17): CLI mount caller 1 connection / 9 statements (option A) -> 0 / 0; daemon bind unchanged at 1 snapshot statement. Production LOC: 139305 -> 139503 (delta +198); core 38706 -> 38728 (+22), engine 63714 -> 63893 (+179), api 20817 -> 20814 (-3), connect 16068 unchanged; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: python3 tools/production_loc.py --root on the parent (git archive HEAD) and the staged tree (git checkout-index -a). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…s (layerfs-project #64) Owner decision 2026-11-06: one eager idle reader, opened after the daemon reported ready, never delaying readiness. A fresh daemon held only its writer at its first mount, which then opened the Workspace's reader (one connection, the light check). After a `Hello` answered ControlReady has been sent, the daemon now opens one reader into its idle readers on a thread of its own; the first mount takes it (one usable check) and opens no connection. - store-service `ReadSet::prefill` / `Store::prefill_reader`: once per Store (later calls return at once), only with shared readers (PostgreSQL), only when no reader is idle, under the idle cap of 4. One attempt: a failure is counted (`ReuseWork::prefill_failures`) and returned, nothing is kept and nothing opens it again; the first mount then opens its own. `ReuseWork::readers_prefilled` counts the opened one. - daemon `application/connection.rs`: after the ControlReady `Hello` reply is written, `reader_after_ready` spawns the open; a failure goes to the daemon's log. owner.rs and bootstrap.rs are untouched. - A mount that arrives before the idle reader is open opens its own, as before; the late one joins the idle readers. - Server connections per idle PostgreSQL daemon after create: 1 -> 2 (the writer and the idle reader). The maximum is unchanged: writer, lane, control reader, one per mounted Workspace, 4 idle. Tests: store-service `postgresql_open_cost` gains the eager-reader rows (open once, first mount takes it, a refused open is one counted failure and is not tried again); `postgresql_phase_attribution` models it as `create.daemon_idle_reader`; daemon `named_mount` checks a SQLite Store opens none. No tests deleted. Counts (PostgreSQL 17): first mount after readiness 1 connection / 4 round trips -> 0 / 1 (the idle reader's usable check); the open itself, 1 / 4, runs after the create's readiness reply. Create's counts are unchanged. Production LOC: 139503 -> 139590 (delta +87); engine 63893 -> 63980 (+87), core 38728, api 20814 and connect 16068 unchanged; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: python3 tools/production_loc.py --root on the parent (git archive HEAD) and the staged tree (git checkout-index -a). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e reader after readiness Conflicts resolved: persistence postgresql/history/statements.rs (kept #95's by-id and #64's named statements) and docs store-calls/service.md (kept #64's eager-reader text with #95's validate note). Production LOC: 139330 -> 139615 (delta +285), first-parent comparison (pg-r2 2bb6782 -> merged tree), counted with tools/production_loc.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…d demands (layerfs-project #95) The page-skipping difference walk (`compare/sorted.rs`) read each page it expanded in a demand of its own. A Commit's logical count and binding hints walk the inode table and the changed directory listings with it, so on a Store every expanded page cost a locator read (and often an inode-value read): C16 (10,000 new files) paid 108 locator reads in `logical.count` and 173 in `bindings`. Beyond 4,096 new objects the Save's locator cache no longer holds them, so each was a round trip. A page the walk opens is now read in one demand with the pages after it on the same side and level that the walk will certainly expand: only while no page of the other frontier is above that level, and leaving out any identity the other frontier shows (it may be skipped as shared). Each page read ahead is then expanded exactly once, so the walk reads no page it did not read before, none twice, and holds at most `READ_AHEAD_PAGES` = 128 (128 x MAXIMUM_PAGE_BYTES = 1 MiB). No cache bound changes (READ_OBJECT_LIMIT, the 2 MiB published packs and the pooled-value bounds stay), no format change. Counts (PostgreSQL `postgresql_phase_attribution`, fresh daemon per case, held lane unchanged): C16 `logical.count` 150 -> 54 round trips (locator reads 108 -> 10), `bindings` 237 -> 76 (173 -> 20); C16 Commit 552 -> 304 RT on top of the two earlier #95 commits (576 at 435f0a9). C01, C13 unchanged in these phases (their walks fit one demand). The remaining 40 inode-value reads per walk are per pooled leaf in Storage (`encoding/pool/read.rs`), outside this change. SQLite: 2-9 fewer statements per Commit (`postgresql_round_trips`). Pipelining is not done: the synchronous `postgres` client does not expose its inner `tokio_postgres` client, so pipelining independent statements needs the session layer rewritten on `tokio-postgres`; out of scope for #95 (recorded in `store/postgresql.md`). New test: content `compare_read_ahead` (10,000-row change: 189 inode pages in 8 demands, 40 directory pages in 4; every page read was expanded, once; a one-row change reads only its path). No tests deleted. Scope: content `compare/sorted.rs`; docs `core/compare-and-merge/compare.md`, `engine/store/postgresql.md`. Production LOC: 139313 -> 139378 (delta +65) (tools/production_loc.py --root on the exported parent and staged trees; combined core+engine+connect+api; reference absent (retired R9c; last counted 65,417 at tree 498dd19)) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…walk Production LOC: 139615 -> 139680 (delta +65), first-parent comparison (pg-r2 55f9b45 -> merged tree), counted with tools/production_loc.py; core (content compare/sorted.rs) +65. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

910f686) Brings the coordinator's pg-r2 head (before the mount merge) under the #95 Storage batching work, as the coordinator asked. No conflicts. Production LOC: 139378 -> 139395 (delta +17) (first-parent comparison 61748eb -> merged tree; tools/production_loc.py --root on both exported trees; combined core+engine+connect+api; reference absent (retired R9c; last counted 65,417 at tree 498dd19)) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…fs-project #64) `BindRefusal::request` became a `BindIntent` in 9962df8 (option B), and two daemon test targets still read `.branch` / `.workspace` on it, so they no longer compiled: `store_reader_lifecycle` (a failed bind returns the original request) and `installed_store` (a duplicate bind after stop). Both now require `BindIntent::Id` and compare the same two fields, so each assertion is as strong as before and also checks the variant. The route they drive (bind by identity) survives; no tests deleted. Production LOC: 139590 -> 139590 (delta 0); test-only change. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: python3 tools/production_loc.py --root on the parent (git archive HEAD) and the staged tree (git checkout-index -a). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Tests only. Production LOC: 139680 -> 139680 (delta 0), first-parent comparison, counted with tools/production_loc.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ent (layerfs-project #96) A Sandbox create's reach check (control `store/reader.rs` `reach`, on the launching side's held read session) finds its project with `layer_stack_named`: one `layer_stack` row by its unique name. On PostgreSQL that ran as a read transaction (BEGIN REPEATABLE READ with SET LOCAL search_path, the SELECT, COMMIT). It is one statement, so it now runs as `Mode::Statement` (one autocommit statement, one snapshot) with a schema-qualified text, as #95 did for a Commit or Layer by id. SQLite is unchanged (`read_statement` is its one read transaction). Create counts (`postgresql_case_attribution`, C01/C13/C14/C16, server statements via pg_stat_statements; base 91198bf): - create.reach: 5 statements / 5 round trips -> 2 statements / 3 round trips (light check + prepare + SELECT), 1 connection - create.daemon_open: 1 statement, 1 connection (unchanged) - create total: 6 statements / 2 connections -> 3 / 2 (with a held reach session: 5/1 -> 2/1) - create.daemon_idle_reader (#64, after readiness, off the create path): 1 connection, 1 statement, unchanged and not counted above `postgresql_open_cost` "host create (attach + project lookup)": 1 connection, 8 -> 6 round trips. Kept: the held reach session (it still serves a status's head and is reused by the next create within 60 s; closing it would save nothing per create) and the daemon's light-checked open (one statement; lower needs dropping the check or an eager connection). Nothing moves into mount or Commit. Docs: store/postgresql.md lists the read under `Statement`; api/sdk/sandbox/create.md gives the new counts, the idle reader, and drops the stale mount use of the held session (#64 option B). Granted minimal edit in persistence history reads (`history/catalog.rs`, `backend/postgresql/history/statements.rs`, one new SQL text). Tests (Linux, PostgreSQL 17.11): persistence postgresql_open_check 3/3, postgresql_commit_round_trips 1/1, postgresql_acquisition 4/4, postgresql_identical_ids 3/3, layerstack_removal 2/2; store-service postgresql_round_trips 2/2, postgresql_open_cost 1/1, postgresql_open 1/1, postgresql_phase_attribution (case) 1/1, operator_history (SQLite) 2/2; project init_postgresql 1/1; api postgres_connect 2/2, postgres_refusals 4/4, postgres_sandbox_reach 1/1 (with layerfs-api/machine). Clippy -D warnings for layerfs-persistence with postgresql, all targets. No tests deleted. Production LOC: 139680 -> 139684 (delta +4) Scope: tools/production_loc.py (pinned) on the parent tree and the staged tree, combined core/engine/connect/api production source (includes shipped runtime SQL); reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…greSQL statement Production LOC: 139680 -> 139684 (delta +4), first-parent comparison (pg-r2 91198bf -> merged tree), counted with tools/production_loc.py; engine +4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…s-project #95) Reading a pooled inode leaf rebuilds its physical bodies from the chain, then looks up the stored values its ordinals name in one call. A demand of many leaves therefore made one value lookup per leaf whose values the reader did not hold: C16's logical count and binding hints, after the difference walk's read-ahead (61748eb), still paid 40 `inode_value` reads each. The leaf read is now two steps the demand can batch. `PoolReader::rebuild` walks and rebuilds one chain (the same checks and work allowance as before); `prepare_leaves` rebuilds the pooled leaves of the demand from the current one on, in resolve order, and looks up the ordinals of all of them in one call. A window ends at 128 leaves, at 1 MiB of rebuilt bodies, or before the leaf whose values would pass the 7,182 values the reader retains (POOLED_VALUE_CACHE_BYTES / 73); the retained set is dropped whole first when it would pass that bound, as `stored_values` does. Each leaf's own reconstruction then takes its prepared chain and still rebuilds and checks every canonical element by its identity, so a wrong value can only fail the read. A failure while preparing is the demand's failure, returned once (no second attempt). The demand loop in `read/objects.rs` prepares a window when it reaches a leaf not prepared; the cohort's packs are already cached by the prefetch. No bound changed, no format change. Counts (PostgreSQL `postgresql_phase_attribution`, fresh daemon per case, held lane unchanged): C16 `logical.count` 54 -> 17 round trips and `bindings` 76 -> 39 (`inode_value` reads 40 -> 3 each); C16 Commit 304 -> 230 (576 at 435f0a9); C13 first Commit 66 -> 65, C01 unchanged. `postgresql_round_trips` `read.stat_files_1000` 48 -> 30 statements on both engines; SQLite Commit statements unchanged or lower. Resident peak of the 64 MiB reads unchanged (153-157 MB). New test: storage `stored_value_reads` `one_demand_of_many_leaves_looks_up_their_values_together` (300 leaves in one demand: at most 3 value lookups; a wrong value still fails). No tests deleted. Scope: storage `encoding/pool/read.rs`, `read/objects.rs`; doc `core/physical-storage/encoding.md`. Production LOC: 139395 -> 139532 (delta +137) (tools/production_loc.py --root on the exported parent and staged trees; combined core+engine+connect+api; reference absent (retired R9c; last counted 65,417 at tree 498dd19)) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…s together Production LOC: 139684 -> 139821 (delta +137), first-parent comparison (pg-r2 dee3eec -> merged tree), counted with tools/production_loc.py; core (storage encoding/pool/read.rs, read/objects.rs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Brings in #582-#593 (SQLite idle WAL checkpoint, ring admission, chain-root rebase, PG Init cleanup). One conflict: api/crates/layerfs-control/src/store/open.rs, both sides only added functions (#64 holds; #86 idle-checkpoint helpers); kept both. Production LOC: 139821 -> 140178 (delta +357), first-parent comparison (pg-r2 76a32a9 -> merged tree), counted with tools/production_loc.py; the delta is the upstream changes. Checked on the merged tree: Clippy -D warnings (persistence, store-service, daemon, control, storage, content, workspace, history, bridge with postgresql/machine) and rustfmt clean; 219 test binaries of storage, content, workspace, persistence and store-service run one by one against PostgreSQL 17: 218 pass (1,020 tests), filesystem_attributes fails 1/13 only because its fixture path is relative to the crate directory (known, same at base). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Brings layerfs-project #84's storage changes (#92 directory/extent page deltas, pack framing v26; #91 inode-value blocks and #90 narrow PostgreSQL rows, schema v2) under round 2. Conflict only in core/crates/layerfs-storage/tests/stored_value_reads.rs: both sides appended a test; both kept. Tree 87f25ed, as verified in the quick verification (layerfs-benchmark#10). Production LOC: 140178 -> 141111 (delta +933) Scope: combined first-party production source, tools/production_loc.py --json on git archive / checkout-index trees, first parent 048d604. Second parent e20c6a2 (140338). Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

`cargo +1.85.1 fmt --all -- --check` fails on layerfs_v2_0_0 after #594 (#92, head c5604ef) in six files; the #84 final integration 7691ac2 carries the same drift. Formatting only: no token changes. Applied with `cargo +1.85.1 fmt --all`; the whole workspace is fmt-clean afterwards. Files: core/crates/layerfs-content/src/file/construction/runs.rs, core/crates/layerfs-content/src/file/edit/engine.rs, core/crates/layerfs-content/src/file/mapping/build.rs, core/crates/layerfs-storage/src/encoding/pool/branch.rs, core/crates/layerfs-storage/tests/page_program_deltas.rs, core/crates/layerfs-storage/tests/physical_formats.rs. Production LOC: 141111 -> 141118 (delta +7) Scope: combined first-party production source, tools/production_loc.py --json on the parent and staged checkout-index trees. The +7 is rustfmt splitting long lines in the three content files and one storage file, not new code. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

yifanxuaaa and others added 27 commits November 6, 2026 12:46

Merge pg-r2-mount f773d9b: adapt daemon tests to BindIntent (#64)

91198bf

Tests only. Production LOC: 139680 -> 139680 (delta 0), first-parent comparison, counted with tools/production_loc.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

yhcedpn assigned yifanxuaaa Nov 6, 2026

yhcedpn added the enhancement New feature or request label Nov 6, 2026

yhcedpn added this to the LayerFS v2.0.0 milestone Nov 6, 2026

yifanxuaaa and others added 2 commits November 6, 2026 23:30

yifanxuaaa merged commit 9c21b68 into layerfs_v2_0_0 Nov 6, 2026
2 of 4 checks passed

Navigation Menu

Repository navigation

PostgreSQL round 2: fewer connections and round trips in create, mount, Commit and reads (layerfs-project #61) - #597

PostgreSQL round 2: fewer connections and round trips in create, mount, Commit and reads (layerfs-project #61)#597
yifanxuaaa merged 29 commits into
layerfs_v2_0_0from
pg-r2

yifanxuaaa commented Nov 6, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

yifanxuaaa commented Nov 6, 2026

Changes (each merged and gated on its own)

Evidence

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants