Repository navigation
Gap stage 1: append-only blocks for inode values and binding hints, schema v2 (#91) - #595
Merged
Merged
Conversation
… schema v2) Gap stage 1 of #84. inode_value and binding_hint kept one database row per entry; PostgreSQL pays ~28 B of tuple header and line pointer plus a B-tree entry for each. Schema version 2 keeps them in immutable, append-only blocks of at most 4,000 bytes (one per 4 KiB SQLite page, two per 8 KiB PostgreSQL page, bytea STORAGE PLAIN): - inode_value_block(first_ordinal, last_ordinal, body): consecutive ordinals, implicit inside the block; per entry the value's byte 0 as tag, LEB128 reference count, content root, and either the attribute root or a per-block mode-table index, zig-zag mtime delta and nanoseconds; any other 73 bytes are stored whole. Decoding rebuilds the exact canonical bytes. - binding_hint_block(grp, seq, body): one write's hints of the 64 serials of group child >> 6, child/parent deltas and shared name prefixes; readers merge a group's blocks and drop repeats. - Every body ends with a CRC32C checked on every decode. Field packing only, no compression (codec frozen). Blocks are encoded before the statement that holds the lane; the publication stays one statement on PostgreSQL. An insert whose ordinals a stored block holds inserts nothing and fails the publication (a stored value is never replaced). Reads stay one statement: per run of ordinals, the last block starting at or before it and the blocks to its end. Schema version 2: PostgreSQL SCHEMA_VERSION 2; SQLite user_version 36-38 (42-44 with acquisition tables). Version 1 Stores (SQLite 16-18/22-24, PostgreSQL 1) are refused at open. tools/store_blocks.py adds the diagnostic decoder (attribution from a closed SQLite copy or exported PostgreSQL rows) and an offline SQLite v1 -> v2 migration of a closed copy; the merge-forms-root.sqlite fixture was converted with it. Golden block vectors are asserted by the engine tests and the tool's tests. Tests: new binding_hint_blocks.rs; sqlite_inode_values.rs reworked for blocks (exact round trip of every value shape, block bounds, checksum, golden vector); versions updated in sqlite_group_rows*.rs, postgresql_open_check.rs (a version 1 schema is refused), sqlite_ordinal_lookahead.rs and old_format_open.rs. No test deleted. Production LOC: 139405 -> 139919 (delta +514) (tools/production_loc.py combined scope: core/engine/connect/api src + runtime SQL; parent tree vs staged tree) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A file below the Store's small_file_threshold_bytes is one whole-file object: building it again yields the same object id, which wave admission finds stored and does not encode again. Its file_digest row cost more than it saved (D10: 15,524 of 15,535 rows were small). record_file_digests now drops rows below the threshold and file_roots answers none for a smaller length, with no statement when a window asks only small lengths. The threshold comes from store_policy at open (both backends). Tests: daemon file_digests asserts the floor (a small file is offered, not kept; relink reuses only the chunked files); postgresql_identical_ids checks the floor on SQLite and PostgreSQL. Production LOC: 139919 -> 139957 (delta +38) (tools/production_loc.py combined, parent vs commit tree, rebased on 8b701f4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The memo filter mixed only bytes 0..8, 9..17 and 41..49 of a value, so two values that differ in mtime low bytes or nanoseconds shared one entry: the first hid the second, and each later copy of the second in the same Save took a new ordinal (D10: 953 of 16,801 repeated values). The filter now mixes all 73 bytes; candidates are still checked byte for byte. Test: stored_value_reads one_save_stores_values_that_differ_only_in_mtime_once_each (fails on the old filter). Production LOC: 139957 -> 139960 (delta +3) (tools/production_loc.py combined, parent vs commit tree, rebased on 8b701f4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…x index (#90) object_location becomes pack_id bigint, canonical_length integer, role, group_number and record_number smallint, then object_id bytea PLAIN, with no unique key on the id: a B-tree object_location_prefix on the id's first eight bytes (signed big-endian bigint) finds candidates and the full id is compared in the heap. A location is inserted only when its id is absent (INSERT ... WHERE NOT EXISTS) inside the one publication statement that already holds the store_lane row, so exactly one location per id is chosen under the lane; the lost-race list and the core's Collision read-back are unchanged. The content_signature foreign key needed a unique key: the same statement now refuses a signature that names neither a location of its publication nor a stored one. Locate and digest_roots join through the prefix (one statement each, as before). Bounded bytea/text columns of the small tables (and the acquisition tables) are STORAGE PLAIN, so those tables get no TOAST relation. Measured and not taken: explicit 8,100 B PLAIN pack chunk rows (D10 packs 44.0 MB against 43.2 MB with EXTERNAL TOAST: one chunk per page plus a partly filled tail page per pack); dropping the composite UNIQUEs (they back the same-stack composite foreign keys); file_digest narrowing (the #91 digest floor already removes almost all its rows). Tests: postgresql_narrow_locations (prefix-sharing ids, lost race, repeated id refused, signature check, TOAST set, prefix index used); postgresql_open_check refuses a widened location column on the full check. Production LOC: 139960 -> 139999 (delta +39) (tools/production_loc.py combined, parent vs staged tree) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on named (#91) A SQLite Store of user_version 16 to 18 or 22 to 24 (one row per inode value and binding hint) was refused at open as Malformed. It is now Refused with a status that says what it is and names the optional offline conversion (tools/store_blocks.py migrate on a closed copy). Other unknown versions stay Malformed. The README states that conversion is optional and that a PostgreSQL schema version 1 Store has no conversion tool. Test: sqlite_inode_values refuses versions 16-18 and 22-24 with that status, writable and read-only. Production LOC: 139999 -> 140008 (delta +9) (tools/production_loc.py combined, parent vs staged tree) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…index (#90) With no unique key on object_id, the planner may answer a large batch (thousands of ids against a table of tens of thousands of rows) with a hash join or hash anti join that scans the whole table and computes the prefix expression for every row. DIAGNOSTIC on a copy of D10's 51,742 locations, 4,000 stored ids: the insert-if-absent took 22.7 ms that way, against 4.9 ms as one correlated index probe per id and 8.5 ms for the schema version 1 ON CONFLICT on the 32-byte primary key. OFFSET 0 in the location subqueries (publication insert and signature check, locate, digest_roots) keeps each one a probe per id through object_location_prefix: 3 buffers per id. Production LOC: 140008 -> 140008 (delta +0) (tools/production_loc.py combined, parent vs staged tree) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gap stage 2: narrow PostgreSQL object_location under an 8-byte-prefix index (layerfs-project #90), stacked on #91; lands with #595 (schema v2). Validated together in the #84 final integration 7691ac2 (layerfs-benchmark#11) and with round 2 in the quick verification (layerfs-benchmark#10). Backup checks B1-B4 PASS (schema v1 backups refused at open naming tools/store_blocks.py migrate). Production LOC: 139960 -> 140008 (delta +48) Scope: combined first-party production source, tools/production_loc.py --json on git archive trees, first parent 017507c (opt/91-gap-blocks). Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yifanxuaaa
added a commit
that referenced
this pull request
Nov 6, 2026
Brings layerfs-project #84's storage changes (#92 directory/extent page deltas, pack framing v26; #91 inode-value blocks and #90 narrow PostgreSQL rows, schema v2) under round 2. Conflict only in core/crates/layerfs-storage/tests/stored_value_reads.rs: both sides appended a test; both kept. Tree 87f25ed, as verified in the quick verification (layerfs-benchmark#10). Production LOC: 140178 -> 141111 (delta +933) Scope: combined first-party production source, tools/production_loc.py --json on git archive / checkout-index trees, first parent 048d604. Second parent e20c6a2 (140338). Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yifanxuaaa
added a commit
that referenced
this pull request
Nov 6, 2026
PostgreSQL round 2 (layerfs-project #61: #94-#98, #64, #66): fewer connections and round trips in create, mount, Commit and reads, now on top of #84's storage changes (#594, #595/#596). Quick verification on this exact tree dd191ba (layerfs-benchmark#10): backup B1-B4 51/51; fs-bench-quick q84-2 + q84-2x 36/36 PASS with PostgreSQL counts equal to the round-2 control except C16 Commit 251 -> 224 statements, no aggregate or per-row timing flag; fs-storage-quick Q1/Q5/Q6 x SQLite/PostgreSQL 6/6 PASS, every metric 'same' as the #84 final. Clippy -D warnings and cargo fmt --check clean. Production LOC: 140338 -> 141118 (delta +780) Scope: combined first-party production source, tools/production_loc.py --json on git archive trees, first parent e20c6a2. +773 round 2 (#94-#98, #64) and +7 from a rustfmt-only commit on #92 files. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
inode_valueandbinding_hintnow store many entries per row: append-only ≤4,000 B blocks, stored PLAIN with a CRC32C, the same block format on SQLite and PostgreSQL. This is schema v2. On fs-storage-quick, Q1 is −5.5% (SQLite) / −13.7% (PostgreSQL) and Q6 is −3.1% / −8.4%. D10 (diagnostic) goes from 52.9 to 49.8 MB on SQLite and from 70.4 to 61.0 MB on PostgreSQL.Gap stage 1: fixes Ephemeral-AI-Lab/layerfs-project#91 (parent Ephemeral-AI-Lab/layerfs-project#84). Base:
layerfs_v2_0_0@8b701f42f0. It must merge together with the stacked #90 PR (schema v2 on PostgreSQL). Merging #91 alone would force #90 to a v3.Commits
b973426ee0Blocks:tools/store_blocks.pyfor the optional offline SQLite migration and DIAGNOSTIC attribution.54486095d6File digests:file_digestis kept only for chunked files. Small files are whole-file objects found by id at wave admission. +38.017507ccf2Save value memo: now keyed on the whole 73-byte value. This fixes a pre-existing bug: values differing only in mtime low bytes or nanoseconds got duplicate ordinals in one Save (D10: 953 extra rows). It wasted space but never returned wrong values. +3.Codec frozen: field packing only. There is no zstd, LZ, dictionary or TOAST compression on these rows, and bodies are
STORAGE PLAIN.Measured and not done:
Multi-writer contract (#84 gap design R1–R9)
postgresql/objects/publish.rs:46–79). Hints are append-only.inode_valuescall (metadata/reads.rs:100–122); onejson_eachstatement on SQLite.publish.rs:117); still one statement.INSERT … WHERE NOT EXISTSonly, never UPDATE. Hint seq+1 under the lane.Tests
222 test executables pass on the stacked head (persistence incl. PostgreSQL, storage, daemon, store-service, workspace, project). clippy
-D warnings, fmt, the boundary guard andtools/test_store_blocks.pyall pass.fs-storage-quick (qgap-c2b, against the Phase 2 baseline
02133596ae; 12/12 PASS, no regressions)On D10 PostgreSQL,
inode_valuegoes 9.39 → 2.84 MB andbinding_hint3.26 → 0.43 MB.Speed gate (fs-bench-quick, Docker, interleaved control/candidate/candidate/control, quiet host)
catof the committed 64 MiB file): 102.9/103.3 → 116.0/120.1 ms, ratio 1.12.Migration
Refused, namingtools/store_blocks.py migrate. The migration is optional and offline: it writes a checked v2 copy from a closed v1 copy.🤖 Generated with Claude Code