Skip to content

Gap stage 1: append-only blocks for inode values and binding hints, schema v2 (#91) - #595

Merged
yifanxuaaa merged 7 commits into
layerfs_v2_0_0from
opt/91-gap-blocks
Nov 6, 2026
Merged

yifanxuaaa merged 7 commits into
layerfs_v2_0_0from
opt/91-gap-blocks

Conversation

@yifanxuaaa

Copy link
Copy Markdown
Contributor

inode_value and binding_hint now store many entries per row: append-only ≤4,000 B blocks, stored PLAIN with a CRC32C, the same block format on SQLite and PostgreSQL. This is schema v2. On fs-storage-quick, Q1 is −5.5% (SQLite) / −13.7% (PostgreSQL) and Q6 is −3.1% / −8.4%. D10 (diagnostic) goes from 52.9 to 49.8 MB on SQLite and from 70.4 to 61.0 MB on PostgreSQL.

Gap stage 1: fixes Ephemeral-AI-Lab/layerfs-project#91 (parent Ephemeral-AI-Lab/layerfs-project#84). Base: layerfs_v2_0_0 @ 8b701f42f0. It must merge together with the stacked #90 PR (schema v2 on PostgreSQL). Merging #91 alone would force #90 to a v3.

Commits

  • b973426ee0 Blocks:
    • inode-value blocks with implicit ordinals, a per-block mode table, varint/zig-zag fields, and an exact 73-byte canonical round trip, so no object id changes;
    • binding-hint blocks;
    • schema v2 on both backends (v1 is refused);
    • tools/store_blocks.py for the optional offline SQLite migration and DIAGNOSTIC attribution.
    • LOC 139405 → 139919 (+514).
  • 54486095d6 File digests: file_digest is kept only for chunked files. Small files are whole-file objects found by id at wave admission. +38.
  • 017507ccf2 Save value memo: now keyed on the whole 73-byte value. This fixes a pre-existing bug: values differing only in mtime low bytes or nanoseconds got duplicate ordinals in one Save (D10: 953 extra rows). It wasted space but never returned wrong values. +3.

Codec frozen: field packing only. There is no zstd, LZ, dictionary or TOAST compression on these rows, and bodies are STORAGE PLAIN.

Measured and not done:

  • Cross-Save ordinal reuse: needs a value index costing about what it saves.
  • Small-pack coalescing: at most 0.8%, and needs a pack-grammar change.

Multi-writer contract (#84 gap design R1–R9)

Rule How it holds
R1 Blocks are written inside the publication statement, so they commit with their packs (postgresql/objects/publish.rs:46–79). Hints are append-only.
R2 One statement per inode_values call (metadata/reads.rs:100–122); one json_each statement on SQLite.
R3 Location path unchanged.
R4 Lost-race and Collision unchanged. An overlapping block returns NULL, which becomes Integrity.
R5 Blocks are encoded before the lane (publish.rs:117); still one statement.
R6 INSERT … WHERE NOT EXISTS only, never UPDATE. Hint seq+1 under the lane.
R7 No compaction.
R8 Ordinals and groups never enter hashed ids.
R9 Same statement count; hint writes use fewer statements.

Tests

222 test executables pass on the stacked head (persistence incl. PostgreSQL, storage, daemon, store-service, workspace, project). clippy -D warnings, fmt, the boundary guard and tools/test_store_blocks.py all pass.

fs-storage-quick (qgap-c2b, against the Phase 2 baseline 02133596ae; 12/12 PASS, no regressions)

Case SQLite net PostgreSQL net
Q1 4,796,416 → 4,534,272 (−5.5%) 6,119,424 → 5,283,840 (−13.7%)
Q3 same 942,080 → 753,664 (−20.0%)
Q4 same 1,343,488 → 1,155,072 (−14.0%)
Q6 24,399,872 → 23,650,304 (−3.1%) 28,999,680 → 26,550,272 (−8.4%)

On D10 PostgreSQL, inode_value goes 9.39 → 2.84 MB and binding_hint 3.26 → 0.43 MB.

Speed gate (fs-bench-quick, Docker, interleaved control/candidate/candidate/control, quiet host)

Migration

  • SQLite v1 Stores: refused at open with Refused, naming tools/store_blocks.py migrate. The migration is optional and offline: it writes a checked v2 copy from a closed v1 copy.
  • PostgreSQL v1 schemas: refused, with no conversion tool. Ship it for new Stores.

🤖 Generated with Claude Code

yifanxuaaa and others added 6 commits November 6, 2026 18:11
… schema v2)

Gap stage 1 of #84. inode_value and binding_hint kept one database row per
entry; PostgreSQL pays ~28 B of tuple header and line pointer plus a B-tree
entry for each. Schema version 2 keeps them in immutable, append-only blocks
of at most 4,000 bytes (one per 4 KiB SQLite page, two per 8 KiB PostgreSQL
page, bytea STORAGE PLAIN):

- inode_value_block(first_ordinal, last_ordinal, body): consecutive ordinals,
  implicit inside the block; per entry the value's byte 0 as tag, LEB128
  reference count, content root, and either the attribute root or a
  per-block mode-table index, zig-zag mtime delta and nanoseconds; any other
  73 bytes are stored whole. Decoding rebuilds the exact canonical bytes.
- binding_hint_block(grp, seq, body): one write's hints of the 64 serials of
  group child >> 6, child/parent deltas and shared name prefixes; readers
  merge a group's blocks and drop repeats.
- Every body ends with a CRC32C checked on every decode.

Field packing only, no compression (codec frozen). Blocks are encoded before
the statement that holds the lane; the publication stays one statement on
PostgreSQL. An insert whose ordinals a stored block holds inserts nothing
and fails the publication (a stored value is never replaced). Reads stay one
statement: per run of ordinals, the last block starting at or before it and
the blocks to its end.

Schema version 2: PostgreSQL SCHEMA_VERSION 2; SQLite user_version 36-38
(42-44 with acquisition tables). Version 1 Stores (SQLite 16-18/22-24,
PostgreSQL 1) are refused at open. tools/store_blocks.py adds the diagnostic
decoder (attribution from a closed SQLite copy or exported PostgreSQL rows)
and an offline SQLite v1 -> v2 migration of a closed copy; the
merge-forms-root.sqlite fixture was converted with it. Golden block vectors
are asserted by the engine tests and the tool's tests.

Tests: new binding_hint_blocks.rs; sqlite_inode_values.rs reworked for
blocks (exact round trip of every value shape, block bounds, checksum,
golden vector); versions updated in sqlite_group_rows*.rs,
postgresql_open_check.rs (a version 1 schema is refused),
sqlite_ordinal_lookahead.rs and old_format_open.rs. No test deleted.

Production LOC: 139405 -> 139919 (delta +514) (tools/production_loc.py combined
scope: core/engine/connect/api src + runtime SQL; parent tree vs staged tree)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A file below the Store's small_file_threshold_bytes is one whole-file
object: building it again yields the same object id, which wave admission
finds stored and does not encode again. Its file_digest row cost more than
it saved (D10: 15,524 of 15,535 rows were small). record_file_digests now
drops rows below the threshold and file_roots answers none for a smaller
length, with no statement when a window asks only small lengths. The
threshold comes from store_policy at open (both backends).

Tests: daemon file_digests asserts the floor (a small file is offered,
not kept; relink reuses only the chunked files); postgresql_identical_ids
checks the floor on SQLite and PostgreSQL.

Production LOC: 139919 -> 139957 (delta +38) (tools/production_loc.py
combined, parent vs commit tree, rebased on 8b701f4)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The memo filter mixed only bytes 0..8, 9..17 and 41..49 of a value, so two
values that differ in mtime low bytes or nanoseconds shared one entry: the
first hid the second, and each later copy of the second in the same Save
took a new ordinal (D10: 953 of 16,801 repeated values). The filter now
mixes all 73 bytes; candidates are still checked byte for byte.

Test: stored_value_reads one_save_stores_values_that_differ_only_in_mtime_once_each
(fails on the old filter).

Production LOC: 139957 -> 139960 (delta +3) (tools/production_loc.py
combined, parent vs commit tree, rebased on 8b701f4)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…x index (#90)

object_location becomes pack_id bigint, canonical_length integer, role,
group_number and record_number smallint, then object_id bytea PLAIN, with
no unique key on the id: a B-tree object_location_prefix on the id's first
eight bytes (signed big-endian bigint) finds candidates and the full id is
compared in the heap. A location is inserted only when its id is absent
(INSERT ... WHERE NOT EXISTS) inside the one publication statement that
already holds the store_lane row, so exactly one location per id is chosen
under the lane; the lost-race list and the core's Collision read-back are
unchanged. The content_signature foreign key needed a unique key: the same
statement now refuses a signature that names neither a location of its
publication nor a stored one. Locate and digest_roots join through the
prefix (one statement each, as before).

Bounded bytea/text columns of the small tables (and the acquisition
tables) are STORAGE PLAIN, so those tables get no TOAST relation.

Measured and not taken: explicit 8,100 B PLAIN pack chunk rows (D10 packs
44.0 MB against 43.2 MB with EXTERNAL TOAST: one chunk per page plus a
partly filled tail page per pack); dropping the composite UNIQUEs (they
back the same-stack composite foreign keys); file_digest narrowing (the
#91 digest floor already removes almost all its rows).

Tests: postgresql_narrow_locations (prefix-sharing ids, lost race,
repeated id refused, signature check, TOAST set, prefix index used);
postgresql_open_check refuses a widened location column on the full check.

Production LOC: 139960 -> 139999 (delta +39) (tools/production_loc.py
combined, parent vs staged tree)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on named (#91)

A SQLite Store of user_version 16 to 18 or 22 to 24 (one row per inode
value and binding hint) was refused at open as Malformed. It is now
Refused with a status that says what it is and names the optional offline
conversion (tools/store_blocks.py migrate on a closed copy). Other unknown
versions stay Malformed. The README states that conversion is optional and
that a PostgreSQL schema version 1 Store has no conversion tool.

Test: sqlite_inode_values refuses versions 16-18 and 22-24 with that
status, writable and read-only.

Production LOC: 139999 -> 140008 (delta +9) (tools/production_loc.py
combined, parent vs staged tree)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…index (#90)

With no unique key on object_id, the planner may answer a large batch
(thousands of ids against a table of tens of thousands of rows) with a
hash join or hash anti join that scans the whole table and computes the
prefix expression for every row. DIAGNOSTIC on a copy of D10's 51,742
locations, 4,000 stored ids: the insert-if-absent took 22.7 ms that way,
against 4.9 ms as one correlated index probe per id and 8.5 ms for the
schema version 1 ON CONFLICT on the 32-byte primary key. OFFSET 0 in the
location subqueries (publication insert and signature check, locate,
digest_roots) keeps each one a probe per id through object_location_prefix:
3 buffers per id.

Production LOC: 140008 -> 140008 (delta +0) (tools/production_loc.py
combined, parent vs staged tree)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yhcedpn yhcedpn added the enhancement New feature or request label Nov 6, 2026
@yhcedpn yhcedpn added this to the LayerFS v2.0.0 milestone Nov 6, 2026
Gap stage 2: narrow PostgreSQL object_location under an 8-byte-prefix index (layerfs-project #90), stacked on #91; lands with #595 (schema v2).

Validated together in the #84 final integration 7691ac2 (layerfs-benchmark#11) and with round 2 in the quick verification (layerfs-benchmark#10). Backup checks B1-B4 PASS (schema v1 backups refused at open naming tools/store_blocks.py migrate).

Production LOC: 139960 -> 140008 (delta +48)
Scope: combined first-party production source, tools/production_loc.py --json on git archive trees, first parent 017507c (opt/91-gap-blocks). Reference absent (retired R9c; last counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yifanxuaaa
yifanxuaaa merged commit e20c6a2 into layerfs_v2_0_0 Nov 6, 2026
2 of 5 checks passed
yifanxuaaa added a commit that referenced this pull request Nov 6, 2026
Brings layerfs-project #84's storage changes (#92 directory/extent page
deltas, pack framing v26; #91 inode-value blocks and #90 narrow PostgreSQL
rows, schema v2) under round 2. Conflict only in
core/crates/layerfs-storage/tests/stored_value_reads.rs: both sides
appended a test; both kept. Tree 87f25ed, as verified in the quick
verification (layerfs-benchmark#10).

Production LOC: 140178 -> 141111 (delta +933)
Scope: combined first-party production source, tools/production_loc.py
--json on git archive / checkout-index trees, first parent 048d604.
Second parent e20c6a2 (140338). Reference absent (retired R9c; last
counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yifanxuaaa added a commit that referenced this pull request Nov 6, 2026
PostgreSQL round 2 (layerfs-project #61: #94-#98, #64, #66): fewer connections and round trips in create, mount, Commit and reads, now on top of #84's storage changes (#594, #595/#596).

Quick verification on this exact tree dd191ba (layerfs-benchmark#10): backup B1-B4 51/51; fs-bench-quick q84-2 + q84-2x 36/36 PASS with PostgreSQL counts equal to the round-2 control except C16 Commit 251 -> 224 statements, no aggregate or per-row timing flag; fs-storage-quick Q1/Q5/Q6 x SQLite/PostgreSQL 6/6 PASS, every metric 'same' as the #84 final. Clippy -D warnings and cargo fmt --check clean.

Production LOC: 140338 -> 141118 (delta +780)
Scope: combined first-party production source, tools/production_loc.py --json on git archive trees, first parent e20c6a2. +773 round 2 (#94-#98, #64) and +7 from a rustfmt-only commit on #92 files. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants