Repository navigation
FRE-14b — read a 0.1.x rows-mode tree into a store, and record that it was read - #96
Merged
Merged
Conversation
14a closed the door on a 0.1.x data directory: the open now refuses a root whose series sit at `<subject>/<aspect>/measurements.db` instead of creating the three control-plane databases it was missing over the top of the operator's measurements and reporting an empty store. The other half of that box is a leg that reads the tree, because a refusal that is also the end of the road leaves the measurements stranded in a format nothing current can open. `weftdb::rows_mode` is that leg: `discover` reports the series a tree holds and lists, with reasons, everything it will not carry — the `metadata.db` catalog, the databases 0.1.x kept beside the rows (`ingestion.db`, `pipeline.db`, `correlations.db`, `patterns.db`, `processed_*.db`, `unprocessed_*.db`) and the `dictionaries/` directories. It opens each series database and checks its shape before any store is opened, so a directory that is not what 0.1.x wrote fails there rather than half-way through a write. An aspect's name comes from its directory, and its value column has to be `value`. 0.1.x's own server created `measurements (id TEXT NOT NULL, dataset_id TEXT NOT NULL, timestamp INTEGER NOT NULL, value TEXT NOT NULL)` (`v0.1.0:weftdb/src/types/aspect.rs:475`, repeated at `:1195`) and writes nothing into that file that names the series; the fixture's page images print the same DDL. An earlier draft of this reader also took "whichever column is named after the aspect", on a story about a 2322 migration renaming the column — there is no such migration in the tag, and the story was mine rather than the source's. It was also worse than unattested: an aspect named `id`, `dataset_id` or `timestamp` had been read out of the column holding its identifiers — the release writes `tx_id.as_uuid()` into `id` and the dataset's UUID into `dataset_id` (`v0.1.0:weftdb/src/types/database/inputs.rs:136-142`) — and every row of that import would have been an identifier passing for a measurement, with nothing afterwards able to say so. `pick_value_column` requires the pair the DDL created and refuses the rest, naming the columns it found. `a_series_database_that_is_not_the_shape_0_1_x_wrote_is_refused` pins the refusal, and `an_aspect_named_after_an_identifier_column_is_still_read_from_value` pins the side that is legal: an aspect actually called `dataset_id` is a name 0.1.x allowed, named by its directory, read from `value` and from nowhere else. The timestamp unit is declared, never inferred, and the fixture is why that is not a theoretical rule. rows-mode stamped `timestamp_millis()` into the `INTEGER` (`v0.1.0:weftdb/src/types/database/inputs.rs:142`, `:187`) whatever the aspect's recorded `resolution` claimed — that column is the sampling grid, not a unit — while 0.1.x's batch path kept seconds, which is why the *other* tree in the same fixture, from the same release and the same `rows.csv`, is second-scale and migrates to a store that declares `TimeUnit::Seconds`. Same grid, two trees, two scales, a factor of a thousand apart: a reader that guessed from the size of the numbers had a plausible wrong answer in both directions, and the wrong answer looks like a date. `TIMESTAMP_UNIT`/`TIMESTAMP_TIME_UNIT` say millis with their citation, the fixture test asserts the import declares millis while the migrated store declares seconds. The warning could not go in the fixture's own README: every file under `compat/fixtures/` is hash-pinned in `MANIFEST.sha256`, and a fixture set is append-only — the release's bytes are what the compatibility gates compare, prose included. So the sentence sits in `ROADMAP.md`'s FRE-14 line, where the lane an operator follows actually passes. The trap closes from the store's side too: `an_aspect_declared_at_another_timestamp_scale_refuses_the_reading` takes an aspect the operator declared by hand at `Seconds` that holds no rows yet, and refuses the reading with both schemas printed rather than filing its rows a thousand-fold away from the instants somebody sized their queries around. Sealing does not use `rows.chunks`: `chunks_keeping_instants_together` extends a chunk until a group of rows sharing one instant ends, so a segment can come out a few rows longer than the caller asked for. That is the store's hazard rather than a legacy one. 0.1.x allowed several rows at one instant — `ImportedAspect::duplicate_timestamps` counts them — and the store treats that group differently depending where it lands: rows sharing an instant inside one segment survive maintenance, while rows sharing an instant across two segments whose windows intersect are folded by `reconcile_segment` under `merge_newer_wins`, which keeps only the more recently sealed one (`weftdb/src/types/segment_store.rs:3666-3672`). A fixed-size chunk that cut a group in two would hand it to a compaction pass this reading does not control, days after an import that had already looked clean. A second reading of the same tree refuses (`AlreadyImported`) rather than skipping, and the `ImportRecord` is what makes that answer available: one `store_meta` row under `rows_mode_import:<database>:<subject>` in the store's own index, holding the tree's path, the counts per aspect, the encoding each aspect was declared under, `duplicate_timestamps`, and what stayed behind. It is read back after the store is closed and reopened, so the idempotency is the store's rather than a process-local memory. Refuse-or-duplicate is the only honest pair of options once a reading has happened: 0.1.x's row ids are UUID transaction ids that this leg does not carry, and the store keeps no record of which rows came from a migration, so a migrated row and a row the store sealed itself are indistinguishable. The tree is read, not written, but it is not opened read-only either, and the reason is worth keeping: a WAL-mode database — the shipped fixture has a `measurements.db-wal` beside every series — can need WAL recovery at open, which a read-only connection cannot perform, so read-only is a failure mode rather than a guarantee. What holds the line is tests: `reading_the_tree_leaves_every_file_it_found_as_it_found_it` compares the tree's files before and after, and the fixture test compares the copy against the release's own bytes file for file. The boundaries the slice does not close are in the module doc: one aspect is held in memory whole because the encoding decision (`recommend_encoding`, tolerance zero) and `seal_declared` both take whole columns; and a reading that stops partway leaves the aspects that already crossed in the store with no record, so the next attempt refuses on `AspectNotEmpty` instead of finishing the job — pinned by a test, and 14c's to handle. 14c is the route an operator runs: the CLI/server path, the numbered migrations that register `metadata.db`'s schema, and a reading of a tree too big for RAM.
physics515
force-pushed
the
fre14/rows-mode-import
branch
from
October 22, 2026 02:37
0a11f3f to
ece7597
Compare
Base automatically changed from
fre14/rows-mode-refusal
to
xname/declare-validation
October 22, 2026 03:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FRE-14b — give the 0.1.x rows-mode tree a reader
Second part of FRE-14, stacked on the 14a branch (
fre14/rows-mode-refusal, PR #95). 14a closed thedoor: a 0.1.x data directory is refused at
openinstead of being opened as an empty store over thetop of the operator's data. This is the leg that reads it, so the refusal does not have to be the last
word. 14c (the route an operator runs) is the remaining part of the box.
What lands
weftdb::rows_mode(new module,weftdb/src/types/rows_mode.rs), re-exported asweftdb::rows_mode:discover(tree) -> LegacyTree— reports the series at<subject>/<aspect>/measurements.db: rowcount, span, how many rows share an instant; and
NotCarried { path, reason }for everything thereading leaves in the tree — the
metadata.dbcatalog, the databases 0.1.x kept beside the rows(
pipeline.db,correlations.db,patterns.db,processed_*.db,unprocessed_*.db) and thedictionaries/directories. It opens each series database and checks its shape first, so a treethat is not what 0.1.x wrote fails before a store is opened, naming the columns it did find.
import_into(store, tree, subject, rows_per_segment) -> ImportRecord— reads one subject into astore already scoped to that
(database, subject); declares each aspect under the narrowestencoding that holds the tree's values exactly (
recommend_encoding, tolerance zero); seals inchunks; writes the record into the store's own index under
rows_mode_import:<database>:<subject>(record_key).read_import_record(store)— reads it back after a reopen, so the idempotency is the store's, nota process-local memory.
RowsModeError, insideanyhow::Error,downcast_refrecovers it):NoCatalog,NoSeries,RowShape,ValueNotDecimal,ScopeMismatch,NoSubject,AspectNotEmpty,SchemaConflict,AlreadyImported. Every one names the file or the aspect it stopped on.Decisions worth reviewing
An aspect's name comes from its directory, and the value column must be
value. 0.1.x's DDL(
v0.1.0:weftdb/src/types/aspect.rs:475-480) createsmeasurements (id TEXT, dataset_id TEXT, timestamp INTEGER, value TEXT)and writes nothing that names the series into it. An earlier revisionof this branch accepted "whichever column is named after the aspect", on a story about a 2322 migration
renaming the column — there is no such migration in the tag, and the story was mine rather than the
source's. It was also a hazard: an aspect named
id,dataset_idortimestampwould have had itsidentifiers read out as its measurements, with nothing afterwards able to say so.
pick_value_columnnow requires the pair that DDL created and refuses the rest;
a_series_database_that_is_not_the_shape_0_1_x_wrote_is_refusedpins that, andan_aspect_named_after_an_identifier_column_is_still_read_from_valuepins the other side: an aspectactually called
dataset_idis a legal series whose name comes from its directory and whose valuescome from
value, never from the column that shares its name.The timestamp unit is declared, never derived.
TIMESTAMP_UNIT/TIMESTAMP_TIME_UNITsay millisbecause 0.1.x stamped
timestamp_millis()into theINTEGER(
v0.1.0:weftdb/src/types/database/inputs.rs:142,:187) whatever the aspect's recordedresolutionclaimed — that column is the sampling grid. The other tree in the same fixture, from the same release
and the same
rows.csv, is second-scale, because 0.1.x's batch path kept seconds while its rows-modepath kept millis. Same fixture, two trees, two units, factor of a thousand: an import that infers from
the size of the numbers has a way to be wrong that looks plausible on both sides. The fixture test
asserts the rows-mode import declares
Millisand the migrated store declaresSeconds; the constantdoc, the module doc and
compat/fixtures/v0.1.0/README.mdall say so. From the store's side the sametrap is closed by
an_aspect_declared_at_another_timestamp_scale_refuses_the_reading: an aspect theoperator declared by hand at
Seconds, holding no rows yet, refuses the reading with both schemasprinted instead of filing its rows a thousand-fold off.
Chunks never split rows that share an instant. This one is a store hazard rather than a legacy one,
and it is why
rows.chunks()is not used:reconcile_segmentfolds two segments whose windowsintersect under
merge_newer_wins, keeping only the newer row at an instant both hold(
weftdb/src/types/segment_store.rs:3666-3672). A group of same-instant rows cut across two segmentswould therefore be trimmed by a compaction this reading does not control, after the import had already
looked clean.
chunks_keeping_instants_togetherextends a chunk pastrows_per_segmentuntil the groupends, so a segment may come out a few rows longer than asked — the cheaper side.
Reading the same tree twice refuses (
AlreadyImported) rather than skipping. After a reading, amigrated row and a row the store sealed itself are indistinguishable — 0.1.x's ids are UUID transaction
ids that are not carried, and the store has no record of which rows came from where. Refuse or
duplicate is the only honest pair of options. The
ImportRecordis what makes the refusal possible atall, and it records the counts, the encodings chosen,
duplicate_timestamps, and what stayed behind.The tree keeps its bytes. Each series database is opened bare, deliberately not through the control
plane's
connect/enable_mvcc_full(which switches journal mode, i.e. writes). A read-only open is notused because a WAL-mode database — the shipped fixture has a
measurements.db-walbeside every series —can need WAL recovery at open, which a read-only connection cannot perform. The guarantee is held by
test instead: every file found before the reading is byte-identical after it, and the fixture test
compares the copy against the release's own bytes file for file.
Test evidence
the_v010_rows_mode_tree_reads_into_a_store_of_its_own_database(weftdb/tests/compat_v010.rs) — thetree the v0.1.0 binary wrote: 371 rows (237
price, 54temp, 80flow— the non-null subset ofrows.csv; rows-mode has no nulls, the release's generator skipped them) carried into a store of thattree's own database and read back as the same exact decimals at instants a thousand times the grid's
numbers;
not_carriedcovers every.dbthe tree holds; a second reading refuses and writes nothing;the tree is byte-identical afterwards.
weftdb/src/types/rows_mode.rstests: 13 — shape refusals (missing catalog, wrong value column,not-a-tree), the identifier-name case above, the round trip of rows and of the record through a
reopen,
ScopeMismatch,AspectNotEmpty,SchemaConflict,AlreadyImported,ValueNotDecimal(including what a partway reading leaves behind), byte-identity of the tree after a reading, and
a_chunk_never_splits_rows_that_share_an_instant.compat/fixtures/v0.1.0/are unchanged by this branch — the tree isread, never rewritten, and
git statusshows only itsREADME.mdgaining the warning about thetwo timestamp scales.
cargo +stable test -p weftdb --lib,--test compat_v010,cargo +nightly fmt --all -- --check,cargo +stable clippy -p weftdb --all-targets,cargo +stable doc -p weftdb --no-deps: .Not in this branch
metadata.db'sschema, and a tree too big to hold one aspect in RAM — the encoding decision here reads the whole
column into memory.
metadata.dbcatalog is not read at all in this slice: its schema is what 14c registers, andguessing it now would be a schema this branch has no reason to pin.
in the module doc as a known limitation.