Skip to content

FRE-14b — read a 0.1.x rows-mode tree into a store, and record that it was read - #96

Merged
physics515 merged 1 commit into
xname/declare-validationfrom
fre14/rows-mode-import
Oct 22, 2026
Merged

physics515 merged 1 commit into
xname/declare-validationfrom
fre14/rows-mode-import

Conversation

@physics515

Copy link
Copy Markdown
Collaborator

FRE-14b — give the 0.1.x rows-mode tree a reader

Second part of FRE-14, stacked on the 14a branch (fre14/rows-mode-refusal, PR #95). 14a closed the
door: a 0.1.x data directory is refused at open instead of being opened as an empty store over the
top of the operator's data. This is the leg that reads it, so the refusal does not have to be the last
word. 14c (the route an operator runs) is the remaining part of the box.

What lands

  • weftdb::rows_mode (new module, weftdb/src/types/rows_mode.rs), re-exported as weftdb::rows_mode:
    • discover(tree) -> LegacyTree — reports the series at <subject>/<aspect>/measurements.db: row
      count, span, how many rows share an instant; and NotCarried { path, reason } for everything the
      reading leaves in the tree — the metadata.db catalog, the databases 0.1.x kept beside the rows
      (pipeline.db, correlations.db, patterns.db, processed_*.db, unprocessed_*.db) and the
      dictionaries/ directories. It opens each series database and checks its shape first, so a tree
      that is not what 0.1.x wrote fails before a store is opened, naming the columns it did find.
    • import_into(store, tree, subject, rows_per_segment) -> ImportRecord — reads one subject into a
      store already scoped to that (database, subject); declares each aspect under the narrowest
      encoding that holds the tree's values exactly (recommend_encoding, tolerance zero); seals in
      chunks; writes the record into the store's own index under
      rows_mode_import:<database>:<subject> (record_key).
    • read_import_record(store) — reads it back after a reopen, so the idempotency is the store's, not
      a process-local memory.
  • Typed refusals (RowsModeError, inside anyhow::Error, downcast_ref recovers it):
    NoCatalog, NoSeries, RowShape, ValueNotDecimal, ScopeMismatch, NoSubject, AspectNotEmpty,
    SchemaConflict, AlreadyImported. Every one names the file or the aspect it stopped on.

Decisions worth reviewing

An aspect's name comes from its directory, and the value column must be value. 0.1.x's DDL
(v0.1.0:weftdb/src/types/aspect.rs:475-480) creates measurements (id TEXT, dataset_id TEXT, timestamp INTEGER, value TEXT) and writes nothing that names the series into it. An earlier revision
of this branch accepted "whichever column is named after the aspect", on a story about a 2322 migration
renaming the column — there is no such migration in the tag, and the story was mine rather than the
source's. It was also a hazard: an aspect named id, dataset_id or timestamp would have had its
identifiers read out as its measurements, with nothing afterwards able to say so. pick_value_column
now requires the pair that DDL created and refuses the rest;
a_series_database_that_is_not_the_shape_0_1_x_wrote_is_refused pins that, and
an_aspect_named_after_an_identifier_column_is_still_read_from_value pins the other side: an aspect
actually called dataset_id is a legal series whose name comes from its directory and whose values
come from value, never from the column that shares its name.

The timestamp unit is declared, never derived. TIMESTAMP_UNIT / TIMESTAMP_TIME_UNIT say millis
because 0.1.x stamped timestamp_millis() into the INTEGER
(v0.1.0:weftdb/src/types/database/inputs.rs:142, :187) whatever the aspect's recorded resolution
claimed — that column is the sampling grid. The other tree in the same fixture, from the same release
and the same rows.csv, is second-scale, because 0.1.x's batch path kept seconds while its rows-mode
path kept millis. Same fixture, two trees, two units, factor of a thousand: an import that infers from
the size of the numbers has a way to be wrong that looks plausible on both sides. The fixture test
asserts the rows-mode import declares Millis and the migrated store declares Seconds; the constant
doc, the module doc and compat/fixtures/v0.1.0/README.md all say so. From the store's side the same
trap is closed by an_aspect_declared_at_another_timestamp_scale_refuses_the_reading: an aspect the
operator declared by hand at Seconds, holding no rows yet, refuses the reading with both schemas
printed instead of filing its rows a thousand-fold off.

Chunks never split rows that share an instant. This one is a store hazard rather than a legacy one,
and it is why rows.chunks() is not used: reconcile_segment folds two segments whose windows
intersect under merge_newer_wins, keeping only the newer row at an instant both hold
(weftdb/src/types/segment_store.rs:3666-3672). A group of same-instant rows cut across two segments
would therefore be trimmed by a compaction this reading does not control, after the import had already
looked clean. chunks_keeping_instants_together extends a chunk past rows_per_segment until the group
ends, so a segment may come out a few rows longer than asked — the cheaper side.

Reading the same tree twice refuses (AlreadyImported) rather than skipping. After a reading, a
migrated row and a row the store sealed itself are indistinguishable — 0.1.x's ids are UUID transaction
ids that are not carried, and the store has no record of which rows came from where. Refuse or
duplicate is the only honest pair of options. The ImportRecord is what makes the refusal possible at
all, and it records the counts, the encodings chosen, duplicate_timestamps, and what stayed behind.

The tree keeps its bytes. Each series database is opened bare, deliberately not through the control
plane's connect/enable_mvcc_full (which switches journal mode, i.e. writes). A read-only open is not
used because a WAL-mode database — the shipped fixture has a measurements.db-wal beside every series —
can need WAL recovery at open, which a read-only connection cannot perform. The guarantee is held by
test instead: every file found before the reading is byte-identical after it, and the fixture test
compares the copy against the release's own bytes file for file.

Test evidence

  • the_v010_rows_mode_tree_reads_into_a_store_of_its_own_database (weftdb/tests/compat_v010.rs) — the
    tree the v0.1.0 binary wrote: 371 rows (237 price, 54 temp, 80 flow — the non-null subset of
    rows.csv; rows-mode has no nulls, the release's generator skipped them) carried into a store of that
    tree's own database and read back as the same exact decimals at instants a thousand times the grid's
    numbers; not_carried covers every .db the tree holds; a second reading refuses and writes nothing;
    the tree is byte-identical afterwards.
  • weftdb/src/types/rows_mode.rs tests: 13 — shape refusals (missing catalog, wrong value column,
    not-a-tree), the identifier-name case above, the round trip of rows and of the record through a
    reopen, ScopeMismatch, AspectNotEmpty, SchemaConflict, AlreadyImported, ValueNotDecimal
    (including what a partway reading leaves behind), byte-identity of the tree after a reading, and
    a_chunk_never_splits_rows_that_share_an_instant.
  • Fixture gate: the bytes under compat/fixtures/v0.1.0/ are unchanged by this branch — the tree is
    read, never rewritten, and git status shows only its README.md gaining the warning about the
    two timestamp scales.
  • cargo +stable test -p weftdb --lib, --test compat_v010, cargo +nightly fmt --all -- --check,
    cargo +stable clippy -p weftdb --all-targets, cargo +stable doc -p weftdb --no-deps: .

Not in this branch

  • 14c: the route an operator runs (CLI/server), the numbered migrations that register metadata.db's
    schema, and a tree too big to hold one aspect in RAM — the encoding decision here reads the whole
    column into memory.
  • The metadata.db catalog is not read at all in this slice: its schema is what 14c registers, and
    guessing it now would be a schema this branch has no reason to pin.
  • A tree whose directory name disagrees with its own catalog row is trusted to its directory — stated
    in the module doc as a known limitation.

14a closed the door on a 0.1.x data directory: the open now refuses a root whose
series sit at `<subject>/<aspect>/measurements.db` instead of creating the three
control-plane databases it was missing over the top of the operator's
measurements and reporting an empty store. The other half of that box is a leg
that reads the tree, because a refusal that is also the end of the road leaves
the measurements stranded in a format nothing current can open. `weftdb::rows_mode`
is that leg: `discover` reports the series a tree holds and lists, with reasons,
everything it will not carry — the `metadata.db` catalog, the databases 0.1.x
kept beside the rows (`ingestion.db`, `pipeline.db`, `correlations.db`,
`patterns.db`, `processed_*.db`, `unprocessed_*.db`) and the `dictionaries/`
directories. It opens each series database and checks its shape before any store
is opened, so a directory that is not what 0.1.x wrote fails there rather than
half-way through a write.

An aspect's name comes from its directory, and its value column has to be
`value`. 0.1.x's own server created `measurements (id TEXT NOT NULL,
dataset_id TEXT NOT NULL, timestamp INTEGER NOT NULL, value TEXT NOT NULL)`
(`v0.1.0:weftdb/src/types/aspect.rs:475`, repeated at `:1195`) and writes nothing
into that file that names the series; the fixture's page images print the same
DDL. An earlier draft of this reader also took "whichever column is named after
the aspect", on a story about a 2322 migration renaming the column — there is no
such migration in the tag, and the story was mine rather than the source's. It
was also worse than unattested: an aspect named `id`, `dataset_id` or `timestamp`
had been read out of the column holding its identifiers — the release writes
`tx_id.as_uuid()` into `id` and the dataset's UUID into `dataset_id`
(`v0.1.0:weftdb/src/types/database/inputs.rs:136-142`) — and every row of that
import would have been an identifier passing for a measurement, with nothing
afterwards able to say so. `pick_value_column` requires the pair the DDL created
and refuses the rest, naming the columns it found.
`a_series_database_that_is_not_the_shape_0_1_x_wrote_is_refused` pins the refusal,
and `an_aspect_named_after_an_identifier_column_is_still_read_from_value` pins the
side that is legal: an aspect actually called `dataset_id` is a name 0.1.x allowed,
named by its directory, read from `value` and from nowhere else.

The timestamp unit is declared, never inferred, and the fixture is why that is
not a theoretical rule. rows-mode stamped `timestamp_millis()` into the `INTEGER`
(`v0.1.0:weftdb/src/types/database/inputs.rs:142`, `:187`) whatever the aspect's
recorded `resolution` claimed — that column is the sampling grid, not a unit —
while 0.1.x's batch path kept seconds, which is why the *other* tree in the same
fixture, from the same release and the same `rows.csv`, is second-scale and
migrates to a store that declares `TimeUnit::Seconds`. Same grid, two trees, two
scales, a factor of a thousand apart: a reader that guessed from the size of the
numbers had a plausible wrong answer in both directions, and the wrong answer
looks like a date. `TIMESTAMP_UNIT`/`TIMESTAMP_TIME_UNIT` say millis with their
citation, the fixture test asserts the import declares millis while the migrated
store declares seconds. The warning could not go in the fixture's own README:
every file under `compat/fixtures/` is hash-pinned in `MANIFEST.sha256`, and a
fixture set is append-only — the release's bytes are what the compatibility gates
compare, prose included. So the sentence sits in `ROADMAP.md`'s FRE-14 line,
where the lane an operator follows actually passes. The trap closes from the store's side too:
`an_aspect_declared_at_another_timestamp_scale_refuses_the_reading` takes an aspect
the operator declared by hand at `Seconds` that holds no rows yet, and refuses the
reading with both schemas printed rather than filing its rows a thousand-fold away
from the instants somebody sized their queries around.

Sealing does not use `rows.chunks`: `chunks_keeping_instants_together` extends a
chunk until a group of rows sharing one instant ends, so a segment can come out a
few rows longer than the caller asked for. That is the store's hazard rather than
a legacy one. 0.1.x allowed several rows at one instant — `ImportedAspect::duplicate_timestamps`
counts them — and the store treats that group differently depending where it
lands: rows sharing an instant inside one segment survive maintenance, while rows
sharing an instant across two segments whose windows intersect are folded by
`reconcile_segment` under `merge_newer_wins`, which keeps only the more recently
sealed one (`weftdb/src/types/segment_store.rs:3666-3672`). A fixed-size chunk
that cut a group in two would hand it to a compaction pass this reading does not
control, days after an import that had already looked clean.

A second reading of the same tree refuses (`AlreadyImported`) rather than
skipping, and the `ImportRecord` is what makes that answer available: one
`store_meta` row under `rows_mode_import:<database>:<subject>` in the store's own
index, holding the tree's path, the counts per aspect, the encoding each aspect
was declared under, `duplicate_timestamps`, and what stayed behind. It is read
back after the store is closed and reopened, so the idempotency is the store's
rather than a process-local memory. Refuse-or-duplicate is the only honest pair of
options once a reading has happened: 0.1.x's row ids are UUID transaction ids that
this leg does not carry, and the store keeps no record of which rows came from a
migration, so a migrated row and a row the store sealed itself are indistinguishable.

The tree is read, not written, but it is not opened read-only either, and the
reason is worth keeping: a WAL-mode database — the shipped fixture has a
`measurements.db-wal` beside every series — can need WAL recovery at open, which a
read-only connection cannot perform, so read-only is a failure mode rather than a
guarantee. What holds the line is tests: `reading_the_tree_leaves_every_file_it_found_as_it_found_it`
compares the tree's files before and after, and the fixture test compares the copy
against the release's own bytes file for file. The boundaries the slice does not
close are in the module doc: one aspect is held in memory whole because the
encoding decision (`recommend_encoding`, tolerance zero) and `seal_declared` both
take whole columns; and a reading that stops partway leaves the aspects that
already crossed in the store with no record, so the next attempt refuses on
`AspectNotEmpty` instead of finishing the job — pinned by a test, and 14c's to
handle. 14c is the route an operator runs: the CLI/server path, the numbered
migrations that register `metadata.db`'s schema, and a reading of a tree too big
for RAM.
@physics515
physics515 force-pushed the fre14/rows-mode-import branch from 0a11f3f to ece7597 Compare October 22, 2026 02:37
Base automatically changed from fre14/rows-mode-refusal to xname/declare-validation October 22, 2026 03:22
@physics515
physics515 merged commit 8ef41cc into xname/declare-validation Oct 22, 2026
18 of 21 checks passed
@physics515
physics515 deleted the fre14/rows-mode-import branch October 22, 2026 03:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant