Repository navigation
Land the W2 stack: X-NAME, ROB-2, PAC-3, X-FRAME, FRE-14a/14b - #97
Merged
Merged
Conversation
1.0 plan X-NAME (absorbs FRE-7a, SEC-0). `encode` (#72) and `validate` plus the declare check (#66) are already on main; the gap this closes is the rule that can only be applied to a name that is not in use yet. `aspect_name::validate_new` is `validate` plus: no ASCII uppercase letter. `price` and `PRICE` are one file name on a case-insensitive or case-folding filesystem (NTFS, APFS by default, ext4 with `casefold`), and the legacy `<aspect>-<id>.weftseg` scheme writes the aspect name into that name literally, so two aspects would share one frame file and the second one's frames would be read as the first's. The module docs already called that collision out as something the encoded write-once names remove; enforcing it at creation removes it for the names written today. `POST /api/v1/storage/aspects` is the only caller: declaring is where a name is born. Reads, seals, maintenance and the store-wide sweep keep plain `validate`, so an aspect that a older version put in the catalog with a capital in it is never stranded. The characters only Windows forbids (`< > : " | ? *`) are deliberately NOT added to the creation rules on Unix. `job:rate5m` is a name in use, aspect_name.rs blesses it on Unix, and `encode` escapes `:` one layer down, which is where D-S10 makes the frame name; the plan moves the containment check into D-S10 for the same reason. Unicode NFC/NFD aliasing is left as documented: `encode` escapes every non-ASCII byte, so new frame names cannot collide that way, and rejecting it at declare would need a normalisation table this crate does not carry.
…ked where the name is born
ROB-2 in wave 2, per release plan C12 and C22: an engine-origin panic while a write is in flight ends the process instead of being answered. `weftdb::exec::control_plane_write` was already the write-scope flag the plan asks for, and it already wraps the whole control-plane write path — every `IndexTxn` operation runs inside it, which weftdb's own `every_op_runs_inside_the_control_plane_write_scope` asserts — so the missing half was the hook, and C12 puts the hook in the binary: `weftdb` holds the engine types but never exits a process. `panic_hook::install` chains the hook it replaces, and when `in_control_plane_write()` says the panicking task was inside the region it names the panic and calls `std::process::abort()`. Startup calls it immediately after `init_tracing`, because `set_hook` hands back the hook it replaced: installing ours first would leave the tracing subscriber's hook calling ours and print every panic twice. Abort rather than the poison path's exit 70: that status is chosen by a known condition with a known operator action, and FRE-6a's point is that this failure must not arrive as an HTTP response, so the exit should carry a signal a supervisor sees. The hook also deliberately does not write the poison marker. Most of the region reads before it writes — `txn.plans()` prepares inside the scope, `SegmentIndex::open` runs inside it — so a panic there does not imply a torn commit, and the marker persists in the store directory until an operator deletes a file, which would turn a benign panic into an outage. The abort buys the restart, and the restart already runs the open-time verification that can judge the store. Nothing is caught: a panic outside the region is reported and left unwinding, so wave 6's ROB-3 catch layer stays the only catch layer and read-path panics keep answering 500. The legacy metadata lane (`weftdb/src/types/aspect.rs`) is untouched; the plan gives that to ROB-5. `tests/control_plane_panic.rs` checks both halves in a child process, the way `poison_exit.rs` does: inside the region, through `control_plane_write` with a yield between entry and panic, the child dies by SIGABRT with the panic and the reason on stderr; outside it, in a spawned task, the child exits 0 — the chained hook still printed the panic, the abort line is absent, and the process kept running.
… process
The box's wording names the mechanism ("a turso-origin panic aborts") but not the boundary it
hangs on, so the note records that the scope was already in the library (`weftdb::exec`'s
control-plane write flag, with weftdb's own test proving every index-transaction operation
runs inside it) and that #92 supplied only the hook C12 puts in the binary — plus the two
decisions a reviewer should be able to disagree with: abort rather than the poison path's
exit 70, and no poison marker for a panic that may have happened during the reading half of
the region.
The arrow marks the split the plan keeps: ROB-5 wraps the legacy metadata lane
(`weftdb/src/types/aspect.rs`), which this change leaves alone.
…PU stack PAC-3. turso's `mimalloc` feature installs a `#[global_allocator]` in turso's own crate root, so the workspace's turso line made `weftdb` — a published library — decide the process allocator of every program that links it, including one with an allocator of its own or none at all. splimes' default `gpu` feature did the same one layer over: `gpu` -> `wgpu` -> the Vulkan, Metal and DX12 backends, in the dependency graph of five library crates. Our own binaries lose nothing. `weft-server` and `weft-tui` now name mimalloc 0.1.52 in their own roots, which is the edit the 1.0 plan's L-MAIN lane already schedules (docs/release/1.0-plan.md:126, "Later main.rs edits, one per wave: PAC-3 (W2, `#[global_allocator]`). `weft-server` and `weft-bench` ask splimes for `gpu` directly, and `weftdb`'s interpolation benches get it as a dev-dependency, so every GPU path this repository compiles and exercises is still compiled and exercised. splimes' `serde` stays on, because `weftdb::BatchedMeasurement` serialises a `splimes::Resolution`; turso's other default, `fts`, stays off as it already was, since it is full-text search and pulling it in grows the graph with tantivy, tantivy-columnar and zstd-sys. `weftdb`'s tracing-subscriber moves to dev-dependencies — only test code installs a subscriber. The non-dev dependency graphs, counted as distinct packages before and after: weft-line-protocol and weft-reduce 77 -> 25, weftdb 266 -> 227, weft-orchestration 267 -> 231. Two crate-doc lines in weftdb/src/lib.rs named splimes' GPU-only `calibrate` and are reworded, since the doc job denies broken intra-doc links.
…cator and GPU stack PR #93. Two decisions were being made for every program that links these crates: turso's mimalloc feature installs a #[global_allocator] in turso's own crate root, so asking for it on the published library weftdb chose the allocator of every embedder's process; and splimes' default features pull wgpu, which sat in the normal graph of four libraries that never call a GPU API. The allocator now belongs to the executables, which is where the plan puts this slice (docs/release/1.0-plan.md:126). GPU stays opt-in at the dependency level. Distinct packages in the normal graph: weft-line-protocol and weft-reduce 77 -> 25, weft-orchestration 267 -> 231, weftdb 266 -> 227. Recorded in the note: the two keeps (splimes serde for BatchedMeasurement, turso fts left off as it already was), and that cargo bench -p weftdb is no longer allocator-comparable across this commit.
ROB-2 — end the process when a control-plane write panics
PAC-3: libraries stop choosing their users' allocator and GPU stack
The plan's X-FRAME row (docs/release/1.0-plan.md:142) asks for a frame
registry, the v7 envelope, the read_frame*/describe_frame family and the
FrameFault classes, absorbing FRE-4, TAG-2 and ROB-8's fault_class naming.
weft_physical_type::frame_registry is now the only place that says which
frame version means what: 5 single-block, 6 paged, 7 envelope, and the rule
in the module doc that a number is never reused or renumbered once it has
shipped. The envelope is a header and a trailer around an existing frame,
WEFTSEG | u16 7 | u16 body_version | extension table | frame | crc32, and it
stores that frame verbatim, its own magic, version, body and CRC intact. That
is a deliberate constraint: segment_store.rs promises an unconfigured
deployment's bytes are byte-for-byte what previous versions wrote, and the
byte-pinning tests stay green because a v5 or v6 body is untouched. A legacy
reader pointed at an envelope still says UnsupportedVersion { found: 7 }
rather than misparsing it.
An extension carries a critical flag. Unknown and critical means refusal:
the body was written on the assumption that the extension would be applied,
so decoding without it serves data under a contract nobody honoured. Unknown
and optional means ignored. KNOWN_EXTENSIONS is empty on purpose - the
envelope ships with no extensions, and the numbers are a permanent namespace.
SegmentStore chose its decoder from the index row's format_version, so
anything that was not 6 - including the unwrap_or(0) in segment_index.rs -
sent a paged frame to the single-block reader. It now asks the file: the six
decode sites go through weftdb/src/types/frame_read.rs, which probes the
header, calls the matching read_frame_*, and returns failures as
weftdb::FrameFaultError in the anyhow chain under the unchanged
"decoding [paged ]segment <path>: ..." sentence. weft-server maps the class
by type: a frame ahead of this build is 503 incompatible_store, any other
frame fault is 500 frame_undecodable, the code X-ERR gave a wire sentence but
no producer.
No writer opts into 7 yet: seals still write 5 and 6. This slice is the
mechanism and the classes; the opt-in and the C5 quarantine behaviour belong
to D-S12 and the ingest lanes.
Tests: weft-physical-type 306 lib (14 new: envelope round-trip, a legacy
reader refusing an envelope, critical refusal, optional ignore, header/count
disagreement caught by CRC, describe_frame on a future version, fault
classification, the read family over a wrapped frame); weftdb frame_read 6
new; weft-server --test frame_fault 2 new, corrupting a real sealed frame
and asserting 500 frame_undecodable with details.fault "integrity" and 503
incompatible_store with "newer_format" after re-stamping the trailer CRC.
Name frame versions once, and read a frame from what the file says
A WeftDB 0.1.x data directory keeps one database per aspect: `metadata.db` at the top, then `<subject>/<aspect>/measurements.db` two levels down. That top `metadata.db` is one of `CONTROL_PLANE_FILES`, so `control_plane_present` said yes and the open went ahead: it created the three control-plane databases it was missing, made `segments/`, wrote a `STORE_FORMAT` marker beside the old data, ran the layout-1 migrations, and returned an empty store. Every measurement vanished without an error, and the marker then described a tree that was not there. The probe for the shape runs as step 4a of `open_on`: after the marker gate, before any database is opened, and only when the root has no marker — which is exactly where a 0.1.x root sits, since markers begin at layout 2. It reports the `<subject>/<aspect>` pairs it found in `StoreError::RowsModeDataDirectory`, which X-ERR already maps to 503 `incompatible_store` and CLI exit 65, so no new status code is involved. By then the open has taken `LOCK` and written `LOCK.holder`, and the variant's doc says those two files are all it touches: the test asserts the rest of the directory is unchanged, that no marker is added, and that a second open refuses the same way. A refusal that annotated the directory would be read by the next version as understanding it, which is the lie this closes. The other half of the box is the upgrade test, and it compares against the release rather than against this build: the fixture carries v0.1.0's own `aspect_stats.txt` and `stats.json`, so the migrated store has to count 3/725/14 bytes-of-rows for `price` over `0..4070900000`, 2/72/7 for `temp`, 1/80/0 for `flow`, and 877 rows / 21 nulls / 3220 bytes store-wide. Then the directory is moved with `rename` and reopened: no migration runs again, the same integers come back, and the six frames under `segments/` are still six. `ordinal()` is the whole of the registry's ordering, and its `unwrap_or(0)` is silent: an id that does not start with a number is ordinal 0, so it is neither pending nor after and simply never runs. Nothing pinned the format, so the registry test now requires four digits, an underscore, and a name. The earlier note for this slice said `migrations_lock.rs` holds a `MAX_MIGRATION_ID` that should be tied to the registry; that file and constant do not exist in this tree, and the note is corrected rather than the phantom wired up. Reading the rows themselves is FRE-14b, and it is a slice of its own: 0.1.x wrote eight databases per aspect, and `correlations` and `patterns` have no frame equivalent at all.
14a closed the door on a 0.1.x data directory: the open now refuses a root whose series sit at `<subject>/<aspect>/measurements.db` instead of creating the three control-plane databases it was missing over the top of the operator's measurements and reporting an empty store. The other half of that box is a leg that reads the tree, because a refusal that is also the end of the road leaves the measurements stranded in a format nothing current can open. `weftdb::rows_mode` is that leg: `discover` reports the series a tree holds and lists, with reasons, everything it will not carry — the `metadata.db` catalog, the databases 0.1.x kept beside the rows (`ingestion.db`, `pipeline.db`, `correlations.db`, `patterns.db`, `processed_*.db`, `unprocessed_*.db`) and the `dictionaries/` directories. It opens each series database and checks its shape before any store is opened, so a directory that is not what 0.1.x wrote fails there rather than half-way through a write. An aspect's name comes from its directory, and its value column has to be `value`. 0.1.x's own server created `measurements (id TEXT NOT NULL, dataset_id TEXT NOT NULL, timestamp INTEGER NOT NULL, value TEXT NOT NULL)` (`v0.1.0:weftdb/src/types/aspect.rs:475`, repeated at `:1195`) and writes nothing into that file that names the series; the fixture's page images print the same DDL. An earlier draft of this reader also took "whichever column is named after the aspect", on a story about a 2322 migration renaming the column — there is no such migration in the tag, and the story was mine rather than the source's. It was also worse than unattested: an aspect named `id`, `dataset_id` or `timestamp` had been read out of the column holding its identifiers — the release writes `tx_id.as_uuid()` into `id` and the dataset's UUID into `dataset_id` (`v0.1.0:weftdb/src/types/database/inputs.rs:136-142`) — and every row of that import would have been an identifier passing for a measurement, with nothing afterwards able to say so. `pick_value_column` requires the pair the DDL created and refuses the rest, naming the columns it found. `a_series_database_that_is_not_the_shape_0_1_x_wrote_is_refused` pins the refusal, and `an_aspect_named_after_an_identifier_column_is_still_read_from_value` pins the side that is legal: an aspect actually called `dataset_id` is a name 0.1.x allowed, named by its directory, read from `value` and from nowhere else. The timestamp unit is declared, never inferred, and the fixture is why that is not a theoretical rule. rows-mode stamped `timestamp_millis()` into the `INTEGER` (`v0.1.0:weftdb/src/types/database/inputs.rs:142`, `:187`) whatever the aspect's recorded `resolution` claimed — that column is the sampling grid, not a unit — while 0.1.x's batch path kept seconds, which is why the *other* tree in the same fixture, from the same release and the same `rows.csv`, is second-scale and migrates to a store that declares `TimeUnit::Seconds`. Same grid, two trees, two scales, a factor of a thousand apart: a reader that guessed from the size of the numbers had a plausible wrong answer in both directions, and the wrong answer looks like a date. `TIMESTAMP_UNIT`/`TIMESTAMP_TIME_UNIT` say millis with their citation, the fixture test asserts the import declares millis while the migrated store declares seconds. The warning could not go in the fixture's own README: every file under `compat/fixtures/` is hash-pinned in `MANIFEST.sha256`, and a fixture set is append-only — the release's bytes are what the compatibility gates compare, prose included. So the sentence sits in `ROADMAP.md`'s FRE-14 line, where the lane an operator follows actually passes. The trap closes from the store's side too: `an_aspect_declared_at_another_timestamp_scale_refuses_the_reading` takes an aspect the operator declared by hand at `Seconds` that holds no rows yet, and refuses the reading with both schemas printed rather than filing its rows a thousand-fold away from the instants somebody sized their queries around. Sealing does not use `rows.chunks`: `chunks_keeping_instants_together` extends a chunk until a group of rows sharing one instant ends, so a segment can come out a few rows longer than the caller asked for. That is the store's hazard rather than a legacy one. 0.1.x allowed several rows at one instant — `ImportedAspect::duplicate_timestamps` counts them — and the store treats that group differently depending where it lands: rows sharing an instant inside one segment survive maintenance, while rows sharing an instant across two segments whose windows intersect are folded by `reconcile_segment` under `merge_newer_wins`, which keeps only the more recently sealed one (`weftdb/src/types/segment_store.rs:3666-3672`). A fixed-size chunk that cut a group in two would hand it to a compaction pass this reading does not control, days after an import that had already looked clean. A second reading of the same tree refuses (`AlreadyImported`) rather than skipping, and the `ImportRecord` is what makes that answer available: one `store_meta` row under `rows_mode_import:<database>:<subject>` in the store's own index, holding the tree's path, the counts per aspect, the encoding each aspect was declared under, `duplicate_timestamps`, and what stayed behind. It is read back after the store is closed and reopened, so the idempotency is the store's rather than a process-local memory. Refuse-or-duplicate is the only honest pair of options once a reading has happened: 0.1.x's row ids are UUID transaction ids that this leg does not carry, and the store keeps no record of which rows came from a migration, so a migrated row and a row the store sealed itself are indistinguishable. The tree is read, not written, but it is not opened read-only either, and the reason is worth keeping: a WAL-mode database — the shipped fixture has a `measurements.db-wal` beside every series — can need WAL recovery at open, which a read-only connection cannot perform, so read-only is a failure mode rather than a guarantee. What holds the line is tests: `reading_the_tree_leaves_every_file_it_found_as_it_found_it` compares the tree's files before and after, and the fixture test compares the copy against the release's own bytes file for file. The boundaries the slice does not close are in the module doc: one aspect is held in memory whole because the encoding decision (`recommend_encoding`, tolerance zero) and `seal_declared` both take whole columns; and a reading that stops partway leaves the aspects that already crossed in the store with no record, so the next attempt refuses on `AspectNotEmpty` instead of finishing the job — pinned by a test, and 14c's to handle. 14c is the route an operator runs: the CLI/server path, the numbered migrations that register `metadata.db`'s schema, and a reading of a tree too big for RAM.
The registry, its two anti-drift tests and the startup report are all in this PR, so the W1 line is checked here rather than in a follow-up: 51 VarSpec entries, tiers 7 Stable / 17 Tuning / 2 Experimental / 25 Test, and a coverage walk over the five crates that ship binaries. Signed-off-by: Justin Icenhour <creative515.justin@gmail.com>
#91 closed after its head branch became #92-#96's merge target, so five slices of W2 sat on a branch off main with their boxes ticked. This merge puts the code, the ticks and the wave status back in agreement: X-NAME, ROB-2, PAC-3, X-FRAME, and FRE-14's refusal and rows-mode import, plus main's roadmap edits, the cfg(unix) on the symlink probe, and PAC-5a's tick, whose registry arrived in #88 without its line. Signed-off-by: Justin Icenhour <creative515.justin@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this lands
The W2 stack, onto
main.#91 (X-NAME → main) was closed after its head branch
xname/declare-validationhad become themerge target for #92–#96. So five slices of W2 sat on a branch off
mainwith their ROADMAPboxes ticked: X-NAME, ROB-2, PAC-3, X-FRAME, and FRE-14's two first slices (the 0.1.x
data-directory refusal, and the rows-mode import).
mainhad nothing past #90.This branch is that stack merged with
main, plus three things that were only ever local:#[cfg(unix)]ona_rows_mode_probe_does_not_follow_a_symlink—std::os::unix::fs::symlinkis the whole point of the test, and Windows' equivalent needs developer mode or elevation.
b5d0667), whose registry merged in PAC-5a: one registry for every variable the workspace reads #88 without its line being checked.Merge shape
origin/maininto the stack tip, then the cherry-pick, then the roadmap note.ROADMAP.mdwasthe only conflict —
maincarried the owner's W0 ticks, the branch carried the W2 slices. Bothsides are kept.
Verification
Every slice ran green as its own PR (#92–#96). The added content here is docs plus one
cfg.CI on this branch is the gate that matters: it is the first time these five slices compile
together with
main.