diff --git a/CHANGELOG.md b/CHANGELOG.md index f719ef20..2bdad7c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -273,9 +273,42 @@ only the control plane, not the segment frames. still part of the file name — `job:rate5m` is a name in use — and that cross-platform hazard is closed one layer down instead, where `aspect_name::encode` escapes `:` for the write-once frame names (D-S10). +- **Our executables say what allocator they run on** (1.0 plan PAC-3). `weft-server` and + `weft-tui` each declare `#[global_allocator] static ALLOCATOR: mimalloc::MiMalloc` in their own + `main.rs`, through a direct `mimalloc` dependency rather than one inherited from a library. + Same allocator, same version, same measured behaviour — the decision is just made in the place + that owns the process (see the dependency entry under Changed). ### Changed +- **Library crates stop deciding their users' allocator and GPU stack** (1.0 plan PAC-3). No + library in this workspace enables turso's `mimalloc` feature or splimes' default `gpu` feature + any more, because both are decisions a dependency makes for the whole process. turso 0.8.1 + installs a `#[global_allocator]` in its own crate root when `mimalloc` is on, so every program + linking `weftdb` — an embedder with its own allocator, a program that wants the system one — was + switched to mimalloc without being asked; `gpu` drags `wgpu`, and with it the Vulkan, Metal and + DX12 backends, into the graph of everything that depends on splimes. Counting the distinct + packages in each crate's non-dev dependency graph, before and after: `weft-line-protocol` and + `weft-reduce` 77 → 25, `weftdb` 266 → 227, `weft-orchestration` 267 → 231. Our own GPU users — + `weft-server` and `weft-bench`, which call `prewarm_gpu`, `calibrate` and `gpu_pool_stats` — + enable `gpu` themselves, as do `weftdb`'s interpolation benches through a dev-dependency, so + nothing this repository runs lost its GPU path. A library user who wants either asks for it: an + allocator by declaring one in their own binary, the GPU by enabling `splimes/gpu`. ⚠️ Rust code + that reached the GPU *through* `weftdb` — a `splimes::Backend::Gpu` while depending on `weftdb` + alone — names `splimes` itself now. One default stays on deliberately: splimes' `serde`, because + `weftdb::BatchedMeasurement` derives `Serialize`/`Deserialize` over a `splimes::Resolution` field. + turso's other default, `fts`, stays off, as it already was: it is turso's full-text search, and + turning it on grows the graph with tantivy, tantivy-columnar, zstd-sys and the rest of that tree, + which the first attempt at this dependency line showed in the lock diff. + One number to be aware of: `cargo bench -p weftdb` is criterion, not test code, so it used to + inherit mimalloc through turso and now runs on the system allocator — a `weftdb` bench figure + from before this release is not comparable with one taken after. `weft-bench` never linked + `weftdb` and never had mimalloc, so the Weft-Bench series, including the nightly regression + track, does not move. +- **`tracing-subscriber` is a dev-dependency of `weftdb`, not a dependency.** No code in the + library installs a subscriber outside `#[cfg(test)]`, so every program linking `weftdb` was + building registry, fmt and env-filter code it never called. + - **Statuses split where the §4 table names a different condition** (1.0 plan X-ERR). A batch that will not seal under an aspect's declared schema no longer answers `400` whatever the reason: a length mismatch between the two columns, a zero `rows_per_page`, and a backwards diff --git a/Cargo.lock b/Cargo.lock index 44e5cb6b..5efe79b4 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4647,7 +4647,6 @@ version = "0.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7de8cbeace6385bf3fcb346d4aace8d9ba0de5cffe5e5a0ed210beebd671c208" dependencies = [ - "mimalloc", "thiserror 2.0.21", "tracing", "tracing-subscriber", @@ -5222,6 +5221,7 @@ dependencies = [ "axum", "bigdecimal", "chrono", + "mimalloc", "opentelemetry", "opentelemetry-otlp", "opentelemetry_sdk", @@ -5252,6 +5252,7 @@ dependencies = [ "crossterm", "dirs", "futures", + "mimalloc", "num-traits", "ratatui", "splimes", diff --git a/Cargo.toml b/Cargo.toml index 29dced62..65853f77 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -45,7 +45,18 @@ rand = "0.10" rayon = "1" serde = { version = "1", features = ["derive"] } serde_json = "1" -turso = { version = "0.8", default-features = false, features = ["mimalloc"] } +# The allocator our own executables run on, named by each of them rather than inherited: see +# `weft-server/src/main.rs` and `weft-tui/src/main.rs`. PAC-3. +mimalloc = "0.1" +# `mimalloc` is deliberately not enabled here: turso's `mimalloc` feature installs a +# process-wide `#[global_allocator]` from its own crate root, so asking for it through a +# library replaces the allocator of every program that merely links that library — including one +# that sets its own, or wants the system default. Nothing in this workspace enables it any more; +# an executable that wants mimalloc depends on it and declares it in its own root, and a library +# user who wants it does the same. turso's other default, `fts`, was already off in this workspace +# and stays off — it is the full-text-search feature, and turning it on here would pull tantivy, +# tantivy-columnar, zstd-sys and the rest of that tree into every build. PAC-3. +turso = { version = "0.8", default-features = false } tempfile = "3" thiserror = "2" tokio = { version = "1", features = ["full"] } @@ -60,7 +71,13 @@ crossterm = "0.29" sysinfo = "0.39" weftdb = { path = "./weftdb", version = "0.1.0" } # Developed in its own repository (github.com/basic-automation/splimes), on its own roadmap. -splimes = "1" +# `default-features = false` because splimes' defaults turn on its `gpu` backend, which pulls +# `wgpu` — Vulkan, Metal, DX12 and a shader compiler — into every crate that links a library +# in this workspace, including the published `weftdb`. The GPU path is the executable's +# decision: `weft-server` and `weft-bench` enable `splimes/gpu`, and `weft-orchestration` +# exposes it as its own opt-in `gpu-codecs` feature. `serde` stays on, because +# `weftdb::BatchedMeasurement`'s derived serde needs splimes' own implementations. PAC-3. +splimes = { version = "1", default-features = false, features = ["serde"] } weft-line-protocol = { path = "./weft-line-protocol", version = "0.1.0" } weft-physical-type = { path = "./weft-physical-type", version = "0.1.0" } weft-reduce = { path = "./weft-reduce", version = "0.1.0" } diff --git a/ROADMAP.md b/ROADMAP.md index d022587a..a7e0b21d 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1158,7 +1158,7 @@ any write, never purge an unproven frame), with its unfinished work on `durabili - [x] **X-ERR** One error envelope and code table in `error.rs` (absorbs SEC-2, FRE-6a, ROB-1, TAG-0's envelope) — *#90: `weft-server/src/error.rs` (685 lines) is the one module that names a refusal. `Code`'s 42 variants are asserted by name **and** status against the plan's §4 table, and the nine synonyms that table drops (`invalid_request`, `unauthorized`, `store_unconfigured`, `maintenance_busy`, `limit_exceeded`, `panic`, `segment_invalid`, `not_found`, `too_large`) are asserted absent, so a later lane cannot quietly reintroduce one. The envelope is `{"error","code","request_id"}` plus optional `details`/`limit`/`committed` on **every** path: `StrictJson`/`StrictQuery` replace axum's extractors, so a wrong `Content-Type` is `unsupported_media_type` and a malformed body or query is `invalid_json`/`bad_request` instead of axum's empty 400, and `Router::fallback` answers `route_not_found` naming the method, the path, where the aspect list is and where `docs/INTEGRATION.md` is. Classification is by **type**: `classify_read`, `classify_ingest` and `classify_typed` downcast `weftdb::InvalidAspectName`, `weftdb::StoreError`, `SealError`, `ReduceError` and `SketchError`, and the four `SegmentStore` seal wrappers (`weftdb/src/types/segment_store.rs:2782/2798/2814/2826`) now wrap with `anyhow::Error::from(error).context(format!("seal failed: {shown}"))` — `anyhow`'s `Display` prints only the outermost context, so the sentence stays byte-identical while `downcast_ref::()` starts working; a reworded wrapper can no longer move a status. Two message rules survive, each commented with the slice that retires it: `no declared schema` (D-S13 types it as `UnknownAspect`, a store-lane edit) and `out-of-order timestamp` (`weft-arrow-store/src/lib.rs:233`'s untyped `bail!`, D-S10/D-S14). `ConvertError` is direction-dependent, so there are two classifiers and `weft-server/tests/error_contract.rs` asserts both halves of the same `Parquet(…)` — a client `400` ingesting, a server `500` reading. Statuses split where §4 names a different condition: `ToleranceExceeded` is `422 value_not_representable` with `details.max_abs_error`/`details.tolerance` (was a `400` carrying a `seal failed` sentence), `Encode(EncodeError::NotFinite)` is `400 not_finite` with `details.row`, a name the store cannot address is `400 invalid_aspect_name` with `details.aspect`/`details.reason`, a store this binary will not open is `503 incompatible_store` (exit 65 in the §4 C7 table), and a downsample `ReduceError` is 422/400 instead of 500. Two findings worth keeping: `weft_physical_type::EncodeError` names only `Overflow` and `NotFinite`, so a wildcard arm beside them was dead code and is gone — a new arm is meant to stop the build where someone picks the §4 code; and `ApiError::message()` returns the cause **by design** (it is the sentence for the log), so the sanitisation tests assert `wire_message()` is the code's fixed text — `internal` "the server could not complete the request", `incompatible_store` "the store on disk uses a format this server cannot open". `axum`'s `CatchPanicLayer` is dropped (FRE-6a: a panic is a process death with the exit code a supervisor sees, not a body), and `ConvertError`/`ReduceError`/`SketchError` gained `#[non_exhaustive]`.* - [x] → **X-NAME** `aspect_name::encode` and `validate_new` at HTTP declare (absorbs FRE-7a, SEC-0) — *partly on main: `validate` and the declare check (#66), `encode` (#72). #91 is the rest: `weftdb::aspect_name::validate_new` (`weftdb/src/types/aspect_name.rs`) runs `validate`'s chain and then refuses an ASCII uppercase letter with a new `AspectNameReason::UppercaseAsciiLetter` (the enum is `#[non_exhaustive]`, so the variant is not a breaking change). Order is asserted: the capital rule runs **after** `reason()`, so `../Price` is refused for escaping the directory, not for its capital — the more dangerous reason is never hidden. The hole is real because the legacy frame file is `-.weftseg`, which writes the name literally, and NTFS, APFS (by default) and ext4 with `casefold` fold ASCII case: two aspects differing only in case share a file, whichever seal lands second overwrites the other's bytes, and reads serve one series' values for the other's name. `encode` (#72) removes that collision for the frames D-S10 will write; this closes the declaration of a new collision. Applied at **creation only** — `POST /api/v1/storage/aspects` — so `SegmentStore::declare`, every read, seal, maintenance and store-wide sweep keep plain `validate` and a store that already holds `Price` stays readable, writable and compactable (the test asserts both halves: the refusal, then a raw catalog declare of `Price` followed by a 201 write). Two things are deliberately **not** refused: the characters only Windows forbids (`< > : " | ? *`), because `job:rate5m` is blessed on Unix by this module's own docs and is a shape in use, and the cross-platform hazard belongs one layer down where `encode` escapes `:` for write-once frame names (D-S10, the slice the plan names for it); and Unicode NFC/NFD aliases, which need `unicode-normalization` this crate does not carry, while `encode` escapes every non-ASCII byte anyway. A refusal keeps the #90 shape: `400 invalid_aspect_name` with `details.aspect` and `details.reason`.* - [x] → **ROB-2** Storage-engine panic fail-stop: a turso-origin panic aborts — *#92: the write scope was already here — `weftdb::exec::control_plane_write`, and every `IndexTxn` operation runs inside it, which weftdb's own `every_op_runs_inside_the_control_plane_write_scope` asserts — so C12's "hook and knob" was the half that was missing, and it lives in the binary: `weft_server::panic_hook::install()` chains the hook it replaces, and when `in_control_plane_write()` says the panicking task held the scope it names the panic and calls `std::process::abort()`: SIGABRT for a supervisor, not the poison path's exit 70, and never an HTTP response (FRE-6a). Startup calls it immediately after `init_tracing`, because `set_hook` hands back the hook it replaced. It deliberately does not write the poison marker: that region reads before it writes (`txn.plans()`, `SegmentIndex::open`), and the marker survives the panic until an operator deletes a file. Nothing is caught, so ROB-3 stays the only catch layer and read-path panics still answer 500; the legacy `weftdb/src/types/aspect.rs` lane is ROB-5's. Proved by `weft-server/tests/control_plane_panic.rs`, which re-executes the test binary the way `poison_exit.rs` does.* - - [ ] **PAC-3** Library crate hygiene: no global allocator or wgpu features imposed on library users + - [x] **PAC-3** Library crate hygiene: no global allocator or wgpu features imposed on library users — *#93: two decisions were being made for every program that links these crates. `turso` 0.8.1 puts a `#[global_allocator]` in its own crate root behind its `mimalloc` feature, so the workspace's `features = ["mimalloc"]` on a **published library** (`weftdb/Cargo.toml`) chose the allocator of every embedder's process; and `splimes`' default features pull `wgpu`, so `wgpu` sat in the normal graph of `weftdb`, `weft-line-protocol`, `weft-reduce` and `weft-orchestration` — 52 packages of Vulkan, Metal and DX12 code for two crates that never call a GPU API. The allocator moved to the place that owns the process: `weft-server/src/main.rs` and `weft-tui/src/main.rs` each declare `static ALLOCATOR: mimalloc::MiMalloc` through a direct `mimalloc` dependency, so the same allocator on the same version still runs everything we benchmark and ship, and `weftdb` now carries no allocator feature at all — the plan's own lane row names this slice at `main.rs` (`docs/release/1.0-plan.md:126`). GPU is opt-in the same way: the workspace asks splimes for `default-features = false, features = ["serde"]`, `weft-server` and `weft-bench` add `gpu` themselves, and `weftdb`'s interpolation benches take it as a dev-dependency. Distinct packages in the normal graph: `weft-line-protocol` and `weft-reduce` 77 → 25, `weft-orchestration` 267 → 231, `weftdb` 266 → 227; `cargo tree -p weftdb -i wgpu -e normal` and `-i mimalloc` now return nothing. Two keeps are deliberate: splimes' `serde` stays on because `weftdb::BatchedMeasurement` holds a `splimes::Resolution` and derives `Serialize`/`Deserialize`, and turso's other default, `fts`, stays off exactly as it already was — an attempt to re-add it pulled `tantivy`, `zstd`, `datasketches` and friends, +435 lines of `Cargo.lock`, for full-text search nothing in this workspace uses. `tracing-subscriber` became a dev-dependency of `weftdb` (only test code installs a subscriber). One consequence for our own numbers: `cargo bench -p weftdb` used to inherit mimalloc and now runs on the system allocator, so those nine criterion benches are not comparable across this commit; the nightly regression track (EVI-5) uses `weft-bench`, which never linked `weftdb` and never had mimalloc, so no baseline moves there.* - [ ] L-FMT: **X-FRAME** Frame registry, v7 envelope, `read_frame*`/`describe_frame`, `FrameFault` classes (absorbs FRE-4, TAG-2) - [ ] Legacy lane: **FRE-14** Legacy data-directory marker, numbered migrations and measurement upgrade test - [ ] **PAC-2** Portable Linux binaries on a glibc 2.28 baseline diff --git a/weft-bench/Cargo.toml b/weft-bench/Cargo.toml index 09835685..fa01ce8d 100644 --- a/weft-bench/Cargo.toml +++ b/weft-bench/Cargo.toml @@ -33,7 +33,10 @@ serde = { workspace = true, features = ["derive"] } serde_json = { workspace = true } rand = { workspace = true } rand_chacha = { workspace = true } -splimes = { workspace = true, features = ["tokio"] } +# The benchmark's engine metadata and its gpu-memory workload call splimes' GPU entry points +# (`prewarm_gpu`, `calibrate`, `gpu_pool_stats`), so it opts into the backend that the library +# crates no longer carry. PAC-3. +splimes = { workspace = true, features = ["tokio", "gpu"] } sysinfo = { workspace = true } tokio = { workspace = true, features = ["full"] } diff --git a/weft-server/Cargo.toml b/weft-server/Cargo.toml index 41ab159b..4f01dcc3 100644 --- a/weft-server/Cargo.toml +++ b/weft-server/Cargo.toml @@ -42,6 +42,10 @@ axum = { workspace = true } bigdecimal = { workspace = true } chrono = { workspace = true, features = ["serde"] } weftdb = { workspace = true } # control-plane segment store backing the storage query endpoints +# The process allocator this binary chooses, stated here rather than inherited from a library +# dependency (PAC-3); see `main.rs`. The same mimalloc that every benchmark of this server, and +# every release build of it, has measured. +mimalloc = { workspace = true } weft-arrow = { workspace = true } # reconstructed-series -> Arrow IPC / Parquet bytes for the interpolate endpoint (arrow types stay inside weft-arrow) weft-arrow-store = { workspace = true } # stored range -> Arrow IPC bytes (keeps the arrow-* tree off the lean core) weft-line-protocol = { workspace = true } @@ -49,7 +53,9 @@ weft-physical-type = { workspace = true } # AspectSchema / PhysicalType / TimeU weft-reduce = { workspace = true } # canonical downsample reduction (Aggregation / reduce / bucket_start) serde = { workspace = true, features = ["derive"] } serde_json = { workspace = true } -splimes = { workspace = true, features = ["tokio"] } +# The GPU backend is this binary's decision, not the libraries': `weft-server/src/gpu.rs` +# calibrates splimes' backends at startup. PAC-3. +splimes = { workspace = true, features = ["tokio", "gpu"] } tokio = { workspace = true, features = ["full"] } tracing = { workspace = true } # per-request engine spans on the compute path (Phase 3 instrumentation) tracing-subscriber = { workspace = true } # RUST_LOG-driven span/event subscriber (env-filter from the workspace dep) diff --git a/weft-server/src/main.rs b/weft-server/src/main.rs index 34a76808..d0dbbfb9 100644 --- a/weft-server/src/main.rs +++ b/weft-server/src/main.rs @@ -8,6 +8,14 @@ mod startup; +// The process allocator is this binary's choice, not a side effect of what a library it links +// happens to depend on (PAC-3). turso used to carry it: its `mimalloc` feature installs a +// `#[global_allocator]` in turso's own crate root, so any program linking `weftdb` got mimalloc +// whether it wanted it or not. `weftdb` no longer asks for it, and an executable that wants it — +// every one of ours, on every machine we benchmark and ship — says so here. +#[global_allocator] +static ALLOCATOR: mimalloc::MiMalloc = mimalloc::MiMalloc; + #[tokio::main] async fn main() -> anyhow::Result<()> { startup::run().await diff --git a/weft-tui/Cargo.toml b/weft-tui/Cargo.toml index 98e7329c..9216d283 100644 --- a/weft-tui/Cargo.toml +++ b/weft-tui/Cargo.toml @@ -18,6 +18,9 @@ publish = false ratatui = { workspace = true } crossterm = { workspace = true } weftdb = { workspace = true } +# The process allocator this binary chooses, stated here rather than inherited from a library +# dependency (PAC-3); see `main.rs`. +mimalloc = { workspace = true } splimes = { workspace = true } tokio = { workspace = true } anyhow = { workspace = true } diff --git a/weft-tui/src/main.rs b/weft-tui/src/main.rs index 48d2141f..7b62a6ad 100644 --- a/weft-tui/src/main.rs +++ b/weft-tui/src/main.rs @@ -1,6 +1,13 @@ #![warn(clippy::pedantic, clippy::nursery, clippy::all)] #![allow(clippy::multiple_crate_versions, clippy::used_underscore_binding, clippy::similar_names, clippy::module_name_repetitions, clippy::module_inception)] +// The process allocator is this binary's choice, not a side effect of what a library it links +// happens to depend on (PAC-3). turso used to carry it: its `mimalloc` feature installs a +// `#[global_allocator]` in turso's own crate root, so any program linking `weftdb` got mimalloc +// whether it wanted it or not. `weftdb` no longer asks for it; this binary says so itself. +#[global_allocator] +static ALLOCATOR: mimalloc::MiMalloc = mimalloc::MiMalloc; + use std::fs::OpenOptions; use tracing_subscriber::{fmt, prelude::*, EnvFilter}; diff --git a/weftdb/Cargo.toml b/weftdb/Cargo.toml index 4ab3d3ba..ca4afb8c 100644 --- a/weftdb/Cargo.toml +++ b/weftdb/Cargo.toml @@ -50,13 +50,17 @@ serde_json = { workspace = true } md5 = { workspace = true } sysinfo = { workspace = true } tracing = { workspace = true } -tracing-subscriber = { workspace = true } [dev-dependencies] csv = { workspace = true } criterion = { workspace = true , features = ["html_reports"] } tempfile = { workspace = true } serial_test = { workspace = true } +# The interpolation benches compare splimes' CPU, rayon and GPU backends, so they need the GPU +# path that the library deliberately does not enable for its users. +splimes = { workspace = true, features = ["gpu"] } +# Only the tests install a subscriber; the library logs through the `tracing` facades. +tracing-subscriber = { workspace = true } [target.'cfg(unix)'.dev-dependencies] # Only to keep the fault tests' deliberately aborted child from dumping core. diff --git a/weftdb/src/lib.rs b/weftdb/src/lib.rs index 26bf0f7f..910589e0 100644 --- a/weftdb/src/lib.rs +++ b/weftdb/src/lib.rs @@ -207,9 +207,11 @@ //! - **Advanced Interpolation**: Linear, cubic, quadratic, and polynomial spline methods //! (splimes 1.0), run on tokio's blocking pool so they never occupy an async worker //! - **Interpolation Backends**: splimes picks one CPU thread or the rayon pool by output -//! grid size, and the GPU (`wgpu`) only above the thresholds `splimes::calibrate()` +//! grid size, and its GPU backend only above thresholds its own `calibrate` function //! measures; a program calls it once at startup (`weft-server` does), and without it -//! interpolation stays on the CPU +//! interpolation stays on the CPU. That backend is splimes' opt-in `gpu` feature, which +//! this crate does **not** enable: enabling it here would compile Vulkan, Metal and DX12 +//! support into every program that links `weftdb`, whether or not it has a GPU. //! - **MVCC Concurrency**: Concurrent writes with `BEGIN CONCURRENT` transactions //! - **Pattern Recognition**: Extract and match recurring patterns in time-series data //! - **Event Detection**: Detect significant occurrences and predict future events @@ -255,7 +257,9 @@ //! - **Rust Version**: 1.95 or later, on stable //! - **Hardware**: `x86_64`, ARM64 architectures //! - **GPU Support**: Optional, through splimes' `wgpu` backend (Vulkan, Metal, DX12), -//! used once `splimes::calibrate()` has run +//! used once splimes' `calibrate` function has run. It is opt-in at the dependency level: +//! `weftdb` does not enable splimes' `gpu` feature, so a build that wants the GPU asks +//! for it explicitly. #![recursion_limit = "1024"] #![warn(clippy::pedantic, clippy::nursery, clippy::all)]