WeftDB

The time-series database for data that wasn't sampled on a clean grid.

Most time-series databases assume your data arrives on a regular cadence. Real data often doesn't. Sensors drop out. Markets trade in bursts. Devices batch-upload after hours offline. When you ask those databases for a value between two samples, you get a gap, a NULL, or a 0 — and the work of reconstructing the signal lands in your application code.

WeftDB puts that reconstruction in the database. Ask for any resolution you like and it returns a continuous series, interpolated with the spline method you choose, where every point is labelled raw, interpolated, or extrapolated — so a synthetic value is never silently mistaken for an observed one.

curl -s -X POST http://127.0.0.1:8080/api/v1/interpolate \
  -H 'content-type: application/json' \
  -d '{"spline":"linear","resolution":"seconds",
       "points":[{"timestamp":"1970-01-01T00:00:00Z","value":0.0},
                 {"timestamp":"1970-01-01T00:01:00Z","value":60.0}]}'

Two samples a minute apart come back as 61 labelled, per-second points — the two observed endpoints marked raw, the 59 between them interpolated.

---

Contents

Start here · Why WeftDB · Quickstart · Is WeftDB right for you? · Project status

Use it · What you get · HTTP API · Configuration · Rust library · Terminal UI

Understand it · Performance · How it works · Benchmarking · Development

---

Why WeftDB

Interpolation is a first-class query, not post-processing. Resampling, gap filling and upsampling run next to the data, over typed columnar segments, on one CPU thread, the rayon pool or — once calibrated — the GPU, chosen by grid size, not in a client loop pulling raw rows across the wire.

Precision is declared, never silently lost. Values are logically BigDecimal. Each aspect declares a physical encoding (F64, F32, ScaledI64, ScaledI128, Decimal128, BigDecimalText) and an error bound. A value the encoding cannot represent within that bound is rejected, not quietly rounded. If you are storing money or calibrated instrument readings, this is the difference between a database you can audit and one you can't.

Synthetic data is always marked as synthetic. Every reconstructed point carries its provenance. Downstream code can filter, weight or refuse interpolated values on purpose instead of by accident.

Claims come with benchmarks. Every performance number in this document links to the harness that produced it, and the harness ships in the repo so you can re-run it on your hardware. Where WeftDB doesn't win, this README says so.

---

Quickstart

Prerequisites: Rust 1.95 or newer on stable. A GPU with a wgpu backend (Vulkan / Metal / DX12) is optional — without one the engine runs on the CPU (one thread or the rayon pool) automatically.

cargo build --release

Prefer not to build from source? Each GitHub release, from v0.1.0 on, carries pre-built weft-server, weft-tui and weft-bench archives for Linux (x86_64 and aarch64), macOS (x86_64 and Apple silicon) and Windows (x86_64), and a SHA256SUMS file to check them with sha256sum -c SHA256SUMS. The Linux binaries are built on Ubuntu 24.04: weft-server and weft-bench need glibc 2.34 or newer, and weft-tui needs glibc 2.39 or newer. The macOS and Windows binaries are not signed, so Gatekeeper and SmartScreen will warn before running them. Read the release's known limitations (v0.1.0's are in its changelog) before you run it.

1. Start the server

WEFT_SEGMENT_STORE_ROOT=./store cargo run --release -p weft-server
# weft-server v0.1.0 listening on http://127.0.0.1:8080

WEFT_SEGMENT_STORE_ROOT is what enables the persistent /storage endpoints; without it the compute endpoints still work statelessly.

2. Interpolate without storing anything

The compute endpoints take points in the request, so you can evaluate WeftDB before committing any data to it:

curl -s -X POST http://127.0.0.1:8080/api/v1/interpolate \
  -H 'content-type: application/json' \
  -d '{"spline":"cubic","resolution":"seconds",
       "points":[{"timestamp":"1970-01-01T00:00:00Z","value":0.0},
                 {"timestamp":"1970-01-01T00:00:30Z","value":30.0},
                 {"timestamp":"1970-01-01T00:01:00Z","value":60.0}]}'

Already have InfluxDB Line Protocol? Point it at the ILP sibling instead:

printf 'cpu,host=a load=0 1000000000\ncpu,host=a load=60 1000000060\n' | \
  curl -s -X POST \
    'http://127.0.0.1:8080/api/v1/interpolate/ilp?field=load&precision=s&spline=linear&resolution=seconds' \
    -H 'content-type: text/plain' --data-binary @-

3. Store a series and query it back

Declare the aspect's schema once — this is where you choose precision:

curl -s -X POST http://127.0.0.1:8080/api/v1/storage/aspects \
  -H 'content-type: application/json' \
  -d '{"name":"price","physical_type":"scaled_i64","scale":2,
       "value_tolerance":"0","timestamp_unit":"millis"}'

scale: 2 with value_tolerance: "0" means exactly two decimal places, no rounding permitted — a third decimal is rejected at ingest rather than silently truncated.

# ingest
curl -s -X POST http://127.0.0.1:8080/api/v1/storage/price/points \
  -H 'content-type: application/json' \
  -d '{"points":[{"timestamp":1000,"value":"1.25"},
                 {"timestamp":2000,"value":"2.50"},
                 {"timestamp":3000,"value":"3.75"}]}'

# read the range back (start/end are required, in the aspect's declared unit)
curl -s 'http://127.0.0.1:8080/api/v1/storage/price/points?start=0&end=9999&limit=10'

# read one instant
curl -s 'http://127.0.0.1:8080/api/v1/storage/price/at?t=2000'

# bucket it
curl -s 'http://127.0.0.1:8080/api/v1/storage/price/downsample?resolution=seconds&agg=avg,max,p95'

Every read surface also speaks CSV, Arrow IPC and Parquet — append .csv, .arrow or .parquet, or use the …/range.parquet variants, and pipe straight into pandas, Polars or DuckDB.

---

Is WeftDB right for you?

Being honest about this saves you an evaluation.

A good fit

• Irregular or gappy series where you need values between the samples — sensor networks with dropouts, market data, batch-uploaded device telemetry.
• Resampling and gap filling as a hot path, run often enough that doing it in application code is costing you real latency or CPU.
• Values where precision is not negotiable — prices, billing quantities, calibrated scientific readings — and a silent f64 round-trip is unacceptable.
• Workloads that care which points are real, because a synthetic value would otherwise contaminate an alert, a model, or an invoice.
• Point lookups and narrow windows over large columnar segments, which WeftDB optimises specifically (see Performance).

A poor fit — today

• You need a mature, hardened production database. WeftDB has no TLS, no authentication, and no RBAC yet. Do not expose it to an untrusted network.
• Generic OLAP over wide tables. WeftDB is not competing with ClickHouse or DuckDB on broad analytical scans, and doesn't claim to.
• Regular, evenly-sampled data with no gaps. If interpolation isn't your problem, WeftDB's core advantage doesn't apply and a conventional TSDB is the simpler choice.
• Per-measurement tags / labels. Not implemented yet, so tag-based filtering and multi-dimensional grouping aren't available.
• High-cardinality dimensional slicing in the Prometheus sense — that's not the shape of WeftDB's data model.
• Managed hosting. There is none; you run it yourself.

---

Project status

Pre-beta. Expect sharp edges. WeftDB is an engineering-grade codebase — over 1,000 tests pass, the storage format is CRC-checksummed and versioned, and the benchmarks are real and reproducible — but it has not been hardened for production deployment.

v0.1.0 is the pre-beta baseline release: pre-built binaries on the GitHub release, and no crates on crates.io. Later releases test their upgrade, rollback and compatibility against it. Its known limitations include no authentication or TLS, incomplete crash consistency, and a store layout that the next release upgrades one way.

Area	State
Storage engine, interpolation, HTTP API, Arrow/Parquet	Working, tested, benchmarked
Backup & restore (control plane)	Working, with verification and a rehearsal drill
Observability — Prometheus metrics, OpenTelemetry traces	Working
Crash consistency — power-loss durability, crash-safe maintenance	In progress (design)
Security — TLS, auth, RBAC, audit logs	Not started
Pre-built binaries (Linux, macOS, Windows)	Released: v0.1.0, on GitHub
Library crates on crates.io	Not published yet
Packaging — Docker, Helm	Not started
Client SDKs (Python, TypeScript)	Not started
Per-measurement tags	Not started

The full work queue lives in ROADMAP.md. It is the single source of truth for status and priorities, and it records negative results alongside wins.

---

What you get

Query

• Interpolation on read — reconstruct any series onto any grid from nanoseconds to years, with Linear / Quadratic / Cubic / Polynomial splines. The engine picks single-threaded CPU, parallel (Rayon) or — once calibrated — GPU by grid size, and degrades the method gracefully (Cubic → Quadratic → Linear) when data is too sparse to support it.
• Provenance on every point — raw, interpolated or extrapolated.
• Downsampling into epoch-aligned buckets: min/max/avg/sum/first/last, exact nearest-rank p50/p90/p95/p99, three time-weighted averages (LOCF for change-only sensors, trapezoidal for continuous signals, and an LOCF variant that carries the last sample to the bucket's end), plus bounded-error sketch_p* percentiles that are mergeable and cost a fixed amount of memory per bucket however many samples land in it.
• Point and range reads that skip work: a point lookup on a regular column is resolved in closed form without materialising timestamps at all.

Store

• Typed columnar segments (.weftseg) — written once, never rewritten: maintenance (reconcile, split, overlap merge, squash, compaction) writes its output to new frames, fsyncs them, and swaps them for its inputs in one control-plane transaction, so a crash leaves either the inputs or the outputs, and a replaced frame is deleted only once no read can still be using it. CRC-checksummed, page-indexed, with per-segment and per-page min/max statistics for data skipping and a null/quality column.
• Compression that adapts per column — timestamps pick among delta-of-delta, RLE, Gorilla, fixed and per-block bit-packing; values pick among varint, bit-packing, per-block adaptive packing, frame-of-reference and an opt-in delta cascade. The codec each column actually used is reported, and bytes/point is the realized figure, not an estimate.
• Late and out-of-order data — detected, counted, and reconciled within a segment or across segments with last-writer-wins semantics, on demand or by background sweep.
• Backup and restore of the control plane, with two verification modes (one safe under concurrent writes) and a non-destructive restore drill so you can answer "is my backup actually restorable?" without risking the live store.

Integrate

• HTTP API with JSON, CSV, Arrow IPC, Parquet and InfluxDB Line Protocol in and out — so TSBS/Influx/QuestDB-shaped writers work unchanged, and results land directly in Arrow-native tooling.
• Prometheus /metrics with latency histograms, plus a live p50/p95/p99 profile endpoint and OpenTelemetry trace export.
• Rust libraries — use the engine, the storage layer or the reductions directly, without the server.
• Terminal UI for importing CSV, browsing and plotting aspects, and running compression interactively.

---

Performance

Every number below comes from a benchmark in this repo, named so you can re-run it. The exception is any figure measured on the real BTC corpus: that input is a local file the repository does not ship (see Ingest), so those figures cannot be reproduced from a clone. All were measured on one developer workstation — treat them as shape, not as a specification for your hardware.

Reads WeftDB optimises specifically

Operation	WeftDB	Naive full decode	Gain
Point lookup, 100k-row segment	56 µs	2.60 ms	~46×
Point lookup, paged frame	33 µs	279 µs	~8.5×
Batch of 64 instants (one pass vs 64 single lookups)	67 µs	3.57 ms	~54×
100-row window over 100k rows	58 µs	2.73 ms	~47×

Source: weft-physical-type/benches/pointread.rs, re-measured 2026-10-08 after the word-wise bit reads and the slicing-by-16 frame CRC. The wins come from skipping the value column entirely until a specific row is needed, and from resolving regular timestamp columns in closed form.

Ingest

Sealing 1M rows of a real 1-minute financial series into a columnar segment: 4.57 s — 218,963 rows/sec at 4.22 bytes/point.

That corpus (database/datasets/btc_1min.csv, a BTC/USD one-minute series) is a local, gitignored file, not distributed with this repository and not required to build or test — the suite skips the tests that use it when it is absent, and so do the benches that load it by default. Every "real BTC" figure in this README comes from it, so none of them can be reproduced from a clone. Whether those numbers are cleared for publication or retired is an open item with counsel before 1.0 (1.0 plan, Appendix B). The ingest figure is quoted anyway because a synthetic generator materially flatters the result: on the same workload the generated corpus reports 1.71 bytes/point against the real 4.22, roughly 2.5× too optimistic. Storage numbers measured on generated data are not reported as results.

Reductions

At 500k points, hour buckets, correctness-gated (weft-bench --downsample; re-measured 2026-10-08, full table under Compute endpoints): avg runs at 2.08M points/sec; time-weighted average costs about 1.39× that. The approximate sketch_p99 is ~2.6× faster than exact p99 (327 ms vs 852 ms) and stays within its 1% bound, while using bounded memory per bucket.

Where it doesn't pay off

A deliberately-kept honest result: the bit-sliced (transposed) value codec once measured ~5.7× faster than the linear bit-unpack at the kernel level, but only because the linear decoder tested one bit at a time. Once that decoder moved to word-wise reads, the linear layout unpacked faster (1.09 vs 1.95 ms per 1 Mi values, benches/bitunpack.rs). End to end, the bit-sliced layout costs +0.08% bytes and reads no faster than the linear codec (benches/transposed_read.rs), so it ships off by default (and is now behind the bitsliced-codec build feature, pending patent review). A kernel speedup measured against a weak baseline is not a result, and this README would rather say so than quote the 5.7×.

Exact decimals are not free yet either. Hourly avg over 1M real BTC closes (the local corpus described under Ingest, not in the repository), all three paths proven to produce the same buckets:

path	time	vs f64
f64 loop (lossy baseline)	2.14 ms	1×
weft_reduce::reduce_scaled (exact, on the stored ScaledI64 mantissas)	13.52 ms	~6.3×
weft_reduce::reduce (exact, per-sample BigDecimal)	60.04 ms	~28×

QuestDB documents ~2× for its DECIMAL, so the precision wedge still costs more here. Before this work the shipped path measured 100.65 ms (~43×). Two changes closed most of the gap: integer accumulation, and computing each bucket's avg by integer long division that reproduces bigdecimal's quotient digit for digit. Per sealed 1M-row segment (decode + six reductions), the stored-range downsample went from 127.6 ms to 59.7 ms. The server's GET /api/v1/storage/{aspect}/downsample now takes the integer path for ScaledI64 segments when every requested reduction is streaming or a sketch_p*, and falls back otherwise (weft-reduce/benches/decimal_tax.rs, load average ~11).

---

HTTP API

weft-server is the axum HTTP surface — Weft-Bench, TSBS-style harnesses, Grafana and SDKs all drive the engine through it. It carries no vendor-specific dependencies.

See the Quickstart to start it.

Bind address defaults to 127.0.0.1:8080 (WEFT_SERVER_ADDR overrides). Setting WEFT_SEGMENT_STORE_ROOT opens a Storage v2 segment store and enables the /storage endpoints (without it they answer 503, and GET /ready reports the segment_store dependency). A store root belongs to one server at a time: the server holds the root's LOCK file while it runs, and a second server pointed at the same root exits at startup with an error naming the first one's pid and host.

The root's STORE_FORMAT file records the store's layout and the oldest layouts a WeftDB must know to read and to write it (plain JSON; see weftdb::StoreFormat), and segment_index.db's store_meta table keeps a copy. The server reads the marker before it opens any database (a root without one, the copy, before it writes anything), and refuses a store a newer WeftDB wrote (IncompatibleLayout) without changing anything but LOCK and LOCK.holder. A store written before the marker existed is layout 1 and is upgraded in place on first open, through the registered migrations (store_migrations in segment_index.db lists those applied); its frames are not rewritten. A layout-1 store whose index records a frame outside segments/ (possible only through the aspect-name traversal of earlier builds) is refused with UnsafeLegacyPath, naming the aspects, before any migration applies or a marker is written, and nothing is moved: drop or re-seal those aspects with the build that wrote them first.

While bound to a loopback address, the server only answers requests addressed to a loopback host: Host must be localhost, an address in 127.0.0.0/8 or [::1] (anything else is a 421), and a state-changing request whose Origin is not a loopback origin is a 403. That keeps web pages from non-loopback origins, open in a local browser, from driving the server; a page served by another local web server (any loopback origin, on any port) is still trusted until authentication lands. Clients that send no Origin, like curl, are unaffected. /health and /ready follow the same rule, so a health probe must send a loopback Host such as localhost or 127.0.0.1, as every HTTP/1.1 client does; an HTTP/1.0 probe that sends no Host gets a 421, so configure it to send one or set WEFT_ALLOW_ANY_HOST=1. Behind a local reverse proxy that forwards a different Host, set WEFT_ALLOW_ANY_HOST=1; a proxy that keeps Host: 127.0.0.1 but forwards the Origin of a browser UI served from a non-loopback origin needs it too, or that UI's state-changing requests get a 403. A non-loopback bind is not guarded: there is no authentication yet, so keep such a server off untrusted networks.

Service endpoints

Method & path	Purpose
GET /health	Liveness.
GET /ready	Readiness, including the segment-store dependency check. With a store, poisoned / restart_required report that a segment-index COMMIT failed in a way that may still have committed, so the store refuses writes until the server restarts (reads keep working, and ready stays true); poison_reason says which transaction and error.
GET /metrics	Prometheus text exposition — request/error/output counters, ingest counters (weft_ingest_*), and latency histograms for the compute and storage-ingest paths.
GET /debug/profile/current	Live p50/p95/p99 latency snapshot per instrumented path.

Compute endpoints

The flagship interpolation-on-read path and its reduction counterpart. Each accepts JSON points or an InfluxDB-Line-Protocol body, and each returns JSON, CSV, Arrow IPC, or Parquet:

Endpoint	Purpose
POST /api/v1/interpolate	Reconstruct an irregular series onto a regular grid (spline = linear | quadratic | cubic | polynomial; resolution = nanoseconds..years). Every output point carries a kind — raw / interpolated / extrapolated. A grid of more than 10,000,000 points (MAX_INTERPOLATE_OUTPUT_POINTS, or WEFT_MAX_INTERPOLATE_POINTS) is refused with 400 before anything is computed; split larger ranges across requests. The response's spline is the method that ran: with fewer distinct timestamps than the requested method needs (cubic 4, quadratic 3, polynomial degree + 1) it is the simpler method splimes stepped down to.
POST /api/v1/interpolate/point	Evaluate the reconstructed signal at a single instant, labelled raw/interpolated/extrapolated.
POST /api/v1/downsample	Reduce samples into epoch-grid-aligned buckets — min/max/avg/sum/first/last, nearest-rank percentiles p50/p90/p95/p99, the three time-weighted averages (twa, LOCF dwell-weighting, for change-only sensors; twa_linear, trapezoidal Σ½(vᵢ+vᵢ₊₁)Δtᵢ/ΣΔtᵢ, for irregularly-sampled continuous signals; twa_bucket_end, LOCF that additionally carries the bucket's last sample to its grid end — twa gives that sample no weight, so a sensor reporting 0 then 100 a minute into an hour bucket reads twa=0 but twa_bucket_end=98.33), and the approximate sketch_p50/p90/p95/p99 (see below); all reductions computed in BigDecimal by the shared weft-reduce crate. Aliases: median→p50, time_weighted_avg/twa_locf→twa, time_weighted_avg_linear→twa_linear, twa_locf_end→twa_bucket_end, sketch_median→sketch_p50. Only non-empty buckets are emitted.
POST /api/v1/{interpolate,downsample}/ilp	The same, fed an ILP text/plain body (the TSBS/InfluxDB/QuestDB wire format); field, precision (ns/us/ms/s), and the compute knobs are query parameters. interpolation= is accepted as an alias for spline= (the canonical spline wins if both are given).
POST /api/v1/{interpolate,downsample}/{csv,arrow,parquet} and …/ilp/{csv,arrow,parquet}	The same computations with CSV (text/csv), Arrow IPC stream, or Parquet output — so a harness feeding line protocol pulls results in any of the four formats.

What the reductions cost. Measured on the shipped harness (500k points, 5 reps, hour buckets, correctness PASS throughout — weft-bench --downsample --ds-points 500000 --ds-aggs <agg>; re-measured 2026-10-08 after the avg division change, at load average ~19). These use the generated series, whose values are exact binary expansions of floats (~50 significant digits), so they are a pessimistic bound. On real two-decimal prices (--ds-csv, 1M BTC/USD one-minute closes from the local corpus described under Ingest, which is not distributed with the repository; hourly avg,p99,twa) the same reduction runs at 4.24M points/sec against 0.56M points/sec for a generated series of the same size (weft-bench --downsample --ds-csv database/datasets/btc_1min.csv --csv-value-col 4 --csv-skip 3000000 --ds-points 1000000 --ds-bucket h --ds-aggs avg,p99,twa --reps 5):

reduction	p50	throughput	note
avg	240.8 ms	2,075,477 points/sec	the streaming baseline — no bucket materialized
twa	334.1 ms	1,515,029 points/sec	1.39× the streaming cost: dwell-weighting needs the time-ordered samples
twa_bucket_end	332.1 ms	1,518,520 points/sec	within noise of twa — one extra weight, effectively free
twa_linear	407.8 ms	1,197,606 points/sec	1.22× twa: an extra BigDecimal add + divide per interval for the trapezoidal mean

Exact vs sketch percentiles — which to ask for. The p50/p90/p95/p99 reductions are exact nearest-rank: they return an actual observed BigDecimal from the bucket, but they materialize and sort the whole bucket, and two buckets' results cannot be combined. The sketch_p50/p90/p95/p99 reductions instead stream every value into a DDSketch — a 1% relative-error bound (weft_reduce::SKETCH_ALPHA), bounded memory regardless of bucket size (values fold in as they arrive; SKETCH_MAX_BINS caps the store absolutely), and an exactly mergeable structure, so a p99 can be computed over a large or streaming bucket. Measured on the shipped harness (500k points, 5 reps, correctness PASS): sketch_p99 runs at 1,528,361 points/sec (p50 = 327.30 ms) versus exact p99 at 584,469 points/sec (p50 = 852.39 ms) — ~2.6× faster (weft-bench --downsample --ds-points 500000 --ds-aggs sketch_p99 vs --ds-aggs p99).

The approximation is declared, never silent — WeftDB's precision principle. The sketch shares the exact percentiles' nearest-rank convention (⌈q·n⌉), so sketch_p* and p* name the same sample at every bucket size and the sketch is always within the 1% bound of the exact answer — the two are substitutable. (DDSketch's reference rank is ⌊q·(n-1)⌋, which on a small bucket picks a different sample; WeftDB deliberately does not inherit that.) The exact percentiles remain the default and cost nothing on a small bucket, so prefer them there; reach for a sketch when a bucket is too large to materialize or the result must merge.

curl -s -X POST http://127.0.0.1:8080/api/v1/interpolate \
  -H 'content-type: application/json' \
  -d '{
        "spline": "linear",
        "resolution": "seconds",
        "points": [
          { "timestamp": "1970-01-01T00:00:00Z", "value": 0.0 },
          { "timestamp": "1970-01-01T00:01:00Z", "value": 60.0 }
        ]
      }'

printf 'cpu,host=a load=0 1000000000\ncpu,host=a load=60 1000000060\n' | \
  curl -s -X POST \
    'http://127.0.0.1:8080/api/v1/interpolate/ilp?field=load&precision=s&spline=linear&resolution=seconds' \
    -H 'content-type: text/plain' --data-binary @-

Malformed payloads, unknown tokens, fewer than two usable points, or a zero-span series return 400 with an {"error": "..."} body.

Numeric boundary: the compute endpoints' wire values are f64 — the documented transport boundary (the logical BigDecimal is narrowed only at the HTTP edge, and the columnar value columns are honestly typed Float64 — no false precision). The storage endpoints below are lossless.

Storage endpoints (Storage v2 over HTTP)

Catalog management, ingest, and stored-range reads against the .weftseg segment store. The no-silent-downcast guarantee holds end to end: a value unrepresentable under the declared encoding/tolerance is rejected 400.

Endpoint	Purpose
POST /api/v1/storage/aspects	Declare an aspect's schema — physical encoding, value tolerance, timestamp unit. The name also names the aspect's files on disk, so it must be at most 160 bytes with no /, \, control characters, bidirectional controls, line separators or invisible formatting characters (the zero-width joiner and non-joiner are allowed), leading ., trailing . or space, and must not be a Windows device name (CON, NUL, COM1, …, also with an extension or a : suffix, as in nul:x); a server running on Windows also refuses <, >, :, ", `
GET /api/v1/storage/aspects · …/{aspect}/schema · …/catalog	List declared schemas; read one aspect's schema; report the store's (database, subject) scope + registered hierarchy.
POST /api/v1/storage/{aspect}/points	Ingest a JSON batch (dense or nullable; single-block or paged via rows_per_page). Set require_sorted to reject an out-of-order batch (400, naming the first backwards row) instead of sealing it. Every ingest response reports time_sorted (whether the sealed segment stored in monotonic order).
POST /api/v1/storage/{aspect}/{ilp,parquet,csv}	Ingest an ILP payload, a Parquet file, or CSV rows into a declared aspect — all sealing through the same schema-enforced path, and all honouring require_sorted (query param) + reporting time_sorted.
GET /api/v1/storage/{aspect}/range · …/value-range	Stream a stored time window / value band as Arrow IPC (application/vnd.apache.arrow.stream).
GET …/range.parquet · …/value-range.parquet	The same windows as Parquet files.
GET …/range.csv · …/value-range.csv	The same windows as CSV (lossless decimal-text values; empty field = null).
GET /api/v1/storage/{aspect}/points · …/value-points	Lossless JSON reads with declarative pagination — offset/limit (+ take and 1-based page aliases), with total/count/offset in the body. For stable forward iteration the body also carries an opaque next_cursor while rows remain; a client re-issues with ?cursor=<token> (supersedes offset/page; malformed → 400) until it is absent.
GET /api/v1/storage/{aspect}/at?t=	Single-instant point lookup: the present value at exactly t (lossless decimal text) or a found:false miss. An order-signal-driven read planner resolves each candidate segment with its persisted time_sorted flag — binary search on a sorted segment, linear scan only on an out-of-order one — after pruning the index to the files spanning t.
GET /api/v1/storage/{aspect}/at-multi?t=,,	Batch point lookup: a comma-separated list of instants resolved in one pass (points:[{timestamp,value,found}] in query order). The index is pruned once by the batch's whole span and each segment's timestamp column decoded once for the whole batch, so N instants sharing a segment cost one decode, not N.
GET·POST /api/v1/storage/{aspect}/downsample?start=&end=&resolution=&agg=	Stored-range downsample — the same reduction set as POST /api/v1/downsample (above), but over the aspect's persisted segments instead of a request-supplied series, so a client aggregates a large stored range without fetching it. Bounded memory: the index is pruned by time and each surviving segment folds into its own mergeable partial reduction, merged once — only one segment's rows are ever resident, so a range far larger than RAM reduces, and with a sketch_p* the per-bucket state is bounded too. Omitted bounds span all stored history. …/downsample.csv · …/downsample.arrow · …/downsample.parquet serve the identical reduction in the compute endpoints' columnar formats. Traced as a downsample.range stage span. The pruned segments are reduced concurrently: measured 4.69× faster than the sequential fold (16 segments over 200k rows, 39.88 ms → 8.50 ms; sketch_p99 49.49 ms → 9.42 ms, 5.25×) — see database/benches/downsample_range.rs. A single pruned segment takes an inline fast path (concurrency there costs more than it buys). Materialized partials (opt-in): with WEFT_SEGMENT_PARTIAL_BASE set, each segment carries a .weftpart partial-reduction sidecar written at seal (a sealed segment is immutable, so its partial never goes stale), and a downsample of the bounded reductions (min/max/avg/sum/first/last + sketch_p*) merges the stored partials instead of decoding the value column — measured 3.2× faster (16 segments / 200k rows, 8.39 ms → 2.60 ms, non-overlapping CIs) and re-keyed to any coarser nesting resolution (minutes→hours→days) with no decode. Rollup tiers (opt-in): WEFT_SEGMENT_PARTIAL_TIERS materializes coarser tiers beside the base (e.g. hours,days), each re-keyed from the tier below, so a coarse query folds the coarsest matching tier instead of the whole fine base — the in-storage analogue of TimescaleDB's hierarchical continuous aggregates. Measured ~1.16× on a DAY query over a MINUTES base (16 segments / 200k rows, 7.30 ms → 6.30 ms, non-overlapping CIs). The win grows with the base-bucket count per segment, and that scaling is now measured rather than asserted: holding rows and segments fixed while widening the sample stride (so only the base buckets per segment change) gives 1.15× at ~208 base buckets → 1.27× at ~3125 → 1.38× at ~12500 (bench_tiered_span). Honest read: the win is real and climbs, but a ~60× increase in base buckets buys only 1.15×→1.38×, so the elided re-key is not the dominant cost and the tiers' extra .weftpart bytes should be weighed against it — see database/benches/downsample_range.rs. A reconcile/split/squash regenerates the sidecar (tiers and all) so the acceleration survives a rewrite.
POST /api/v1/storage/{aspect}/reconcile	Reconciliation pass (mode in the response). Default (intra-segment): rewrite every out-of-order segment of the aspect into a time-sorted one in place. ?threshold=N gates it on unsorted_segments >= N (QuestDB-style split-count trigger). ?hot_cold=true reconciles cold segments but defers the hot tail until the backlog reaches the threshold. ?overlaps=true instead runs the cross-segment overlap merge (newer-wins) — reconciled is then the number of segments merged away, and ?split_min_bytes=N makes that merge split-not-rewrite (carve off a dominant cold prefix instead of rewriting the whole component). Returns triggered/reconciled/cold_reconciled/hot_reconciled and the post-pass unsorted_segments + overlapping_segments. Maintenance operations on one aspect take turns: while another one holds the aspect (a daemon pass, another request), the request waits up to 30 s, then answers 409 Conflict without touching the aspect; retry.
POST /api/v1/storage/{aspect}/squash	Squash the aspect's segments into one (newer-wins), bounding split-path fragmentation. ?max_segments=N gates it (squash only when the count exceeds N). Returns triggered/removed/segment_count. Waits for a busy aspect and answers 409 as reconcile does.
POST /api/v1/storage/{aspect}/compact?target_rows=N	Size-targeted compaction — coalesce the aspect's segments toward ~N rows per segment (leaving already-large segments untouched), holding fragmentation near the read-optimal size rather than folding to one (which squash does). Motivated by the downsample_range knee (one giant segment reads slower than several mid-sized ones). target_rows is required (absent → 400). Returns removed/segment_count. The manual counterpart of the WEFT_COMPACT_TARGET_ROWS daemon pass. Waits for a busy aspect and answers 409 as reconcile does.
POST /api/v1/storage/reconcile	Store-wide reconciliation sweep across every declared aspect: threshold (default), ?hot_cold=true, or ?overlaps=true (+ ?split_min_bytes=N for split-not-rewrite). Returns mode, aspects_scanned / aspects_reconciled / segments_reconciled (+ cold/hot split), the post-sweep store-wide unsorted_segments + overlapping_segments, and failed: a [{aspect, error}] list of the aspects whose pass failed (empty on a clean sweep). A failing aspect (e.g. a torn or truncated frame) no longer aborts the sweep: every other aspect is still swept and the response is 200, so check failed to tell a partial sweep from a clean one (earlier versions returned 500 at the first failing aspect and left every aspect after it in name order unswept). Only an unreadable aspect list is still a 500. An aspect another maintenance operation holds is waited for (30 s in all, across the sweep); if some are still held then, the others are swept and the answer is 409 Conflict naming the held ones. (The background daemon never waits: it skips a busy aspect until its next tick.) The manual counterpart to the background reconcile daemon (WEFT_RECONCILE_INTERVAL_SECS / WEFT_RECONCILE_THRESHOLD / WEFT_RECONCILE_HOT_COLD / WEFT_RECONCILE_OVERLAPS / WEFT_RECONCILE_SPLIT_MIN_BYTES / WEFT_RECONCILE_MAX_SPLITS).
POST /api/v1/storage/backup	Online control-plane backup (Phase 7.4) — snapshot the store's four control-plane DBs (segment_index/metadata/aspect_catalog/catalog) to fresh files via Turso's stable VACUUM INTO. Lands in <base>/<label> where <base> is WEFT_BACKUP_DIR or <store_root>/backups and <label> is a traversal-guarded ?label= ([A-Za-z0-9][A-Za-z0-9._-]{0,99}: 1 to 100 letters, digits, ., _ or -, starting with a letter or digit and not ending with .) or, without one, a generated manual-<unix_millis>. The backup-<digits> form is reserved for the backup daemon (retention counts and prunes exactly those names), so a ?label= in that form is a 400, and no snapshot taken through this endpoint is ever pruned by retention. ?verify=source|snapshot picks the verification: source (the default) cross-checks each copy's user-table set + row counts against the live control plane — the strongest check, but it assumes a quiescent store; snapshot verifies each copy on its own terms (it reopens and fully scans every row of every table) without re-reading the source, which is what an online backup taken while ingest continues needs. An unknown token → 400. Returns per-DB {name, tables, rows, bytes} + total_rows/total_bytes + the verify mode that ran. The backup is built in a .partial-* directory beside its label and appears under the label, with a MANIFEST.json, only once complete and durable. A label outside that grammar → 400 (the grammar also keeps out .partial-*, .deleting-* and .restore-drill-*, the names of unfinished backups, prunes and drills); an existing target dir → 409 with "code": "already_exists"; no store → 503. Control plane only — the .weftseg measurement frames are not part of this backup (hard-constraint #3).
POST /api/v1/storage/restore/drill?label=<backup>	Restore drill (Phase 7.4) — rehearse restoring a backup on a running server, so an operator can answer "is my backup actually restorable?" without risking anything. Restores the named backup into a throwaway directory beside the backups, verifies every restored database at its destination (it opens, and every row of every table reads), returns per-DB {name, tables, rows, bytes} + total_rows/total_bytes/restorable, then deletes the rehearsal copy (cleanup_error says why, if it could not; it is null otherwise). Non-destructive by construction — it cannot touch the live store, and restoring over a live control plane is deliberately not offered (the library primitive refuses to clobber; pointing WEFT_SEGMENT_STORE_ROOT at a restored copy is a deployment decision, not an HTTP call). Unknown backup → 404; a label outside the backup label grammar (so also traversal and the staging names .partial-*, .deleting-*, .restore-drill-*) → 400, while a daemon snapshot's backup-<digits> is accepted, since a drill only reads it; a rehearsal directory already taken by a drill started in the same millisecond → 409 with "code": "already_exists" (retry; nothing was restored or removed); a backup that will not restore → 500 carrying the failure, which is a failed drill rather than a server bug.
GET /api/v1/storage/{aspect}/stats · …/storage/stats	Materialized per-aspect and store-wide rollups, including the realized bytes/point (the north-star cost term), an unsorted_segments order-health count (segments that would force a linear scan on a point lookup), and an overlapping_segments count (time-overlapping segments — the cross-segment order-health signal, computed by an index scan). Rollup fields are served from the control plane without opening a segment.

Metrics

GET /metrics renders Prometheus text exposition (v0.0.4): shared weft_interpolate_* / weft_downsample_* counters across each endpoint family, weft_ingest_* counters (requests/errors/rows/segments) on the ingest paths, weft_reconcile_* counters (_passes_total / _segments_reconciled_total) over the out-of-order reconciliation passes (manual, store-wide, and the background daemon — threshold-held calls excluded) plus _failed_passes_total, the per-aspect passes a store-wide sweep (daemon tick or POST /api/v1/storage/reconcile) skipped after an error (the daemon also logs one WARN line per failed aspect), weft_backup_* counters (_snapshots_total / _bytes_written_total) over the online control-plane backups, and latency histograms over the compute and seal paths. GET /debug/profile/current serves the same timing data as a live p50/p95/p99 snapshot.

Backup & restore

The control plane (the catalog, per-aspect schema, segment index and metadata rollup databases) has an online backup, two verification modes, an unattended daemon, and a restore. Per the storage boundary this covers the control plane only — the .weftseg measurement frames are not part of a snapshot.

• Online snapshot. POST /api/v1/storage/backup copies all four databases with Turso's VACUUM INTO while the store stays fully usable, and verifies every copy before returning (see the endpoint catalog above for the parameters).
• Two verifications, because "verified" means different things under load. ?verify=source (default) cross-checks each copy against the live control plane — the strongest check, valid while the store is quiescent. ?verify=snapshot never re-reads the source; it reopens each copy and fully scans every row of every table, so it is correct while writers are committing. A snapshot-only backup taken with six concurrent ingests in flight returns 200 with the concurrent seals visible in the copy.
• A backup directory holds exactly its four databases and a MANIFEST.json. Verifying a snapshot reopens it, which leaves zero-byte -wal / -log journal sidecars beside the copy; these are swept after every verification, in both modes. Only a sidecar that is exactly zero bytes is removed — a non-empty -wal holds un-checkpointed frames, which is data, and is never touched. Verification also requires each database's own tables, so an empty file never passes.
• A backup appears only once it is complete and durable. It is built in a .partial-{label}-{nonce}/ directory; each database is verified and fsynced, the manifest (format, creation time, each file's size, tables and rows) is written and fsynced, and the directory is fsynced, renamed to its label, and the parent fsynced. A backup that fails with an error before the rename removes its build directory again; a crash there leaves the .partial-* directory, which is never counted or restored, and which the backup daemon's sweep removes (where the daemon is not enabled, remove it by hand). Once renamed, only the parent's fsync is left: if that fails, the call reports an error (500 over HTTP) although the complete backup is already under its label, so a retry with the same label is refused (409). The next backup's fsync of the same parent makes it durable.
• What a snapshot costs is dominated by the filesystem, not by the vacuum. Benchmarked in database/benches/backup_cost.rs: on tmpfs a whole four-database backup-and-verify runs in 8.62 ms at one aspect, 6.18 ms at 16 and 16.79 ms at 128 (snapshot verification; source verification is within noise of those), so the vacuum's own work is milliseconds and scales with control-plane rows rather than being a fixed per-database floor. On a real disk it is orders of magnitude slower, because Turso's VACUUM INTO fsyncs the destination and runs a TRUNCATE checkpoint before returning. On this project's btrfs volume (load average ~12) the background daemon's ticks landed 3.0–4.5 s apart at a 2 s configured interval. A tick fires late only when the one before it outlasts the interval (tokio's default Burst catch-up), so each snapshot and its prune took that long, and they ran back to back. Set WEFT_BACKUP_INTERVAL_SECS from what your storage can sustain, not from how much data you have; point WEFT_BENCH_BACKUP_DIR at your own volume to measure it.
• Unattended backups with retention. Set WEFT_BACKUP_INTERVAL_SECS to run a background daemon that snapshots into a fresh backup-<unix_millis> directory each tick (verifying snapshot-only, since it backs up a live store), and WEFT_BACKUP_KEEP to retain only the newest N. Retention is deliberately narrow: it only ever removes daemon-generated backup-<digits> directories, a form the API refuses to create (a directory you rename into it by hand is treated as generated and can be pruned, as it always could), so a snapshot you took through the API (?label=nightly, or an unlabelled manual-<unix_millis>) is never a prune candidate and never counts toward N, and it runs only after a successful snapshot, so a run of failures cannot prune your last good backup away. It counts only complete backups (a manifest, or all four databases from before manifests), and renames a pruned backup to .deleting-* before removing its files, so an interrupted prune never leaves a half-removed backup. A backup-<digits> directory stamped more than 24 h past the clock, or past the directory's own modification time, did not come from the daemon: retention neither counts nor removes it, and logs a warning each time it sees it. The modification-time check keeps such a directory ignored after the clock catches up with its stamp, as long as the directory is left unmodified and the filesystem reports modification times; remove it by hand when the warning appears. At start and on every tick the daemon removes .partial-*, .deleting-* and .restore-drill-* entries untouched for an hour. Both paths record weft_backup_* metrics.
• Restore, with a drill. weftdb::restore_control_plane(backup_dir, root) puts a snapshot back into a store root and verifies each restored file at its destination — what matters is that the file the store will open actually reads. It refuses an incomplete backup directory (or one whose files no longer match its manifest, or one with a staging name) and refuses to overwrite an existing control plane, pre-flighting both across all four files before writing anything, so a rejected restore leaves nothing behind. Each file is copied to <name>.tmp, fsynced and verified; only once all four verify are they renamed into place, one after another, and the root fsynced. A failure before the renames leaves only .tmp files, which a retry replaces; one part-way through them leaves some databases in place, which a retry refuses to overwrite, so clear the root first. Restoring beside the original segments/ frames reconstitutes a working store: the round-trip test reopens the restored root and reads its measurements back.
• Drills on a live server. POST /api/v1/storage/restore/drill?label= rehearses a restore of the named backup into a throwaway directory, verifies it, reports restorable, and cleans up — non-destructive by construction, so the question "is my backup actually restorable?" can be asked of production. Restoring over a live control plane is not offered: that is a deployment decision (point WEFT_SEGMENT_STORE_ROOT at a restored copy and restart), not an HTTP call.

---

Configuration

WeftDB is configured primarily through environment variables:

Variable	Used by	Purpose	Default
WEFT_DATA_DIR	weftdb, weft-tui	Root directory for database files and the TUI log (weft-tui.log).	portable per-user default (see below)
TEST_DATA_DIR	weftdb	Highest-priority override for the database root (used by the test suite).	unset
WEFT_EXTRACTED_BATCH_RETENTION_SECS	weftdb	How long (seconds) a legacy aspect remembers the batches pattern extraction consumed, so the incremental build does not queue them again. Each extraction deletes older records, keeping about one row per resolution step of this period. Set it above the longest interval between pipeline runs of an aspect plus the longest ingest call; a non-positive or unparsable value keeps the default.	172800 (48 hours)
WEFT_SERVER_ADDR	weft-server	HTTP bind address.	127.0.0.1:8080
WEFT_ALLOW_ANY_HOST	weft-server	Truthy (1/true/yes/on) → turn off the loopback request guard. On a loopback bind the server otherwise answers only requests whose Host is localhost, 127.0.0.0/8 or [::1] (421 otherwise) and refuses state-changing requests from non-loopback web origins (403). Set it behind a local reverse proxy that forwards a different Host, or one that forwards the non-loopback Origin of a browser UI it fronts. A non-loopback bind is never guarded.	unset (guarded on a loopback bind)
WEFT_GPU_CALIBRATE	weft-server	Startup calibration. By default, once the listener is bound, the server runs splimes::calibrate() once in the background on a blocking thread: it starts the GPU if there is one, times the single-thread, rayon and GPU backends on grids up to 16 Mi points (several seconds, a few hundred MB), and sets where Backend::Auto switches between them. Requests are served from the start and interpolate on the CPU with splimes' default thresholds until it finishes. A CPU/software adapter (llvmpipe, lavapipe, WARP) is not calibrated, with a log line saying why; force calibrates it anyway. 0 (or false/no/off) skips calibration. It logs the adapter (or why there is none) and the thresholds, and never fails startup; skipped or failed, interpolation stays on the CPU with splimes' defaults.	unset (calibrate, except a software adapter)
WEFT_MAX_INTERPOLATE_POINTS	weft-server	The most output points one /api/v1/interpolate* request may produce; a larger grid is a 400 naming its size and the limit, refused before anything is allocated. A positive integer, read once at startup; anything else (including 0) stops the server from starting with a message naming the variable.	10000000
WEFT_SEGMENT_STORE_ROOT	weft-server	Root of the Storage v2 segment store; enables the /storage endpoints.	unset (storage endpoints answer 503)
WEFT_ON_AMBIGUOUS_COMMIT	weft-server	What the server does once its segment store is write-poisoned (after a COMMIT that may or may not have committed): poison keeps serving, refusing every write until the server restarts while reads keep working and GET /ready reports poisoned; exit logs and exits with status 70, for a supervisor that restarts the server. The restart's open settles the transaction either way. Read once, at startup; the weftdb library never reads it and never exits the process.	unset (poison)
WEFT_RECONCILE_INTERVAL_SECS	weft-server	Background reconcile daemon sweep interval in seconds; 0/unset disables it.	unset (disabled)
WEFT_RECONCILE_THRESHOLD	weft-server	unsorted_segments backlog an aspect must reach before the daemon reconciles it.	1
WEFT_RECONCILE_HOT_COLD	weft-server	Truthy → the daemon reconciles cold segments each tick and defers the hot tail until the threshold.	unset (all-or-nothing)
WEFT_RECONCILE_OVERLAPS	weft-server	Truthy → each daemon tick also merges cross-segment time-overlap groups.	unset (disabled)
WEFT_RECONCILE_SPLIT_MIN_BYTES	weft-server	Split-not-rewrite floor (bytes) for the daemon's overlap merge; a dominant cold prefix clearing it is split off rather than rewritten.	unset (default 50 MiB floor → full-rewrite)
WEFT_RECONCILE_MAX_SPLITS	weft-server	Segment-count cap; each tick also squashes every aspect over it into one segment (bounds split-path fragmentation).	unset (no squash)
WEFT_COMPACT_TARGET_ROWS	weft-server	Target segment size in rows for the daemon's size-aware compaction pass; each tick coalesces every aspect's segments toward ~this many rows per segment (leaving already-large segments alone), holding fragmentation near the read-optimal size instead of folding to one (WEFT_RECONCILE_MAX_SPLITS). Motivated by the downsample_range knee (one giant segment reads slower than several mid-sized ones). Needs the daemon interval set.	unset (no compaction)
WEFT_BACKUP_DIR	weft-server	Directory each control-plane backup is written under — shared by the manual POST …/storage/backup endpoint and the background backup daemon, so hand-taken and automatic snapshots live together.	<store_root>/backups
WEFT_BACKUP_INTERVAL_SECS	weft-server	Background control-plane backup daemon interval in seconds; 0/unset disables it. Each tick writes a verified snapshot into a fresh backup-<unix_millis> directory, using the concurrent-write-safe snapshot verification (it backs up a live store). Needs a segment store configured.	unset (disabled)
WEFT_BACKUP_KEEP	weft-server	How many daemon-generated snapshots to retain; after each successful backup the oldest backup-<digits> directories beyond this many are removed. Only daemon-generated names are ever counted or pruned — a snapshot taken through POST …/storage/backup (a ?label= one, or an unlabelled manual-<unix_millis> one) is never a candidate — and a generated-looking directory stamped more than 24 h past the clock or past its own modification time is ignored (not counted, not removed, logged as a warning; remove it by hand).	unset (retain everything)
BTC_TEST_MAX_ROWS / BTC_TEST_FULL	weftdb (tests)	Row cap for the bounded, hermetic default run of test_create_btc_1min_database (into a temp data dir), or BTC_TEST_FULL=1 for the historical whole-corpus load into the shared data dir.	5000 / unset
WEFT_SEGMENT_CHECKPOINT_STRIDE	weft-server	Rows between entries of the sealed timestamp checkpoint index — trades a little size for much faster point lookups on sorted, irregular columns (~3.45× single-block; see the checkpointed-frames feature above). Applies only where it pays: sorted + irregular + at least WEFT_SEGMENT_CHECKPOINT_MIN_ROWS rows. ~1024 is the sweet spot (stride barely moves speed but does move size).	unset (no index; frames byte-for-byte as before)
WEFT_SEGMENT_CHECKPOINT_MIN_ROWS	weft-server	Row floor below which a segment is never checkpointed (a small column decodes trivially, so an index would be pure cost).	8192
WEFT_SEGMENT_CHECKPOINT_MAX_CODEC_OVERHEAD	weft-server	Ceiling on the timestamp-codec override a checkpointed seal will accept (blocked / best bytes; 1.0 = only when free). A checkpointed frame must use the range-decodable per-block codec, which is ~free where that codec already wins but ~3.5× on a Gorilla-shaped and ~14× on an RLE-shaped column — this refuses those seals rather than silently bloating them.	1.25
WEFT_SEGMENT_TRANSPOSED_MAX_OVERHEAD	weft-server	Ceiling on the size overhead the bit-sliced (transposed) value codec may pay against the size-selected codec (transposed / best bytes; 1.0 = only when free, and a value below 1.0 is meaningful — the bit-sliced layout can be a strict size win, since it pays one width header per 1024-value tile where the blocked codec pays one per 64 values). Stores the value column bit-plane-major. Measured byte-neutral end to end on a 1M-row column and no faster to read than the linear codec (see Where it doesn't pay off), so it is off by default. Needs a build with the bitsliced-codec feature; any other build ignores the variable and logs a warning.	unset (no transposed codec; frames byte-for-byte as before)
WEFT_SEGMENT_PARTIAL_BASE	weft-server	Resolution token (seconds/minutes/hours/…) at which each sealed segment materializes a partial-reduction .weftpart sidecar — a stored mergeable partial of the bounded reductions. A stored-range downsample of those reductions then merges the sidecars instead of decoding the value column (measured 3.2×), re-keyed to any coarser nesting resolution. A sealed segment is immutable, so the sidecar never goes stale; a rewrite (reconcile/split/squash) regenerates it.	unset (no sidecar; downsample_range decodes as before)
WEFT_SEGMENT_PARTIAL_MIN_ROWS	weft-server	Row floor below which a segment gets no partial sidecar (a tiny segment's partial saves too little decode to be worth the extra file).	4096
WEFT_SEGMENT_PARTIAL_TIERS	weft-server	Comma-separated fine→coarse rollup resolutions (e.g. hours,days) materialized beside the WEFT_SEGMENT_PARTIAL_BASE partial, each re-keyed from the tier below (up to four kept; a finer/non-nesting entry is skipped). A coarse stored-range downsample then folds the coarsest matching tier instead of re-keying the whole fine base (measured ~1.16× on a DAY query over a MINUTES base). No effect unless WEFT_SEGMENT_PARTIAL_BASE is set.	unset (base-only sidecar)
RUST_LOG	all	tracing filter. On weft-server it drives a per-request root span (request{method,path,request_id}, echoed as x-request-id) that every per-stage span nests under, each with busy/idle timing: the compute paths (interpolate.parse/compute/serialize under interpolate.engine, downsample.parse/reduce); the storage read paths (storage.{range,value_range,point}.read + .serialize, with a format field over JSON/CSV/Arrow/Parquet); the ingest paths (storage.ingest.parse/normalize/seal for ILP/CSV/JSON, and storage.ingest.parquet for the Parquet decode+seal); the background reconcile daemon (reconcile.tick{kind,…,aspects,segments}); and the control-plane writes on the seal path (control_plane.index.insert{aspect,id,rows_changed}, control_plane.index.delete{…}, control_plane.metadata.put{aspect,rows_changed}), whose rows_changed is libSQL's own affected-row count (Statement::n_change()) rather than an inference — so a trace shows whether a catalog write actually changed anything.	weft_tui=debug,database=debug,info
OTEL_EXPORTER_OTLP_ENDPOINT	weft-server	When set (e.g. http://localhost:4317), export tracing spans to an OpenTelemetry collector over OTLP/gRPC in addition to the RUST_LOG fmt output. Unset → no exporter, no network dependency; a misconfigured/absent collector never blocks startup. scripts/verify-otlp.sh verifies delivery end-to-end against a local Jaeger container (starting one if needed).	unset (export disabled)
SKIP_SLOW_TESTS	tests	Set to 1 to skip long-running tests.	unset

The WEFT_SEGMENT_* variables configure the store weft-server opens; the server maps them to weftdb::SegmentStoreOptions. A program embedding weftdb gets them only by asking: SegmentStore::open uses SegmentStoreOptions::default() (every one of them unset), and SegmentStore::open_with_options(root, SegmentStoreOptions::from_env()) reads them as the server does.

The database root directory is resolved in this order:

1. TEST_DATA_DIR, if set (explicit override for the test suite);
2. otherwise WEFT_DATA_DIR, if set;
3. otherwise a portable, per-user default — ~/.weftdb/data (falling back to the platform data directory + weftdb, e.g. %APPDATA%\weftdb on Windows, if the home directory cannot be determined).

No paths are hard-coded: data lands in a writable, machine-independent location out of the box, and setting WEFT_DATA_DIR relocates both the databases and the TUI log together. The resolution functions are exported as weftdb::data_dir() (resolved) and weftdb::default_data_dir() (the raw default).

The splimes crate also exposes build features:

Feature	Effect
gpu (default)	The wgpu backend. Without it, interpolation always runs on the CPU.
serde (default)	Serialize/Deserialize for Point, PointKind, Resolution and Spline.
tokio	Interpolator::run_async, which runs an interpolation on tokio's blocking pool. WeftDB enables it: every async caller (the weftdb read and compression paths, weft-server, the bench adapter) interpolates off the async workers.

The GPU is started at runtime, not at build time: weft-server calibrates it in the background at startup (WEFT_GPU_CALIBRATE, above), and an embedding program calls splimes::calibrate() or splimes::prewarm_gpu() itself.

WeftDB's own build features are all off by default, sit outside the 1.0 semver promise, and are pending patent review. A default build reads every segment written under the default configuration (WEFT_SEGMENT_TRANSPOSED_MAX_OVERHEAD unset). A segment written with that variable set, as builds from before v0.1.0 did whenever it was set, needs bitsliced-codec. The release binaries are built without either feature, except that weft-bench enables experimental-codecs.

Feature	Crate(s)	Effect
bitsliced-codec	weft-physical-type, forwarded by weftdb and weft-server	Compiles in the opt-in bit-sliced value codec, so WEFT_SEGMENT_TRANSPOSED_MAX_OVERHEAD takes effect. Without it, a segment written with the codec fails to read with an error naming the feature.
experimental-codecs	weft-physical-type (enabled by weft-bench)	Advisory codecs never written to disk: Gorilla-XOR, Chimp, Chimp128 and Elf for f64, and the Sprintz FIRE timestamp forecaster. Benchmark what-ifs only.

---

Rust library

WeftDB can be used directly as a set of libraries. They are not on crates.io yet (v0.1.0 ships binaries only), so depend on them from this repository, pinned to a release tag. splimes, the interpolation engine, is on crates.io:

[dependencies]
# the database: capture, query, interpolate
weftdb = { git = "https://github.com/basic-automation/weftdb", tag = "v0.1.0" }
# the analytics pipeline
weft-orchestration = { git = "https://github.com/basic-automation/weftdb", tag = "v0.1.0" }
# the interpolation engine on its own
splimes = "1"

Crate	Depend on it when you want
weftdb	The database — capture measurements, query and interpolate stored series.
splimes	Spline interpolation over irregular series, standalone. No database.
weft-orchestration	The batching → patterns → events → correlation → signals pipeline.
weft-physical-type	Declared numeric encodings and the .weftseg columnar format.
weft-reduce	Downsampling reductions over an epoch-aligned bucket grid.
weft-line-protocol	To parse InfluxDB Line Protocol without an InfluxDB client.
weft-arrow	Segments as Apache Arrow RecordBatch, Arrow IPC or Parquet.
weft-arrow-store	A stored range read straight into an Arrow RecordBatch.

weft-server, weft-tui and weft-bench are binaries rather than libraries, so they are not published to crates.io — take them from the archives on a GitHub release or build them from this repository.

The examples below are illustrative — run cargo doc --open for the authoritative, version-matched API.

Capture Measurements

use weftdb::{Database, DatasetId, InputMeasurement, Resolution};
use bigdecimal::BigDecimal;
use std::str::FromStr;
use chrono::{TimeZone, Utc};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let db = Database::new("my_sensors").await?;
    let sensor = db.observe_subject("temperature_sensor_001").await?;
    let temp = db.track_aspect(&sensor.id(), "ambient_temp", &Resolution::Seconds, None).await?;

    let measurements = vec![
        InputMeasurement::new(Utc.with_ymd_and_hms(2024, 1, 1, 12, 0, 0).unwrap(), BigDecimal::from_str("22.5")?),
        InputMeasurement::new(Utc.with_ymd_and_hms(2024, 1, 1, 12, 1, 0).unwrap(), BigDecimal::from_str("22.8")?),
    ];
    db.batch_capture_measurements(temp.id(), DatasetId::new(), measurements).await?;
    Ok(())
}

Query & Interpolate

use weftdb::{Database, Resolution, Spline};
use chrono::{TimeZone, Utc};
use futures::StreamExt;

let db = Database::existing("my_sensors").await?;

// Interpolated point at a specific time:
let point = db.analyze_point(&aspect_id, query_time, &Resolution::Seconds, &Spline::Linear).await?;

// Streamed, interpolated range:
let mut stream = db.analyze_range(&aspect_id, start, end, Resolution::Seconds, Spline::Linear).await?;
while let Some(result) = stream.next().await {
    let p = result?;
    println!("{}: {}", p.timestamp, p.value);
}

Run an Analytics Pipeline

use weft_orchestration::Pipeline;
use weftdb::Database;
use splimes::Spline;

let database = Database::existing("PlantTelemetry").await?;
// ... resolve `aspect_id` for the "outlet_pressure" aspect of pump-station-3 ...

let mut pipeline = Pipeline::builder(database.clone(), aspect_id)
    .spline_method(Spline::Linear)
    .batch_size(24)                              // 24-hour batches
    .add_dictionary("daily", "Daily patterns", constraints)
    .with_monthly_increase_detector(0.05)        // 5% monthly increase
    .with_peak_detector("Pressure Peaks")
    .build()
    .await?;

pipeline.run().await?;                            // full pipeline (incremental)
println!("Signals generated: {}", pipeline.signals().len());

For finer control, the individual steps — prepare_data, extract_patterns, detect_events, correlate_events, generate_signals — can be called directly.

---

Terminal UI

weft-tui is the interactive way to explore WeftDB (the HTTP server is the primary commercial surface; the TUI serves demos and debugging). Launch it with cargo run --release -p weft-tui. It is a stateful, keyboard-driven application that walks you through:

• Database / Subject / Aspect creation — with name validation and resolution selection (+/- to step through resolutions from nanoseconds to years).
• CSV import — point it at a CSV file and watch live loading progress (rows loaded / scanned).
• Browsing — select an existing database, subject, and aspect.
• Plotting — visualize an aspect's series directly in the terminal.
• Compression — choose a mode (time-based, size-based, or combined), configure tiers/targets/aggressiveness in a form, and monitor a live compression dashboard (phase, tier progress, aggressiveness, compression ratio) plus last-run stats.

Logs are written to weft-tui.log (see WEFT_DATA_DIR), and the last 50 log lines are also shown in-app.

---

How it works

The parts below are reference material — you do not need them to use WeftDB.

Architecture

┌──────────────────────────────┐  ┌──────────────────────────────────────┐
│           weft-tui            │  │              weft-server               │
│  Ratatui terminal UI (app)   │  │  axum HTTP API: interpolate ·         │
│  create DB · import · plot   │  │  downsample · storage ingest/query ·  │
│  · compress                  │  │  catalog · /metrics · /debug/profile  │
└──────────────┬───────────────┘  └──────┬───────────────────┬───────────┘
               │ uses                     │ uses              │ columnar I/O
               ▼                          ▼                   ▼
┌──────────────────────────────┐   ┌────────────────────────────────────┐
│    weft-orchestration    │   │   weft-arrow / weft-arrow-store      │
│  batch → patterns → events   │   │   Arrow IPC + Parquet interchange  │
│  → correlations → signals    │   └──────────────────┬─────────────────┘
└───────────────┬──────────────┘                      │
                ▼                                     ▼
┌─────────────────────────────────────────────────────────────────────┐
│                              database                                │
│   Turso/libSQL control plane (catalog · metadata · segment index)   │
│   + typed columnar `.weftseg` segment store (measurement hot path)   │
│   Subject/Aspect/Measurement · Patterns · Events · Compression      │
└───────────────┬─────────────────────────────────────┬───────────────┘
                │ physical types + segments           │ numerics
                ▼                                     ▼
┌──────────────────────────────┐   ┌──────────────────────────────────┐
│      weft-physical-type       │   │             splimes              │
│  encodings · schemas ·       │   │   Spline interpolation engine    │
│  timestamp codecs · .weftseg  │   │  CPU · Rayon · GPU (calibrated)  │
└──────────────────────────────┘   └──────────────────────────────────┘

weft-bench drives the whole stack from outside through the same public APIs a customer would use; weft-line-protocol is the shared ILP dialect between the harness and the server. Dependencies flow downward; the arrow-* tree lives only in the weft-arrow* leaf crates.

---

Data Model

WeftDB organizes data hierarchically. Each level maps to a directory or database file on disk, so an aspect's storage is fully self-contained.

Concept	Description	Example
Database	Top-level container for all data.	Crypto, my_sensors
Subject	A logical entity being observed.	pump-station-3, temperature_sensor_001
Aspect	A specific measurement type of a subject, with its own resolution, declared physical encoding, and (optional) compression config.	open, ambient_temp
Measurement	A timestamped, arbitrary-precision value.	(2024-01-01T12:00Z, 22.5)

On-Disk Layout

{data_dir}/
└── {database_name}/
    ├── metadata.db                  # database metadata
    └── {subject_name}/
        └── {aspect_name}/
            ├── measurements.db        # raw measurements
            ├── unprocessed_batches.db # batching queue
            ├── processed_batches.db
            ├── patterns.db            # extracted patterns
            ├── events.db              # detected events
            ├── correlations.db        # pattern↔event links
            ├── pipeline.db            # persisted pipeline config & state
            └── dictionaries/
                └── {dictionary_name}.db

The Storage v2 segment store (see Physical Types & Columnar Storage) adds, per store root: catalog.db (database/subject/aspect hierarchy + declared schemas), per-aspect metadata.db rollups, segment_index.db (the prune-before-open segment index), and segments/*.weftseg (the typed columnar measurement files).

Higher-level analytics types build on this foundation:

Type	Purpose
Batch	A group of measurements processed together.
Pattern	An extracted recurring shape, with the occurrences where it appears.
Dictionary	A collection of patterns sharing similarity constraints.
Event	A detected occurrence with one or more manifestations (time spans).
Correlation	A link between a pattern and an event.
Signal	A probability-weighted prediction derived from a correlation.

---

Interpolation Engine

splimes is the numerical heart of WeftDB. Given a set of Points and a target time range + resolution, it produces an interpolated series using one of four spline methods, choosing the execution backend automatically, and labels every output point Raw, Interpolated or Extrapolated (PointKind). It interpolates and extrapolates — single-instant lookups and full ranges — on the CPU (one thread or rayon's pool) and the GPU. WeftDB depends on splimes 1.0; its own migration guide lists what changed from 0.1.

Spline Methods

Method	Min. points	Notes
Linear	2	Straight-line interpolation.
Quadratic	3	Second-degree.
Cubic	4	Smooth third-degree splines.
Polynomial(degree, bounds_factor)	degree + 1	Degree 1–8, with optional bounds damping of the extrapolation.

If a method needs more distinct points than are available, splimes steps down automatically (Cubic → Quadratic → Linear; Polynomial(d) → Polynomial(n − 1)), never upgrading beyond what you requested, and reports the method it used.

Backend Selection

Every call is synchronous; Backend::Auto (the default) picks the backend by output grid size, using AutoThresholds:

Grid size	Backend
below parallel_min_points (default 64 Ki)	CPU, the calling thread
from parallel_min_points	Parallel, rayon's pool
from gpu_min_points (default never), once the GPU has been started	GPU (wgpu compute shaders), falling back to rayon if it fails

Auto never starts the GPU itself. splimes::calibrate() starts it, times every backend on this machine and sets the thresholds to the measured crossovers; weft-server runs it once in the background at startup, skipping a CPU/software adapter (WEFT_GPU_CALIBRATE=0 skips it, force calibrates any adapter). Otherwise:

• splimes::prewarm_gpu() starts the GPU and returns its GpuInfo (adapter, API, device type, driver, f64 support); then set thresholds with splimes::set_auto_thresholds.
• splimes::configure_gpu(GpuConfig) — before the GPU's first use — sizes its buffer pool and per-dispatch chunk (max_pool_bytes, chunk_points, low_power; presets GpuConfig::minimal(), low_memory(), DEFAULT, high_performance()). Calling it after the GPU has started is an Error::GpuAlreadyConfigured, never a silent no-op; splimes::gpu_config() reports what is in force.
• splimes::gpu_pool_stats() returns the buffer pool's counters once the GPU is up.
• The GPU computes in f64 where the adapter supports it; f32 must be asked for (Interpolator::gpu_precision(Precision::F32)).

use splimes::{Interpolator, Resolution, Spline};

// Backend::Auto picks the backend by grid size; from async code, run it off the
// executor (`run_async` needs the `tokio` feature):
let series = Interpolator::new(Spline::Cubic, Resolution::Seconds).run_async(points, start, end).await?;
for (timestamp, value, kind) in series.iter() {
    // kind: PointKind::Raw | Interpolated | Extrapolated
}

---

Physical Types & Columnar Storage

WeftDB's storage hot path is its own typed columnar format; Turso/libSQL serves as the control plane (catalog, metadata, segment index, pipeline state) and never holds bulk measurements. BigDecimal remains the logical/API type everywhere.

Physical type system (weft-physical-type)

• Six numeric encodings — F64, F32, ScaledI64, ScaledI128, Decimal128, BigDecimalText — each declaring storage width, hot-path/GPU eligibility, and conversion behavior. Every encode reports an explicit Exactness (exact / lossy-with-measured-residual); to_logical is always available.
• No silent downcast, enforced — AspectSchema declares, per aspect, the physical encoding, a permitted per-value error bound, and the timestamp unit. Sealing a segment under a declared schema errors (SealError::Encode / SealError::ToleranceExceeded) rather than losing precision beyond the declared bound. Over HTTP this surfaces as a 400 — never a downcast.
• Advisory encoding selection — recommend_encoding picks the narrowest safe encoding within a tolerance, and columnar estimators report expected bytes/point (the north-star cost term) before anything is written.
• Timestamp codecs — integer-epoch timestamps (TimeUnit = seconds/millis/micros/nanos) with lossless delta and delta-of-delta transforms and five interchangeable second-difference codecs — per-value zig-zag + LEB128 varint, RLE, fixed-width bit-packing, a Gorilla-style variable-length codec, and per-block adaptive (dynamic) bit-packing — chosen per column by a smallest-wins selector. Each wins a different regime: bit-packing on regular/small-jitter series, RLE on long constant runs, Gorilla on scattered single jitter (isolated moderate spikes among regular intervals, where RLE cannot form runs and bit-packing must widen every value), and per-block adaptive bit-packing on a mixed-magnitude stream (a contiguous wide region among narrow runs, where a single global width overpays — 184 B vs 384–961 B for the other codecs on a 256-value mixed corpus). A regular 1000-point column packs to ~12 bytes total.
• Value-column codecs — a value column is stored under its schema-declared physical encoding, and a ScaledI64 column additionally chooses, via a self-describing selector byte, between a per-value zig-zag varint, fixed-width bit-packing of its mantissas, per-block adaptive (blocked) bit-packing, and per-block Frame-of-Reference (FOR) packing — bit-packing a regular scaled series (e.g. a 2-decimal price ramp) to 37% below the varint, the blocked codec confining a wide burst to the blocks it spans on a mixed-magnitude column, and FOR (subtract each block's minimum, pack the unsigned residual to the block's range — the standard FastLanes/ALP move) collapsing values clustered at a high base (a sensor reading near a fixed offset: mantissas near 1e9 store 88% below the bit-packed alternative). Each codec is chosen only when strictly smallest, so a regular column is byte-for-byte unchanged; the realized figure is reported as StorageEstimate.realized_value_bytes / value_codec. The timestamp column keeps an advisory FOR estimate (for_estimated_bytes) — second differences are near-zero, so FOR rarely wins there. Every linear bit-packed codec decodes with a word-wise field read (one unaligned load and a shift/mask per value), so 1 Mi mantissas unpack in 1.09 ms (fixed width) / 1.83 ms (per-block) (weft-physical-type/benches/bitunpack.rs), and 1 Mi real BTC closes stored under FOR at scale 8 decode to floats in 1.66 ms (weft-physical-type/benches/alp_vs_f64_codecs.rs).
• Two-level delta cascade (opt-in) — a trending ScaledI64 column (a counter or monotone sensor whose magnitude every single-level codec pays for) is additionally compressible by a cascade: delta-transform the mantissas, then pack the differences with the smallest inner packer (varint / bit-pack / blocked / FOR / RLE). It is realized on disk as the VAL_CODEC_DELTA_CASCADE codec-chain descriptor (write_value_column_cascading, read back by the ordinary reader) and surfaced advisory-first in the bench (StorageEstimate.advisory_delta_cascade_value_bytes, schema v15). It is opt-in — the cascade beats even FOR broadly, so folding it into the default selector is a headline change held for owner sign-off; the default codec choice is unchanged.
• Bit-sliced (transposed) value codec (opt-in, bitsliced-codec build feature) — VAL_CODEC_TRANSPOSED stores a ScaledI64 column's mantissas bit-plane-major in 1024-value tiles, so the decoder rebuilds eight lanes per plane byte with branch-free table spreads and a tile's width header drops the empty high bit-planes of a small-magnitude column. The decode kernel is checked against an external yardstick, the published fastlanes crate, at equal bit width (weft-physical-type/benches/fastlanes_yardstick.rs, 1 Mi values): WeftDB's decoder takes 0.78 / 1.50 / 2.32 ms at widths 3 / 10 / 20 against fastlanes' 0.33 ms — still 2.4–7× behind, and that gap is an open roadmap item, not a design choice. It carries its own size function (transposed_value_bytes) and selector entry (best_value_codec_transposed(max_overhead)) rather than reusing the blocked figure, is random-access capable (so it does not regress the streaming point read), and is requested through FrameOptions / Segment::write_to_with or, from a deployment, WEFT_SEGMENT_TRANSPOSED_MAX_OVERHEAD. Honest result: no read-path win over the linear codec. Measured on a 1M-row zero-straddling column (weft-physical-type/benches/transposed_read.rs, 2026-10-08, transposed vs linear): frame bytes +0.08% (1,251,057 vs 1,250,076), full decode 7.81 ms vs 6.87 ms; with the slicing-by-16 frame CRC, windowed 1000-row range 638 µs vs 632 µs and single point read 820 µs vs 805 µs. So the layout is byte-neutral and no faster (slightly slower) on reads; it stays opt-in and the default codec choice is unchanged. Note the codec is chosen only when a strict size win or within the caller's overhead ceiling — the ceiling may legitimately be set below 1.0, because per-tile widths with one header per 1024 values can beat both a global width and the blocked codec's one-header-per-64. It is a bit-sliced layout, not the FastLanes layout (which keeps each value's bits together), and it is compiled only with the bitsliced-codec feature, pending patent review; without it the writer never selects it and reading a segment that uses it fails with an error naming the feature.
• Block-level random access — the fixed-layout value codecs support decoding a single value (or a sub-range) without materializing the whole column: weftseg::read_value_at(bytes, i) reads only the block covering row i (skipping earlier blocks by their headers) for the blocked/FOR codecs, reads bit i * width directly for the fixed-width bit-pack codec, and decodes only the covering tile for the bit-sliced codec — the point-lookup / late-materialization lever. weftseg::read_value_range(bytes, start, len) is its windowed sibling: every fixed-layout codec locates a value by walking its block/tile headers from the start of the stream, so resolving a window one value at a time re-walks that chain per row. The range read does the decode once, which is what keeps a windowed read on the 1024-value-tile bit-sliced layout from costing a whole tile decode per row. weftseg::read_segment_point(bytes, t) wires this up to the framed single-block segment — it skips the value block by its framing, decodes only the timestamps to find the row, and unpacks the one covering value block — so a point lookup never materializes the value column on a per-block codec (equal to a full decode + value_at for every frame; the non-block codecs fall back to that). read_paged_segment_point does the same for a paged frame, first pruning pages on their indexed min/max timestamp so only the surviving page is touched. A regular (constant-stride) timestamp column is resolved in closed form — ts[i] = first + i*step, so the row for an instant is O(1) with no timestamp materialization at all. Measured ~46× faster point lookup on a 100k-row single-block FOR segment (56 µs vs 2.60 ms) and ~8.5× faster on the paged frame (33 µs vs 279 µs), with the closed-form timestamp path a further ~3.4× over an irregular column (56 µs vs 188 µs), identical bytes on disk (weft-physical-type/benches/pointread.rs).
• Checkpointed frames — sublinear point lookup on an irregular sorted column (opt-in; write_segment_checkpointed / write_paged_segment_checkpointed). The closed form above only serves a regular column; an irregular one has no closed form, so a probe had to decode the whole delta-of-delta timestamp stream. A checkpointed frame persists a sparse (row, timestamp, delta) index every stride rows next to a range-decodable per-block dod stream, so a probe binary-searches the index and decodes only the blocks it walks — the timestamp column is never materialized. Measured ~3.45× faster point reads on a 100k-row sorted-irregular frame (352.94 ms → 102.42 ms for 64 probes) for +0.41% frame bytes at stride 1024; a paged frame gains only ~1.14× (68.70 → 60.41 ms), because page pruning already bounds its decode to rows_per_page (weft-physical-type/benches/dodsearch.rs). The codec tag is additive — every previously written frame still reads, with no format-version bump — and a checkpointed frame is byte-for-byte equivalent in behaviour to a plain one on every read. Opt in by setting WEFT_SEGMENT_CHECKPOINT_STRIDE (see the configuration table). The seal path then writes the index only where it genuinely pays, on three counts: the column is sorted and irregular (a regular one already resolves O(1) closed-form, an out-of-order one cannot be binary-searched), it has at least WEFT_SEGMENT_CHECKPOINT_MIN_ROWS rows, and the codec override is affordable — a checkpointed frame must use the range-decodable per-block codec, which costs ~3.5× on a Gorilla-shaped column and ~14× on an RLE-shaped one, so those are refused rather than bloated (WEFT_SEGMENT_CHECKPOINT_MAX_CODEC_OVERHEAD). It is off by default, so an unconfigured store's bytes/point are unchanged; making it the default is an open roadmap decision.
• Realized headline bytes/point — every headline bytes/point figure (Segment::bytes_per_point, StorageEstimate.bytes_per_point / total_bytes_per_point, the bench HTML val B/pt) reports the codec actually written to disk, not a fixed-width estimate (bench schema v10). The naive len * width figure is retained beside it (Segment::logical_value_bytes, StorageEstimate.estimated_value_bytes) as the uncompressed baseline, so realized / logical reads directly as the storage compression ratio.

Segment store (.weftseg + weftdb::SegmentStore)

• Sealed, checksummed segments — a Segment binds a typed value column and a delta-of-delta timestamp column with min/max timestamp/value stats, row/null counts, and a format version. The on-disk frame is hand-rolled, versioned, and CRC-32-verified before parse, so a corrupt or truncated file fails fast.
• Multi-codec timestamps — the timestamp block writes its second-difference stream under whichever of five codecs is smallest — fixed-width bit-packing, per-value zig-zag varint, RLE, Gorilla variable-length, or per-block adaptive (dynamic) bit-packing — chosen by a self-describing selector byte the reader dispatches on, routed through the single source of truth so the reported codec name always matches the bytes on disk. A regular series packs to a handful of bytes on disk (a 1000-point regular block stores in <20 bytes), so the bytes/point saving is realized, not just estimated. The value block carries its own codec selector too, so a ScaledI64 column stores its mantissas under whichever of varint, fixed-width bit-packing, per-block adaptive (blocked) bit-packing, or per-block Frame-of-Reference (FOR) packing is smallest on disk.
• Order semantics — every segment records whether its timestamps are monotonic (time_sorted). Ingest can enforce order (require_sorted rejects an out-of-order batch rather than sealing it), and the per-aspect/store rollups carry an unsorted_segments count so out-of-order data is visible before it costs a point-lookup scan.
• Order-signal read planner — a single-instant point lookup (SegmentStore::read_point) prunes the index to the segments spanning the instant and resolves each with its persisted time_sorted flag: a sorted segment is binary-searched, an out-of-order one linear-scanned (the only sound search on unsorted timestamps). Both frame kinds resolve through the streaming point read (weftseg::read_segment_point / read_paged_segment_point, below) — no value-column materialization on a per-block codec, and a paged frame prunes pages on their indexed min/max timestamp without decoding a column byte before touching the one surviving page. SegmentStore::read_points resolves a batch of instants in one pass — the index is pruned once and each segment's timestamp column decoded once for the whole batch, so N instants sharing a segment cost one decode, not N (~54× faster for 64 instants on a 100k-row FOR frame than 64 single lookups — 67 µs vs 3.57 ms, weft-physical-type/benches/pointread.rs).
• Intra-segment reconciliation — SegmentStore::reconcile_segment/reconcile_aspect rewrite an out-of-order segment into a sorted one in place (stable sort by timestamp, re-sealed at the same id, frame kind preserved), so it drops out of the unsorted_segments count and its point lookups binary-search.
• Threshold-triggered reconciliation — the unsorted_segments backlog drives a QuestDB-style trigger so the rewrite is paid only once out-of-order data is worth compacting, never on every late row: reconcile_aspect_if_unsorted_exceeds gates a single-aspect pass on the backlog, reconcile_all_over_threshold sweeps every declared aspect over the threshold, and a background timer daemon (WEFT_RECONCILE_INTERVAL_SECS / WEFT_RECONCILE_THRESHOLD) runs the sweep unattended. A threshold of 0 clamps to 1 (any out-of-order segment). Passes and the segments they rewrite are counted in weft_reconcile_* metrics.
• Hot/cold reconciliation — reconcile_aspect_hot_cold/reconcile_all_hot_cold reconcile every cold (sealed) out-of-order segment on each pass but defer the hot tail (the most-recently-sealed segment) until the backlog reaches the threshold, so the actively-appended segment is not rewritten on every tick — QuestDB's "squash non-active partitions each commit, defer the active one" policy. Enabled on the daemon with WEFT_RECONCILE_HOT_COLD and on the reconcile endpoints with ?hot_cold=true.
• Cross-segment overlap signal — a sorted segment whose time window a later-sealed segment re-enters (late data landing in an already-covered window) is not counted by unsorted_segments; SegmentIndex::overlapping_count reports these time-overlapping segments as a distinct overlapping_segments order-health count in the per-aspect and store-wide rollups.
• Cross-segment overlap merge — SegmentStore::reconcile_overlaps/reconcile_all_overlaps collapse each connected group of time-overlapping segments into one time-sorted segment, folding members oldest→newest with newer-wins (last-writer-wins / upsert) dedup on shared timestamps — the same answer a point lookup already gives across overlaps, and QuestDB's DEDUP UPSERT "last write wins on the designated timestamp" semantics. Drives overlapping_segments to zero; exposed on the reconcile endpoints with ?overlaps=true and as a background daemon sweep (WEFT_RECONCILE_OVERLAPS).
• Split-not-rewrite merge — SegmentStore::split_segment carves a sorted segment at a boundary into a cold prefix (kept id) + hot suffix (new id), and reconcile_overlaps_with_policy consults a size-based SplitPolicy: when an overlap component's cold prefix clears a byte floor and outweighs its hot suffix, only the suffix is merged and the large cold prefix is left untouched, so later late arrivals re-entering the hot window never rewrite it — QuestDB's partition-split write-amp bound. The default reconcile_overlaps keeps QuestDB's 50 MiB floor (small components full-rewrite); a floor is selectable with ?split_min_bytes= on the per-aspect and store-wide reconcile endpoints and WEFT_RECONCILE_SPLIT_MIN_BYTES on the daemon.
• Squash — SegmentStore::squash_aspect/squash_aspect_if_exceeds/ squash_all_over_threshold fold an aspect's segments back into one (newer-wins), bounding the fragmentation repeated split carve-offs create — QuestDB's max.splits squash trigger. Exposed as POST …/{aspect}/squash?max_segments= and on the daemon with WEFT_RECONCILE_MAX_SPLITS (squash aspects over the cap each tick, after the overlap merge that accumulates the splits).
• Null/quality column — a per-row presence bitmap (zero bytes for fully dense columns) makes gaps real: present values are stored densely and reconstructed as None on read.
• Paged segments — rows partition into fixed-height pages, each independently encoded with its own stats, behind a per-page index table: a reader prunes pages on min/max time/value without touching a column byte and seeks straight to the pages it needs.
• Data skipping — segment-level and page-level pruning by time, by value band, and by quality (segments/pages that overlap a window but hold only nulls there are skipped). A range read over a regular (constant-stride) block-coded segment goes further — read_segment_range computes the row window in closed form and unpacks only the present values inside it, so a selective range decodes ~window values, not the whole segment (~47× faster for a 100-row window over a 100k-row FOR frame — 58 µs vs 2.73 ms, weft-physical-type/benches/pointread.rs).
• Control plane — SegmentIndexStore persists one descriptor per sealed segment in libSQL and answers range queries with SQL pruning; CatalogStore/AspectCatalog register the database → subject → aspect hierarchy with declared schemas (declare once, seal by lookup — an undeclared aspect is refused, not guessed); per-aspect metadata.db rollups make aspect-wide stats (including realized bytes/point) an O(1) read, always re-derivable from the segment index.

Arrow & Parquet interchange (weft-arrow, weft-arrow-store)

• Lossless Arrow batches — sealed segments and stored reads convert to/from Apache Arrow RecordBatches: a lossless text form, plus a typed fast path (F64 → Float64, F32 → Float32, scaled integers → exact Decimal128). Paged segments stream one batch per page.
• Wire formats — Arrow IPC stream bytes and Parquet files (embedding the Arrow schema so WeftDB's time-unit/encoding metadata survives), both directions: export a stored range, or ingest a Parquet file into a declared aspect under the full no-silent-downcast guarantee.
• Lean core preserved — the arrow-*/parquet dependency tree lives only in these leaf crates; splimes/weftdb/weft-physical-type stay arrow-free.

---

The Analytics Pipeline

The weft-orchestration crate provides a fluent Pipeline API. Each aspect has exactly one pipeline, persisted to its pipeline.db, so runs are incremental and conflict-free. A pipeline can host multiple dictionaries and multiple detectors.

 Measurements ─▶ Batches ─▶ Processed Batches
                                   │
                                   ▼
                          Dictionaries (1..N)        Event Detectors (1..N)
                          extract patterns           peaks · valleys ·
                                   │                  thresholds · monthly · …
                                   ▼                          │
                               Patterns ───── correlate ──────┤
                                                              ▼
                                  Events ─▶ Correlations ─▶ Signals
                                                              │
                                                              ▼
                                          query_probability(event, time)

Built-in event detectors (in weft_orchestration::detectors):

Detector	Description
detect_monthly_increase	Months where the value rises ≥ a threshold start-to-end.
detect_peaks / detect_all_peaks	Global / all local maxima.
detect_valleys / detect_all_valleys	Global / all local minima.
detect_threshold_crossing_up / _down	Upward / downward threshold crossings.
detect_drawdown	Significant value drops.

You can also register custom detectors with the event_detector_fn! macro, and run pipelines across many aspects in parallel with run_all_pipelines / run_subject_pipelines.

---

Dataset Compression

Long-running deployments accumulate huge volumes of raw measurements. The compression module (weftdb::compression) reduces storage while preserving the shape of the data, configured per-aspect via CompressionConfig:

• Time-based — recent data within a "pure" window (e.g. the last 7 years) stays uncompressed; older data is progressively compressed across configurable tiers with Linear, Exponential, or Custom aggressiveness scaling.
• Size-based — compress until total storage fits within a target (e.g. 10 GB), hitting the oldest data hardest. Takes precedence over time-based.

Compression works by interpolating to a coarser resolution and applying slope-change simplification to drop redundant points. Newly inserted data inside a compressed range marks a dirty region for targeted re-compression. Progress is reported live (phase, tier, aggressiveness, time range) — surfaced in the TUI.

use weftdb::{CompressionConfig, TimeBasedCompressionConfig};

let config = CompressionConfig::time_based(
    TimeBasedCompressionConfig::default_seven_years()
);
let aspect = db.track_aspect(&subject_id, "temperature", &Resolution::Hours, Some(config)).await?;
let summary = aspect.compress(&db).await?;

---

Benchmarking

Weft-Bench (weft-bench) is WeftDB's reproducible, correctness-gated benchmark harness — both an internal engineering suite and a public, customer-runnable diagnostic. Every number it emits ships with the dataset seed, the workload profile, and a correctness verdict.

Policy: WeftDB benchmarks guide real engineering decisions, not synthetic wins. A latency number is only publishable when its correctness check passes.

What it does today:

• Seeded workload profiles — the flagship interpolation-heavy-irregular profile generates an irregularly-spaced, gap-containing series (ChaCha8Rng, published seed → byte-for-byte reproducible). The underlying analytic signal is shape-selectable (MultiSine | Sawtooth | Step | DampedSine) and doubles as the accuracy ground truth.
• Storage, read & aggregation workloads — beside the flagship interpolation run, four parallel workloads exercise WeftDB's own hot paths, each with its own correctness gate and p50/p95/p99 latency: point_lookup (--point-lookup) seals a .weftseg segment and times the streaming point read (read_segment_point/read_segment_points, single or batch, single-block or paged via --pl-rows-per-page, regular closed-form vs irregular), gated against the full-decode value — also over a real corpus with --pl-csv (1M real BTC closes: a 128-instant batch at p50 2.43 ms single-block vs 0.90 ms paged at 8,192 rows/page, weft-bench --point-lookup --pl-csv database/datasets/btc_1min.csv --csv-value-col 4 --csv-skip 3000000 --pl-rows 1000000 --pl-queries 128 [--pl-rows-per-page 8192] --reps 50); range_fetch (--range-fetch, real corpus via --rf-csv: 32 × 100-row windows over 1M real BTC closes at p50 60.8 ms single-block vs 17.8 ms paged) times the windowed range read; compression (--compression) reports realized bytes/point, the value-column compression ratio, and decode throughput, gated on an exact round-trip — over a seeded shape, or a real corpus with --comp-csv <FILE> (--csv-value-col, --csv-skip; values read as exact decimal text; --pl-csv does the same for point_lookup). On 1M real BTC/USD one-minute closes (rows 3M onward of btc_1min.csv) it measures 4.22 B/point realized (scaled_for, exact), full-decode 20.8M points/sec (p50 52.7 ms, 5 reps), and an advisory decimal-exponent FOR footprint of 1.70 B/point (weft-bench --compression --comp-csv database/datasets/btc_1min.csv --csv-value-col 4 --csv-skip 3000000 --comp-rows 1000000 --reps 5); and downsample (--downsample) times WeftDB's canonical weft-reduce reduction into grid-aligned buckets with a --ds-aggs selector over min/max/avg/sum/first/last/p50…p99/twa/twa_linear/twa_bucket_end/sketch_p50…sketch_p99. --ds-parallel <N> reduces in N chunks via mergeable partial reductions (identical buckets to serial, asserted by test) — measured 14.7× at 64 chunks on a quiet 16-core box (441.2 ms → 30.1 ms, 1,134,659 → 16,921,104 points/sec, --ds-aggs sketch_p99, 500k points, 5 reps, correctness PASS); a re-run on 2026-10-08 at load average ~19 measured 327.3 → 48.5 ms (6.7×). The real-corpus figures in this bullet come from btc_1min.csv, a local file not distributed with the repository (see Ingest); point --pl-csv / --rf-csv / --comp-csv / --ds-csv at a CSV of your own to run the same workloads on real data. The underlying point/range read speedups are quantified at the codec layer in weft-physical-type/benches/pointread.rs.
• Vendor-neutral adapters — every system is driven through the SystemAdapter trait: the WeftDB reference adapter (splimes' Interpolator on Backend::Auto), a precision-aware portable linear baseline (fair-protocol class C), and a forward-fill/LOCF baseline (class B — the in-process mirror of FILL(previous) / locf()).
• Quality beside speed — reconstructions are scored against the analytic ground truth: RMSE / MAE / max-error / bias. Honest finding, surfaced not hidden: on smooth multisine (and step) WeftDB's cubic leads on RMSE, but on sawtooth the portable linear baseline out-accuracies the cubic, which overshoots sharp discontinuities.
• Fair-protocol statistics — p50/p95/p99 + min/max/mean/stddev with seeded bootstrap confidence intervals, and an end-to-end timing breakdown proving the latency distribution times only the adapter call, never setup.
• Storage estimate — each result carries a StorageEstimate (recommended encoding + value/timestamp/total bytes/point) computed by weft-physical-type over the actual columns. It also surfaces advisory potential-saving estimates for codecs not yet realized on disk: the best-of f64 value codec (Gorilla / Chimp / Chimp128 XOR, advisory_best_f64_bytes / _codec) for a lossy F64 column, and the Sprintz FIRE forecaster's footprint on the timestamp column (advisory_fire_timestamp_bytes), and the decimal-exponent FOR value codec (advisory_dfor_value_bytes, schema v17, present only when it beats the realized codec — e.g. a ScaledI64 column whose scale is forced by a few high-precision values; real BTC closes measure 13.58 vs 33.74 bits/value in weft-physical-type/benches/alp_vs_f64_codecs.rs), plus two timestamp-column what-ifs (schema v18), each present only when it beats the realized codec: advisory_common_multiple_timestamp_bytes (the GCD of the deltas factored out — millisecond-precise instants stored in microseconds measure 20 → 10 bits/value in weft-physical-type/benches/timestamp_vs_pco.rs) and advisory_delta_for_timestamp_bytes (first-order deltas under per-block FOR, for independent random intervals: 20.0 → 19.4 bits/value) — each a what-if number the adopt-or-drop decision reads, never a realized headline claim. The f64 codecs and FIRE live behind weft-physical-type's experimental-codecs feature, which only weft-bench enables.
• Reports — a BenchReport JSON artifact (run metadata + a best-effort hardware probe: CPU model, cores, RAM, and the kind/file system/mount of the disk under the working directory) under reports/json/, plus a self-contained HTML view (--html) with the most-accurate row highlighted.
• The engine the server runs — an interpolation run first calls splimes::calibrate() once, as weft-server does at startup (skipping a CPU/software adapter the same way), so Backend::Auto uses rayon and the GPU where they are faster on this machine. The calibration, GPU and thresholds are printed and recorded in the report's metadata.engine (schema v16); --no-gpu-calibrate skips it and records splimes' defaults. There is no counterpart to the server's WEFT_GPU_CALIBRATE=force: on a software adapter the bench always skips calibration, so there it cannot reproduce a server started with force.
• ILP / TSBS input — a .lp / TSBS file drives the same harness, correctness gate, and reporting via the shared weft-line-protocol parser.

# Benchmark a line-protocol file, WeftDB vs the portable baselines, JSON + HTML report:
cargo run -p weft-bench -- \
    --input data.lp --field usage --spline cubic --resolution minutes --compare --html

# Seeded synthetic mode — the only mode with a known ground truth, so it also scores accuracy:
cargo run -p weft-bench -- \
    --synthetic --points 300 --noise 0 --shape sawtooth \
    --spline cubic --resolution minutes --reps 5 --compare

# Storage/read/aggregation workloads over WeftDB's own hot paths:
cargo run -p weft-bench -- --point-lookup --pl-rows 100000 --pl-queries 128 --reps 50
cargo run -p weft-bench -- --range-fetch --rf-window 100 --rf-windows 32 --reps 30
cargo run -p weft-bench -- --compression --comp-shape clustered --reps 20
cargo run --release -p weft-bench -- --compression --comp-csv database/datasets/btc_1min.csv --csv-value-col 4 --csv-skip 3000000 --comp-rows 1000000 --reps 5
cargo run -p weft-bench -- --downsample --ds-bucket m --ds-aggs min,max,p99,twa --reps 20

The process exits non-zero when any result's correctness gate fails, so scripted callers can gate on it. External-engine competitor adapters (DuckDB, ClickHouse, InfluxDB 3, QuestDB, TimescaleDB) are the next tracked step in ROADMAP.md.

---

Development

# Run the full test suite:
cargo test

# Skip long-running tests:
SKIP_SLOW_TESTS=1 cargo test          # PowerShell: $env:SKIP_SLOW_TESTS=1; cargo test

# Test a single crate:
cargo test -p weftdb

The workspace ships extensive Criterion benchmark suites alongside the Weft-Bench harness (see Benchmark Harness):

Crate	Benchmarks
weftdb	interpolation_benchmarks, integration_interpolation_benchmarks, optimized_interpolation_benchmarks, strategy_selection_benchmarks, new_api_benchmarks, cache_performance_benchmarks

cargo bench -p weftdb --bench cache_performance_benchmarks

These benches create their databases in a temporary data dir of their own, which they remove when they finish, never in ~/.weftdb/data. That dir is under the system temp dir, often a tmpfs; to bench another filesystem, set TEST_DATA_DIR to a directory on it (the benches remove the databases they create there, not the directory).

HTML reports are generated under target/criterion/. The interpolation engine's own benchmarks live in the splimes repository.

Formatting uses nightly-only rustfmt options (imports_granularity, group_imports), so the style gate runs on nightly even though the crates themselves build on stable:

cargo +nightly fmt --all

---

Project layout

WeftDB/
├── Cargo.toml                  # workspace manifest (members + shared deps)
├── weftdb/                     # time-series database (Turso/libSQL control plane + segment store)
│   └── src/types/
│       ├── database/           # connection, config, inputs, outputs, pipeline
│       ├── compression/        # tiered dataset compression
│       ├── batches/            # batching & analysis
│       └── …                   # patterns, events, correlations, signals, …
├── weft-orchestration/         # pipeline orchestration & event detectors
├── weft-physical-type/         # physical encodings, schemas, timestamp codecs, .weftseg
├── weft-arrow/                 # Arrow/Parquet interchange for segments
├── weft-arrow-store/           # Arrow/Parquet bridge over the segment store
├── weft-line-protocol/         # InfluxDB Line Protocol parser (shared dialect)
├── weft-reduce/                # downsampling reductions over a bucket grid
├── weft-server/                # axum HTTP API server
├── weft-bench/                 # benchmark harness (adapters, profiles, reports)
└── weft-tui/                   # terminal UI binary

The complete work queue and design constraints live in ROADMAP.md — the single source of truth.

---

Crates

WeftDB is a Rust Cargo workspace, from a low-level numerical engine up to an HTTP server and an interactive application:

Crate	Role
splimes (own repo, from crates.io)	Spline interpolation engine — Linear / Quadratic / Cubic / Polynomial methods on the CPU (one thread or rayon) or the GPU, chosen by grid size, with raw / interpolated / extrapolated provenance on every point.
weftdb	Time-series database: Turso (libSQL) control plane (catalog, metadata, segment index; MVCC concurrent writes) + WeftDB's own typed columnar .weftseg segment store on the measurement hot path, plus the pattern-recognition types and tiered dataset compression.
weft-orchestration	High-level pipeline that chains batching → pattern extraction → event detection → correlation → signal generation, with built-in detectors and parallel execution.
weft-physical-type	Vendor-neutral physical type system — schema-declared numeric encodings with explicit exactness, timestamp codecs, and the .weftseg columnar segment format (single-block and paged).
weft-arrow / weft-arrow-store	Apache Arrow / Parquet interchange for sealed segments and stored reads, kept in leaf crates so the arrow-* dependency tree never reaches the hot-path core.
weft-line-protocol	Dependency-free InfluxDB Line Protocol parser shared by the server and the benchmark harness.
weft-reduce	Vendor-neutral downsampling reductions, computable over parts and merged (reduce_partial/PartialReduction, exact for every reduction) — min/max/avg/sum/first/last + nearest-rank p50/p90/p95/p99 + three time-weighted averages (LOCF, linear/trapezoidal, LOCF-to-bucket-end) + mergeable bounded-error sketch_p* percentiles, over an epoch-aligned bucket grid, computed in BigDecimal. A PartialReduction is serde-serializable (so a segment's partial can be persisted and merged later, in place of re-reading it) and re-bucketable to any coarser nesting resolution (rebucket/grids_nest). Shared by the HTTP downsample endpoint and the benchmark harness.
weft-server	Benchmark-grade axum HTTP API — interpolation, downsampling, storage ingest/query, catalog management, Prometheus metrics, live latency profiles.
weft-bench	Reproducible, correctness-gated benchmark harness — the roadmap's spine; both an internal suite and a customer-runnable diagnostic.
weft-tui	Terminal user interface (Ratatui + Crossterm) for creating databases, importing CSV data, browsing and plotting aspects, and running compression.

The platform is designed for real-time and large-scale workloads: measurements are stored with BigDecimal logical precision, interpolation scales from a handful of points to millions, from one CPU thread to the rayon pool and, where calibration measured it faster, the GPU, and the storage layer uses bulk transactions, cached connections, and typed columnar segments throughout.

---

Tuning tips

• Batch your writes. batch_capture_measurements uses bulk transactions and is dramatically faster than single inserts.
• Connections are cached. The database layer reuses connections through a TTL/LRU cache (default: 50 connections, 30-minute TTL) and starts MVCC BEGIN CONCURRENT transactions automatically, with retry + backoff on transient failures.
• Calibrate the backends once. weft-server does it in the background at startup; a program embedding the libraries calls splimes::calibrate() (or splimes::prewarm_gpu() plus splimes::set_auto_thresholds) once, off any latency-critical path. Without it, interpolation never uses the GPU.
• Pick sensible batch sizes for pipelines (typically 24–100 for hourly data).
• Let the engine choose. Backend::Auto (behind analyze_range and every endpoint) already selects the backend by grid size — overriding is rarely necessary.
• Pipeline pattern loading is memory-aware: batch sizes adapt to available RAM to avoid exhaustion on large datasets.

---

Roadmap

ROADMAP.md is the single source of truth for everything planned, in progress, and done — a phase-based [ ]/[x] checkbox queue organized around the benchmark-led commercial thesis (north star: dollars per billion interpolated output points at a p95 latency target). Shipped capabilities are documented here as features; every performance claim links to a benchmark artifact.

---

License

Copyright (c) 2025-2026 Justin Icenhour.

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)

at your option.

Two crates include code adapted from Apache-2.0 projects — the Chimp codecs in weft-physical-type and the DDSketch quantile sketch in weft-reduce. Those portions stay under the Apache License 2.0 whichever option you choose. NOTICE lists them, and each crate's THIRD-PARTY-NOTICES file carries the attribution and the license text.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Every pull request also needs either a DCO sign-off on each commit or a signed CLA; see Contribution terms.