Skip to content

feat(server): preview plans from the synced warehouse copy - #1437

Merged
chio-labs-delivery[bot] merged 17 commits into
mainfrom
feat/plan-preview-observations
Nov 13, 2026
Merged

chio-labs-delivery[bot] merged 17 commits into
mainfrom
feat/plan-preview-observations

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Why

Every UI Run button plans against the warehouse first, resuming a suspended warehouse just to show a plan. The server already syncs the metadata a preview needs.

Changes

  • Successful builds record display-only observations (cursor MAX, described expression columns keyed to exact SQL), append-only and pruned by retention.
  • The server serves a location a synced copy: catalog, fingerprints, latest run, observations and when every built schema last synced cleanly.
  • A requested plan runs sqb plan on that copy and never connects. Windows start at the last build's cursor; freshness, retention, table types, unmirrored sources and moved windows are unknown; --defer-to is refused.
  • The plan page and docs show sync time, unknowns and window starts.

DuckShop: no case added; a new feature, not a fix for a recorded hazard.

Verification

Live planning is unchanged. Base and head, one toolchain, run alternately on one machine (3 runs each, load 1.4 of 8 cores; medians). plan --json minus timings is byte-identical:

  • DuckShop (DuckDB copy, 252 models, built): wall 0.996x, CPU 1.008x
  • dense 3,000: wall 0.976x, CPU 0.978x
  • fresh 3,000: wall 0.999x, CPU 0.992x

Preview equals live on DuckShop from a synced copy: all 252 models plan the same action, reason and backfill. Lookback and replay windows are listed unknown, and an edited expression is planned as declared.

Fensu XSB078 keeps observations out of planning and builds. XSB079 limits the synced copy to the listed preview branches.

  • E2E test_plan_preview.py: a real server and location with the warehouse write-locked. Preview equals live; an unbuilt cursor, lookback and a freshness block are unknown; an edited expression plans declared columns.
  • Probes: an unmirrored source, replay on change and a raised cursor_start are unknown. Sync time stays unset until every schema syncs (SQLite, Postgres).
  • UI: svelte-check 0 errors, 205 vitest tests pass, screenshots checked. make check-ci passes.

After a build, each successful incremental model's cursor MAX is read with
the planner's own cursor SQL and appended to the observations table in its
state schema. Failures only warn.
A build keeps the columns planning described for each type-enforced
expression source and appends them as source_columns observations in the
state schema of every model reading the source. Plans record nothing.
Location sync reads _sqlbuild_observations as its own history kind and
the server keeps the rows in sqb_warehouse_observations, keyed by the
observation identity, with kind and node columns for latest-value reads.
…ions

Add Fensu rule XSB078: Rust code may name observation state only in the
files that write, prune, mirror or display it. An E2E build ignores a
wrong cursor_max observation and appends from the live MAX.
A source_columns value is now {sql_hash, columns}, the SHA-256 of the
exact expression SQL planning described, so a preview can tell when the
expression changed since the observation and report it as unknown.
Add POST .../metadata/preview-copy for a location's fenced reader. It
returns the target's catalog relations, each node's newest fingerprint
with its definition, newest succeeded run and newest observation per
kind, and synced_at: the oldest of the newest clean syncs of catalog,
fingerprints and consumption, or null until all three synced cleanly.
A location's requested plan now fetches the server's synced copy and runs
sqb plan against an adapter that answers listings, columns and fingerprints
from it, never connecting to the warehouse. Cursor targets use the newest
successful build's cursor_max, inputs are read when the run starts, and
expression sources use the columns described for the same SQL. The plan
reports when the copy synced and what stays unknown until the run, and
refuses --defer-to. Fensu XSB079 confines the synced copy to sqb plan.
…window start

A location plans a build from the server's synced copy of warehouse
metadata. The plan page now says when that copy synced, lists what stays
unknown until the run, and shows each incremental model's window start
running to the latest data when the run starts.
Record why the remaining per-table state bindings, the SQL Server keyed
state DDL overrides, the observation ingest projection and the copy's
Python object builders look alike.
…lpers

Name the preview cases' outcomes expected_ and keep the Fensu rules module
to its rule functions.
…project shape

Table types and retention stay unknown instead of reading the warehouse,
and sources outside the mirrored schemas are not probed. The copy counts
as synced only once every schema the snapshot builds into has synced each
required table. A model whose lookback, backfill duration or microbatches
move its start reports its window as unknown.
…previews

A preview observes no source freshness and warns about none, and reports a
cursor window as unknown when lookback, replay on change, a start override,
or a cursor_start floor past the last build moves the run's start. Cursor
intrinsics bind to the bounds the run resolves, and typed source
projections render through the facade.
@kvlonge kvlonge added the full-verify Run the full Verify matrix on this pull request label Nov 13, 2026
@kvlonge
kvlonge disabled auto-merge November 13, 2026 04:45

kvlonge commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

CodeQL's cleartext-transmission alert on post_metadata covers behaviour that already exists on main: location requests, bearer token included, can go to a remote http:// server. This PR only adds a caller with the same body shape. The alert stays open, and whether to restrict plain HTTP is tracked as a separate follow-up.

…ark inputs

Keep the preview's runtime-resolved bounds to models whose watermark inputs
it cleared, so a model live planning cannot bound still fails the preview.
XSB079 now also confines the synced copy's refusal error.
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 13, 2026
Merged via the queue into main with commit 45005ee Nov 13, 2026
102 of 105 checks passed
@chio-labs-delivery
chio-labs-delivery Bot deleted the feat/plan-preview-observations branch November 13, 2026 07:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-verify Run the full Verify matrix on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant