Repository navigation
feat(server): preview plans from the synced warehouse copy - #1437
Merged
Merged
Conversation
After a build, each successful incremental model's cursor MAX is read with the planner's own cursor SQL and appended to the observations table in its state schema. Failures only warn.
A build keeps the columns planning described for each type-enforced expression source and appends them as source_columns observations in the state schema of every model reading the source. Plans record nothing.
Location sync reads _sqlbuild_observations as its own history kind and the server keeps the rows in sqb_warehouse_observations, keyed by the observation identity, with kind and node columns for latest-value reads.
…ions Add Fensu rule XSB078: Rust code may name observation state only in the files that write, prune, mirror or display it. An E2E build ignores a wrong cursor_max observation and appends from the live MAX.
A source_columns value is now {sql_hash, columns}, the SHA-256 of the
exact expression SQL planning described, so a preview can tell when the
expression changed since the observation and report it as unknown.
Add POST .../metadata/preview-copy for a location's fenced reader. It returns the target's catalog relations, each node's newest fingerprint with its definition, newest succeeded run and newest observation per kind, and synced_at: the oldest of the newest clean syncs of catalog, fingerprints and consumption, or null until all three synced cleanly.
A location's requested plan now fetches the server's synced copy and runs sqb plan against an adapter that answers listings, columns and fingerprints from it, never connecting to the warehouse. Cursor targets use the newest successful build's cursor_max, inputs are read when the run starts, and expression sources use the columns described for the same SQL. The plan reports when the copy synced and what stays unknown until the run, and refuses --defer-to. Fensu XSB079 confines the synced copy to sqb plan.
…mption and observations
…window start A location plans a build from the server's synced copy of warehouse metadata. The plan page now says when that copy synced, lists what stays unknown until the run, and shows each incremental model's window start running to the latest data when the run starts.
Record why the remaining per-table state bindings, the SQL Server keyed state DDL overrides, the observation ingest projection and the copy's Python object builders look alike.
…lpers Name the preview cases' outcomes expected_ and keep the Fensu rules module to its rule functions.
…project shape Table types and retention stay unknown instead of reading the warehouse, and sources outside the mirrored schemas are not probed. The copy counts as synced only once every schema the snapshot builds into has synced each required table. A model whose lookback, backfill duration or microbatches move its start reports its window as unknown.
…previews A preview observes no source freshness and warns about none, and reports a cursor window as unknown when lookback, replay on change, a start override, or a cursor_start floor past the last build moves the run's start. Cursor intrinsics bind to the bounds the run resolves, and typed source projections render through the facade.
Contributor
Author
|
CodeQL's cleartext-transmission alert on |
…ark inputs Keep the preview's runtime-resolved bounds to models whose watermark inputs it cleared, so a model live planning cannot bound still fails the preview. XSB079 now also confines the synced copy's refusal error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Every UI Run button plans against the warehouse first, resuming a suspended warehouse just to show a plan. The server already syncs the metadata a preview needs.
Changes
sqb planon that copy and never connects. Windows start at the last build's cursor; freshness, retention, table types, unmirrored sources and moved windows are unknown;--defer-tois refused.DuckShop: no case added; a new feature, not a fix for a recorded hazard.
Verification
Live planning is unchanged. Base and head, one toolchain, run alternately on one machine (3 runs each, load 1.4 of 8 cores; medians).
plan --jsonminus timings is byte-identical:Preview equals live on DuckShop from a synced copy: all 252 models plan the same action, reason and backfill. Lookback and replay windows are listed unknown, and an edited expression is planned as declared.
Fensu XSB078 keeps observations out of planning and builds. XSB079 limits the synced copy to the listed preview branches.
test_plan_preview.py: a real server and location with the warehouse write-locked. Preview equals live; an unbuilt cursor, lookback and a freshness block are unknown; an edited expression plans declared columns.cursor_startare unknown. Sync time stays unset until every schema syncs (SQLite, Postgres).make check-cipasses.