Repository navigation
feat(plan): plan from synced state, depend on jobs or tags, and faster plans - #1580
Merged
Merged
Conversation
`sqb plan --timings`, and the timings a plan document carries, left discovering the project, evaluating the configured rules and opening the warehouse connection inside Other. Record them as their own steps, "Discovering the project", "Evaluating rules" and "Connecting to the warehouse", so Other is only time no step covers. A location's plan from its synced state uses the project it already discovered, compiled and checked, and opens no connection, so those steps do not appear there.
Planning listed every schema with SHOW TABLES and then SHOW VIEWS, one after the other. A schema-scoped reader can now hand the catalog its listing as independent reads and combine their results, and Snowflake does so, so the catalog runs every schema's TABLES and VIEWS reads together within the adapter's inspection concurrency. Builds keep the listing path that records schema listing versions. The schema existence check behind a failed SHOW is remembered per plan; callers asking while it runs now wait and share its answer, so two failed reads of a missing schema check it once.
A live plan read every incremental model's cursor bounds with its own statement, so a project with hundreds of cursor models ran hundreds of round trips, eight at a time on Snowflake. Read up to 50 physical cursor reads in one UNION ALL statement, one row per read tagged with its position, at the adapter's inspection concurrency. A read without a minimum repeats its maximum, so no branch casts a bare NULL. When a batch fails, its reads run their own statements, so one unreadable relation is still the only one reported unavailable. A single read keeps its single statement. The eligible-maximum reads stay one statement each: they filter rows rather than read metadata, and batched they ran one after another inside a single statement.
…ve dialects On Snowflake without QUOTED_IDENTIFIERS_IGNORE_CASE, and on Postgres, every source with inspected warehouse columns counted as changed, so planning analysed every selected model downstream of a source again, even when the warehouse columns matched the declared ones. The blanket rule came with physical rebinding, where warehouse names are bound with their exact case: a column stored as lowercase "id" is not the unquoted id a model selects. Compare each listed column's bound identity (quoted names exactly, unquoted names folded the way the dialect folds them) with the compile-time shape instead, so that case still rebinds and rejects the reader, and matching shapes add no work.
…print A plan with declared model migrations read the newest fingerprint of each origin by rebuilding a lookup key from every model fingerprint in the state schema, twice per migration, and resolved each origin name by walking every project model. On a 10K-model project with about a hundred `migrate_from` models that was close to five seconds of a location's warm preview. Read each schema's fingerprint keys once, when the schema's fingerprints are inspected, index them by built relation name and by node name, and resolve origin names through one name index per plan. Lookups choose the same fingerprint as before: the candidates keep their stored order, so ties resolve the same way. The 10K preview guard now also bounds the warm preview at 10 seconds.
…ning workload The 10K guards timed compile and a location's preview from its synced state, but nothing timed a live `sqb plan`, and the fixture had a few dozen incremental models, five Python files and no migrations. A live plan that analysed every model again against inspected source columns, and read each cursor bound with its own statement, went unnoticed. Each preview case now also plans the project in-process against an offline Snowflake warehouse loaded from the same synced catalog, one source in fifty carrying an undeclared column. It bounds the models analysed again against source columns, the warehouse statements sent (cursor bounds read in full batches) and the live plan's time as a multiple of the warm preview. The fake warehouse answers the column description of expression sources. Generator 4 adds an append-incremental history of every third timestamped intermediate table (583 models, 112 capping their automatic start, 112 renamed with migrate_from) and a Python export asset for every fifth mart (433 assets in 241 modules sharing one helper module).
…res them A source declared `VARCHAR` lists as `VARCHAR(16777216)` on Snowflake, and one declared `INTEGER` or `NUMBER` lists as `NUMBER(38,0)`, so every such source counted as changed and planning analysed every selected model reading it again. Compare a declared type with the listed one through the analysis dialect's engine types instead of their text: synonyms, default string lengths, a timestamp precision left unstated, and Snowflake's bare NUMBER or DECIMAL as NUMBER(38,0) all match. A shorter VARCHAR or a different scale still counts as a change. Dialects without a native analysis module keep the exact text comparison. The 10K live-plan guard's offline warehouse now lists source columns with Snowflake's own spellings.
…olumn Physical rebinding now analyses a source's readers again only when its listed columns differ from the declared ones, so the session-setting case listed the declared column unchanged and saw nothing to rebind; the plan still stopped on the compile-time B002. List the column with a different length so the case rebinds and checks the setting again.
…n change them When a source's warehouse columns differed from its declared ones, planning analysed every selected model reading it again, and every model downstream of those. Warehouses routinely carry columns a source does not declare, so on a real project this meant re-analysing about 1,600 models on every plan, almost all of them unchanged. When the listing only adds columns, every declared column present with the type it stores, analyse a reader again only if the new columns can change its result: it projects the source's columns through a star, directly or through CTEs, subqueries and set operations; it binds an unqualified name in a scope where the source sits beside another relation; it names one of the added columns anywhere; or it uses a construct the walk does not follow, or SQL the parser cannot read. Everything else keeps its compile-time analysis. A re-analysed model passes on only the difference its own new shape makes, so its readers are judged the same way, and a model whose shape did not change stops the cascade. A missing or retyped column still re-analyses every reader. On a 10K-model fixture with one source in fifty carrying an extra column, re-analysis drops from 519 models to 52; the guard's expected range follows.
Every plan lists and describes the warehouse's schemas live, which is most of a plan's time on a large project, while `sqb dev` already keeps a synced copy of the catalog in the project's control store. A target can now set `plan_from = "synced_state"`, and `--plan-from live|synced_state` on `sqb plan` and `sqb build` overrides it for one invocation. The default stays `live`. Planning from synced state reads the target's mirrored catalog from the control store, then checks it with one INFORMATION_SCHEMA.TABLES read per database: the listing it returns is used as is, so created and dropped tables are seen, and columns come from the copy except for tables whose LAST_DDL is later than the copy observed them (less five minutes for clock skew), which are described again. With more than 128 such tables the schema's columns are read live. Cursor bounds and incremental state are still read live. When the project has no control store, no location syncs the target for the project, the copy is empty or the check fails, the plan reads live and says why on stderr. Otherwise it reports how long the check took and how many tables it described again. Planning never writes to the control store.
…e warm preview The live plan was bounded at six times the warm preview. Once the warm preview got faster, live plans that had not changed came out at 5.3-6.4 times it on a loaded machine, so the ratio failed for reasons that have nothing to do with live planning. Bound the live plan by the location's publish instead, which compiles the same project on the same machine. Measured at 1.22-1.40 across six runs, quiet and loaded; the limit is 1.8. The warm preview keeps its 10 second limit.
Tasks, assets and checks can declare depends_on=job("orders_job") or
tag("finance"), each with an optional exclude= of ordinary selectors.
Compile resolves them into edges to the set's end nodes: its models,
seeds, sources, loaders, tasks and assets, less the exclude, that
nothing else in the set reads. Planning, selector expansion,
--changed, sqb dag and job group waits then treat them like named
dependencies, and a model named and reached through a set is one edge.
Compile fails for an unknown job or tag, a set its exclude empties, a
loader declaring one, a code location prefix (reserved), a set node
reading the dependant's output (naming the nearest reader and the
exclude that breaks the cycle), and a job group the new edges would make
wait in a loop, as groups wait job by job today; that error names the
nodes closing the loop and suggests moving them to another job.
Each node's identity records the end nodes it resolved, and the plan
lists what it now also waits on, or no longer waits on, since its last
build, in text and as wait_changes in --json. Published Python node
definitions carry set_dependencies for the UI's lineage to use.
A plan of an unchanged project still compiled in 6-7 s with every model's analysis answered by the compile cache: the cache covers analysis, not the per-model work around it. The largest piece was building each model's analysis text, which wraps typed SQL macro calls in markers and parses the whole model to check each marker lands where it parses, up to once per marked call. Every compile did this twice per model, once for the analysis request and once for the compiled model. Keep the verified marker set per (dialect, SQL, calls) for the process, a bounded memo the second read and later compiles in long-lived processes reuse. On a 2,280-model project, compiled-model assembly drops from 0.49 s to 0.05 s per compile.
A plan read the warehouse columns of every built model in the project, whatever it selected: on a 2,400-model Snowflake project that is one SHOW COLUMNS IN SCHEMA of 131K rows, measured at 3-29 s, even for a one-model plan. Read columns only for the relations of the selection and what it reads, the models whose changes the plan and its stale warnings detect; an unnarrowed plan still reads them all. Snowflake reads up to 200 relations with concurrent SHOW COLUMNS IN TABLE (about 8 ms a relation at the inspection concurrency, 200 in 1.7 s) instead of from 33, so a small selection no longer lists the schema; the sequential direct path keeps 32. INFORMATION_SCHEMA.COLUMNS with an IN list is not an option there: 7.8 s for 30 tables. Parsing the listing's type JSON is memoised by text: a 131K-row schema holds 135 distinct texts, 0.28 s down to 0.02 s.
The Python node decorators, two-field key readers and the development location read each touch an existing near-miss cluster; each keeps its own record type or SQL, with the shared work already in one owner.
kvlonge
enabled auto-merge
November 18, 2026 17:14
# Conflicts: # crates/sqlbuild-python-project/src/native_host/_helpers/mod.rs # crates/sqlbuild-python-project/src/native_host/main/mod.rs
Moves the set_dependencies publication into ProjectSelection so the Python extension crate stays within its code-line cap after merging main.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Most of a plan's time on a large warehouse is reading metadata, and an unchanged project still paid for per-model work the cache doesn't cover. Python nodes also could not wait on a whole job or tag.
Changes
plan_from = "synced_state"(and--plan-from) plans from the target's synced catalog, checked live (Snowflake; other adapters plan live and say why); the default stayslive.job(...)ortag(...); plans show what a node now also waits on.Not ported: a per-process memo for shared Python helpers. Main already shares identity work per discovery pass, the remaining gain is unmeasured, and it needs a
[[stays]]approval; it returns only if asked.Golden:
failure/target-unknown-keynow listsplan_fromamong a target's allowed keys.DuckShop: no edge case added. The two plan fixes compare live warehouse column metadata with declared sources, which DuckShop's edge area does not exercise; offline Snowflake planner tests cover them.
Verification
make check-ci.