Skip to content

feat(plan): plan from synced state, depend on jobs or tags, and faster plans - #1580

Merged
kvlonge merged 18 commits into
mainfrom
perf/plan-speed-main
Nov 18, 2026
Merged

kvlonge merged 18 commits into
mainfrom
perf/plan-speed-main

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Why

Most of a plan's time on a large warehouse is reading metadata, and an unchanged project still paid for per-model work the cache doesn't cover. Python nodes also could not wait on a whole job or tag.

Changes

  • plan_from = "synced_state" (and --plan-from) plans from the target's synced catalog, checked live (Snowflake; other adapters plan live and say why); the default stays live.
  • Cursor bounds are read in batched UNION ALL statements; tables and views are listed together; migration origins and source rebinding avoid rescans and needless re-analysis.
  • A plan reads columns only for the selection and what it reads. Snowflake describes up to 200 relations one by one before listing a schema, and memoises type parsing.
  • Typed SQL macro markers are verified once per process.
  • Tasks, assets and checks can depend on job(...) or tag(...); plans show what a node now also waits on.
  • Plan timings cover discovery, rules and the connection.

Not ported: a per-process memo for shared Python helpers. Main already shares identity work per discovery pass, the remaining gain is unmeasured, and it needs a [[stays]] approval; it returns only if asked.

Golden: failure/target-unknown-key now lists plan_from among a target's allowed keys.

DuckShop: no edge case added. The two plan fixes compare live warehouse column metadata with declared sources, which DuckShop's edge area does not exercise; offline Snowflake planner tests cover them.

Verification

  • 621 targeted unit, integration and plan E2E tests (incl. synced state on DuckDB); native set-dependency and typed-call tests; make check-ci.
  • Same machine, base main vs this branch, 3 alternating runs each, load 2.4-3.7 on 8 cores, offline Snowflake, 2,400 models, 8 ms per statement, warm compile cache. Full plan: wall 2.67 s to 1.81 s, CPU 2.62 s to 1.82 s, cursor-bound statements 615 to 13. One-model plan: 0.76 s to 0.73 s.

`sqb plan --timings`, and the timings a plan document carries, left
discovering the project, evaluating the configured rules and opening the
warehouse connection inside Other. Record them as their own steps,
"Discovering the project", "Evaluating rules" and "Connecting to the
warehouse", so Other is only time no step covers. A location's plan from
its synced state uses the project it already discovered, compiled and
checked, and opens no connection, so those steps do not appear there.
Planning listed every schema with SHOW TABLES and then SHOW VIEWS, one
after the other. A schema-scoped reader can now hand the catalog its
listing as independent reads and combine their results, and Snowflake
does so, so the catalog runs every schema's TABLES and VIEWS reads
together within the adapter's inspection concurrency. Builds keep the
listing path that records schema listing versions.

The schema existence check behind a failed SHOW is remembered per plan;
callers asking while it runs now wait and share its answer, so two
failed reads of a missing schema check it once.
A live plan read every incremental model's cursor bounds with its own
statement, so a project with hundreds of cursor models ran hundreds of
round trips, eight at a time on Snowflake.

Read up to 50 physical cursor reads in one UNION ALL statement, one row
per read tagged with its position, at the adapter's inspection
concurrency. A read without a minimum repeats its maximum, so no branch
casts a bare NULL. When a batch fails, its reads run their own
statements, so one unreadable relation is still the only one reported
unavailable. A single read keeps its single statement.

The eligible-maximum reads stay one statement each: they filter rows
rather than read metadata, and batched they ran one after another
inside a single statement.
…ve dialects

On Snowflake without QUOTED_IDENTIFIERS_IGNORE_CASE, and on Postgres, every
source with inspected warehouse columns counted as changed, so planning
analysed every selected model downstream of a source again, even when the
warehouse columns matched the declared ones.

The blanket rule came with physical rebinding, where warehouse names are
bound with their exact case: a column stored as lowercase "id" is not the
unquoted id a model selects. Compare each listed column's bound identity
(quoted names exactly, unquoted names folded the way the dialect folds
them) with the compile-time shape instead, so that case still rebinds and
rejects the reader, and matching shapes add no work.
…print

A plan with declared model migrations read the newest fingerprint of
each origin by rebuilding a lookup key from every model fingerprint in
the state schema, twice per migration, and resolved each origin name
by walking every project model. On a 10K-model project with about a
hundred `migrate_from` models that was close to five seconds of a
location's warm preview.

Read each schema's fingerprint keys once, when the schema's
fingerprints are inspected, index them by built relation name and by
node name, and resolve origin names through one name index per plan.
Lookups choose the same fingerprint as before: the candidates keep
their stored order, so ties resolve the same way.

The 10K preview guard now also bounds the warm preview at 10 seconds.
…ning workload

The 10K guards timed compile and a location's preview from its synced
state, but nothing timed a live `sqb plan`, and the fixture had a few
dozen incremental models, five Python files and no migrations. A live
plan that analysed every model again against inspected source columns,
and read each cursor bound with its own statement, went unnoticed.

Each preview case now also plans the project in-process against an
offline Snowflake warehouse loaded from the same synced catalog, one
source in fifty carrying an undeclared column. It bounds the models
analysed again against source columns, the warehouse statements sent
(cursor bounds read in full batches) and the live plan's time as a
multiple of the warm preview. The fake warehouse answers the column
description of expression sources.

Generator 4 adds an append-incremental history of every third
timestamped intermediate table (583 models, 112 capping their automatic
start, 112 renamed with migrate_from) and a Python export asset for every
fifth mart (433 assets in 241 modules sharing one helper module).
…res them

A source declared `VARCHAR` lists as `VARCHAR(16777216)` on Snowflake,
and one declared `INTEGER` or `NUMBER` lists as `NUMBER(38,0)`, so
every such source counted as changed and planning analysed every
selected model reading it again.

Compare a declared type with the listed one through the analysis
dialect's engine types instead of their text: synonyms, default string
lengths, a timestamp precision left unstated, and Snowflake's bare
NUMBER or DECIMAL as NUMBER(38,0) all match. A shorter VARCHAR or a
different scale still counts as a change. Dialects without a native
analysis module keep the exact text comparison.

The 10K live-plan guard's offline warehouse now lists source columns
with Snowflake's own spellings.
…olumn

Physical rebinding now analyses a source's readers again only when its
listed columns differ from the declared ones, so the session-setting
case listed the declared column unchanged and saw nothing to rebind;
the plan still stopped on the compile-time B002. List the column with a
different length so the case rebinds and checks the setting again.
…n change them

When a source's warehouse columns differed from its declared ones,
planning analysed every selected model reading it again, and every
model downstream of those. Warehouses routinely carry columns a source
does not declare, so on a real project this meant re-analysing about
1,600 models on every plan, almost all of them unchanged.

When the listing only adds columns, every declared column present with
the type it stores, analyse a reader again only if the new columns can
change its result: it projects the source's columns through a star,
directly or through CTEs, subqueries and set operations; it binds an
unqualified name in a scope where the source sits beside another
relation; it names one of the added columns anywhere; or it uses a
construct the walk does not follow, or SQL the parser cannot read.
Everything else keeps its compile-time analysis. A re-analysed model
passes on only the difference its own new shape makes, so its readers
are judged the same way, and a model whose shape did not change stops
the cascade. A missing or retyped column still re-analyses every reader.

On a 10K-model fixture with one source in fifty carrying an extra
column, re-analysis drops from 519 models to 52; the guard's expected
range follows.
Every plan lists and describes the warehouse's schemas live, which is
most of a plan's time on a large project, while `sqb dev` already keeps
a synced copy of the catalog in the project's control store.

A target can now set `plan_from = "synced_state"`, and `--plan-from
live|synced_state` on `sqb plan` and `sqb build` overrides it for one
invocation. The default stays `live`. Planning from synced state reads
the target's mirrored catalog from the control store, then checks it
with one INFORMATION_SCHEMA.TABLES read per database: the listing it
returns is used as is, so created and dropped tables are seen, and
columns come from the copy except for tables whose LAST_DDL is later
than the copy observed them (less five minutes for clock skew), which
are described again. With more than 128 such tables the schema's
columns are read live. Cursor bounds and incremental state are still
read live.

When the project has no control store, no location syncs the target
for the project, the copy is empty or the check fails, the plan reads
live and says why on stderr. Otherwise it reports how long the check
took and how many tables it described again. Planning never writes to
the control store.
…e warm preview

The live plan was bounded at six times the warm preview. Once the warm
preview got faster, live plans that had not changed came out at 5.3-6.4
times it on a loaded machine, so the ratio failed for reasons that have
nothing to do with live planning.

Bound the live plan by the location's publish instead, which compiles
the same project on the same machine. Measured at 1.22-1.40 across six
runs, quiet and loaded; the limit is 1.8. The warm preview keeps its
10 second limit.
Tasks, assets and checks can declare depends_on=job("orders_job") or
tag("finance"), each with an optional exclude= of ordinary selectors.
Compile resolves them into edges to the set's end nodes: its models,
seeds, sources, loaders, tasks and assets, less the exclude, that
nothing else in the set reads. Planning, selector expansion,
--changed, sqb dag and job group waits then treat them like named
dependencies, and a model named and reached through a set is one edge.

Compile fails for an unknown job or tag, a set its exclude empties, a
loader declaring one, a code location prefix (reserved), a set node
reading the dependant's output (naming the nearest reader and the
exclude that breaks the cycle), and a job group the new edges would make
wait in a loop, as groups wait job by job today; that error names the
nodes closing the loop and suggests moving them to another job.

Each node's identity records the end nodes it resolved, and the plan
lists what it now also waits on, or no longer waits on, since its last
build, in text and as wait_changes in --json. Published Python node
definitions carry set_dependencies for the UI's lineage to use.
A plan of an unchanged project still compiled in 6-7 s with every model's
analysis answered by the compile cache: the cache covers analysis, not the
per-model work around it. The largest piece was building each model's
analysis text, which wraps typed SQL macro calls in markers and parses the
whole model to check each marker lands where it parses, up to once per
marked call. Every compile did this twice per model, once for the analysis
request and once for the compiled model.

Keep the verified marker set per (dialect, SQL, calls) for the process, a
bounded memo the second read and later compiles in long-lived processes
reuse. On a 2,280-model project, compiled-model assembly drops from 0.49 s
to 0.05 s per compile.
A plan read the warehouse columns of every built model in the project,
whatever it selected: on a 2,400-model Snowflake project that is one
SHOW COLUMNS IN SCHEMA of 131K rows, measured at 3-29 s, even for a
one-model plan.

Read columns only for the relations of the selection and what it reads,
the models whose changes the plan and its stale warnings detect; an
unnarrowed plan still reads them all.

Snowflake reads up to 200 relations with concurrent SHOW COLUMNS IN TABLE
(about 8 ms a relation at the inspection concurrency, 200 in 1.7 s)
instead of from 33, so a small selection no longer lists the schema; the
sequential direct path keeps 32. INFORMATION_SCHEMA.COLUMNS with an IN
list is not an option there: 7.8 s for 30 tables.

Parsing the listing's type JSON is memoised by text: a 131K-row schema
holds 135 distinct texts, 0.28 s down to 0.02 s.
The Python node decorators, two-field key readers and the development
location read each touch an existing near-miss cluster; each keeps its
own record type or SQL, with the shared work already in one owner.
@kvlonge
kvlonge enabled auto-merge November 18, 2026 17:14
# Conflicts:
#	crates/sqlbuild-python-project/src/native_host/_helpers/mod.rs
#	crates/sqlbuild-python-project/src/native_host/main/mod.rs
Moves the set_dependencies publication into ProjectSelection so the
Python extension crate stays within its code-line cap after merging main.
@kvlonge
kvlonge added this pull request to the merge queue Nov 18, 2026
@kvlonge
kvlonge removed this pull request from the merge queue due to a manual request Nov 18, 2026
@kvlonge
kvlonge added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 1fb37f7 Nov 18, 2026
35 checks passed
@kvlonge
kvlonge deleted the perf/plan-speed-main branch November 18, 2026 19:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant