Repository navigation
Commit 1fb37f7
authored
feat(plan): plan from synced state, depend on jobs or tags, and faster plans (#1580)
* feat(plan): time project discovery, rules and the warehouse connection
`sqb plan --timings`, and the timings a plan document carries, left
discovering the project, evaluating the configured rules and opening the
warehouse connection inside Other. Record them as their own steps,
"Discovering the project", "Evaluating rules" and "Connecting to the
warehouse", so Other is only time no step covers. A location's plan from
its synced state uses the project it already discovered, compiled and
checked, and opens no connection, so those steps do not appear there.
* perf(plan): list each schema's tables and views at the same time
Planning listed every schema with SHOW TABLES and then SHOW VIEWS, one
after the other. A schema-scoped reader can now hand the catalog its
listing as independent reads and combine their results, and Snowflake
does so, so the catalog runs every schema's TABLES and VIEWS reads
together within the adapter's inspection concurrency. Builds keep the
listing path that records schema listing versions.
The schema existence check behind a failed SHOW is remembered per plan;
callers asking while it runs now wait and share its answer, so two
failed reads of a missing schema check it once.
* perf(plan): read cursor bounds in batched UNION ALL statements
A live plan read every incremental model's cursor bounds with its own
statement, so a project with hundreds of cursor models ran hundreds of
round trips, eight at a time on Snowflake.
Read up to 50 physical cursor reads in one UNION ALL statement, one row
per read tagged with its position, at the adapter's inspection
concurrency. A read without a minimum repeats its maximum, so no branch
casts a bare NULL. When a batch fails, its reads run their own
statements, so one unreadable relation is still the only one reported
unavailable. A single read keeps its single statement.
The eligible-maximum reads stay one statement each: they filter rows
rather than read metadata, and batched they ran one after another
inside a single statement.
* fix(plan): rebind sources only on new column evidence on case-sensitive dialects
On Snowflake without QUOTED_IDENTIFIERS_IGNORE_CASE, and on Postgres, every
source with inspected warehouse columns counted as changed, so planning
analysed every selected model downstream of a source again, even when the
warehouse columns matched the declared ones.
The blanket rule came with physical rebinding, where warehouse names are
bound with their exact case: a column stored as lowercase "id" is not the
unquoted id a model selects. Compare each listed column's bound identity
(quoted names exactly, unquoted names folded the way the dialect folds
them) with the compile-time shape instead, so that case still rebinds and
rejects the reader, and matching shapes add no work.
* perf(plan): look up migration origins without rescanning every fingerprint
A plan with declared model migrations read the newest fingerprint of
each origin by rebuilding a lookup key from every model fingerprint in
the state schema, twice per migration, and resolved each origin name
by walking every project model. On a 10K-model project with about a
hundred `migrate_from` models that was close to five seconds of a
location's warm preview.
Read each schema's fingerprint keys once, when the schema's
fingerprints are inspected, index them by built relation name and by
node name, and resolve origin names through one name index per plan.
Lookups choose the same fingerprint as before: the candidates keep
their stored order, so ties resolve the same way.
The 10K preview guard now also bounds the warm preview at 10 seconds.
* test(perf): guard live plans of the 10K project and give it real planning workload
The 10K guards timed compile and a location's preview from its synced
state, but nothing timed a live `sqb plan`, and the fixture had a few
dozen incremental models, five Python files and no migrations. A live
plan that analysed every model again against inspected source columns,
and read each cursor bound with its own statement, went unnoticed.
Each preview case now also plans the project in-process against an
offline Snowflake warehouse loaded from the same synced catalog, one
source in fifty carrying an undeclared column. It bounds the models
analysed again against source columns, the warehouse statements sent
(cursor bounds read in full batches) and the live plan's time as a
multiple of the warm preview. The fake warehouse answers the column
description of expression sources.
Generator 4 adds an append-incremental history of every third
timestamped intermediate table (583 models, 112 capping their automatic
start, 112 renamed with migrate_from) and a Python export asset for every
fifth mart (433 assets in 241 modules sharing one helper module).
* fix(plan): compare inspected source column types as the warehouse stores them
A source declared `VARCHAR` lists as `VARCHAR(16777216)` on Snowflake,
and one declared `INTEGER` or `NUMBER` lists as `NUMBER(38,0)`, so
every such source counted as changed and planning analysed every
selected model reading it again.
Compare a declared type with the listed one through the analysis
dialect's engine types instead of their text: synonyms, default string
lengths, a timestamp precision left unstated, and Snowflake's bare
NUMBER or DECIMAL as NUMBER(38,0) all match. A shorter VARCHAR or a
different scale still counts as a change. Dialects without a native
analysis module keep the exact text comparison.
The 10K live-plan guard's offline warehouse now lists source columns
with Snowflake's own spellings.
* test(plan): rebind the quoted-case session case on a retyped source column
Physical rebinding now analyses a source's readers again only when its
listed columns differ from the declared ones, so the session-setting
case listed the declared column unchanged and saw nothing to rebind;
the plan still stopped on the compile-time B002. List the column with a
different length so the case rebinds and checks the setting again.
* perf(plan): analyse readers again only when a source's new columns can change them
When a source's warehouse columns differed from its declared ones,
planning analysed every selected model reading it again, and every
model downstream of those. Warehouses routinely carry columns a source
does not declare, so on a real project this meant re-analysing about
1,600 models on every plan, almost all of them unchanged.
When the listing only adds columns, every declared column present with
the type it stores, analyse a reader again only if the new columns can
change its result: it projects the source's columns through a star,
directly or through CTEs, subqueries and set operations; it binds an
unqualified name in a scope where the source sits beside another
relation; it names one of the added columns anywhere; or it uses a
construct the walk does not follow, or SQL the parser cannot read.
Everything else keeps its compile-time analysis. A re-analysed model
passes on only the difference its own new shape makes, so its readers
are judged the same way, and a model whose shape did not change stops
the cascade. A missing or retyped column still re-analyses every reader.
On a 10K-model fixture with one source in fifty carrying an extra
column, re-analysis drops from 519 models to 52; the guard's expected
range follows.
* feat(plan): let a target plan from its synced copy of warehouse state
Every plan lists and describes the warehouse's schemas live, which is
most of a plan's time on a large project, while `sqb dev` already keeps
a synced copy of the catalog in the project's control store.
A target can now set `plan_from = "synced_state"`, and `--plan-from
live|synced_state` on `sqb plan` and `sqb build` overrides it for one
invocation. The default stays `live`. Planning from synced state reads
the target's mirrored catalog from the control store, then checks it
with one INFORMATION_SCHEMA.TABLES read per database: the listing it
returns is used as is, so created and dropped tables are seen, and
columns come from the copy except for tables whose LAST_DDL is later
than the copy observed them (less five minutes for clock skew), which
are described again. With more than 128 such tables the schema's
columns are read live. Cursor bounds and incremental state are still
read live.
When the project has no control store, no location syncs the target
for the project, the copy is empty or the check fails, the plan reads
live and says why on stderr. Otherwise it reports how long the check
took and how many tables it described again. Planning never writes to
the control store.
* test(perf): bound the 10K live plan by the location's publish, not the warm preview
The live plan was bounded at six times the warm preview. Once the warm
preview got faster, live plans that had not changed came out at 5.3-6.4
times it on a loaded machine, so the ratio failed for reasons that have
nothing to do with live planning.
Bound the live plan by the location's publish instead, which compiles
the same project on the same machine. Measured at 1.22-1.40 across six
runs, quiet and loaded; the limit is 1.8. The warm preview keeps its
10 second limit.
* feat(python-nodes): depend on a job or a tag
Tasks, assets and checks can declare depends_on=job("orders_job") or
tag("finance"), each with an optional exclude= of ordinary selectors.
Compile resolves them into edges to the set's end nodes: its models,
seeds, sources, loaders, tasks and assets, less the exclude, that
nothing else in the set reads. Planning, selector expansion,
--changed, sqb dag and job group waits then treat them like named
dependencies, and a model named and reached through a set is one edge.
Compile fails for an unknown job or tag, a set its exclude empties, a
loader declaring one, a code location prefix (reserved), a set node
reading the dependant's output (naming the nearest reader and the
exclude that breaks the cycle), and a job group the new edges would make
wait in a loop, as groups wait job by job today; that error names the
nodes closing the loop and suggests moving them to another job.
Each node's identity records the end nodes it resolved, and the plan
lists what it now also waits on, or no longer waits on, since its last
build, in text and as wait_changes in --json. Published Python node
definitions carry set_dependencies for the UI's lineage to use.
* perf(compile): verify a model's typed SQL macro markers once per process
A plan of an unchanged project still compiled in 6-7 s with every model's
analysis answered by the compile cache: the cache covers analysis, not the
per-model work around it. The largest piece was building each model's
analysis text, which wraps typed SQL macro calls in markers and parses the
whole model to check each marker lands where it parses, up to once per
marked call. Every compile did this twice per model, once for the analysis
request and once for the compiled model.
Keep the verified marker set per (dialect, SQL, calls) for the process, a
bounded memo the second read and later compiles in long-lived processes
reuse. On a 2,280-model project, compiled-model assembly drops from 0.49 s
to 0.05 s per compile.
* perf(plan): read target columns for the selection and its upstream only
A plan read the warehouse columns of every built model in the project,
whatever it selected: on a 2,400-model Snowflake project that is one
SHOW COLUMNS IN SCHEMA of 131K rows, measured at 3-29 s, even for a
one-model plan.
Read columns only for the relations of the selection and what it reads,
the models whose changes the plan and its stale warnings detect; an
unnarrowed plan still reads them all.
Snowflake reads up to 200 relations with concurrent SHOW COLUMNS IN TABLE
(about 8 ms a relation at the inspection concurrency, 200 in 1.7 s)
instead of from 33, so a small selection no longer lists the schema; the
sequential direct path keeps 32. INFORMATION_SCHEMA.COLUMNS with an IN
list is not an option there: 7.8 s for 30 tables.
Parsing the listing's type JSON is memoised by text: a 131K-row schema
holds 135 distinct texts, 0.28 s down to 0.02 s.
* chore(fensu): record the intentional mirrors the plan work touches
The Python node decorators, two-field key readers and the development
location read each touch an existing near-miss cluster; each keeps its
own record type or SQL, with the shared work already in one owner.
* refactor(graph): publish set dependencies from the project selection
Moves the set_dependencies publication into ProjectSelection so the
Python extension crate stays within its code-line cap after merging main.
* test(goldens): list plan_from among a target's allowed keys1 parent 4decdcc commit 1fb37f7
178 files changed
Lines changed: 6529 additions & 449 deletions
File tree
- benchmarks/duckshop-10k
- duckshop10k
- tests
- crates
- sqlbuild-analysis/src
- analysis_route
- main
- tests
- node_set_dependencies
- _helpers
- main
- tests
- planner_warehouse
- main
- tests
- source_drift
- _helpers
- main
- tests
- sql_macros
- _helpers/typing
- tests
- sqlbuild-cli/src
- argv
- tests
- dispatch
- sqlbuild-config/src
- effective
- project
- _helpers
- tests
- sqlbuild-core/src/cli_text
- _helpers
- sqlbuild-host/src/entries
- _helpers
- sqlbuild-orchestration/src/warehouse_metadata
- main/location_reads
- sqlbuild-python-base/src
- records
- _helpers
- compiler_records
- contract_records
- sqlbuild-python-compile/src/compile_driver/_helpers/setup
- sqlbuild-python-front/src/discovery
- _helpers
- records
- snapshots
- main/records
- sqlbuild-python-planner/src
- live_reads
- _helpers
- main
- plan_output/_helpers
- planner_migrations
- _helpers
- discovery
- model
- planning
- main
- planner_phases
- planner_warehouse
- _helpers/inspection
- sqlbuild-python-project/src
- compiled_project
- _helpers
- main
- graph
- _helpers/python_nodes
- main/python_nodes
- native_host
- _helpers
- main
- semantic_shapes/_helpers
- sqlbuild-python/src/bindings/_helpers
- build_command
- compile_driver
- location_command
- plan_command
- src/sqlbuild
- .agents/skills/sqlbuild/references/docs
- cli
- concepts
- python-nodes
- adapters/snowflake
- _helpers
- classes
- adapter/relations
- classes
- compiler/discovery/_helpers/validation
- executor/python_nodes/_helpers
- python_nodes
- _helpers
- main
- tests
- e2e/src/sqlbuild/cli/commands/main/plan
- goldens/compiler/failure
- integration/src/sqlbuild/compiler
- pipeline
- planner
- _helpers
- main
- unit/src/sqlbuild
- adapters/snowflake/inspection
- adapter/relations
- python_nodes/decorators
- website/src/content/docs/docs
- cli
- concepts
- python-nodes
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
38 | | - | |
39 | | - | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
40 | 42 | | |
41 | 43 | | |
42 | | - | |
| 44 | + | |
43 | 45 | | |
44 | 46 | | |
45 | 47 | | |
| |||
51 | 53 | | |
52 | 54 | | |
53 | 55 | | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
54 | 63 | | |
55 | 64 | | |
56 | 65 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| 16 | + | |
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
| |||
89 | 90 | | |
90 | 91 | | |
91 | 92 | | |
| 93 | + | |
92 | 94 | | |
93 | 95 | | |
94 | 96 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
42 | | - | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
43 | 45 | | |
44 | 46 | | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
49 | 51 | | |
50 | 52 | | |
51 | 53 | | |
| |||
78 | 80 | | |
79 | 81 | | |
80 | 82 | | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
81 | 86 | | |
82 | 87 | | |
83 | 88 | | |
| |||
93 | 98 | | |
94 | 99 | | |
95 | 100 | | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
96 | 113 | | |
97 | 114 | | |
98 | 115 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
490 | 490 | | |
491 | 491 | | |
492 | 492 | | |
| 493 | + | |
| 494 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
| 28 | + | |
27 | 29 | | |
28 | 30 | | |
29 | 31 | | |
| |||
0 commit comments