Repository navigation
Commit 3b34bce
authored
feat!: require explicit, typed references in macros, hooks and Python nodes (#858)
* feat!: accept typed reference arguments in macros and reject macro-generated references
Macros now accept __ref(), __source() and __seed() as arguments, including
inside lists and dictionaries, and receive sqlbuild.refs typed values. The
reference renders at the call site, so the graph, lineage and scopes see it
as written in the model. References that appear in a macro's expanded output
fail compile with P006 unless [references] enforce_explicit = false.
BREAKING CHANGE: macros that emit __ref(), __source() or __seed() in their
output now fail compile; pass the reference in as a typed argument.
* feat!: add Python hook reads and reject hard-coded project relation names
@hook(reads=...) declares the models, sources and seeds a Python hook
reads; ctx.relation(...) resolves them and raises for undeclared refs.
Declared reads order the hooked model after the resources it reads, and a
read that depends on the hooked model fails compile with P007.
Literal SQL passed to ctx.query or ctx.execute_sql in tasks, assets and
hooks that names a project relation fails compile with P008. At run time
the same check parses SQL in the adapter dialect and records a P008
warning in run output, the summary and the JSON run results; the query
still runs. [references] enforce_explicit = false disables both.
Build summaries now print warning details before the completion line and
count a node that succeeded with warnings as WARN.
BREAKING CHANGE: Python tasks, assets and hooks that hard-code project
relation names in literal SQL now fail compile; declare the dependency and
use ctx.relation(...).
* docs: document typed macro references, hook reads and [references] enforce_explicit
* feat!: accept seed() and SQL references in checks, and check loader and check SQL
Tasks, assets and checks may declare model(), source() and seed() in
depends_on and resolve them with ctx.relation(...). Seed dependencies
order read-side tasks after the seed, appear as edges in sqb dag --json
and Dagster, and are validated by sqb check. A check with SQL
dependencies runs at the end of sqb build when those resources ran in the
build, and its checked assets include them.
Selector expansion now crosses the SQL/Python boundary: +task:x also
selects the models, sources and seeds x declares and their upstream, and
x+ on a SQL resource also selects the tasks and assets that read it.
Literal SQL in checks and loaders that names a project relation now fails
compile with P008, and dynamic SQL produces a P008 run-time warning, as
for tasks and assets. Loader help points to ctx.source(...) or
ctx.loader(...), and a model or seed named in loader SQL is reported as
unreadable because loaders run before models and seeds. ctx.source() and
ctx.loader() count as resolved. Unqualified matches add a help to
qualify an external table that shares the name. Loader warnings appear in
the build summary and in the JSON run results.
BREAKING CHANGE: Python checks and loaders that hard-code project relation
names in literal SQL now fail compile; use ctx.relation(...) in checks and
ctx.source(...) or ctx.loader(...) in loaders.
BREAKING CHANGE: +<python node> selectors now include the SQL resources
the node declares, and <sql resource>+ selectors include the tasks and
assets that declare it.
* docs: document seed and SQL references in checks, and loader relation rules
Checks may depend on models, sources and seeds; tasks and assets may depend
on seeds; selector expansion follows declared SQL references; loaders read
upstream relations through ctx.source() and ctx.loader(); Dagster does not
add hook reads to its asset graph.
* fix: report every P006, P007 and P008 violation in one compile
Macro-generated references, hook reads of a hook's own dependant, and
hard-coded relation names in Python SQL now become compile diagnostics
instead of stopping at the first one. One compile reports each violation
once (per consumer, macro and reference for P006; per hook read for P007;
per node, line and relation for P008) in deterministic order, in the
terminal and in sqb compile --json with has_errors. Commands that plan or
build still fail before planning and list every error.
Re-expanding model SQL with mocked macros for SQL tests no longer checks
macro-generated references again; the model's own expansion reports them.
* fix: report every P004, P005 and scope placement violation in one compile
Single-model singular audits (P004), attached audits reading a resource
built from their target (P005), declarations not visible from their
consumer (S006), and declaration placement errors (S007, S008, S009,
S010, S024) now become individual compile diagnostics instead of stopping
at the first one or folding every scope finding into one P001. Each
carries its file path and, where it has one, the model or audit name, so
the terminal marks the right rows FAIL and sqb compile --json reports
every error with has_errors. Commands that plan or build still fail
before planning and list every error.
Project-loading failures stay fatal: audits/ and _sqlbuild/ layout
errors, unknown seed YAML keys, and declaration-index parse, name and
duplicate errors.
* fix: expand both-way selectors across the SQL and Python boundary from the selected roots
+x+ selected the downstream of x's upstream, so a task or model selector
also ran sibling tasks that read x's upstream. Upstream and downstream
expansion now both start from the selected resources, so +x+ is x, its
ancestors and its descendants, and SQL-only selections are unchanged.
* fix: resolve Python reads of managed sources through source deferral
Python hooks resolved ctx.relation(source(...)) from the plan's load map,
so under defer_sources_to a hook read the local source while the model it
ran on read the deferred one. Tasks, assets and checks only deferred a
source when a selected SQL model also read it, and sqb check never
deferred.
The plan now carries the source entries Python reads resolve, each as a
selected SQL read of that source sees it. Hooks, tasks, assets, checks,
the check existence guard and the hard-coded relation guard use it, and
sqb check plans with source deferral like sqb build.
* fix: report P006 for references a macro constructs itself
Every SqlResourceRef formatted while a macro ran rendered as an approved
call-site placeholder, so a macro that built model("orders") itself
compiled without P006. Only references written as arguments at the call
site, including inside nested macro arguments, render as approved
placeholders now; any other reference renders as its reference call and
is reported as macro-generated, or becomes an ordinary reference with
[references] enforce_explicit = false.
* fix: record every typed macro argument as a dependency of its caller
A reference passed to a macro became a dependency only if the macro
rendered it, so @unused(__ref("orders")) or an unchosen dictionary or list
entry left the caller without that dependency. Every __ref(), __source()
and __seed() written as a macro argument, including inside nested macro
arguments, is now a dependency of the calling model, audit or function,
so it orders, selects and appears in the DAG like a rendered reference.
* fix: check literal SQL passed to ctx.query and ctx.execute_sql by keyword
Compile-time P008 only inspected the first positional argument, so
ctx.query(sql="...") and ctx.execute_sql(sql="...") skipped the
hard-coded project relation check. The sql keyword and a literal
**{"sql": ...} unpack are now checked like the positional form.
* feat: resolve references in SQL hooks and treat them as hook reads
__ref(), __source() and __seed() in inline and named SQL hook SQL, written
directly or passed as typed macro arguments, previously compiled without
being recorded and reached the warehouse unresolved. They now resolve at
plan time exactly like references in model SQL, including adapter quoting,
source deferral, --defer and target overrides.
Each reference is also a hook read with the same rules as Python
@hook(reads=...): the hooked model runs after the resources it reads when
both are in the build, reads never change selection, a read of a resource
built from the hooked model fails with P007, and an unknown reference fails
to compile. Macro-generated references in hook SQL still fail with P006.1 parent 5bde694 commit 3b34bce
170 files changed
Lines changed: 7026 additions & 554 deletions
File tree
- src/sqlbuild
- .agents/skills/sqlbuild/references
- docs
- cli
- concepts
- models/hooks
- python-nodes
- cli
- commands
- _helpers
- build_execution
- build_python_nodes
- check
- classes
- output/_helpers
- compiler
- auditing/main
- compile
- _helpers
- analysis
- assembly
- attachment
- diagnostics
- explicit_references
- named_declarations
- render
- classes
- main
- dag/_helpers
- discovery
- _helpers
- filesystem
- validation
- yml
- graph
- _helpers
- main
- pipeline
- _helpers
- main
- planner
- _helpers
- command_planning
- graph
- output
- resolve
- scenario
- warehouse
- python_nodes/_helpers
- references
- _helpers
- main
- source_freshness/_helpers
- executor
- build
- _helpers
- classes
- load
- _helpers
- main
- python_nodes
- _helpers
- classes
- run
- _helpers
- execution
- materializations
- scenario/main
- python_nodes
- _helpers
- main
- spec/contracts
- tests
- e2e/src/sqlbuild/cli/commands/main/build
- no_tests_no_audits
- integration/src/sqlbuild/executor
- build/custom
- run
- incremental
- microbatch
- view
- unit/src/sqlbuild
- cli/commands
- _helpers/check
- compiler
- compile/_helpers
- discovery
- _helpers
- planner/_helpers
- planning
- resolve
- python_nodes/_helpers
- references/main
- executor
- python_nodes/classes
- run/_helpers
- website/src/content/docs/docs
- cli
- concepts
- models/hooks
- python-nodes
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
| 72 | + | |
72 | 73 | | |
73 | 74 | | |
74 | 75 | | |
75 | 76 | | |
76 | 77 | | |
77 | 78 | | |
| 79 | + | |
| 80 | + | |
78 | 81 | | |
79 | 82 | | |
80 | 83 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
91 | | - | |
| 91 | + | |
92 | 92 | | |
93 | 93 | | |
94 | | - | |
| 94 | + | |
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
54 | | - | |
| 54 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| 14 | + | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
| |||
77 | 78 | | |
78 | 79 | | |
79 | 80 | | |
| 81 | + | |
| 82 | + | |
80 | 83 | | |
81 | 84 | | |
82 | 85 | | |
| |||
87 | 90 | | |
88 | 91 | | |
89 | 92 | | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
90 | 135 | | |
91 | 136 | | |
92 | 137 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
9 | 20 | | |
10 | 21 | | |
11 | 22 | | |
| |||
29 | 40 | | |
30 | 41 | | |
31 | 42 | | |
32 | | - | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
33 | 84 | | |
34 | 85 | | |
35 | 86 | | |
| |||
73 | 124 | | |
74 | 125 | | |
75 | 126 | | |
| 127 | + | |
76 | 128 | | |
77 | 129 | | |
78 | 130 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| 16 | + | |
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
| |||
163 | 164 | | |
164 | 165 | | |
165 | 166 | | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
166 | 207 | | |
167 | 208 | | |
168 | 209 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| |||
430 | 431 | | |
431 | 432 | | |
432 | 433 | | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
433 | 459 | | |
434 | 460 | | |
435 | 461 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
124 | 124 | | |
125 | 125 | | |
126 | 126 | | |
127 | | - | |
| 127 | + | |
128 | 128 | | |
129 | 129 | | |
130 | 130 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
| 19 | + | |
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
42 | 57 | | |
43 | 58 | | |
44 | 59 | | |
| |||
79 | 94 | | |
80 | 95 | | |
81 | 96 | | |
82 | | - | |
| 97 | + | |
83 | 98 | | |
84 | 99 | | |
85 | 100 | | |
| |||
89 | 104 | | |
90 | 105 | | |
91 | 106 | | |
92 | | - | |
| 107 | + | |
93 | 108 | | |
94 | 109 | | |
95 | 110 | | |
| |||
101 | 116 | | |
102 | 117 | | |
103 | 118 | | |
104 | | - | |
| 119 | + | |
105 | 120 | | |
106 | 121 | | |
107 | 122 | | |
| |||
114 | 129 | | |
115 | 130 | | |
116 | 131 | | |
117 | | - | |
| 132 | + | |
118 | 133 | | |
119 | 134 | | |
120 | 135 | | |
121 | 136 | | |
122 | 137 | | |
123 | | - | |
| 138 | + | |
124 | 139 | | |
125 | 140 | | |
126 | 141 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
268 | 268 | | |
269 | 269 | | |
270 | 270 | | |
| 271 | + | |
| 272 | + | |
271 | 273 | | |
272 | 274 | | |
273 | 275 | | |
| |||
0 commit comments