Skip to content

Commit 3b34bce

Browse files
authored
feat!: require explicit, typed references in macros, hooks and Python nodes (#858)
* feat!: accept typed reference arguments in macros and reject macro-generated references Macros now accept __ref(), __source() and __seed() as arguments, including inside lists and dictionaries, and receive sqlbuild.refs typed values. The reference renders at the call site, so the graph, lineage and scopes see it as written in the model. References that appear in a macro's expanded output fail compile with P006 unless [references] enforce_explicit = false. BREAKING CHANGE: macros that emit __ref(), __source() or __seed() in their output now fail compile; pass the reference in as a typed argument. * feat!: add Python hook reads and reject hard-coded project relation names @hook(reads=...) declares the models, sources and seeds a Python hook reads; ctx.relation(...) resolves them and raises for undeclared refs. Declared reads order the hooked model after the resources it reads, and a read that depends on the hooked model fails compile with P007. Literal SQL passed to ctx.query or ctx.execute_sql in tasks, assets and hooks that names a project relation fails compile with P008. At run time the same check parses SQL in the adapter dialect and records a P008 warning in run output, the summary and the JSON run results; the query still runs. [references] enforce_explicit = false disables both. Build summaries now print warning details before the completion line and count a node that succeeded with warnings as WARN. BREAKING CHANGE: Python tasks, assets and hooks that hard-code project relation names in literal SQL now fail compile; declare the dependency and use ctx.relation(...). * docs: document typed macro references, hook reads and [references] enforce_explicit * feat!: accept seed() and SQL references in checks, and check loader and check SQL Tasks, assets and checks may declare model(), source() and seed() in depends_on and resolve them with ctx.relation(...). Seed dependencies order read-side tasks after the seed, appear as edges in sqb dag --json and Dagster, and are validated by sqb check. A check with SQL dependencies runs at the end of sqb build when those resources ran in the build, and its checked assets include them. Selector expansion now crosses the SQL/Python boundary: +task:x also selects the models, sources and seeds x declares and their upstream, and x+ on a SQL resource also selects the tasks and assets that read it. Literal SQL in checks and loaders that names a project relation now fails compile with P008, and dynamic SQL produces a P008 run-time warning, as for tasks and assets. Loader help points to ctx.source(...) or ctx.loader(...), and a model or seed named in loader SQL is reported as unreadable because loaders run before models and seeds. ctx.source() and ctx.loader() count as resolved. Unqualified matches add a help to qualify an external table that shares the name. Loader warnings appear in the build summary and in the JSON run results. BREAKING CHANGE: Python checks and loaders that hard-code project relation names in literal SQL now fail compile; use ctx.relation(...) in checks and ctx.source(...) or ctx.loader(...) in loaders. BREAKING CHANGE: +<python node> selectors now include the SQL resources the node declares, and <sql resource>+ selectors include the tasks and assets that declare it. * docs: document seed and SQL references in checks, and loader relation rules Checks may depend on models, sources and seeds; tasks and assets may depend on seeds; selector expansion follows declared SQL references; loaders read upstream relations through ctx.source() and ctx.loader(); Dagster does not add hook reads to its asset graph. * fix: report every P006, P007 and P008 violation in one compile Macro-generated references, hook reads of a hook's own dependant, and hard-coded relation names in Python SQL now become compile diagnostics instead of stopping at the first one. One compile reports each violation once (per consumer, macro and reference for P006; per hook read for P007; per node, line and relation for P008) in deterministic order, in the terminal and in sqb compile --json with has_errors. Commands that plan or build still fail before planning and list every error. Re-expanding model SQL with mocked macros for SQL tests no longer checks macro-generated references again; the model's own expansion reports them. * fix: report every P004, P005 and scope placement violation in one compile Single-model singular audits (P004), attached audits reading a resource built from their target (P005), declarations not visible from their consumer (S006), and declaration placement errors (S007, S008, S009, S010, S024) now become individual compile diagnostics instead of stopping at the first one or folding every scope finding into one P001. Each carries its file path and, where it has one, the model or audit name, so the terminal marks the right rows FAIL and sqb compile --json reports every error with has_errors. Commands that plan or build still fail before planning and list every error. Project-loading failures stay fatal: audits/ and _sqlbuild/ layout errors, unknown seed YAML keys, and declaration-index parse, name and duplicate errors. * fix: expand both-way selectors across the SQL and Python boundary from the selected roots +x+ selected the downstream of x's upstream, so a task or model selector also ran sibling tasks that read x's upstream. Upstream and downstream expansion now both start from the selected resources, so +x+ is x, its ancestors and its descendants, and SQL-only selections are unchanged. * fix: resolve Python reads of managed sources through source deferral Python hooks resolved ctx.relation(source(...)) from the plan's load map, so under defer_sources_to a hook read the local source while the model it ran on read the deferred one. Tasks, assets and checks only deferred a source when a selected SQL model also read it, and sqb check never deferred. The plan now carries the source entries Python reads resolve, each as a selected SQL read of that source sees it. Hooks, tasks, assets, checks, the check existence guard and the hard-coded relation guard use it, and sqb check plans with source deferral like sqb build. * fix: report P006 for references a macro constructs itself Every SqlResourceRef formatted while a macro ran rendered as an approved call-site placeholder, so a macro that built model("orders") itself compiled without P006. Only references written as arguments at the call site, including inside nested macro arguments, render as approved placeholders now; any other reference renders as its reference call and is reported as macro-generated, or becomes an ordinary reference with [references] enforce_explicit = false. * fix: record every typed macro argument as a dependency of its caller A reference passed to a macro became a dependency only if the macro rendered it, so @unused(__ref("orders")) or an unchosen dictionary or list entry left the caller without that dependency. Every __ref(), __source() and __seed() written as a macro argument, including inside nested macro arguments, is now a dependency of the calling model, audit or function, so it orders, selects and appears in the DAG like a rendered reference. * fix: check literal SQL passed to ctx.query and ctx.execute_sql by keyword Compile-time P008 only inspected the first positional argument, so ctx.query(sql="...") and ctx.execute_sql(sql="...") skipped the hard-coded project relation check. The sql keyword and a literal **{"sql": ...} unpack are now checked like the positional form. * feat: resolve references in SQL hooks and treat them as hook reads __ref(), __source() and __seed() in inline and named SQL hook SQL, written directly or passed as typed macro arguments, previously compiled without being recorded and reached the warehouse unresolved. They now resolve at plan time exactly like references in model SQL, including adapter quoting, source deferral, --defer and target overrides. Each reference is also a hook read with the same rules as Python @hook(reads=...): the hooked model runs after the resources it reads when both are in the build, reads never change selection, a read of a resource built from the hooked model fails with P007, and an unknown reference fails to compile. Macro-generated references in hook SQL still fail with P006.
1 parent 5bde694 commit 3b34bce

170 files changed

Lines changed: 7026 additions & 554 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Makefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,12 +69,15 @@ E2E_DUCKDB_BUILD_CORE_PATHS := \
6969
tests/e2e/src/sqlbuild/cli/commands/main/build/test_dag_json_behavior.py \
7070
tests/e2e/src/sqlbuild/cli/commands/main/build/test_enum_contract.py \
7171
tests/e2e/src/sqlbuild/cli/commands/main/build/test_execution_limits.py \
72+
tests/e2e/src/sqlbuild/cli/commands/main/build/test_explicit_references.py \
7273
tests/e2e/src/sqlbuild/cli/commands/main/build/test_expression_sources.py \
7374
tests/e2e/src/sqlbuild/cli/commands/main/build/test_lifecycle_commands.py \
7475
tests/e2e/src/sqlbuild/cli/commands/main/build/test_manifest_artifact_gating.py \
7576
tests/e2e/src/sqlbuild/cli/commands/main/build/test_no_tests_no_audits_flags.py \
7677
tests/e2e/src/sqlbuild/cli/commands/main/build/test_plan_command_surface.py \
7778
tests/e2e/src/sqlbuild/cli/commands/main/build/test_python_hooks.py \
79+
tests/e2e/src/sqlbuild/cli/commands/main/build/test_python_source_deferral.py \
80+
tests/e2e/src/sqlbuild/cli/commands/main/build/test_sql_hook_reads.py \
7881
tests/e2e/src/sqlbuild/cli/commands/main/build/test_python_node_identity.py \
7982
tests/e2e/src/sqlbuild/cli/commands/main/build/test_sql_hook_identity.py \
8083
tests/e2e/src/sqlbuild/cli/commands/main/build/test_query_change_tracking.py \

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -88,10 +88,10 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in
8888
- [Loaders](concepts/python-nodes/loaders.md) (`concepts/python-nodes/loaders`) - Load external data into source tables with Python functions.
8989
- [Tasks](concepts/python-nodes/tasks.md) (`concepts/python-nodes/tasks`) - Run Python computation and side effects as nodes in the SQLBuild graph.
9090
- [Assets](concepts/python-nodes/assets.md) (`concepts/python-nodes/assets`) - Produce or observe external artifacts as nodes in the SQLBuild graph.
91-
- [Checks](concepts/python-nodes/checks.md) (`concepts/python-nodes/checks`) - Validate tasks, assets, and loaders with Python checks.
91+
- [Checks](concepts/python-nodes/checks.md) (`concepts/python-nodes/checks`) - Validate tasks, assets, loaders, and SQL relations with Python checks.
9292
- [Factories](concepts/python-nodes/factories.md) (`concepts/python-nodes/factories`) - Generate Python nodes programmatically with @factory.
9393
- [Providers](concepts/python-nodes/providers.md) (`concepts/python-nodes/providers`) - Shared runtime services for Python nodes and hooks.
94-
- [SQL References](concepts/python-nodes/sql-references.md) (`concepts/python-nodes/sql-references`) - Read SQL models and sources from Python nodes without creating SQL dependencies.
94+
- [SQL References](concepts/python-nodes/sql-references.md) (`concepts/python-nodes/sql-references`) - Read SQL models, sources, and seeds from Python nodes without creating SQL dependencies.
9595

9696
## Integrations
9797

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/check.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,4 +51,4 @@ Results are written to `target/run/checks/python_checks.json`.
5151

5252
## Relationship to build
5353

54-
`sqb build` runs relevant Python checks by default after the tasks, assets, and loaders they depend on have run. `sqb build --no-audits` skips checks alongside audits. `sqb check` runs only checks, on demand.
54+
`sqb build` runs relevant Python checks by default after the tasks, assets, loaders, models, sources, and seeds they depend on have run. `sqb build --no-audits` skips checks alongside audits. `sqb check` runs only checks, on demand.

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ Online: https://sqlbuild.com/docs/concepts/macros/
1111
- Create a macro
1212
- Call a macro from SQL
1313
- Arguments
14+
- Typed reference arguments
1415
- Keep implementation details private
1516
- Use macros in tests
1617
- Use macros in hooks
@@ -77,6 +78,8 @@ Macro calls accept Python literal values:
7778
- Dictionaries: `{"key": "value"}`
7879
- `None`
7980
- The result of another macro call
81+
- `__ref("model")`, `__source("source")`, and `__seed("seed")` references, alone or inside lists
82+
and dictionaries
8083

8184
Positional and keyword arguments are supported:
8285

@@ -87,6 +90,48 @@ Positional and keyword arguments are supported:
8790
Use quoted strings when passing SQL expressions such as column names. The macro decides how to
8891
place that text into its returned SQL.
8992

93+
## Typed reference arguments
94+
95+
Pass the models, sources, and seeds a macro reads as reference arguments:
96+
97+
```sql
98+
SELECT * FROM @union_all([__ref("orders_eu"), __ref("orders_us")])
99+
```
100+
101+
The macro receives each reference as a typed `SqlResourceRef` from `sqlbuild.refs`, the same type
102+
Python nodes use. It has `kind` and `name` attributes, and formatting it into SQL renders the
103+
relation with the adapter's quoting:
104+
105+
```python
106+
# models/marts/_sqlbuild/_macros/unions.py
107+
from sqlbuild.refs import SqlResourceRef
108+
109+
def union_all(relations: list[SqlResourceRef]) -> str:
110+
return " UNION ALL ".join(f"SELECT order_id FROM {relation}" for relation in relations)
111+
```
112+
113+
The dependency belongs to the model that makes the call. The graph, column lineage, and
114+
declaration scopes treat `@union_all([__ref("orders_eu"), __ref("orders_us")])` exactly like a
115+
model that wrote `__ref("orders_eu")` and `__ref("orders_us")` itself.
116+
117+
### References in macro output
118+
119+
A macro must not produce references on its own. Any `__ref()`, `__source()`, or `__seed()` in a
120+
macro's expanded output fails to compile with `P006`, whether the macro's Python code wrote it or
121+
it arrived as a quoted string argument such as `@wrap('__ref("stg_orders")')`:
122+
123+
```
124+
error[P006]: model:order_summary depends on model:stg_orders through macro orders_base()
125+
--> models/marts/_sqlbuild/_macros/orders.py
126+
= help: write the reference in the model, or pass it in: @orders_base(__ref("stg_orders"))
127+
= help: while migrating a project, allow macro-generated references with [references] enforce_explicit = false in sqlbuild_project.toml
128+
```
129+
130+
A dependency hidden inside a macro is invisible to reviewers of the model file and bypasses
131+
declaration scopes. For projects migrating from other tools,
132+
[`[references] enforce_explicit = false`](project-configuration.md#references) allows
133+
macro-generated references project-wide.
134+
90135
## Keep implementation details private
91136

92137
Only public functions owned by the file are exported as macros. Prefix helpers, constants, classes,

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks/python.md‎

Lines changed: 53 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,17 @@
66
77
Online: https://sqlbuild.com/docs/concepts/models/hooks/python/
88

9+
## Contents
10+
11+
- Define a Python hook
12+
- Declared reads
13+
- Invoke a Python hook
14+
- Signature validation
15+
- Hook context
16+
- Providers
17+
- Skips and failures
18+
- Identity and diagnostics
19+
920
Python hooks run model lifecycle logic that needs Python control flow, providers, warehouse queries, logging, or explicit skip decisions.
1021

1122
For shared lifecycle ordering, failure timing, naming, and identity rules, see the [Hooks overview](../hooks.md).
@@ -29,7 +40,47 @@ def notify_complete(ctx, channel="#data-builds"):
2940
)
3041
```
3142

32-
By default, the hook name is the function name. The decorator accepts optional `name` and `description` arguments. A Python file may define multiple decorated hooks.
43+
By default, the hook name is the function name. The decorator accepts optional `name`, `description`, and `reads` arguments. A Python file may define multiple decorated hooks.
44+
45+
## Declared reads
46+
47+
A hook that queries other models, sources, or seeds declares them with `reads` and resolves each
48+
one with `ctx.relation(...)`:
49+
50+
```python
51+
from sqlbuild.hooks import hook
52+
from sqlbuild.refs import model
53+
54+
@hook(reads=model("dim_customers"))
55+
def refresh_lookup(ctx):
56+
customers = ctx.relation(model("dim_customers"))
57+
ctx.execute_sql(f"CREATE OR REPLACE TABLE customer_lookup AS SELECT * FROM {customers}")
58+
```
59+
60+
`reads` accepts one reference or a list of `model(...)`, `source(...)`, and `seed(...)`
61+
references from `sqlbuild.refs`. `ctx.relation(...)` returns the adapter-qualified relation for
62+
the current run, following deferral like SQL references do. Resolving a reference that is not
63+
declared raises, as it does for [Python nodes](../../python-nodes/sql-references.md).
64+
65+
Declared reads order the build: a model that runs the hook waits for the resources the hook reads
66+
when both are in the build. In a partial build, unselected reads are not built, and the hook reads
67+
their existing relations. A hook cannot read a resource built from the model that runs it, because
68+
that model would have to wait for its own dependants; that fails to compile with `P007`. Reading
69+
an unknown model, source, or seed fails to compile. Use `ctx.destination` for the hook's own model.
70+
71+
The Dagster integration does not add declared reads to its asset graph, the same as reads of
72+
attached audits. Within one `sqb build` the ordering holds; when Dagster runs the hooked model in a
73+
separate build from the resources it reads, the hook reads their existing relations.
74+
75+
Declaring reads does not change who may call the hook: the model must still see the hook under the
76+
usual [scope rules](../hooks.md#project-layout).
77+
78+
SQL hooks declare reads by referencing resources in their SQL; see
79+
[Read models, sources, and seeds](sql.md#read-models-sources-and-seeds).
80+
81+
Literal project relation names in `ctx.query()` and `ctx.execute_sql()` SQL fail to compile, and
82+
dynamically built names produce run-time warnings; see
83+
[Hard-coded relation names](../../python-nodes/sql-references.md#hard-coded-relation-names).
3384

3485
Files named `__init__.py` or beginning with `_` are skipped. Imported decorated functions are not registered again from the importing module.
3586

@@ -73,6 +124,7 @@ Python hooks declare a `HookContext` parameter named `ctx`, `context`, `_ctx`, o
73124
| `ctx.target` | Active target |
74125
| `ctx.vars` | Effective project variables |
75126
| `ctx.destination` | Destination relation metadata |
127+
| `ctx.relation(ref)` | Adapter-qualified relation for a [declared read](#declared-reads) |
76128
| `ctx.adapter_name` | Active adapter name |
77129
| `ctx.adapter` | Adapter instance |
78130
| `ctx.connection` | Live connection |

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks/sql.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ Online: https://sqlbuild.com/docs/concepts/models/hooks/sql/
1313
- SQL hook arguments
1414
- Inline SQL hooks
1515
- Compile-time context
16+
- Read models, sources, and seeds
1617
- Validation and diagnostics
1718

1819
SQL hooks submit one rendered SQL payload to the adapter before or after model materialization. Use a named SQL hook for reusable behavior and `inline_sql(...)` for short model-specific payloads.
@@ -163,6 +164,46 @@ LANGUAGE SQL
163164
AS 'SELECT 1'
164165
```
165166

167+
## Read models, sources, and seeds
168+
169+
SQL hooks reference other resources the same way model SQL does, with `__ref("...")`,
170+
`__source("...")`, and `__seed("...")`, written directly or passed as typed macro arguments:
171+
172+
```sql
173+
MODEL (
174+
materialized table,
175+
pre_hooks [
176+
inline_sql('DELETE FROM @@CTX:destination.qualified WHERE order_id IN (SELECT order_id FROM __ref("cancelled_orders"))'),
177+
],
178+
post_hooks [sql("record_counts")],
179+
);
180+
181+
SELECT order_id FROM __ref("stg_orders")
182+
```
183+
184+
**`models/marts/_sqlbuild/_hooks/sql/record_counts.sql`**
185+
186+
```sql
187+
HOOK ();
188+
189+
@count_rows(__ref("stg_orders"))
190+
```
191+
192+
At plan time each reference resolves exactly like one in model SQL: adapter quoting, source
193+
deferral, `--defer`, and target overrides all apply. Every reference is also a hook read, with the
194+
same rules as [declared Python hook reads](python.md#declared-reads):
195+
196+
- The model that runs the hook waits for the resources it reads when both are in the build. Reads
197+
do not change selection; in a partial build the hook reads the existing relations.
198+
- A hook cannot read a resource built from its own model; that fails to compile with `P007`. Use
199+
`@@CTX:destination.qualified` for the hook's own model.
200+
- Referencing an unknown model, source, or seed fails to compile.
201+
- A reference passed to a macro is a read even when the macro does not render it. A reference a
202+
macro writes into its output instead fails with `P006`, as in model SQL.
203+
204+
The model must still see a named hook under the usual
205+
[scope rules](../hooks.md#project-layout).
206+
166207
## Validation and diagnostics
167208

168209
SQLBuild strictly validates hook resource syntax, arguments, interpolation, macros, metadata, and non-empty payloads. It does not classify executable statement kinds or implement vendor SQL grammar. Each rendered hook payload is passed to the adapter in one `execute` call; whether a driver accepts multiple statements or client-side batch separators is adapter and warehouse behavior. Use separate hook entries when portable ordering between statements matters.

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ Online: https://sqlbuild.com/docs/concepts/project-configuration/
1616
- Settings
1717
- Rules
1818
- Scopes
19+
- References
1920
- Project variables
2021
- Janitor
2122
- Scenario
@@ -430,6 +431,31 @@ enforce_placement = false
430431
Visibility errors are unaffected. See
431432
[Where to Put Declarations](declaration-scopes/placement.md).
432433

434+
## References
435+
436+
Every reference to a model, source, or seed must be explicit and typed. By default:
437+
438+
- A macro whose expanded output contains `__ref()`, `__source()`, or `__seed()` fails to compile
439+
with `P006`. Pass the reference to the macro as an argument instead; see
440+
[Typed reference arguments](macros.md#typed-reference-arguments).
441+
- Literal SQL passed to `ctx.query()` or `ctx.execute_sql()` in a task, asset, check, loader, or
442+
Python hook that names a project relation fails to compile with `P008`.
443+
- The same check runs on SQL sent at run time and reports a `P008` warning; the query still runs.
444+
See [Hard-coded relation names](python-nodes/sql-references.md#hard-coded-relation-names).
445+
446+
One compile reports every `P006`, `P007`, and `P008` violation in the project, in the terminal and in
447+
`sqb compile --json`.
448+
449+
While migrating a project from another tool, turn these checks off for the whole project:
450+
451+
```toml
452+
[references]
453+
enforce_explicit = false # default true
454+
```
455+
456+
The switch has no per-model or per-folder form. Undeclared `ctx.relation(...)` calls still fail,
457+
and declared hook reads still order the build.
458+
433459
## Project variables
434460

435461
Variables are simple string substitutions available in model SQL via the `@@name` syntax:

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/assets.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,7 @@ See [Tasks](tasks.md#retries) for the full retry policy fields.
124124
| Parameter | Description |
125125
|-----------|-------------|
126126
| `name` | Override the node name (defaults to the function name) |
127-
| `depends_on` | Upstream nodes (function, tuple, or list); `model()`/`source()` for read-only SQL refs |
127+
| `depends_on` | Upstream nodes (function, tuple, or list); `model()`/`source()`/`seed()` for read-only SQL refs |
128128
| `columns` | Column declarations for the produced artifact |
129129
| `column_lineage` | Map of asset column to upstream `{node, column}` references |
130130
| `tags` | Labels for selection and grouping |

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/checks.md‎

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
# Checks
44

5-
> Validate tasks, assets, and loaders with Python checks.
5+
> Validate tasks, assets, loaders, and SQL relations with Python checks.
66
77
Online: https://sqlbuild.com/docs/concepts/python-nodes/checks/
88

@@ -16,7 +16,7 @@ Online: https://sqlbuild.com/docs/concepts/python-nodes/checks/
1616
- Running checks
1717
- Checks vs audits
1818

19-
Checks are Python nodes that validate other Python nodes. They are the Python analog of SQL [audits](../audits.md): audits validate SQL relations, checks validate the output of tasks, assets, and loaders. See [Python Nodes](overview.md) for the shared model.
19+
Checks are Python nodes that validate the output of tasks, assets, and loaders, or the data in SQL models, sources, and seeds. They are the Python analog of SQL [audits](../audits.md). See [Python Nodes](overview.md) for the shared model.
2020

2121
Checks are separate graph nodes, not callbacks embedded in a task or asset. A check declares what it validates through `depends_on`.
2222

@@ -39,6 +39,21 @@ def check_orders_exported(ctx: CheckContext):
3939

4040
The check receives a `CheckContext` and reads its dependencies' persisted results with `ctx.result_of(...)`.
4141

42+
A check can also validate SQL relations. Declare them with `model()`, `source()`, or `seed()` from `sqlbuild.refs`, and resolve them with `ctx.relation(...)`:
43+
44+
```python
45+
from sqlbuild.checks import check
46+
from sqlbuild.refs import model
47+
48+
@check(depends_on=model("fact_orders"))
49+
def orders_present(ctx):
50+
orders = ctx.relation(model("fact_orders"))
51+
count = ctx.query(f"SELECT count(*) FROM {orders}").fetchone()[0]
52+
return ctx.pass_() if count else ctx.fail("no orders")
53+
```
54+
55+
Do not hard-code relation names in check SQL; see [Hard-coded relation names](sql-references.md#hard-coded-relation-names).
56+
4257
## Results
4358

4459
Return a result through the context helpers, or a bool shorthand:
@@ -79,7 +94,7 @@ def orders_freshness(ctx):
7994
## What checks can depend on
8095

8196
- Checks may depend on **tasks, assets, and loaders**.
82-
- Checks may **not** depend on SQL models, sources, seeds, or functions. Use SQL [audits](../audits.md) to validate SQL relations.
97+
- Checks may depend on SQL **models, sources, and seeds** through `model()`, `source()`, and `seed()` references. They may **not** depend on functions.
8398
- Checks may **not** depend on other checks.
8499
- Checks may **not** depend on a terminal source loader directly. Validate loaded source data with a source audit instead.
85100

@@ -89,7 +104,7 @@ A check that depends on a single node is displayed grouped under that node. Mult
89104

90105
| Parameter | Description |
91106
|-----------|-------------|
92-
| `depends_on` | Required. Tasks/assets/loaders to validate (function, tuple, or list) |
107+
| `depends_on` | Required. Tasks/assets/loaders to validate, and `model()`/`source()`/`seed()` references (function, reference, tuple, or list) |
93108
| `name` | Override the node name (defaults to the function name) |
94109
| `severity` | `error` (default) or `warn` |
95110
| `tags` | Labels for selection and grouping |
@@ -101,7 +116,7 @@ Checks do not support `columns`, `column_lineage`, or `retry`.
101116

102117
## Running checks
103118

104-
Checks run automatically during `sqb build` when their Python dependencies run. They are skipped when `--no-audits` is passed. To run checks on their own, use [`sqb check`](../../cli/check.md):
119+
Checks run automatically at the end of `sqb build` when all their Python and SQL dependencies ran in that build. They are skipped when `--no-audits` is passed. To run checks on their own, use [`sqb check`](../../cli/check.md):
105120

106121
```bash
107122
# Run all checks
@@ -114,13 +129,13 @@ sqb check --select +check_orders_exported
114129
sqb check --select tag:exports
115130
```
116131

117-
`sqb check` rejects selecting non-check nodes; use `sqb build` to run tasks and assets. Check results are written to `target/run/checks/python_checks.json`, and `sqb check --json` prints them to stdout.
132+
`sqb check` rejects selecting non-check nodes; use `sqb build` to run tasks and assets. The models, sources, and seeds a check depends on must already exist; `sqb check` does not build them. Check results are written to `target/run/checks/python_checks.json`, and `sqb check --json` prints them to stdout.
118133

119134
## Checks vs audits
120135

121136
| | Checks | Audits |
122137
|---|--------|--------|
123-
| Validates | Python tasks, assets, loaders | SQL relations |
138+
| Validates | Python tasks, assets, loaders, and SQL relations | SQL relations |
124139
| Authored in | `python/` (Python) | `MODEL()` headers / `audits/` (SQL) |
125140
| Run by | `sqb build`, `sqb check` | `sqb build`, `sqb audit` |
126141
| Severity | `error`, `warn` | `error`, `warn` |

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/loaders.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,6 +268,8 @@ def raw_account_metrics(ctx: LoaderContext) -> list[dict[str, object]]:
268268
]
269269
```
270270

271+
Read upstream relations through `ctx.loader(...)` and `ctx.source(...)`, not by writing their table names in SQL. SQLBuild rejects literal SQL that names a project source, model, or seed with `P008`, and warns at run time when dynamic SQL does; loaders run before models and seeds and cannot read them. See [Hard-coded relation names](sql-references.md#hard-coded-relation-names).
272+
271273
Dependencies form a DAG. SQLBuild schedules loaders in topological order and executes independent loaders concurrently when `--concurrency` is set.
272274

273275
Intermediate loaders (those referenced only via `depends_on`, with no managed source of the same name) are given synthetic source entries and write to `__loader__<name>` tables by default. Only the terminal loader - the one whose name matches a managed source - populates that source; intermediate loaders feed it. Use the `destination` parameter on the decorator to override the intermediate relation:

0 commit comments

Comments
 (0)