Repository navigation
Commit 2909eda
authored
feat: publish Python nodes for jobs, roles and server runs (#1472)
* test: reproduce Python selectors in published job selections
* wip: resolve published Python selectors with the CLI's unified resolver
Work in progress, not ready for review.
- Publish-time job, sensor-condition and role selections resolve with the
same unified SQL/Python resolver `sqb build --select` uses.
- Python snapshot nodes publish deps (Python and SQL) and tags; an
intermediate loader publishes the source node it writes, and loader
chains list it in `loads`.
- Server-side ad-hoc selection rebuilds the Python selection graph from the
snapshot and resolves with the same resolver.
- The worker selects Python nodes by typed key.
Validated so far: test_orchestration_snapshot.py passes (99 tests,
including the Python selector reproduction). The Rust unit tests for the
ad-hoc planner and planner scope have not finished yet; no review and no
full gate.
* feat: let a selected Python node read an unselected upstream's result
A Python node can now be selected without its Python upstream in every
path that uses the shared resolver: sqb build --select, published jobs,
sensors, roles and ad-hoc runs. It reads the upstream's latest persisted
result, and fails at run time with an error naming the upstream and the
command to build it when the upstream has never succeeded.
An intermediate loader without a write_strategy that returns rows now
writes them to __loader__<name> as a table, as the loaders docs describe.
A self-managed intermediate that returns nothing keeps its behaviour.
The published snapshot rebuilds the Python selection graph from node deps,
and job groups order jobs across Python and Python-to-SQL edges.
Validated locally: python_selection and adhoc planner Rust tests, clippy
on both crates, fensu check, the worker claim tests, orchestration snapshot,
Python node selection E2E, load E2E (sqb load and sqb build on the docs
example) and the python_nodes and executor/load unit tests. Not yet run:
make check-ci, the full suite, review.
* feat: select Python-only jobs and SQL tests across CLI and server runs
- `job:` and bare job names resolve jobs through the Python-aware resolver
publication uses, so a job that selects only Python nodes builds them.
- Server ad-hoc runs accept `test:` selectors; a selected SQL test reports
as a run node rather than as an attached check.
- A provider function whose context is positional-only (`ctx, /`) receives
it by position instead of failing at run time.
- A role must grant the intermediate loader a job loads to run that job.
* fix: keep Python selections runnable on workers and require checked nodes
Selections now publish a managed source instead of its loader, ad-hoc
test, audit, seed and freshness runs resolve SQL only, a selected SQL
test reports its own outcome, and a selected model folds in its failing
test. A selector that matches a Python check must also select the nodes
the check depends on, in jobs, roles and the CLI alike.
* fix: refuse selecting a Python check on a source no loader writes
A build never loads such a source, so a matched check on it would be
accepted and then never run. The resolver now names the check and the
source and points to sqb check.
* refactor: share upstream result reads between loader and node contexts
Loader and Python node contexts read upstream results through one
mixin, and the selection entry points build their context in one place.
Regenerates the packaged docs skill for the selector and check pages.1 parent 4502c30 commit 2909eda
77 files changed
Lines changed: 2152 additions & 361 deletions
File tree
- benchmarks/duckshop/edge_cases
- loaders
- orchestration
- crates
- sqlbuild-analysis/src
- graph
- _helpers
- main
- planner_scope
- main
- tests
- python_selection
- _helpers
- main
- tests
- sqlbuild-cli/src/adhoc
- _helpers
- tests
- sqlbuild-orchestration/src/runs
- _helpers/requests
- sqlbuild-python/src/bindings/_helpers
- compiled_project
- graph
- location_command
- native_host
- src/sqlbuild
- .agents/skills/sqlbuild/references/docs/concepts
- python-nodes
- executor
- load
- _helpers
- main
- node_results
- classes
- python_nodes
- _helpers
- provider/_helpers
- tests
- e2e/src/sqlbuild/cli/commands/main
- build
- no_tests_no_audits
- load
- integration/src/sqlbuild/compiler/compiled_project
- publication
- person
- unit/src/sqlbuild
- compiler/python_nodes/_helpers
- provider/_helpers
- website/src/content/docs/docs/concepts
- python-nodes
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
2 | 5 | | |
3 | 6 | | |
4 | 7 | | |
5 | 8 | | |
6 | | - | |
| 9 | + | |
7 | 10 | | |
8 | 11 | | |
9 | 12 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
12 | 19 | | |
13 | 20 | | |
| 21 | + | |
14 | 22 | | |
15 | 23 | | |
16 | 24 | | |
17 | 25 | | |
18 | 26 | | |
19 | | - | |
20 | | - | |
21 | | - | |
| 27 | + | |
22 | 28 | | |
23 | 29 | | |
24 | 30 | | |
| |||
0 commit comments