Skip to content

feat: publish Python nodes for jobs, roles and server runs - #1472

Merged
kvlonge merged 9 commits into
mainfrom
feat/python-nodes-in-snapshot
Nov 14, 2026
Merged

kvlonge merged 9 commits into
mainfrom
feat/python-nodes-in-snapshot

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Why

Python-only jobs failed with S010, server ad-hoc runs could not select SQL tests, and an intermediate loader with no write_strategy did not write the __loader__<name> table the docs describe.

Changes

  • Python nodes are published in the snapshot. Jobs, roles and ad-hoc runs use the CLI's resolver, so Python-only jobs build and +job: expands over Python edges.
  • A selection reaching a managed source publishes the source, never loader:, so workers can run it.
  • Ad-hoc test, audit, seed, freshness and build --no-python resolve SQL only. Ad-hoc runs accept test:, and a selected SQL test reports its own outcome.
  • A selected Python node reads its unselected upstream's last successful result.
  • An intermediate loader with no write_strategy defaults to table into __loader__<name>.
  • A provider with a positional-only ctx receives it by position.

Behaviour change: a selector that matches a Python check is accepted when it also selects the nodes the check depends on, and the check runs after them. Otherwise it fails with an error that names the check and the missing node. Previously, any selected check was refused. A job error names the job but still fails publication for the whole location; per-job errors are a larger change.

Verification

  • E2E: Python-only jobs, the intermediate-loader docs example, and a check selected with and without its task.
  • Integration: a worker job reaching a managed source through +, ad-hoc build test: and test with a failing test, accepted and rejected check jobs, and role grants. Each fails on the old code.
  • Targeted pytest (294 passed), cargo test for analysis and cli, clippy -D warnings, fensu check, and make check-ci on the final head.

DuckShop: new edge cases under benchmarks/duckshop/edge_cases/ (loaders/intermediate_default_accounts, orchestration/python_node_jobs, orchestration/upstream_loaded_orders.py, python_pipeline_*, intermediate_loader_roles.toml).

Work in progress, not ready for review.

- Publish-time job, sensor-condition and role selections resolve with the
  same unified SQL/Python resolver `sqb build --select` uses.
- Python snapshot nodes publish deps (Python and SQL) and tags; an
  intermediate loader publishes the source node it writes, and loader
  chains list it in `loads`.
- Server-side ad-hoc selection rebuilds the Python selection graph from the
  snapshot and resolves with the same resolver.
- The worker selects Python nodes by typed key.

Validated so far: test_orchestration_snapshot.py passes (99 tests,
including the Python selector reproduction). The Rust unit tests for the
ad-hoc planner and planner scope have not finished yet; no review and no
full gate.
A Python node can now be selected without its Python upstream in every
path that uses the shared resolver: sqb build --select, published jobs,
sensors, roles and ad-hoc runs. It reads the upstream's latest persisted
result, and fails at run time with an error naming the upstream and the
command to build it when the upstream has never succeeded.

An intermediate loader without a write_strategy that returns rows now
writes them to __loader__<name> as a table, as the loaders docs describe.
A self-managed intermediate that returns nothing keeps its behaviour.

The published snapshot rebuilds the Python selection graph from node deps,
and job groups order jobs across Python and Python-to-SQL edges.

Validated locally: python_selection and adhoc planner Rust tests, clippy
on both crates, fensu check, the worker claim tests, orchestration snapshot,
Python node selection E2E, load E2E (sqb load and sqb build on the docs
example) and the python_nodes and executor/load unit tests. Not yet run:
make check-ci, the full suite, review.
- `job:` and bare job names resolve jobs through the Python-aware resolver
  publication uses, so a job that selects only Python nodes builds them.
- Server ad-hoc runs accept `test:` selectors; a selected SQL test reports
  as a run node rather than as an attached check.
- A provider function whose context is positional-only (`ctx, /`) receives
  it by position instead of failing at run time.
- A role must grant the intermediate loader a job loads to run that job.
…odes

Selections now publish a managed source instead of its loader, ad-hoc
test, audit, seed and freshness runs resolve SQL only, a selected SQL
test reports its own outcome, and a selected model folds in its failing
test. A selector that matches a Python check must also select the nodes
the check depends on, in jobs, roles and the CLI alike.
A build never loads such a source, so a matched check on it would be
accepted and then never run. The resolver now names the check and the
source and points to sqb check.
Loader and Python node contexts read upstream results through one
mixin, and the selection entry points build their context in one place.
Regenerates the packaged docs skill for the selector and check pages.
…snapshot

# Conflicts:
#	crates/sqlbuild-python/src/bindings/_helpers/compiled_project/snapshot_execution.rs
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 14, 2026
@kvlonge
kvlonge removed this pull request from the merge queue due to a manual request Nov 14, 2026
@kvlonge
kvlonge added this pull request to the merge queue Nov 14, 2026
Merged via the queue into main with commit 2909eda Nov 14, 2026
30 checks passed
@kvlonge
kvlonge deleted the feat/python-nodes-in-snapshot branch November 14, 2026 08:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant