Repository navigation
feat: publish Python nodes for jobs, roles and server runs - #1472
Merged
Merged
Conversation
Work in progress, not ready for review. - Publish-time job, sensor-condition and role selections resolve with the same unified SQL/Python resolver `sqb build --select` uses. - Python snapshot nodes publish deps (Python and SQL) and tags; an intermediate loader publishes the source node it writes, and loader chains list it in `loads`. - Server-side ad-hoc selection rebuilds the Python selection graph from the snapshot and resolves with the same resolver. - The worker selects Python nodes by typed key. Validated so far: test_orchestration_snapshot.py passes (99 tests, including the Python selector reproduction). The Rust unit tests for the ad-hoc planner and planner scope have not finished yet; no review and no full gate.
A Python node can now be selected without its Python upstream in every path that uses the shared resolver: sqb build --select, published jobs, sensors, roles and ad-hoc runs. It reads the upstream's latest persisted result, and fails at run time with an error naming the upstream and the command to build it when the upstream has never succeeded. An intermediate loader without a write_strategy that returns rows now writes them to __loader__<name> as a table, as the loaders docs describe. A self-managed intermediate that returns nothing keeps its behaviour. The published snapshot rebuilds the Python selection graph from node deps, and job groups order jobs across Python and Python-to-SQL edges. Validated locally: python_selection and adhoc planner Rust tests, clippy on both crates, fensu check, the worker claim tests, orchestration snapshot, Python node selection E2E, load E2E (sqb load and sqb build on the docs example) and the python_nodes and executor/load unit tests. Not yet run: make check-ci, the full suite, review.
- `job:` and bare job names resolve jobs through the Python-aware resolver publication uses, so a job that selects only Python nodes builds them. - Server ad-hoc runs accept `test:` selectors; a selected SQL test reports as a run node rather than as an attached check. - A provider function whose context is positional-only (`ctx, /`) receives it by position instead of failing at run time. - A role must grant the intermediate loader a job loads to run that job.
…odes Selections now publish a managed source instead of its loader, ad-hoc test, audit, seed and freshness runs resolve SQL only, a selected SQL test reports its own outcome, and a selected model folds in its failing test. A selector that matches a Python check must also select the nodes the check depends on, in jobs, roles and the CLI alike.
A build never loads such a source, so a matched check on it would be accepted and then never run. The resolver now names the check and the source and points to sqb check.
Loader and Python node contexts read upstream results through one mixin, and the selection entry points build their context in one place. Regenerates the packaged docs skill for the selector and check pages.
…snapshot # Conflicts: # crates/sqlbuild-python/src/bindings/_helpers/compiled_project/snapshot_execution.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Python-only jobs failed with S010, server ad-hoc runs could not select SQL tests, and an intermediate loader with no
write_strategydid not write the__loader__<name>table the docs describe.Changes
+job:expands over Python edges.loader:, so workers can run it.test,audit,seed,freshnessandbuild --no-pythonresolve SQL only. Ad-hoc runs accepttest:, and a selected SQL test reports its own outcome.write_strategydefaults totableinto__loader__<name>.ctxreceives it by position.Behaviour change: a selector that matches a Python check is accepted when it also selects the nodes the check depends on, and the check runs after them. Otherwise it fails with an error that names the check and the missing node. Previously, any selected check was refused. A job error names the job but still fails publication for the whole location; per-job errors are a larger change.
Verification
+, ad-hocbuild test:andtestwith a failing test, accepted and rejected check jobs, and role grants. Each fails on the old code.cargo testfor analysis and cli, clippy-D warnings,fensu check, andmake check-cion the final head.DuckShop: new edge cases under
benchmarks/duckshop/edge_cases/(loaders/intermediate_default_accounts,orchestration/python_node_jobs,orchestration/upstream_loaded_orders.py,python_pipeline_*,intermediate_loader_roles.toml).