Repository navigation
feat!: order job groups by lineage only, limit Python descriptions, skip checks under --no-audits - #1476
Merged
Conversation
Work in progress, not ready for review. - Publish-time job, sensor-condition and role selections resolve with the same unified SQL/Python resolver `sqb build --select` uses. - Python snapshot nodes publish deps (Python and SQL) and tags; an intermediate loader publishes the source node it writes, and loader chains list it in `loads`. - Server-side ad-hoc selection rebuilds the Python selection graph from the snapshot and resolves with the same resolver. - The worker selects Python nodes by typed key. Validated so far: test_orchestration_snapshot.py passes (99 tests, including the Python selector reproduction). The Rust unit tests for the ad-hoc planner and planner scope have not finished yet; no review and no full gate.
A Python node can now be selected without its Python upstream in every path that uses the shared resolver: sqb build --select, published jobs, sensors, roles and ad-hoc runs. It reads the upstream's latest persisted result, and fails at run time with an error naming the upstream and the command to build it when the upstream has never succeeded. An intermediate loader without a write_strategy that returns rows now writes them to __loader__<name> as a table, as the loaders docs describe. A self-managed intermediate that returns nothing keeps its behaviour. The published snapshot rebuilds the Python selection graph from node deps, and job groups order jobs across Python and Python-to-SQL edges. Validated locally: python_selection and adhoc planner Rust tests, clippy on both crates, fensu check, the worker claim tests, orchestration snapshot, Python node selection E2E, load E2E (sqb load and sqb build on the docs example) and the python_nodes and executor/load unit tests. Not yet run: make check-ci, the full suite, review.
- `job:` and bare job names resolve jobs through the Python-aware resolver publication uses, so a job that selects only Python nodes builds them. - Server ad-hoc runs accept `test:` selectors; a selected SQL test reports as a run node rather than as an attached check. - A provider function whose context is positional-only (`ctx, /`) receives it by position instead of failing at run time. - A role must grant the intermediate loader a job loads to run that job.
…odes Selections now publish a managed source instead of its loader, ad-hoc test, audit, seed and freshness runs resolve SQL only, a selected SQL test reports its own outcome, and a selected model folds in its failing test. A selector that matches a Python check must also select the nodes the check depends on, in jobs, roles and the CLI alike.
A build never loads such a source, so a matched check on it would be accepted and then never run. The resolver now names the check and the source and points to sqb check.
Loader and Python node contexts read upstream results through one mixin, and the selection entry points build their context in one place. Regenerates the packaged docs skill for the selector and check pages.
…dits JobGroup no longer takes after: a group's order comes only from the lineage between the nodes its jobs build, and declaring after fails with an error that names the group. build --no-audits now skips Python checks alongside audits, as documented.
…tions A Python task, asset, loader, check, hook, provider or UDF description that wraps past [format] max_description_lines is a P027 compile error at its declaration, and a job, job group, sensor or calendar schedule with one cannot publish.
…snapshot # Conflicts: # crates/sqlbuild-python/src/bindings/_helpers/compiled_project/snapshot_execution.rs
…th lineage-only groups Rust orchestration fixtures declare the node lineage their group waits come from. A Python description limit falls back to the defaults without a project directory, and a calendar schedule that never decorated a function no longer breaks publication for other targets; DuckShop's calendar edge case now declares one.
…p JobGroup after JobGroup no longer has an after parameter at all, so passing it fails at the call site. Calendar names and descriptions are read through the native host edge, the admission fixture builds stage lineage without branching, and the description-limit test keeps its cases beside the test.
kvlonge
marked this pull request as ready for review
November 14, 2026 09:47
kvlonge
enabled auto-merge
November 14, 2026 09:50
# Conflicts: # tests/e2e/src/sqlbuild/cli/commands/main/build/no_tests_no_audits/test_build_no_tests_no_audits.py
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A job group could be ordered by node lineage and by a hand-written
afterlist, and the two could disagree. Python descriptions had no length limit, while SQL headers already respect[format] max_description_lines.sqb build --no-auditswas documented to skip checks, but it still ran Python checks.Changes
JobGroupno longer takesafter. A group's order comes only from the lineage between the nodes its jobs build. The server rejects group waits that lineage does not produce; the job graph draws lineage waits.after=. If an order had no lineage behind it, declare the missing dependency between the nodes, or run the jobs from separate triggers.max_description_linesatline_widthis a compile error. A job, job group, sensor or calendar schedule whose description is too long cannot publish.sqb build --no-auditsskips Python checks as well as audits.AttributeError; a clear error is a follow-up.Verification
--no-auditscase fails on 0.223.0 and passes here.failure/python-node-description-too-long.jsonfor the P027 corpus case.--no-audits; one case now expects checks skipped.sqlbuild-orchestrationnative tests pass on PostgreSQL.make check-cipasses.DuckShop: the calendar edge case adds a production-only calendar that decorates no function. Publication failed on it before; it passes now. No DuckShop group used
after.