Skip to content

feat!: order job groups by lineage only, limit Python descriptions, skip checks under --no-audits - #1476

Merged
kvlonge merged 23 commits into
mainfrom
feat/orchestration-step2
Nov 14, 2026
Merged

kvlonge merged 23 commits into
mainfrom
feat/orchestration-step2

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Why

A job group could be ordered by node lineage and by a hand-written after list, and the two could disagree. Python descriptions had no length limit, while SQL headers already respect [format] max_description_lines. sqb build --no-audits was documented to skip checks, but it still ran Python checks.

Changes

  • Breaking: JobGroup no longer takes after. A group's order comes only from the lineage between the nodes its jobs build. The server rejects group waits that lineage does not produce; the job graph draws lineage waits.
    • Migration: delete after=. If an order had no lineage behind it, declare the missing dependency between the nodes, or run the jobs from separate triggers.
  • P027: a Python task, asset, loader, check, hook, provider or function description that wraps past max_description_lines at line_width is a compile error. A job, job group, sensor or calendar schedule whose description is too long cannot publish.
  • sqb build --no-audits skips Python checks as well as audits.
  • A calendar schedule declared for another target without decorating a function no longer breaks publication.
  • Docs: format, model configuration and Project Python pages.
  • Unchanged: an undecorated calendar on its own target still fails with a raw AttributeError; a clear error is a follow-up.

Verification

  • The new --no-audits case fails on 0.223.0 and passes here.
  • 159 targeted tests pass.
  • New golden failure/python-node-description-too-long.json for the P027 corpus case.
  • E2E: build cases asserting a check ran drop --no-audits; one case now expects checks skipped.
  • sqlbuild-orchestration native tests pass on PostgreSQL.
  • The rerun wait-boundary test fails when ancestor waits are read as direct edges only.
  • make check-ci passes.

DuckShop: the calendar edge case adds a production-only calendar that decorates no function. Publication failed on it before; it passes now. No DuckShop group used after.

Work in progress, not ready for review.

- Publish-time job, sensor-condition and role selections resolve with the
  same unified SQL/Python resolver `sqb build --select` uses.
- Python snapshot nodes publish deps (Python and SQL) and tags; an
  intermediate loader publishes the source node it writes, and loader
  chains list it in `loads`.
- Server-side ad-hoc selection rebuilds the Python selection graph from the
  snapshot and resolves with the same resolver.
- The worker selects Python nodes by typed key.

Validated so far: test_orchestration_snapshot.py passes (99 tests,
including the Python selector reproduction). The Rust unit tests for the
ad-hoc planner and planner scope have not finished yet; no review and no
full gate.
A Python node can now be selected without its Python upstream in every
path that uses the shared resolver: sqb build --select, published jobs,
sensors, roles and ad-hoc runs. It reads the upstream's latest persisted
result, and fails at run time with an error naming the upstream and the
command to build it when the upstream has never succeeded.

An intermediate loader without a write_strategy that returns rows now
writes them to __loader__<name> as a table, as the loaders docs describe.
A self-managed intermediate that returns nothing keeps its behaviour.

The published snapshot rebuilds the Python selection graph from node deps,
and job groups order jobs across Python and Python-to-SQL edges.

Validated locally: python_selection and adhoc planner Rust tests, clippy
on both crates, fensu check, the worker claim tests, orchestration snapshot,
Python node selection E2E, load E2E (sqb load and sqb build on the docs
example) and the python_nodes and executor/load unit tests. Not yet run:
make check-ci, the full suite, review.
- `job:` and bare job names resolve jobs through the Python-aware resolver
  publication uses, so a job that selects only Python nodes builds them.
- Server ad-hoc runs accept `test:` selectors; a selected SQL test reports
  as a run node rather than as an attached check.
- A provider function whose context is positional-only (`ctx, /`) receives
  it by position instead of failing at run time.
- A role must grant the intermediate loader a job loads to run that job.
…odes

Selections now publish a managed source instead of its loader, ad-hoc
test, audit, seed and freshness runs resolve SQL only, a selected SQL
test reports its own outcome, and a selected model folds in its failing
test. A selector that matches a Python check must also select the nodes
the check depends on, in jobs, roles and the CLI alike.
A build never loads such a source, so a matched check on it would be
accepted and then never run. The resolver now names the check and the
source and points to sqb check.
Loader and Python node contexts read upstream results through one
mixin, and the selection entry points build their context in one place.
Regenerates the packaged docs skill for the selector and check pages.
…dits

JobGroup no longer takes after: a group's order comes only from the
lineage between the nodes its jobs build, and declaring after fails with
an error that names the group. build --no-audits now skips Python checks
alongside audits, as documented.
…tions

A Python task, asset, loader, check, hook, provider or UDF description
that wraps past [format] max_description_lines is a P027 compile error at
its declaration, and a job, job group, sensor or calendar schedule with
one cannot publish.
…snapshot

# Conflicts:
#	crates/sqlbuild-python/src/bindings/_helpers/compiled_project/snapshot_execution.rs
…th lineage-only groups

Rust orchestration fixtures declare the node lineage their group waits come from.
A Python description limit falls back to the defaults without a project directory,
and a calendar schedule that never decorated a function no longer breaks publication
for other targets; DuckShop's calendar edge case now declares one.
…p JobGroup after

JobGroup no longer has an after parameter at all, so passing it fails at the call site.
Calendar names and descriptions are read through the native host edge, the admission
fixture builds stage lineage without branching, and the description-limit test keeps its
cases beside the test.
@kvlonge
kvlonge marked this pull request as ready for review November 14, 2026 09:47
@kvlonge
kvlonge enabled auto-merge November 14, 2026 09:50
@kvlonge
kvlonge added this pull request to the merge queue Nov 14, 2026
@kvlonge
kvlonge removed this pull request from the merge queue due to a manual request Nov 14, 2026
# Conflicts:
#	tests/e2e/src/sqlbuild/cli/commands/main/build/no_tests_no_audits/test_build_no_tests_no_audits.py
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 14, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 14, 2026
@kvlonge
kvlonge added this pull request to the merge queue Nov 14, 2026
Merged via the queue into main with commit 4e32d70 Nov 14, 2026
30 checks passed
@kvlonge
kvlonge deleted the feat/orchestration-step2 branch November 14, 2026 12:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant