Skip to content

feat!: warehouse round trips, kind: selectors and whole-graph compile refusals - #1550

Merged
kvlonge merged 20 commits into
mainfrom
feat/warehouse-round-trips
Nov 17, 2026
Merged

kvlonge merged 20 commits into
mainfrom
feat/warehouse-round-trips

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Why

A loader could not depend on a task or asset that reads SQL, nodes could not be selected by kind, and sqb compile passed projects every plan then refused.

Changes

  • [settings] allow_warehouse_round_trips (off by default, refused in sqlbuild_local.toml) lets a loader depend on Python that reads SQL. One sqb build runs that SQL, the task, the loader and the models reading its source; a failed upstream hard-skips the load, one that did not run skips it softly.
  • kind:<kind> selects every node of one kind.
  • sqb compile fails on every whole-graph refusal a plan makes, with the plan's message.
  • Job groups skip members not declared for the target; Python edges into a declared loader point at its source.

Breaking: sqb compile and compile --dag now fail on projects plan and build already refused: a loader after a task or asset that reads SQL, unless allow_warehouse_round_trips = true, and SQL cycles.

Replayed onto the split binding crates. Hand resolutions: ProjectSelection lives in graph/models.rs; native_run_lifecycle, project_graph_error and execution_cycle are public entries; the _with variants sit beside execution_deps/execution_origins.

Added: the regenerated selectors.md; one reader for a Python node's SQL dependencies (two copies had diverged); one function per planner identity entry; tests expecting the new compile refusals.

No DuckShop case, to keep the replay faithful: cases for the three fixes (off-target job-group members, loader edges, whole-graph refusals) and the round trip are in #1553.

Verification

  • Rust tests for python_selection, python_plan, graph, dag_artifact and config pass.
  • Targeted unit, integration and DuckDB E2E tests pass.
  • Review asked whether a round-trip loader is dropped when its source is not selected. It is not: -s loader:X is refused with S007 naming source:X; -s source:X fails because the task has no result; -s +source:X runs the round trip.
  • make check-ci passes.

Remove the declarative dlt (`dlt_sources`) and ingestr (a source's `ingestr`
block) integrations, the loaders SQLBuild generated for them, and the external
loader connection mode they used: loaders that ran before SQLBuild opened its
warehouse connections, with their own scheduling checks. Every loader now runs
on a SQLBuild connection. The `dlt` and `ingestr` extras, the
`external_source_load` and `ingestr_command` lifecycle operations and the
plan's `integration_kind` field are gone.

A sources file that still declares `dlt_sources`, or a source that still
declares `ingestr`, fails compile with an error that points to `@loader`.

BREAKING CHANGE: the dlt and ingestr integrations are removed. A project that
declares `dlt_sources` or a source `ingestr` block no longer compiles; declare
the source with `managed: true` and load it with a Python `@loader` named after
the source. The `dlt` and `ingestr` package extras no longer exist.
Drop the dlt and ingestr integration pages, their sidebar entries and screenshots, and every mention in the loader, source, adapter, configuration and feature comparison pages, and regenerate the skill references.
…_trips

A loader may now depend on a task or asset that reads SQL, directly or
through other Python nodes, when the project sets `[settings]
allow_warehouse_round_trips = true` in sqlbuild_project.toml. The setting is
off by default and refused in sqlbuild_local.toml, like
require_sql_analysis. With it off, the selection that plans or builds such
a loader stops with an error that names the loader and the SQL-reading task
or asset and explains the round trip. A loader depending on Python that
reads no SQL is unchanged and always allowed.

A task or asset that reads anything built from the source its own
downstream loader fills is a cycle, refused with its path whatever the
setting. Server-side ad-hoc previews resolve round trips; the launched
command enforces the project's setting.
…reading its source

A loader whose selected upstream task or asset reads SQL no longer runs in
pre-SQL ingress. Its source load waits in the SQL build for the SQL that
upstream reads, so one `sqb build` runs the model, then the task, then the
loader, then the models reading the source. A failed or skipped upstream task
or asset hard-skips the load and everything downstream of the source; an
upstream that did not run because the SQL it reads was not rebuilt, as under
`--changed`, skips the load softly so models selected for their own changes
still build. A source load now honours the scheduler's blocked keys, and a
hard-skipped load blocks its dependents as the loader docs describe.

`ctx.loader()` refuses a task or asset with a message pointing to
`ctx.result_of(node_function=...)`, which loaders already share with tasks
and assets.
The DAG artifact draws a loader that a source declares as that source's node, but Python dependency edges still named loader:<name>, which has no node, so a task upstream of such a loader dangled. Those edges now end at the source, as loader-to-loader edges already did.
…nd assets

Rewrite the SQL boundary section for loaders depending on any Python node, add the warehouse round trip rules and the allow_warehouse_round_trips setting, and update the loader reference for depends_on, ctx.result_of and ctx.loader.
`kind:model`, `kind:seed`, `kind:task` and the other node kinds the catalog
names select every node of that kind. The selector is part of the one selector
grammar, so `sqb build -s`, job selections, lineage and the server's ad-hoc
resolver all accept it, and it composes with `+`, `,` and `--exclude` like
`tag:`. An unknown kind names the closest kinds; a kind with no nodes is an
error, as an unused tag is.

The selector docs now also say how `+` binds inside a comma intersection: it
expands the part it is written on before the parts are intersected.
…e walk

Every selection adds the functions and seeds upstream of what it selects. That
walked each selected node's upstream separately, so a tag matching 2,000 models
in a 10,000-model project walked the shared staging layers 2,000 times. One walk
from every selected node reaches the same nodes. Evaluating such a tag drops
from about 75 ms to 27 ms; the selection is unchanged.
Resolving a selection collected all graph keys before looking at it, and the
unified resolution resolves each token on its own, so a selection of N nodes
collected every key N times.
…e plan's own error

Compile now runs the checks every plan and job resolution run over the
whole graph, whatever is selected:

- the planner's execution-order cycle (model refs, source loaders, SQL
  test, audit-gate and hook edges), through the same ordering and edge
  origins as plan scope;
- the SQL/Python boundary validation (cross-graph dependency cycles,
  warehouse round trips without the setting, dependencies on terminal
  loaders, invalid SQL references), through python_sql_graph_error,
  which wraps the selection's validate_boundaries.

Execution gate edges are computed once for both the dependencies and
their origins, which also saves the duplicate read in plan scope.
Lifecycle classification and run selection both read each node's
sql_deps; they now share one reader instead of two copies that had
started to read the reference kind differently.
The execution readers that share precomputed gate edges get their own
entry modules, the cycle-or-order helper moves to planner_identity's
helpers, and the lifecycle crossing budget follows its calls into
native_run_lifecycle.
…asks

Compile now refuses a project whose SQL graph has a cycle, and refuses a
loader that runs after a task reading SQL unless warehouse round trips are
allowed. Update the cyclic compile case to expect a failing exit code and
let the Python compile fixture allow round trips.
…trips

# Conflicts:
#	crates/sqlbuild-analysis/src/python_plan/main/run_lifecycle.rs
#	crates/sqlbuild-analysis/src/python_plan/models.rs
#	crates/sqlbuild-analysis/src/python_plan/tests/helpers.rs
#	crates/sqlbuild-analysis/src/python_plan/tests/test_run_lifecycle.rs
#	crates/sqlbuild-analysis/src/python_plan/tests/test_types.rs
#	crates/sqlbuild-core/src/lifecycle_events/constants.rs
#	crates/sqlbuild-python-front/src/discovery/_helpers/project/project_facts.rs
#	crates/sqlbuild-python-project/src/graph/_helpers/python_nodes/python_lifecycle.rs
#	tests/unit/src/sqlbuild/executor/load/_test_types.py
@kvlonge
kvlonge marked this pull request as ready for review November 17, 2026 16:25
@kvlonge
kvlonge added this pull request to the merge queue Nov 17, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 17, 2026
…on-nodes integration project

The unknown-local-key golden now lists allow_warehouse_round_trips among
the [settings] keys, next to require_sql_analysis. The Dagster and Rivers
Python-nodes project has a loader after a task that reads SQL, which
compile now refuses unless the project allows warehouse round trips.
`sqb compile` now fails on every whole-graph refusal that `sqb plan` and
`sqb build` already made. Projects that compiled before but could never
build now fail compile, including `sqb compile --dag`.

BREAKING CHANGE: `sqb compile` fails when a loader depends on a task or asset that reads SQL, unless the project sets `allow_warehouse_round_trips = true` under `[settings]` in sqlbuild_project.toml, and when the SQL graph has a cycle. Plan and build refused both before; compile and `compile --dag` passed them.
@kvlonge kvlonge changed the title feat: warehouse round trips, kind: selectors and whole-graph compile refusals feat!: warehouse round trips, kind: selectors and whole-graph compile refusals Nov 17, 2026
…trips

# Conflicts:
#	tests/e2e/src/sqlbuild/cli/commands/main/build/_test_types.py
#	tests/e2e/src/sqlbuild/cli/commands/main/build/helpers.py
@kvlonge
kvlonge disabled auto-merge November 17, 2026 18:31
@kvlonge
kvlonge added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit 78a786a Nov 17, 2026
34 checks passed
@kvlonge
kvlonge deleted the feat/warehouse-round-trips branch November 17, 2026 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant