Repository navigation
feat!: warehouse round trips, kind: selectors and whole-graph compile refusals - #1550
Merged
Merged
Conversation
Remove the declarative dlt (`dlt_sources`) and ingestr (a source's `ingestr` block) integrations, the loaders SQLBuild generated for them, and the external loader connection mode they used: loaders that ran before SQLBuild opened its warehouse connections, with their own scheduling checks. Every loader now runs on a SQLBuild connection. The `dlt` and `ingestr` extras, the `external_source_load` and `ingestr_command` lifecycle operations and the plan's `integration_kind` field are gone. A sources file that still declares `dlt_sources`, or a source that still declares `ingestr`, fails compile with an error that points to `@loader`. BREAKING CHANGE: the dlt and ingestr integrations are removed. A project that declares `dlt_sources` or a source `ingestr` block no longer compiles; declare the source with `managed: true` and load it with a Python `@loader` named after the source. The `dlt` and `ingestr` package extras no longer exist.
Drop the dlt and ingestr integration pages, their sidebar entries and screenshots, and every mention in the loader, source, adapter, configuration and feature comparison pages, and regenerate the skill references.
…_trips A loader may now depend on a task or asset that reads SQL, directly or through other Python nodes, when the project sets `[settings] allow_warehouse_round_trips = true` in sqlbuild_project.toml. The setting is off by default and refused in sqlbuild_local.toml, like require_sql_analysis. With it off, the selection that plans or builds such a loader stops with an error that names the loader and the SQL-reading task or asset and explains the round trip. A loader depending on Python that reads no SQL is unchanged and always allowed. A task or asset that reads anything built from the source its own downstream loader fills is a cycle, refused with its path whatever the setting. Server-side ad-hoc previews resolve round trips; the launched command enforces the project's setting.
…reading its source A loader whose selected upstream task or asset reads SQL no longer runs in pre-SQL ingress. Its source load waits in the SQL build for the SQL that upstream reads, so one `sqb build` runs the model, then the task, then the loader, then the models reading the source. A failed or skipped upstream task or asset hard-skips the load and everything downstream of the source; an upstream that did not run because the SQL it reads was not rebuilt, as under `--changed`, skips the load softly so models selected for their own changes still build. A source load now honours the scheduler's blocked keys, and a hard-skipped load blocks its dependents as the loader docs describe. `ctx.loader()` refuses a task or asset with a message pointing to `ctx.result_of(node_function=...)`, which loaders already share with tasks and assets.
The DAG artifact draws a loader that a source declares as that source's node, but Python dependency edges still named loader:<name>, which has no node, so a task upstream of such a loader dangled. Those edges now end at the source, as loader-to-loader edges already did.
…nd assets Rewrite the SQL boundary section for loaders depending on any Python node, add the warehouse round trip rules and the allow_warehouse_round_trips setting, and update the loader reference for depends_on, ctx.result_of and ctx.loader.
`kind:model`, `kind:seed`, `kind:task` and the other node kinds the catalog names select every node of that kind. The selector is part of the one selector grammar, so `sqb build -s`, job selections, lineage and the server's ad-hoc resolver all accept it, and it composes with `+`, `,` and `--exclude` like `tag:`. An unknown kind names the closest kinds; a kind with no nodes is an error, as an unused tag is. The selector docs now also say how `+` binds inside a comma intersection: it expands the part it is written on before the parts are intersected.
…e walk Every selection adds the functions and seeds upstream of what it selects. That walked each selected node's upstream separately, so a tag matching 2,000 models in a 10,000-model project walked the shared staging layers 2,000 times. One walk from every selected node reaches the same nodes. Evaluating such a tag drops from about 75 ms to 27 ms; the selection is unchanged.
Resolving a selection collected all graph keys before looking at it, and the unified resolution resolves each token on its own, so a selection of N nodes collected every key N times.
…e plan's own error Compile now runs the checks every plan and job resolution run over the whole graph, whatever is selected: - the planner's execution-order cycle (model refs, source loaders, SQL test, audit-gate and hook edges), through the same ordering and edge origins as plan scope; - the SQL/Python boundary validation (cross-graph dependency cycles, warehouse round trips without the setting, dependencies on terminal loaders, invalid SQL references), through python_sql_graph_error, which wraps the selection's validate_boundaries. Execution gate edges are computed once for both the dependencies and their origins, which also saves the duplicate read in plan scope.
…r SQL-reading tasks
Lifecycle classification and run selection both read each node's sql_deps; they now share one reader instead of two copies that had started to read the reference kind differently.
The execution readers that share precomputed gate edges get their own entry modules, the cycle-or-order helper moves to planner_identity's helpers, and the lifecycle crossing budget follows its calls into native_run_lifecycle.
…asks Compile now refuses a project whose SQL graph has a cycle, and refuses a loader that runs after a task reading SQL unless warehouse round trips are allowed. Update the cyclic compile case to expect a failing exit code and let the Python compile fixture allow round trips.
…trips # Conflicts: # crates/sqlbuild-analysis/src/python_plan/main/run_lifecycle.rs # crates/sqlbuild-analysis/src/python_plan/models.rs # crates/sqlbuild-analysis/src/python_plan/tests/helpers.rs # crates/sqlbuild-analysis/src/python_plan/tests/test_run_lifecycle.rs # crates/sqlbuild-analysis/src/python_plan/tests/test_types.rs # crates/sqlbuild-core/src/lifecycle_events/constants.rs # crates/sqlbuild-python-front/src/discovery/_helpers/project/project_facts.rs # crates/sqlbuild-python-project/src/graph/_helpers/python_nodes/python_lifecycle.rs # tests/unit/src/sqlbuild/executor/load/_test_types.py
kvlonge
marked this pull request as ready for review
November 17, 2026 16:25
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 17, 2026
…on-nodes integration project The unknown-local-key golden now lists allow_warehouse_round_trips among the [settings] keys, next to require_sql_analysis. The Dagster and Rivers Python-nodes project has a loader after a task that reads SQL, which compile now refuses unless the project allows warehouse round trips.
`sqb compile` now fails on every whole-graph refusal that `sqb plan` and `sqb build` already made. Projects that compiled before but could never build now fail compile, including `sqb compile --dag`. BREAKING CHANGE: `sqb compile` fails when a loader depends on a task or asset that reads SQL, unless the project sets `allow_warehouse_round_trips = true` under `[settings]` in sqlbuild_project.toml, and when the SQL graph has a cycle. Plan and build refused both before; compile and `compile --dag` passed them.
…trips # Conflicts: # tests/e2e/src/sqlbuild/cli/commands/main/build/_test_types.py # tests/e2e/src/sqlbuild/cli/commands/main/build/helpers.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A loader could not depend on a task or asset that reads SQL, nodes could not be selected by kind, and
sqb compilepassed projects every plan then refused.Changes
[settings] allow_warehouse_round_trips(off by default, refused insqlbuild_local.toml) lets a loader depend on Python that reads SQL. Onesqb buildruns that SQL, the task, the loader and the models reading its source; a failed upstream hard-skips the load, one that did not run skips it softly.kind:<kind>selects every node of one kind.sqb compilefails on every whole-graph refusal a plan makes, with the plan's message.Breaking:
sqb compileandcompile --dagnow fail on projects plan and build already refused: a loader after a task or asset that reads SQL, unlessallow_warehouse_round_trips = true, and SQL cycles.Replayed onto the split binding crates. Hand resolutions:
ProjectSelectionlives ingraph/models.rs;native_run_lifecycle,project_graph_errorandexecution_cycleare public entries; the_withvariants sit besideexecution_deps/execution_origins.Added: the regenerated
selectors.md; one reader for a Python node's SQL dependencies (two copies had diverged); one function per planner identity entry; tests expecting the new compile refusals.No DuckShop case, to keep the replay faithful: cases for the three fixes (off-target job-group members, loader edges, whole-graph refusals) and the round trip are in #1553.
Verification
-s loader:Xis refused with S007 namingsource:X;-s source:Xfails because the task has no result;-s +source:Xruns the round trip.make check-cipasses.