Skip to content

test: add DuckShop cases for warehouse round trips and off-target group members - #1553

Merged
kvlonge merged 18 commits into
mainfrom
test/duckshop-round-trip-cases
Nov 17, 2026
Merged

kvlonge merged 18 commits into
mainfrom
test/duckshop-round-trip-cases

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Why

The warehouse round-trip PR replays its commits unchanged, so it adds no DuckShop edge cases. The round trip and its three fixes (off-target job-group members, loader edges, whole-graph compile refusals) each involve a project shape, so DuckShop gets a matching case for each.

Changes

  • edge_cases/loaders/round_trip/project/: orders → score task → partner loader → partner_scores → scored_orders. The round-trip build E2E now copies this project instead of writing inline strings.
  • edge_cases/loaders/round_trip/refusals/: a task cycle through the loader's source, a cycle of model refs, and a task depending on a terminal loader. The compile-refusal E2E reads them from here.
  • A dag --json E2E over the round-trip project checks that the task's edge ends at source:partner_scores and that every edge ends at a node.
  • edge_cases/orchestration/off_target_members/: a job group whose member is declared only for prod. A new integration case builds the group under dev (member skipped) and prod (member runs).

Draft until the round-trip PR merges; its commits show here until then.

Verification

  • The round-trip build, compile-refusal, dag --json and job-selector tests pass (26).
  • With the job-group fix reverted, the new off-target case fails with an unknown-job selection error.
  • make check-ci on the final head.

Remove the declarative dlt (`dlt_sources`) and ingestr (a source's `ingestr`
block) integrations, the loaders SQLBuild generated for them, and the external
loader connection mode they used: loaders that ran before SQLBuild opened its
warehouse connections, with their own scheduling checks. Every loader now runs
on a SQLBuild connection. The `dlt` and `ingestr` extras, the
`external_source_load` and `ingestr_command` lifecycle operations and the
plan's `integration_kind` field are gone.

A sources file that still declares `dlt_sources`, or a source that still
declares `ingestr`, fails compile with an error that points to `@loader`.

BREAKING CHANGE: the dlt and ingestr integrations are removed. A project that
declares `dlt_sources` or a source `ingestr` block no longer compiles; declare
the source with `managed: true` and load it with a Python `@loader` named after
the source. The `dlt` and `ingestr` package extras no longer exist.
Drop the dlt and ingestr integration pages, their sidebar entries and screenshots, and every mention in the loader, source, adapter, configuration and feature comparison pages, and regenerate the skill references.
…_trips

A loader may now depend on a task or asset that reads SQL, directly or
through other Python nodes, when the project sets `[settings]
allow_warehouse_round_trips = true` in sqlbuild_project.toml. The setting is
off by default and refused in sqlbuild_local.toml, like
require_sql_analysis. With it off, the selection that plans or builds such
a loader stops with an error that names the loader and the SQL-reading task
or asset and explains the round trip. A loader depending on Python that
reads no SQL is unchanged and always allowed.

A task or asset that reads anything built from the source its own
downstream loader fills is a cycle, refused with its path whatever the
setting. Server-side ad-hoc previews resolve round trips; the launched
command enforces the project's setting.
…reading its source

A loader whose selected upstream task or asset reads SQL no longer runs in
pre-SQL ingress. Its source load waits in the SQL build for the SQL that
upstream reads, so one `sqb build` runs the model, then the task, then the
loader, then the models reading the source. A failed or skipped upstream task
or asset hard-skips the load and everything downstream of the source; an
upstream that did not run because the SQL it reads was not rebuilt, as under
`--changed`, skips the load softly so models selected for their own changes
still build. A source load now honours the scheduler's blocked keys, and a
hard-skipped load blocks its dependents as the loader docs describe.

`ctx.loader()` refuses a task or asset with a message pointing to
`ctx.result_of(node_function=...)`, which loaders already share with tasks
and assets.
The DAG artifact draws a loader that a source declares as that source's node, but Python dependency edges still named loader:<name>, which has no node, so a task upstream of such a loader dangled. Those edges now end at the source, as loader-to-loader edges already did.
…nd assets

Rewrite the SQL boundary section for loaders depending on any Python node, add the warehouse round trip rules and the allow_warehouse_round_trips setting, and update the loader reference for depends_on, ctx.result_of and ctx.loader.
`kind:model`, `kind:seed`, `kind:task` and the other node kinds the catalog
names select every node of that kind. The selector is part of the one selector
grammar, so `sqb build -s`, job selections, lineage and the server's ad-hoc
resolver all accept it, and it composes with `+`, `,` and `--exclude` like
`tag:`. An unknown kind names the closest kinds; a kind with no nodes is an
error, as an unused tag is.

The selector docs now also say how `+` binds inside a comma intersection: it
expands the part it is written on before the parts are intersected.
…e walk

Every selection adds the functions and seeds upstream of what it selects. That
walked each selected node's upstream separately, so a tag matching 2,000 models
in a 10,000-model project walked the shared staging layers 2,000 times. One walk
from every selected node reaches the same nodes. Evaluating such a tag drops
from about 75 ms to 27 ms; the selection is unchanged.
Resolving a selection collected all graph keys before looking at it, and the
unified resolution resolves each token on its own, so a selection of N nodes
collected every key N times.
…e plan's own error

Compile now runs the checks every plan and job resolution run over the
whole graph, whatever is selected:

- the planner's execution-order cycle (model refs, source loaders, SQL
  test, audit-gate and hook edges), through the same ordering and edge
  origins as plan scope;
- the SQL/Python boundary validation (cross-graph dependency cycles,
  warehouse round trips without the setting, dependencies on terminal
  loaders, invalid SQL references), through python_sql_graph_error,
  which wraps the selection's validate_boundaries.

Execution gate edges are computed once for both the dependencies and
their origins, which also saves the duplicate read in plan scope.
Lifecycle classification and run selection both read each node's
sql_deps; they now share one reader instead of two copies that had
started to read the reference kind differently.
The execution readers that share precomputed gate edges get their own
entry modules, the cycle-or-order helper moves to planner_identity's
helpers, and the lifecycle crossing budget follows its calls into
native_run_lifecycle.
…asks

Compile now refuses a project whose SQL graph has a cycle, and refuses a
loader that runs after a task reading SQL unless warehouse round trips are
allowed. Update the cyclic compile case to expect a failing exit code and
let the Python compile fixture allow round trips.
…up members

The round-trip project and its refusal variants move into DuckShop's
loader edge cases, and the round-trip build, compile-refusal and new
dag --json tests read them from there. A job group with a member
declared only for another target gets an orchestration fixture and an
integration case under both targets.
@kvlonge
kvlonge marked this pull request as ready for review November 17, 2026 19:45
@kvlonge
kvlonge enabled auto-merge November 17, 2026 19:45
@kvlonge
kvlonge disabled auto-merge November 17, 2026 19:54
@kvlonge
kvlonge added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit fa63ff6 Nov 17, 2026
33 of 58 checks passed
@kvlonge
kvlonge deleted the test/duckshop-round-trip-cases branch November 17, 2026 20:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant