Skip to content

fix: follow the Python fan-in rule for round-trip loads - #1559

Merged
kvlonge merged 4 commits into
mainfrom
fix/round-trip-skip-modes-pr
Nov 17, 2026
Merged

kvlonge merged 4 commits into
mainfrom
fix/round-trip-skip-modes-pr

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Why

A warehouse round-trip load (task or asset → loader → source → models) gated on its upstreams more strictly than any other Python node, and in one case more loosely. A task's default soft skip hard-skipped the load and blocked every model reading the source. The load skipped at the first gating upstream that did not succeed, even when another succeeded. A succeeding task behind a soft-skipped one could let the load run on stale results. With --no-python the skip reason was misleading.

Changes

  • The load follows the Python fan-in rule over the tasks and assets it reads directly (or through other loaders): a failed or hard-skipped gate anywhere skips it hard; a direct gate that succeeded lets it run; otherwise it skips softly, and models reading the source build from its last loaded data when another of their inputs ran.
  • With --no-python, a read-side gate that did not run soft-skips the load even if an ingress gate succeeded, and the reason names --no-python.
  • Loader and Python node docs, and the generated skill references, describe the rule. DuckShop gains round-trip variants for a transitive gate and an ingress gate.

Verification

  • Eight new cases in test_warehouse_round_trips.py cover soft and hard skips, three fan-ins, a transitive gate, --no-python and an ingress gate; the last two failed before their fix. A Rust unit case covers the direct-gate set.
  • Round-trip, DAG JSON, job selector and graph boundary tests pass, as do the Rust python_plan tests.
  • make check-ci passes on this head.

With --no-python the task gating a round-trip loader never runs, and the
load was soft-skipped with "the SQL it reads was not rebuilt", which is not
why. The reason now says --no-python skips read-side tasks and assets.
A round-trip load was hard-skipped whenever its gating task or asset
skipped, so a task's default soft skip blocked every model reading the
source. The load now takes the upstream's skip mode: a soft skip skips
only the load, and models reading the source still build from its last
loaded data. A hard skip or failure still blocks them.
…eams

A round-trip load skipped at the first gating task or asset that did not
succeed. It now follows the rule every other Python node follows: a
failed or hard-skipped upstream skips it hard, any successful upstream
lets it run, and it skips softly only when every upstream skipped softly
or did not run.
…ectly

A load ran when any task or asset anywhere upstream of it succeeded, so a
succeeding task behind a soft-skipped one loaded that one's stale result.
Only the tasks and assets a loader reads directly, or through other
loaders, can now let it run; a failed or hard-skipped one anywhere still
skips it hard. With --no-python, a read-side gate that did not run
soft-skips the load even when an ingress gate succeeded.
@kvlonge
kvlonge enabled auto-merge November 17, 2026 21:36
@kvlonge
kvlonge added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit 66652f6 Nov 17, 2026
34 checks passed
@kvlonge
kvlonge deleted the fix/round-trip-skip-modes-pr branch November 17, 2026 22:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant