Repository navigation
fix: follow the Python fan-in rule for round-trip loads - #1559
Merged
Merged
Conversation
With --no-python the task gating a round-trip loader never runs, and the load was soft-skipped with "the SQL it reads was not rebuilt", which is not why. The reason now says --no-python skips read-side tasks and assets.
A round-trip load was hard-skipped whenever its gating task or asset skipped, so a task's default soft skip blocked every model reading the source. The load now takes the upstream's skip mode: a soft skip skips only the load, and models reading the source still build from its last loaded data. A hard skip or failure still blocks them.
…eams A round-trip load skipped at the first gating task or asset that did not succeed. It now follows the rule every other Python node follows: a failed or hard-skipped upstream skips it hard, any successful upstream lets it run, and it skips softly only when every upstream skipped softly or did not run.
…ectly A load ran when any task or asset anywhere upstream of it succeeded, so a succeeding task behind a soft-skipped one loaded that one's stale result. Only the tasks and assets a loader reads directly, or through other loaders, can now let it run; a failed or hard-skipped one anywhere still skips it hard. With --no-python, a read-side gate that did not run soft-skips the load even when an ingress gate succeeded.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A warehouse round-trip load (task or asset → loader → source → models) gated on its upstreams more strictly than any other Python node, and in one case more loosely. A task's default soft skip hard-skipped the load and blocked every model reading the source. The load skipped at the first gating upstream that did not succeed, even when another succeeded. A succeeding task behind a soft-skipped one could let the load run on stale results. With
--no-pythonthe skip reason was misleading.Changes
--no-python, a read-side gate that did not run soft-skips the load even if an ingress gate succeeded, and the reason names--no-python.Verification
test_warehouse_round_trips.pycover soft and hard skips, three fan-ins, a transitive gate,--no-pythonand an ingress gate; the last two failed before their fix. A Rust unit case covers the direct-gate set.python_plantests.make check-cipasses on this head.