Repository navigation
fix(plan): read deferred upstreams from the deferred target and trust synced records from their last scan - #1584
Merged
Merged
Conversation
… synced records from their last scan When deferring, unselected model and seed upstreams must exist in the deferred target, matched under their deferred relation names in every database the deferred locations use. S301 names the target each missing upstream was looked for in. A synced plan treats a mirrored record as current from the start of the last catalog scan that found it unchanged, timed by the server's clock, keeping the clock-skew allowance.
…ved it A synced plan trusted a mirrored record from the start time in its scan id, which the location's clock sets. A location clock running ahead could push that past a later change and skip describing a table that changed. Each catalog scan now stamps the records it re-delivers unchanged with the store's receipt time, and the plan trusts a record from the earlier of its scan's start and that receipt, keeping the clock-skew allowance.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
With
--defer-to, unselected upstreams were checked in the active target, so one that existed only in the deferred target was reported missing, and an aliased ormigrate_fromupstream could be judged by the wrong name. A synced plan trusted a mirrored record only from its first write, so an unchanged table a later scan re-confirmed was described live on every plan.Changes
migrate_fromorigins resolve correctly. The deferred target is listed once per database it uses.migrate_fromorigin leaves an upstream missing, S301 now reports it in place of the old "could not list origin" warning.defer_toreaches the deferral inputs, including dbt interop.received_at), so no location clock can date a record after its page read.edge_cases/deferred_reads, an aliased deferred upstream.Verification
received_at. The synced-scan same-columns case fails on main; its live-describe controls pass on both.make check-cipass.