Skip to content

fix(plan): read deferred upstreams from the deferred target and trust synced records from their last scan - #1584

Merged
kvlonge merged 2 commits into
mainfrom
fix/defer-reads-and-synced-scans
Nov 18, 2026
Merged

kvlonge merged 2 commits into
mainfrom
fix/defer-reads-and-synced-scans

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Why

With --defer-to, unselected upstreams were checked in the active target, so one that existed only in the deferred target was reported missing, and an aliased or migrate_from upstream could be judged by the wrong name. A synced plan trusted a mirrored record only from its first write, so an unchanged table a later scan re-confirmed was described live on every plan.

Changes

  • When deferring, unselected model and seed upstreams must exist in the deferred target. Each one is matched by its deferred location's database, schema and name, so aliases and migrate_from origins resolve correctly. The deferred target is listed once per database it uses.
  • S301 prints one header per target. Models and seeds name the deferred target; functions name the active one, where they are checked. When an unlistable migrate_from origin leaves an upstream missing, S301 now reports it in place of the old "could not list origin" warning.
  • defer_to reaches the deferral inputs, including dbt interop.
  • A synced record counts as current from the later of its write time and the start of the last scan that found it unchanged. The 5-minute skew allowance stays; no new state.
  • Scan IDs are minted on the server's clock, and a record's scan start is capped at when the store received it (catalog re-scans now refresh received_at), so no location clock can date a record after its page read.
  • DuckShop: edge_cases/deferred_reads, an aliased deferred upstream.

Verification

  • New tests, each confirmed to fail without the change: a clone E2E test with an aliased upstream in the deferred target, deferred-location listing, S301 wording that names the target for a model and a function, a CLI test and a synced-plan case with a location clock 10 minutes fast, and a catalog re-scan refreshing received_at. The synced-scan same-columns case fails on main; its live-describe controls pass on both.
  • 1,139 targeted pytest tests, related Rust tests and make check-ci pass.

… synced records from their last scan

When deferring, unselected model and seed upstreams must exist in the
deferred target, matched under their deferred relation names in every
database the deferred locations use. S301 names the target each missing
upstream was looked for in.

A synced plan treats a mirrored record as current from the start of the
last catalog scan that found it unchanged, timed by the server's clock,
keeping the clock-skew allowance.
@kvlonge
kvlonge enabled auto-merge November 18, 2026 19:57
@kvlonge
kvlonge added this pull request to the merge queue Nov 18, 2026
@kvlonge
kvlonge removed this pull request from the merge queue due to a manual request Nov 18, 2026
…ved it

A synced plan trusted a mirrored record from the start time in its scan
id, which the location's clock sets. A location clock running ahead could
push that past a later change and skip describing a table that changed.

Each catalog scan now stamps the records it re-delivers unchanged with
the store's receipt time, and the plan trusts a record from the earlier
of its scan's start and that receipt, keeping the clock-skew allowance.
@kvlonge
kvlonge added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 77a12cc Nov 18, 2026
34 checks passed
@kvlonge
kvlonge deleted the fix/defer-reads-and-synced-scans branch November 18, 2026 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant