Skip to content

[Bug] check_compiles accepts a plan whose view over an ingestion-time chain cannot be refreshed on its own #1291

Open

Description

@zzylol

Example

sum_over_time(m[5m] offset 1h) every minute, 1,000 gauge series at one sample per second, no raw data kept (raw_data_retained: false). Take the plan in which the query's TimeShift(1h) is itself a materialized view over an ingestion-time chain of 60 s windows:

0 Scan(m)                         ingestion time
1 SummaryTumblingWindowChain(60s) ingestion time, materialized (view 1)
2 TimeShift(1h)                   ingestion time, materialized (view 2)
3 TimeRange(5m) -> merge -> finalize   query time

Stage 3 does not select this plan today (it only duplicates the chain), so the plan was forced in a review harness (cases f01 offset 1h, f02 offset 90s, f03 offset 5m, each with view 2's retention set to 300,000 ms and to 3,900,000 ms).

Check Result
installable_plan_of / InstallablePlan::validate pass
InstallablePlan::check_compiles pass
Refresh of view 2 alone, view 1 read as a stored materialized view compile: invalid DAG: materialized view 1 is read unbounded

So the install checks accept a plan whose view refresh does not compile. They should either refuse it, or every refresh they accept should compile.

Current behavior

check_compiles (crates/planner/src/installable/mod.rs, executor_compiles and executor_refreshes) compiles each ingestion refresh group as one DAG, with every view of the group as a root, plus each view that is not refreshed at ingestion on its own. Views 1 and 2 both get the refresh window of the chain (60 s): refresh_windows takes the union of windows_below of each view, and for view 2 that is the chain's window. So they are one group and are compiled together, with the chain as a root that is computed from raw rows. The chain is never compiled as a materialized-view slot read by view 2.

When view 2 is refreshed with view 1 as a stored slot, the executor's physical_planner::compile checks the slot's read window with window_read_above(1, ..) (crates/executor/src/physical_planner/mod.rs, materialized view slots). The only reader of view 1 is view 2, a TimeShift without @, which is a Through reader and a compile root, so the walk maps it to Whole and returns None: "read unbounded".

Expected behavior and invariant

A plan that passes check_compiles can be refreshed view by view: for each materialized view, the DAG of the view's refresh, with the views below it as materialized-view slots, compiles. Or, if a runtime must refresh one group in one run, IngestionRefreshGroup and check_compiles say so, and the views of one group are never refreshed apart.

Options:

  1. check_compiles also compiles each view's own refresh (roots: the view; slots: the views it reads, MaterializedView::inputs), and refuses the plan when one fails.
  2. The executor accepts a materialized-view slot read by a root TimeShift view: the slot is read through the shift's offset plus the window its own readers read (the shift's retention_ms already holds both since the shared retention walk starts at the producer's own offset).

Option 1 makes the checks match a view-by-view runtime. Option 2 makes such a refresh work. They can both be done.

Minimal reproduction

The forced plans and the harness are review scratch, not committed. A unit test can build the DAG above by hand (the installer tests' chain_plan helper builds the chain, with an optional TimeShift view over it), then compile view 2's refresh with view 1 as a slot through asap_executor::physical_planner::compile.

Impact

Latent today: no workload in the review sets (round 1, round 2, a01 to a22) selects a view over an ingestion-time chain. A planner change that makes such a view cheaper would produce plans that install and then fail at their first refresh.

Regression test

A planner installer test with the DAG above: check_compiles returns Err naming view 2 (option 1), or view 2's refresh compiles and its rows equal the query-time result (option 2). It fails today: check_compiles returns Ok(()).

End-to-end verification

Rerun the forced f01 to f03 plans through install, refresh and query: either install refuses them, or the answers equal the exact reference.

Activity

  1. added
    bugSomething isn't working
    windowWindow functions, frames, panes, or temporal semantics
    on Nov 13, 2026
  2. zzylol commented on Nov 13, 2026

    @zzylol
    ContributorAuthor

    How #1236 now handles a stored TimeShift, and what option 2 has to change

    #1236 (head after round 5) gives a TimeShift without @ that is materialized across evaluations at ingestion time its own read kind, TimeIndexedRead::MaterializedTimeShift { offset_ms } (from TimeIndexedRead::of(operator, timing, lifetime)):

    • Its readers read its stored rows offset_ms later, so its own retention_ms is offset + range. For the f01 example above, sum_over_time(m[5m] offset 1h), that is 3,900,000 ms.
    • Each refresh reads its input's newest output, so the shared walk stops at it: a chain below it is read through {0, 0}, its newest window, and gets retention_ms 0. The store always keeps the rows appended last, so that keeps one window. Over raw data, which has no newest output, the walk reads unbounded.

    The executor still refuses this refresh. Its slot walk (physical_planner::time_window::window_read_above) reads every consumer outside a window unbounded. A materialized shift is such a consumer, so a chain slot under it is read unbounded and the compile fails with "materialized view 1 is read unbounded". Over a raw Scan, the refresh compiles and reads every row. The executor test a_materialized_time_shift_over_a_chain_reads_the_chain_unbounded pins this.

    Option 2 has to change two things:

    1. Make the executor's walk read a chain slot under a MaterializedTimeShift as its newest window, i.e. accept {0, 0} there, and update that test. In refactor(stage3): baseline selection MILP and its PhysicalASAPDAG; executor θ, native chains and a time-indexed store (#1207 PR 6) #1236's review, removing the refusal made the refresh ask for the chain's rows over (x, x], which fails with "no rows tiling (x, x]": today the executor reads {0, 0} literally, as an empty window, not as "the newest window".
    2. Define how the shift stamps the rows it stores, so that its readers' TimeRange over it, read offset_ms later, finds them. With a long chain retention and no refusal, the forced f01 to f03 plans ran and answered 0 of 12 series.
  3. zzylol commented on Nov 14, 2026

    @zzylol
    ContributorAuthor

    Approved design: one fix for #1291 and #1292

    This comment is posted on both #1291 and #1292. They are fixed together, in one PR that closes both, after #1300 merges. No code is written until then.

    Problem, by example

    Take f01's plan for sum_over_time(m[5m] offset 1h), evaluated every 60 s, with no raw data kept:

    0 Scan(m)                              ingestion time
    1 SummaryTumblingWindowChain(60 s)     ingestion time, stored (view 1), retention 0
    2 TimeShift(1h)                        ingestion time, stored (view 2), retention 3,900,000
    3 TimeRange(5m) -> SummaryMerge -> FinalizeExactAccumulator   query time
    

    Measured on #1300's head (48819b44a), with a hand-built plan of this shape and with the review harness on the forced plans f01–f03 and g01–g04/g13:

    Runtime Refresh of view 2 Query at T Answer
    View by view (view 1 is a stored slot under view 2) refused: "materialized view 1 is read unbounded". With the refusal removed, slot 1 is read over (e, e], an empty request, and view 2 stores 0 rows. refused, or 0 rows
    Both views refreshed in one run (as check_compiles compiles them) compiles view 2 is read over (T − 60 s, T] (from window_below) instead of (T − 65m, T − 60m], and TimeRange(5m) then drops every row [], 0 of 12 series

    What should happen: the refresh at e reads the chain's output keyed in (e − 60 s, e], and the query at T reads view 2's outputs keyed in (T − 65m, T − 60m]. The result is Σ m over (T − 65m, T − 60m], which is equal to the exact answer.

    Why it happens

    1. Time semantics are implemented in several places, and they disagree.
    2. Width 0 has no stated meaning.
      • The planner reads the chain under a stored shift through {0, 0}, meaning "the newest output", and gives the chain retention 0. That relies on an unwritten rule: the store always keeps the newest output.
      • The executor turns {0, 0} into the empty request (e, e].
    3. The failure is silent.
      • InMemoryDataStore::read_materialized_view returns no rows, and no error, for (e, e].
      • check_compiles accepts the plan.

    Proposal (approved)

    1. Every time window is over the time keys of stored outputs.

    • Keys. A chain's output is keyed by its window end. Any other stored output is keyed by the end of the interval it summarizes.
    • Reads. A read {o, w} at evaluation time t means the outputs whose key lies in (t − o − w, t − o].
    • Width 0 is no longer allowed. "The newest output" is {o, output_interval}, which for a chain of width c is {o, c}.
    • Cost.
      • Chains are unchanged: div_ceil(c, c) is 1 retained window, the same as with retention 0.
      • A non-chain stored result that is read only at its current time keeps one more output interval. That is the correct cost: today a reader at an offset o is given retention o, and the store, which drops outputs keyed at or before newest − retention, drops the output keyed at exactly t − o.

    2. The planner computes each reader's window once and writes it into the plan.

    pub struct PhysicalASAPDAGEdge {
        pub producer: PhysicalASAPNodeId,
        pub consumer: PhysicalASAPNodeId,
        pub input_index: usize,
        /// Which of the producer's time-indexed outputs this consumer reads, relative to the
        /// consumer's evaluation time t: time keys in (t - ends_before_evaluation_ms - length_ms,
        /// t - ends_before_evaluation_ms]. None = every output (or the producer is not time-indexed).
        pub read_window: Option<TimeWindow>,
    }

    Example, with the plan above:

    Edge read_window Meaning
    chain → stored offset 1h shift {0, 60_000} at refresh e, the chain output keyed in (e − 60 s, e]
    shift → TimeRange(5m) {3_600_000, 300_000} at query T, the shift outputs keyed in (T − 65m, T − 60m]

    3. Fail loudly.

    Scope and acceptance

    In scope (one PR):

    Acceptance:

    • The f01-shaped executor test answers the direct sums in both runtimes (view by view, and both views in one run). Today it is refused, or answers [].
    • Forced f01–f03 and g01–g04/g13 match exact on all 12 series. They cover:
      • offset < range: 90 s on [5m];
      • offset = range: 5m on [5m];
      • offset > range: 1h on [5m], and 2m on [1m];
      • phases 15, 20 and 30 s;
      • other readers of the same chain.
    • A read whose window is not covered by the stored outputs errors, and does not return 0 rows.
    • check_compiles refuses an edge whose read_window reaches past its producer's retention.
    • An execution-vs-exact test of a stored offset shift runs in CI.

    Status and related work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingwindowWindow functions, frames, panes, or temporal semantics

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions