Repository navigation
Description
Example
sum_over_time(m[5m] offset 1h) every minute, 1,000 gauge series at one sample per second, no raw data kept (raw_data_retained: false). Take the plan in which the query's TimeShift(1h) is itself a materialized view over an ingestion-time chain of 60 s windows:
0 Scan(m) ingestion time
1 SummaryTumblingWindowChain(60s) ingestion time, materialized (view 1)
2 TimeShift(1h) ingestion time, materialized (view 2)
3 TimeRange(5m) -> merge -> finalize query time
Stage 3 does not select this plan today (it only duplicates the chain), so the plan was forced in a review harness (cases f01 offset 1h, f02 offset 90s, f03 offset 5m, each with view 2's retention set to 300,000 ms and to 3,900,000 ms).
| Check | Result |
|---|---|
installable_plan_of / InstallablePlan::validate |
pass |
InstallablePlan::check_compiles |
pass |
| Refresh of view 2 alone, view 1 read as a stored materialized view | compile: invalid DAG: materialized view 1 is read unbounded |
So the install checks accept a plan whose view refresh does not compile. They should either refuse it, or every refresh they accept should compile.
Current behavior
check_compiles (crates/planner/src/installable/mod.rs, executor_compiles and executor_refreshes) compiles each ingestion refresh group as one DAG, with every view of the group as a root, plus each view that is not refreshed at ingestion on its own. Views 1 and 2 both get the refresh window of the chain (60 s): refresh_windows takes the union of windows_below of each view, and for view 2 that is the chain's window. So they are one group and are compiled together, with the chain as a root that is computed from raw rows. The chain is never compiled as a materialized-view slot read by view 2.
When view 2 is refreshed with view 1 as a stored slot, the executor's physical_planner::compile checks the slot's read window with window_read_above(1, ..) (crates/executor/src/physical_planner/mod.rs, materialized view slots). The only reader of view 1 is view 2, a TimeShift without @, which is a Through reader and a compile root, so the walk maps it to Whole and returns None: "read unbounded".
Expected behavior and invariant
A plan that passes check_compiles can be refreshed view by view: for each materialized view, the DAG of the view's refresh, with the views below it as materialized-view slots, compiles. Or, if a runtime must refresh one group in one run, IngestionRefreshGroup and check_compiles say so, and the views of one group are never refreshed apart.
Options:
check_compilesalso compiles each view's own refresh (roots: the view; slots: the views it reads,MaterializedView::inputs), and refuses the plan when one fails.- The executor accepts a materialized-view slot read by a root
TimeShiftview: the slot is read through the shift's offset plus the window its own readers read (the shift'sretention_msalready holds both since the shared retention walk starts at the producer's own offset).
Option 1 makes the checks match a view-by-view runtime. Option 2 makes such a refresh work. They can both be done.
Minimal reproduction
The forced plans and the harness are review scratch, not committed. A unit test can build the DAG above by hand (the installer tests' chain_plan helper builds the chain, with an optional TimeShift view over it), then compile view 2's refresh with view 1 as a slot through asap_executor::physical_planner::compile.
Impact
Latent today: no workload in the review sets (round 1, round 2, a01 to a22) selects a view over an ingestion-time chain. A planner change that makes such a view cheaper would produce plans that install and then fail at their first refresh.
Regression test
A planner installer test with the DAG above: check_compiles returns Err naming view 2 (option 1), or view 2's refresh compiles and its rows equal the query-time result (option 2). It fails today: check_compiles returns Ok(()).
End-to-end verification
Rerun the forced f01 to f03 plans through install, refresh and query: either install refuses them, or the answers equal the exact reference.
Activity
zzylol commentedon Nov 13, 2026
How #1236 now handles a stored
TimeShift, and what option 2 has to change#1236 (head after round 5) gives a
TimeShiftwithout@that is materialized across evaluations at ingestion time its own read kind,TimeIndexedRead::MaterializedTimeShift { offset_ms }(fromTimeIndexedRead::of(operator, timing, lifetime)):offset_mslater, so its ownretention_msis offset + range. For the f01 example above,sum_over_time(m[5m] offset 1h), that is 3,900,000 ms.{0, 0}, its newest window, and getsretention_ms0. The store always keeps the rows appended last, so that keeps one window. Over raw data, which has no newest output, the walk reads unbounded.The executor still refuses this refresh. Its slot walk (
physical_planner::time_window::window_read_above) reads every consumer outside a window unbounded. A materialized shift is such a consumer, so a chain slot under it is read unbounded and the compile fails with "materialized view 1 is read unbounded". Over a rawScan, the refresh compiles and reads every row. The executor testa_materialized_time_shift_over_a_chain_reads_the_chain_unboundedpins this.Option 2 has to change two things:
MaterializedTimeShiftas its newest window, i.e. accept{0, 0}there, and update that test. In refactor(stage3): baseline selection MILP and its PhysicalASAPDAG; executor θ, native chains and a time-indexed store (#1207 PR 6) #1236's review, removing the refusal made the refresh ask for the chain's rows over(x, x], which fails with "no rows tiling (x, x]": today the executor reads{0, 0}literally, as an empty window, not as "the newest window".TimeRangeover it, readoffset_mslater, finds them. With a long chain retention and no refusal, the forced f01 to f03 plans ran and answered 0 of 12 series.zzylol commentedon Nov 14, 2026
Approved design: one fix for #1291 and #1292
This comment is posted on both #1291 and #1292. They are fixed together, in one PR that closes both, after #1300 merges. No code is written until then.
Problem, by example
Take f01's plan for
sum_over_time(m[5m] offset 1h), evaluated every 60 s, with no raw data kept:Measured on #1300's head (
48819b44a), with a hand-built plan of this shape and with the review harness on the forced plans f01–f03 and g01–g04/g13:(e, e], an empty request, and view 2 stores 0 rows.check_compilescompiles them)(T − 60 s, T](fromwindow_below) instead of(T − 65m, T − 60m], andTimeRange(5m)then drops every row[], 0 of 12 seriesWhat should happen: the refresh at e reads the chain's output keyed in
(e − 60 s, e], and the query at T reads view 2's outputs keyed in(T − 65m, T − 60m]. The result is Σ m over(T − 65m, T − 60m], which is equal to the exact answer.Why it happens
crates/types/src/plan/time_window.rs) already decides each reader's window, and Stage 3 charges retention for it.CompileGraph::current_time_read_needs_every_rowreturnstruefor every node, where the planner returnstrueonly for raw data;window_below;counter_window,node_below_shiftsandbelow_time_shifts([Refactor] Executor time walks (counter_window, node_below_shifts, below_time_shifts) should use the shared time_window walks #1292) are three more copies.{0, 0}, meaning "the newest output", and gives the chain retention 0. That relies on an unwritten rule: the store always keeps the newest output.{0, 0}into the empty request(e, e].InMemoryDataStore::read_materialized_viewreturns no rows, and no error, for(e, e].check_compilesaccepts the plan.Proposal (approved)
1. Every time window is over the time keys of stored outputs.
{o, w}at evaluation time t means the outputs whose key lies in(t − o − w, t − o].{o, output_interval}, which for a chain of width c is{o, c}.div_ceil(c, c)is 1 retained window, the same as with retention 0.newest − retention, drops the output keyed at exactly t − o.2. The planner computes each reader's window once and writes it into the plan.
Example, with the plan above:
read_windowoffset 1hshift{0, 60_000}(e − 60 s, e]TimeRange(5m){3_600_000, 300_000}(T − 65m, T − 60m]read_windowinto store requests. The executor's walks are deleted:CompileGraph's window walks,window_belowfor view reads, and [Refactor] Executor time walks (counter_window, node_below_shifts, below_time_shifts) should use the shared time_window walks #1292'scounter_window,node_below_shiftsandbelow_time_shifts.check_compileschecks statically that each edge's window lies within its producer's retention.3. Fail loudly.
Scope and acceptance
In scope (one PR):
read_windowonPhysicalASAPDAGEdge).plan_offillsread_window.read_window, and the executor's own walks are deleted ([Refactor] Executor time walks (counter_window, node_below_shifts, below_time_shifts) should use the shared time_window walks #1292).Acceptance:
[].check_compilesrefuses an edge whoseread_windowreaches past its producer's retention.Status and related work
{o, 0}from a stored shift as the chain's newest window". That option was prototyped against fix(executor): run each node by its plan Lifetime, so roots drain in any order #1300's head:Lifetime, which this builds on;@anchors cannot be executed or verified #1306:@range queries; a shift with@still reads every output (read_window: None);