Repository navigation
Open
Labels
cost-modelCost formulas, statistics, ranking, and selectionCost formulas, statistics, ranking, and selectionenhancementNew feature or requestNew feature or requestwindowWindow functions, frames, panes, or temporal semanticsWindow functions, frames, panes, or temporal semantics
Description
Problem, by example
Three query shapes now run, but the window of raw data they read is not bounded where the executor reads it, where the installable plan tells the deployment how long to keep raw data, or where Stage 3 prices it:
| Query (exact) | Rows the query needs | Executor Scan window |
raw_data_retention() |
Stage 3 prices the scan as |
|---|---|---|---|---|
max_over_time(rate(m[1m])[10m:1m]) |
(T - 11m, T] |
(T - 11m, T] |
unbounded (keep every row) | (T - 1m, T] (the inner range only) |
max_over_time(m[5m] @ end()) in a range query [S, E] |
(E - 5m, E] |
unbounded | unbounded | unbounded: needs the source's rows_held, and fails without it (pricing source m needs its rows_held) |
SQL SELECT service, asap_rate(latency, ts, 300000) FROM metrics GROUP BY service |
(T - 5m, T] |
unbounded (the whole table) | unbounded | the widest window read, or the source's rows_held |
Every answer is correct (each TimeRange and series window filters its rows); the cost is reading and keeping more raw data than needed, and Stage 3 under-pricing subqueries.
Why it happens
ScanWindow(crates/types/src/ir/physical_export.rs) is relative to the evaluation time:Within { end_offset_ms, width_ms }orUnbounded. An@anchor is absolute, soPhysicalASAPDAG::scan_windowsreturnsUnboundedfor it.scan_windowswalks aScan's consumers throughTimeShifts to aTimeRange; aProjectbetween them (SQLasap_rate) makes itUnbounded.- A subquery re-times the ranges below it.
query_time_scan_windows(crates/types/src/ir/materialized_view.rs) makes such a scan unbounded for retention; the executor widens its own read window (Subquery::scan_window); Stage 3's pricing (asap_costplan_costthroughscan_window, and the selection MILP throughwindow_read_throughin crates/plan-selection/src/groups.rs) still sees the inner range only. The two must change together, or the MILP's objective andplan_costdisagree (the check in plan selection).
Proposal
ScanWindowgains the subquery span: a scan read under subqueries of rangesR_iand offsetso_ireads(T - Σo - ΣR - off - r, T - Σo - off]; compute it in one walk shared byscan_windows, retention and both pricers.- An anchored window:
ScanWindow::Anchored { at: AtModifier, end_offset_ms, width_ms }, resolved by the deployment from the query (@ <t>fixed,start()/end()from the request range); retention for@ start()/end()is the window plus the range span. - A
Project/Filterthat keeps the time column unchanged passes the window through.
Scope and acceptance
- Each row of the table above gets a bounded executor window and retention, and Stage 3 prices the same window it hands the deployment (the MILP objective equals
plan_cost). - Tests:
scan_window/raw_data_retentionfor the three queries;plan_installableofmax_over_time(m[5m] @ end())withoutrows_heldsucceeds.
Related
#955 (closed; raw retention groups, @ lookbacks), the time-operators PR (exec/x4-time-operators), executor gaps 29 and 30.
Metadata
Metadata
Assignees
Labels
cost-modelCost formulas, statistics, ranking, and selectionCost formulas, statistics, ranking, and selectionenhancementNew feature or requestNew feature or requestwindowWindow functions, frames, panes, or temporal semanticsWindow functions, frames, panes, or temporal semantics
Activity
zzylol commentedon Nov 15, 2026
The time-operators PR (#1250) is closed as superseded, so the three shapes in the table above do not run on main (
59365ce80):value operation has no native implementation);@ end()/@ <t>and SQLasap_rateare refused by the install checkcheck_raw_window_below:an @ anchor below the range, andan operator other than TimeShift below the range.The execution side is re-filed as #1318 (subqueries), #1306 (
@, commented there), #748 (counter@) and #1319 (TimeRangeover computed rows). This issue keeps the bounded scan window, retention and Stage 3 pricing for those shapes.Subquery::scan_windowandquery_time_scan_windows, named above, are from the closed branch.