Skip to content

Bounded scan windows, retention and Stage 3 pricing for subqueries, @ anchors and ranges over projected rows #1249

Open

Description

@zzylol

Problem, by example

Three query shapes now run, but the window of raw data they read is not bounded where the executor reads it, where the installable plan tells the deployment how long to keep raw data, or where Stage 3 prices it:

Query (exact) Rows the query needs Executor Scan window raw_data_retention() Stage 3 prices the scan as
max_over_time(rate(m[1m])[10m:1m]) (T - 11m, T] (T - 11m, T] unbounded (keep every row) (T - 1m, T] (the inner range only)
max_over_time(m[5m] @ end()) in a range query [S, E] (E - 5m, E] unbounded unbounded unbounded: needs the source's rows_held, and fails without it (pricing source m needs its rows_held)
SQL SELECT service, asap_rate(latency, ts, 300000) FROM metrics GROUP BY service (T - 5m, T] unbounded (the whole table) unbounded the widest window read, or the source's rows_held

Every answer is correct (each TimeRange and series window filters its rows); the cost is reading and keeping more raw data than needed, and Stage 3 under-pricing subqueries.

Why it happens

  • ScanWindow (crates/types/src/ir/physical_export.rs) is relative to the evaluation time: Within { end_offset_ms, width_ms } or Unbounded. An @ anchor is absolute, so PhysicalASAPDAG::scan_windows returns Unbounded for it.
  • scan_windows walks a Scan's consumers through TimeShifts to a TimeRange; a Project between them (SQL asap_rate) makes it Unbounded.
  • A subquery re-times the ranges below it. query_time_scan_windows (crates/types/src/ir/materialized_view.rs) makes such a scan unbounded for retention; the executor widens its own read window (Subquery::scan_window); Stage 3's pricing (asap_cost plan_cost through scan_window, and the selection MILP through window_read_through in crates/plan-selection/src/groups.rs) still sees the inner range only. The two must change together, or the MILP's objective and plan_cost disagree (the check in plan selection).

Proposal

  • ScanWindow gains the subquery span: a scan read under subqueries of ranges R_i and offsets o_i reads (T - Σo - ΣR - off - r, T - Σo - off]; compute it in one walk shared by scan_windows, retention and both pricers.
  • An anchored window: ScanWindow::Anchored { at: AtModifier, end_offset_ms, width_ms }, resolved by the deployment from the query (@ <t> fixed, start()/end() from the request range); retention for @ start()/end() is the window plus the range span.
  • A Project/Filter that keeps the time column unchanged passes the window through.

Scope and acceptance

  • Each row of the table above gets a bounded executor window and retention, and Stage 3 prices the same window it hands the deployment (the MILP objective equals plan_cost).
  • Tests: scan_window/raw_data_retention for the three queries; plan_installable of max_over_time(m[5m] @ end()) without rows_held succeeds.

Related

#955 (closed; raw retention groups, @ lookbacks), the time-operators PR (exec/x4-time-operators), executor gaps 29 and 30.

Activity

  1. added
    enhancementNew feature or request
    cost-modelCost formulas, statistics, ranking, and selection
    windowWindow functions, frames, panes, or temporal semantics
    on Nov 7, 2026
  2. zzylol commented on Nov 15, 2026

    @zzylol
    ContributorAuthor

    The time-operators PR (#1250) is closed as superseded, so the three shapes in the table above do not run on main (59365ce80):

    • the subquery fails in the executor (value operation has no native implementation);
    • @ end() / @ <t> and SQL asap_rate are refused by the install check check_raw_window_below: an @ anchor below the range, and an operator other than TimeShift below the range.

    The execution side is re-filed as #1318 (subqueries), #1306 (@, commented there), #748 (counter @) and #1319 (TimeRange over computed rows). This issue keeps the bounded scan window, retention and Stage 3 pricing for those shapes. Subquery::scan_window and query_time_scan_windows, named above, are from the closed branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    cost-modelCost formulas, statistics, ranking, and selectionenhancementNew feature or requestwindowWindow functions, frames, panes, or temporal semantics

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions