Repository navigation
Open
Labels
enhancementNew feature or requestNew feature or requestsqlSQL front-end lowering (DataFusion → L2)SQL front-end lowering (DataFusion → L2)windowWindow functions, frames, panes, or temporal semanticsWindow functions, frames, panes, or temporal semantics
Description
Problem, by example
SQL asap_rate over a time-indexed table plans and then fails at install. Table metrics(ts TIMESTAMP, service UTF8, latency DOUBLE), time index ts, unique key (ts, service). Workload: one repeating query every 60 s at phase 0, exact, ingesting at 100 rows/s. plan_installable on main (59365ce80):
SELECT service, asap_rate(latency, ts, 300000) AS r FROM metrics GROUP BY service- Today:
installable plan: raw-data window of node 2 is unsupported: an operator other than TimeShift below the range. - Expected: the plan installs, and each
servicegets the rate oflatencyover(T - 5m, T].
The frontend lowers asap_rate as TimeRange(Project(Scan)): the Project puts the table's columns into the time-series layout the rate reads. So the TimeRange reads computed rows, not a Scan.
Why it happens
There are two refusals, in this order:
- Planner install check.
ViewIndex::check_raw_window_below(crates/types/src/plan/materialized_view/raw_data.rs) walks down from everyTimeRangethat reads raw data. It accepts onlyTimeShifts without@down to theScan, so aProjectgivesan operator other than TimeShift below the range. The check also runs on query-time windows, which no ingestion run refreshes. It exists for windows that a materialized view's refresh reads. - Executor. The executor's
TimeRangelowering expects to reach aScanthroughTimeShifts. This was observed on the closed branchexec/x4-time-operators, which had to extend the lowering to read any time-indexed input.
Proposal
- Apply the raw-data window shape check only to windows that a view's refresh reads (ingestion time). At query time the executor reads any shape. The closed branch did this.
- In the executor, lower a
TimeRangeover any time-indexed input. Keep the rows whose time column lies in the window the plan gives: the edgeread_windowof the approved time-walk design ([Bug] check_compiles accepts a plan whose view over an ingestion-time chain cannot be refreshed on its own #1291/[Refactor] Executor time walks (counter_window, node_below_shifts, below_time_shifts) should use the shared time_window walks #1292), not a new executor walk. AProjectorFilterthat keeps the time column unchanged passes the window through, as Bounded scan windows, retention and Stage 3 pricing for subqueries, @ anchors and ranges over projected rows #1249 proposes forScanWindow.
Scope and acceptance
- In: the query above, and a per-entity
SummaryAggover such a range. Each is planned exactly withplan_installableand run by the executor, and the answer matches the rate computed by hand over the rows in(T - 5m, T]. The closed branch has a test that can be ported:sql_asap_rate_reads_its_window_of_projected_rowsin itscrates/planner/tests/time_operators.rs. - Out: a bounded scan window, retention and Stage 3 pricing for this shape (Bounded scan windows, retention and Stage 3 pricing for subqueries, @ anchors and ranges over projected rows #1249 row 3; today the whole table is read).
Status and related work
- Re-filed from the closed time-operators PR (feat(executor): PromQL subqueries, @ anchors and time ranges over computed rows #1250, branch
exec/x4-time-operators, its "gap 30"), which is superseded by the approvedread_windowdesign. - Depends on the [Bug] check_compiles accepts a plan whose view over an ingestion-time chain cannot be refreshed on its own #1291/[Refactor] Executor time walks (counter_window, node_below_shifts, below_time_shifts) should use the shared time_window walks #1292 time-walk PR.
- Related: Bounded scan windows, retention and Stage 3 pricing for subqueries, @ anchors and ranges over projected rows #1249 (bounded scan windows, retention and pricing), Stage 3 selects plans whose raw-data windows installation refuses (subquery over a range) #1045 (install-time raw-data window refusals for a selected plan).
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestsqlSQL front-end lowering (DataFusion → L2)SQL front-end lowering (DataFusion → L2)windowWindow functions, frames, panes, or temporal semanticsWindow functions, frames, panes, or temporal semantics
Activity