Repository navigation
Description
Problem, by example
Table t(ts TIMESTAMP, k UTF8, a DOUBLE NULL), time index ts, no unique key. Workload: one repeating query every 10 s at phase 0, exact (max_error 0), data arrival {"ingesting": {"rows_per_second": 100.0}}, source facts t: {series_count: 10, sample_interval: 10000, rows_held: 1000}. plan_installable on main (59365ce80):
| Query (exact) | Today |
|---|---|
SELECT sum(a) FROM t |
installable plan: materialized view 2 reads a source outside a raw-data window |
SELECT max(a) FROM t |
the same error |
SELECT count(*) FROM t |
the same error |
SELECT corr(a, a) FROM t |
… materialized view 1 reads a source outside a raw-data window |
SELECT k, corr(a, b) FROM t GROUP BY k (with a column b added) |
… materialized view 1 reads a source outside a raw-data window |
SELECT k, sum(a) FROM t GROUP BY k |
installs |
The same queries over the table declared "at_rest" install. Expected: each query installs with the ingesting arrival too, or Stage 3 never selects a plan that installation refuses.
The plan Stage 3 selects for SELECT sum(a) FROM t (from e2e_plan):
0 Scan(t) IngestionTime Raw lifetime None
1 Aggregate sum(a), no keys IngestionTime Raw lifetime None
2 Project IngestionTime Raw lifetime AcrossEvaluations(Memory), retention 0
For the grouped sum it selects a query-time SummaryAgg → FinalizeExactAccumulator → Project, which installs.
Why it happens
- Installation refuses a view whose refresh reads a source
Scanother than through a raw-dataTimeRange:InstallableDagError::ViewReadsUnwindowedSource(crates/types/src/plan/materialized_view/mod.rs). A whole SQL table has no window an ingestion run can refresh. - Stage 2 still offers ingestion time to the aggregate over the whole table, and Stage 3 selects it as the cheapest plan. Nothing before installation checks the view's raw-data window. This is the same class of bug as Stage 3 selects plans whose raw-data windows installation refuses (subquery over a range) #1045 (a subquery over a range), with a different cause: a windowless table read instead of a re-timed range.
Proposal
Options:
- Stage 2 does not offer
IngestionTimeto a node whose input reaches aScanof a SQL table without aTimeRangein between (no raw-data window). The selected plan is then the query-time one, as for"at_rest". - Stage 3 rejects a candidate whose views fail the installation check, as Stage 3 selects plans whose raw-data windows installation refuses (subquery over a range) #1045 proposes for
check_raw_data_windows. This is one rule for both bugs, but it is applied later, after the candidates are built.
Option 1 removes candidates that can never install, so Stage 3 does less work. Option 2 also covers #1045.
Scope and acceptance
- Done when every query in the table installs with the ingesting arrival and gives the same answer as with
"at_rest". Failing-first test:plan_installableofSELECT sum(a) FROM twith the workload above. - Out of scope: maintaining a whole-table aggregate incrementally at ingestion. That needs a runtime that feeds a view every appended row, which no installed plan describes today.
Related
#1045 (Stage 3 selects plans whose raw-data windows installation refuses), the reduce-aggregates executor PR (#1242), whose SQL tests declare the table at rest because of this bug.
Activity
zzylol commentedon Nov 15, 2026
Root cause check on main (59365ce). The queries in the table above get two different plan shapes:
sum,max,count(*),avg,stddev. The plan isScan→Aggregate(IngestionTime, None) →Project(IngestionTime, AcrossEvaluations,retention_ms0). The view holds one aggregate row, the right size for a running aggregate. This is the SQL case of An ingestion-time view with no time window below it (SQL table, PromQL offset over raw rows) is planned but refused at install #1288, whose recommendation is that the executor refreshes such a view (accumulates each run's rows into the state), not that Stage 2 stops offering it.corr(a, b). TheAggregateruns at query time over a raw-rowProject(IngestionTime, AcrossEvaluations,retention_ms0). That view is a copy of the table's rows, but the retention walk gives it 0 andasap-costcharges it and its reader for one row. The plan costs 1.19e-4/s, against 7.8e-4/s for the query-time plan, which is charged for 1,000 rows.SELECT a FROM tgets the same kind of view, as its root. That plan is wrong even for an installer that accepted views over unwindowed sources. Filed separately as SQL whole-table read: Stage 3 stores the table's raw rows as an ingestion-time view with retention 0 and costs it as one row (SELECT corr(a, b) FROM t) #1321, with the cause inwindows_readers_needandcurrent_time_read_needs_every_row.feat(executor): run grouped stddev/variance, corr, group, count_value…
zzylol commentedon Nov 15, 2026
Closing as a duplicate of #1288.
The plan shape this issue reports for
sum,max,count(*),avgandstddev(an ingestion-timeAggregateover the whole table, keptAcrossEvaluations) is the SQL case of #1288: a materialized view with no raw-data window below it, whichPhysicalASAPDAG::materialized_viewsrefuses withViewReadsUnwindowedSource.Decision: the fix follows #1288. The plan stays valid, and the executor learns to refresh a running-aggregate view over a whole SQL table (each ingestion run accumulates its rows into the view's state). We do not take option 1 here (Stage 2 stops offering
IngestionTimeover an unwindowed table), because an executor gap must not restrict the planner's design: removing the candidate would hide a plan that is cheaper once the executor supports it.The root-cause comment above (#1320 (comment)) carries over to #1288. The
corr(a, b)/SELECT a FROM tcosting bug it found is tracked separately in #1321 and stays open.feat(executor): run grouped stddev/variance, corr, group, count_value…
feat(executor): grouped stddev/variance, corr, group, count_values, e…