Skip to content

[Bug] Stage 3 selects an ingestion-time view over a whole SQL table, which installation refuses (SELECT sum(a) FROM t) #1320

Closed as duplicate

Description

@zzylol

Problem, by example

Table t(ts TIMESTAMP, k UTF8, a DOUBLE NULL), time index ts, no unique key. Workload: one repeating query every 10 s at phase 0, exact (max_error 0), data arrival {"ingesting": {"rows_per_second": 100.0}}, source facts t: {series_count: 10, sample_interval: 10000, rows_held: 1000}. plan_installable on main (59365ce80):

Query (exact) Today
SELECT sum(a) FROM t installable plan: materialized view 2 reads a source outside a raw-data window
SELECT max(a) FROM t the same error
SELECT count(*) FROM t the same error
SELECT corr(a, a) FROM t … materialized view 1 reads a source outside a raw-data window
SELECT k, corr(a, b) FROM t GROUP BY k (with a column b added) … materialized view 1 reads a source outside a raw-data window
SELECT k, sum(a) FROM t GROUP BY k installs

The same queries over the table declared "at_rest" install. Expected: each query installs with the ingesting arrival too, or Stage 3 never selects a plan that installation refuses.

The plan Stage 3 selects for SELECT sum(a) FROM t (from e2e_plan):

0 Scan(t)                     IngestionTime Raw  lifetime None
1 Aggregate sum(a), no keys   IngestionTime Raw  lifetime None
2 Project                     IngestionTime Raw  lifetime AcrossEvaluations(Memory), retention 0

For the grouped sum it selects a query-time SummaryAgg → FinalizeExactAccumulator → Project, which installs.

Why it happens

  • Installation refuses a view whose refresh reads a source Scan other than through a raw-data TimeRange: InstallableDagError::ViewReadsUnwindowedSource (crates/types/src/plan/materialized_view/mod.rs). A whole SQL table has no window an ingestion run can refresh.
  • Stage 2 still offers ingestion time to the aggregate over the whole table, and Stage 3 selects it as the cheapest plan. Nothing before installation checks the view's raw-data window. This is the same class of bug as Stage 3 selects plans whose raw-data windows installation refuses (subquery over a range) #1045 (a subquery over a range), with a different cause: a windowless table read instead of a re-timed range.

Proposal

Options:

  1. Stage 2 does not offer IngestionTime to a node whose input reaches a Scan of a SQL table without a TimeRange in between (no raw-data window). The selected plan is then the query-time one, as for "at_rest".
  2. Stage 3 rejects a candidate whose views fail the installation check, as Stage 3 selects plans whose raw-data windows installation refuses (subquery over a range) #1045 proposes for check_raw_data_windows. This is one rule for both bugs, but it is applied later, after the candidates are built.

Option 1 removes candidates that can never install, so Stage 3 does less work. Option 2 also covers #1045.

Scope and acceptance

  • Done when every query in the table installs with the ingesting arrival and gives the same answer as with "at_rest". Failing-first test: plan_installable of SELECT sum(a) FROM t with the workload above.
  • Out of scope: maintaining a whole-table aggregate incrementally at ingestion. That needs a runtime that feeds a view every appended row, which no installed plan describes today.

Related

#1045 (Stage 3 selects plans whose raw-data windows installation refuses), the reduce-aggregates executor PR (#1242), whose SQL tests declare the table at rest because of this bug.

Activity

  1. added
    bugSomething isn't working
    optimizerWorkload optimization and plan selection
    sqlSQL front-end lowering (DataFusion → L2)
    on Nov 15, 2026
  2. zzylol commented on Nov 15, 2026

    @zzylol
    ContributorAuthor

    Root cause check on main (59365ce). The queries in the table above get two different plan shapes:

  3. zzylol commented on Nov 15, 2026

    @zzylol
    ContributorAuthor

    Closing as a duplicate of #1288.

    The plan shape this issue reports for sum, max, count(*), avg and stddev (an ingestion-time Aggregate over the whole table, kept AcrossEvaluations) is the SQL case of #1288: a materialized view with no raw-data window below it, which PhysicalASAPDAG::materialized_views refuses with ViewReadsUnwindowedSource.

    Decision: the fix follows #1288. The plan stays valid, and the executor learns to refresh a running-aggregate view over a whole SQL table (each ingestion run accumulates its rows into the view's state). We do not take option 1 here (Stage 2 stops offering IngestionTime over an unwindowed table), because an executor gap must not restrict the planner's design: removing the candidate would hide a plan that is cheaper once the executor supports it.

    The root-cause comment above (#1320 (comment)) carries over to #1288. The corr(a, b) / SELECT a FROM t costing bug it found is tracked separately in #1321 and stays open.

  4. reopened this on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingoptimizerWorkload optimization and plan selectionsqlSQL front-end lowering (DataFusion → L2)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions