Skip to content

An ingestion-time view with no time window below it (SQL table, PromQL offset over raw rows) is planned but refused at install #1288

Open

Description

@zzylol

Example

Two cases, one SQL and one PromQL. Both were observed with the review harness on the Stage 3 baseline branch.

SQL: a table with no time window. The data arrives at 1,000 rows/s and the query runs every 10 s:

SELECT approx_percentile_cont(bytes, 0.9) FROM flows

(rank error 0.01; SELECT count(DISTINCT dst) FROM flows at error 0.05 behaves the same.)

  • Today: Stage 3 selects 0 Scan(flows) (IngestionTime, None) → 1 Aggregate over the whole table (IngestionTime, None) → 2 Project (IngestionTime, AcrossEvaluations), costed at 0.0076/s. plan_installable refuses it: Uninstallable(ViewReadsUnwindowedSource { node_id: 2 }). Both queries get the same plan and the same refusal.
  • Expected: either a plan the runtime can install (for example the query-time plan, which reads the table at each evaluation), or the ingestion-time view kept, with the executor refreshing a view over the whole table as rows arrive.

PromQL: an offset over raw rows. The query runs every 60 s, raw data is not kept (raw_data_retained: false), and there are 1,000 series at 1 sample/s:

sum_over_time(m[5m] offset 90s)
  • Today: Stage 3 selects 0 Scan(m) (IngestionTime, None) → 1 TimeShift(90s) (IngestionTime, AcrossEvaluations, retention_ms 300,000) → 2 TimeRange(300s) → SummaryAgg → Finalize. The TimeShift sits between the Scan and the range, so the materialized view at node 1 reads raw rows with no window below it. plan_installable refuses it: Uninstallable(ViewReadsUnwindowedSource { node_id: 1 }).
  • Expected: the same plan installs, with the executor refreshing a shifted view of raw rows that keeps the last 5 minutes (its retention_ms).

Why

PhysicalASAPDAG::materialized_views (crates/types/src/plan/materialized_view/views.rs, inputs_of_view) refuses a view that reads a Scan other than through a raw-data TimeRange or a chain's tumbling window. An ingestion run refreshes the window that ends at the run's end. An unwindowed SQL table has no window, and a TimeShift placed directly over the Scan has none below it either.

Stage 2 still offers these ingestion-time candidates: views over an unwindowed table, and (IngestionTime, AcrossEvaluations) TimeShift views over a raw Scan. Stage 3 does not know the installer refuses them, so it picks them when they are cheapest.

For the PromQL case, Stage 3 also under-costs the view today: it charges one ingestion run's rows instead of the rows its retention_ms covers. That cost fix is in progress in the #1207 PR 6 branch. After it, these views are charged their full retention, so the planner may select them less often. They remain valid plans, so the installer still has to accept them.

What works nearby: the same SQL queries with a time window (WHERE ts > now() - INTERVAL '5 minutes') or with a declared rows_held plan and install. For the PromQL query, the cheapest plan without ingestion-time TimeShift views is a chain plan (0.1043/s against 0.01225/s for the selected view under today's cost).

Proposal

The planner must not be restricted by today's executor. Two options:

  1. Executor: refresh these views in the executor.
    • A view over an unwindowed SQL source becomes an all-time accumulation. Each ingestion run merges its window's rows into the stored state (a running exact aggregate or a sketch merge), and the view keeps one state.
    • A shifted view of raw rows keeps the rows its retention_ms covers, indexed by time. Each ingestion run appends its rows and drops rows older than the retention.
    • The installer then accepts both, with retention_ms covering what the view keeps.
  2. Cost and Stage 2 (if 1 is out of reach): cost these views by what they keep and do, and keep the planner's choice. Only the executor work in 1 makes them installable.

Option 1 is the recommendation.

Scope and acceptance

In scope: an ingestion-time view that reads a Scan with no window below it. This covers SQL tables with no time window and PromQL views placed between the Scan and the range, such as a TimeShift for offset.

Acceptance: the SQL queries above and sum_over_time(m[5m] offset 90s) (with raw data not kept) plan, install (plan_installable returns Ok), and answer equal to the exact reference over the same rows. Each is a planner integration test that fails today.

Related

Found by the functional reviews of the Stage 3 baseline (#1236, PR 6 of #1207):

  • SQL: workloads s01 and x13.
  • PromQL: workload y13. y05, y08 and y18 (raw data not kept) and y02 and z03 (raw data kept) select the same kind of TimeShift view.

Executor gaps are fixed in the executor, not by restricting the planner.

Activity

  1. added
    bugSomething isn't working
    sqlSQL front-end lowering (DataFusion → L2)
    windowWindow functions, frames, panes, or temporal semantics
    on Nov 12, 2026
  2. zzylol commented on Nov 12, 2026

    @zzylol
    ContributorAuthor

    Four more workloads of the same review hit the same refusal (ViewReadsUnwindowedSource) on the Stage 3 baseline branch, all over the unwindowed table flows:

    • SELECT src, sum(bytes) AS s FROM flows GROUP BY src ORDER BY s DESC LIMIT 5
    • SELECT src, sum(-bytes) AS s FROM flows GROUP BY src ORDER BY s DESC LIMIT 5 (twice, with and without an accuracy target)
    • SELECT (SELECT approx_percentile_cont(bytes, 0.9) FROM flows)

    The acceptance test should cover a grouped top-k and a scalar subquery too.

  3. changed the title [-]An ingestion-time view over a SQL table with no time window is planned but refused at install[/-] [+]An ingestion-time view with no time window below it (SQL table, PromQL offset over raw rows) is planned but refused at install[/+] on Nov 13, 2026
  4. added
    promqlPromQL front-end lowering (L1→L2)
    on Nov 13, 2026
  5. zzylol commented on Nov 15, 2026

    @zzylol
    ContributorAuthor

    Decision: running-aggregate views over a whole SQL table are fixed here, in the executor. The executor refreshes such a view by accumulating each ingestion run's rows into the view's state; Stage 2 keeps offering the ingestion-time candidate, because an executor gap must not restrict the planner's design.

    #1320 reported the same plan shape for SELECT sum(a) FROM t (and max, count(*), avg, stddev) and is closed as a duplicate of this issue. Its root-cause check on main is here: #1320 (comment). The acceptance test should cover those queries too, with the ingesting arrival from #1320 (rows_per_second 100, rows_held 1000) giving the same answers as with "at_rest". The separate corr(a, b) costing bug is #1321.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpromqlPromQL front-end lowering (L1→L2)sqlSQL front-end lowering (DataFusion → L2)windowWindow functions, frames, panes, or temporal semantics

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions