Skip to content

Stage 3 does not share a result between queries on nested evaluation grids: the same exact sum every 60 s and every 120 s runs twice #1385

Open

Description

@zzylol

Problem, by example

The same exact query asked every 60 s and every 120 s (both at phase 0) is planned as two independent pipelines, although every evaluation of the 120 s copy happens at the same time as an evaluation of the 60 s copy and could read its result.

Workload: crates/integration-tests/tests/plan_sharing_fixtures/slower_repeat_exact_sum.json:

  • sum (sum_over_time(m_gauge[2m])), fixed_interval_at 60,000 ms, phase 0, exact;
  • the same query, fixed_interval_at 120,000 ms, phase 0, exact;
  • m_gauge: 10,000 gauge series, one sample every 15 s (1,333 rows/s), raw data kept.

Measured on main ca4be93f0 with default CostInputs (plan_sharing::slower_repeat_reuses_faster_sum and a local dump of the selected plan and its asap_cost::plan_cost):

Plan Nodes Cost per second
First query alone Scan → TimeRange(2m) → SummaryAgg(exact Sum) → FinalizeExactAccumulator, every 60 s 0.05901
Both queries, selected the pipeline above every 60 s, plus a second copy of all four nodes every 120 s 0.08851
Both queries, witness the first query's four nodes only; the second query's root is the first query's FinalizeExactAccumulator 0.08851 as priced today; 0.05901 if one run at each even minute serves both queries

The witness DAG validates (PhysicalASAPDAG::validate). Today plan_cost rates its nodes at 0.0250 runs/s (1/60 + 1/120) instead of 0.0167 runs/s (1/60), so it is priced the same as two copies, and Stage 3, which uses the same rule, has no reason to share. The missed saving is 0.0295/s, 33% of the selected plan.

With raw data not kept (raw_data_retained: false), the two queries share one ingestion-time SummaryTumblingWindowChain, but each still runs its own TimeRange(120s) → SummaryMerge → FinalizeExactAccumulator → Aggregate(sum) over it: 1.8965/s, against 1.5560/s for the first query alone.

Expected: the second query reads the first query's result at the times both evaluate, and the plan costs about 0.0590/s (raw data kept) or 1.556/s (raw data not kept). This is the behavior #925 fixed before Stage 3 was rebuilt over execution contexts.

Why it happens

An execution context is one evaluation grid:

  • QueryWorkloadEntry::evaluation_grid_ms (crates/types/src/workload/query.rs) is the greatest common divisor of a query's interval and phase: 60,000 ms for the first query and 120,000 ms for the second.
  • queries_of_context (crates/types/src/stages/physical/execution_context.rs) puts only the queries with exactly that grid into one context.
  • computations (crates/cost/src/plan/execution_demand/mod.rs) runs a node once per grid among the queries that reach it and adds the grids' rates. Stage 3 builds its contexts $\kappa$ the same way (SelectableExecutions, executions_per_second).

So the 60 s and 120 s grids are two contexts even though the 120 s grid's instants are a subset of the 60 s grid's. A node read by both runs in both contexts, and nothing in the model says that the runs at the even minutes coincide.

The concepts doc defines $\kappa$ as "a class of executions that happen at the same times". Grids that nest (one grid's instants all lie on another) break the "one grid, one context" shortcut for that definition.

Proposal

Define a context by the set of queries that evaluate at an instant, not by one grid:

  • For the grids of the queries reaching a node, partition their instants by which queries evaluate there. In the example: instants at odd minutes are evaluated by the first query only (0.0083/s), and instants at even minutes by both (0.0083/s). A node read by both runs 0.0167/s in total.
  • The rate of each class follows from the grids' least common multiples (inclusion-exclusion, as distinct_instants_per_request already does for range steps).
  • Stage 3 indexes $x_{\kappa,v}$ and $y_{\kappa,u,c,i}$ by these classes, and computations rates nodes the same way, so the MILP objective still equals plan_cost.

Alternative: keep one context per grid, but let a context whose grid is a multiple of another (same phase) read the finer context's execution at the shared instants. This adds edges between contexts and needs the same rate split, so it is no simpler.

This changes how Stage 3 and asap-cost define $\kappa$, so it needs a design decision before code.

Scope and acceptance

  • In scope: fixed-interval and scheduled queries whose instants coincide on nested grids, with raw data kept or not.
  • Out of scope: estimated-rate and one-off queries (no known grid), and @ anchors (their own request contexts).
  • Acceptance: plan_sharing::slower_repeat_reuses_faster_sum passes (both queries cost at most 0.1% more than the first alone), and a unit test of computations rates the witness's nodes at 1/60 runs per second.

Status and related work

Activity

  1. added
    bugSomething isn't working
    cost-modelCost formulas, statistics, ranking, and selection
    optimizerWorkload optimization and plan selection
    workloadBatch, repeating, or multi-query workload modeling
    on Nov 16, 2026
  2. zzylol commented on Nov 16, 2026

    @zzylol
    ContributorAuthor

    The stale-test PR #1389 leaves plan_sharing::slower_repeat_reuses_faster_sum failing; this issue is the fix it needs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CSEbugSomething isn't workingcost-modelCost formulas, statistics, ranking, and selectionoptimizerWorkload optimization and plan selectionworkloadBatch, repeating, or multi-query workload modeling

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions