Repository navigation
Description
Problem, by example
The same exact query asked every 60 s and every 120 s (both at phase 0) is planned as two independent pipelines, although every evaluation of the 120 s copy happens at the same time as an evaluation of the 60 s copy and could read its result.
Workload: crates/integration-tests/tests/plan_sharing_fixtures/slower_repeat_exact_sum.json:
sum (sum_over_time(m_gauge[2m])),fixed_interval_at60,000 ms, phase 0, exact;- the same query,
fixed_interval_at120,000 ms, phase 0, exact; m_gauge: 10,000 gauge series, one sample every 15 s (1,333 rows/s), raw data kept.
Measured on main ca4be93f0 with default CostInputs (plan_sharing::slower_repeat_reuses_faster_sum and a local dump of the selected plan and its asap_cost::plan_cost):
| Plan | Nodes | Cost per second |
|---|---|---|
| First query alone | Scan → TimeRange(2m) → SummaryAgg(exact Sum) → FinalizeExactAccumulator, every 60 s |
0.05901 |
| Both queries, selected | the pipeline above every 60 s, plus a second copy of all four nodes every 120 s | 0.08851 |
| Both queries, witness | the first query's four nodes only; the second query's root is the first query's FinalizeExactAccumulator |
0.08851 as priced today; 0.05901 if one run at each even minute serves both queries |
The witness DAG validates (PhysicalASAPDAG::validate). Today plan_cost rates its nodes at 0.0250 runs/s (1/60 + 1/120) instead of 0.0167 runs/s (1/60), so it is priced the same as two copies, and Stage 3, which uses the same rule, has no reason to share. The missed saving is 0.0295/s, 33% of the selected plan.
With raw data not kept (raw_data_retained: false), the two queries share one ingestion-time SummaryTumblingWindowChain, but each still runs its own TimeRange(120s) → SummaryMerge → FinalizeExactAccumulator → Aggregate(sum) over it: 1.8965/s, against 1.5560/s for the first query alone.
Expected: the second query reads the first query's result at the times both evaluate, and the plan costs about 0.0590/s (raw data kept) or 1.556/s (raw data not kept). This is the behavior #925 fixed before Stage 3 was rebuilt over execution contexts.
Why it happens
An execution context is one evaluation grid:
-
QueryWorkloadEntry::evaluation_grid_ms(crates/types/src/workload/query.rs) is the greatest common divisor of a query's interval and phase: 60,000 ms for the first query and 120,000 ms for the second. -
queries_of_context(crates/types/src/stages/physical/execution_context.rs) puts only the queries with exactly that grid into one context. -
computations(crates/cost/src/plan/execution_demand/mod.rs) runs a node once per grid among the queries that reach it and adds the grids' rates. Stage 3 builds its contexts$\kappa$ the same way (SelectableExecutions,executions_per_second).
So the 60 s and 120 s grids are two contexts even though the 120 s grid's instants are a subset of the 60 s grid's. A node read by both runs in both contexts, and nothing in the model says that the runs at the even minutes coincide.
The concepts doc defines
Proposal
Define a context by the set of queries that evaluate at an instant, not by one grid:
- For the grids of the queries reaching a node, partition their instants by which queries evaluate there. In the example: instants at odd minutes are evaluated by the first query only (0.0083/s), and instants at even minutes by both (0.0083/s). A node read by both runs 0.0167/s in total.
- The rate of each class follows from the grids' least common multiples (inclusion-exclusion, as
distinct_instants_per_requestalready does for range steps). - Stage 3 indexes
$x_{\kappa,v}$ and$y_{\kappa,u,c,i}$ by these classes, andcomputationsrates nodes the same way, so the MILP objective still equalsplan_cost.
Alternative: keep one context per grid, but let a context whose grid is a multiple of another (same phase) read the finer context's execution at the shared instants. This adds edges between contexts and needs the same rate split, so it is no simpler.
This changes how Stage 3 and asap-cost define
Scope and acceptance
- In scope: fixed-interval and scheduled queries whose instants coincide on nested grids, with raw data kept or not.
- Out of scope: estimated-rate and one-off queries (no known grid), and
@anchors (their own request contexts). - Acceptance:
plan_sharing::slower_repeat_reuses_faster_sumpasses (both queries cost at most 0.1% more than the first alone), and a unit test ofcomputationsrates the witness's nodes at 1/60 runs per second.
Status and related work
- Regression of [Bug] The same query at a longer repeat interval recomputes instead of reusing the shorter-interval result #925 (closed 2026-10-27), which fixed the same example in the earlier search.
- The test fails on every commit from refactor(stage3): baseline selection MILP and its PhysicalASAPDAG; executor θ, native chains and a time-indexed store (#1207 PR 6) #1236 (the first Stage 3 of the Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 refactor that selects a plan) to main; under the earlier unit costs it selected two ingestion-time pipelines, 0.0124/s against 0.0083/s for the first query alone, also with no sharing.
- Execution contexts come from the Stage 3 refactor (Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207, Stage 3 MILP: implement the formulation variables x, y, z, t and rows (A)–(I) as named code over explicit edges #1204). This query-time plan is selected since the measured unit costs (fix(cost): charge each operation the executor's measured time, so a sketch plan is no longer predicted cheaper than a faster exact plan #1355, fix(cost): charge sketch states and merges, raw batch visits, grouping keys, distinct counts and KMV at their measured costs, so the measured-best plan is selected #1363), under which the first query's query-time pipeline (0.0590/s) is about 26 times cheaper than its ingestion-time chain (1.556/s).
- Stage 3 overprices unselected plans with split twin readers or a shared OneEvaluation scan #1335 covers other mismatches in how Stage 3 rates executions within one context (twin readers, scan windows); this issue is about executions across contexts.
Activity
zzylol commentedon Nov 16, 2026
The stale-test PR #1389 leaves
plan_sharing::slower_repeat_reuses_faster_sumfailing; this issue is the fix it needs.