Repository navigation
Closed
Labels
bugSomething isn't workingSomething isn't workingpromqlPromQL front-end lowering (L1→L2)PromQL front-end lowering (L1→L2)
Description
Problem, by example
The data is 1,000 series of a gauge latency sampled every 1 s, arriving at 2,000 rows per second. Each query below runs every 10 s with the built-in models and default CostInputs:
| Query | plan_installable, raw data kept |
plan_installable, raw data not kept |
|---|---|---|
deriv(latency[1h]) |
the executor cannot compile the plan's query run: invalid DAG: node 2: per-entity aggregate Deriv has no native implementation |
no valid candidate: ... Infeasible |
predict_linear(latency[1h], 60) |
... per-entity aggregate PredictLinear { seconds: 60.0 } has no native implementation |
same infeasible selection |
delta(latency[1h]) |
... per-entity aggregate Delta has no native implementation |
same infeasible selection |
idelta(latency[5m]) |
... per-entity aggregate IDelta has no native implementation |
same infeasible selection |
sum by (job) (delta(latency[1h])) |
... per-entity aggregate Delta has no native implementation |
same infeasible selection |
These come from a throwaway planner test that calls plan_installable for each query, on the head of #1355 (008704da2) and with its review fixes; the results are the same on both.
The planner plans each query as Scan → TimeRange → Aggregate { reduction: per entity, measures: [Deriv] } (and the same for the others). The executor then refuses to compile that plan. Expected: each query installs and answers as Prometheus does.
Why it happens
- The executor's per-entity
Aggregatelowering (crates/executor/src/physical_planner/mod.rs, the arm that returnsper-entity aggregate ... has no native implementation) maps onlyCount,Sum,Avg,Min,Max,Quantile,Cardinality,FrequencyL2andFrequencyEntropyto aReduction. That list isagg_has_exact_per_entity_kernel(crates/types/src/ir/operator/agg_intent.rs). - The executor already has a windowed aggregate for some of these:
Operator::window(crates/executor/src/operators/aggregate/mod.rs) acceptsRate,Deriv,PredictLinearandIncreaseover a(start, end)window. The physical planner never builds it, because the per-entityAggregateover aTimeRangehas no window bounds at compile time: they depend on each evaluation's time. rateandincreaseinstall today only because Stage 2 offers an exact Rate / Increase accumulator (SummaryAgg+FinalizeExactAccumulator) for them.deriv,predict_linear,deltaandideltahave no accumulator, so the per-entityAggregateis their only plan.agg_has_exact_per_entity_kernel's doc says "Stage 3 rejects a plan holding any other exact per-entity aggregate, so both read this one list". Only the executor reads it; Stage 3 selects these plans and installation refuses them, as the table shows.
Proposal
- Executor: lower a per-entity
Aggregatewhose measure isRate,Increase,Deriv,PredictLinear,DeltaorIDeltato a kernel that reads each series' samples in time order over the evaluation's window (theTimeRange's bounds at run time), reusingOperator::window's computation for the first four. AddDeltaandIDelta(delta: last minus first, extrapolated as Prometheus does;idelta: the last two samples). - Fix the stale claim in
agg_has_exact_per_entity_kernel's doc: say that only the executor reads it, or add the Stage 3 check it describes. The planner should not be restricted by the executor's current list, so the preferred fix is the executor kernels above plus a doc that says what the list is. irate,resetsandchangesare Executor: irate, resets and changes have no native implementation, so the planner plans them but none installs #1345.
Scope and acceptance
- In scope: per-series
rate,increase,deriv,predict_linear,delta,ideltaover a range, as a per-entityAggregateat query time over raw data. - Out of scope: tumbling-window chain states for
deriv/predict_linear/delta(a separate accumulator design). - Acceptance: a planner test plans each query above with raw data kept, installs it and answers as Prometheus does on a small series. It fails today.
Related
- Executor: irate, resets and changes have no native implementation, so the planner plans them but none installs #1345:
irate,resetsandchangeshave no native implementation. - Plans whose operators have no executor kernel (KMV, IRate) count as valid in Stage 2/3 #701: Stage 2/3 count plans whose operators have no kernel as valid.
- fix(cost): charge each operation the executor's measured time, so a sketch plan is no longer predicted cheaper than a faster exact plan #1355 charges
rate,increase,derivandpredict_linearmeasures in a hash aggregate as much as the Rate accumulator per row: an estimate of their work, not a way to steer selection away from them.
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingpromqlPromQL front-end lowering (L1→L2)PromQL front-end lowering (L1→L2)
Activity