Repository navigation
Description
Problem
A PromQL grouping that names a label an inner without already removed is legal. Prometheus treats the label as absent: a no-op for without, the empty label for by. Three shapes of this still fail. All of them fail loudly; none gives a wrong answer.
These were run through e2e_plan and plan_installable on one metric m (10 series), with the exact target and the 5 % target. Both targets give the same result.
| Query | What should happen | What happens today |
|---|---|---|
sum by (instance) (sum without (job) (sum without (instance) (m))) |
Same as sum by (instance) (sum without (job, instance) (m)): one series {} holding the total |
Lowering fails: column resolution failed: column `instance` not found in schema (have: ["sum"]) |
avg without (instance) (avg without (instance) (m)) |
Same as avg without (instance) (m), because the outer without (instance) is a no-op |
Plan selection fails: no valid candidate: … no plan gives every query output a candidate whose every input has an admissible producer (the selection MILP is infeasible); candidates left out because they could not be costed: inconsistent operator statistics: ungrouped HashAggregate must have one zero-key-width group |
stddev without (instance) (stddev without (instance) (m)) |
The outer without (instance) is a no-op, so each inner series is its own group and every value is 0 |
Same plan-selection error as avg |
For comparison, sum without (instance) (sum without (instance) (m)) plans and installs today.
sum by (instance) (sum without (instance) (m)) (one without level) does plan. The plan then fails to install with invalid DAG: node 4: invalid DAG: native output type differs from Planner output. This is the same absent-label rule on the by side, so the issue covers it too.
Why it happens
- Only one
withoutlevel is checked.labels_removed_by_withoutincrates/frontend-common/src/resolve.rsreturns the keys of thewithoutaggregate directly below.- It sees through
Filterand through aProjectthat keeps every label. - It does not look further down. For the outer
sum by (instance), the aggregate below issum without (job), so the rule finds onlyjob. Theinstanceremoved one level lower is missed, and resolvinginstancefails.
- A
withoutwith no keys left is costed as a global aggregate.- The outer
avg without (instance)resolves toGroupKeys::without([]), becauseinstanceis absent. without([])groups by every remaining label.- The analytical cost check in
crates/cost/src/node/analytical.rs(grouping_key_count == 0) treats it as ungrouped and requiresgroup_count == 1. The statistics have many groups, so every candidate is left out and the MILP is infeasible. sumtakes a different path and is not affected.avgandstddevreach the genericHashAggregatecosting.
- The outer
byover an absent label builds a different output type in the planner and the executor.- The planner resolves the absent
bykey to no column. - The executor's native output for that node does not match the planner's schema.
- This was not traced further.
- The planner resolves the absent
Proposal
- Make
labels_removed_by_withoutcollect labels through nestedwithoutaggregates. Awithoutaggregate removes its own keys plus whatever its input had already removed. Stop at any operator that can re-add a label, such aslabel_replace, a binary operator, orcount_values. - Treat a
withoutreduction as grouped in cost statistics even when it has zero keys. Better still, have the frontend lowerwithout([])over a series identity as a per-series pass-through: an aggregate whose groups are the input series. - For
byover an absent label: either emit the empty label column the planner schema declares, or drop the key from thebylist so both sides agree onby ().
Scope and acceptance
- In scope: the three shapes above, and the one-level
bycase. - Out of scope: labels re-added by
label_replaceor a binary operator's side. - Done when planner end-to-end tests (
crates/planner/tests,reduce_aggregates) for each query in the table plan, install, and match the reference evaluator's answer for the equivalent single-withoutquery.
Related
- The absent-label rule (
labels_removed_by_without) and the double-withoutcases come from feat(executor): grouped stddev/variance, corr, group, count_values, exact topk and without grouping #1242. - PromQL aggregates grouped with without (...) get no plan: the executor rejects "grouping without requires resolved label columns" #861 is the earlier
withoutexecutor gap. - A global PromQL aggregate over another aggregate answers one row for an empty input #1348 is the separate global-aggregate-over-empty-input mismatch.
Activity