Skip to content

PromQL: a grouping over a label an inner without removed fails (nested without, double avg/stddev without, by over an absent label) #1349

Open

Description

@zzylol

Problem

A PromQL grouping that names a label an inner without already removed is legal. Prometheus treats the label as absent: a no-op for without, the empty label for by. Three shapes of this still fail. All of them fail loudly; none gives a wrong answer.

These were run through e2e_plan and plan_installable on one metric m (10 series), with the exact target and the 5 % target. Both targets give the same result.

Query What should happen What happens today
sum by (instance) (sum without (job) (sum without (instance) (m))) Same as sum by (instance) (sum without (job, instance) (m)): one series {} holding the total Lowering fails: column resolution failed: column `instance` not found in schema (have: ["sum"])
avg without (instance) (avg without (instance) (m)) Same as avg without (instance) (m), because the outer without (instance) is a no-op Plan selection fails: no valid candidate: … no plan gives every query output a candidate whose every input has an admissible producer (the selection MILP is infeasible); candidates left out because they could not be costed: inconsistent operator statistics: ungrouped HashAggregate must have one zero-key-width group
stddev without (instance) (stddev without (instance) (m)) The outer without (instance) is a no-op, so each inner series is its own group and every value is 0 Same plan-selection error as avg

For comparison, sum without (instance) (sum without (instance) (m)) plans and installs today.

sum by (instance) (sum without (instance) (m)) (one without level) does plan. The plan then fails to install with invalid DAG: node 4: invalid DAG: native output type differs from Planner output. This is the same absent-label rule on the by side, so the issue covers it too.

Why it happens

  1. Only one without level is checked.
    • labels_removed_by_without in crates/frontend-common/src/resolve.rs returns the keys of the without aggregate directly below.
    • It sees through Filter and through a Project that keeps every label.
    • It does not look further down. For the outer sum by (instance), the aggregate below is sum without (job), so the rule finds only job. The instance removed one level lower is missed, and resolving instance fails.
  2. A without with no keys left is costed as a global aggregate.
    • The outer avg without (instance) resolves to GroupKeys::without([]), because instance is absent.
    • without([]) groups by every remaining label.
    • The analytical cost check in crates/cost/src/node/analytical.rs (grouping_key_count == 0) treats it as ungrouped and requires group_count == 1. The statistics have many groups, so every candidate is left out and the MILP is infeasible.
    • sum takes a different path and is not affected. avg and stddev reach the generic HashAggregate costing.
  3. by over an absent label builds a different output type in the planner and the executor.
    • The planner resolves the absent by key to no column.
    • The executor's native output for that node does not match the planner's schema.
    • This was not traced further.

Proposal

  1. Make labels_removed_by_without collect labels through nested without aggregates. A without aggregate removes its own keys plus whatever its input had already removed. Stop at any operator that can re-add a label, such as label_replace, a binary operator, or count_values.
  2. Treat a without reduction as grouped in cost statistics even when it has zero keys. Better still, have the frontend lower without([]) over a series identity as a per-series pass-through: an aggregate whose groups are the input series.
  3. For by over an absent label: either emit the empty label column the planner schema declares, or drop the key from the by list so both sides agree on by ().

Scope and acceptance

  • In scope: the three shapes above, and the one-level by case.
  • Out of scope: labels re-added by label_replace or a binary operator's side.
  • Done when planner end-to-end tests (crates/planner/tests, reduce_aggregates) for each query in the table plan, install, and match the reference evaluator's answer for the equivalent single-without query.

Related

Activity

  1. added
    bugSomething isn't working
    promqlPromQL front-end lowering (L1→L2)
    on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpromqlPromQL front-end lowering (L1→L2)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions