Repository navigation
Description
Problem
Heavy-hitter promotion in canonicalize is skipped whenever a parent reads the ranked node by column position. For example, with one repeating PromQL query:
| Query | Canonical shape today | Summary candidates for the ranking |
|---|---|---|
topk(3, sum by (job) (m)) |
Aggregate { [TopK { k: 3 }] } over Aggregate { [Sum] } |
heavy-hitter summaries (TopK is a summary-capable intent) |
sort_desc(topk(3, sum by (job) (m))) |
Sort { Limit { Sort { Aggregate { [Sum] } } } }: the inner Limit { Sort } stays |
none: only the exact sort then limit |
topk(3, sum by (job) (m)) > 5 |
Filter { Limit { Sort { Aggregate { [Sum] } } } } |
none |
These shapes are the lowered trees lower_promql_query_workload returns today (checked on the without grouping PR's head). The same holds under any operator that reads the top-k's columns by position, such as arithmetic (topk(3, …) * 2) or an outer aggregate.
Why
canonicalize (crates/frontend-common/src/canonicalize.rs) computes read_by_position, the nodes some consumer reads by column position, and promotes a Limit { Sort { Aggregate } } only for a node outside that set (columns_may_change). Promotion changes the node's columns, so a parent bound to the old positions would read another column. Refusing is correct but loses the rewrite.
Series identity realization (with_promql_series_identity, crates/types/src/ir/schema/series_identity.rs) solves the same problem the other way: when it inserts the identity column into a node, it renumbers every parent's column references with NonASAPOp::map_input_columns (and an Aggregate's having by the aggregate's output), and fails only when a parent reads a column the rewrite drops.
Proposal
Let promotion renumber parents instead of refusing:
- For a node in
read_by_position, compute where each column of the old output sits in the promoted output, matched by name, aspositions_after_realizationdoes for the identity column. - If every column a parent reads is kept, rewrite the node and renumber each parent's references with
map_input_columns. - Refuse only when a parent reads a column the promotion does not keep.
The two rewrites would then share one rule: a rewrite that moves a node's columns renumbers its readers. A shared helper (positions by name, then map_input_columns) could serve both.
Scope and acceptance
- In scope: heavy-hitter promotion under positional readers.
- Out of scope: other canonicalization rules, which do not change columns.
- Done when
sort_desc(topk(3, sum by (job) (m)))canonicalizes its inner ranking toAggregate { [TopK { k: 3 }] }, a test checks the parent's sort key reads the ranked value after promotion, and the exact answers over a sample input stay equal to today's.
Related
- Found by the round-5 abstraction review of feat(executor): grouped stddev/variance, corr, group, count_values, exact topk and without grouping #1242, which added the renumbering to series identity realization.
Activity