Repository navigation
Description
Problem, by example
The workload crates/integration-tests/tests/plan_sharing_fixtures/percentile_filtered_and_by_city.json, over the table metrics(ts, city, host, value) with 100 series, every 10 s, 1% target:
-- A: the filtered quantile
SELECT percentile_cont(0.9) WITHIN GROUP (ORDER BY value) FROM metrics
WHERE ts > now() - INTERVAL '5' MINUTE AND city = 'NYC'
-- B: the same quantile by city
SELECT city, percentile_cont(0.9) WITHIN GROUP (ORDER BY value) FROM metrics
WHERE ts > now() - INTERVAL '5' MINUTE GROUP BY cityUnder the default CostInputs (with the measured unit costs of #1355), both queries are planned exactly (scan, keep each group's values, sort). B alone costs 2.1164 per second; A and B together cost 2.2742, 7.5% more: A scans and sorts NYC's rows again, although B's output already holds NYC's exact 0.9 quantile.
Expected: A reads B's row for city = 'NYC' (a filter on B's output, dropping the city column), and the two cost about what B costs alone. With summaries' CPU costed free (summary_cpu_free), the sketch plans are selected, A reads B's states through a compensating filter, and together they cost 2.1167 against B's 2.1164. Numbers from plan_sharing::grouped_filtered_quantile_plans (cost= lines with --nocapture).
Why it happens
- Pass 2's rollup rule (
rollup_fromincrates/stage1-pass2-reuse-candidates/src/rules/rollup/rewrite.rs) does find B as A's finer group, with the compensating filtercity = 'NYC'on B's key. It always buildsRollup(by A's keys)(Filter(compensation, B)). - For an exact
Aggregate, the rollup re-aggregates each measure withrolled_up(same file), which has a rollup only forSum,Count,MinandMax. A quantile has none, so the rule returns without a candidate. - But here no re-aggregation is needed: the compensating filter pins the one key A drops (
city) to a single value, so at most one B row survives and that row is A's answer. View matching calls this the same-level case (the dropped group columns are all fixed by constant conditions): the substitute is a filter and projection over the view, with no aggregate, and it works for holistic measures such as a quantile. - For summary states the same rule already works (A merges B's NYC state), which is why the sketch plans share.
Proposal
- In
rollup_from, when every key A drops is pinned by an equality in the compensation, offerProject(Filter(compensation, B))(A's columns, under A's names) instead of the rollup, for any measure. It is exact, so it carries no accuracy cost. - Keep the SQL semantics of a global aggregate over no row: if no B row matches, A returns one NULL row (a
MIN-style wrapper, as the earlier filtered-group rule did), which the memo's schema check should cover.
Scope and acceptance
- In scope: equality compensations that pin every dropped key, over an exact grouped
Aggregate, SQL and PromQL. Out of scope:INlists or ranges (they still need a rollup, which a quantile does not have). - Acceptance:
plan_sharing::grouped_filtered_quantile_plans' last assertion (A and B together cost more than B alone under the default costs) fails because A reuses B; change it to expecttogether ≤ B × 1.001.
Related
- fix(cost): charge each operation the executor's measured time, so a sketch plan is no longer predicted cheaper than a faster exact plan #1355: its test pins today's no-reuse under the default costs.
- Reuse a finer aggregate through a compensating filter and a rollup of values or states (view matching) #950 (closed): the view-matching rollup rule this extends; its case A is this example's shape.
- Share one scan of a source across readers with different predicates, labels and time ranges (subsumption derivations) #951: shared scans across readers with different predicates.
Activity
fix(cost): per-family compaction merges, one CPU-operation unit, revi…
fix(cost): per-family compaction merges, one CPU-operation unit, revi…
fix(cost): per-family compaction merges, one CPU-operation unit, revi…
fix(cost): per-family compaction merges, one CPU-operation unit, revi…
fix(cost): charge each operation the executor's measured time, so a s…