Repository navigation
Description
On #1239, a coarser aggregate over a shorter window is not served from a finer aggregate's tumbling-window chain. Main offers it.
Reproduce
r5.json (SQL, since PromQL has no cross-series quantile over a window):
{"query_workload": {"language": {"sql": "datafusion_sql"}, "repeating_queries": [{"query": "SELECT job, approx_percentile_cont(value, 0.9) AS value FROM data WHERE ts >= now() - INTERVAL '5 minutes' GROUP BY job", "demand": {"fixed_interval_at": {"interval": 60000, "evaluation_phase": 0}}, "requirements": {"accuracy": {"max_error": 0.01, "failure_probability": 0.001}}, "predictability": {"predictable": {"known_at": null}}}, {"query": "SELECT job, host, approx_percentile_cont(value, 0.9) AS value FROM data WHERE ts >= now() - INTERVAL '10 minutes' GROUP BY job, host", "demand": {"fixed_interval_at": {"interval": 60000, "evaluation_phase": 0}}, "requirements": {"accuracy": {"max_error": 0.01, "failure_probability": 0.001}}, "predictability": {"predictable": {"known_at": null}}}]}, "data_workload": {"arrival": {"ingesting": {"rows_per_second": 666.6666666666666}}, "sources": {"data": {"series_count": 10000, "metric_type": "gauge", "sample_interval": 15000, "distinct_values_per_label": {"job": 10, "host": 1000}}}}}asap_planner --workload r5.json --run-until stage1_pass2 \
--sql-catalog-table '{"name":"data","columns":[{"name":"ts","type":"timestamp"},{"name":"job","type":"utf8"},{"name":"host","type":"utf8"},{"name":"value","type":"float64"}],"time_index":0}'(For main, replace each fixed_interval_at with "fixed_interval": 60000.)
q1: approx_percentile_cont(value, 0.9) ... [5m] GROUP BY job. q2: the same over [10m] GROUP BY job, host.
Expected
A candidate for q1's state that reads q2's (job, host) chain: merge the five 60 s windows, then merge by job.
Observed
main 3608c22a2 |
#1239 9e0f50179 |
|
|---|---|---|
q1's state (Aggregate quantile by job, [5m]) |
120 Rollup candidates: SummaryMerge(by job) ← SummaryMerge(5 windows) ← SummaryTumblingWindowChain Kll of the (job, host) state |
none |
| Chains | 240 (every divisor width) | G15 (by job, 60 s) and G16 (by job, host, 60 s); neither reads the other |
Stage 3 picks no chain on either (ingest-time memory cost), so today's plans match (~0.0593/s). The candidate matters once a chain is selected.
Cause
rules/rollup.rs clusters aggregates by equal definition, which includes the input TimeRange, so [5m] and [10m] are never in one cluster. rules/chains.rs then builds one chain per state. No rule composes the two: a coarse state over window W reading a finer state's chain over a different window.
Control: the same queries with both windows [5m] (same data) roll up on #1239, and Stage 3 picks it (0.0267/s vs main's 0.0299/s).
Activity