Skip to content

Rollup across different windows through a shared chain is not offered (#1239) #1282

Open

Description

@milindsrivastava1997

On #1239, a coarser aggregate over a shorter window is not served from a finer aggregate's tumbling-window chain. Main offers it.

Reproduce

r5.json (SQL, since PromQL has no cross-series quantile over a window):

{"query_workload": {"language": {"sql": "datafusion_sql"}, "repeating_queries": [{"query": "SELECT job, approx_percentile_cont(value, 0.9) AS value FROM data WHERE ts >= now() - INTERVAL '5 minutes' GROUP BY job", "demand": {"fixed_interval_at": {"interval": 60000, "evaluation_phase": 0}}, "requirements": {"accuracy": {"max_error": 0.01, "failure_probability": 0.001}}, "predictability": {"predictable": {"known_at": null}}}, {"query": "SELECT job, host, approx_percentile_cont(value, 0.9) AS value FROM data WHERE ts >= now() - INTERVAL '10 minutes' GROUP BY job, host", "demand": {"fixed_interval_at": {"interval": 60000, "evaluation_phase": 0}}, "requirements": {"accuracy": {"max_error": 0.01, "failure_probability": 0.001}}, "predictability": {"predictable": {"known_at": null}}}]}, "data_workload": {"arrival": {"ingesting": {"rows_per_second": 666.6666666666666}}, "sources": {"data": {"series_count": 10000, "metric_type": "gauge", "sample_interval": 15000, "distinct_values_per_label": {"job": 10, "host": 1000}}}}}
asap_planner --workload r5.json --run-until stage1_pass2 \
  --sql-catalog-table '{"name":"data","columns":[{"name":"ts","type":"timestamp"},{"name":"job","type":"utf8"},{"name":"host","type":"utf8"},{"name":"value","type":"float64"}],"time_index":0}'

(For main, replace each fixed_interval_at with "fixed_interval": 60000.)

q1: approx_percentile_cont(value, 0.9) ... [5m] GROUP BY job. q2: the same over [10m] GROUP BY job, host.

Expected

A candidate for q1's state that reads q2's (job, host) chain: merge the five 60 s windows, then merge by job.

Observed

main 3608c22a2 #1239 9e0f50179
q1's state (Aggregate quantile by job, [5m]) 120 Rollup candidates: SummaryMerge(by job) ← SummaryMerge(5 windows) ← SummaryTumblingWindowChain Kll of the (job, host) state none
Chains 240 (every divisor width) G15 (by job, 60 s) and G16 (by job, host, 60 s); neither reads the other

Stage 3 picks no chain on either (ingest-time memory cost), so today's plans match (~0.0593/s). The candidate matters once a chain is selected.

Cause

rules/rollup.rs clusters aggregates by equal definition, which includes the input TimeRange, so [5m] and [10m] are never in one cluster. rules/chains.rs then builds one chain per state. No rule composes the two: a coarse state over window W reading a finer state's chain over a different window.

Control: the same queries with both windows [5m] (same data) roll up on #1239, and Stage 3 picks it (0.0267/s vs main's 0.0299/s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions