Skip to content

Showcase milestones: every workload at least as fast as exact, with measured costs #1329

Open

Description

@zzylol

Problem, by example

quantile_over_time(0.99, latency[1h]) over 1,000 series sampled at 1 Hz, evaluated every 10 s, with a 1% relative target (δ = 0.01). On main (59365ce80) the planner selects a DDSketch built from raw data at query time. Measured on the executor, that plan runs at 0.50× the speed of the exact plan: it is twice as slow as the plan it was supposed to beat. The exact plan was a candidate, so Stage 3 ranked the two wrongly: its cost estimate is wrong.

Today's baseline (main 59365ce80, 1h window, 1,000 series, measured on one core of an idle 56-core server; speedup = exact p50 latency / selected-plan p50 latency):

Query Setting Speedup
quantile 0.99 1 Hz, every 10 s 0.50×
joint workload of 5 1 Hz, every 10 s 0.91×
sum_over_time 1 Hz, every 10 s 3.5×
sum rate / sum_over_time 10 Hz, every 60 s 128–145×

Goal: the showcase rule

For every workload in a fixed suite, the selected plan must be:

  • never slower than exact: measured cost ≤ the exact plan's measured cost × 1.05, and
  • within its target: measured error ≤ the workload's accuracy target.

The exact plan is always a candidate, so a violation of the first condition means the cost estimate is wrong, not that no good plan exists.

The check is asap_run bench --suite showcase (#1317).

The suite

Group Workloads
Easy sum(rate(…)), sum_over_time, count_over_time
Hard quantile_over_time, avg(quantile_over_time(…)), max_over_time
Mixed the joint workload of 5; S1 dashboard subsets (W-RangeHeavy, W-RollupHeavy)
Data settings (each workload) 1 Hz every 10 s; 10 Hz every 60 s; raw data retained and not retained
Counterexample a 1m window (stored state cannot pay off; the rule still holds, by choosing exact)

Milestones

Milestone Work Value shown Pass criteria
M0 Baseline with asap_run (#1317, after #1300) The suite runs end to end from one command; today's numbers recorded Answers correct (exact arms match; approximate arms reported); numbers recorded
M1 Accurate costs: measured cost table (#1317 calibrate), #1313, #1314 The planner never picks a plan slower than exact Never worse than exact (rule holds on the whole suite); exact-vs-selected ranking ≥ 90% (36/85 today); predicted / measured within 1.5× per plan kind
M2 #1312 merge speed Stored-window plans get faster where merges dominate M1 still holds; sum-family speedups at 1 Hz improve
M3 #1315 nested windows 100× at the common setting, not only at 10 Hz Sum family ≥ 100× at 1 Hz every 10 s
M4 Measured accuracy (#1317 MeasuredSketchAccuracy) Smaller sketches at the same target Measured error ≤ target, with memory below M3's
M5 #1316 latency objective A declared latency bound is met Measured latency ≤ the declared bound
#1207 done The formulation tracking issue completes The whole evaluation suite plans and runs All S1 workloads plan and run, answers correct, never worse than exact

Process

  • Every PR runs a small showcase (100 series, 10 minutes of data) as a 7th review check, after asap_run exists, and puts the result table in the PR body. A PR that breaks the rule on the small suite says why.
  • Each milestone runs the suite at full scale. Results go under calibration/runs/<machine-key>/<run-id>/, and docs/showcase.md keeps one table row per milestone (speedup, error, memory per workload).

Related

#1312, #1313, #1314, #1315, #1316, #1317, #1324, #1300, #1207.

Activity

  1. added
    cost-modelCost formulas, statistics, ranking, and selection
    testingCorpus, regression, differential, or benchmark tests
    on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

cost-modelCost formulas, statistics, ranking, and selectiontestingCorpus, regression, differential, or benchmark teststracking issue

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions