Repository navigation
Description
Problem, by example
quantile_over_time(0.99, latency[1h]) over 1,000 series sampled at 1 Hz, evaluated every 10 s, with a 1% relative target (δ = 0.01). On main (59365ce80) the planner selects a DDSketch built from raw data at query time. Measured on the executor, that plan runs at 0.50× the speed of the exact plan: it is twice as slow as the plan it was supposed to beat. The exact plan was a candidate, so Stage 3 ranked the two wrongly: its cost estimate is wrong.
Today's baseline (main 59365ce80, 1h window, 1,000 series, measured on one core of an idle 56-core server; speedup = exact p50 latency / selected-plan p50 latency):
| Query | Setting | Speedup |
|---|---|---|
| quantile 0.99 | 1 Hz, every 10 s | 0.50× |
| joint workload of 5 | 1 Hz, every 10 s | 0.91× |
| sum_over_time | 1 Hz, every 10 s | 3.5× |
| sum rate / sum_over_time | 10 Hz, every 60 s | 128–145× |
- The exact-vs-selected ranking is right in 36 of 85 default cases (58 of 85 with no raw data retained).
- Measured CPU and latency are 13–960× the built-in unit costs ([Design] Price CPU and latency from a measured cost table per state family (scan, update, merge), not built-in unit costs #1314), and the error differs by plan kind, so rankings flip.
Goal: the showcase rule
For every workload in a fixed suite, the selected plan must be:
- never slower than exact: measured cost ≤ the exact plan's measured cost × 1.05, and
- within its target: measured error ≤ the workload's accuracy target.
The exact plan is always a candidate, so a violation of the first condition means the cost estimate is wrong, not that no good plan exists.
The check is asap_run bench --suite showcase (#1317).
The suite
| Group | Workloads |
|---|---|
| Easy | sum(rate(…)), sum_over_time, count_over_time |
| Hard | quantile_over_time, avg(quantile_over_time(…)), max_over_time |
| Mixed | the joint workload of 5; S1 dashboard subsets (W-RangeHeavy, W-RollupHeavy) |
| Data settings (each workload) | 1 Hz every 10 s; 10 Hz every 60 s; raw data retained and not retained |
| Counterexample | a 1m window (stored state cannot pay off; the rule still holds, by choosing exact) |
Milestones
| Milestone | Work | Value shown | Pass criteria |
|---|---|---|---|
| M0 | Baseline with asap_run (#1317, after #1300) |
The suite runs end to end from one command; today's numbers recorded | Answers correct (exact arms match; approximate arms reported); numbers recorded |
| M1 | Accurate costs: measured cost table (#1317 calibrate), #1313, #1314 |
The planner never picks a plan slower than exact | Never worse than exact (rule holds on the whole suite); exact-vs-selected ranking ≥ 90% (36/85 today); predicted / measured within 1.5× per plan kind |
| M2 | #1312 merge speed | Stored-window plans get faster where merges dominate | M1 still holds; sum-family speedups at 1 Hz improve |
| M3 | #1315 nested windows | 100× at the common setting, not only at 10 Hz | Sum family ≥ 100× at 1 Hz every 10 s |
| M4 | Measured accuracy (#1317 MeasuredSketchAccuracy) |
Smaller sketches at the same target | Measured error ≤ target, with memory below M3's |
| M5 | #1316 latency objective | A declared latency bound is met | Measured latency ≤ the declared bound |
| #1207 done | The formulation tracking issue completes | The whole evaluation suite plans and runs | All S1 workloads plan and run, answers correct, never worse than exact |
Process
- Every PR runs a small showcase (100 series, 10 minutes of data) as a 7th review check, after
asap_runexists, and puts the result table in the PR body. A PR that breaks the rule on the small suite says why. - Each milestone runs the suite at full scale. Results go under
calibration/runs/<machine-key>/<run-id>/, anddocs/showcase.mdkeeps one table row per milestone (speedup, error, memory per workload).
Related
#1312, #1313, #1314, #1315, #1316, #1317, #1324, #1300, #1207.
Activity