Repository navigation
Description
Example
Today, measuring whether a plan is fast, and whether the planner priced it right, takes a one-off harness. For example, to check that sum by (job) (rate(requests_total[1h])) over 1,000 series at 10 Hz, evaluated every 60 s, is about 128× faster planned than exact, someone has to:
- write a test-only binary that builds an in-memory store, generates data, installs the plan, ingests and evaluates;
- time it by hand and compare its answers with the exact plan;
- guess the planner's unit costs, because nothing fills
MeasuredOperationCosts(crates/cost/src/measured.rs). With the table empty, the built-in formulas are 13–960× too low for CPU and latency ([Design] Price CPU and latency from a measured cost table per state family (scan, update, merge), not built-in unit costs #1314); - invent the source facts (series count, sample interval, label counts, value magnitudes) by hand, although the data to compute them from is at hand.
There is also no reference showing how to drive the executor end to end. The one example outside the repo calls an older executor API.
Proposal: one devtools binary, asap_run, with four subcommands
asap_run execute --workload w.json [--data facts|file.jsonl] --at <ms>
# Reference use of the executor: plan → install → ingest (ingestion time) → evaluate at a time; print answers.
# Commented step by step; the rustdoc tutorials link to it.
asap_run calibrate [--families all]
# Microbenchmarks every operator × phase × state family (sketch params / exact kind) on the real executor,
# at two or more sizes; fits ns_per_unit and ns_per_invocation; memory_bytes_per_state from the kernel's
# own memory_bytes() after inserting data shaped by the source facts. Writes a MeasuredOperationCosts JSON,
# and a MeasuredSketchAccuracy JSON (below).
asap_run bench --workload w.json --arms exact,planned,noraw --duration 1h
asap_run bench --suite showcase [--small]
# Runs each arm end to end: query latency, ingestion and query CPU, memory; each arm's answers against exact
# (measured error); and the planner's predicted cost next to the measured one (ratio per plan kind, and
# whether exact vs selected are ranked right). `--suite showcase` runs the fixed suite of #1329 (below).
asap_run facts --data <file> --workload w.json [--trust-observed-bounds]
# Computes SourceFacts from historical or ingested data, like a database's ANALYZE, so that future
# queries are planned on the statistics of history. Writes the workload's data_workload.sources.
All four share one driver module in devtools: data generation (below), the store, install / ingest / evaluate, and per-thread CPU timing.
calibrate and bench answer different questions. calibrate asks what each part costs, and its output is an input to the planner. bench asks whether the total the planner adds up from those parts matches a real run. That end-to-end check catches errors unit costs cannot, such as an operator the cost model misses or a node charged twice.
asap_run facts: source facts from data
Example: asap_run facts --data requests.jsonl --workload w.json, where w.json holds sum by (job, instance) (rate(requests_total[5m])), reads the samples and fills sources.requests_total with, among others, series_count: 1000, sample_interval: 1000 and a distinct_values_per_label_set entry for {job, instance}.
Statistics, per source:
| Statistic | How it is computed | SourceFacts field |
|---|---|---|
| series count | distinct label sets | series_count |
| sample interval | median gap between a series' samples | sample_interval |
| distinct values per label | distinct values of each label | distinct_values_per_label |
| distinct values per label set | distinct combinations, for each by (…) set the workload's queries use |
distinct_values_per_label_set (the existing field) |
| value magnitudes | min and max absolute value per column | column_magnitudes (#1324) |
| value signs | whether every value of a column has one sign | column_signs |
| distinct values per evaluation | the most distinct values one evaluation window saw | max_distinct_per_evaluation |
| stream norms | the observed norms of each stream | stream_norms |
Two kinds of facts, written differently:
- Facts that affect only cost are written directly:
series_count,sample_interval, the label-set counts used for pricing,column_magnitudes. A wrong value makes a plan more or less expensive, never wrong. - Facts a guarantee relies on are printed as suggestions: signs, distinct bounds, norms, and the label counts used to split δ across groups. Bounds get a ×2 margin over the observed value. They are written only with
--trust-observed-boundsor after the user confirms, because an observed value is not a bound: the next hour's data may exceed it, and then the guarantee is false.
Optional sources: the VictoriaMetrics or Prometheus HTTP APIs (series and label values, or pulling a sample of the data), feeding the same statistics code. The planner library only reads SourceFacts; every adapter lives in devtools.
Data generation: reuse sketch-bench's aqpbm-datagen
asap_run generates data with sketch-bench's aqpbm-datagen, not with a generator of its own. The dependency is a git dependency pinned by rev to the commit of the open sketch-bench PR that makes aqpbm-datagen a standalone library; it does not wait for that PR to merge, and moves to a merged commit later. Only asap-devtools depends on it; the planner library and the executor do not.
Measured accuracy
calibrate also measures each sketch configuration's error on the executor's own sketches, and writes a MeasuredSketchAccuracy table. Stage 1 reads it through the existing hook AccuracyEvidence::measured_guarantee (crates/accuracy/src/evidence.rs; today only NoMeasuredGuarantees implements it).
// crates/accuracy/src/measured.rs
/// Errors measured on the executor's own sketches, by `asap_run calibrate`.
/// Default: empty, every sketch keeps its closed-form guarantee.
pub struct MeasuredSketchAccuracy { pub entries: Vec<MeasuredAccuracyEntry> }
/// One sketch configuration, one readout.
pub struct MeasuredAccuracyEntry {
/// The exact parameters measured; only these match.
pub sketch: SketchParams,
/// The readout, e.g. Quantile { q: 0.99 }; only the same statistic matches.
pub statistic: SketchStatistic,
pub accuracy_metric: AccuracyMetric,
/// Independent trials; every recorded δ is at least 3 / trials.
pub trials: u32,
/// Ascending δ; the error does not increase as δ grows.
pub error_at: Vec<ErrorAtFailureProbability>,
}
pub struct ErrorAtFailureProbability {
pub failure_probability: f64,
/// The (1 − δ) quantile of per-value errors, the worst over the measured merge counts.
pub error: f64,
}Example table:
{"entries": [{"sketch": {"Kll": {"k": 800}}, "statistic": {"Quantile": {"q": 0.99}}, "accuracy_metric": "rank", "trials": 1000, "error_at": [{"failure_probability": 0.01, "error": 0.00176}, {"failure_probability": 0.05, "error": 0.00121}]}]}- Lookup: for a request at failure probability δ, the guarantee is the error of a recorded δ' ≤ δ (the largest such δ'), for the same
sketch,statisticandaccuracy_metric. There is no extrapolation: a δ below every recorded δ', or a configuration with no entry, gets the closed-form guarantee. - Raw results: the per-merge-count results stay in
calibrate-run.json; the table keeps only the worst over merge counts.
Where results live
The defaults below can be overridden with ASAP_CALIBRATION_DIR. calibration/ is in .gitignore.
calibration/
<machine-key>/ # e.g. xeon-gold-6142-56c-251g, from CPU model, cores and memory; never a hostname
machine.json # CPU model, cores, memory, OS
costs.json # MeasuredOperationCosts, read by the planner
accuracy.json # MeasuredSketchAccuracy, read by the planner
calibrate-run.json # parameters, raw samples and fits of this calibration
runs/<machine-key>/<run-id>/ # one directory per bench run; run-id = start time
workload.json plan.txt results.csv summary.md
- Readers:
asap-plannerandasap_rundefault to the current machine'scosts.jsonandaccuracy.json, and--costs <path>/--accuracy <path>override them. - Executor version: the tables are always measured with the executor in the same checkout, and are not tied to a commit. After an executor change, rerun
asap_run calibrate, which overwrites the machine's tables. Machine information lives inmachine.json. - No table for this machine: they print a warning that names the machine key and the
asap_run calibratecommand. They then fall back to the theoretical cost and error estimates, which may be inaccurate. - The planner library is unchanged. Path and machine resolution live in the devtools CLIs, and the library only takes
CostInputsand anAccuracyEvidence.
The showcase gate: asap_run bench --suite showcase
The fixed suite and rule of #1329: for every workload, the selected plan's measured cost ≤ the exact plan's measured cost × 1.05, and its measured error ≤ its target.
- Suite:
sum(rate),sum_over_time,count_over_time;quantile_over_time,avg(quantile_over_time),max_over_time; the joint workload of 5 and the S1 dashboard subsets (W-RangeHeavy, W-RollupHeavy); at 1 Hz every 10 s and 10 Hz every 60 s, with raw data retained and not; plus a 1m-window counterexample. --small(100 series, 10 minutes of data) is the per-PR check; the full scale runs once per milestone.- Output: one row per workload and setting (exact and selected cost, speedup, measured error vs target, pass / fail), into
runs/<machine-key>/<run-id>/summary.md.
Deletes
-
The one-off demo harness.
-
Any separate executor driver elsewhere, which moves to
asap_runonce it is checked for duplication. -
In the
calibratePR:CostTableProvenanceand theprovenancefield ofMeasuredOperationCosts(crates/cost/src/measured.rs, re-exported fromasap-costandasap-planner): the table is not tied to a commit, and machine information lives inmachine.json;crates/devtools/src/rqe_atomic_costs.rsandcrates/devtools/src/bin/convert_rqe_atomic_costs.rs;- the legacy array format in
crates/cost/src/measured.rs(the reader'svalue.is_array()branch and its docs).
sketch-bench's atomic-costs output stays, but the planner does not read it.
Order
Starts after #1300, which fixes how the executor runs each node. Related: #1312, #1313, #1314, #1316, #1324 (column_magnitudes), #1329 (showcase milestones).
Activity