Skip to content

Reusable layers: aqpbm-usecase crate, System trait, typed outputs, scripts by role - #246

Merged
GordonYuanyc merged 12 commits into
mainfrom
refactor/reusable-layers
Nov 11, 2026
Merged

GordonYuanyc merged 12 commits into
mainfrom
refactor/reusable-layers

Conversation

@GordonYuanyc

@GordonYuanyc GordonYuanyc commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Splits the benchmark along its reuse boundaries. The lower crates were already clean; the work is in the upper layer (planeval, systemeval, scripts, Grafana).

Changes, one commit each

Use-case layer independent of ASAPPlanner

  • planeval: each query declares its truth — QuerySpec.truth holds the statistic, metric and label matchers (plus top-k's k and label); the window is window_ms. score() no longer takes an Arm or plans anything, and systemeval's placeholder Arm is gone. Truth is still never parsed from PromQL: aqpbm-planeval/tests/declared_truth.rs lowers every workload query with ASAPPlanner's PromQL frontend and asserts the declared truth equals the lowered plan's (checked to fail when one truth is edited). This reverses "truth comes from the planner IR" from planeval: score answers through aqpbm-core; windowed ground truth #238; the test keeps the two from drifting.
  • usecase: a crate for specs, samples, truth and scoring — new aqpbm-usecase (spec, data, accuracy, score, workloads/), depending only on aqpbm-core and aqpbm-datagen. planeval keeps plan, run, sketch_analytics. systemeval now depends on aqpbm-usecase; cargo tree -p aqpbm-systemeval has no ASAPPlanner or DataFusion.
  • usecase: planeval and systemeval write typed, versioned output — serde structs in aqpbm_usecase::output with schema_version: 1. Key names and which keys an unsupported query carries are unchanged.

Systems under test

  • systemeval: a System trait — write (default: Remote Write POST), make visible through t, status. One file per system under src/system/, listed in SYSTEMS; --system values come from that list.

Scripts and dashboards

  • scripts: split into use_cases, datasets, studies and sketch_runs, then dataset_tools instead of datasets — the root .gitignore ignores every datasets/ directory, so new files under scripts/datasets/ would never be added. Repository-root lookups and the one cross-directory import follow the move.
  • use cases: one module per use case — scripts/use_cases/cases/uc*.py holds each case's rules, its table entry and its dashboard parts. run_use_case.py and run_use_case_on_systems.py find cases by listing the package; generate_dashboards.py builds every dashboard from them. Canary's rules stay in rollout.py, which uc4_canary_release.py uses.

Sketch layer

  • cli: a library behind approxbench — aqpbm_cli::{sketchbench, trials, repeat, …}; main.rs keeps the allocator and dispatch.
  • core: flatten_record moves next to MergedRecord.
  • cli: atomic_costs and erp move out of aqpbm-core.
  • rust-microbench: asap_sketchlib from the workspace's git rev (a05b35c, as the workspace) instead of ../../asap_sketchlib; the root manifest comment drops sketchlib-tool.

Not changed

  • Crate, binary and prefix names (sketch-bench, approxbench, aqpbm-).
  • Time-series generation stays in aqpbm-usecase, not aqpbm-datagen.
  • approx_rows.py row format and system process management stay in Python.
  • The app dashboards (alert, decision, rollout, top-k) still name their panels' queries inside generate_dashboards.py; what moved to the case modules is the composition, specs, labels, rule sets, scaling thresholds and canary query colours.
  • docs/*.md and configs/workloads/README.md are untouched and now out of date in places: docs/aqpbm-planeval.md, docs/aqpbm-core.md, docs/aqpbm-cli.md, configs/workloads/README.md (still runs scripts/run_external_sweep.py, now scripts/dataset_tools/run_external_sweep.py), and any page citing scripts/<file> or aqpbm-planeval/workloads.

Verification

  • Cold cargo build --release --workspace: no warnings. cargo test --workspace passes. cargo clippy --workspace --all-targets --all-features --locked -- -D warnings and cargo fmt --check pass. Every Python test file passes.
  • planeval against a build of main: exact runs of use cases 1, 2 and 5 are identical apart from timing fields and the new schema_version. An approximate use case 2 run has the same structure and plans; its values differ, as they do between two runs of main's binary.
  • generate_dashboards.py output is byte-identical to the committed dashboards.
  • run_use_case.py 5 --arm noraw-ignore-eps0.05 runs end to end.
  • Throughput, main vs this branch, interleaved, median of 3 × 10 runs, 1 M uniform keys: cms +0.4%, countsketch +0.2%, hll +0.7%, kll-cdf +0.3%.

🤖 Generated with Claude Code

GordonYuanyc and others added 12 commits November 11, 2026 00:58
Files move as they are; repository-root lookups and the one import that
crosses directories (data_root, video_spec) follow them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ead it

Each scripts/use_cases/cases/uc*.py holds the use case's rules, its table
entry (spec, arms, derive, signals) and its dashboard parts. The runners
find the cases by listing the package, and generate_dashboards.py builds
every dashboard from them; the generated dashboards are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A query's truth (statistic, metric, label matchers; top-k's k and label)
is now a field of the spec, and its window is the query's window_ms, so
score() needs neither an arm nor the planner. A planeval test lowers every
workload query with ASAPPlanner's PromQL frontend and checks the declared
truth equals what the lowered plan holds. systemeval drops the arm it built
only to score. Bound queries and the video spec writer carry the field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The root .gitignore ignores every datasets/ directory (the data root), so
a new script under scripts/datasets/ would never be added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…planeval

aqpbm-usecase holds spec.rs, data.rs, accuracy.rs, score.rs and the
workloads, and depends only on aqpbm-core and aqpbm-datagen. planeval keeps
the ASAPPlanner side (plan, run, sketch_analytics); systemeval now depends
on aqpbm-usecase, so its build no longer includes ASAPPlanner. Tests that
need no planner move with the code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t comment

The path dependency on ../../asap_sketchlib only built with that checkout
next to this one; the git rev is the one the workspace pins. The root
manifest's first line drops the sketchlib-tool name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A system writes a Remote Write body (the default posts it to
/api/v1/write), makes samples up to a time visible, and may report a
status. ASAPQuery-backend and VictoriaMetrics each implement it in their
own file and are listed in SYSTEMS, which --system draws its values from.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both documents are now serde structs in aqpbm_usecase::output with a
schema_version, in place of the json! objects built in each main.rs. The
key names, and which keys an unsupported query carries, are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…spatch

run_sketchbench and its helpers move to sketchbench.rs, and every module
(trials, repeat, the subcommands) is reachable as aqpbm_cli::*, so another
Rust program can call them without spawning approxbench.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fold of one invocation's records into a MergedRecord is part of the
result format, so it lives with the record types; approxbench's --flat and
flatten subcommand call it from there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mmands

Both shape benchmark records into ASAPQuery's optimizer inputs (the
atomic cost table and the error-resource profile), which only approxbench's
atomic-costs and erp subcommands produce; aqpbm-core keeps measurement and
the result format.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… script

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@GordonYuanyc
GordonYuanyc merged commit ee0e05c into main Nov 11, 2026
2 checks passed
@GordonYuanyc
GordonYuanyc deleted the refactor/reusable-layers branch November 11, 2026 08:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant