Repository navigation
Reusable layers: aqpbm-usecase crate, System trait, typed outputs, scripts by role - #246
Merged
Merged
Conversation
Files move as they are; repository-root lookups and the one import that crosses directories (data_root, video_spec) follow them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ead it Each scripts/use_cases/cases/uc*.py holds the use case's rules, its table entry (spec, arms, derive, signals) and its dashboard parts. The runners find the cases by listing the package, and generate_dashboards.py builds every dashboard from them; the generated dashboards are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A query's truth (statistic, metric, label matchers; top-k's k and label) is now a field of the spec, and its window is the query's window_ms, so score() needs neither an arm nor the planner. A planeval test lowers every workload query with ASAPPlanner's PromQL frontend and checks the declared truth equals what the lowered plan holds. systemeval drops the arm it built only to score. Bound queries and the video spec writer carry the field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The root .gitignore ignores every datasets/ directory (the data root), so a new script under scripts/datasets/ would never be added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…planeval aqpbm-usecase holds spec.rs, data.rs, accuracy.rs, score.rs and the workloads, and depends only on aqpbm-core and aqpbm-datagen. planeval keeps the ASAPPlanner side (plan, run, sketch_analytics); systemeval now depends on aqpbm-usecase, so its build no longer includes ASAPPlanner. Tests that need no planner move with the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t comment The path dependency on ../../asap_sketchlib only built with that checkout next to this one; the git rev is the one the workspace pins. The root manifest's first line drops the sketchlib-tool name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A system writes a Remote Write body (the default posts it to /api/v1/write), makes samples up to a time visible, and may report a status. ASAPQuery-backend and VictoriaMetrics each implement it in their own file and are listed in SYSTEMS, which --system draws its values from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both documents are now serde structs in aqpbm_usecase::output with a schema_version, in place of the json! objects built in each main.rs. The key names, and which keys an unsupported query carries, are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…spatch run_sketchbench and its helpers move to sketchbench.rs, and every module (trials, repeat, the subcommands) is reachable as aqpbm_cli::*, so another Rust program can call them without spawning approxbench. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fold of one invocation's records into a MergedRecord is part of the result format, so it lives with the record types; approxbench's --flat and flatten subcommand call it from there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mmands Both shape benchmark records into ASAPQuery's optimizer inputs (the atomic cost table and the error-resource profile), which only approxbench's atomic-costs and erp subcommands produce; aqpbm-core keeps measurement and the result format. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… script Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Splits the benchmark along its reuse boundaries. The lower crates were already clean; the work is in the upper layer (planeval, systemeval, scripts, Grafana).
Changes, one commit each
Use-case layer independent of ASAPPlanner
planeval: each query declares its truth—QuerySpec.truthholds the statistic, metric and label matchers (plus top-k'skand label); the window iswindow_ms.score()no longer takes anArmor plans anything, and systemeval's placeholderArmis gone. Truth is still never parsed from PromQL:aqpbm-planeval/tests/declared_truth.rslowers every workload query with ASAPPlanner's PromQL frontend and asserts the declared truth equals the lowered plan's (checked to fail when one truth is edited). This reverses "truth comes from the planner IR" from planeval: score answers through aqpbm-core; windowed ground truth #238; the test keeps the two from drifting.usecase: a crate for specs, samples, truth and scoring— newaqpbm-usecase(spec,data,accuracy,score,workloads/), depending only onaqpbm-coreandaqpbm-datagen. planeval keepsplan,run,sketch_analytics. systemeval now depends onaqpbm-usecase;cargo tree -p aqpbm-systemevalhas no ASAPPlanner or DataFusion.usecase: planeval and systemeval write typed, versioned output— serde structs inaqpbm_usecase::outputwithschema_version: 1. Key names and which keys an unsupported query carries are unchanged.Systems under test
systemeval: a System trait— write (default: Remote Write POST), make visible through t, status. One file per system undersrc/system/, listed inSYSTEMS;--systemvalues come from that list.Scripts and dashboards
scripts: split into use_cases, datasets, studies and sketch_runs, thendataset_tools instead of datasets— the root.gitignoreignores everydatasets/directory, so new files underscripts/datasets/would never be added. Repository-root lookups and the one cross-directory import follow the move.use cases: one module per use case—scripts/use_cases/cases/uc*.pyholds each case's rules, its table entry and its dashboard parts.run_use_case.pyandrun_use_case_on_systems.pyfind cases by listing the package;generate_dashboards.pybuilds every dashboard from them. Canary's rules stay inrollout.py, whichuc4_canary_release.pyuses.Sketch layer
cli: a library behind approxbench—aqpbm_cli::{sketchbench, trials, repeat, …};main.rskeeps the allocator and dispatch.core: flatten_record moves next to MergedRecord.cli: atomic_costs and erp move out of aqpbm-core.rust-microbench: asap_sketchlib from the workspace's git rev(a05b35c, as the workspace) instead of../../asap_sketchlib; the root manifest comment dropssketchlib-tool.Not changed
sketch-bench,approxbench,aqpbm-).aqpbm-usecase, notaqpbm-datagen.approx_rows.pyrow format and system process management stay in Python.generate_dashboards.py; what moved to the case modules is the composition, specs, labels, rule sets, scaling thresholds and canary query colours.docs/*.mdandconfigs/workloads/README.mdare untouched and now out of date in places:docs/aqpbm-planeval.md,docs/aqpbm-core.md,docs/aqpbm-cli.md,configs/workloads/README.md(still runsscripts/run_external_sweep.py, nowscripts/dataset_tools/run_external_sweep.py), and any page citingscripts/<file>oraqpbm-planeval/workloads.Verification
cargo build --release --workspace: no warnings.cargo test --workspacepasses.cargo clippy --workspace --all-targets --all-features --locked -- -D warningsandcargo fmt --checkpass. Every Python test file passes.main: exact runs of use cases 1, 2 and 5 are identical apart from timing fields and the newschema_version. An approximate use case 2 run has the same structure and plans; its values differ, as they do between two runs ofmain's binary.generate_dashboards.pyoutput is byte-identical to the committed dashboards.run_use_case.py 5 --arm noraw-ignore-eps0.05runs end to end.mainvs this branch, interleaved, median of 3 × 10 runs, 1 M uniform keys: cms +0.4%, countsketch +0.2%, hll +0.7%, kll-cdf +0.3%.🤖 Generated with Claude Code