Skip to content

One script for the synthetic use cases; systems runner takes any spec, error budgets and raw samples - #248

Merged
GordonYuanyc merged 3 commits into
mainfrom
run-any-spec-on-systems
Nov 13, 2026
Merged

GordonYuanyc merged 3 commits into
mainfrom
run-any-spec-on-systems

Conversation

@GordonYuanyc

Copy link
Copy Markdown
Collaborator
  • scripts/use_cases/run_all_use_cases.sh (new): builds planeval, runs each use case on its synthetic workload one after another, writes the rows to target/grafana-data, then serves the dashboards. --use-case 1,2,..., --quick (one approximate arm per use case) and --no-grafana. On an 8-core laptop: build about 11 min, --quick about 11 min, every arm about 1 h 40 min; peak memory under 4 GB.
  • tools/grafana/run.sh: Grafana's state and the Infinity plugin live under target/grafana-home instead of the user's Grafana; Grafana is found through Homebrew or GRAFANA_HOME; --check installs the plugin and exits.
  • scripts/use_cases/run_use_case_on_systems.py: --spec runs any spec file and --arm (repeatable) sets the error budgets; VictoriaMetrics runs once per spec. Arms that keep raw samples tell the planner so and give the ASAPQuery-backend data plane a VictoriaMetrics instance holding the raw samples.
  • docs/aqpbm-planeval.md, docs/phase2-use-cases/canary.md: script paths under scripts/use_cases/, and the current dashboard URL.

Checked: run_all_use_cases.sh with --quick and with every arm from an empty target, all 5 phase 2 dashboards return rows; test_run_use_case_on_systems.py passes.

🤖 Generated with Claude Code

GordonYuanyc and others added 3 commits November 12, 2026 23:36
--spec runs a spec file on VictoriaMetrics and ASAPQuery-backend; --arm picks
the error budgets. Queries that do not plan together plan alone, one instance
each; queries the planner or the data plane refuses are recorded as unsupported.
Each budget writes a summary with per-query agreement against VictoriaMetrics
and the planner's time and memory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oards

run_all_use_cases.sh builds planeval, runs the use cases one after another and starts Grafana.
tools/grafana/run.sh keeps Grafana's state and the Infinity plugin under target/grafana-home and
finds Grafana through Homebrew or GRAFANA_HOME.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and current dashboards

Arms that keep raw samples tell the planner so and give the ASAPQuery-backend data plane a
VictoriaMetrics instance holding the raw samples. The docs use the scripts/use_cases paths and the
canary-release dashboard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@GordonYuanyc
GordonYuanyc merged commit c956127 into main Nov 13, 2026
1 of 2 checks passed
@GordonYuanyc
GordonYuanyc deleted the run-any-spec-on-systems branch November 13, 2026 09:07
GordonYuanyc added a commit that referenced this pull request Nov 16, 2026
… needless_range_loop

Clippy in Rust 1.100 rejects indexing self.quantiles by the loop variable, which fails
format-and-lint on main since #248 and on this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GordonYuanyc added a commit that referenced this pull request Nov 16, 2026
… by default (#251)

* grafana: describe each use case's data and rules on its dashboard; synthetic runs only by default

Each use case dashboard now opens with its scenario figure, a plain description of the
synthetic data (every column, its usual values and the events injected over time), and every
query and rule with what it does. The rule dropdown names the threshold rules by their
condition, entity dropdowns use each use case's own word (server, container, subpopulation,
client), and injected problem periods and the times approx decides differently from exact are
always drawn on the time charts.

generate_dashboards.py takes --phase, --out, --no-figure and --real-data; by default it shows
synthetic runs only, which is what run_all_use_cases.sh produces. run.sh takes the same --phase
and --real-data and serves the scenario figures on port 8001. The overview table gains its own
row, split plan columns and a target error column; panel heights follow their text. Dataset and
policy come from the run name, so the dashboards read only fields the current runner writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* polars_shared: fill the quantile array through iter_mut, for clippy's needless_range_loop

Clippy in Rust 1.100 rejects indexing self.quantiles by the loop variable, which fails
format-and-lint on main since #248 and on this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant