Skip to content

ci: compare PR benchmarks with base#2296

Open
zeitlinger wants to merge 1 commit into
mainfrom
ci/pr-benchmark-base-comparison
Open

ci: compare PR benchmarks with base#2296
zeitlinger wants to merge 1 commit into
mainfrom
ci/pr-benchmark-base-comparison

Conversation

@zeitlinger

@zeitlinger zeitlinger commented Jul 9, 2026

Copy link
Copy Markdown
Member

Summary

Make PR benchmark comments answer whether the PR is faster or slower than its base commit, while keeping wall-clock time close to a single benchmark run.

Changes

  • Run base and head benchmarks in parallel matrix jobs.
  • Add a lightweight summary job that downloads both raw JMH artifacts and generates the PR comment.
  • Add a Comparison with base table to the generated README.md / PR comment.
  • Mark each benchmark as faster, slower, or within noise using JMH confidence intervals when available.
  • Capture and show runner hardware for both base and head jobs, including a note that parallel jobs may run on different hardware.
  • Rename the existing per-run relative column to Within run so it is clear that it is not the base comparison.
  • Store baseline-results.json in the uploaded benchmark artifact for inspection.

Validation

  • Ran mise run lint:fix.
  • Ran python3 -m py_compile .mise/tasks/generate_benchmark_summary.py.
  • Ran git diff --check.
  • Replayed summary generation with the latest perf: optimize classic-only histogram observe path #2252 benchmark artifact and the published benchmark baseline; the output now shows HistogramBenchmark.prometheusClassic as -78.6% / slower.
  • Verified no-baseline summary generation still omits the comparison section.

Tradeoff

Parallel base/head jobs are faster wall-clock than measuring both commits sequentially on one runner, but runner hardware can differ. The comment now prints both runner descriptions and calls out that caveat.

@zeitlinger zeitlinger marked this pull request as ready for review July 9, 2026 13:55
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
@zeitlinger zeitlinger force-pushed the ci/pr-benchmark-base-comparison branch from 086ff4a to 25a8b0b Compare July 9, 2026 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant