ci: shard the benchmark workflow across parallel jobs#958
Draft
zeroshade wants to merge 1 commit into
Draft
Conversation
The Benchmarks workflow ran `go test -bench=. ./...` sequentially over every package, and because `-timeout` applies per package the wall-clock time summed to roughly three hours. Split the run so it can be parallelized: - bench.sh gains --run (benchmark a subset of packages into a .dat) and --aggregate (combine .dat files into one JSON) modes; the legacy "<dir> [--json]" interface is unchanged. - bench_shard.sh emits a GitHub Actions matrix that buckets the packages containing benchmarks into shards. - benchmark.yml becomes setup -> benchmark (shard matrix) -> combine. Each shard uploads its .dat; combine merges them into a single bench_stats.json and, on push to main, uploads once to Conbench. - bench_adapt.py reuses an existing bench_stats.json instead of re-running the suite. Aggregating into one JSON preserves a single Conbench run (no fragmentation).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The
Benchmarksworkflow runsgo test -bench=. ./...sequentially across everypackage. Because
go test's-timeoutis applied per package, the wall-clock timeis the sum over all packages — recent
mainruns have taken roughly 3.3 hours(198–208 min).
What changes are included in this PR?
Split the benchmark run so it can be parallelized, then combine the results into a
single upload:
ci/scripts/bench.sh— adds--run(benchmark a subset of packages, writingraw output to a
.dat) and--aggregate(merge one or more.datfiles into asingle
bench_stats.jsonviagobenchdata) modes. The existingbench.sh <dir> [--json|-json]interface is unchanged, so nothing else that callsit needs to change.
ci/scripts/bench_shard.sh(new) — prints a GitHub Actions matrix that bucketsthe packages containing benchmarks into N shards.
.github/workflows/benchmark.yml— reworked into three jobs:setup(computethe shard matrix) →
benchmark(matrix; each shard runs its packages and uploadsits
.dat) →combine(download all.dat, aggregate into onebench_stats.json,and — only on push to
main— upload once to Conbench).ci/scripts/bench_adapt.py— reuses an existingbench_stats.json(produced bycombine) instead of re-running the whole suite.Because the shards are merged into one JSON and uploaded once, Conbench still sees a
single run (no
run_idfragmentation).Are these changes tested?
Locally:
shellcheckclean on both scripts;actionlintclean on the workflow;py_compileOK on
bench_adapt.py.--runon two packages, then--aggregateproduced one
bench_stats.jsoncontaining both suites, in the exact shapebench_adapt.pyconsumes.bench.sh <dir> --jsonpath still runs → aggregates → cleans up.Opened as a draft to exercise the reworked workflow in CI end to end (it triggers
on changes to these files).
Are there any user-facing changes?
No. This only touches CI / benchmark tooling.
Notes / follow-ups
shard can hold two heavy packages (e.g.
arrow/compute+parquet/internal/encoding)and become the long pole. The per-package
-timeout(40m) remains the hard floor forany single package. Once this runs, per-shard timings can seed a runtime-weighted
split or tune the shard count.