Skip to content

ci: shard the benchmark workflow across parallel jobs#958

Draft
zeroshade wants to merge 1 commit into
apache:mainfrom
zeroshade:bench-parallel-shards
Draft

ci: shard the benchmark workflow across parallel jobs#958
zeroshade wants to merge 1 commit into
apache:mainfrom
zeroshade:bench-parallel-shards

Conversation

@zeroshade

Copy link
Copy Markdown
Member

Rationale for this change

The Benchmarks workflow runs go test -bench=. ./... sequentially across every
package. Because go test's -timeout is applied per package, the wall-clock time
is the sum over all packages — recent main runs have taken roughly 3.3 hours
(198–208 min).

What changes are included in this PR?

Split the benchmark run so it can be parallelized, then combine the results into a
single upload:

  • ci/scripts/bench.sh — adds --run (benchmark a subset of packages, writing
    raw output to a .dat) and --aggregate (merge one or more .dat files into a
    single bench_stats.json via gobenchdata) modes. The existing
    bench.sh <dir> [--json|-json] interface is unchanged, so nothing else that calls
    it needs to change.
  • ci/scripts/bench_shard.sh (new) — prints a GitHub Actions matrix that buckets
    the packages containing benchmarks into N shards.
  • .github/workflows/benchmark.yml — reworked into three jobs: setup (compute
    the shard matrix) → benchmark (matrix; each shard runs its packages and uploads
    its .dat) → combine (download all .dat, aggregate into one bench_stats.json,
    and — only on push to main — upload once to Conbench).
  • ci/scripts/bench_adapt.py — reuses an existing bench_stats.json (produced by
    combine) instead of re-running the whole suite.

Because the shards are merged into one JSON and uploaded once, Conbench still sees a
single run (no run_id fragmentation).

Are these changes tested?

Locally:

  • shellcheck clean on both scripts; actionlint clean on the workflow; py_compile
    OK on bench_adapt.py.
  • Verified the split→merge end to end: ran --run on two packages, then --aggregate
    produced one bench_stats.json containing both suites, in the exact shape
    bench_adapt.py consumes.
  • The legacy bench.sh <dir> --json path still runs → aggregates → cleans up.

Opened as a draft to exercise the reworked workflow in CI end to end (it triggers
on changes to these files).

Are there any user-facing changes?

No. This only touches CI / benchmark tooling.

Notes / follow-ups

  • Sharding is currently round-robin by package, not runtime-weighted, so a single
    shard can hold two heavy packages (e.g. arrow/compute + parquet/internal/encoding)
    and become the long pole. The per-package -timeout (40m) remains the hard floor for
    any single package. Once this runs, per-shard timings can seed a runtime-weighted
    split or tune the shard count.

The Benchmarks workflow ran `go test -bench=. ./...` sequentially over every
package, and because `-timeout` applies per package the wall-clock time summed
to roughly three hours.

Split the run so it can be parallelized:

- bench.sh gains --run (benchmark a subset of packages into a .dat) and
  --aggregate (combine .dat files into one JSON) modes; the legacy
  "<dir> [--json]" interface is unchanged.
- bench_shard.sh emits a GitHub Actions matrix that buckets the packages
  containing benchmarks into shards.
- benchmark.yml becomes setup -> benchmark (shard matrix) -> combine. Each shard
  uploads its .dat; combine merges them into a single bench_stats.json and, on
  push to main, uploads once to Conbench.
- bench_adapt.py reuses an existing bench_stats.json instead of re-running the
  suite.

Aggregating into one JSON preserves a single Conbench run (no fragmentation).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant