Skip to content

ci(perf-gate): cumulative budget against the last release tag - #295

Merged
unbalancedparentheses merged 5 commits into
mainfrom
ci/perf-gate-release
Nov 21, 2026
Merged

unbalancedparentheses merged 5 commits into
mainfrom
ci/perf-gate-release

Conversation

@unbalancedparentheses

@unbalancedparentheses unbalancedparentheses commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

The perf gate compared each PR only with its merge base (≤ +3% instructions/op). Regressions that each passed could therefore add up across a release cycle.

Release budget. The same job now also measures the last release tag (git describe --tags --abbrev=0 --match 'v*', today v0.8.0). It uses the same driver (the head's copy) and the same harness. It fails when any gated benchmark grows more than 3% over the release (--release, --release-threshold / HOTPATH_RELEASE_THRESHOLD).

  • The job summary table shows benchmark, release, base, head, Δ vs base, Δ vs release, raw spread and the result (base / release).
  • The release binary is cached per release commit, driver hash, toolchain and features (actions/cache). PR caches can only restore what main saved, so pushes to main run hotpath.py build-release to warm the cache; on a cache hit that step does nothing. When the merge base is the release commit, the base binary is reused.
  • The release build uses the same --features as the base and head (default: the crate's defaults). chore!: drop the no-op *-experiments features (0.9) #281 removed only the *-experiments features, which the driver never used. If a future driver uses an API the release lacks, the gate fails with a message saying so rather than a bare build error.

threads_request_seq noise. Valgrind runs one guest thread at a time. A thread blocked in a syscall (futex, park) lets the other run, so the host scheduler decides how the requester and the actor interleave. On an idle runner a woken thread finds a free CPU at once and the count repeats. On a loaded host it does not. The two-thread benchmark now takes the median of 9 run pairs; the async ones stay at 3, since their counts are identical run to run.

Measured with ten back-to-back gates on fede, with 24 extra busy loops on top of the host's own load (load average 25–80 on 30 CPUs). Base and head have identical runtime code, so the true Δ vs base is 0:

method run-pair spread Δ vs base, 10 gates Δ vs v0.8.0, 10 gates
before: median of 3 5–6% −1.5 … +1.5 up to +3.8 (false fail)
min of 9 per size (tried, rejected) 5–6% −1.6 … +3.5 (false fail)
pinned to one CPU (tried, rejected) 1.5% on fede, 10% on the CI runner +1.3 … +1.9 on fede CI: one false fail (+3.2%)
median of 9, unpinned (this PR) 5–6% −0.8 … +1.4 −0.2 … +1.6

On the CI runner the unpinned count repeats: run-pair spread 1 Ir/op, Δ vs base 0.0%. The minimum fails because the noise is in both sizes, so the difference of the minima is not a lower bound. Pinning makes every hand-off depend on wakeup preemption.

On fede, current main is +1.2% vs v0.8.0 on threads_request_seq (after #284, #287 and #282) and between −0.9% and −8.8% on the async benchmarks (#296). A full gate takes about 85 s on fede; on CI the release binary now comes from the cache.

Docs: docs/TESTING.md (Performance gate) and benchmarks/README.md. CHANGELOG bullet added.

Model: no change because this PR changes CI and the benchmark script only, no modelled runtime behaviour.

The gate compared each PR with its merge base only, so regressions under
the 3% threshold could add up across a release cycle. The same job now
also measures the last v* tag with the same driver and harness and fails
when any gated benchmark grows more than 3% over it. The table shows
release, base and head with both deltas. The release binary is cached per
release commit, driver and toolchain, warmed on pushes to main.

Callgrind runs are pinned to one CPU, and threads_request_seq takes the
median of nine run pairs: under load the host scheduler moved it by up
to 3.7%, as much as the threshold.
…st_seq

Pinning helped on loaded fede but on the CI runner it turned a
repeatable count (run pairs within 0.05%) into a 10% spread and a false
failure. Unpinned, the median of nine pairs kept the change against an
identical base within -0.4% to +1.1% across ten gates under heavy load.
@unbalancedparentheses
unbalancedparentheses marked this pull request as ready for review November 21, 2026 17:04
@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

This PR looks solid. The cumulative release budget design, cache key structure (commit + driver hash + toolchain + features), reuse of base_bin when release == base, and the empirical justification for increasing --contended-reps to 9 for threads_request_seq are all well thought out.

One minor nit:

  • benchmarks/hotpath.py:172: csv = open(args.csv, "w") if args.csv else None is never explicitly closed before gate() returns. While sys.exit() right after will flush and close open descriptors, closing csv (e.g., via if csv: csv.close() before returning or a contextlib.ExitStack) would be cleaner.

Automated review by Gemini · gemini-4-argon · custom prompt · 8.3k in / 3.5k out · ~$0.05

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: PR #295, cumulative release budget for perf-gate

The design is sound. A per-PR budget alone lets small regressions add up, and comparing against the last tag closes that gap. Moving to the median of nine run pairs for threads_request_seq is well justified by the data in TESTING.md. Most of the problems are in how the release binary is cached.

Bugs / correctness

1. The workflow cache key and the script's cache key can disagree, and the cache then never repairs itself.

  • perf-gate.yml, "Release binary cache" step: the key is release_sha + hashFiles(driver.rs, rust-toolchain.toml).
  • The toolchain is pinned in the workflow itself (dtolnay/rust-toolchain@1.93.0), not in rust-toolchain.toml. If that file doesn't exist, hashFiles contributes nothing.
  • The script's release_key (hotpath.py, around line 81) hashes rustc -V instead.
  • Effect after a toolchain bump in the workflow:
    • actions/cache reports a primary-key hit.
    • build_release misses on its own key, rebuilds, deletes the old file and writes the new one.
    • actions/cache does not re-save on a primary-key hit.
    • Every PR and every push then rebuilds the release binary until the release or the driver changes.
  • Fix: put the toolchain version in the workflow key. For example, add hashFiles('.github/workflows/perf-gate.yml'), or a rustc -V output captured in an earlier step.

2. The cache file is written non-atomically.

  • In build_release, shutil.copy(binary, cached) writes straight to the final name.
  • A cancelled job, or a timeout now that the limit is 50 minutes, can leave a truncated binary that later runs trust because cached.exists() is true.
  • Fix: copy to cached.with_suffix(".tmp"), then call os.replace.

3. git describe failure breaks every run, including the skip path.

  • perf-gate.yml, scope step: release=$(git describe ... --match 'v*' HEAD) runs before the event or scope checks.
  • If no v* tag is reachable, the step fails. Examples: a shallow or tagless checkout, or a fork without tags.
  • That turns the "always passes when skipped" required check red on docs-only PRs.
  • Fix: compute release only when it is needed, or tolerate failure (|| true) and omit --release.
  • Also note that describe returns the nearest reachable tag. A release tagged on a branch that was never merged back (such as a hotfix) won't be picked up.

4. Every new benchmark that uses a new API is now blocked until the next release.

  • hotpath.py main(): a CalledProcessError from build_release aborts the gate.
  • The driver must compile against the last release. So a PR that adds a benchmark for a new API fails perf-gate, and so does every PR after it, until the next tag.
  • Two smaller issues with the same handler:
    • It also catches unrelated failures, such as git worktree add, and reports them as "driver does not build".
    • The message wording is fine, but this constraint should be called out explicitly in TESTING.md.
  • Options:
    • Let the driver list release-only-safe benchmarks via cfg/feature detection.
    • Fall back to skipping the release comparison, with a warning, when the release build fails.

Minor / maintainability

  • CSV file handle is never closed (gate, csv = open(args.csv, "w")). Use with open(...), or try/finally, so the data is flushed before sys.exit. The name csv also shadows the stdlib module name.
  • contended() doesn't match what the report and docstring say. The predicate is not name.startswith("async_ct_"), but the report says "for the two-thread threads_*". Any future gated benchmark that isn't async_ct_ will silently get 9 reps. Use name.startswith("threads_"), or have the driver list contended benchmarks.
  • Push runs do unnecessary work. The toolchain install and the build-release step run on every push to main, even on a cache hit. Give the cache step an id and gate build-release on steps.<id>.outputs.cache-hit != 'true'. Note that release_key calls rustc, so the toolchain step must stay whenever the build runs.
  • Tag names are interpolated into shell. run: interpolates ${{ steps.scope.outputs.release }} directly. The risk is low because tags are maintainer-controlled, but passing it through env: is the safer habit.
  • per_op docstring has an unwrapped overlong line: "...three reached +3.8%. The minimum is not a stable estimate: ...". Reflow it.
  • Internal hostname in public docs. docs/TESTING.md names "fede". Prefer "a heavily loaded 30-CPU host".
  • build-release mode computes args.reps and accepts --csv/--only silently. This is harmless, but rejecting irrelevant flags would make misuse obvious.

Looks good

  • release_key resolves ^{commit}, so a tag name (in build-release) and a SHA (in gate) produce the same key.
  • Reusing base_bin when release == base avoids a redundant build.
  • The finally block now also cleans up the release worktree.
  • Recording the rejected approaches (the minimum, CPU pinning) and their measured spreads in TESTING.md and next to instructions() is valuable context.

Automated review by Claude · claude-opus-5-5 · custom prompt · 11.3k in / 5.0k out · ~$0.15

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

  • [P2] Include compiler flags in the release cache key — benchmarks/hotpath.py:81. The key omits RUSTFLAGS and CARGO_ENCODED_RUSTFLAGS, although Cargo honors them during builds. Reusing --release-cache after changing optimization or CPU flags compares freshly built base/head binaries against a release built with different settings, potentially producing false passes or failures. Include these build inputs in the key or explicitly normalize them for every build.

No actor runtime code changes are included in this PR.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt · 94.6k in / 698 out · ~$0.08

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

The cumulative release budget is a sound idea, and pairing callgrind samples then taking the median is the right fix for threads_request_seq. A few gate bugs will make that check flaky or permanently more expensive than the warm-cache design intends.

Bugs

  1. Cache hit is not refreshed, so a bad entry sticks. In .github/workflows/perf-gate.yml the actions/cache key is release_sha + hashFiles(driver.rs, rust-toolchain.toml), while benchmarks/hotpath.py looks up a different name (release_key, ~line 81: commit prefix, SHA-256 of the driver, SHA-256 of rustc -V). actions/cache does not save again on hit. If those keys disagree, or the restored dir is empty/partial, build_release (~lines 100–104) rebuilds, deletes every hotpath-* file, writes a new binary, and the updated dir is thrown away. Every later PR repeats that. build_release also unlinks all hotpath-* entries before shutil.copy; a failed copy on a cache miss can persist an empty dir under that key. Put rustc -V (and features, if you ever pass them) in the Actions key, or treat the Python filename as the only key and skip the gate’s release build when steps.*.outputs.cache-hit == 'true' only after checking that file exists.

  2. git describe runs before the skip and is unquoted (workflow scope step, ~lines 44–51). A docs-only PR used to pass the required check without tags. Now git describe --tags --abbrev=0 --match 'v*' HEAD failing fails the job even when run=false. Describe from the PR head also is not “latest release on main” unless checkout is the merge ref (refs/pull/N/merge); a head-only checkout measures an older ancestor tag. Quote the tag (--release "${{ steps.scope.outputs.release }}") in both run: scripts.

  3. Release build failure aborts the per-PR gate. In main() the CalledProcessError handler (~line 345) sys.exits before gate(), and any failure (worktree, missing tag object, cargo) is reported as “driver does not build against the last release”. There is no fallback, so a harness change that uses a post-release API turns this required check red until somebody tags. Confirm current benchmarks/hotpath/driver.rs still builds at the current v* tag; otherwise this PR is red on every PR and every main warm. Narrow the except to the cargo build, and still emit the base-vs-head table.

  4. Timeout is tight for the new work. Three trees instead of two, and contended benches go from 3 to 9 pairs (--contended-reps default 9). That is roughly 2× the old callgrind count; timeout-minutes only moved 40 → 50. On a cache miss the PR job also does a full release cargo build.

Correctness / maintainability

contended() (~line 135) is not name.startswith("async_ct_"), but the changelog and docs/TESTING.md only justify nine pairs for threads_request_seq. Any other list-gate name silently gets 9× valgrind and the summary still says threads_*. Match the specific benchmark.

release == base skips build_release and reuses base samples (gate(), release_bin == base_bin). That comparison is fine, but then only the main warm job fills the cache for that key. The warm job still installs the toolchain on every main push even on cache hit, because release_key shells out to rustc -V.

open(args.csv) in gate() is never closed; use a context manager so a mid-loop failure still flushes.

The actor lifecycle, cancellation, and message-ordering paths are untouched. The measurement change itself (back-to-back small/large pairs, median, no CPU pin, fair-sched=yes) matches the noise analysis in the docs.


Automated review by Grok (xAI) · grok-4.7 · custom prompt · 9.1k in / 21.2k out · ~$0.15

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

Overall, the release/base/head measurement flow and paired-median calculation look consistent. I found three issues in the release-cache handling:

  • [P2] Feature cache keys can collide — benchmarks/hotpath.py:87. Replacing every non-alphanumeric sequence with _ means distinct feature arguments such as foo-bar, foo_bar, and foo bar produce the same key. Because build_release() trusts an existing file at lines 96–99, a gate can silently measure a release binary built with different features. Hash the original feature argument instead of using lossy sanitization.

  • [P3] Shared-cache cleanup is racy — benchmarks/hotpath.py:100-104. Two processes using the same --release-cache can delete each other’s binaries: one process may return cached, then another removes it through stale.unlink() before Valgrind executes it. The copy is also visible before it completes. CI’s per-job temporary directory avoids this, but the CLI exposes shared cache directories. Prefer retaining keyed entries, or use locking and atomic temporary-file-plus-rename publication if cleanup is required.

  • [P3] Release failures are reported too broadly as driver incompatibilities — benchmarks/hotpath.py:345-352. The CalledProcessError handler also covers rustc -V, git rev-parse, worktree creation, and non-compilation Cargo failures such as dependency downloads. Those all produce the misleading “driver does not build” message. Preserve the original command/error and limit the compatibility hint to the actual driver compilation failure.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt · 7.3k in / 8.6k out · ~$0.10

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

The PR is mostly Python/CI harness code, so no Rust memory-safety or data-race issues. Overall the approach is reasonable: adding a cumulative release-budget gate and caching the release binary is a good idea.

A few specific concerns:

  1. build_release cache cleanup is destructive for shared cache directories
    In benchmarks/hotpath.py, build_release does:

    for stale in cache.glob("hotpath-*"):
        stale.unlink()

    If a user points --release-cache at a directory that contains binaries for multiple releases or feature sets (which the docstring implies is possible by keying on release commit, driver and toolchain), this deletes all of them before copying the new binary. In CI the cache directory is per-key, so it works, but locally it can delete unrelated cache entries. Better to remove only the specific cached file or use a separate subdirectory per key.

  2. CSV file is opened without a context manager and never explicitly closed
    In gate:

    csv = open(args.csv, "w") if args.csv else None

    The file is never closed. It likely flushes on interpreter exit, but using with open(...) as csv: is clearer and avoids possible lost writes if the process is killed or exceptions occur.

  3. Release measurement happens after base/head for each benchmark
    In gate, for each benchmark, per_op is called for base and head first, then for release (if not equal to base). If the host load changes during the job, the release measurement may be taken under different conditions than base/head, potentially biasing the release comparison. The code already accepts median-of-9 for the contended threads_request_seq to mitigate load noise, but temporal separation between base/head and release is still a risk. Consider interleaving the three sides per repetition or randomizing the order.

  4. Ensure benchmarks/hotpath.py is in the changed-path filter in the workflow
    The workflow’s changed detection determines whether the perf gate runs. This PR modifies benchmarks/hotpath.py itself. If the filter only checks runtime source/driver/toolchain paths and not the harness, changes like this won’t be exercised by CI. I can’t see the full filter from the diff, but please confirm benchmarks/hotpath.py is included.

  5. CSV output omits release rows when release == base
    In gate, when release_bin == base_bin, the release samples are reused (r, rs = b, bs), but no CSV row is written for the release side. The CSV header still lists side, so downstream consumers may expect three sides for every benchmark. Either write the release row too, or document that it is omitted when identical to base.

  6. Minor cleanup noise in build-release/gate finally blocks
    The finally blocks always try to remove work/release, even when the release worktree was never created (cache hit or release not used). git worktree remove failures are captured and ignored, so it’s harmless, but it can be avoided with an existence check or a flag.

The per_op refactor and the median-of-9 for contended benchmarks look correct, and the workflow cache key logic is sound for the CI use case.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt · 7.9k in / 35.8k out · ~$0.15

@unbalancedparentheses
unbalancedparentheses added this pull request to the merge queue Nov 21, 2026
Merged via the queue into main with commit 96803e5 Nov 21, 2026
43 of 44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant