Repository navigation
ci(perf-gate): cumulative budget against the last release tag - #295
Conversation
The gate compared each PR with its merge base only, so regressions under the 3% threshold could add up across a release cycle. The same job now also measures the last v* tag with the same driver and harness and fails when any gated benchmark grows more than 3% over it. The table shows release, base and head with both deltas. The release binary is cached per release commit, driver and toolchain, warmed on pushes to main. Callgrind runs are pinned to one CPU, and threads_request_seq takes the median of nine run pairs: under load the host scheduler moved it by up to 3.7%, as much as the threshold.
ee65424 to
9077423
Compare
…st_seq Pinning helped on loaded fede but on the CI runner it turned a repeatable count (run pairs within 0.05%) into a 10% spread and a false failure. Unpinned, the median of nine pairs kept the change against an identical base within -0.4% to +1.1% across ten gates under heavy load.
🤖 Gemini Code ReviewThis PR looks solid. The cumulative release budget design, cache key structure (commit + driver hash + toolchain + features), reuse of One minor nit:
Automated review by Gemini · gemini-4-argon · custom prompt · 8.3k in / 3.5k out · ~$0.05 |
🤖 Claude Code ReviewReview: PR #295, cumulative release budget for
|
🤖 Codex Code Review
No actor runtime code changes are included in this PR. Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt · 94.6k in / 698 out · ~$0.08 |
🤖 Grok Code ReviewThe cumulative release budget is a sound idea, and pairing callgrind samples then taking the median is the right fix for Bugs
Correctness / maintainability
The actor lifecycle, cancellation, and message-ordering paths are untouched. The measurement change itself (back-to-back small/large pairs, median, no CPU pin, Automated review by Grok (xAI) · grok-4.7 · custom prompt · 9.1k in / 21.2k out · ~$0.15 |
🤖 Kimi Code ReviewOverall, the release/base/head measurement flow and paired-median calculation look consistent. I found three issues in the release-cache handling:
Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt · 7.3k in / 8.6k out · ~$0.10 |
🤖 DeepSeek Code ReviewThe PR is mostly Python/CI harness code, so no Rust memory-safety or data-race issues. Overall the approach is reasonable: adding a cumulative release-budget gate and caching the release binary is a good idea. A few specific concerns:
The Automated review by DeepSeek · deepseek-v4-pro · custom prompt · 7.9k in / 35.8k out · ~$0.15 |
The perf gate compared each PR only with its merge base (≤ +3% instructions/op). Regressions that each passed could therefore add up across a release cycle.
Release budget. The same job now also measures the last release tag (
git describe --tags --abbrev=0 --match 'v*', todayv0.8.0). It uses the same driver (the head's copy) and the same harness. It fails when any gated benchmark grows more than 3% over the release (--release,--release-threshold/HOTPATH_RELEASE_THRESHOLD).actions/cache). PR caches can only restore what main saved, so pushes to main runhotpath.py build-releaseto warm the cache; on a cache hit that step does nothing. When the merge base is the release commit, the base binary is reused.--featuresas the base and head (default: the crate's defaults). chore!: drop the no-op *-experiments features (0.9) #281 removed only the*-experimentsfeatures, which the driver never used. If a future driver uses an API the release lacks, the gate fails with a message saying so rather than a bare build error.threads_request_seqnoise. Valgrind runs one guest thread at a time. A thread blocked in a syscall (futex, park) lets the other run, so the host scheduler decides how the requester and the actor interleave. On an idle runner a woken thread finds a free CPU at once and the count repeats. On a loaded host it does not. The two-thread benchmark now takes the median of 9 run pairs; the async ones stay at 3, since their counts are identical run to run.Measured with ten back-to-back gates on fede, with 24 extra busy loops on top of the host's own load (load average 25–80 on 30 CPUs). Base and head have identical runtime code, so the true Δ vs base is 0:
On the CI runner the unpinned count repeats: run-pair spread 1 Ir/op, Δ vs base 0.0%. The minimum fails because the noise is in both sizes, so the difference of the minima is not a lower bound. Pinning makes every hand-off depend on wakeup preemption.
On fede, current main is +1.2% vs v0.8.0 on
threads_request_seq(after #284, #287 and #282) and between −0.9% and −8.8% on the async benchmarks (#296). A full gate takes about 85 s on fede; on CI the release binary now comes from the cache.Docs:
docs/TESTING.md(Performance gate) andbenchmarks/README.md. CHANGELOG bullet added.Model: no change because this PR changes CI and the benchmark script only, no modelled runtime behaviour.