Repository navigation
docs: Explain how to read CI benchmark bot comments - #4225
Conversation
Co-authored-by: Nathaniel Tucker <me@ntucker.me>
|
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4225 +/- ##
=======================================
Coverage 98.10% 98.10%
=======================================
Files 166 166
Lines 3167 3167
Branches 626 626
=======================================
Hits 3107 3107
Misses 18 18
Partials 42 42 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Co-authored-by: Nathaniel Tucker <me@ntucker.me>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Motivation
The "Benchmark React", "Benchmark", and "Benchmark Spread" PR comments compare one GitHub runner against the last stored master run on a different runner. Runner speed alone can shift a whole run, so these comments read like regressions when nothing changed. For example, #4097 measured 1.00 in a same-machine A/B. The existing variance thresholds assume same-machine runs.
Solution
Docs only. No workflow changes and no changeset, since nothing under
packages/*changes.examples/benchmark-react/README.md: a new "Reading the CI comment" section, the only full write-up. It has a table that maps what the comment shows to what to do (treat it as runner noise, or run an A/B), the local A/B commands for React and Node, and a note that the path filter can leave the stored baseline stale.examples/benchmark/README.mdand.cursor/rules/benchmarking.mdc: one-line pointers to that section..claude/rules/benchmarking.mdis regenerated withyarn build:agent-rules.