Skip to content

docs: Explain how to read CI benchmark bot comments - #4225

Merged
ntucker merged 2 commits into
masterfrom
cursor/benchmark-bot-reading-notes-3b65
Oct 6, 2026
Merged

ntucker merged 2 commits into
masterfrom
cursor/benchmark-bot-reading-notes-3b65

Conversation

@ntucker

@ntucker ntucker commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

The "Benchmark React", "Benchmark", and "Benchmark Spread" PR comments compare one GitHub runner against the last stored master run on a different runner. Runner speed alone can shift a whole run, so these comments read like regressions when nothing changed. For example, #4097 measured 1.00 in a same-machine A/B. The existing variance thresholds assume same-machine runs.

Solution

Docs only. No workflow changes and no changeset, since nothing under packages/* changes.

  • examples/benchmark-react/README.md: a new "Reading the CI comment" section, the only full write-up. It has a table that maps what the comment shows to what to do (treat it as runner noise, or run an A/B), the local A/B commands for React and Node, and a note that the path filter can leave the stored baseline stale.
  • examples/benchmark/README.md and .cursor/rules/benchmarking.mdc: one-line pointers to that section. .claude/rules/benchmarking.md is regenerated with yarn build:agent-rules.
Open in Web Open in Cursor 

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
@changeset-bot

changeset-bot Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 72b362b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@vercel

vercel Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs-site Ignored Ignored Preview Oct 6, 2026 12:26am UTC

Request Review

@codecov

codecov Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.10%. Comparing base (599c3dc) to head (72b362b).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #4225   +/-   ##
=======================================
  Coverage   98.10%   98.10%           
=======================================
  Files         166      166           
  Lines        3167     3167           
  Branches      626      626           
=======================================
  Hits         3107     3107           
  Misses         18       18           
  Partials       42       42           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
@ntucker
ntucker marked this pull request as ready for review October 6, 2026 00:35
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@ntucker
ntucker merged commit 0eece91 into master Oct 6, 2026
23 checks passed
@ntucker
ntucker deleted the cursor/benchmark-bot-reading-notes-3b65 branch October 6, 2026 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants