[prrao87]

Why

The contributing notes asked every code change for before-and-after numbers from both benchmark suites, but didn't explain how to get them. The suite runners pin lance-csr to a PyPI release, so running ./run_benchmarks.sh as it is times that release, not the contributor's branch. Contributors and their coding agents had nothing to follow, and nothing explained why we ask for a rationale in the first place. This PR turns the contributing guide into one that asks for the reasons behind a change and shows, step by step, how to produce the evidence.

What changed

• The contributing guide now opens with the questions every PR should answer (the problem, who hits it, the evidence, why it belongs here, what it costs, what else was tried). A table maps each kind of change to the evidence it needs.
• It describes a benchmark harness that builds release wheels from main and from the branch, and runs each suite's lance-csr runner against each wheel with uv run --with, in both memory and lazy modes. Runs alternate between main and the branch, are saved with pytest-benchmark and compared with pytest-benchmark compare. The run helper stops if it can't find the wheel, so it can't quietly time the pinned release instead.
• It sets realistic expectations for noise: a query's time can move by 15 to 20% between runs, in either direction. Changes in that range don't count as wins or regressions. If several queries get slower by more than that, the contributor investigates and explains why. There is no automatic rejection threshold.
• It includes a brief you can paste into a coding agent. The brief tells the agent not to touch queries, expected answers or runner settings, and to report every query.
• The README section is now a short summary that links to the guide, and the PR template asks for the rationale and has a per-query results table. The agent guidelines point agents to the harness before they claim a speedup. The docs site's contributing page no longer says that micro-benchmark numbers are enough on their own.

Validation

• Tried the wheel swap in a scratch copy of the 30-query runner, using stub wheels: uv run --frozen --with <wheel> loaded the named wheel even when both wheels had the same version, picked up a wheel rebuilt in place, and a plain uv run still loaded the pinned 0.2.0.
• Ran the guide's run and compare snippets under bash and zsh with a fake uv. Arguments, storage paths and the missing-wheel guard all behaved the same in both shells.
• uvx typos@1.26.0 . and uv run --project docs --locked zensical build --strict passed locally.
• Ran the whole workflow end to end with real release wheels, using the v0.2.0 tag as main and v0.3.1 as the branch: two release builds, the import check, 16 alternating runs across both suites and modes (all passed their result checks), and the compare step. Two bugs turned up and are fixed: pytest-benchmark compare rejected an absolute glob, so the guide now passes --storage with a relative glob; and it shortens run names to 12 characters, so runs are now saved as main-memory rather than memory-main, keeping main and branch visible.

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

Deploying lance-csr with    Cloudflare Pages

Latest commit:	a8f81d3
Status:	✅  Deploy successful!
Preview URL:	https://2a85e410.lance-csr.pages.dev
Branch Preview URL:	https://docs-contributing-benchmark.lance-csr.pages.dev

View logs