diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index d8734b0901..14a0149f81 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -2,6 +2,12 @@ +## 🔬 Oracle Coverage + + + + + ## ✅ Checklist - [ ] I have tested this code locally with `pnpm test`. diff --git a/AGENTS.md b/AGENTS.md index 5449b360d3..7518574818 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -475,8 +475,9 @@ applicable oracle coverage. ### Always Add Tests for Bugs -**Key Principle:** Reproduce a bug in a test before fixing it. Prefer extending -an oracle as described below over adding an isolated unit test. This ensures: +**Key Principle:** Reproduce a bug in a test before fixing it, then follow the +[applicable oracle coverage requirements](#require-applicable-oracle-coverage). +The evidence must show: - The bug is actually fixed - The bug doesn't regress in the future @@ -486,11 +487,14 @@ an oracle as described below over adding an isolated unit test. This ensures: ```typescript // Found a bug with fetchSnapshot resolving after up-to-date message -// Should add a test: +// First isolate the failure in a focused regression: test('ignores snapshot that resolves after up-to-date message', async () => { // Reproduce the corner case // Verify it's handled correctly }) + +// Then extend the applicable primary oracle so the law is exercised across +// nearby legal histories. ``` ### Treat Every Review Bug as a Test Gap @@ -519,7 +523,15 @@ intended checkpoint, then passes with the fix. At closeout, state what the evidence covers. Record remaining in-scope histories or paths in the oracle coverage map with an owner and needed witness. A reachable in-scope counterexample keeps the class open. Passing random runs -is not a universal proof. +is not a universal proof. A bug-fix pull request is not ready while a known +in-scope part of the declared bug class remains open. + +Every bug-fix pull request must show how it evaluated and closed the declared +bug class. State the violated law and its authority. Name the legal histories, +production paths, and public observation checkpoints that the oracle exercises. +Include the original or hostile-mutant RED result and the repaired GREEN result. +State any remaining limits. A passing reproduction for one trace does not +satisfy this requirement. ### Keep Oracles Independent @@ -544,19 +556,37 @@ that an oracle. Both copies can encode the same wrong assumption. the oracle's reference semantics. More generated traces cannot repair a wrong reference model. -### Prefer Oracle Coverage Over Isolated Regressions +### Require Applicable Oracle Coverage An oracle that checks general laws across generated states and histories is a -stronger form of coverage than a unit test for one specific example. Prefer -extending an existing oracle when it can cover the behavior. Add the missing -model rule, generator dimension, state transition, or observable assertion; -adding more pinned examples alone does not generalize the oracle. - -Use a focused regression to isolate and shrink a failure, then keep it as a -replay example for the broader oracle where possible. Verify that the expanded -oracle fails without the fix and passes with it. Keep valuable unit tests, but -do not treat them as a substitute for applicable oracle coverage. If an oracle -is not practical for the behavior, explain why a focused test is sufficient. +stronger form of coverage than a unit test for one specific example. Every +production change that adds, removes, or changes observable behavior must +extend the primary oracle when that oracle can express the governing law. Add +the missing model rule, generator dimension, state transition, or observable +assertion; adding more pinned examples alone does not generalize the oracle. Do +not submit only an isolated unit test for behavior owned by an applicable +oracle. An internal refactor does not require new oracle coverage when existing +applicable oracles already exercise the unchanged contract. + +Applicability follows the contract ownership recorded in +`docs/contributing/oracle-coverage.md`, not the limits of the owner's current +histories, production paths, or observations. A missing history, path, or +observation is a coverage gap in the primary owner. Add a separate oracle owner +only when the boundary requires a different independent judgment. Add a +receiving witness when a claim crosses an adapter, framework, provider, or other +integration boundary. + +For a bug fix, use a focused regression to isolate and shrink the failure, then +keep it as a replay example for the broader oracle. Verify that the expanded +oracle fails without the fix and passes with it. Keep valuable unit tests as +supplemental coverage, but do not treat them as a substitute for applicable +oracle coverage. If no existing oracle applies, create one when the behavior +admits a practical independent model. For a change that is not a bug fix, a +focused test alone is acceptable only when an oracle is not practical for the +behavior. The pull request must identify the relevant coverage-map entry and +explain why extending its owner, creating a new independent oracle, or adding a +receiving witness is not practical. Bug fixes cannot use this focused-test-only +exception. They must provide evidence that closes the entire declared bug class. ### Name Oracle Files for Discovery diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index cc6f65a9e7..23669fff9c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -54,6 +54,21 @@ If you have been assigned to fix an issue or develop a new feature, please follo ``` - Implement your changes and tests in the relevant package or example. +- For a production change that adds, removes, or changes observable behavior, + extend the primary oracle when it can express the governing law. A focused + unit test can supplement the oracle but cannot replace it. Internal refactors + need no new oracle coverage when existing oracles exercise the unchanged + contract. For a change that is not a bug fix, a focused test alone is + acceptable only when an oracle is not practical. Identify the relevant + coverage-map entry and explain why you cannot extend its owner, create an + independent oracle, or add a receiving witness. See the + [oracle coverage map](docs/contributing/oracle-coverage.md) and + [oracle test guide](docs/contributing/oracle-tests.md). +- For a bug fix, state how the pull request evaluated and closed the declared + bug class. Include the violated law, exercised histories, production paths, + observation checkpoints, RED/GREEN evidence, and remaining limits. A passing + reproduction for one trace is not sufficient. The focused-test-only exception + does not apply to bug fixes. Resolve known in-scope gaps before submitting. - Document your changes in the appropriate doc page. - Git stage your required changes and commit them. - Submit a PR for review.