From 08462df2439f33b5189de730a80d1a09840ac414 Mon Sep 17 00:00:00 2001 From: Isaac Date: Fri, 16 Oct 2026 08:47:40 -0700 Subject: [PATCH 1/5] docs: require applicable oracle coverage --- AGENTS.md | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5449b360d3..4a04fedeb9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -475,8 +475,11 @@ applicable oracle coverage. ### Always Add Tests for Bugs -**Key Principle:** Reproduce a bug in a test before fixing it. Prefer extending -an oracle as described below over adding an isolated unit test. This ensures: +**Key Principle:** Reproduce a bug in a test before fixing it. When an existing +oracle can express the violated law, extending that oracle is required. An +isolated unit test may supplement the oracle, but it does not replace it. If no +oracle applies, add one when an independent model is practical; otherwise, +explain in the pull request why a focused test is sufficient. This ensures: - The bug is actually fixed - The bug doesn't regress in the future @@ -544,19 +547,23 @@ that an oracle. Both copies can encode the same wrong assumption. the oracle's reference semantics. More generated traces cannot repair a wrong reference model. -### Prefer Oracle Coverage Over Isolated Regressions +### Require Applicable Oracle Coverage An oracle that checks general laws across generated states and histories is a -stronger form of coverage than a unit test for one specific example. Prefer -extending an existing oracle when it can cover the behavior. Add the missing -model rule, generator dimension, state transition, or observable assertion; -adding more pinned examples alone does not generalize the oracle. +stronger form of coverage than a unit test for one specific example. Every bug +fix must extend the primary oracle when that oracle can express the violated +law. Add the missing model rule, generator dimension, state transition, or +observable assertion; adding more pinned examples alone does not generalize the +oracle. Do not submit only an isolated unit test for behavior owned by an +applicable oracle. Use a focused regression to isolate and shrink a failure, then keep it as a -replay example for the broader oracle where possible. Verify that the expanded -oracle fails without the fix and passes with it. Keep valuable unit tests, but -do not treat them as a substitute for applicable oracle coverage. If an oracle -is not practical for the behavior, explain why a focused test is sufficient. +replay example for the broader oracle. Verify that the expanded oracle fails +without the fix and passes with it. Keep valuable unit tests as supplemental +coverage, but do not treat them as a substitute for applicable oracle coverage. +If no existing oracle applies, create one when the behavior admits a practical +independent model. A focused test alone is acceptable only when an oracle is not +practical for the behavior; explain that exception in the pull request. ### Name Oracle Files for Discovery From 82ce1fa2b1ca7ec5f633e6fb8852b04969bb38a5 Mon Sep 17 00:00:00 2001 From: Isaac Date: Fri, 16 Oct 2026 09:02:43 -0700 Subject: [PATCH 2/5] docs: clarify oracle coverage requirements --- .github/pull_request_template.md | 4 +++ AGENTS.md | 53 ++++++++++++++++++++------------ CONTRIBUTING.md | 10 ++++++ 3 files changed, 47 insertions(+), 20 deletions(-) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index d8734b0901..db649c4459 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -2,6 +2,10 @@ +## 🔬 Oracle Coverage + + + ## ✅ Checklist - [ ] I have tested this code locally with `pnpm test`. diff --git a/AGENTS.md b/AGENTS.md index 4a04fedeb9..f2f0cf39c1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -475,11 +475,9 @@ applicable oracle coverage. ### Always Add Tests for Bugs -**Key Principle:** Reproduce a bug in a test before fixing it. When an existing -oracle can express the violated law, extending that oracle is required. An -isolated unit test may supplement the oracle, but it does not replace it. If no -oracle applies, add one when an independent model is practical; otherwise, -explain in the pull request why a focused test is sufficient. This ensures: +**Key Principle:** Reproduce a bug in a test before fixing it, then follow the +[applicable oracle coverage requirements](#require-applicable-oracle-coverage). +The evidence must show: - The bug is actually fixed - The bug doesn't regress in the future @@ -489,11 +487,14 @@ explain in the pull request why a focused test is sufficient. This ensures: ```typescript // Found a bug with fetchSnapshot resolving after up-to-date message -// Should add a test: +// First isolate the failure in a focused regression: test('ignores snapshot that resolves after up-to-date message', async () => { // Reproduce the corner case // Verify it's handled correctly }) + +// Then extend the applicable primary oracle so the law is exercised across +// nearby legal histories. ``` ### Treat Every Review Bug as a Test Gap @@ -550,20 +551,32 @@ that an oracle. Both copies can encode the same wrong assumption. ### Require Applicable Oracle Coverage An oracle that checks general laws across generated states and histories is a -stronger form of coverage than a unit test for one specific example. Every bug -fix must extend the primary oracle when that oracle can express the violated -law. Add the missing model rule, generator dimension, state transition, or -observable assertion; adding more pinned examples alone does not generalize the -oracle. Do not submit only an isolated unit test for behavior owned by an -applicable oracle. - -Use a focused regression to isolate and shrink a failure, then keep it as a -replay example for the broader oracle. Verify that the expanded oracle fails -without the fix and passes with it. Keep valuable unit tests as supplemental -coverage, but do not treat them as a substitute for applicable oracle coverage. -If no existing oracle applies, create one when the behavior admits a practical -independent model. A focused test alone is acceptable only when an oracle is not -practical for the behavior; explain that exception in the pull request. +stronger form of coverage than a unit test for one specific example. Every +production change that adds, removes, or changes observable behavior must +extend the primary oracle when that oracle can express the governing law. Add +the missing model rule, generator dimension, state transition, or observable +assertion; adding more pinned examples alone does not generalize the oracle. Do +not submit only an isolated unit test for behavior owned by an applicable +oracle. An internal refactor does not require new oracle coverage when existing +applicable oracles already exercise the unchanged contract. + +Applicability follows the contract ownership recorded in +`docs/contributing/oracle-coverage.md`, not the limits of the owner's current +histories, production paths, or observations. A missing history, path, or +observation is a coverage gap in the primary owner. Add a separate oracle owner +only when the boundary requires a different independent judgment. Add a +receiving witness when a claim crosses an adapter, framework, provider, or other +integration boundary. + +For a bug fix, use a focused regression to isolate and shrink the failure, then +keep it as a replay example for the broader oracle. Verify that the expanded +oracle fails without the fix and passes with it. Keep valuable unit tests as +supplemental coverage, but do not treat them as a substitute for applicable +oracle coverage. If no existing oracle applies, create one when the behavior +admits a practical independent model. A focused test alone is acceptable only +when an oracle is not practical for the behavior. The pull request must identify +the relevant coverage-map entry and explain why extending its owner, creating a +new independent oracle, or adding a receiving witness is not practical. ### Name Oracle Files for Discovery diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index cc6f65a9e7..dd495514ca 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -54,6 +54,16 @@ If you have been assigned to fix an issue or develop a new feature, please follo ``` - Implement your changes and tests in the relevant package or example. +- For a production change that adds, removes, or changes observable behavior, + extend the primary oracle when it can express the governing law. A focused + unit test can supplement the oracle but cannot replace it. Internal refactors + need no new oracle coverage when existing oracles exercise the unchanged + contract. If an oracle is not practical, explain why the focused test is + sufficient in the pull request. Identify the relevant coverage-map entry and + explain why you cannot extend its owner, create an independent oracle, or add + a receiving witness. See the + [oracle coverage map](docs/contributing/oracle-coverage.md) and + [oracle test guide](docs/contributing/oracle-tests.md). - Document your changes in the appropriate doc page. - Git stage your required changes and commit them. - Submit a PR for review. From 13925e62f83fd1bdba5015046535c8c381c598db Mon Sep 17 00:00:00 2001 From: Isaac Date: Fri, 16 Oct 2026 09:05:51 -0700 Subject: [PATCH 3/5] docs: clarify focused-test evidence limits --- AGENTS.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f2f0cf39c1..8a3ec055ed 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -523,7 +523,9 @@ intended checkpoint, then passes with the fix. At closeout, state what the evidence covers. Record remaining in-scope histories or paths in the oracle coverage map with an owner and needed witness. A reachable in-scope counterexample keeps the class open. Passing random runs -is not a universal proof. +is not a universal proof. When the focused-test exception applies, do not claim +that the bug class is closed. State which trace the focused test proves and +record the remaining oracle gap. ### Keep Oracles Independent @@ -576,7 +578,9 @@ oracle coverage. If no existing oracle applies, create one when the behavior admits a practical independent model. A focused test alone is acceptable only when an oracle is not practical for the behavior. The pull request must identify the relevant coverage-map entry and explain why extending its owner, creating a -new independent oracle, or adding a receiving witness is not practical. +new independent oracle, or adding a receiving witness is not practical. This +exception permits the specific repair, but a focused test alone does not prove +that the broader bug class is closed. ### Name Oracle Files for Discovery From cba359fe78cf0b7a278c4bc8eb0ebbeeb95aec0a Mon Sep 17 00:00:00 2001 From: Isaac Date: Fri, 16 Oct 2026 09:06:43 -0700 Subject: [PATCH 4/5] docs: require bug-class closure evidence --- .github/pull_request_template.md | 2 ++ AGENTS.md | 7 +++++++ CONTRIBUTING.md | 4 ++++ 3 files changed, 13 insertions(+) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index db649c4459..d317f4be1c 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -6,6 +6,8 @@ + + ## ✅ Checklist - [ ] I have tested this code locally with `pnpm test`. diff --git a/AGENTS.md b/AGENTS.md index 8a3ec055ed..af4f576880 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -527,6 +527,13 @@ is not a universal proof. When the focused-test exception applies, do not claim that the bug class is closed. State which trace the focused test proves and record the remaining oracle gap. +Every bug-fix pull request must show how it evaluated and closed the declared +bug class. State the violated law and its authority. Name the legal histories, +production paths, and public observation checkpoints that the oracle exercises. +Include the original or hostile-mutant RED result and the repaired GREEN result. +State any remaining limits. A passing reproduction for one trace does not +satisfy this requirement. + ### Keep Oracles Independent An oracle is useful only when its expected result comes from a source independent diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dd495514ca..9ae41f63aa 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -64,6 +64,10 @@ If you have been assigned to fix an issue or develop a new feature, please follo a receiving witness. See the [oracle coverage map](docs/contributing/oracle-coverage.md) and [oracle test guide](docs/contributing/oracle-tests.md). +- For a bug fix, state how the pull request evaluated and closed the declared + bug class. Include the violated law, exercised histories, production paths, + observation checkpoints, RED/GREEN evidence, and remaining limits. A passing + reproduction for one trace is not sufficient. - Document your changes in the appropriate doc page. - Git stage your required changes and commit them. - Submit a PR for review. From 250d3959ccfe82b4dd0220c22d5e100eab761895 Mon Sep 17 00:00:00 2001 From: Isaac Date: Fri, 16 Oct 2026 09:30:41 -0700 Subject: [PATCH 5/5] docs: require complete bug-class evidence for fixes --- .github/pull_request_template.md | 4 ++-- AGENTS.md | 17 ++++++++--------- CONTRIBUTING.md | 11 ++++++----- 3 files changed, 16 insertions(+), 16 deletions(-) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index d317f4be1c..14a0149f81 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -4,9 +4,9 @@ ## 🔬 Oracle Coverage - + - + ## ✅ Checklist diff --git a/AGENTS.md b/AGENTS.md index af4f576880..7518574818 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -523,9 +523,8 @@ intended checkpoint, then passes with the fix. At closeout, state what the evidence covers. Record remaining in-scope histories or paths in the oracle coverage map with an owner and needed witness. A reachable in-scope counterexample keeps the class open. Passing random runs -is not a universal proof. When the focused-test exception applies, do not claim -that the bug class is closed. State which trace the focused test proves and -record the remaining oracle gap. +is not a universal proof. A bug-fix pull request is not ready while a known +in-scope part of the declared bug class remains open. Every bug-fix pull request must show how it evaluated and closed the declared bug class. State the violated law and its authority. Name the legal histories, @@ -582,12 +581,12 @@ keep it as a replay example for the broader oracle. Verify that the expanded oracle fails without the fix and passes with it. Keep valuable unit tests as supplemental coverage, but do not treat them as a substitute for applicable oracle coverage. If no existing oracle applies, create one when the behavior -admits a practical independent model. A focused test alone is acceptable only -when an oracle is not practical for the behavior. The pull request must identify -the relevant coverage-map entry and explain why extending its owner, creating a -new independent oracle, or adding a receiving witness is not practical. This -exception permits the specific repair, but a focused test alone does not prove -that the broader bug class is closed. +admits a practical independent model. For a change that is not a bug fix, a +focused test alone is acceptable only when an oracle is not practical for the +behavior. The pull request must identify the relevant coverage-map entry and +explain why extending its owner, creating a new independent oracle, or adding a +receiving witness is not practical. Bug fixes cannot use this focused-test-only +exception. They must provide evidence that closes the entire declared bug class. ### Name Oracle Files for Discovery diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9ae41f63aa..23669fff9c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -58,16 +58,17 @@ If you have been assigned to fix an issue or develop a new feature, please follo extend the primary oracle when it can express the governing law. A focused unit test can supplement the oracle but cannot replace it. Internal refactors need no new oracle coverage when existing oracles exercise the unchanged - contract. If an oracle is not practical, explain why the focused test is - sufficient in the pull request. Identify the relevant coverage-map entry and - explain why you cannot extend its owner, create an independent oracle, or add - a receiving witness. See the + contract. For a change that is not a bug fix, a focused test alone is + acceptable only when an oracle is not practical. Identify the relevant + coverage-map entry and explain why you cannot extend its owner, create an + independent oracle, or add a receiving witness. See the [oracle coverage map](docs/contributing/oracle-coverage.md) and [oracle test guide](docs/contributing/oracle-tests.md). - For a bug fix, state how the pull request evaluated and closed the declared bug class. Include the violated law, exercised histories, production paths, observation checkpoints, RED/GREEN evidence, and remaining limits. A passing - reproduction for one trace is not sufficient. + reproduction for one trace is not sufficient. The focused-test-only exception + does not apply to bug fixes. Resolve known in-scope gaps before submitting. - Document your changes in the appropriate doc page. - Git stage your required changes and commit them. - Submit a PR for review.