From cd1e4e278262fcf8bc17f0db66eb07b5f873ade2 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 5 Oct 2026 09:45:51 +0000 Subject: [PATCH 01/19] Acceptance process: make the cohort runnable and markable mechanically Reader isolation and the packet, a marking section (question 2, traceable misses, unanswered questions, the same passage, delegated marking), thresholds for any cohort size, row precedence, the record, the reviewer and agent-surface rebuild in CONTRIBUTING.md, and how the process itself changes. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 65 +++++++++++++++++++++++++++++++++------ CHANGELOG.md | 22 +++++++++++++ CONTRIBUTING.md | 18 +++++++---- acceptance/README.md | 15 ++++----- acceptance/runs/README.md | 8 ++--- 5 files changed, 101 insertions(+), 27 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 31c8df0..f0395b3 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -11,8 +11,15 @@ test is run with a low-strength model so that clarity is measured at the floor, - **At least five independent readers**, each a fresh instance of a low-strength model (the smallest current model in its family, for example Haiku), with no shared context and no access to the change history, the critique findings or each other. -- Each reader receives **only the changed document or documents** and the fixed question set - below. Nothing else is read first. +- **Each reader is a separate process with no tools and no context but its packet**, started + outside any repository so that no project instruction or settings file reaches it: for + example `claude -p` with every tool disabled, run from an empty directory. A reader that + shares the editor's context, such as an agent dispatched from the editor's own session, is not + a cold reader, and its answers do not count towards a run. +- Each reader receives **only its packet**: the changed document, the document's question set + and the fixture the questions name, each in full under its file name, in the reader's prompt. + The question set carries the instruction to the reader, and nothing is added around it. + Every reader in a run receives the same packet. Nothing else is read first. - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. @@ -61,23 +68,49 @@ about any of them is the editor's decision. An earlier version of this test aske what would make a clause clearer and for a clarity rating; both were removed on the author's instruction for that reason, and neither is reinstated. +## Marking + +The editor marks each answer sheet against the document's key, or delegates the marking to a +separate marker that has seen neither the drafting nor the review, and receives only the +document at the commit the readers read, its key, this file and the answer sheets. A delegated +marker's verdict stands. Each sheet is marked question by question, as follows. + +- **Questions 1 and 3** are marked as the key states for each item. +- **Question 2.** An answer is correct when its sentence states what the key's answer states, + including anything the key says an answer is wrong to omit, and its quotation is a sentence of + the document that supports it. The quotation need not be the one the key gives. +- **A miss traceable to the text** is one whose reasoning quotes the document. A miss whose + reasoning quotes nothing, or which misreads the fixture, is the reader's. +- **An unanswered question.** A keyed question left unanswered is a miss, and is not traceable + to the text. Question 4 left unanswered counts as no passage quoted. +- **The same passage.** Two quotations under question 4 are of the same passage when they share + a sentence of the document. Quotations of different sentences are different passages, even + within one paragraph or clause. + +Where a mark needs a judgement these rules do not settle, the marker makes it, and the record +states the judgement and why. + ## Pass criteria -The change passes when all of the following hold on a single run: +The change passes when all of the following hold on a single run. With a cohort of five, "every +reader" is five of five and "all but one" is four of five. | Criterion | Threshold | | ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | -| Load-bearing statements (question 2) | **5 of 5** readers correct on every one | -| Restatement and application (questions 1 and 3) | **At least 4 of 5** readers correct on each keyed item | +| Load-bearing statements (question 2) | **Every** reader correct on every one | +| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | | A wrong answer traceable to the text | **None.** A miss whose reasoning quotes the document is a finding, whatever the count | | A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it | -| Anything from one reader only | Recorded; applied at the editor's discretion; does not block | +| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | + +The rows are applied in order, so a miss traceable to the text blocks even when only one reader +made it; the last row covers only what no earlier row has caught. The asymmetry is deliberate. A load-bearing statement is the sentence the whole document exists to make, and one reader in five getting it wrong means one agent in five will act on the wrong reading; unanimity is the only acceptable result there. Everywhere else a single divergent reader -is more likely to be the reader than the text, so four of five is tolerated, but a miss that the -reader can justify from the document is always the document's fault. +is more likely to be the reader than the text, so one such reader is tolerated, but a miss that +the reader can justify from the document is always the document's fault. ## The bar only moves up @@ -107,8 +140,9 @@ A wording change after the run means a fresh run; readers are never reused. Each run has a file under `acceptance/runs/`, named `NNN-.md` with `NNN` the next number in the one sequence shared by the three documents, stating: the document, the pull request and the -commit read, the cohort size and model, the run number in that pull request, the marks per -reader per question, the findings applied, and whether it passed. The changelog line for the +commit read, the cohort size and model, how the readers were isolated, the run number in that +pull request, who marked, the marks per reader per question, any judgement made in marking, the +findings applied, and whether it passed. The changelog line for the change names the file. CI refuses a specification change whose pull request adds no run file; it checks for at least one, and a pull request that changes two documents adds a run for each. @@ -116,3 +150,14 @@ Each published version's changelog entry carries the lines of the changes it con version's cohort record is the set of runs those lines name. A version with no such line has not been accepted and MUST NOT be described as published. The question sets and answer keys live under `acceptance/` in this repository so that a run can be repeated and compared. + +## Changes to this process + +This file, `acceptance/README.md` and the steps in `CONTRIBUTING.md` are not specifications, and +a change to them is not tested by a cohort: the cohort measures whether a low-strength reader can +follow a specification, and the process is followed by the editor. Such a change names the runs +or the case that showed the need for it, has the same independent review as any other, and is +recorded by a line under `Unreleased` in the changelog's entry for the acceptance process. It is +bound by the section on the bar: it may raise a criterion, never relax one. A change to a +question set, key or fixture is not a change to this process; `acceptance/README.md` governs +those. diff --git a/CHANGELOG.md b/CHANGELOG.md index f5f8122..05324bb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -133,3 +133,25 @@ First published version, the draft label dropped on the same day as the method s custom rules, a single-rule harness, identifier resolution, a derived listing of active defences, the project record, self-application, and the conformance definition in section 11. Published before the acceptance test in `ACCEPTANCE.md` existed; 0.2.0 is the first version to pass it. + +## Acceptance process (ACCEPTANCE.md) + +Not a specification and not versioned. Changes are recorded here as `ACCEPTANCE.md` describes, +and need no cohort. + +### Unreleased + +- The cohort can be run and marked without judgement calls that the rules leave open. Each + reader is a separate process with no tools and no context but its packet, and the packet is + the document, its question set and the fixture the questions name, which `ACCEPTANCE.md` and + `acceptance/README.md` previously disagreed on. A new marking section settles what a correct + load-bearing answer needs, what makes a miss traceable to the text, an unanswered question and + "the same passage", and lets the editor delegate marking to a separate marker whose verdict + stands. Thresholds read "every reader" and "all but one", the same as five of five and four of + five for a cohort of five, and the pass criteria rows apply in order. The run record states + how readers were isolated and who marked. `CONTRIBUTING.md` names the reviewer, the + agent-facing rebuild after an Appendix A change, and how a change to this process is made. + Prompted by an agent-driven run of the process that could not follow it mechanically; the + rulings in [run 16](acceptance/runs/016-deferral-clause-run-1.md), + [run 17](acceptance/runs/017-deferral-clause-run-2.md) and + [run 18](acceptance/runs/018-deferral-clause-run-3.md) are written in. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fa1743e..bc249de 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,7 +29,8 @@ review. It is a patch version when released. SHOULD or MAY in a numbered section or appendix. It needs everything a clarity edit needs, plus the five tests in SCOPE.md answered in the pull request, and it is at least a minor version when released. `tools/changes.py` tells the two apart mechanically and CI runs it on every pull -request. +request. A change to the acceptance process itself is neither, and ACCEPTANCE.md says how it is +made. ## What a pull request contains @@ -39,6 +40,9 @@ request. request template lays them out. - The observed case that prompted it, for either kind. A clarity edit still has a reason: a reader who took the passage the wrong way. +- Where a specification's Appendix A changed, the agent-facing files rebuilt with + `python3 build-agent-surfaces.py` and committed with the change; CI runs it with `--check` + and fails when they are stale. - `python3 spec-qa.py` clean and the unit tests passing, which CI checks for you. A pull request is a suggestion, not a demand. The documents have one editor, who keeps ownership @@ -55,9 +59,10 @@ survives, the change is yours. the editor sets the pre-release header. 2. **Scope.** For an obligation, the editor confirms each SCOPE.md test. A failed test closes the pull request with a row in `DECLINED.md`. -3. **Review and rewrite.** An independent review of the diff against the document, then the - editor's rewrite, pushed to the pull request branch where the contributor allows it or to an - editor's branch that supersedes it. +3. **Review and rewrite.** An independent review of the diff against the whole document, by a + reviewer who did not draft the change, then the editor's rewrite, pushed to the pull request + branch where the contributor allows it or to an editor's branch that supersedes it. A review + finding applied after a cohort has run is a wording change, and a fresh run follows. 4. **Keys.** Where the change touches a sentence an acceptance key quotes, or adds a load-bearing obligation, the editor updates the key before the cohort runs, never after. 5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text. Its record is @@ -67,8 +72,9 @@ survives, the change is yours. version. A wording change after the run means a fresh run. 6. **Merge** into `next` by merge commit, and the branch is deleted. -Steps 2 to 5 are done by a person with a Claude Code session, not by CI, and they happen in -batches. A quiet pull request is not a declined one; a declined one says so. +Steps 2 to 5 are the editor's, not CI's, and they happen in batches. The editor may have an +agent draft or review, and may delegate marking as ACCEPTANCE.md allows, in which case the +marker's verdict stands; every other decision in these steps stays the editor's. A quiet pull request is not a declined one; a declined one says so. ## Versions and releases diff --git a/acceptance/README.md b/acceptance/README.md index 4cfb6ec..57a2880 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -14,13 +14,14 @@ the record of every run lives in `runs/`. ## How a run works 1. The editor identifies which documents the change touched. Only those are tested. -2. For each, five or more fresh low-strength readers each receive the document's current text, - its `QUESTIONS.md`, and the fixture the questions name. Nothing else: no key, no changelog, no - critique findings, no other reader's answers. -3. The editor marks each answer sheet against `KEY.md` and applies the pass criteria in - `ACCEPTANCE.md`. -4. The result is recorded as a file under [runs/](runs/), with cohort size, model, run number, - marks and the findings applied, and the change's `Unreleased` changelog line names that file. +2. For each, five or more fresh low-strength readers, each isolated as `ACCEPTANCE.md` describes, + each receive the document's current text, its `QUESTIONS.md`, and the fixture the questions + name. Nothing else: no key, no changelog, no critique findings, no other reader's answers. +3. The editor, or a separate marker the editor delegates to, marks each answer sheet against + `KEY.md` under the marking rules in `ACCEPTANCE.md` and applies its pass criteria. +4. The result is recorded as a file under [runs/](runs/), with cohort size, model, how the + readers were isolated, run number, who marked, marks and the findings applied, and the + change's `Unreleased` changelog line names that file. A run that fails is followed by a change to the text, never to the questions, and a fresh cohort. diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 4994f74..3435821 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -6,10 +6,10 @@ share. Runs 1 to 15 predate this directory and are summarised in the changelog e the toolchain specification; numbering here continues from 16. A record states, in this order: the document; the pull request and the commit the readers were -given; the cohort size and model; the run number within that pull request; the marks per reader -per question against the key; the passages quoted as confusing and by how many readers; the -findings applied to the text as a result; and the verdict against the criteria in -`ACCEPTANCE.md`. Answer sheets are not pasted in full; the marks are the record. +given; the cohort size and model; how the readers were isolated; the run number within that +pull request; who marked; the marks per reader per question against the key, with any judgement +made in marking; the passages quoted as confusing and by how many readers; the findings applied +to the text as a result; and the verdict against the criteria in `ACCEPTANCE.md`. Answer sheets are not pasted in full; the marks are the record. The `Unreleased` changelog line for the change names the record, and `tools/changes.py` refuses a specification change whose pull request adds no file here. From e7a0c5d6e6c5eb4e9866368f88c7c2c59f56c972 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 5 Oct 2026 09:56:07 +0000 Subject: [PATCH 02/19] Acceptance process: apply the first review One rule for a miss traceable to the text: the reasoning quotes the document, or the key says the miss counts; the key can add, never take away, and a quoting miss is traceable even where it misreads the fixture. Quoting is defined, question 2 answers must assert nothing the key marks wrong, unanswered keyed items count one miss each, and "the same passage" is settled for fragments, whole paragraphs and passages named unquoted. A delegated marker receives the question set and fixture too, and the editor supplies the record's facts. Every pass-criteria row must hold; the last covers only what the others do not. Answer sheets, the reader command and a delegated marker's instruction are kept with the record. The process files are named; review findings that change the document mean a fresh run, the step 5 header does not. The three record lists agree, lines are rewrapped, and the changelog names the case from the repository. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 92 ++++++++++++++++++++++++--------------- CHANGELOG.md | 34 +++++++++------ CONTRIBUTING.md | 10 +++-- acceptance/README.md | 6 +-- acceptance/runs/README.md | 14 +++--- 5 files changed, 95 insertions(+), 61 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index f0395b3..e0ba89a 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -17,9 +17,10 @@ test is run with a low-strength model so that clarity is measured at the floor, shares the editor's context, such as an agent dispatched from the editor's own session, is not a cold reader, and its answers do not count towards a run. - Each reader receives **only its packet**: the changed document, the document's question set - and the fixture the questions name, each in full under its file name, in the reader's prompt. - The question set carries the instruction to the reader, and nothing is added around it. - Every reader in a run receives the same packet. Nothing else is read first. + and the fixture the questions name, each in full in the reader's prompt and headed by its file + name. The question set carries the instruction to the reader; no preamble, wrapper or other + text is added beyond the file names. Every reader in a run receives the same packet. Nothing + else is read first. - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. @@ -44,9 +45,9 @@ document, so that marking is mechanical. Each reader answers all of them. from different sources; how many grades a project that ships tooling has. 3. **Application.** A fixture case supplied with the questions, graded clause by clause with - Yes, Partial or No and one line of reasoning. Keyed against the intended verdicts; a divergence - counts against the document only when the reader's reasoning shows the clause was read - differently, not when the fixture was misread. + Yes, Partial or No and one line of reasoning. Keyed against the intended verdicts; whether a + divergence counts against the document is settled by the rule for a miss traceable to the + text, under [Marking](#marking). 4. **Confusion.** Any passage the reader found confusing on its one reading, quoted with its clause number, and nothing else: not what would make it clearer, not a proposed change. @@ -71,21 +72,36 @@ instruction for that reason, and neither is reinstated. ## Marking The editor marks each answer sheet against the document's key, or delegates the marking to a -separate marker that has seen neither the drafting nor the review, and receives only the -document at the commit the readers read, its key, this file and the answer sheets. A delegated -marker's verdict stands. Each sheet is marked question by question, as follows. - +separate marker that has seen neither the drafting nor the review. Whoever marks is the marker +in this file. A delegated marker receives only these: the document at the commit the readers +read, its question set, the fixture the questions name and its key; this file and +`acceptance/runs/README.md`, with earlier run records of other pull requests for the record's +shape; the answer sheets; and, from the editor, the facts the record states that the sheets do +not, which are the pull request and the commit, the cohort size and model, the command the +readers were started with and the run number. The record states those facts as the editor +supplied them. A delegated marker's verdict stands. Each sheet is marked question by question, +as follows. + +- **Quoting.** A reader quotes the document when it gives words of the document verbatim, a + whole sentence or part of one. Naming a clause number, or paraphrasing, is not quoting. - **Questions 1 and 3** are marked as the key states for each item. - **Question 2.** An answer is correct when its sentence states what the key's answer states, - including anything the key says an answer is wrong to omit, and its quotation is a sentence of - the document that supports it. The quotation need not be the one the key gives. -- **A miss traceable to the text** is one whose reasoning quotes the document. A miss whose - reasoning quotes nothing, or which misreads the fixture, is the reader's. -- **An unanswered question.** A keyed question left unanswered is a miss, and is not traceable - to the text. Question 4 left unanswered counts as no passage quoted. -- **The same passage.** Two quotations under question 4 are of the same passage when they share - a sentence of the document. Quotations of different sentences are different passages, even - within one paragraph or clause. + including anything the key says an answer is wrong to omit, asserts nothing the key says makes + an answer wrong, and quotes the document in support. The quotation need not be the one the key + gives. +- **A miss traceable to the text** is a miss whose reasoning quotes the document, or a miss the + key says counts against the document. A miss whose reasoning quotes the document is traceable + even where it also misreads the fixture, and even where the key says that miss is the reader's + or counts only with a particular quotation: the key can add to the misses that count against + the document, never take one away. Every other miss is the reader's. +- **An unanswered question.** Each keyed item left unanswered is one miss, not traceable to the + text, so a sheet that leaves three items of question 3 blank has three misses. Question 4 left + unanswered counts as no passage quoted. +- **The same passage.** A quotation under question 4 covers every sentence of the document it + quotes any part of, and a passage named without a quotation, by clause number or description, + covers every sentence of what it names. Two passages are the same when they cover a sentence in + common. Passages with no sentence in common are different, even within one paragraph or + clause. Where a mark needs a judgement these rules do not settle, the marker makes it, and the record states the judgement and why. @@ -99,12 +115,13 @@ reader" is five of five and "all but one" is four of five. | ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | | Load-bearing statements (question 2) | **Every** reader correct on every one | | Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | -| A wrong answer traceable to the text | **None.** A miss whose reasoning quotes the document is a finding, whatever the count | +| A miss traceable to the text | **None.** Any such miss, as Marking defines it, is a finding, whatever the count | | A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it | | Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | -The rows are applied in order, so a miss traceable to the text blocks even when only one reader -made it; the last row covers only what no earlier row has caught. +Every row must hold. The first four apply whatever the count, and the last covers only what none +of them catches: a miss traceable to the text blocks even when only one reader made it, and the +second row's tolerance of one reader does not excuse it. The asymmetry is deliberate. A load-bearing statement is the sentence the whole document exists to make, and one reader in five getting it wrong means one agent in five will act on the wrong @@ -140,11 +157,15 @@ A wording change after the run means a fresh run; readers are never reused. Each run has a file under `acceptance/runs/`, named `NNN-.md` with `NNN` the next number in the one sequence shared by the three documents, stating: the document, the pull request and the -commit read, the cohort size and model, how the readers were isolated, the run number in that -pull request, who marked, the marks per reader per question, any judgement made in marking, the -findings applied, and whether it passed. The changelog line for the -change names the file. CI refuses a specification change whose pull request adds no run file; -it checks for at least one, and a pull request that changes two documents adds a run for each. +commit read, the cohort size and model, how the readers were isolated and the command they were +started with, the run number in that pull request, who marked and, for a delegated marker, how it +was started, the marks per reader per question, any judgement made in marking, the passages +quoted as confusing and by how many readers, the findings applied, and whether it passed. Each +reader's answer sheet is kept unedited beside the record, one file per reader in a directory +`NNN-/`, with the instruction a delegated marker was given, so that an owner can check the +isolation and the marks. The changelog line for the change names the file. CI refuses a +specification change whose pull request adds no run file; it checks for at least one, and a pull +request that changes two documents adds a run for each. Each published version's changelog entry carries the lines of the changes it contains, so a version's cohort record is the set of runs those lines name. A version with no such line has not @@ -153,11 +174,12 @@ under `acceptance/` in this repository so that a run can be repeated and compare ## Changes to this process -This file, `acceptance/README.md` and the steps in `CONTRIBUTING.md` are not specifications, and -a change to them is not tested by a cohort: the cohort measures whether a low-strength reader can -follow a specification, and the process is followed by the editor. Such a change names the runs -or the case that showed the need for it, has the same independent review as any other, and is -recorded by a line under `Unreleased` in the changelog's entry for the acceptance process. It is -bound by the section on the bar: it may raise a criterion, never relax one. A change to a -question set, key or fixture is not a change to this process; `acceptance/README.md` governs -those. +This file, `acceptance/README.md`, `acceptance/runs/README.md` and `CONTRIBUTING.md` are the +process files, and a change to any part of them is a change to this process. They are not +specifications, and a change to them is not tested by a cohort: the cohort measures whether a +low-strength reader can follow a specification, and the process is followed by the editor. Such +a change names the runs or the case that showed the need for it, has the same independent review +as any other, and is recorded by a line under `Unreleased` in the changelog's entry for the +acceptance process. It is bound by the section on the bar: it may raise a criterion, never relax +one. A change to a question set, key or fixture is not a change to this process; +`acceptance/README.md` governs those. diff --git a/CHANGELOG.md b/CHANGELOG.md index 05324bb..7df15d9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -134,24 +134,30 @@ custom rules, a single-rule harness, identifier resolution, a derived listing of the project record, self-application, and the conformance definition in section 11. Published before the acceptance test in `ACCEPTANCE.md` existed; 0.2.0 is the first version to pass it. -## Acceptance process (ACCEPTANCE.md) +## Acceptance process -Not a specification and not versioned. Changes are recorded here as `ACCEPTANCE.md` describes, -and need no cohort. +Not a specification and not versioned. Changes to the process files `ACCEPTANCE.md` names are +recorded here as it describes, and need no cohort. ### Unreleased - The cohort can be run and marked without judgement calls that the rules leave open. Each reader is a separate process with no tools and no context but its packet, and the packet is - the document, its question set and the fixture the questions name, which `ACCEPTANCE.md` and - `acceptance/README.md` previously disagreed on. A new marking section settles what a correct - load-bearing answer needs, what makes a miss traceable to the text, an unanswered question and - "the same passage", and lets the editor delegate marking to a separate marker whose verdict - stands. Thresholds read "every reader" and "all but one", the same as five of five and four of - five for a cohort of five, and the pass criteria rows apply in order. The run record states - how readers were isolated and who marked. `CONTRIBUTING.md` names the reviewer, the - agent-facing rebuild after an Appendix A change, and how a change to this process is made. - Prompted by an agent-driven run of the process that could not follow it mechanically; the - rulings in [run 16](acceptance/runs/016-deferral-clause-run-1.md), + the document, its question set and the fixture the questions name, with nothing added around + them. A new marking section says what a delegated marker receives and that its verdict stands, + what counts as quoting, what a correct load-bearing answer needs, which misses are traceable + to the text and how the key's statements bear on that, how unanswered items count, and when + two confusing passages are the same. Thresholds read "every reader" and "all but one", the + same as five of five and four of five for a cohort of five; every pass-criteria row must hold, + and the last covers only what the others do not. The run record states how readers were + isolated and who marked, and the answer sheets are kept beside it. `CONTRIBUTING.md` requires + a reviewer who did not draft the change, says which review findings mean a fresh run, adds + the agent-facing rebuild after an Appendix A change, and points to how a change to the process + files is made. Prompted by gaps in the text as it stood: `ACCEPTANCE.md` named no fixture in + what a reader receives whilst `acceptance/README.md` did, the thresholds were written as five + of five and four of five for a cohort of at least five, two pass-criteria rows both covered a + single reader's traceable miss, and nothing settled an unanswered question or "the same + passage". [Run 16](acceptance/runs/016-deferral-clause-run-1.md), [run 17](acceptance/runs/017-deferral-clause-run-2.md) and - [run 18](acceptance/runs/018-deferral-clause-run-3.md) are written in. + [run 18](acceptance/runs/018-deferral-clause-run-3.md) ruled on these case by case, and their + rulings are written in. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bc249de..a555d7f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,8 +29,8 @@ review. It is a patch version when released. SHOULD or MAY in a numbered section or appendix. It needs everything a clarity edit needs, plus the five tests in SCOPE.md answered in the pull request, and it is at least a minor version when released. `tools/changes.py` tells the two apart mechanically and CI runs it on every pull -request. A change to the acceptance process itself is neither, and ACCEPTANCE.md says how it is -made. +request. A change to the process files ACCEPTANCE.md names, this file among them, is neither, +and ACCEPTANCE.md says how it is made. ## What a pull request contains @@ -62,7 +62,8 @@ survives, the change is yours. 3. **Review and rewrite.** An independent review of the diff against the whole document, by a reviewer who did not draft the change, then the editor's rewrite, pushed to the pull request branch where the contributor allows it or to an editor's branch that supersedes it. A review - finding applied after a cohort has run is a wording change, and a fresh run follows. + finding that changes the document's text after a cohort has run is a wording change, and a + fresh run follows; the header the editor sets at step 5 is not a wording change. 4. **Keys.** Where the change touches a sentence an acceptance key quotes, or adds a load-bearing obligation, the editor updates the key before the cohort runs, never after. 5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text. Its record is @@ -74,7 +75,8 @@ survives, the change is yours. Steps 2 to 5 are the editor's, not CI's, and they happen in batches. The editor may have an agent draft or review, and may delegate marking as ACCEPTANCE.md allows, in which case the -marker's verdict stands; every other decision in these steps stays the editor's. A quiet pull request is not a declined one; a declined one says so. +marker's verdict stands; every other decision in these steps stays the editor's. A quiet pull +request is not a declined one; a declined one says so. ## Versions and releases diff --git a/acceptance/README.md b/acceptance/README.md index 57a2880..c7e8d04 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -19,9 +19,9 @@ the record of every run lives in `runs/`. name. Nothing else: no key, no changelog, no critique findings, no other reader's answers. 3. The editor, or a separate marker the editor delegates to, marks each answer sheet against `KEY.md` under the marking rules in `ACCEPTANCE.md` and applies its pass criteria. -4. The result is recorded as a file under [runs/](runs/), with cohort size, model, how the - readers were isolated, run number, who marked, marks and the findings applied, and the - change's `Unreleased` changelog line names that file. +4. The result is recorded as a file under [runs/](runs/), stating what `ACCEPTANCE.md` lists, + with the answer sheets kept beside it, and the change's `Unreleased` changelog line names + that file. A run that fails is followed by a change to the text, never to the questions, and a fresh cohort. diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 3435821..6f0fddc 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -5,11 +5,15 @@ share. Runs 1 to 15 predate this directory and are summarised in the changelog e 1.0.0 and 1.0.1 of the method specification, 1.0.0 of the detector specification and 0.2.0 of the toolchain specification; numbering here continues from 16. -A record states, in this order: the document; the pull request and the commit the readers were -given; the cohort size and model; how the readers were isolated; the run number within that -pull request; who marked; the marks per reader per question against the key, with any judgement -made in marking; the passages quoted as confusing and by how many readers; the findings applied -to the text as a result; and the verdict against the criteria in `ACCEPTANCE.md`. Answer sheets are not pasted in full; the marks are the record. +A record states what `ACCEPTANCE.md` lists, in this order: the document; the pull request and +the commit the readers were given; the cohort size and model; how the readers were isolated and +the command they were started with; the run number within that pull request; who marked and, for +a delegated marker, how it was started; the marks per reader per question against the key, with +any judgement made in marking; the passages quoted as confusing and by how many readers; the +findings applied to the text as a result; and the verdict against the criteria in +`ACCEPTANCE.md`. Answer sheets are not pasted into the record, and the marks are the record; the +sheets are kept unedited beside it in a directory `NNN-/`, one file per reader, with the +instruction a delegated marker was given. The `Unreleased` changelog line for the change names the record, and `tools/changes.py` refuses a specification change whose pull request adds no file here. From ad97dfaff012079ea282188de7a90a96c5433d86 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 5 Oct 2026 12:18:40 +0000 Subject: [PATCH 03/19] Acceptance process: apply the owner's decisions on review 1 - Process changes are recorded in acceptance/CHANGES.md; the fourth section leaves CHANGELOG.md, which is again for the three specifications only, so tools/versions.py no longer reads a phantom toolchain Unreleased entry. - The isolation rule and the kept answer sheets apply from run 019; runs 16 to 18 stand as recorded. - A delegated marker's verdict is final unless the editor shows a rule was misapplied, then re-marked with the reason recorded; any open judgement takes the stricter reading. - The claude -p example carries the flags that isolate a reader: --tools "", --setting-sources "", --strict-mcp-config, --no-session-persistence, from an empty directory, packet inline. - A cohort is an odd number of readers, at least five; the editor may choose a larger one for a more load-bearing change. Fewer than five counting readers voids the run. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 47 +++++++++++++++++++++++++++++-------------- CHANGELOG.md | 28 -------------------------- CONTRIBUTING.md | 8 +++++--- acceptance/CHANGES.md | 35 ++++++++++++++++++++++++++++++++ acceptance/README.md | 12 ++++++----- 5 files changed, 79 insertions(+), 51 deletions(-) create mode 100644 acceptance/CHANGES.md diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index e0ba89a..548d732 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -8,22 +8,36 @@ test is run with a low-strength model so that clarity is measured at the floor, ## The cohort -- **At least five independent readers**, each a fresh instance of a low-strength model (the - smallest current model in its family, for example Haiku), with no shared context and no - access to the change history, the critique findings or each other. +- **An odd number of independent readers, at least five**, each a fresh instance of a + low-strength model (the smallest current model in its family, for example Haiku), with no + shared context and no access to the change history, the critique findings or each other. The + editor chooses the size, and may choose a larger cohort the more load-bearing the change is. + - **Each reader is a separate process with no tools and no context but its packet**, started - outside any repository so that no project instruction or settings file reaches it: for - example `claude -p` with every tool disabled, run from an empty directory. A reader that - shares the editor's context, such as an agent dispatched from the editor's own session, is not - a cold reader, and its answers do not count towards a run. + from an empty directory outside any repository, with no settings files, no MCP servers and no + saved session, so that no project instruction or settings file reaches it. For example, with + the packet given inline as the prompt: + + ``` + claude -p --model haiku --tools "" --setting-sources "" --strict-mcp-config --no-session-persistence "" + ``` + + A reader that shares the editor's context, such as an agent dispatched from the editor's own + session, is not a cold reader, and its answers do not count towards a run. If fewer than five + readers' answers count, the run is void and a fresh cohort runs. + - Each reader receives **only its packet**: the changed document, the document's question set and the fixture the questions name, each in full in the reader's prompt and headed by its file name. The question set carries the instruction to the reader; no preamble, wrapper or other text is added beyond the file names. Every reader in a run receives the same packet. Nothing else is read first. + - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. +The isolation rule above and the answer sheets kept under [The record](#the-record) apply to runs +from 019 onwards; runs 16 to 18 stand as recorded. + ## The question set Written before the run, with an **answer key** for every keyed question drawn verbatim from the @@ -79,8 +93,9 @@ read, its question set, the fixture the questions name and its key; this file an shape; the answer sheets; and, from the editor, the facts the record states that the sheets do not, which are the pull request and the commit, the cohort size and model, the command the readers were started with and the run number. The record states those facts as the editor -supplied them. A delegated marker's verdict stands. Each sheet is marked question by question, -as follows. +supplied them. A delegated marker's verdict is final unless the editor shows that a rule in this +file was misapplied; the editor then re-marks what the misapplied rule affected, and the record +says which rule and why. Each sheet is marked question by question, as follows. - **Quoting.** A reader quotes the document when it gives words of the document verbatim, a whole sentence or part of one. Naming a clause number, or paraphrasing, is not quoting. @@ -103,8 +118,9 @@ as follows. common. Passages with no sentence in common are different, even within one paragraph or clause. -Where a mark needs a judgement these rules do not settle, the marker makes it, and the record -states the judgement and why. +Where a mark needs a judgement these rules do not settle, the marker makes it, taking the stricter +reading, the one under which the change is less likely to pass, and the record states the +judgement and why. ## Pass criteria @@ -179,7 +195,8 @@ process files, and a change to any part of them is a change to this process. The specifications, and a change to them is not tested by a cohort: the cohort measures whether a low-strength reader can follow a specification, and the process is followed by the editor. Such a change names the runs or the case that showed the need for it, has the same independent review -as any other, and is recorded by a line under `Unreleased` in the changelog's entry for the -acceptance process. It is bound by the section on the bar: it may raise a criterion, never relax -one. A change to a question set, key or fixture is not a change to this process; -`acceptance/README.md` governs those. +as any other, and is recorded by a line under `Unreleased` in +[acceptance/CHANGES.md](acceptance/CHANGES.md), not in the changelog, which is for the three +specifications. It is bound by the section on the bar: it may raise a criterion, never relax one. +A change to a question set, key or fixture is not a change to this process; `acceptance/README.md` +governs those. diff --git a/CHANGELOG.md b/CHANGELOG.md index 7df15d9..f5f8122 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -133,31 +133,3 @@ First published version, the draft label dropped on the same day as the method s custom rules, a single-rule harness, identifier resolution, a derived listing of active defences, the project record, self-application, and the conformance definition in section 11. Published before the acceptance test in `ACCEPTANCE.md` existed; 0.2.0 is the first version to pass it. - -## Acceptance process - -Not a specification and not versioned. Changes to the process files `ACCEPTANCE.md` names are -recorded here as it describes, and need no cohort. - -### Unreleased - -- The cohort can be run and marked without judgement calls that the rules leave open. Each - reader is a separate process with no tools and no context but its packet, and the packet is - the document, its question set and the fixture the questions name, with nothing added around - them. A new marking section says what a delegated marker receives and that its verdict stands, - what counts as quoting, what a correct load-bearing answer needs, which misses are traceable - to the text and how the key's statements bear on that, how unanswered items count, and when - two confusing passages are the same. Thresholds read "every reader" and "all but one", the - same as five of five and four of five for a cohort of five; every pass-criteria row must hold, - and the last covers only what the others do not. The run record states how readers were - isolated and who marked, and the answer sheets are kept beside it. `CONTRIBUTING.md` requires - a reviewer who did not draft the change, says which review findings mean a fresh run, adds - the agent-facing rebuild after an Appendix A change, and points to how a change to the process - files is made. Prompted by gaps in the text as it stood: `ACCEPTANCE.md` named no fixture in - what a reader receives whilst `acceptance/README.md` did, the thresholds were written as five - of five and four of five for a cohort of at least five, two pass-criteria rows both covered a - single reader's traceable miss, and nothing settled an unanswered question or "the same - passage". [Run 16](acceptance/runs/016-deferral-clause-run-1.md), - [run 17](acceptance/runs/017-deferral-clause-run-2.md) and - [run 18](acceptance/runs/018-deferral-clause-run-3.md) ruled on these case by case, and their - rulings are written in. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a555d7f..b7d8456 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,8 +29,9 @@ review. It is a patch version when released. SHOULD or MAY in a numbered section or appendix. It needs everything a clarity edit needs, plus the five tests in SCOPE.md answered in the pull request, and it is at least a minor version when released. `tools/changes.py` tells the two apart mechanically and CI runs it on every pull -request. A change to the process files ACCEPTANCE.md names, this file among them, is neither, -and ACCEPTANCE.md says how it is made. +request. A change to the process files ACCEPTANCE.md names, this file among them, is neither: +ACCEPTANCE.md says how it is made, and it is recorded in `acceptance/CHANGES.md`, not in +CHANGELOG.md. ## What a pull request contains @@ -75,7 +76,8 @@ survives, the change is yours. Steps 2 to 5 are the editor's, not CI's, and they happen in batches. The editor may have an agent draft or review, and may delegate marking as ACCEPTANCE.md allows, in which case the -marker's verdict stands; every other decision in these steps stays the editor's. A quiet pull +marker's verdict is final unless the editor shows a rule was misapplied, as ACCEPTANCE.md says; +every other decision in these steps stays the editor's. A quiet pull request is not a declined one; a declined one says so. ## Versions and releases diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md new file mode 100644 index 0000000..af466a9 --- /dev/null +++ b/acceptance/CHANGES.md @@ -0,0 +1,35 @@ +# Changes to the acceptance process + +Changes to the process files that [ACCEPTANCE.md](../ACCEPTANCE.md) names, recorded here as it +describes. The process is not a specification and is not versioned, and a change to it is not +tested by a cohort. Changes to the three specifications are recorded in +[CHANGELOG.md](../CHANGELOG.md). + +## Unreleased + +- The cohort can be run and marked without judgement calls that the rules leave open. A cohort + is an odd number of readers, at least five, and the editor may choose a larger one for a more + load-bearing change. Each reader is a separate process with no tools, no settings files, no + MCP servers and no context but its packet, started from an empty directory outside any + repository, and the example command shows the flags that do this; a reader sharing the + editor's context does not count, and a run left with fewer than five counting readers is void. + The packet is the document, its question set and the fixture the questions name, with nothing + added around them. A new marking section says what a delegated marker receives, that its + verdict is final unless the editor shows a rule was misapplied, that the marker takes the + stricter reading wherever it makes an open judgement, what counts as quoting, what a correct + load-bearing answer needs, which misses are traceable to the text and how the key's statements + bear on that, how unanswered items count, and when two confusing passages are the same. + Thresholds read "every reader" and "all but one", the same as five of five and four of five for + a cohort of five; every pass-criteria row must hold, and the last covers only what the others + do not. The run record states how readers were isolated, the command they were started with + and who marked, and the answer sheets are kept beside it; the isolation rule and the kept + sheets apply from run 019, and runs 16 to 18 stand as recorded. Process changes are recorded + in this file, not in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the + change, says which review findings mean a fresh run, adds the agent-facing rebuild after an + Appendix A change, and points to how a change to the process files is made. Prompted by gaps in + the text as it stood: `ACCEPTANCE.md` named no fixture in what a reader receives whilst + `acceptance/README.md` did, the thresholds were written as five of five and four of five for a + cohort of at least five, two pass-criteria rows both covered a single reader's traceable miss, + and nothing settled an unanswered question or "the same passage". [Run + 16](runs/016-deferral-clause-run-1.md), [run 17](runs/017-deferral-clause-run-2.md) and + [run 18](runs/018-deferral-clause-run-3.md) ruled on these case by case. diff --git a/acceptance/README.md b/acceptance/README.md index c7e8d04..b5ae59d 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -2,8 +2,9 @@ The question sets and answer keys that [ACCEPTANCE.md](../ACCEPTANCE.md) requires. One directory per specification document, each holding the fixed questions a cold reader answers and the key an -editor marks against. The fixture cases the application question uses live in `fixtures/`, and -the record of every run lives in `runs/`. +editor marks against. The fixture cases the application question uses live in `fixtures/`, the +record of every run lives in `runs/`, and changes to the acceptance process itself are recorded +in [CHANGES.md](CHANGES.md). | Document | Questions | Key | | ------------------ | ------------------------------------------------ | ------------------------------------ | @@ -14,9 +15,10 @@ the record of every run lives in `runs/`. ## How a run works 1. The editor identifies which documents the change touched. Only those are tested. -2. For each, five or more fresh low-strength readers, each isolated as `ACCEPTANCE.md` describes, - each receive the document's current text, its `QUESTIONS.md`, and the fixture the questions - name. Nothing else: no key, no changelog, no critique findings, no other reader's answers. +2. For each, an odd number of fresh low-strength readers, at least five, each isolated as + `ACCEPTANCE.md` describes, each receive the document's current text, its `QUESTIONS.md`, and + the fixture the questions name. Nothing else: no key, no changelog, no critique findings, no + other reader's answers. 3. The editor, or a separate marker the editor delegates to, marks each answer sheet against `KEY.md` under the marking rules in `ACCEPTANCE.md` and applies its pass criteria. 4. The result is recorded as a file under [runs/](runs/), stating what `ACCEPTANCE.md` lists, From d45df88760e4f2d9ef4bbe1666b85df4b4addc1e Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 20:34:04 +0100 Subject: [PATCH 04/19] Acceptance process: answer sheets posted on the pull request, not kept in the repository The owner chose to keep the repository from growing by five or more sheet files a run. The record links one pull request comment holding every sheet unedited and the instruction a delegated marker was given. A process change carries no run placeholder: it is recorded in acceptance/CHANGES.md and is not tested by a cohort, as ACCEPTANCE.md already says. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 13 +++++++------ acceptance/CHANGES.md | 5 +++-- acceptance/README.md | 4 ++-- acceptance/runs/README.md | 6 +++--- 4 files changed, 15 insertions(+), 13 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 548d732..307e00d 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -35,8 +35,8 @@ test is run with a low-strength model so that clarity is measured at the floor, - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. -The isolation rule above and the answer sheets kept under [The record](#the-record) apply to runs -from 019 onwards; runs 16 to 18 stand as recorded. +The isolation rule above and the answer sheets posted under [The record](#the-record) apply to +runs from 019 onwards; runs 16 to 18 stand as recorded. ## The question set @@ -176,10 +176,11 @@ the one sequence shared by the three documents, stating: the document, the pull commit read, the cohort size and model, how the readers were isolated and the command they were started with, the run number in that pull request, who marked and, for a delegated marker, how it was started, the marks per reader per question, any judgement made in marking, the passages -quoted as confusing and by how many readers, the findings applied, and whether it passed. Each -reader's answer sheet is kept unedited beside the record, one file per reader in a directory -`NNN-/`, with the instruction a delegated marker was given, so that an owner can check the -isolation and the marks. The changelog line for the change names the file. CI refuses a +quoted as confusing and by how many readers, the findings applied, whether it passed, and a link +to the answer sheets. The answer sheets are not kept in this repository: every reader's sheet, +unedited, and the instruction a delegated marker was given are posted in one comment on the pull +request, so that an owner can check the isolation and the marks. The changelog line for the +change names the file. CI refuses a specification change whose pull request adds no run file; it checks for at least one, and a pull request that changes two documents adds a run for each. diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index af466a9..0cd5100 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -22,8 +22,9 @@ tested by a cohort. Changes to the three specifications are recorded in Thresholds read "every reader" and "all but one", the same as five of five and four of five for a cohort of five; every pass-criteria row must hold, and the last covers only what the others do not. The run record states how readers were isolated, the command they were started with - and who marked, and the answer sheets are kept beside it; the isolation rule and the kept - sheets apply from run 019, and runs 16 to 18 stand as recorded. Process changes are recorded + and who marked, and links the answer sheets, which are posted unedited on the pull request + rather than kept in the repository; the isolation rule and the posted sheets apply from run + 019, and runs 16 to 18 stand as recorded. Process changes are recorded in this file, not in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the change, says which review findings mean a fresh run, adds the agent-facing rebuild after an Appendix A change, and points to how a change to the process files is made. Prompted by gaps in diff --git a/acceptance/README.md b/acceptance/README.md index b5ae59d..bb387a5 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -22,8 +22,8 @@ in [CHANGES.md](CHANGES.md). 3. The editor, or a separate marker the editor delegates to, marks each answer sheet against `KEY.md` under the marking rules in `ACCEPTANCE.md` and applies its pass criteria. 4. The result is recorded as a file under [runs/](runs/), stating what `ACCEPTANCE.md` lists, - with the answer sheets kept beside it, and the change's `Unreleased` changelog line names - that file. + with the answer sheets posted on the pull request and linked from it, and the change's + `Unreleased` changelog line names that file. A run that fails is followed by a change to the text, never to the questions, and a fresh cohort. diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 6f0fddc..fd7f56c 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -11,9 +11,9 @@ the command they were started with; the run number within that pull request; who a delegated marker, how it was started; the marks per reader per question against the key, with any judgement made in marking; the passages quoted as confusing and by how many readers; the findings applied to the text as a result; and the verdict against the criteria in -`ACCEPTANCE.md`. Answer sheets are not pasted into the record, and the marks are the record; the -sheets are kept unedited beside it in a directory `NNN-/`, one file per reader, with the -instruction a delegated marker was given. +`ACCEPTANCE.md`, with a link to the answer sheets. Answer sheets are not kept in this repository, +and the marks are the record; the sheets, unedited, and the instruction a delegated marker was +given are posted in one comment on the pull request, which the record links. The `Unreleased` changelog line for the change names the record, and `tools/changes.py` refuses a specification change whose pull request adds no file here. From 6cc2fb69b1d90ee0a9c51822847e6414cfa8ec49 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 20:47:13 +0100 Subject: [PATCH 05/19] Acceptance process: apply the editor's rulings on review 2 Defines a wording change once, keyed items, quoting, the same passage and cold readers; makes the reader packet, isolation command and set-aside and void-run rules exact; has the marker write the record; posts the exact prompt and the marker's instruction with the sheets; restores the paraphrase rule for traceable misses; removes the keys' limits on which misses count against the document; adds the enforcing checks to the process files and says how a process change moves through the steps. acceptance/CHANGES.md loses its Unreleased heading and names the pull request. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 189 +++++++++++++++++++++--------------- CONTRIBUTING.md | 16 +-- acceptance/CHANGES.md | 65 +++++++------ acceptance/README.md | 8 +- acceptance/detector/KEY.md | 3 +- acceptance/method/KEY.md | 6 +- acceptance/runs/README.md | 18 ++-- acceptance/toolchain/KEY.md | 4 +- 8 files changed, 176 insertions(+), 133 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 307e00d..83afafb 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -8,35 +8,44 @@ test is run with a low-strength model so that clarity is measured at the floor, ## The cohort -- **An odd number of independent readers, at least five**, each a fresh instance of a +- **An odd number of independent readers started, at least five**, each a fresh instance of a low-strength model (the smallest current model in its family, for example Haiku), with no shared context and no access to the change history, the critique findings or each other. The - editor chooses the size, and may choose a larger cohort the more load-bearing the change is. + editor chooses the size, and may choose a larger cohort the more obligations the change + touches. -- **Each reader is a separate process with no tools and no context but its packet**, started - from an empty directory outside any repository, with no settings files, no MCP servers and no - saved session, so that no project instruction or settings file reaches it. For example, with - the packet given inline as the prompt: +- **Each reader is a separate process with no tools and no project context, only its packet**, + started from an empty directory outside any repository, with no settings files, no `CLAUDE.md`, + skills, plugins, hooks or MCP servers, and no saved session, so that no project instruction or + settings file reaches it. The model tool's own fixed system prompt is not project context. For + example, with the packet on standard input: ``` - claude -p --model haiku --tools "" --setting-sources "" --strict-mcp-config --no-session-persistence "" + claude -p --safe-mode --model haiku --tools "" --setting-sources "" --strict-mcp-config --no-session-persistence --output-format json < packet.md ``` - A reader that shares the editor's context, such as an agent dispatched from the editor's own - session, is not a cold reader, and its answers do not count towards a run. If fewer than five - readers' answers count, the run is void and a fresh cohort runs. - -- Each reader receives **only its packet**: the changed document, the document's question set - and the fixture the questions name, each in full in the reader's prompt and headed by its file - name. The question set carries the instruction to the reader; no preamble, wrapper or other - text is added beyond the file names. Every reader in a run receives the same packet. Nothing - else is read first. + The JSON output gives the model version each reader reported. A reader isolated in this way is + a cold reader. A reader whose process failed or returned nothing, or that shared the editor's + context, such as an agent dispatched from the editor's own session, is set aside and its + answers do not count; no reader is set aside for any other cause, and the record shows the + cause. If fewer than five readers' answers count, the run is void and a fresh cohort runs, + unless the readers whose answers count already fail a criterion, when the run fails. A void run + takes a run number, has a record and posted sheets like any other, and counts towards the + number of runs. + +- Each reader receives **only its packet**: the document's question set, the changed document + and the fixture the questions name, each in full in the reader's prompt, the question set + first. Each file is preceded by exactly one line, `===== FILE: =====`, where `` is + the name the question set gives the file (so `fixtures/remediation-ledger.md`, and + `QUESTIONS.md` for the question set itself). The question set carries the instruction to the + reader, and nothing else is added. Every reader in a run receives the same packet. Nothing else + is read first. - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. -The isolation rule above and the answer sheets posted under [The record](#the-record) apply to -runs from 019 onwards; runs 16 to 18 stand as recorded. +The isolation rule above and the answer sheets posted under [The record](#the-record) apply from +run 19; runs 16 to 18 stand as recorded. ## The question set @@ -89,55 +98,65 @@ The editor marks each answer sheet against the document's key, or delegates the separate marker that has seen neither the drafting nor the review. Whoever marks is the marker in this file. A delegated marker receives only these: the document at the commit the readers read, its question set, the fixture the questions name and its key; this file and -`acceptance/runs/README.md`, with earlier run records of other pull requests for the record's -shape; the answer sheets; and, from the editor, the facts the record states that the sheets do -not, which are the pull request and the commit, the cohort size and model, the command the -readers were started with and the run number. The record states those facts as the editor -supplied them. A delegated marker's verdict is final unless the editor shows that a rule in this -file was misapplied; the editor then re-marks what the misapplied rule affected, and the record -says which rule and why. Each sheet is marked question by question, as follows. - -- **Quoting.** A reader quotes the document when it gives words of the document verbatim, a - whole sentence or part of one. Naming a clause number, or paraphrasing, is not quoting. +`acceptance/runs/README.md`, with earlier run records of other pull requests as examples of +style only; the answer sheets; and, from the editor, the facts the record states that the sheets +do not, which are the pull request and the commit, the cohort size and model, how the readers +were isolated and the command they were started with, how the marker was started, and the run +number. The marker writes the record, stating those facts as the editor supplied them, and the +editor adds the findings applied and the link to the posted sheets after marking. A delegated +marker's verdict, its marks and its PASS or FAIL, is final unless the editor shows that a rule +in this file was misapplied; the editor then re-marks what the misapplied rule affected, and the +record says which rule and why. Each sheet is marked question by question, as follows. + +- **Keyed items.** Question 1 as a whole is one keyed item, and so is each item of question 2 + and each clause verdict and each summary verdict of question 3. +- **Quoting.** A reader quotes the document when it gives five or more consecutive words of the + document, verbatim apart from markdown link syntax and emphasis markers, as a quotation, in + quotation marks or a block quote. Naming a clause number, or paraphrasing, is not quoting. - **Questions 1 and 3** are marked as the key states for each item. - **Question 2.** An answer is correct when its sentence states what the key's answer states, including anything the key says an answer is wrong to omit, asserts nothing the key says makes an answer wrong, and quotes the document in support. The quotation need not be the one the key gives. -- **A miss traceable to the text** is a miss whose reasoning quotes the document, or a miss the - key says counts against the document. A miss whose reasoning quotes the document is traceable - even where it also misreads the fixture, and even where the key says that miss is the reader's - or counts only with a particular quotation: the key can add to the misses that count against - the document, never take one away. Every other miss is the reader's. +- **A miss traceable to the text** is a miss whose reasoning quotes the document or states, in + the reader's own words, what the document says or requires, or a miss the key says counts + against the document. Such a miss is traceable even where it also misreads the fixture, and + whatever the key says of it: the key can add to the misses that count against the document, + never take one away. Only a miss whose reasoning states nothing about the document, such as a + misread fixture alone, is the reader's. Question 1 has no reasoning, so a miss there is + traceable only where the key says it counts. - **An unanswered question.** Each keyed item left unanswered is one miss, not traceable to the text, so a sheet that leaves three items of question 3 blank has three misses. Question 4 left - unanswered counts as no passage quoted. + unanswered counts as no passage quoted. A truncated sheet is marked as it stands; a reader + whose process failed or returned nothing is set aside, as [The cohort](#the-cohort) says. - **The same passage.** A quotation under question 4 covers every sentence of the document it - quotes any part of, and a passage named without a quotation, by clause number or description, - covers every sentence of what it names. Two passages are the same when they cover a sentence in - common. Passages with no sentence in common are different, even within one paragraph or - clause. + quotes any part of, in every place the quoted words appear, and a passage named without a + quotation, by clause number or description, covers every sentence of what it names. A heading, + list item or table row counts as one sentence. Two passages are the same when they cover a + sentence in common or lie in the same paragraph, and a list item or table row is its own + paragraph. Where a mark needs a judgement these rules do not settle, the marker makes it, taking the stricter -reading, the one under which the change is less likely to pass, and the record states the -judgement and why. +reading, the one that records more misses or findings, and the record states the judgement and +why. ## Pass criteria -The change passes when all of the following hold on a single run. With a cohort of five, "every -reader" is five of five and "all but one" is four of five. +The change passes when all of the following hold on a single run. The thresholds apply to the +readers whose answers count; with five such readers, "every reader" is five of five and "all but +one" is four of five. -| Criterion | Threshold | -| ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | -| Load-bearing statements (question 2) | **Every** reader correct on every one | -| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | -| A miss traceable to the text | **None.** Any such miss, as Marking defines it, is a finding, whatever the count | -| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it | -| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | +| Criterion | Threshold | +| ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | +| Load-bearing statements (question 2) | **Every** reader correct on every one | +| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | +| A miss traceable to the text | **None.** Any such miss, as [Marking](#marking) defines it, is a finding, whatever the count | +| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort on the changed text passes this row | +| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | -Every row must hold. The first four apply whatever the count, and the last covers only what none -of them catches: a miss traceable to the text blocks even when only one reader made it, and the -second row's tolerance of one reader does not excuse it. +Every row must hold, and the last covers only what none of the others catches. The first, third +and fourth rows block on what a single reader, or for the fourth two readers, did, and the second +row's tolerance of one reader never excuses a miss under the third. The asymmetry is deliberate. A load-bearing statement is the sentence the whole document exists to make, and one reader in five getting it wrong means one agent in five will act on the wrong @@ -149,8 +168,9 @@ the reader can justify from the document is always the document's fault. No criterion in this document is ever relaxed to let a change through. If the cohort is confused, the test is working and the text is wrong; the fix is to the text, never to the threshold, the -question, the fixture or the key. A threshold may be raised, a keyed question may be added, and a -key may be corrected where it misquoted the document; none of those makes a failing run pass. The +question, the fixture or the key. A threshold may be raised, a keyed question may be added, a key +may be corrected where it misquoted the document, and a key's limit on which misses count against +the document may be removed, which raises the bar; none of those makes a failing run pass. The one time a threshold was relaxed, after three runs, the relaxation was reversed on the author's instruction and is recorded here so that it is not tried again. Removing the clarity rating and the request for suggestions was not a relaxation: it changed what the cohort is asked, not how @@ -164,25 +184,36 @@ something. ## When it runs -Once per pull request, on the final text, after the editor's rewrite and any key update. It is -the last gate before the merge to `next`, so that every change on `next` is one the cohort has -accepted, and the release branch needs no cohort of its own unless it changes specification text. -A wording change after the run means a fresh run; readers are never reused. +On the final text of each pull request, after the editor's rewrite and any key update, and again +after every failure, void run or wording change. It is the last gate before the merge to `next`, +so that every change on `next` is one the cohort has accepted, and the release branch needs no +cohort of its own unless it changes specification text. A wording change is any change to the +document's text after the readers read it, from any source (a review finding, the editor, or +`next` merged into the branch), except the lines `PUBLISHING.md` and step 5 of `CONTRIBUTING.md` +have the editor set for a version: the header, the status paragraph and the citation. A wording +change means a fresh run; readers are never reused. ## The record Each run has a file under `acceptance/runs/`, named `NNN-.md` with `NNN` the next number in the one sequence shared by the three documents, stating: the document, the pull request and the -commit read, the cohort size and model, how the readers were isolated and the command they were -started with, the run number in that pull request, who marked and, for a delegated marker, how it -was started, the marks per reader per question, any judgement made in marking, the passages -quoted as confusing and by how many readers, the findings applied, whether it passed, and a link -to the answer sheets. The answer sheets are not kept in this repository: every reader's sheet, -unedited, and the instruction a delegated marker was given are posted in one comment on the pull -request, so that an owner can check the isolation and the marks. The changelog line for the -change names the file. CI refuses a -specification change whose pull request adds no run file; it checks for at least one, and a pull -request that changes two documents adds a run for each. +commit read, the cohort size and model, with the model version each reader reported, how the +readers were isolated and the command they were started with, the run number in that pull +request, who marked and, for a delegated marker, how it was started, the marks per reader per +question, any judgement made in marking, the passages quoted as confusing and by how many +readers, the findings applied, whether it passed, and a link to the answer sheets. The answer +sheets are not kept in this repository: every reader's sheet, unedited, with a set-aside +reader's marked as set aside, and the instruction a delegated marker was given, which is its +task prompt and the agent definition it ran under, are posted in one or more consecutive +comments on the pull request, each linked from the record, so that an owner can check the marks. +The comments also hold the exact prompt every reader received and the command and the form of +the working directory the readers were started in; a sheet cannot show how its process was +started, so the record says that the editor attests the rest of the isolation. A comment is not +edited after posting, and a record whose comment was edited is void; that is checkable only +through GitHub's edit history, which is the limit of keeping the sheets outside the repository. +The changelog line for the change names the file. CI refuses a specification change whose pull +request adds no run file; it checks for at least one, and a pull request that changes two +documents adds a run for each. Each published version's changelog entry carries the lines of the changes it contains, so a version's cohort record is the set of runs those lines name. A version with no such line has not @@ -191,13 +222,17 @@ under `acceptance/` in this repository so that a run can be repeated and compare ## Changes to this process -This file, `acceptance/README.md`, `acceptance/runs/README.md` and `CONTRIBUTING.md` are the -process files, and a change to any part of them is a change to this process. They are not -specifications, and a change to them is not tested by a cohort: the cohort measures whether a -low-strength reader can follow a specification, and the process is followed by the editor. Such -a change names the runs or the case that showed the need for it, has the same independent review -as any other, and is recorded by a line under `Unreleased` in -[acceptance/CHANGES.md](acceptance/CHANGES.md), not in the changelog, which is for the three -specifications. It is bound by the section on the bar: it may raise a criterion, never relax one. -A change to a question set, key or fixture is not a change to this process; `acceptance/README.md` +The process files are this file, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md` and the checks that enforce them: `tools/changes.py`, the key-quotation checks +in `spec-qa.py`, the CI workflow and the pull request template. A change to any part of them is +a change to this process. They are not specifications, and a change to them is not tested by a +cohort: the cohort measures whether a low-strength reader can follow a specification, and the +process is followed by the editor. Such a change names the runs or the case that showed the need +for it, and goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, with no scope, key or cohort +step; its review reads every process file it changes. It applies from its merge into `next`; a +run made before then may follow it, and its record says so. It is recorded by a line added at +the top of [acceptance/CHANGES.md](acceptance/CHANGES.md), naming its pull request, not in the +changelog, which is for the three specifications. It is bound by +[the section on the bar](#the-bar-only-moves-up): it may raise a criterion, never relax one. A +change to a question set, key or fixture is not a change to this process; `acceptance/README.md` governs those. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b7d8456..796be43 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -41,9 +41,9 @@ CHANGELOG.md. request template lays them out. - The observed case that prompted it, for either kind. A clarity edit still has a reason: a reader who took the passage the wrong way. -- Where a specification's Appendix A changed, the agent-facing files rebuilt with - `python3 build-agent-surfaces.py` and committed with the change; CI runs it with `--check` - and fails when they are stale. +- Where a specification's Appendix A changed, the files `python3 build-agent-surfaces.py` + writes, rebuilt with it and committed with the change; CI runs it with `--check` and fails + when they are stale. - `python3 spec-qa.py` clean and the unit tests passing, which CI checks for you. A pull request is a suggestion, not a demand. The documents have one editor, who keeps ownership @@ -62,9 +62,9 @@ survives, the change is yours. pull request with a row in `DECLINED.md`. 3. **Review and rewrite.** An independent review of the diff against the whole document, by a reviewer who did not draft the change, then the editor's rewrite, pushed to the pull request - branch where the contributor allows it or to an editor's branch that supersedes it. A review - finding that changes the document's text after a cohort has run is a wording change, and a - fresh run follows; the header the editor sets at step 5 is not a wording change. + branch where the contributor allows it or to an editor's branch that supersedes it. The pull + request names the reviewer and links or contains the review. A wording change, as + ACCEPTANCE.md defines it under "When it runs", means a fresh run. 4. **Keys.** Where the change touches a sentence an acceptance key quotes, or adds a load-bearing obligation, the editor updates the key before the cohort runs, never after. 5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text. Its record is @@ -77,8 +77,8 @@ survives, the change is yours. Steps 2 to 5 are the editor's, not CI's, and they happen in batches. The editor may have an agent draft or review, and may delegate marking as ACCEPTANCE.md allows, in which case the marker's verdict is final unless the editor shows a rule was misapplied, as ACCEPTANCE.md says; -every other decision in these steps stays the editor's. A quiet pull -request is not a declined one; a declined one says so. +every other decision in these steps stays the editor's. A quiet pull request is not a declined +one; a declined one says so. ## Versions and releases diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 0cd5100..81a9ac7 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -1,34 +1,45 @@ # Changes to the acceptance process Changes to the process files that [ACCEPTANCE.md](../ACCEPTANCE.md) names, recorded here as it -describes. The process is not a specification and is not versioned, and a change to it is not -tested by a cohort. Changes to the three specifications are recorded in -[CHANGELOG.md](../CHANGELOG.md). +describes: a line is added at the top for each change, naming its pull request. The process is +not a specification and is not versioned, and a change to it is not tested by a cohort. Changes +to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). -## Unreleased - -- The cohort can be run and marked without judgement calls that the rules leave open. A cohort - is an odd number of readers, at least five, and the editor may choose a larger one for a more - load-bearing change. Each reader is a separate process with no tools, no settings files, no - MCP servers and no context but its packet, started from an empty directory outside any - repository, and the example command shows the flags that do this; a reader sharing the - editor's context does not count, and a run left with fewer than five counting readers is void. - The packet is the document, its question set and the fixture the questions name, with nothing - added around them. A new marking section says what a delegated marker receives, that its - verdict is final unless the editor shows a rule was misapplied, that the marker takes the - stricter reading wherever it makes an open judgement, what counts as quoting, what a correct - load-bearing answer needs, which misses are traceable to the text and how the key's statements - bear on that, how unanswered items count, and when two confusing passages are the same. - Thresholds read "every reader" and "all but one", the same as five of five and four of five for - a cohort of five; every pass-criteria row must hold, and the last covers only what the others - do not. The run record states how readers were isolated, the command they were started with - and who marked, and links the answer sheets, which are posted unedited on the pull request - rather than kept in the repository; the isolation rule and the posted sheets apply from run - 019, and runs 16 to 18 stand as recorded. Process changes are recorded - in this file, not in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the - change, says which review findings mean a fresh run, adds the agent-facing rebuild after an - Appendix A change, and points to how a change to the process files is made. Prompted by gaps in - the text as it stood: `ACCEPTANCE.md` named no fixture in what a reader receives whilst +- Pull request #PR: the cohort can be run and marked without judgement calls that the rules + leave open. A cohort is an odd number of readers started, at least five, larger at the + editor's choice the more obligations the change touches. Each reader is a separate process + with no tools and no project context, started from an empty directory outside any repository + with no settings files, `CLAUDE.md`, skills, plugins, hooks, MCP servers or saved session; the + example command takes the packet on standard input. A reader is set aside only if its process + failed or returned nothing or it shared the editor's context; a run left with fewer than five + counting readers is void unless those readers already fail it, and a void run has a number, a + record and posted sheets and counts towards the number of runs. The packet is the question + set, the document and the fixture, each after one `===== FILE: =====` line, with + nothing else added. A new marking section says what a delegated marker receives, that it + writes the record and its marks and PASS or FAIL are final unless the editor shows a rule was + misapplied, that open judgements take the reading that records more misses or findings, what + a keyed item is, that quoting is five or more consecutive verbatim words given as a quotation, + what a correct load-bearing answer needs, that a miss is traceable to the text when its + reasoning quotes or states what the document says or the key says it counts, and that a key + can add such misses, never remove one, how unanswered items and truncated sheets count, and + that two confusing passages are the same when they share a sentence or a paragraph. The + thresholds apply to the readers whose answers count, read "every reader" and "all but one", + and block on a single reader for the first, third and fourth rows; the fourth row's block lifts + when a fresh cohort on the changed text passes it. A wording change is any change to the text + after the readers read it, except the lines set for a version, and the cohort runs again after + every failure, void run or wording change. The run record states the model version each reader + reported, how readers were isolated, the command and who marked, and links the answer sheets, + the exact prompt and the marker's instruction, posted unedited in comments on the pull request + rather than kept in the repository, with the rest of the isolation attested by the editor; the + isolation rule and the posted sheets apply from run 19, and runs 16 to 18 stand as recorded. + The three keys no longer limit which misses count against the document, and removing such a + limit is named as raising the bar. The process files include the checks that enforce them, and + a process change goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, applies on its merge into + `next` and is recorded in this file, not in the changelog. `CONTRIBUTING.md` requires a + reviewer who did not draft the change, named in the pull request with the review, points to + the definition of a wording change, and adds the rebuild of the files + `build-agent-surfaces.py` writes after an Appendix A change. Prompted by gaps in the text as it + stood: `ACCEPTANCE.md` named no fixture in what a reader receives whilst `acceptance/README.md` did, the thresholds were written as five of five and four of five for a cohort of at least five, two pass-criteria rows both covered a single reader's traceable miss, and nothing settled an unanswered question or "the same passage". [Run diff --git a/acceptance/README.md b/acceptance/README.md index bb387a5..9af1d2f 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -1,10 +1,10 @@ # Acceptance question sets The question sets and answer keys that [ACCEPTANCE.md](../ACCEPTANCE.md) requires. One directory per -specification document, each holding the fixed questions a cold reader answers and the key an -editor marks against. The fixture cases the application question uses live in `fixtures/`, the -record of every run lives in `runs/`, and changes to the acceptance process itself are recorded -in [CHANGES.md](CHANGES.md). +specification document, each holding the fixed questions a +[cold reader](../ACCEPTANCE.md#the-cohort) answers and the key an editor marks against. The +fixture cases the application question uses live in `fixtures/`, the record of every run lives in +`runs/`, and changes to the acceptance process itself are recorded in [CHANGES.md](CHANGES.md). | Document | Questions | Key | | ------------------ | ------------------------------------------------ | ------------------------------------ | diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 132e6cb..6de0386 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -60,8 +60,7 @@ agent summary (those belong to the toolchain specification). Section [8](../../DETECTOR-SPEC.md#8-conformance) verdict: **does not conform**, because [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) fails and partial conformance MUST NOT be described as conformance. A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Partial on the grounds that thirty-eight resolve -has read the clause differently and that counts against the document; a reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes -because they missed the last paragraph of the fixture has misread the fixture, not the document. +has read the clause differently and that counts against the document. Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**. It is a SHOULD, and Lintomatic's failure to check its own identifiers in CI is a missed SHOULD, not a further MUST failure; the MUST that fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 3ca1a80..71f3935 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -56,8 +56,7 @@ a test may serve as the detector; an agent may baseline the remaining instances. Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**, because [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) and [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) were not followed. The **defence does not conform** either, because [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) fails; a reader who says the defence conforms -because the sweep is what failed has missed [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) and that counts against the document only if the -reasoning quotes the section [7](../../SPEC.md#7-conformance) definition of a conforming defence. +because the sweep is what failed has missed [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation). Section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): the **inline ignore on `Report::render()`** was the owner's decision, not the practitioner's. An answer naming instead the decision not to build the wider rule is half right: @@ -67,8 +66,7 @@ An answer naming neither is wrong. A reader who grades [3.5](../../SPEC.md#35-enforce-permanently-and-block) Yes because the rule fails the build has not applied the clause's opening sentence on unrecorded suppression, and that counts against the document. A reader who grades [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) Yes because the URL resolves has read the versioning sentence of [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) -differently, which counts against the document; a reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the -fixture. A reader who attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing +differently, which counts against the document. A reader who attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as covering it, which is a fair reading and is not marked wrong. ## 4. Confusion diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index fd7f56c..9a20bad 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -6,14 +6,16 @@ share. Runs 1 to 15 predate this directory and are summarised in the changelog e the toolchain specification; numbering here continues from 16. A record states what `ACCEPTANCE.md` lists, in this order: the document; the pull request and -the commit the readers were given; the cohort size and model; how the readers were isolated and -the command they were started with; the run number within that pull request; who marked and, for -a delegated marker, how it was started; the marks per reader per question against the key, with -any judgement made in marking; the passages quoted as confusing and by how many readers; the -findings applied to the text as a result; and the verdict against the criteria in -`ACCEPTANCE.md`, with a link to the answer sheets. Answer sheets are not kept in this repository, -and the marks are the record; the sheets, unedited, and the instruction a delegated marker was -given are posted in one comment on the pull request, which the record links. +the commit the readers were given; the cohort size and model, with the model version each reader +reported; how the readers were isolated and the command they were started with; the run number +within that pull request; who marked and, for a delegated marker, how it was started; the marks +per reader per question against the key, with any judgement made in marking; the passages quoted +as confusing and by how many readers; the findings applied to the text as a result; and the +verdict against the criteria in `ACCEPTANCE.md`, with a link to the answer sheets. Earlier +records show the style only; where one differs from this order, this file wins. Answer sheets +are not kept in this repository, and the marks are the record; the sheets, unedited, and the +instruction a delegated marker was given are posted in one or more consecutive comments on the +pull request, each linked from the record. The `Unreleased` changelog line for the change names the record, and `tools/changes.py` refuses a specification change whose pull request adds no file here. diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 5a2602b..91112fa 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -73,9 +73,7 @@ Declaration: **does not affect the verdict**. Key quotation, [9.2](../../TOOLING A reader who says Zigpipe cannot be graded without one has read [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) wrongly. A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes on the grounds that Lintomatic "mostly" conforms has read either [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) -or the detector document's section [8](../../DETECTOR-SPEC.md#8-conformance) differently; it counts against this document only if the -reasoning quotes [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification). A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes because they missed the two undocumented rules -has misread the fixture. +or the detector document's section [8](../../DETECTOR-SPEC.md#8-conformance) differently. ## 4. Confusion From 23e7d8c140d1ebadbec6330f6ff5a10f218d226c Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 21:02:50 +0100 Subject: [PATCH 06/19] Acceptance process: apply the editor's rulings on review 3 The fourth row's block lifts only when no reader of a fresh cohort names the same passage, as on origin/next, and the keys say "names". Quoting has no length floor and excuses only case, quotation-mark style and markup. A question 2 answer states each thing the key's answer states unless the key says it may be left out. A reader is set aside only for empty answer text or shared context, and any set-aside voids the run unless it already fails. A failed text is never rerun unchanged; only version numbers and dates set for a release are excepted from a wording change. A process change names the run it applies from and is followed whole. The marker receives this pull request's earlier runs and the readers set aside; keys and marking name the commit the readers read. The record lists set-asides, the attestation, re-marks, managed settings, early adoption and void runs; posted sheets are whole outputs, never edited, hidden or deleted. Question 1 misses count only where the key says; half right is a miss; detector 7.1's conditional is gone. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 146 ++++++++++++++++++------------------ acceptance/CHANGES.md | 69 ++++++++--------- acceptance/README.md | 2 + acceptance/detector/KEY.md | 8 +- acceptance/method/KEY.md | 7 +- acceptance/runs/README.md | 17 +++-- acceptance/toolchain/KEY.md | 5 +- 7 files changed, 129 insertions(+), 125 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 83afafb..bab883d 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -15,23 +15,23 @@ test is run with a low-strength model so that clarity is measured at the floor, touches. - **Each reader is a separate process with no tools and no project context, only its packet**, - started from an empty directory outside any repository, with no settings files, no `CLAUDE.md`, - skills, plugins, hooks or MCP servers, and no saved session, so that no project instruction or - settings file reaches it. The model tool's own fixed system prompt is not project context. For - example, with the packet on standard input: + started from an empty directory outside any repository, with no user, project or local settings + files, no `CLAUDE.md`, skills, plugins, hooks or MCP servers, and no saved session, so that no + project instruction reaches it. The model tool's own fixed system prompt is not project + context. For example, with the packet on standard input: ``` claude -p --safe-mode --model haiku --tools "" --setting-sources "" --strict-mcp-config --no-session-persistence --output-format json < packet.md ``` The JSON output gives the model version each reader reported. A reader isolated in this way is - a cold reader. A reader whose process failed or returned nothing, or that shared the editor's - context, such as an agent dispatched from the editor's own session, is set aside and its - answers do not count; no reader is set aside for any other cause, and the record shows the - cause. If fewer than five readers' answers count, the run is void and a fresh cohort runs, - unless the readers whose answers count already fail a criterion, when the run fails. A void run - takes a run number, has a record and posted sheets like any other, and counts towards the - number of runs. + a cold reader. A reader whose answer text is empty, or that shared the editor's context, such + as an agent dispatched from the editor's own session, is set aside and its answers do not + count; no reader is set aside for any other cause, and the record shows the cause. If any + reader is set aside, the run is void and a fresh cohort runs, unless the readers whose answers + count already fail a criterion, when the run fails; a run never passes with a reader set + aside. A void run takes a run number, has a record and posted sheets like any other, and + counts towards the number of runs. - Each reader receives **only its packet**: the document's question set, the changed document and the fixture the questions name, each in full in the reader's prompt, the question set @@ -44,9 +44,6 @@ test is run with a low-strength model so that clarity is measured at the floor, - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. -The isolation rule above and the answer sheets posted under [The record](#the-record) apply from -run 19; runs 16 to 18 stand as recorded. - ## The question set Written before the run, with an **answer key** for every keyed question drawn verbatim from the @@ -96,13 +93,13 @@ instruction for that reason, and neither is reinstated. The editor marks each answer sheet against the document's key, or delegates the marking to a separate marker that has seen neither the drafting nor the review. Whoever marks is the marker -in this file. A delegated marker receives only these: the document at the commit the readers -read, its question set, the fixture the questions name and its key; this file and -`acceptance/runs/README.md`, with earlier run records of other pull requests as examples of -style only; the answer sheets; and, from the editor, the facts the record states that the sheets -do not, which are the pull request and the commit, the cohort size and model, how the readers -were isolated and the command they were started with, how the marker was started, and the run -number. The marker writes the record, stating those facts as the editor supplied them, and the +in this file. A delegated marker receives only these: the document, its question set, the +fixture the questions name and its key, each at the commit the readers read; this file and +`acceptance/runs/README.md`; the earlier run records of the same pull request, and those of +other pull requests as examples of style only; the answer sheets; and, from the editor, the +facts the record states that the sheets do not, which are the pull request and the commit, the +cohort size and model, how the readers were isolated and the command they were started with, the +readers set aside and why, how the marker was started, and the run number. The marker writes the record, stating those facts as the editor supplied them, and the editor adds the findings applied and the link to the posted sheets after marking. A delegated marker's verdict, its marks and its PASS or FAIL, is final unless the editor shows that a rule in this file was misapplied; the editor then re-marks what the misapplied rule affected, and the @@ -110,28 +107,30 @@ record says which rule and why. Each sheet is marked question by question, as fo - **Keyed items.** Question 1 as a whole is one keyed item, and so is each item of question 2 and each clause verdict and each summary verdict of question 3. -- **Quoting.** A reader quotes the document when it gives five or more consecutive words of the - document, verbatim apart from markdown link syntax and emphasis markers, as a quotation, in - quotation marks or a block quote. Naming a clause number, or paraphrasing, is not quoting. +- **Quoting.** A reader quotes the document when it gives consecutive words of the document, in + the document's order and of any length, as a quotation, in quotation marks or a block quote. + The words are verbatim apart from letter case, quotation-mark or apostrophe style, and markdown + link or emphasis markup. Naming a clause number, or paraphrasing, is not quoting. - **Questions 1 and 3** are marked as the key states for each item. -- **Question 2.** An answer is correct when its sentence states what the key's answer states, - including anything the key says an answer is wrong to omit, asserts nothing the key says makes - an answer wrong, and quotes the document in support. The quotation need not be the one the key - gives. -- **A miss traceable to the text** is a miss whose reasoning quotes the document or states, in - the reader's own words, what the document says or requires, or a miss the key says counts - against the document. Such a miss is traceable even where it also misreads the fixture, and - whatever the key says of it: the key can add to the misses that count against the document, - never take one away. Only a miss whose reasoning states nothing about the document, such as a - misread fixture alone, is the reader's. Question 1 has no reasoning, so a miss there is - traceable only where the key says it counts. +- **Question 2.** An answer is correct when its sentence states each thing the key's answer + states, unless the key says it may be left out, asserts nothing the key says makes an answer + wrong, and quotes the document in support. The quotation need not be the one the key gives. +- **A miss traceable to the text**, under questions 2 and 3, is a miss whose reasoning quotes the + document or states, in the reader's own words, what the document says or requires. Such a miss + is traceable even where it also misreads the fixture, and whatever the key says of it. Only a + miss whose reasoning states nothing about the document, such as a misread fixture alone, is the + reader's. A miss under any question is also traceable where the key says it counts against the + document: the key can add to these misses, never take one away. Any other miss on question 1 + is the reader's. - **An unanswered question.** Each keyed item left unanswered is one miss, not traceable to the text, so a sheet that leaves three items of question 3 blank has three misses. Question 4 left - unanswered counts as no passage quoted. A truncated sheet is marked as it stands; a reader - whose process failed or returned nothing is set aside, as [The cohort](#the-cohort) says. -- **The same passage.** A quotation under question 4 covers every sentence of the document it - quotes any part of, in every place the quoted words appear, and a passage named without a - quotation, by clause number or description, covers every sentence of what it names. A heading, + unanswered counts as no passage quoted. A sheet with any answer text is marked as it stands, + whatever its process returned; one whose answer text is empty is set aside, as + [The cohort](#the-cohort) says. +- **The same passage.** A quotation under question 4, of any length, covers every sentence + containing any part of the quoted words taken as one run of words, in every place that run + appears, and a passage named without a quotation, by clause number or description, covers + every sentence of what it names. A heading, list item or table row counts as one sentence. Two passages are the same when they cover a sentence in common or lie in the same paragraph, and a list item or table row is its own paragraph. @@ -146,13 +145,13 @@ The change passes when all of the following hold on a single run. The thresholds readers whose answers count; with five such readers, "every reader" is five of five and "all but one" is four of five. -| Criterion | Threshold | -| ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | -| Load-bearing statements (question 2) | **Every** reader correct on every one | -| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | -| A miss traceable to the text | **None.** Any such miss, as [Marking](#marking) defines it, is a finding, whatever the count | -| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort on the changed text passes this row | -| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | +| Criterion | Threshold | +| ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Load-bearing statements (question 2) | **Every** reader correct on every one | +| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | +| A miss traceable to the text | **None.** Any such miss, as [Marking](#marking) defines it, is a finding, whatever the count | +| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage of the changed text that is the same as it, as [Marking](#marking) defines the same passage | +| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | Every row must hold, and the last covers only what none of the others catches. The first, third and fourth rows block on what a single reader, or for the fourth two readers, did, and the second @@ -184,34 +183,38 @@ something. ## When it runs -On the final text of each pull request, after the editor's rewrite and any key update, and again -after every failure, void run or wording change. It is the last gate before the merge to `next`, -so that every change on `next` is one the cohort has accepted, and the release branch needs no -cohort of its own unless it changes specification text. A wording change is any change to the -document's text after the readers read it, from any source (a review finding, the editor, or -`next` merged into the branch), except the lines `PUBLISHING.md` and step 5 of `CONTRIBUTING.md` -have the editor set for a version: the header, the status paragraph and the citation. A wording -change means a fresh run; readers are never reused. +On the final text of each pull request, after the editor's rewrite and any key update. A fresh +cohort runs after every change to the text that follows a failure, after every void run, and +after every wording change; a text that failed is never run again unchanged. It is the last gate +before the merge to `next`, so that every change on `next` is one the cohort has accepted, and +the release branch needs no cohort of its own unless it changes specification text. A wording +change is any change to the document's text after the readers read it, from any source (a +review finding, the editor, or `next` merged into the branch), except a version number or date +that `PUBLISHING.md` or step 5 of `CONTRIBUTING.md` has the editor set, in the header, the status +section, the citation or the closing table. Any other change to those paragraphs is a wording +change. Readers are never reused. ## The record Each run has a file under `acceptance/runs/`, named `NNN-.md` with `NNN` the next number in the one sequence shared by the three documents, stating: the document, the pull request and the commit read, the cohort size and model, with the model version each reader reported, how the -readers were isolated and the command they were started with, the run number in that pull -request, who marked and, for a delegated marker, how it was started, the marks per reader per -question, any judgement made in marking, the passages quoted as confusing and by how many -readers, the findings applied, whether it passed, and a link to the answer sheets. The answer -sheets are not kept in this repository: every reader's sheet, unedited, with a set-aside -reader's marked as set aside, and the instruction a delegated marker was given, which is its -task prompt and the agent definition it ran under, are posted in one or more consecutive -comments on the pull request, each linked from the record, so that an owner can check the marks. -The comments also hold the exact prompt every reader received and the command and the form of -the working directory the readers were started in; a sheet cannot show how its process was -started, so the record says that the editor attests the rest of the isolation. A comment is not -edited after posting, and a record whose comment was edited is void; that is checkable only -through GitHub's edit history, which is the limit of keeping the sheets outside the repository. -The changelog line for the change names the file. CI refuses a specification change whose pull +readers were isolated and the command they were started with, whether the machine applies +managed (policy) settings to the readers, that the editor attests the rest of the isolation, the +readers set aside and the cause, the run number in that pull request, counting void runs, who +marked and, for a delegated marker, how it was started, the marks per reader per question, any +judgement made in marking, on a re-mark which rule and why, the passages quoted as confusing and +by how many readers, the findings applied, whether it was made under a process change before +that change merged, whether it passed, and a link to the answer sheets. The answer sheets are +not kept in this repository: every reader's sheet, which is its whole output as its process +returned it (the JSON, where the command asks for JSON), with a set-aside reader's marked as set +aside, and the instruction a delegated marker was given, which is its task prompt and the agent +definition it ran under, are posted in one or more comments on the pull request, each linked +from the record, so that an owner can check the marks. The comments also hold the exact prompt +every reader received and the command and the form of the working directory the readers were +started in; a sheet cannot show how its process was started, which is why the editor attests the +rest of the isolation. The comments are never edited, hidden or deleted after posting; if one +is, the run is void and a fresh cohort runs. The changelog line for the change names the file. CI refuses a specification change whose pull request adds no run file; it checks for at least one, and a pull request that changes two documents adds a run for each. @@ -229,8 +232,9 @@ a change to this process. They are not specifications, and a change to them is n cohort: the cohort measures whether a low-strength reader can follow a specification, and the process is followed by the editor. Such a change names the runs or the case that showed the need for it, and goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, with no scope, key or cohort -step; its review reads every process file it changes. It applies from its merge into `next`; a -run made before then may follow it, and its record says so. It is recorded by a line added at +step; its review reads every process file it changes. It names the run number it applies from; +a run under it made before it merges follows all of it, never part, and its record says so. It +is recorded by a line added at the top of [acceptance/CHANGES.md](acceptance/CHANGES.md), naming its pull request, not in the changelog, which is for the three specifications. It is bound by [the section on the bar](#the-bar-only-moves-up): it may raise a criterion, never relax one. A diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 81a9ac7..69550db 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -6,42 +6,37 @@ not a specification and is not versioned, and a change to it is not tested by a to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). - Pull request #PR: the cohort can be run and marked without judgement calls that the rules - leave open. A cohort is an odd number of readers started, at least five, larger at the - editor's choice the more obligations the change touches. Each reader is a separate process - with no tools and no project context, started from an empty directory outside any repository - with no settings files, `CLAUDE.md`, skills, plugins, hooks, MCP servers or saved session; the - example command takes the packet on standard input. A reader is set aside only if its process - failed or returned nothing or it shared the editor's context; a run left with fewer than five - counting readers is void unless those readers already fail it, and a void run has a number, a - record and posted sheets and counts towards the number of runs. The packet is the question - set, the document and the fixture, each after one `===== FILE: =====` line, with - nothing else added. A new marking section says what a delegated marker receives, that it - writes the record and its marks and PASS or FAIL are final unless the editor shows a rule was - misapplied, that open judgements take the reading that records more misses or findings, what - a keyed item is, that quoting is five or more consecutive verbatim words given as a quotation, - what a correct load-bearing answer needs, that a miss is traceable to the text when its - reasoning quotes or states what the document says or the key says it counts, and that a key - can add such misses, never remove one, how unanswered items and truncated sheets count, and - that two confusing passages are the same when they share a sentence or a paragraph. The - thresholds apply to the readers whose answers count, read "every reader" and "all but one", - and block on a single reader for the first, third and fourth rows; the fourth row's block lifts - when a fresh cohort on the changed text passes it. A wording change is any change to the text - after the readers read it, except the lines set for a version, and the cohort runs again after - every failure, void run or wording change. The run record states the model version each reader - reported, how readers were isolated, the command and who marked, and links the answer sheets, - the exact prompt and the marker's instruction, posted unedited in comments on the pull request - rather than kept in the repository, with the rest of the isolation attested by the editor; the - isolation rule and the posted sheets apply from run 19, and runs 16 to 18 stand as recorded. - The three keys no longer limit which misses count against the document, and removing such a - limit is named as raising the bar. The process files include the checks that enforce them, and - a process change goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, applies on its merge into - `next` and is recorded in this file, not in the changelog. `CONTRIBUTING.md` requires a - reviewer who did not draft the change, named in the pull request with the review, points to - the definition of a wording change, and adds the rebuild of the files - `build-agent-surfaces.py` writes after an Appendix A change. Prompted by gaps in the text as it - stood: `ACCEPTANCE.md` named no fixture in what a reader receives whilst - `acceptance/README.md` did, the thresholds were written as five of five and four of five for a - cohort of at least five, two pass-criteria rows both covered a single reader's traceable miss, - and nothing settled an unanswered question or "the same passage". [Run + leave open; this applies from run 19, and runs 16 to 18 stand as recorded. A cohort is an odd + number of readers started, at least five, larger at the editor's choice the more obligations + the change touches, each a separate process with no tools and no project context, started from + an empty directory outside any repository with no user, project or local settings files, + `CLAUDE.md`, skills, plugins, hooks, MCP servers or saved session. A reader is set aside only + if its answer text is empty or it shared the editor's context, and a run with a reader set + aside is void, or fails if the readers who count already fail it; a void run has a number, a + record and posted sheets. The packet is the question set, the document and the fixture, each + after one `===== FILE: =====` line. A new marking section says what a delegated marker + receives and that its verdict is final unless a rule was misapplied, that open judgements take + the stricter reading, and defines a keyed item, quoting (consecutive words of any length, + verbatim apart from case, quotation-mark style and markup), a correct load-bearing answer, a + miss traceable to the text, which a key can add to but never remove, an unanswered item and + the same passage. The thresholds apply to the readers whose answers count and block on a single + reader for the first, third and fourth rows, and the fourth row says what a fresh cohort no + longer naming a passage means. A fresh cohort runs after every change following a failure, + every void run and every wording change, which is any change to the text after the readers + read it except a version number or date the editor sets for a release. The record adds the + model version each reader reported, the isolation, managed settings, readers set aside, void + runs and re-marks, and links each reader's whole output and the marker's instruction, posted on + the pull request and never edited, hidden or deleted, or the run is void. The keys are marked at + the commit the readers read and no longer limit which misses count against the document, + removing such a limit is named as raising the bar, and the method key's "half right" is a miss. + The process files include the checks that enforce them, and a process change goes through + steps 1, 3 and 6 of `CONTRIBUTING.md`, names the run it applies from and is recorded here, not + in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the change, points to + the definition of a wording change, and adds the rebuild of the files `build-agent-surfaces.py` + writes after an Appendix A change. Prompted by gaps in the text as it stood: `ACCEPTANCE.md` + named no fixture in what a reader receives whilst `acceptance/README.md` did, the thresholds + were written as five of five and four of five for a cohort of at least five, two pass-criteria + rows both covered a single reader's traceable miss, and nothing settled an unanswered question + or "the same passage". [Run 16](runs/016-deferral-clause-run-1.md), [run 17](runs/017-deferral-clause-run-2.md) and [run 18](runs/018-deferral-clause-run-3.md) ruled on these case by case. diff --git a/acceptance/README.md b/acceptance/README.md index 9af1d2f..0e43599 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -35,6 +35,8 @@ matches the document is a defect in this directory, not a reason to mark a reade Questions are fixed so that runs can be compared across versions. Adding a keyed question is allowed when a new clause is load-bearing; removing or softening one to make a run pass is not. +A process change may remove a limit a key puts on which misses count against the document, as +[the section on the bar](../ACCEPTANCE.md#the-bar-only-moves-up) allows. The reader may quote a passage it found confusing; it is never asked to rate the document, say what would make a passage clearer, or propose a change: the cohort confirms comprehension and does not design the text. diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 6de0386..ee01416 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -1,6 +1,7 @@ # Answer key for DETECTOR-SPEC.md -Marked against the document at the version named in the changelog entry that records the run. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, update the quotation here in the same commit. @@ -65,12 +66,9 @@ has read the clause differently and that counts against the document. Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**. It is a SHOULD, and Lintomatic's failure to check its own identifiers in CI is a missed SHOULD, not a further MUST failure; the MUST that fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). -A reader who marks [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) No because the comment takes no reason has confused [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) with [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), which is -SHOULD; that counts against the document if the reasoning quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason). - ## 4. Confusion No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to +finding that blocks until the text is changed and a fresh cohort no longer names it. What to change is the editor's decision; the reader is not asked and any suggestion it volunteers is discarded unread. diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 71f3935..837b819 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -1,6 +1,7 @@ # Answer key for SPEC.md -Marked against the document at the version named in the changelog entry that records the run. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, update the quotation here in the same commit. @@ -59,7 +60,7 @@ Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**, because the sweep is what failed has missed [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation). Section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): the **inline ignore on `Report::render()`** was the owner's decision, not the -practitioner's. An answer naming instead the decision not to build the wider rule is half right: +practitioner's. An answer naming instead the decision not to build the wider rule is half right, which is a miss: that is also listed in section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) as the owner's to take, and an answer naming both is correct. An answer naming neither is wrong. @@ -72,6 +73,6 @@ passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as cove ## 4. Confusion No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to +finding that blocks until the text is changed and a fresh cohort no longer names it. What to change is the editor's decision; the reader is not asked and any suggestion it volunteers is discarded unread. diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 9a20bad..4af8149 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -7,15 +7,18 @@ the toolchain specification; numbering here continues from 16. A record states what `ACCEPTANCE.md` lists, in this order: the document; the pull request and the commit the readers were given; the cohort size and model, with the model version each reader -reported; how the readers were isolated and the command they were started with; the run number -within that pull request; who marked and, for a delegated marker, how it was started; the marks -per reader per question against the key, with any judgement made in marking; the passages quoted -as confusing and by how many readers; the findings applied to the text as a result; and the +reported; how the readers were isolated and the command they were started with; whether the +machine applies managed (policy) settings to the readers; that the editor attests the rest of the +isolation; the readers set aside and the cause; the run number within that pull request, counting +void runs; who marked and, for a delegated marker, how it was started; the marks per reader per +question against the key, with any judgement made in marking and, on a re-mark, which rule and +why; the passages quoted as confusing and by how many readers; the findings applied to the text +as a result; whether the run was made under a process change before that change merged; and the verdict against the criteria in `ACCEPTANCE.md`, with a link to the answer sheets. Earlier records show the style only; where one differs from this order, this file wins. Answer sheets -are not kept in this repository, and the marks are the record; the sheets, unedited, and the -instruction a delegated marker was given are posted in one or more consecutive comments on the -pull request, each linked from the record. +are not kept in this repository, and the marks are the record; the sheets, each the reader's +whole output, and the instruction a delegated marker was given are posted in one or more +comments on the pull request, each linked from the record, and never edited, hidden or deleted. The `Unreleased` changelog line for the change names the record, and `tools/changes.py` refuses a specification change whose pull request adds no file here. diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 91112fa..1568db2 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -1,6 +1,7 @@ # Answer key for TOOLING-SPEC.md -Marked against the document at the version named in the changelog entry that records the run. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, update the quotation here in the same commit. @@ -78,6 +79,6 @@ or the detector document's section [8](../../DETECTOR-SPEC.md#8-conformance) dif ## 4. Confusion No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to +finding that blocks until the text is changed and a fresh cohort no longer names it. What to change is the editor's decision; the reader is not asked and any suggestion it volunteers is discarded unread. From dba74615373fbce126b640e38275227d8a64f33a Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 21:13:54 +0100 Subject: [PATCH 07/19] Acceptance process: apply the editor's rulings on review 4 A question 1 bullet that quotes the document makes its miss traceable; a reasoning that names the clause a case fails or meets states what the document requires; a question 2 quotation must itself state part of the key's answer. The fourth row matches a fresh reader's passage against the earlier passage's paragraph as it reads after the change, the record quotes confusing passages verbatim, and a delegated marker gets the earlier runs' documents and the three facts it was missing. An edited comment voids only a run not already failed, before merge. A process change applies to runs after it merges, or to earlier runs made under it at a named commit. A set-aside reader's traceable miss fails the run. The keys count passages named, not only quoted; every keyed answer of question 3 is a keyed item; "the same passage" covers the sentences an occurrence of the whole quotation overlaps. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 81 ++++++++++++++++++++----------------- acceptance/CHANGES.md | 23 ++++++----- acceptance/detector/KEY.md | 8 ++-- acceptance/method/KEY.md | 8 ++-- acceptance/runs/README.md | 5 ++- acceptance/toolchain/KEY.md | 8 ++-- 6 files changed, 73 insertions(+), 60 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index bab883d..7b5663a 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -30,7 +30,8 @@ test is run with a low-strength model so that clarity is measured at the floor, count; no reader is set aside for any other cause, and the record shows the cause. If any reader is set aside, the run is void and a fresh cohort runs, unless the readers whose answers count already fail a criterion, when the run fails; a run never passes with a reader set - aside. A void run takes a run number, has a record and posted sheets like any other, and + aside. A set-aside reader's sheet is still marked, and a miss traceable to the text in it fails + the run. A void run takes a run number, has a record and posted sheets like any other, and counts towards the number of runs. - Each reader receives **only its packet**: the document's question set, the changed document @@ -96,17 +97,21 @@ separate marker that has seen neither the drafting nor the review. Whoever marks in this file. A delegated marker receives only these: the document, its question set, the fixture the questions name and its key, each at the commit the readers read; this file and `acceptance/runs/README.md`; the earlier run records of the same pull request, and those of -other pull requests as examples of style only; the answer sheets; and, from the editor, the -facts the record states that the sheets do not, which are the pull request and the commit, the -cohort size and model, how the readers were isolated and the command they were started with, the -readers set aside and why, how the marker was started, and the run number. The marker writes the record, stating those facts as the editor supplied them, and the -editor adds the findings applied and the link to the posted sheets after marking. A delegated -marker's verdict, its marks and its PASS or FAIL, is final unless the editor shows that a rule -in this file was misapplied; the editor then re-marks what the misapplied rule affected, and the -record says which rule and why. Each sheet is marked question by question, as follows. +other pull requests as examples of style only; the document at the commit each earlier run of +the pull request read; the answer sheets; and, from the editor, the facts the record states that +the sheets do not, which are the pull request and the commit, the cohort size and model, how the +readers were isolated and the command they were started with, whether the machine applies +managed (policy) settings to the readers, that the editor attests the rest of the isolation, the +readers set aside and why, how the marker was started, the run number, and whether the run was +made under a process change before that change merged and, if so, at which commit. The marker +writes the record, stating those facts as the editor supplied them, and the editor adds the +findings applied and the link to the posted sheets after marking. A delegated marker's verdict, +its marks and its PASS or FAIL, is final unless the editor shows that a rule in this file was +misapplied; the editor then re-marks what the misapplied rule affected, and the record says +which rule and why. Each sheet is marked question by question, as follows. - **Keyed items.** Question 1 as a whole is one keyed item, and so is each item of question 2 - and each clause verdict and each summary verdict of question 3. + and each answer of question 3 for which the key gives an intended answer. - **Quoting.** A reader quotes the document when it gives consecutive words of the document, in the document's order and of any length, as a quotation, in quotation marks or a block quote. The words are verbatim apart from letter case, quotation-mark or apostrophe style, and markdown @@ -114,24 +119,27 @@ record says which rule and why. Each sheet is marked question by question, as fo - **Questions 1 and 3** are marked as the key states for each item. - **Question 2.** An answer is correct when its sentence states each thing the key's answer states, unless the key says it may be left out, asserts nothing the key says makes an answer - wrong, and quotes the document in support. The quotation need not be the one the key gives. + wrong, and quotes in support words of the document that by themselves state at least one + thing the key's answer states. The quotation need not be the one the key gives. - **A miss traceable to the text**, under questions 2 and 3, is a miss whose reasoning quotes the document or states, in the reader's own words, what the document says or requires. Such a miss is traceable even where it also misreads the fixture, and whatever the key says of it. Only a miss whose reasoning states nothing about the document, such as a misread fixture alone, is the - reader's. A miss under any question is also traceable where the key says it counts against the - document: the key can add to these misses, never take one away. Any other miss on question 1 - is the reader's. + reader's. A reasoning that only reports facts of the fixture, attributing no requirement, + condition or meaning to the document or a clause of it, states nothing about the document; + naming the clause a case fails or meets attributes a requirement to it. A miss under any + question is also traceable where the key says it counts against the document: the key can add + to these misses, never take one away. A miss on question 1 is also traceable where the bullet + it turns on quotes the document; any other miss on question 1 is the reader's. - **An unanswered question.** Each keyed item left unanswered is one miss, not traceable to the text, so a sheet that leaves three items of question 3 blank has three misses. Question 4 left unanswered counts as no passage quoted. A sheet with any answer text is marked as it stands, whatever its process returned; one whose answer text is empty is set aside, as [The cohort](#the-cohort) says. -- **The same passage.** A quotation under question 4, of any length, covers every sentence - containing any part of the quoted words taken as one run of words, in every place that run - appears, and a passage named without a quotation, by clause number or description, covers - every sentence of what it names. A heading, - list item or table row counts as one sentence. Two passages are the same when they cover a +- **The same passage.** A quotation under question 4, of any length, covers every sentence that + an occurrence of the quoted words, taken as one run, overlaps, at every place that run occurs, + and a passage named without a quotation, by clause number or description, covers every + sentence of what it names. A heading, list item or table row counts as one sentence. Two passages are the same when they cover a sentence in common or lie in the same paragraph, and a list item or table row is its own paragraph. @@ -145,13 +153,13 @@ The change passes when all of the following hold on a single run. The thresholds readers whose answers count; with five such readers, "every reader" is five of five and "all but one" is four of five. -| Criterion | Threshold | -| ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Load-bearing statements (question 2) | **Every** reader correct on every one | -| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | -| A miss traceable to the text | **None.** Any such miss, as [Marking](#marking) defines it, is a finding, whatever the count | -| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage of the changed text that is the same as it, as [Marking](#marking) defines the same passage | -| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | +| Criterion | Threshold | +| ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Load-bearing statements (question 2) | **Every** reader correct on every one | +| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | +| A miss traceable to the text | **None.** Any such miss, as [Marking](#marking) defines it, is a finding, whatever the count | +| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage that covers a sentence of the paragraph the earlier passage lay in, as that paragraph reads in the changed document, or a sentence the change kept from it | +| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | Every row must hold, and the last covers only what none of the others catches. The first, third and fourth rows block on what a single reader, or for the fourth two readers, did, and the second @@ -203,10 +211,10 @@ readers were isolated and the command they were started with, whether the machin managed (policy) settings to the readers, that the editor attests the rest of the isolation, the readers set aside and the cause, the run number in that pull request, counting void runs, who marked and, for a delegated marker, how it was started, the marks per reader per question, any -judgement made in marking, on a re-mark which rule and why, the passages quoted as confusing and -by how many readers, the findings applied, whether it was made under a process change before -that change merged, whether it passed, and a link to the answer sheets. The answer sheets are -not kept in this repository: every reader's sheet, which is its whole output as its process +judgement made in marking, on a re-mark which rule and why, the passages quoted as confusing, +each with the words quoted, verbatim, and its clause, and by how many readers, the findings +applied, whether it was made under a process change before that change merged and, if so, at +which commit, whether it passed, and a link to the answer sheets. The answer sheets are not kept in this repository: every reader's sheet, which is its whole output as its process returned it (the JSON, where the command asks for JSON), with a set-aside reader's marked as set aside, and the instruction a delegated marker was given, which is its task prompt and the agent definition it ran under, are posted in one or more comments on the pull request, each linked @@ -214,8 +222,9 @@ from the record, so that an owner can check the marks. The comments also hold th every reader received and the command and the form of the working directory the readers were started in; a sheet cannot show how its process was started, which is why the editor attests the rest of the isolation. The comments are never edited, hidden or deleted after posting; if one -is, the run is void and a fresh cohort runs. The changelog line for the change names the file. CI refuses a specification change whose pull -request adds no run file; it checks for at least one, and a pull request that changes two +is before the change merges, the run is void and a fresh cohort runs, unless the run had already +failed, when it stays failed. The changelog line for the change names the file. CI refuses a +specification change whose pull request adds no run file; it checks for at least one, and a pull request that changes two documents adds a run for each. Each published version's changelog entry carries the lines of the changes it contains, so a @@ -232,10 +241,10 @@ a change to this process. They are not specifications, and a change to them is n cohort: the cohort measures whether a low-strength reader can follow a specification, and the process is followed by the editor. Such a change names the runs or the case that showed the need for it, and goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, with no scope, key or cohort -step; its review reads every process file it changes. It names the run number it applies from; -a run under it made before it merges follows all of it, never part, and its record says so. It -is recorded by a line added at -the top of [acceptance/CHANGES.md](acceptance/CHANGES.md), naming its pull request, not in the +step; its review reads every process file it changes. It applies to every run made after it +merges; a run made before then may be made under it, follows all of it at a named commit, never +part, and its record says so and names that commit. It is recorded by a line added at the top +of [acceptance/CHANGES.md](acceptance/CHANGES.md), naming its pull request, not in the changelog, which is for the three specifications. It is bound by [the section on the bar](#the-bar-only-moves-up): it may raise a criterion, never relax one. A change to a question set, key or fixture is not a change to this process; `acceptance/README.md` diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 69550db..b555ee6 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -6,32 +6,35 @@ not a specification and is not versioned, and a change to it is not tested by a to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). - Pull request #PR: the cohort can be run and marked without judgement calls that the rules - leave open; this applies from run 19, and runs 16 to 18 stand as recorded. A cohort is an odd + leave open; this applies to runs made after it merges, and to run 19 if made under it, and + runs 16 to 18 stand as recorded. A cohort is an odd number of readers started, at least five, larger at the editor's choice the more obligations the change touches, each a separate process with no tools and no project context, started from an empty directory outside any repository with no user, project or local settings files, `CLAUDE.md`, skills, plugins, hooks, MCP servers or saved session. A reader is set aside only if its answer text is empty or it shared the editor's context, and a run with a reader set - aside is void, or fails if the readers who count already fail it; a void run has a number, a - record and posted sheets. The packet is the question set, the document and the fixture, each + aside is void, or fails if the readers who count already fail it or its own sheet has a miss + traceable to the text; a void run has a number, a record and posted sheets. The packet is the question set, the document and the fixture, each after one `===== FILE: =====` line. A new marking section says what a delegated marker receives and that its verdict is final unless a rule was misapplied, that open judgements take the stricter reading, and defines a keyed item, quoting (consecutive words of any length, - verbatim apart from case, quotation-mark style and markup), a correct load-bearing answer, a - miss traceable to the text, which a key can add to but never remove, an unanswered item and - the same passage. The thresholds apply to the readers whose answers count and block on a single + verbatim apart from case, quotation-mark style and markup), a correct load-bearing answer, + whose quotation must itself state part of the key's answer, a miss traceable to the text, + including a question 1 bullet that quotes the document, which a key can add to but never + remove, an unanswered item and the same passage. The thresholds apply to the readers whose answers count and block on a single reader for the first, third and fourth rows, and the fourth row says what a fresh cohort no longer naming a passage means. A fresh cohort runs after every change following a failure, every void run and every wording change, which is any change to the text after the readers read it except a version number or date the editor sets for a release. The record adds the model version each reader reported, the isolation, managed settings, readers set aside, void - runs and re-marks, and links each reader's whole output and the marker's instruction, posted on - the pull request and never edited, hidden or deleted, or the run is void. The keys are marked at + runs, re-marks and each confusing passage's words verbatim, and links each reader's whole + output and the marker's instruction, posted on the pull request and never edited, hidden or + deleted, or a run not already failed is void. The keys are marked at the commit the readers read and no longer limit which misses count against the document, removing such a limit is named as raising the bar, and the method key's "half right" is a miss. The process files include the checks that enforce them, and a process change goes through - steps 1, 3 and 6 of `CONTRIBUTING.md`, names the run it applies from and is recorded here, not - in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the change, points to + steps 1, 3 and 6 of `CONTRIBUTING.md`, applies to runs made after it merges, or earlier + ones made under it at a named commit, and is recorded here, not in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the change, points to the definition of a wording change, and adds the rebuild of the files `build-agent-surfaces.py` writes after an Appendix A change. Prompted by gaps in the text as it stood: `ACCEPTANCE.md` named no fixture in what a reader receives whilst `acceptance/README.md` did, the thresholds diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index ee01416..8608f7d 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -68,7 +68,7 @@ own identifiers in CI is a missed SHOULD, not a further MUST failure; the MUST t ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer names it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage quoted. Two or more readers naming the same passage, as +`ACCEPTANCE.md` defines it, is a finding that blocks until the text is changed and a fresh +cohort no longer names it. What to change is the editor's decision; the reader is not asked +and any suggestion it volunteers is discarded unread. diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 837b819..2e37b07 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -72,7 +72,7 @@ passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as cove ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer names it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage quoted. Two or more readers naming the same passage, as +`ACCEPTANCE.md` defines it, is a finding that blocks until the text is changed and a fresh +cohort no longer names it. What to change is the editor's decision; the reader is not asked +and any suggestion it volunteers is discarded unread. diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 4af8149..2d0b96a 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -12,8 +12,9 @@ machine applies managed (policy) settings to the readers; that the editor attest isolation; the readers set aside and the cause; the run number within that pull request, counting void runs; who marked and, for a delegated marker, how it was started; the marks per reader per question against the key, with any judgement made in marking and, on a re-mark, which rule and -why; the passages quoted as confusing and by how many readers; the findings applied to the text -as a result; whether the run was made under a process change before that change merged; and the +why; the passages quoted as confusing, each with the words quoted, verbatim, and its clause, and +by how many readers; the findings applied to the text as a result; whether the run was made +under a process change before that change merged and, if so, at which commit; and the verdict against the criteria in `ACCEPTANCE.md`, with a link to the answer sheets. Earlier records show the style only; where one differs from this order, this file wins. Answer sheets are not kept in this repository, and the marks are the record; the sheets, each the reader's diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 1568db2..cae0efc 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -78,7 +78,7 @@ or the detector document's section [8](../../DETECTOR-SPEC.md#8-conformance) dif ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer names it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage quoted. Two or more readers naming the same passage, as +`ACCEPTANCE.md` defines it, is a finding that blocks until the text is changed and a fresh +cohort no longer names it. What to change is the editor's decision; the reader is not asked +and any suggestion it volunteers is discarded unread. From 5a863aeb9d75ec83b27995359d200b1bb8d2b430 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 22:49:13 +0100 Subject: [PATCH 08/19] Acceptance process: apply the editor's rulings on review 5 - Question 2: a correct answer also asserts nothing the document does not state, so a wrong claim the key does not list is no longer correct. - A reader is set aside when its process reports an error, when the command or working directory recorded for it is not one The cohort requires, or when its reported model version is not the cohort's model; the existing void rule applies. The unanswered-question bullet agrees. - CONTRIBUTING.md step 5 sets the status paragraph and citation to the -dev version with the header, so the version exception covers them. - The sentence after the pass-criteria table agrees with the fourth row. - A delegated marker of a run made under an unmerged process change receives ACCEPTANCE.md and acceptance/runs/README.md at the named commit. - acceptance/CHANGES.md line updated to match. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 27 ++++++++++++++++----------- CONTRIBUTING.md | 3 ++- acceptance/CHANGES.md | 11 +++++++---- 3 files changed, 25 insertions(+), 16 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 7b5663a..8afed4b 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -25,9 +25,12 @@ test is run with a low-strength model so that clarity is measured at the floor, ``` The JSON output gives the model version each reader reported. A reader isolated in this way is - a cold reader. A reader whose answer text is empty, or that shared the editor's context, such - as an agent dispatched from the editor's own session, is set aside and its answers do not - count; no reader is set aside for any other cause, and the record shows the cause. If any + a cold reader. A reader is set aside, and its answers do not count, when its answer text is + empty; when its process reports an error, for example `is_error` in its JSON output or an API + error message in place of answers; when it shared the editor's context, such as an agent + dispatched from the editor's own session; when the command or working directory recorded for + it is not one this section requires; or when the model version it reported is not the + cohort's model. No reader is set aside for any other cause, and the record shows the cause. If any reader is set aside, the run is void and a fresh cohort runs, unless the readers whose answers count already fail a criterion, when the run fails; a run never passes with a reader set aside. A set-aside reader's sheet is still marked, and a miss traceable to the text in it fails @@ -96,7 +99,8 @@ The editor marks each answer sheet against the document's key, or delegates the separate marker that has seen neither the drafting nor the review. Whoever marks is the marker in this file. A delegated marker receives only these: the document, its question set, the fixture the questions name and its key, each at the commit the readers read; this file and -`acceptance/runs/README.md`; the earlier run records of the same pull request, and those of +`acceptance/runs/README.md`, at the named commit where the run is made under a process change +before that change merges; the earlier run records of the same pull request, and those of other pull requests as examples of style only; the document at the commit each earlier run of the pull request read; the answer sheets; and, from the editor, the facts the record states that the sheets do not, which are the pull request and the commit, the cohort size and model, how the @@ -118,8 +122,8 @@ which rule and why. Each sheet is marked question by question, as follows. link or emphasis markup. Naming a clause number, or paraphrasing, is not quoting. - **Questions 1 and 3** are marked as the key states for each item. - **Question 2.** An answer is correct when its sentence states each thing the key's answer - states, unless the key says it may be left out, asserts nothing the key says makes an answer - wrong, and quotes in support words of the document that by themselves state at least one + states, unless the key says it may be left out, asserts nothing the document does not state + and nothing the key says makes an answer wrong, and quotes in support words of the document that by themselves state at least one thing the key's answer states. The quotation need not be the one the key gives. - **A miss traceable to the text**, under questions 2 and 3, is a miss whose reasoning quotes the document or states, in the reader's own words, what the document says or requires. Such a miss @@ -133,9 +137,9 @@ which rule and why. Each sheet is marked question by question, as follows. it turns on quotes the document; any other miss on question 1 is the reader's. - **An unanswered question.** Each keyed item left unanswered is one miss, not traceable to the text, so a sheet that leaves three items of question 3 blank has three misses. Question 4 left - unanswered counts as no passage quoted. A sheet with any answer text is marked as it stands, - whatever its process returned; one whose answer text is empty is set aside, as - [The cohort](#the-cohort) says. + unanswered counts as no passage quoted. A sheet with any answer text is marked as it stands; + one whose answer text is empty, or whose process reports an error, is set aside, as + [The cohort](#the-cohort) says, and is still marked. - **The same passage.** A quotation under question 4, of any length, covers every sentence that an occurrence of the quoted words, taken as one run, overlaps, at every place that run occurs, and a passage named without a quotation, by clause number or description, covers every @@ -161,8 +165,9 @@ one" is four of five. | A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage that covers a sentence of the paragraph the earlier passage lay in, as that paragraph reads in the changed document, or a sentence the change kept from it | | Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | -Every row must hold, and the last covers only what none of the others catches. The first, third -and fourth rows block on what a single reader, or for the fourth two readers, did, and the second +Every row must hold, and the last covers only what none of the others catches. The first and +third rows block on what a single reader did, and the fourth on what two readers did and then +on what any one reader of a fresh cohort names, and the second row's tolerance of one reader never excuses a miss under the third. The asymmetry is deliberate. A load-bearing statement is the sentence the whole document exists diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 796be43..29a3772 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -70,7 +70,8 @@ survives, the change is yours. 5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text. Its record is committed under `acceptance/runs/`, the `Unreleased` changelog line names it, and CI refuses a specification change that lacks one. For an obligation, the editor sets the document's - header to `-dev, unpublished` in the same commit if it still names a published + header to `-dev, unpublished`, and the status paragraph and citation that state + its version to `-dev`, in the same commit if the header still names a published version. A wording change after the run means a fresh run. 6. **Merge** into `next` by merge commit, and the branch is deleted. diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index b555ee6..254c23d 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -12,20 +12,23 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). the change touches, each a separate process with no tools and no project context, started from an empty directory outside any repository with no user, project or local settings files, `CLAUDE.md`, skills, plugins, hooks, MCP servers or saved session. A reader is set aside only - if its answer text is empty or it shared the editor's context, and a run with a reader set + if its answer text is empty, its process reports an error, it shared the editor's context, it + was not started as required or it reported another model, and a run with a reader set aside is void, or fails if the readers who count already fail it or its own sheet has a miss traceable to the text; a void run has a number, a record and posted sheets. The packet is the question set, the document and the fixture, each after one `===== FILE: =====` line. A new marking section says what a delegated marker receives and that its verdict is final unless a rule was misapplied, that open judgements take the stricter reading, and defines a keyed item, quoting (consecutive words of any length, verbatim apart from case, quotation-mark style and markup), a correct load-bearing answer, - whose quotation must itself state part of the key's answer, a miss traceable to the text, + which asserts nothing the document does not state and whose quotation must itself state part + of the key's answer, a miss traceable to the text, including a question 1 bullet that quotes the document, which a key can add to but never remove, an unanswered item and the same passage. The thresholds apply to the readers whose answers count and block on a single - reader for the first, third and fourth rows, and the fourth row says what a fresh cohort no + reader for the first and third rows and on two readers for the fourth, and the fourth row says what a fresh cohort no longer naming a passage means. A fresh cohort runs after every change following a failure, every void run and every wording change, which is any change to the text after the readers - read it except a version number or date the editor sets for a release. The record adds the + read it except a version number or date the editor sets for a release, and step 5 sets the + status paragraph and citation to the `-dev` version with the header. The record adds the model version each reader reported, the isolation, managed settings, readers set aside, void runs, re-marks and each confusing passage's words verbatim, and links each reader's whole output and the marker's instruction, posted on the pull request and never edited, hidden or From b713023c70a4082da5786545eb407e6e3feaffab Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 22:56:58 +0100 Subject: [PATCH 09/19] Acceptance process: the commit named before readers start; sheet-visible set-asides by the marker Two minor points of review 6: a run made under an unmerged process change names its commit before the readers start, so the choice cannot follow the marks; and a delegated marker sets aside, with the cause, a reader whose sheet shows one, the editor supplying only causes the sheets cannot show. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 8afed4b..a6834a7 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -106,9 +106,10 @@ the pull request read; the answer sheets; and, from the editor, the facts the re the sheets do not, which are the pull request and the commit, the cohort size and model, how the readers were isolated and the command they were started with, whether the machine applies managed (policy) settings to the readers, that the editor attests the rest of the isolation, the -readers set aside and why, how the marker was started, the run number, and whether the run was +readers set aside for a cause the sheets cannot show and why, how the marker was started, the run number, and whether the run was made under a process change before that change merged and, if so, at which commit. The marker -writes the record, stating those facts as the editor supplied them, and the editor adds the +writes the record, stating those facts as the editor supplied them and setting aside, with the +cause, any reader whose sheet shows a cause, and the editor adds the findings applied and the link to the posted sheets after marking. A delegated marker's verdict, its marks and its PASS or FAIL, is final unless the editor shows that a rule in this file was misapplied; the editor then re-marks what the misapplied rule affected, and the record says @@ -247,8 +248,8 @@ cohort: the cohort measures whether a low-strength reader can follow a specifica process is followed by the editor. Such a change names the runs or the case that showed the need for it, and goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, with no scope, key or cohort step; its review reads every process file it changes. It applies to every run made after it -merges; a run made before then may be made under it, follows all of it at a named commit, never -part, and its record says so and names that commit. It is recorded by a line added at the top +merges; a run made before then may be made under it, follows all of it at a commit named before +the readers start, never part, and its record says so and names that commit. It is recorded by a line added at the top of [acceptance/CHANGES.md](acceptance/CHANGES.md), naming its pull request, not in the changelog, which is for the three specifications. It is bound by [the section on the bar](#the-bar-only-moves-up): it may raise a criterion, never relax one. A From a6950109940da0f7dfcf521752a6418cc3ac12db Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 22:57:27 +0100 Subject: [PATCH 10/19] Acceptance process: name pull request 13 in acceptance/CHANGES.md Co-Authored-By: Claude Opus 5.5 --- acceptance/CHANGES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 254c23d..68814cb 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -5,7 +5,7 @@ describes: a line is added at the top for each change, naming its pull request. not a specification and is not versioned, and a change to it is not tested by a cohort. Changes to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). -- Pull request #PR: the cohort can be run and marked without judgement calls that the rules +- [Pull request #13](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/13): the cohort can be run and marked without judgement calls that the rules leave open; this applies to runs made after it merges, and to run 19 if made under it, and runs 16 to 18 stand as recorded. A cohort is an odd number of readers started, at least five, larger at the editor's choice the more obligations From bdfd055069b65d40f51cbeb6e73af74b0d255410 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:12:27 +0100 Subject: [PATCH 11/19] Acceptance process: mark question 2 for its gist, judge open marks as a careful reader would Applies the editor's rulings after runs 19 and 20. Question 2 reads the reader's sentence and quotation together and marks the gist of the key's answer, failing only what the document does not state, what the key says makes an answer wrong and what the key says an answer is wrong to omit. An open judgement is made as a careful reader of the document would, not by the reading that records more misses. A passage under question 4 is one the reader found confusing, not one named only to explain another. Each key names what a question 2 answer is wrong to omit, from what it already marked. None of these rules was on next, so none relaxes it. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 19 +++++++++++-------- acceptance/CHANGES.md | 16 ++++++++++------ acceptance/detector/KEY.md | 3 +++ acceptance/method/KEY.md | 14 +++++++------- acceptance/toolchain/KEY.md | 3 +++ 5 files changed, 34 insertions(+), 21 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index a6834a7..1de652c 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -122,10 +122,13 @@ which rule and why. Each sheet is marked question by question, as follows. The words are verbatim apart from letter case, quotation-mark or apostrophe style, and markdown link or emphasis markup. Naming a clause number, or paraphrasing, is not quoting. - **Questions 1 and 3** are marked as the key states for each item. -- **Question 2.** An answer is correct when its sentence states each thing the key's answer - states, unless the key says it may be left out, asserts nothing the document does not state - and nothing the key says makes an answer wrong, and quotes in support words of the document that by themselves state at least one - thing the key's answer states. The quotation need not be the one the key gives. +- **Question 2.** An answer is the reader's sentence and its quotation read together. It is + correct when, so read, it states what the key's answer states, in the document's words or the + reader's own (a paraphrase, a synonym, or a defined term used in its defined sense states what + it means), asserts nothing the document does not state or the key says makes an answer wrong, + omits nothing the key says an answer is wrong to omit, and quotes the document in support. The + quotation need not be the one the key gives. The key names the elements an answer is wrong to + omit; for the rest, the marker reads for the gist of the key's answer, not a checklist. - **A miss traceable to the text**, under questions 2 and 3, is a miss whose reasoning quotes the document or states, in the reader's own words, what the document says or requires. Such a miss is traceable even where it also misreads the fixture, and whatever the key says of it. Only a @@ -141,16 +144,16 @@ which rule and why. Each sheet is marked question by question, as follows. unanswered counts as no passage quoted. A sheet with any answer text is marked as it stands; one whose answer text is empty, or whose process reports an error, is set aside, as [The cohort](#the-cohort) says, and is still marked. -- **The same passage.** A quotation under question 4, of any length, covers every sentence that +- **The same passage.** A passage under question 4 is one the reader says it found confusing, not + one it names only to explain another confusion. A quotation of one, of any length, covers every sentence that an occurrence of the quoted words, taken as one run, overlaps, at every place that run occurs, and a passage named without a quotation, by clause number or description, covers every sentence of what it names. A heading, list item or table row counts as one sentence. Two passages are the same when they cover a sentence in common or lie in the same paragraph, and a list item or table row is its own paragraph. -Where a mark needs a judgement these rules do not settle, the marker makes it, taking the stricter -reading, the one that records more misses or findings, and the record states the judgement and -why. +Where a mark needs a judgement these rules do not settle, the marker makes it as a careful reader +of the document would, and the record states the judgement and why. ## Pass criteria diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 68814cb..15ea7a2 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -17,11 +17,13 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). aside is void, or fails if the readers who count already fail it or its own sheet has a miss traceable to the text; a void run has a number, a record and posted sheets. The packet is the question set, the document and the fixture, each after one `===== FILE: =====` line. A new marking section says what a delegated marker - receives and that its verdict is final unless a rule was misapplied, that open judgements take - the stricter reading, and defines a keyed item, quoting (consecutive words of any length, - verbatim apart from case, quotation-mark style and markup), a correct load-bearing answer, - which asserts nothing the document does not state and whose quotation must itself state part - of the key's answer, a miss traceable to the text, + receives and that its verdict is final unless a rule was misapplied, that an open judgement is + made as a careful reader of the document would, and defines a keyed item, quoting (consecutive + words of any length, verbatim apart from case, quotation-mark style and markup), a correct + load-bearing answer, read with its quotation for the gist of the key's answer, asserting + nothing the document does not state and omitting nothing the key says it is wrong to omit, a + passage found confusing, which excludes one named only to explain another, a miss traceable + to the text, including a question 1 bullet that quotes the document, which a key can add to but never remove, an unanswered item and the same passage. The thresholds apply to the readers whose answers count and block on a single reader for the first and third rows and on two readers for the fourth, and the fourth row says what a fresh cohort no @@ -34,7 +36,9 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). output and the marker's instruction, posted on the pull request and never edited, hidden or deleted, or a run not already failed is void. The keys are marked at the commit the readers read and no longer limit which misses count against the document, - removing such a limit is named as raising the bar, and the method key's "half right" is a miss. + removing such a limit is named as raising the bar; the method key's "half right" is a miss, and + each key names what a question 2 answer is wrong to omit. No marking rule was on `next` + before, so none relaxes it. The process files include the checks that enforce them, and a process change goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, applies to runs made after it merges, or earlier ones made under it at a named commit, and is recorded here, not in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the change, points to diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 8608f7d..c0749a8 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -25,6 +25,9 @@ agent summary (those belong to the toolchain specification). ## 2. Load-bearing statements +No entry names an element an answer is wrong to omit, so each is marked for the gist of its +answer, as `ACCEPTANCE.md` says. + - **2a.** Conformance is satisfying every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression), judged on evidence; a detector that predates the document can conform, and the declaration is optional. Key quotation, section 8: "A Detector Conforms if it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression)." Supporting, [8.1](../../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps): "A diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 2e37b07..8839bc7 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -29,20 +29,20 @@ a test may serve as the detector; an agent may baseline the remaining instances. then sweep and fix every instance, in that order. Key quotations, section [3](../../SPEC.md#3-the-method): "Six clauses, in order."; [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance): "The Practitioner MUST run the Rule across the entire codebase and record the total Instance count before fixing anything, and MUST then fix every Instance found rather - than only the one". An answer that has the fix before the rule, or the sweep before the proof, - is wrong. + than only the one". An answer that omits the rule, the proof or the sweep, or has the fix + before the rule or the sweep before the proof, is wrong. - **2b.** The owner of the codebase, always a human; an agent may not. Key quotation, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): "An Agent MUST NOT Baseline, suppress or knowingly leave an Instance unfixed on its own - authority, whatever the Instance count turns out to be." An answer that lets the practitioner - decide because the count was small, or because the project recorded no rule, is wrong (the - default owner paragraph in the terminology entry for Owner still requires a recorded decision). + authority, whatever the Instance count turns out to be." An answer that omits the owner, or + lets the practitioner decide because the count was small or because the project recorded no + rule, is wrong (the default owner paragraph in the terminology entry for Owner still requires a + recorded decision). - **2c.** All six clauses of section [3](../../SPEC.md#3-the-method) were followed for that defect, and the verdict rests on the reviewer reproducing the red and green runs, not on the report. Key quotations, section [7](../../SPEC.md#7-conformance): "A remediation Conforms if all six clauses of section [3](../../SPEC.md#3-the-method) were followed for that Defect, and the verdict that they were rests on reproduction"; "A verdict on a remediation or a Defence MUST rest on reproduction, not on the report". An answer that accepts the pull request description - as the verdict is wrong, and so is one that states the six clauses and says nothing of - reproduction. + as the verdict is wrong, and so is one that omits reproduction. ## 3. Application: the ledger remediation diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index cae0efc..32b312c 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -30,6 +30,9 @@ conform; the agent summary (section [7](../../TOOLING-SPEC.md#7-agent-context)) ## 2. Load-bearing statements +No entry names an element an answer is wrong to omit, so each is marked for the gist of its +answer, as `ACCEPTANCE.md` says. + - **2a.** At the project level; the toolchain is whatever the project has assembled from third-party, first-party and project-level parts. Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "A project's Toolchain Conforms if every MUST in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record) holds across the assembled parts, wherever From 7f6a196657812b2e0e82dde4ce3973a1d8eb85c3 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:18:30 +0100 Subject: [PATCH 12/19] Acceptance process: review 7, every element of a question 2 answer counts, read for its gist A question 2 answer is wrong to omit any element of the key's answer that answers a part of the question, each read for its gist, as run 16 marked; the keys' "no entry names" lines go. A passage quoted under question 4 counts whatever the reader quoted it for, as on next. A delegated marker takes the process files from next where no process change is pending, and the record gives a description where a confusing passage was named without a quotation. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 13 ++++++------- acceptance/CHANGES.md | 12 +++++------- acceptance/detector/KEY.md | 3 --- acceptance/runs/README.md | 2 +- acceptance/toolchain/KEY.md | 3 --- 5 files changed, 12 insertions(+), 21 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 1de652c..43f23e1 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -100,7 +100,7 @@ separate marker that has seen neither the drafting nor the review. Whoever marks in this file. A delegated marker receives only these: the document, its question set, the fixture the questions name and its key, each at the commit the readers read; this file and `acceptance/runs/README.md`, at the named commit where the run is made under a process change -before that change merges; the earlier run records of the same pull request, and those of +before that change merges, and otherwise as they stand on `next` when the readers start; the earlier run records of the same pull request, and those of other pull requests as examples of style only; the document at the commit each earlier run of the pull request read; the answer sheets; and, from the editor, the facts the record states that the sheets do not, which are the pull request and the commit, the cohort size and model, how the @@ -126,9 +126,9 @@ which rule and why. Each sheet is marked question by question, as follows. correct when, so read, it states what the key's answer states, in the document's words or the reader's own (a paraphrase, a synonym, or a defined term used in its defined sense states what it means), asserts nothing the document does not state or the key says makes an answer wrong, - omits nothing the key says an answer is wrong to omit, and quotes the document in support. The - quotation need not be the one the key gives. The key names the elements an answer is wrong to - omit; for the rest, the marker reads for the gist of the key's answer, not a checklist. + omits no element of the key's answer that answers a part of the question, nor anything the key + says an answer is wrong to omit, and quotes the document in support. Each element is read for + its gist, not its exact words. The quotation need not be the one the key gives. - **A miss traceable to the text**, under questions 2 and 3, is a miss whose reasoning quotes the document or states, in the reader's own words, what the document says or requires. Such a miss is traceable even where it also misreads the fixture, and whatever the key says of it. Only a @@ -144,8 +144,7 @@ which rule and why. Each sheet is marked question by question, as follows. unanswered counts as no passage quoted. A sheet with any answer text is marked as it stands; one whose answer text is empty, or whose process reports an error, is set aside, as [The cohort](#the-cohort) says, and is still marked. -- **The same passage.** A passage under question 4 is one the reader says it found confusing, not - one it names only to explain another confusion. A quotation of one, of any length, covers every sentence that +- **The same passage.** A quotation under question 4, of any length, covers every sentence that an occurrence of the quoted words, taken as one run, overlaps, at every place that run occurs, and a passage named without a quotation, by clause number or description, covers every sentence of what it names. A heading, list item or table row counts as one sentence. Two passages are the same when they cover a @@ -221,7 +220,7 @@ managed (policy) settings to the readers, that the editor attests the rest of th readers set aside and the cause, the run number in that pull request, counting void runs, who marked and, for a delegated marker, how it was started, the marks per reader per question, any judgement made in marking, on a re-mark which rule and why, the passages quoted as confusing, -each with the words quoted, verbatim, and its clause, and by how many readers, the findings +each with the words quoted, verbatim, or the description used where none were quoted, and its clause, and by how many readers, the findings applied, whether it was made under a process change before that change merged and, if so, at which commit, whether it passed, and a link to the answer sheets. The answer sheets are not kept in this repository: every reader's sheet, which is its whole output as its process returned it (the JSON, where the command asks for JSON), with a set-aside reader's marked as set diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 15ea7a2..56e6350 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -20,10 +20,9 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). receives and that its verdict is final unless a rule was misapplied, that an open judgement is made as a careful reader of the document would, and defines a keyed item, quoting (consecutive words of any length, verbatim apart from case, quotation-mark style and markup), a correct - load-bearing answer, read with its quotation for the gist of the key's answer, asserting - nothing the document does not state and omitting nothing the key says it is wrong to omit, a - passage found confusing, which excludes one named only to explain another, a miss traceable - to the text, + load-bearing answer, read with its quotation, asserting nothing the document does not state + and omitting no element of the key's answer that answers the question, each read for its + gist, a miss traceable to the text, including a question 1 bullet that quotes the document, which a key can add to but never remove, an unanswered item and the same passage. The thresholds apply to the readers whose answers count and block on a single reader for the first and third rows and on two readers for the fourth, and the fourth row says what a fresh cohort no @@ -36,9 +35,8 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). output and the marker's instruction, posted on the pull request and never edited, hidden or deleted, or a run not already failed is void. The keys are marked at the commit the readers read and no longer limit which misses count against the document, - removing such a limit is named as raising the bar; the method key's "half right" is a miss, and - each key names what a question 2 answer is wrong to omit. No marking rule was on `next` - before, so none relaxes it. + removing such a limit is named as raising the bar; the method key's "half right" is a miss. + Each marking rule writes down how runs 16 to 18 marked or is stricter, so none relaxes `next`. The process files include the checks that enforce them, and a process change goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, applies to runs made after it merges, or earlier ones made under it at a named commit, and is recorded here, not in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the change, points to diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index c0749a8..8608f7d 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -25,9 +25,6 @@ agent summary (those belong to the toolchain specification). ## 2. Load-bearing statements -No entry names an element an answer is wrong to omit, so each is marked for the gist of its -answer, as `ACCEPTANCE.md` says. - - **2a.** Conformance is satisfying every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression), judged on evidence; a detector that predates the document can conform, and the declaration is optional. Key quotation, section 8: "A Detector Conforms if it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression)." Supporting, [8.1](../../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps): "A diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 2d0b96a..633d90d 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -12,7 +12,7 @@ machine applies managed (policy) settings to the readers; that the editor attest isolation; the readers set aside and the cause; the run number within that pull request, counting void runs; who marked and, for a delegated marker, how it was started; the marks per reader per question against the key, with any judgement made in marking and, on a re-mark, which rule and -why; the passages quoted as confusing, each with the words quoted, verbatim, and its clause, and +why; the passages quoted as confusing, each with the words quoted, verbatim, or the description used where none were quoted, and its clause, and by how many readers; the findings applied to the text as a result; whether the run was made under a process change before that change merged and, if so, at which commit; and the verdict against the criteria in `ACCEPTANCE.md`, with a link to the answer sheets. Earlier diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 32b312c..cae0efc 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -30,9 +30,6 @@ conform; the agent summary (section [7](../../TOOLING-SPEC.md#7-agent-context)) ## 2. Load-bearing statements -No entry names an element an answer is wrong to omit, so each is marked for the gist of its -answer, as `ACCEPTANCE.md` says. - - **2a.** At the project level; the toolchain is whatever the project has assembled from third-party, first-party and project-level parts. Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "A project's Toolchain Conforms if every MUST in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record) holds across the assembled parts, wherever From 7a4bb2dd82c74d5ac206db5a3afa355413106e94 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:25:45 +0100 Subject: [PATCH 13/19] Acceptance process: review 8, a set-aside sheet's answers still count towards a failure A run fails if any criterion fails with a set-aside reader's answered items counted alongside the others, so setting a reader aside can never turn a failing run into a void one. The fourth row's clearing test covers every paragraph the earlier passage lay in. acceptance/CHANGES.md names runs 19 and 20 as made under this change. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 6 +++--- acceptance/CHANGES.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 43f23e1..cb0f011 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -33,8 +33,8 @@ test is run with a low-strength model so that clarity is measured at the floor, cohort's model. No reader is set aside for any other cause, and the record shows the cause. If any reader is set aside, the run is void and a fresh cohort runs, unless the readers whose answers count already fail a criterion, when the run fails; a run never passes with a reader set - aside. A set-aside reader's sheet is still marked, and a miss traceable to the text in it fails - the run. A void run takes a run number, has a record and posted sheets like any other, and + aside. A set-aside reader's sheet is still marked, and the run fails if any criterion fails when its + answered items are counted with the other readers', or if it has a miss traceable to the text. A void run takes a run number, has a record and posted sheets like any other, and counts towards the number of runs. - Each reader receives **only its packet**: the document's question set, the changed document @@ -165,7 +165,7 @@ one" is four of five. | Load-bearing statements (question 2) | **Every** reader correct on every one | | Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | | A miss traceable to the text | **None.** Any such miss, as [Marking](#marking) defines it, is a finding, whatever the count | -| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage that covers a sentence of the paragraph the earlier passage lay in, as that paragraph reads in the changed document, or a sentence the change kept from it | +| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage that covers a sentence of each paragraph the earlier passage lay in, as that paragraph reads in the changed document, or a sentence the change kept from it | | Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | Every row must hold, and the last covers only what none of the others catches. The first and diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 56e6350..eb74588 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -6,7 +6,7 @@ not a specification and is not versioned, and a change to it is not tested by a to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). - [Pull request #13](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/13): the cohort can be run and marked without judgement calls that the rules - leave open; this applies to runs made after it merges, and to run 19 if made under it, and + leave open; this applies to runs made after it merges, and to runs 19 and 20, which were made under it at a695010, and runs 16 to 18 stand as recorded. A cohort is an odd number of readers started, at least five, larger at the editor's choice the more obligations the change touches, each a separate process with no tools and no project context, started from @@ -14,8 +14,8 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). `CLAUDE.md`, skills, plugins, hooks, MCP servers or saved session. A reader is set aside only if its answer text is empty, its process reports an error, it shared the editor's context, it was not started as required or it reported another model, and a run with a reader set - aside is void, or fails if the readers who count already fail it or its own sheet has a miss - traceable to the text; a void run has a number, a record and posted sheets. The packet is the question set, the document and the fixture, each + aside is void, or fails if the readers who count already fail it, if it fails with the set-aside + sheet's answered items counted too, or if that sheet has a miss traceable to the text; a void run has a number, a record and posted sheets. The packet is the question set, the document and the fixture, each after one `===== FILE: =====` line. A new marking section says what a delegated marker receives and that its verdict is final unless a rule was misapplied, that an open judgement is made as a careful reader of the document would, and defines a keyed item, quoting (consecutive From 204f95b0957f20fb869e20d42fb5257b7c59b97e Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:30:15 +0100 Subject: [PATCH 14/19] Acceptance process: review 9, the fourth row's clearing test covers any earlier passage A fresh reader naming a sentence of any paragraph an earlier confusing passage lay in keeps the finding open, as the keys say. "Status paragraph" is used as in CONTRIBUTING.md step 5, and the method key's 2a lists every element an answer is wrong to omit. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 4 ++-- acceptance/method/KEY.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index cb0f011..8c396dd 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -165,7 +165,7 @@ one" is four of five. | Load-bearing statements (question 2) | **Every** reader correct on every one | | Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | | A miss traceable to the text | **None.** Any such miss, as [Marking](#marking) defines it, is a finding, whatever the count | -| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage that covers a sentence of each paragraph the earlier passage lay in, as that paragraph reads in the changed document, or a sentence the change kept from it | +| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage that covers a sentence of any paragraph an earlier passage lay in, as that paragraph reads in the changed document, or a sentence the change kept from one | | Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | Every row must hold, and the last covers only what none of the others catches. The first and @@ -207,7 +207,7 @@ the release branch needs no cohort of its own unless it changes specification te change is any change to the document's text after the readers read it, from any source (a review finding, the editor, or `next` merged into the branch), except a version number or date that `PUBLISHING.md` or step 5 of `CONTRIBUTING.md` has the editor set, in the header, the status -section, the citation or the closing table. Any other change to those paragraphs is a wording +paragraph, the citation or the closing table. Any other change to those paragraphs is a wording change. Readers are never reused. ## The record diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 8839bc7..de03f0d 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -29,7 +29,7 @@ a test may serve as the detector; an agent may baseline the remaining instances. then sweep and fix every instance, in that order. Key quotations, section [3](../../SPEC.md#3-the-method): "Six clauses, in order."; [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance): "The Practitioner MUST run the Rule across the entire codebase and record the total Instance count before fixing anything, and MUST then fix every Instance found rather - than only the one". An answer that omits the rule, the proof or the sweep, or has the fix + than only the one". An answer that omits the class, the rule, the proof, the sweep or the fix of every instance, or has the fix before the rule or the sweep before the proof, is wrong. - **2b.** The owner of the codebase, always a human; an agent may not. Key quotation, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): "An Agent MUST NOT Baseline, suppress or knowingly leave an Instance unfixed on its own From 598908ba9e02b83683c622baa1d9a84e9107f027 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:57:42 +0100 Subject: [PATCH 15/19] Acceptance process: a short quotation under question 4 covers the place the reader names Run 24 showed one-word quotations ("MAY", "configuration") covering every paragraph where the word occurs. A quotation now covers the sentences it overlaps at the clause or place the reader names it by, and every occurrence only where the reader names no place. On next the same passage was undefined and run 16 read it narrowly, so this relaxes nothing there. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 8c396dd..43adb33 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -145,8 +145,9 @@ which rule and why. Each sheet is marked question by question, as follows. one whose answer text is empty, or whose process reports an error, is set aside, as [The cohort](#the-cohort) says, and is still marked. - **The same passage.** A quotation under question 4, of any length, covers every sentence that - an occurrence of the quoted words, taken as one run, overlaps, at every place that run occurs, - and a passage named without a quotation, by clause number or description, covers every + an occurrence of the quoted words, taken as one run, overlaps: at the place the reader names + it, by clause number or description, and where the reader names no place, at every place that + run occurs; a passage named without a quotation, by clause number or description, covers every sentence of what it names. A heading, list item or table row counts as one sentence. Two passages are the same when they cover a sentence in common or lie in the same paragraph, and a list item or table row is its own paragraph. From d88ab30271a9bee188302d19a691780717926e57 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 00:47:46 +0100 Subject: [PATCH 16/19] Acceptance process: name runs 19 to 30 as made under this change Co-Authored-By: Claude Opus 5.5 --- acceptance/CHANGES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index eb74588..18d028b 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -6,7 +6,7 @@ not a specification and is not versioned, and a change to it is not tested by a to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). - [Pull request #13](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/13): the cohort can be run and marked without judgement calls that the rules - leave open; this applies to runs made after it merges, and to runs 19 and 20, which were made under it at a695010, and + leave open; this applies to runs made after it merges, and to runs 19 to 30, which were made under it at the commits their records name, and runs 16 to 18 stand as recorded. A cohort is an odd number of readers started, at least five, larger at the editor's choice the more obligations the change touches, each a separate process with no tools and no project context, started from From 814567b9735140e02434372843d3eb2f063d0385 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 10:51:03 +0100 Subject: [PATCH 17/19] Acceptance process: an earlier run's judgement does not bind where its rule has changed Run 22 made a marking judgement by the stricter reading, a tie-break bdfd055 had already replaced with a careful reader's, and runs 24 to 30 followed it as precedent. The marking section now says a judgement an earlier run recorded does not bind a later run where a rule it rested on has since changed: the marker makes it afresh and the record names the rule. This corrects a misapplied rule and relaxes none. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 4 +++- acceptance/CHANGES.md | 7 +++++-- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 43adb33..eacf840 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -153,7 +153,9 @@ which rule and why. Each sheet is marked question by question, as follows. paragraph. Where a mark needs a judgement these rules do not settle, the marker makes it as a careful reader -of the document would, and the record states the judgement and why. +of the document would, and the record states the judgement and why. A judgement an earlier run +recorded does not bind a later run where a rule it rested on has since changed: the marker makes +it afresh under the rules as they now stand, and the record names the rule that changed. ## Pass criteria diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 18d028b..8735c63 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -18,7 +18,8 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). sheet's answered items counted too, or if that sheet has a miss traceable to the text; a void run has a number, a record and posted sheets. The packet is the question set, the document and the fixture, each after one `===== FILE: =====` line. A new marking section says what a delegated marker receives and that its verdict is final unless a rule was misapplied, that an open judgement is - made as a careful reader of the document would, and defines a keyed item, quoting (consecutive + made as a careful reader of the document would, that an earlier run's judgement does not bind + a later run where a rule it rested on has since changed, and defines a keyed item, quoting (consecutive words of any length, verbatim apart from case, quotation-mark style and markup), a correct load-bearing answer, read with its quotation, asserting nothing the document does not state and omitting no element of the key's answer that answers the question, each read for its @@ -45,6 +46,8 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). named no fixture in what a reader receives whilst `acceptance/README.md` did, the thresholds were written as five of five and four of five for a cohort of at least five, two pass-criteria rows both covered a single reader's traceable miss, and nothing settled an unanswered question - or "the same passage". [Run + or "the same passage". The rule on an earlier run's judgement was prompted by run 22, which + made a judgement by the stricter reading, a tie-break this change had already replaced with a + careful reader's, and which later runs of pull request 14 then followed as precedent. [Run 16](runs/016-deferral-clause-run-1.md), [run 17](runs/017-deferral-clause-run-2.md) and [run 18](runs/018-deferral-clause-run-3.md) ruled on these case by case. From 5cec5573f06a2f82e2890104deb75b18e8144d9c Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 11:17:55 +0100 Subject: [PATCH 18/19] Acceptance process: an earlier run's judgement binds no later run Review 1 of this change, finding 1: the sentence freed a later run only where a rule the judgement rested on had since changed. Run 22's judgement applied a tie-break bdfd055 had replaced before run 22 was made, so no rule changed after it and the sentence missed its own case; it also implied that a judgement binds where its rule has not changed, which no process file said. A judgement an earlier run recorded now binds no later run, one that applied a rule not in force when it was made is never followed, and a record says where its marker departs from an earlier judgement and why. acceptance/CHANGES.md follows. No criterion changes. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 5 +++-- acceptance/CHANGES.md | 8 +++++--- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index eacf840..42e28f3 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -154,8 +154,9 @@ which rule and why. Each sheet is marked question by question, as follows. Where a mark needs a judgement these rules do not settle, the marker makes it as a careful reader of the document would, and the record states the judgement and why. A judgement an earlier run -recorded does not bind a later run where a rule it rested on has since changed: the marker makes -it afresh under the rules as they now stand, and the record names the rule that changed. +recorded binds no later run, and one that applied a rule not in force when it was made is never +followed: the marker makes each judgement afresh under this file as it applies to the run being +marked, and where it departs from an earlier run's judgement, the record says so and why. ## Pass criteria diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 8735c63..433677b 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -18,8 +18,9 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). sheet's answered items counted too, or if that sheet has a miss traceable to the text; a void run has a number, a record and posted sheets. The packet is the question set, the document and the fixture, each after one `===== FILE: =====` line. A new marking section says what a delegated marker receives and that its verdict is final unless a rule was misapplied, that an open judgement is - made as a careful reader of the document would, that an earlier run's judgement does not bind - a later run where a rule it rested on has since changed, and defines a keyed item, quoting (consecutive + made as a careful reader of the document would, that an earlier run's judgement binds no + later run, one that applied a rule not in force when it was made is never followed, and a + record says where its marker departs from an earlier judgement and why, and defines a keyed item, quoting (consecutive words of any length, verbatim apart from case, quotation-mark style and markup), a correct load-bearing answer, read with its quotation, asserting nothing the document does not state and omitting no element of the key's answer that answers the question, each read for its @@ -48,6 +49,7 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). rows both covered a single reader's traceable miss, and nothing settled an unanswered question or "the same passage". The rule on an earlier run's judgement was prompted by run 22, which made a judgement by the stricter reading, a tie-break this change had already replaced with a - careful reader's, and which later runs of pull request 14 then followed as precedent. [Run + careful reader's before run 22 was made, so that the rule it applied was not in force, and which + later runs of pull request 14 then followed as precedent. [Run 16](runs/016-deferral-clause-run-1.md), [run 17](runs/017-deferral-clause-run-2.md) and [run 18](runs/018-deferral-clause-run-3.md) ruled on these case by case. From 5f01f4cbdc9dfa437b0b705f95a578fcb9cd3aad Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 14:49:06 +0100 Subject: [PATCH 19/19] Acceptance process: a departure is recorded against the same pull request's runs Review 2, finding N4. The clause that a judgement applying a rule not in force is "never followed" is dropped: no earlier judgement binds, and the words could be read as forbidding a mark a careful reader reaches afresh. The duty to record a departure now names the earlier runs of the same pull request, since other pull requests' runs reach a marker as examples of style only. Run 22's case, within one pull request, is still covered. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 6 +++--- acceptance/CHANGES.md | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 42e28f3..107a490 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -154,9 +154,9 @@ which rule and why. Each sheet is marked question by question, as follows. Where a mark needs a judgement these rules do not settle, the marker makes it as a careful reader of the document would, and the record states the judgement and why. A judgement an earlier run -recorded binds no later run, and one that applied a rule not in force when it was made is never -followed: the marker makes each judgement afresh under this file as it applies to the run being -marked, and where it departs from an earlier run's judgement, the record says so and why. +recorded binds no later run: the marker makes each judgement afresh under this file as it +applies to the run being marked, and where it departs from a judgement that an earlier run of the +same pull request recorded, the record says so and why. ## Pass criteria diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 433677b..5fe9e20 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -19,8 +19,8 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). after one `===== FILE: =====` line. A new marking section says what a delegated marker receives and that its verdict is final unless a rule was misapplied, that an open judgement is made as a careful reader of the document would, that an earlier run's judgement binds no - later run, one that applied a rule not in force when it was made is never followed, and a - record says where its marker departs from an earlier judgement and why, and defines a keyed item, quoting (consecutive + later run, and that a record says where its marker departs from a judgement an earlier run of + the same pull request recorded, and why, and defines a keyed item, quoting (consecutive words of any length, verbatim apart from case, quotation-mark style and markup), a correct load-bearing answer, read with its quotation, asserting nothing the document does not state and omitting no element of the key's answer that answers the question, each read for its