diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 31c8df0..107a490 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -8,11 +8,43 @@ test is run with a low-strength model so that clarity is measured at the floor, ## The cohort -- **At least five independent readers**, each a fresh instance of a low-strength model (the - smallest current model in its family, for example Haiku), with no shared context and no - access to the change history, the critique findings or each other. -- Each reader receives **only the changed document or documents** and the fixed question set - below. Nothing else is read first. +- **An odd number of independent readers started, at least five**, each a fresh instance of a + low-strength model (the smallest current model in its family, for example Haiku), with no + shared context and no access to the change history, the critique findings or each other. The + editor chooses the size, and may choose a larger cohort the more obligations the change + touches. + +- **Each reader is a separate process with no tools and no project context, only its packet**, + started from an empty directory outside any repository, with no user, project or local settings + files, no `CLAUDE.md`, skills, plugins, hooks or MCP servers, and no saved session, so that no + project instruction reaches it. The model tool's own fixed system prompt is not project + context. For example, with the packet on standard input: + + ``` + claude -p --safe-mode --model haiku --tools "" --setting-sources "" --strict-mcp-config --no-session-persistence --output-format json < packet.md + ``` + + The JSON output gives the model version each reader reported. A reader isolated in this way is + a cold reader. A reader is set aside, and its answers do not count, when its answer text is + empty; when its process reports an error, for example `is_error` in its JSON output or an API + error message in place of answers; when it shared the editor's context, such as an agent + dispatched from the editor's own session; when the command or working directory recorded for + it is not one this section requires; or when the model version it reported is not the + cohort's model. No reader is set aside for any other cause, and the record shows the cause. If any + reader is set aside, the run is void and a fresh cohort runs, unless the readers whose answers + count already fail a criterion, when the run fails; a run never passes with a reader set + aside. A set-aside reader's sheet is still marked, and the run fails if any criterion fails when its + answered items are counted with the other readers', or if it has a miss traceable to the text. A void run takes a run number, has a record and posted sheets like any other, and + counts towards the number of runs. + +- Each reader receives **only its packet**: the document's question set, the changed document + and the fixture the questions name, each in full in the reader's prompt, the question set + first. Each file is preceded by exactly one line, `===== FILE: =====`, where `` is + the name the question set gives the file (so `fixtures/remediation-ledger.md`, and + `QUESTIONS.md` for the question set itself). The question set carries the instruction to the + reader, and nothing else is added. Every reader in a run receives the same packet. Nothing else + is read first. + - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. @@ -37,9 +69,9 @@ document, so that marking is mechanical. Each reader answers all of them. from different sources; how many grades a project that ships tooling has. 3. **Application.** A fixture case supplied with the questions, graded clause by clause with - Yes, Partial or No and one line of reasoning. Keyed against the intended verdicts; a divergence - counts against the document only when the reader's reasoning shows the clause was read - differently, not when the fixture was misread. + Yes, Partial or No and one line of reasoning. Keyed against the intended verdicts; whether a + divergence counts against the document is settled by the rule for a miss traceable to the + text, under [Marking](#marking). 4. **Confusion.** Any passage the reader found confusing on its one reading, quoted with its clause number, and nothing else: not what would make it clearer, not a proposed change. @@ -61,30 +93,103 @@ about any of them is the editor's decision. An earlier version of this test aske what would make a clause clearer and for a clarity rating; both were removed on the author's instruction for that reason, and neither is reinstated. +## Marking + +The editor marks each answer sheet against the document's key, or delegates the marking to a +separate marker that has seen neither the drafting nor the review. Whoever marks is the marker +in this file. A delegated marker receives only these: the document, its question set, the +fixture the questions name and its key, each at the commit the readers read; this file and +`acceptance/runs/README.md`, at the named commit where the run is made under a process change +before that change merges, and otherwise as they stand on `next` when the readers start; the earlier run records of the same pull request, and those of +other pull requests as examples of style only; the document at the commit each earlier run of +the pull request read; the answer sheets; and, from the editor, the facts the record states that +the sheets do not, which are the pull request and the commit, the cohort size and model, how the +readers were isolated and the command they were started with, whether the machine applies +managed (policy) settings to the readers, that the editor attests the rest of the isolation, the +readers set aside for a cause the sheets cannot show and why, how the marker was started, the run number, and whether the run was +made under a process change before that change merged and, if so, at which commit. The marker +writes the record, stating those facts as the editor supplied them and setting aside, with the +cause, any reader whose sheet shows a cause, and the editor adds the +findings applied and the link to the posted sheets after marking. A delegated marker's verdict, +its marks and its PASS or FAIL, is final unless the editor shows that a rule in this file was +misapplied; the editor then re-marks what the misapplied rule affected, and the record says +which rule and why. Each sheet is marked question by question, as follows. + +- **Keyed items.** Question 1 as a whole is one keyed item, and so is each item of question 2 + and each answer of question 3 for which the key gives an intended answer. +- **Quoting.** A reader quotes the document when it gives consecutive words of the document, in + the document's order and of any length, as a quotation, in quotation marks or a block quote. + The words are verbatim apart from letter case, quotation-mark or apostrophe style, and markdown + link or emphasis markup. Naming a clause number, or paraphrasing, is not quoting. +- **Questions 1 and 3** are marked as the key states for each item. +- **Question 2.** An answer is the reader's sentence and its quotation read together. It is + correct when, so read, it states what the key's answer states, in the document's words or the + reader's own (a paraphrase, a synonym, or a defined term used in its defined sense states what + it means), asserts nothing the document does not state or the key says makes an answer wrong, + omits no element of the key's answer that answers a part of the question, nor anything the key + says an answer is wrong to omit, and quotes the document in support. Each element is read for + its gist, not its exact words. The quotation need not be the one the key gives. +- **A miss traceable to the text**, under questions 2 and 3, is a miss whose reasoning quotes the + document or states, in the reader's own words, what the document says or requires. Such a miss + is traceable even where it also misreads the fixture, and whatever the key says of it. Only a + miss whose reasoning states nothing about the document, such as a misread fixture alone, is the + reader's. A reasoning that only reports facts of the fixture, attributing no requirement, + condition or meaning to the document or a clause of it, states nothing about the document; + naming the clause a case fails or meets attributes a requirement to it. A miss under any + question is also traceable where the key says it counts against the document: the key can add + to these misses, never take one away. A miss on question 1 is also traceable where the bullet + it turns on quotes the document; any other miss on question 1 is the reader's. +- **An unanswered question.** Each keyed item left unanswered is one miss, not traceable to the + text, so a sheet that leaves three items of question 3 blank has three misses. Question 4 left + unanswered counts as no passage quoted. A sheet with any answer text is marked as it stands; + one whose answer text is empty, or whose process reports an error, is set aside, as + [The cohort](#the-cohort) says, and is still marked. +- **The same passage.** A quotation under question 4, of any length, covers every sentence that + an occurrence of the quoted words, taken as one run, overlaps: at the place the reader names + it, by clause number or description, and where the reader names no place, at every place that + run occurs; a passage named without a quotation, by clause number or description, covers every + sentence of what it names. A heading, list item or table row counts as one sentence. Two passages are the same when they cover a + sentence in common or lie in the same paragraph, and a list item or table row is its own + paragraph. + +Where a mark needs a judgement these rules do not settle, the marker makes it as a careful reader +of the document would, and the record states the judgement and why. A judgement an earlier run +recorded binds no later run: the marker makes each judgement afresh under this file as it +applies to the run being marked, and where it departs from a judgement that an earlier run of the +same pull request recorded, the record says so and why. + ## Pass criteria -The change passes when all of the following hold on a single run: +The change passes when all of the following hold on a single run. The thresholds apply to the +readers whose answers count; with five such readers, "every reader" is five of five and "all but +one" is four of five. -| Criterion | Threshold | -| ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | -| Load-bearing statements (question 2) | **5 of 5** readers correct on every one | -| Restatement and application (questions 1 and 3) | **At least 4 of 5** readers correct on each keyed item | -| A wrong answer traceable to the text | **None.** A miss whose reasoning quotes the document is a finding, whatever the count | -| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it | -| Anything from one reader only | Recorded; applied at the editor's discretion; does not block | +| Criterion | Threshold | +| ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Load-bearing statements (question 2) | **Every** reader correct on every one | +| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | +| A miss traceable to the text | **None.** Any such miss, as [Marking](#marking) defines it, is a finding, whatever the count | +| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage that covers a sentence of any paragraph an earlier passage lay in, as that paragraph reads in the changed document, or a sentence the change kept from one | +| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | + +Every row must hold, and the last covers only what none of the others catches. The first and +third rows block on what a single reader did, and the fourth on what two readers did and then +on what any one reader of a fresh cohort names, and the second +row's tolerance of one reader never excuses a miss under the third. The asymmetry is deliberate. A load-bearing statement is the sentence the whole document exists to make, and one reader in five getting it wrong means one agent in five will act on the wrong reading; unanimity is the only acceptable result there. Everywhere else a single divergent reader -is more likely to be the reader than the text, so four of five is tolerated, but a miss that the -reader can justify from the document is always the document's fault. +is more likely to be the reader than the text, so one such reader is tolerated, but a miss that +the reader can justify from the document is always the document's fault. ## The bar only moves up No criterion in this document is ever relaxed to let a change through. If the cohort is confused, the test is working and the text is wrong; the fix is to the text, never to the threshold, the -question, the fixture or the key. A threshold may be raised, a keyed question may be added, and a -key may be corrected where it misquoted the document; none of those makes a failing run pass. The +question, the fixture or the key. A threshold may be raised, a keyed question may be added, a key +may be corrected where it misquoted the document, and a key's limit on which misses count against +the document may be removed, which raises the bar; none of those makes a failing run pass. The one time a threshold was relaxed, after three runs, the relaxation was reversed on the author's instruction and is recorded here so that it is not tried again. Removing the clarity rating and the request for suggestions was not a relaxation: it changed what the cohort is asked, not how @@ -98,21 +203,61 @@ something. ## When it runs -Once per pull request, on the final text, after the editor's rewrite and any key update. It is -the last gate before the merge to `next`, so that every change on `next` is one the cohort has -accepted, and the release branch needs no cohort of its own unless it changes specification text. -A wording change after the run means a fresh run; readers are never reused. +On the final text of each pull request, after the editor's rewrite and any key update. A fresh +cohort runs after every change to the text that follows a failure, after every void run, and +after every wording change; a text that failed is never run again unchanged. It is the last gate +before the merge to `next`, so that every change on `next` is one the cohort has accepted, and +the release branch needs no cohort of its own unless it changes specification text. A wording +change is any change to the document's text after the readers read it, from any source (a +review finding, the editor, or `next` merged into the branch), except a version number or date +that `PUBLISHING.md` or step 5 of `CONTRIBUTING.md` has the editor set, in the header, the status +paragraph, the citation or the closing table. Any other change to those paragraphs is a wording +change. Readers are never reused. ## The record Each run has a file under `acceptance/runs/`, named `NNN-.md` with `NNN` the next number in the one sequence shared by the three documents, stating: the document, the pull request and the -commit read, the cohort size and model, the run number in that pull request, the marks per -reader per question, the findings applied, and whether it passed. The changelog line for the -change names the file. CI refuses a specification change whose pull request adds no run file; -it checks for at least one, and a pull request that changes two documents adds a run for each. +commit read, the cohort size and model, with the model version each reader reported, how the +readers were isolated and the command they were started with, whether the machine applies +managed (policy) settings to the readers, that the editor attests the rest of the isolation, the +readers set aside and the cause, the run number in that pull request, counting void runs, who +marked and, for a delegated marker, how it was started, the marks per reader per question, any +judgement made in marking, on a re-mark which rule and why, the passages quoted as confusing, +each with the words quoted, verbatim, or the description used where none were quoted, and its clause, and by how many readers, the findings +applied, whether it was made under a process change before that change merged and, if so, at +which commit, whether it passed, and a link to the answer sheets. The answer sheets are not kept in this repository: every reader's sheet, which is its whole output as its process +returned it (the JSON, where the command asks for JSON), with a set-aside reader's marked as set +aside, and the instruction a delegated marker was given, which is its task prompt and the agent +definition it ran under, are posted in one or more comments on the pull request, each linked +from the record, so that an owner can check the marks. The comments also hold the exact prompt +every reader received and the command and the form of the working directory the readers were +started in; a sheet cannot show how its process was started, which is why the editor attests the +rest of the isolation. The comments are never edited, hidden or deleted after posting; if one +is before the change merges, the run is void and a fresh cohort runs, unless the run had already +failed, when it stays failed. The changelog line for the change names the file. CI refuses a +specification change whose pull request adds no run file; it checks for at least one, and a pull request that changes two +documents adds a run for each. Each published version's changelog entry carries the lines of the changes it contains, so a version's cohort record is the set of runs those lines name. A version with no such line has not been accepted and MUST NOT be described as published. The question sets and answer keys live under `acceptance/` in this repository so that a run can be repeated and compared. + +## Changes to this process + +The process files are this file, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md` and the checks that enforce them: `tools/changes.py`, the key-quotation checks +in `spec-qa.py`, the CI workflow and the pull request template. A change to any part of them is +a change to this process. They are not specifications, and a change to them is not tested by a +cohort: the cohort measures whether a low-strength reader can follow a specification, and the +process is followed by the editor. Such a change names the runs or the case that showed the need +for it, and goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, with no scope, key or cohort +step; its review reads every process file it changes. It applies to every run made after it +merges; a run made before then may be made under it, follows all of it at a commit named before +the readers start, never part, and its record says so and names that commit. It is recorded by a line added at the top +of [acceptance/CHANGES.md](acceptance/CHANGES.md), naming its pull request, not in the +changelog, which is for the three specifications. It is bound by +[the section on the bar](#the-bar-only-moves-up): it may raise a criterion, never relax one. A +change to a question set, key or fixture is not a change to this process; `acceptance/README.md` +governs those. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fa1743e..29a3772 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,7 +29,9 @@ review. It is a patch version when released. SHOULD or MAY in a numbered section or appendix. It needs everything a clarity edit needs, plus the five tests in SCOPE.md answered in the pull request, and it is at least a minor version when released. `tools/changes.py` tells the two apart mechanically and CI runs it on every pull -request. +request. A change to the process files ACCEPTANCE.md names, this file among them, is neither: +ACCEPTANCE.md says how it is made, and it is recorded in `acceptance/CHANGES.md`, not in +CHANGELOG.md. ## What a pull request contains @@ -39,6 +41,9 @@ request. request template lays them out. - The observed case that prompted it, for either kind. A clarity edit still has a reason: a reader who took the passage the wrong way. +- Where a specification's Appendix A changed, the files `python3 build-agent-surfaces.py` + writes, rebuilt with it and committed with the change; CI runs it with `--check` and fails + when they are stale. - `python3 spec-qa.py` clean and the unit tests passing, which CI checks for you. A pull request is a suggestion, not a demand. The documents have one editor, who keeps ownership @@ -55,20 +60,26 @@ survives, the change is yours. the editor sets the pre-release header. 2. **Scope.** For an obligation, the editor confirms each SCOPE.md test. A failed test closes the pull request with a row in `DECLINED.md`. -3. **Review and rewrite.** An independent review of the diff against the document, then the - editor's rewrite, pushed to the pull request branch where the contributor allows it or to an - editor's branch that supersedes it. +3. **Review and rewrite.** An independent review of the diff against the whole document, by a + reviewer who did not draft the change, then the editor's rewrite, pushed to the pull request + branch where the contributor allows it or to an editor's branch that supersedes it. The pull + request names the reviewer and links or contains the review. A wording change, as + ACCEPTANCE.md defines it under "When it runs", means a fresh run. 4. **Keys.** Where the change touches a sentence an acceptance key quotes, or adds a load-bearing obligation, the editor updates the key before the cohort runs, never after. 5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text. Its record is committed under `acceptance/runs/`, the `Unreleased` changelog line names it, and CI refuses a specification change that lacks one. For an obligation, the editor sets the document's - header to `-dev, unpublished` in the same commit if it still names a published + header to `-dev, unpublished`, and the status paragraph and citation that state + its version to `-dev`, in the same commit if the header still names a published version. A wording change after the run means a fresh run. 6. **Merge** into `next` by merge commit, and the branch is deleted. -Steps 2 to 5 are done by a person with a Claude Code session, not by CI, and they happen in -batches. A quiet pull request is not a declined one; a declined one says so. +Steps 2 to 5 are the editor's, not CI's, and they happen in batches. The editor may have an +agent draft or review, and may delegate marking as ACCEPTANCE.md allows, in which case the +marker's verdict is final unless the editor shows a rule was misapplied, as ACCEPTANCE.md says; +every other decision in these steps stays the editor's. A quiet pull request is not a declined +one; a declined one says so. ## Versions and releases diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md new file mode 100644 index 0000000..5fe9e20 --- /dev/null +++ b/acceptance/CHANGES.md @@ -0,0 +1,55 @@ +# Changes to the acceptance process + +Changes to the process files that [ACCEPTANCE.md](../ACCEPTANCE.md) names, recorded here as it +describes: a line is added at the top for each change, naming its pull request. The process is +not a specification and is not versioned, and a change to it is not tested by a cohort. Changes +to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). + +- [Pull request #13](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/13): the cohort can be run and marked without judgement calls that the rules + leave open; this applies to runs made after it merges, and to runs 19 to 30, which were made under it at the commits their records name, and + runs 16 to 18 stand as recorded. A cohort is an odd + number of readers started, at least five, larger at the editor's choice the more obligations + the change touches, each a separate process with no tools and no project context, started from + an empty directory outside any repository with no user, project or local settings files, + `CLAUDE.md`, skills, plugins, hooks, MCP servers or saved session. A reader is set aside only + if its answer text is empty, its process reports an error, it shared the editor's context, it + was not started as required or it reported another model, and a run with a reader set + aside is void, or fails if the readers who count already fail it, if it fails with the set-aside + sheet's answered items counted too, or if that sheet has a miss traceable to the text; a void run has a number, a record and posted sheets. The packet is the question set, the document and the fixture, each + after one `===== FILE: =====` line. A new marking section says what a delegated marker + receives and that its verdict is final unless a rule was misapplied, that an open judgement is + made as a careful reader of the document would, that an earlier run's judgement binds no + later run, and that a record says where its marker departs from a judgement an earlier run of + the same pull request recorded, and why, and defines a keyed item, quoting (consecutive + words of any length, verbatim apart from case, quotation-mark style and markup), a correct + load-bearing answer, read with its quotation, asserting nothing the document does not state + and omitting no element of the key's answer that answers the question, each read for its + gist, a miss traceable to the text, + including a question 1 bullet that quotes the document, which a key can add to but never + remove, an unanswered item and the same passage. The thresholds apply to the readers whose answers count and block on a single + reader for the first and third rows and on two readers for the fourth, and the fourth row says what a fresh cohort no + longer naming a passage means. A fresh cohort runs after every change following a failure, + every void run and every wording change, which is any change to the text after the readers + read it except a version number or date the editor sets for a release, and step 5 sets the + status paragraph and citation to the `-dev` version with the header. The record adds the + model version each reader reported, the isolation, managed settings, readers set aside, void + runs, re-marks and each confusing passage's words verbatim, and links each reader's whole + output and the marker's instruction, posted on the pull request and never edited, hidden or + deleted, or a run not already failed is void. The keys are marked at + the commit the readers read and no longer limit which misses count against the document, + removing such a limit is named as raising the bar; the method key's "half right" is a miss. + Each marking rule writes down how runs 16 to 18 marked or is stricter, so none relaxes `next`. + The process files include the checks that enforce them, and a process change goes through + steps 1, 3 and 6 of `CONTRIBUTING.md`, applies to runs made after it merges, or earlier + ones made under it at a named commit, and is recorded here, not in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the change, points to + the definition of a wording change, and adds the rebuild of the files `build-agent-surfaces.py` + writes after an Appendix A change. Prompted by gaps in the text as it stood: `ACCEPTANCE.md` + named no fixture in what a reader receives whilst `acceptance/README.md` did, the thresholds + were written as five of five and four of five for a cohort of at least five, two pass-criteria + rows both covered a single reader's traceable miss, and nothing settled an unanswered question + or "the same passage". The rule on an earlier run's judgement was prompted by run 22, which + made a judgement by the stricter reading, a tie-break this change had already replaced with a + careful reader's before run 22 was made, so that the rule it applied was not in force, and which + later runs of pull request 14 then followed as precedent. [Run + 16](runs/016-deferral-clause-run-1.md), [run 17](runs/017-deferral-clause-run-2.md) and + [run 18](runs/018-deferral-clause-run-3.md) ruled on these case by case. diff --git a/acceptance/README.md b/acceptance/README.md index 4cfb6ec..0e43599 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -1,9 +1,10 @@ # Acceptance question sets The question sets and answer keys that [ACCEPTANCE.md](../ACCEPTANCE.md) requires. One directory per -specification document, each holding the fixed questions a cold reader answers and the key an -editor marks against. The fixture cases the application question uses live in `fixtures/`, and -the record of every run lives in `runs/`. +specification document, each holding the fixed questions a +[cold reader](../ACCEPTANCE.md#the-cohort) answers and the key an editor marks against. The +fixture cases the application question uses live in `fixtures/`, the record of every run lives in +`runs/`, and changes to the acceptance process itself are recorded in [CHANGES.md](CHANGES.md). | Document | Questions | Key | | ------------------ | ------------------------------------------------ | ------------------------------------ | @@ -14,13 +15,15 @@ the record of every run lives in `runs/`. ## How a run works 1. The editor identifies which documents the change touched. Only those are tested. -2. For each, five or more fresh low-strength readers each receive the document's current text, - its `QUESTIONS.md`, and the fixture the questions name. Nothing else: no key, no changelog, no - critique findings, no other reader's answers. -3. The editor marks each answer sheet against `KEY.md` and applies the pass criteria in - `ACCEPTANCE.md`. -4. The result is recorded as a file under [runs/](runs/), with cohort size, model, run number, - marks and the findings applied, and the change's `Unreleased` changelog line names that file. +2. For each, an odd number of fresh low-strength readers, at least five, each isolated as + `ACCEPTANCE.md` describes, each receive the document's current text, its `QUESTIONS.md`, and + the fixture the questions name. Nothing else: no key, no changelog, no critique findings, no + other reader's answers. +3. The editor, or a separate marker the editor delegates to, marks each answer sheet against + `KEY.md` under the marking rules in `ACCEPTANCE.md` and applies its pass criteria. +4. The result is recorded as a file under [runs/](runs/), stating what `ACCEPTANCE.md` lists, + with the answer sheets posted on the pull request and linked from it, and the change's + `Unreleased` changelog line names that file. A run that fails is followed by a change to the text, never to the questions, and a fresh cohort. @@ -32,6 +35,8 @@ matches the document is a defect in this directory, not a reason to mark a reade Questions are fixed so that runs can be compared across versions. Adding a keyed question is allowed when a new clause is load-bearing; removing or softening one to make a run pass is not. +A process change may remove a limit a key puts on which misses count against the document, as +[the section on the bar](../ACCEPTANCE.md#the-bar-only-moves-up) allows. The reader may quote a passage it found confusing; it is never asked to rate the document, say what would make a passage clearer, or propose a change: the cohort confirms comprehension and does not design the text. diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 132e6cb..8608f7d 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -1,6 +1,7 @@ # Answer key for DETECTOR-SPEC.md -Marked against the document at the version named in the changelog entry that records the run. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, update the quotation here in the same commit. @@ -60,18 +61,14 @@ agent summary (those belong to the toolchain specification). Section [8](../../DETECTOR-SPEC.md#8-conformance) verdict: **does not conform**, because [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) fails and partial conformance MUST NOT be described as conformance. A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Partial on the grounds that thirty-eight resolve -has read the clause differently and that counts against the document; a reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes -because they missed the last paragraph of the fixture has misread the fixture, not the document. +has read the clause differently and that counts against the document. Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**. It is a SHOULD, and Lintomatic's failure to check its own identifiers in CI is a missed SHOULD, not a further MUST failure; the MUST that fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). -A reader who marks [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) No because the comment takes no reason has confused [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) with [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), which is -SHOULD; that counts against the document if the reasoning quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason). - ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage quoted. Two or more readers naming the same passage, as +`ACCEPTANCE.md` defines it, is a finding that blocks until the text is changed and a fresh +cohort no longer names it. What to change is the editor's decision; the reader is not asked +and any suggestion it volunteers is discarded unread. diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 3ca1a80..de03f0d 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -1,6 +1,7 @@ # Answer key for SPEC.md -Marked against the document at the version named in the changelog entry that records the run. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, update the quotation here in the same commit. @@ -28,20 +29,20 @@ a test may serve as the detector; an agent may baseline the remaining instances. then sweep and fix every instance, in that order. Key quotations, section [3](../../SPEC.md#3-the-method): "Six clauses, in order."; [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance): "The Practitioner MUST run the Rule across the entire codebase and record the total Instance count before fixing anything, and MUST then fix every Instance found rather - than only the one". An answer that has the fix before the rule, or the sweep before the proof, - is wrong. + than only the one". An answer that omits the class, the rule, the proof, the sweep or the fix of every instance, or has the fix + before the rule or the sweep before the proof, is wrong. - **2b.** The owner of the codebase, always a human; an agent may not. Key quotation, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): "An Agent MUST NOT Baseline, suppress or knowingly leave an Instance unfixed on its own - authority, whatever the Instance count turns out to be." An answer that lets the practitioner - decide because the count was small, or because the project recorded no rule, is wrong (the - default owner paragraph in the terminology entry for Owner still requires a recorded decision). + authority, whatever the Instance count turns out to be." An answer that omits the owner, or + lets the practitioner decide because the count was small or because the project recorded no + rule, is wrong (the default owner paragraph in the terminology entry for Owner still requires a + recorded decision). - **2c.** All six clauses of section [3](../../SPEC.md#3-the-method) were followed for that defect, and the verdict rests on the reviewer reproducing the red and green runs, not on the report. Key quotations, section [7](../../SPEC.md#7-conformance): "A remediation Conforms if all six clauses of section [3](../../SPEC.md#3-the-method) were followed for that Defect, and the verdict that they were rests on reproduction"; "A verdict on a remediation or a Defence MUST rest on reproduction, not on the report". An answer that accepts the pull request description - as the verdict is wrong, and so is one that states the six clauses and says nothing of - reproduction. + as the verdict is wrong, and so is one that omits reproduction. ## 3. Application: the ledger remediation @@ -56,24 +57,22 @@ a test may serve as the detector; an agent may baseline the remaining instances. Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**, because [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) and [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) were not followed. The **defence does not conform** either, because [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) fails; a reader who says the defence conforms -because the sweep is what failed has missed [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) and that counts against the document only if the -reasoning quotes the section [7](../../SPEC.md#7-conformance) definition of a conforming defence. +because the sweep is what failed has missed [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation). Section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): the **inline ignore on `Report::render()`** was the owner's decision, not the -practitioner's. An answer naming instead the decision not to build the wider rule is half right: +practitioner's. An answer naming instead the decision not to build the wider rule is half right, which is a miss: that is also listed in section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) as the owner's to take, and an answer naming both is correct. An answer naming neither is wrong. A reader who grades [3.5](../../SPEC.md#35-enforce-permanently-and-block) Yes because the rule fails the build has not applied the clause's opening sentence on unrecorded suppression, and that counts against the document. A reader who grades [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) Yes because the URL resolves has read the versioning sentence of [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) -differently, which counts against the document; a reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the -fixture. A reader who attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing +differently, which counts against the document. A reader who attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as covering it, which is a fair reading and is not marked wrong. ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage quoted. Two or more readers naming the same passage, as +`ACCEPTANCE.md` defines it, is a finding that blocks until the text is changed and a fresh +cohort no longer names it. What to change is the editor's decision; the reader is not asked +and any suggestion it volunteers is discarded unread. diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 4994f74..633d90d 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -5,11 +5,21 @@ share. Runs 1 to 15 predate this directory and are summarised in the changelog e 1.0.0 and 1.0.1 of the method specification, 1.0.0 of the detector specification and 0.2.0 of the toolchain specification; numbering here continues from 16. -A record states, in this order: the document; the pull request and the commit the readers were -given; the cohort size and model; the run number within that pull request; the marks per reader -per question against the key; the passages quoted as confusing and by how many readers; the -findings applied to the text as a result; and the verdict against the criteria in -`ACCEPTANCE.md`. Answer sheets are not pasted in full; the marks are the record. +A record states what `ACCEPTANCE.md` lists, in this order: the document; the pull request and +the commit the readers were given; the cohort size and model, with the model version each reader +reported; how the readers were isolated and the command they were started with; whether the +machine applies managed (policy) settings to the readers; that the editor attests the rest of the +isolation; the readers set aside and the cause; the run number within that pull request, counting +void runs; who marked and, for a delegated marker, how it was started; the marks per reader per +question against the key, with any judgement made in marking and, on a re-mark, which rule and +why; the passages quoted as confusing, each with the words quoted, verbatim, or the description used where none were quoted, and its clause, and +by how many readers; the findings applied to the text as a result; whether the run was made +under a process change before that change merged and, if so, at which commit; and the +verdict against the criteria in `ACCEPTANCE.md`, with a link to the answer sheets. Earlier +records show the style only; where one differs from this order, this file wins. Answer sheets +are not kept in this repository, and the marks are the record; the sheets, each the reader's +whole output, and the instruction a delegated marker was given are posted in one or more +comments on the pull request, each linked from the record, and never edited, hidden or deleted. The `Unreleased` changelog line for the change names the record, and `tools/changes.py` refuses a specification change whose pull request adds no file here. diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 5a2602b..cae0efc 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -1,6 +1,7 @@ # Answer key for TOOLING-SPEC.md -Marked against the document at the version named in the changelog entry that records the run. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, update the quotation here in the same commit. @@ -73,13 +74,11 @@ Declaration: **does not affect the verdict**. Key quotation, [9.2](../../TOOLING A reader who says Zigpipe cannot be graded without one has read [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) wrongly. A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes on the grounds that Lintomatic "mostly" conforms has read either [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) -or the detector document's section [8](../../DETECTOR-SPEC.md#8-conformance) differently; it counts against this document only if the -reasoning quotes [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification). A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes because they missed the two undocumented rules -has misread the fixture. +or the detector document's section [8](../../DETECTOR-SPEC.md#8-conformance) differently. ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage quoted. Two or more readers naming the same passage, as +`ACCEPTANCE.md` defines it, is a finding that blocks until the text is changed and a fresh +cohort no longer names it. What to change is the editor's decision; the reader is not asked +and any suggestion it volunteers is discarded unread.