Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
cd1e4e2
Acceptance process: make the cohort runnable and markable mechanically
LTSCommerce Oct 5, 2026
e7a0c5d
Acceptance process: apply the first review
LTSCommerce Oct 5, 2026
ad97dfa
Acceptance process: apply the owner's decisions on review 1
LTSCommerce Oct 5, 2026
d45df88
Acceptance process: answer sheets posted on the pull request, not kep…
LTSCommerce Oct 9, 2026
6cc2fb6
Acceptance process: apply the editor's rulings on review 2
LTSCommerce Oct 9, 2026
23e7d8c
Acceptance process: apply the editor's rulings on review 3
LTSCommerce Oct 9, 2026
dba7461
Acceptance process: apply the editor's rulings on review 4
LTSCommerce Oct 9, 2026
5a863ae
Acceptance process: apply the editor's rulings on review 5
LTSCommerce Oct 9, 2026
b713023
Acceptance process: the commit named before readers start; sheet-visi…
LTSCommerce Oct 9, 2026
a695010
Acceptance process: name pull request 13 in acceptance/CHANGES.md
LTSCommerce Oct 9, 2026
bdfd055
Acceptance process: mark question 2 for its gist, judge open marks as…
LTSCommerce Oct 9, 2026
7f6a196
Acceptance process: review 7, every element of a question 2 answer co…
LTSCommerce Oct 9, 2026
7a4bb2d
Acceptance process: review 8, a set-aside sheet's answers still count…
LTSCommerce Oct 9, 2026
204f95b
Acceptance process: review 9, the fourth row's clearing test covers a…
LTSCommerce Oct 9, 2026
598908b
Acceptance process: a short quotation under question 4 covers the pla…
LTSCommerce Oct 9, 2026
d88ab30
Acceptance process: name runs 19 to 30 as made under this change
LTSCommerce Oct 9, 2026
814567b
Acceptance process: an earlier run's judgement does not bind where it…
LTSCommerce Oct 10, 2026
5cec557
Acceptance process: an earlier run's judgement binds no later run
LTSCommerce Oct 10, 2026
5f01f4c
Acceptance process: a departure is recorded against the same pull req…
LTSCommerce Oct 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
201 changes: 173 additions & 28 deletions ACCEPTANCE.md

Large diffs are not rendered by default.

25 changes: 18 additions & 7 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,9 @@ review. It is a patch version when released.
SHOULD or MAY in a numbered section or appendix. It needs everything a clarity edit needs, plus
the five tests in SCOPE.md answered in the pull request, and it is at least a minor version when
released. `tools/changes.py` tells the two apart mechanically and CI runs it on every pull
request.
request. A change to the process files ACCEPTANCE.md names, this file among them, is neither:
ACCEPTANCE.md says how it is made, and it is recorded in `acceptance/CHANGES.md`, not in
CHANGELOG.md.

## What a pull request contains

Expand All @@ -39,6 +41,9 @@ request.
request template lays them out.
- The observed case that prompted it, for either kind. A clarity edit still has a reason: a
reader who took the passage the wrong way.
- Where a specification's Appendix A changed, the files `python3 build-agent-surfaces.py`
writes, rebuilt with it and committed with the change; CI runs it with `--check` and fails
when they are stale.
- `python3 spec-qa.py` clean and the unit tests passing, which CI checks for you.

A pull request is a suggestion, not a demand. The documents have one editor, who keeps ownership
Expand All @@ -55,20 +60,26 @@ survives, the change is yours.
the editor sets the pre-release header.
2. **Scope.** For an obligation, the editor confirms each SCOPE.md test. A failed test closes the
pull request with a row in `DECLINED.md`.
3. **Review and rewrite.** An independent review of the diff against the document, then the
editor's rewrite, pushed to the pull request branch where the contributor allows it or to an
editor's branch that supersedes it.
3. **Review and rewrite.** An independent review of the diff against the whole document, by a
reviewer who did not draft the change, then the editor's rewrite, pushed to the pull request
branch where the contributor allows it or to an editor's branch that supersedes it. The pull
request names the reviewer and links or contains the review. A wording change, as
ACCEPTANCE.md defines it under "When it runs", means a fresh run.
4. **Keys.** Where the change touches a sentence an acceptance key quotes, or adds a load-bearing
obligation, the editor updates the key before the cohort runs, never after.
5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text. Its record is
committed under `acceptance/runs/`, the `Unreleased` changelog line names it, and CI refuses
a specification change that lacks one. For an obligation, the editor sets the document's
header to `<next version>-dev, unpublished` in the same commit if it still names a published
header to `<next version>-dev, unpublished`, and the status paragraph and citation that state
its version to `<next version>-dev`, in the same commit if the header still names a published
version. A wording change after the run means a fresh run.
6. **Merge** into `next` by merge commit, and the branch is deleted.

Steps 2 to 5 are done by a person with a Claude Code session, not by CI, and they happen in
batches. A quiet pull request is not a declined one; a declined one says so.
Steps 2 to 5 are the editor's, not CI's, and they happen in batches. The editor may have an
agent draft or review, and may delegate marking as ACCEPTANCE.md allows, in which case the
marker's verdict is final unless the editor shows a rule was misapplied, as ACCEPTANCE.md says;
every other decision in these steps stays the editor's. A quiet pull request is not a declined
one; a declined one says so.

## Versions and releases

Expand Down
55 changes: 55 additions & 0 deletions acceptance/CHANGES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Changes to the acceptance process

Changes to the process files that [ACCEPTANCE.md](../ACCEPTANCE.md) names, recorded here as it
describes: a line is added at the top for each change, naming its pull request. The process is
not a specification and is not versioned, and a change to it is not tested by a cohort. Changes
to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md).

- [Pull request #13](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/13): the cohort can be run and marked without judgement calls that the rules
leave open; this applies to runs made after it merges, and to runs 19 to 30, which were made under it at the commits their records name, and
runs 16 to 18 stand as recorded. A cohort is an odd
number of readers started, at least five, larger at the editor's choice the more obligations
the change touches, each a separate process with no tools and no project context, started from
an empty directory outside any repository with no user, project or local settings files,
`CLAUDE.md`, skills, plugins, hooks, MCP servers or saved session. A reader is set aside only
if its answer text is empty, its process reports an error, it shared the editor's context, it
was not started as required or it reported another model, and a run with a reader set
aside is void, or fails if the readers who count already fail it, if it fails with the set-aside
sheet's answered items counted too, or if that sheet has a miss traceable to the text; a void run has a number, a record and posted sheets. The packet is the question set, the document and the fixture, each
after one `===== FILE: <name> =====` line. A new marking section says what a delegated marker
receives and that its verdict is final unless a rule was misapplied, that an open judgement is
made as a careful reader of the document would, that an earlier run's judgement binds no
later run, and that a record says where its marker departs from a judgement an earlier run of
the same pull request recorded, and why, and defines a keyed item, quoting (consecutive
words of any length, verbatim apart from case, quotation-mark style and markup), a correct
load-bearing answer, read with its quotation, asserting nothing the document does not state
and omitting no element of the key's answer that answers the question, each read for its
gist, a miss traceable to the text,
including a question 1 bullet that quotes the document, which a key can add to but never
remove, an unanswered item and the same passage. The thresholds apply to the readers whose answers count and block on a single
reader for the first and third rows and on two readers for the fourth, and the fourth row says what a fresh cohort no
longer naming a passage means. A fresh cohort runs after every change following a failure,
every void run and every wording change, which is any change to the text after the readers
read it except a version number or date the editor sets for a release, and step 5 sets the
status paragraph and citation to the `-dev` version with the header. The record adds the
model version each reader reported, the isolation, managed settings, readers set aside, void
runs, re-marks and each confusing passage's words verbatim, and links each reader's whole
output and the marker's instruction, posted on the pull request and never edited, hidden or
deleted, or a run not already failed is void. The keys are marked at
the commit the readers read and no longer limit which misses count against the document,
removing such a limit is named as raising the bar; the method key's "half right" is a miss.
Each marking rule writes down how runs 16 to 18 marked or is stricter, so none relaxes `next`.
The process files include the checks that enforce them, and a process change goes through
steps 1, 3 and 6 of `CONTRIBUTING.md`, applies to runs made after it merges, or earlier
ones made under it at a named commit, and is recorded here, not in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the change, points to
the definition of a wording change, and adds the rebuild of the files `build-agent-surfaces.py`
writes after an Appendix A change. Prompted by gaps in the text as it stood: `ACCEPTANCE.md`
named no fixture in what a reader receives whilst `acceptance/README.md` did, the thresholds
were written as five of five and four of five for a cohort of at least five, two pass-criteria
rows both covered a single reader's traceable miss, and nothing settled an unanswered question
or "the same passage". The rule on an earlier run's judgement was prompted by run 22, which
made a judgement by the stricter reading, a tie-break this change had already replaced with a
careful reader's before run 22 was made, so that the rule it applied was not in force, and which
later runs of pull request 14 then followed as precedent. [Run
16](runs/016-deferral-clause-run-1.md), [run 17](runs/017-deferral-clause-run-2.md) and
[run 18](runs/018-deferral-clause-run-3.md) ruled on these case by case.
25 changes: 15 additions & 10 deletions acceptance/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# Acceptance question sets

The question sets and answer keys that [ACCEPTANCE.md](../ACCEPTANCE.md) requires. One directory per
specification document, each holding the fixed questions a cold reader answers and the key an
editor marks against. The fixture cases the application question uses live in `fixtures/`, and
the record of every run lives in `runs/`.
specification document, each holding the fixed questions a
[cold reader](../ACCEPTANCE.md#the-cohort) answers and the key an editor marks against. The
fixture cases the application question uses live in `fixtures/`, the record of every run lives in
`runs/`, and changes to the acceptance process itself are recorded in [CHANGES.md](CHANGES.md).

| Document | Questions | Key |
| ------------------ | ------------------------------------------------ | ------------------------------------ |
Expand All @@ -14,13 +15,15 @@ the record of every run lives in `runs/`.
## How a run works

1. The editor identifies which documents the change touched. Only those are tested.
2. For each, five or more fresh low-strength readers each receive the document's current text,
its `QUESTIONS.md`, and the fixture the questions name. Nothing else: no key, no changelog, no
critique findings, no other reader's answers.
3. The editor marks each answer sheet against `KEY.md` and applies the pass criteria in
`ACCEPTANCE.md`.
4. The result is recorded as a file under [runs/](runs/), with cohort size, model, run number,
marks and the findings applied, and the change's `Unreleased` changelog line names that file.
2. For each, an odd number of fresh low-strength readers, at least five, each isolated as
`ACCEPTANCE.md` describes, each receive the document's current text, its `QUESTIONS.md`, and
the fixture the questions name. Nothing else: no key, no changelog, no critique findings, no
other reader's answers.
3. The editor, or a separate marker the editor delegates to, marks each answer sheet against
`KEY.md` under the marking rules in `ACCEPTANCE.md` and applies its pass criteria.
4. The result is recorded as a file under [runs/](runs/), stating what `ACCEPTANCE.md` lists,
with the answer sheets posted on the pull request and linked from it, and the change's
`Unreleased` changelog line names that file.
A run that fails is followed by a change to the text, never to the questions, and a fresh
cohort.

Expand All @@ -32,6 +35,8 @@ matches the document is a defect in this directory, not a reason to mark a reade

Questions are fixed so that runs can be compared across versions. Adding a keyed question is
allowed when a new clause is load-bearing; removing or softening one to make a run pass is not.
A process change may remove a limit a key puts on which misses count against the document, as
[the section on the bar](../ACCEPTANCE.md#the-bar-only-moves-up) allows.
The reader may quote a passage it found confusing; it is never asked to rate the document, say
what would make a passage clearer, or propose a change: the cohort confirms comprehension and
does not design the text.
17 changes: 7 additions & 10 deletions acceptance/detector/KEY.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Answer key for DETECTOR-SPEC.md

Marked against the document at the version named in the changelog entry that records the run.
Marked against the document at the commit the readers read, with this key, its question set and
fixture at that same commit.
Quotations below are from the document; when a clause is reworded, update the quotation here in
the same commit.

Expand Down Expand Up @@ -60,18 +61,14 @@ agent summary (those belong to the toolchain specification).

Section [8](../../DETECTOR-SPEC.md#8-conformance) verdict: **does not conform**, because [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) fails and partial conformance MUST NOT be
described as conformance. A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Partial on the grounds that thirty-eight resolve
has read the clause differently and that counts against the document; a reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes
because they missed the last paragraph of the fixture has misread the fixture, not the document.
has read the clause differently and that counts against the document.

Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**. It is a SHOULD, and Lintomatic's failure to check its
own identifiers in CI is a missed SHOULD, not a further MUST failure; the MUST that fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together).

A reader who marks [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) No because the comment takes no reason has confused [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) with [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), which is
SHOULD; that counts against the document if the reasoning quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason).

## 4. Confusion

No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a
finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to
change is the editor's decision; the reader is not asked and any suggestion it volunteers is
discarded unread.
No keyed answer. Record every passage quoted. Two or more readers naming the same passage, as
`ACCEPTANCE.md` defines it, is a finding that blocks until the text is changed and a fresh
cohort no longer names it. What to change is the editor's decision; the reader is not asked
and any suggestion it volunteers is discarded unread.
Loading
Loading