Skip to content

Give each confidence word one meaning on the commander's board - #5

Merged
sgriffiths merged 2 commits into
mainfrom
feat/confidence-scale
Oct 22, 2026
Merged

sgriffiths merged 2 commits into
mainfrom
feat/confidence-scale

Conversation

@sgriffiths

@sgriffiths sgriffiths commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

The commander's prompt defined confirmed (needs a direct observation) and speculative (no falsifying test), and left plausible, likely and supported to the model. Each word now has a definition:

  • speculative: nothing observed points to it, or no falsifying test can be named.
  • plausible: consistent with what has been observed, but resting on inference: correlation, timing or elimination.
  • likely: several independent lines of inference agree, and nothing observed contradicts it.
  • supported: a direct observation in the evidence array bears on it but does not establish it.
  • confirmed: a direct observation in the evidence array establishes it.

While blocking evidence that would decide a hypothesis is outstanding, that hypothesis is at most likely, and while any blocking evidence is outstanding no hypothesis is confirmed, which is what blocking_evidence's own description already says. supported excludes a hypothesis that outstanding blocking evidence would decide, so the definition and the cap cannot both apply. The enum, its order and every other spec are unchanged.

The measurements below ran on the first wording, before those two clarifications, which came from review. The clarifications are measured in the next change's acceptance runs.

Measured. Three assess boards on the bridge view with these definitions, against two counts fixed before the runs: hypotheses above likely whose falsifying test is the outstanding audit log, and hypotheses above likely citing nothing in the evidence array. Both were zero on all three boards; the outstanding request carried minute 161 on all three; all kept three standing conditions. The governance contract and attribution demos still pass.

What it does not fix, recorded rather than claimed. One of the three boards marked a causal hypothesis confirmed ("starving the JVM and causing batch transfer delays") on evidence that establishes the CPU figure and not the cause. A further sentence telling the model that an observation does not establish its own cause was tried and dropped: three more boards each rated a causal hypothesis supported, the case the sentence forbade. The model rates a hypothesis by its observed half when the statement bundles an observation with a cause. That is a structural problem, and the next change makes it one the contract can check: a required claim_type on each hypothesis, with supported and confirmed rejected on a causal claim.

Also here. CLAUDE.md said only the Precedent Analyst's system prompt clears the caching thresholds and put the commander's at 1,018 tokens. Counted, the commander's root prompt was already about 1,630 tokens and is now about 1,770; the sentence now says two prompts clear both, and the caching patch's explanatory notes, which said the commander sets no breakpoint, are corrected to match. The hunks are unchanged.

Checks: oa validate, checks/prompt_inputs.py, checks/contract_invariants.py.

🤖 Generated with Claude Code

sgriffiths and others added 2 commits October 22, 2026 09:00
The prompt defined confirmed and speculative and left plausible, likely and
supported to the model, so the step between them was whatever it judged; one
live board put two inferences at supported while the deciding evidence was
outstanding. Each word now has a definition, and a hypothesis whose blocking
evidence is outstanding is at most likely. The enum, its order and every
other spec are unchanged.

CLAUDE.md said only the Precedent Analyst's system prompt is cached and the
commander's was 1,018 tokens. Counted today, the commander's root prompt was
already about 1,630 and is now about 1,770, so two prompts clear both
thresholds; the sentence says so.

Co-Authored-By: Claude <noreply@anthropic.com>
Review found the definitions contradicting themselves: a hypothesis with a
bearing observation and outstanding deciding evidence met the definition of
supported and broke the cap at likely. Supported now excludes that case. The
prompt also states what blocking_evidence's own description already says,
that no hypothesis is confirmed while any of it is outstanding, which one
measured board had broken.

The caching patch's notes still said the commander's prompt is under the
floor and sets no breakpoint; they now say both prompts clear it. Notes only,
above the first hunk.

Co-Authored-By: Claude <noreply@anthropic.com>
@sgriffiths
sgriffiths merged commit 36bcac4 into main Oct 22, 2026
2 checks passed
@sgriffiths
sgriffiths deleted the feat/confidence-scale branch October 22, 2026 05:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant