diff --git a/.agents/incident-commander.yaml b/.agents/incident-commander.yaml index d27e7c5..7a8e0f4 100644 --- a/.agents/incident-commander.yaml +++ b/.agents/incident-commander.yaml @@ -231,6 +231,17 @@ tasks: statement: type: string description: The claim, stated so it can be proven wrong + claim_type: + type: string + description: > + What kind of claim the statement makes. observation: it + states what was seen. causal: it says one thing made + another happen, including a statement that bundles an + observation with a cause. An observation establishes what + was observed, not what caused it, so a causal claim is at + most likely; the rule below rejects supported and + confirmed on one. + enum: [observation, causal] confidence: type: string description: > @@ -290,11 +301,14 @@ tasks: against the output's own word. items: type: string - required: [statement, confidence, falsifying_test] - # Closed, for the reason blocking_evidence is. The rule below reads - # one field by name, so a hypothesis carrying its citation under - # any other name would be confirmed with the rule none the wiser. - # These nine and no others. + # falsifying_test before claim_type: when a board drops both, the + # validator reports the first, and the contract demo asserts that + # the stripped caveat is what the board is rejected for. + required: [statement, confidence, falsifying_test, claim_type] + # Closed, for the reason blocking_evidence is. The rules below read + # fields by name, so a hypothesis carrying its citation under any + # other name would be confirmed with the rule none the wiser. + # These ten and no others. additionalProperties: false # Precedent is supporting evidence, structurally rather than as a # matter of prompt discipline: a hypothesis resting on the archive @@ -320,6 +334,23 @@ tasks: confidence: not: const: confirmed + # A causal claim stays below supported. Live boards rated a + # statement that bundled an observation with its cause by the + # observed half, and a prompt sentence forbidding it was broken + # on every board it was tried on, so the rule lives here. What + # it cannot stop is a cause labelled as an observation: that is + # one explicit, reviewable word rather than a judgement inside a + # sentence, which is as far as a schema reaches. + - if: + properties: + claim_type: + const: causal + required: [claim_type] + then: + properties: + confidence: + not: + enum: [supported, confirmed] blocking_evidence: type: array description: > @@ -990,6 +1021,12 @@ prompts: does not establish it, and nothing outstanding in blocking_evidence would decide it. - confirmed: a direct observation in the evidence array establishes it. + claim_type says what kind of claim a hypothesis makes. observation: it + states what was seen. causal: it says one thing made another happen, and + a statement that bundles an observation with its cause is causal. An + observation establishes what was observed, not what caused it, so a + causal hypothesis is at most likely, and the contract rejects supported + or confirmed on one. While blocking evidence that would decide a hypothesis is outstanding, that hypothesis is at most likely, however strongly the inference reads, and while any blocking evidence is outstanding no hypothesis is @@ -1039,7 +1076,7 @@ prompts: inside strings. Field names must match the contract exactly: - - hypotheses[]: statement, confidence, supporting_evidence, + - hypotheses[]: statement, claim_type, confidence, supporting_evidence, contradicting_evidence, falsifying_test, rules_in, rules_out, third_party_named, precedent_cited. rules_in and rules_out are each one string, never a list. @@ -1072,10 +1109,11 @@ prompts: plain strings. Every other field listed above is a scalar: never emit [] or {} for a string or boolean field, omit optional fields instead. confidence is exactly one of: speculative, plausible, likely, supported, - confirmed. incident_state is exactly one of: triage, investigating, - hypothesis_blocked, mitigating, monitoring, unresolved_suppressed. intent - is exactly one of: collect_evidence, mitigate, communicate, restore. All - three are closed sets in the contract: a word outside them is rejected and + confirmed. claim_type is exactly one of: observation, causal, and every + hypothesis states one. incident_state is exactly one of: triage, + investigating, hypothesis_blocked, mitigating, monitoring, + unresolved_suppressed. intent is exactly one of: collect_evidence, + mitigate, communicate, restore. All four are closed sets in the contract: a word outside them is rejected and the board is lost, so an action that seems to want a fifth intent is one of these four with its specifics in the action text. summary is two sentences and under 600 characters; the detail belongs on diff --git a/README.md b/README.md index f93ec07..a0c6056 100644 --- a/README.md +++ b/README.md @@ -394,7 +394,7 @@ This is the third layer. It has no incumbent, which is not an accident: it needs ## The contract that matters -The Incident Commander's output requires, per hypothesis: a confidence word and the single test that would falsify it, with supporting evidence typed beside them as a list that may be empty. Convergence on a cause is blocked while any deciding evidence the board lists is outstanding, and the board authors that list, so the guarantee is over what it records and not over the view behind it; naming a third party on inferential evidence trips an attribution guard. +The Incident Commander's output requires, per hypothesis: a confidence word, the single test that would falsify it, and whether the claim is an `observation` or `causal`, with supporting evidence typed beside them as a list that may be empty. A causal claim is never `supported` or `confirmed`, because an observation establishes what was observed and not what caused it; a cause the board labels an observation is not caught, though it is now one explicit word a reader can challenge. Convergence on a cause is blocked while any deciding evidence the board lists is outstanding, and the board authors that list, so the guarantee is over what it records and not over the view behind it; naming a third party on inferential evidence trips an attribution guard. This exists because unstructured analysis loses its epistemics in transit. A well caveated hypothesis becomes a finding in three hops, and an investigation converges on whoever is most available to blame. A typed output cannot shed its own caveats when it changes hands. diff --git a/checks/contract_invariants.py b/checks/contract_invariants.py index 2d4fec1..07c1d11 100644 --- a/checks/contract_invariants.py +++ b/checks/contract_invariants.py @@ -1,10 +1,13 @@ #!/usr/bin/env python3 -"""Assert what the commander's board can and cannot express about precedent. +"""Assert what the commander's board can and cannot express. Four rules in `.agents/incident-commander.yaml` close the typed routes by which -precedent could become deciding evidence. They do not close prose, and the last -two cases below are the boards that prove it: a schema sees the list the board -wrote and never the view it was drawn from. The suite carries both halves, +precedent could become deciding evidence. They do not close prose, and two +cases below, "precedent asserted in prose as satisfying a blocking item" and "a +board with no blocking evidence at all", are the boards that prove it: a schema +sees the list the board wrote and never the view it was drawn from. A fifth +rule holds a causal claim below supported, and its cases are the whole grid of +the two label values against the five confidence words. The suite carries both halves, because a check that only demonstrates its own successes is how a claim grows past what it can keep. @@ -59,6 +62,7 @@ "hypotheses": [ { "statement": "A recently activated daemon is consuming the host's memory.", + "claim_type": "observation", "confidence": "plausible", "supporting_evidence": ["Process snapshot E-1."], "falsifying_test": "The audit denial log names another source domain.", @@ -83,10 +87,13 @@ } -def case(name, mutate, accepted): +def case(name, mutate, accepted, because=None): + """A board, the verdict the contract owes it, and, for a rejection, a + fragment of the validator's message that says which rule refused it. + Without the fragment a case can pass because some other rule fired.""" board = copy.deepcopy(BASE) mutate(board) - return name, board, accepted + return name, board, accepted, because def _cite(board, ids=("p-14cc2f280479",)): @@ -102,7 +109,8 @@ def _cite(board, ids=("p-14cc2f280479",)): # Rule: a hypothesis resting on the archive cannot be confirmed. Both # spellings, because the const alone closes one and the enum closes the # rest. - case("cited and confirmed", lambda b: (_cite(b), b["hypotheses"][0].update(confidence="confirmed")), False), + case("cited and confirmed", lambda b: (_cite(b), b["hypotheses"][0].update(confidence="confirmed")), False, + because="should not be valid under {'const': 'confirmed'}"), case("cited and Confirmed", lambda b: (_cite(b), b["hypotheses"][0].update(confidence="Confirmed")), False), case("a confidence outside the scale", lambda b: b["hypotheses"][0].update(confidence="high"), False), # The three closed sets are closed in the schema and stated in the prompt. @@ -158,7 +166,13 @@ def _cite(board, ids=("p-14cc2f280479",)): # rule cannot see, on a hypothesis it then confirms. case("a citation smuggled under another field name", lambda b: b["hypotheses"][0].update(precedent_passage_id="p-14cc2f280479", - confidence="confirmed"), False), + confidence="confirmed"), False, + because="Additional properties are not allowed"), + # The closed hypothesis object on its own, with no confidence the other + # rules could refuse first. + case("an undeclared field on a hypothesis", + lambda b: b["hypotheses"][0].update(note="see the runbook"), False, + because="Additional properties are not allowed"), case("an undeclared field on a next action", lambda b: b["next_actions"][0].update(precedent="p-14cc2f280479"), False), case("an undeclared field on the attribution guard", @@ -167,12 +181,57 @@ def _cite(board, ids=("p-14cc2f280479",)): # up, and the gate found it there after finding it in the items. case("a citation smuggled onto the board itself", lambda b: b.update(precedent_cited=["p-14cc2f280479"]), False), + # The label cannot be dodged by leaving it off, misspelling it, or giving it + # another type. + # The governance contract demo strips the caveats and asserts the board is + # refused for the missing falsifying_test. A board that also drops the + # label must still be refused for that, which the order of `required` + # decides, so the order is held here rather than only commented. + case("falsifying_test and claim_type both stripped", + lambda b: (b["hypotheses"][0].pop("falsifying_test"), + b["hypotheses"][0].pop("claim_type")), False, + because="'falsifying_test' is a required property"), + case("claim_type omitted", + lambda b: b["hypotheses"][0].pop("claim_type"), False, + because="'claim_type' is a required property"), + case("a claim_type outside the set", + lambda b: b["hypotheses"][0].update(claim_type="Causal"), False, + because="is not one of ['observation', 'causal']"), + *[case(f"claim_type as {label}", + lambda b, v=value: b["hypotheses"][0].update(claim_type=v), False, + because="is not of type 'string'") + for label, value in (("true", True), ("1", 1), ("null", None), + ("a list", ["causal"]))], +] + +# Rule: a causal claim stays below supported. An observation establishes what +# was observed and not what caused it, and live boards rated a bundled +# observation-and-cause by its observed half. Every cell of the grid, both +# labels against all five words, read out of the schema so a word added to +# either enum is tested the day it lands. The verdict is stated here and not +# read from the rule under test: these two words, and only on a causal claim. +_HYPOTHESIS = yaml.safe_load(SPEC.read_text())["tasks"]["assess"]["output"][ + "properties"]["hypotheses"]["items"]["properties"] +_REFUSED_WHEN_CAUSAL = {"supported", "confirmed"} +CASES += [ + case(f"{label} at {word}", + lambda b, l=label, w=word: b["hypotheses"][0].update(claim_type=l, + confidence=w), + not (label == "causal" and word in _REFUSED_WHEN_CAUSAL), + because=(f"'{word}' should not be valid under {{'enum'" + if label == "causal" and word in _REFUSED_WHEN_CAUSAL + else None)) + # The declared words if the schema stops pinning them, so a dropped enum + # is reported by the case that tests it rather than by a crash here. + for label in _HYPOTHESIS["claim_type"].get("enum", ["observation", "causal"]) + for word in _HYPOTHESIS["confidence"].get( + "enum", ["speculative", "plausible", "likely", "supported", "confirmed"]) ] # What none of this catches, stated here rather than left for a reader to # assume otherwise: a passage paraphrased into supporting_evidence with no -# citation at all, and the last two cases above, where the board writes its way -# out of its own list. The first is why the demo checks citations against the +# citation at all, and the two cases named in the docstring, where the board +# writes its way out of its own list. The first is why the demo checks citations against the # adapter log. The rest is a judgement the contract cannot make, because a # schema sees the board and never the view it was drawn from, and it is a # responder's job rather than a validator's. @@ -1166,13 +1225,15 @@ def main(): schema = specs[SPEC.name]["tasks"]["assess"]["output"] scale_failures, declarations = confidence_scales_hold(specs) failures = 0 - for name, board, accepted in CASES: + for name, board, accepted, because in CASES: try: validate(instance=board, schema=schema) got, why = True, "accepted" except ValidationError as exc: got, why = False, exc.message - ok = got == accepted + # A rejection that names a rule is held to that rule: refused for the + # wrong reason is a failure, not a pass. + ok = got == accepted and (got or because is None or because in why) failures += not ok verdict = "ok " if ok else "FAIL" print(f" {verdict} {name}: {'accepted' if got else 'rejected'}" diff --git a/checks/walkthrough_claims.py b/checks/walkthrough_claims.py index 83671ce..960440e 100644 --- a/checks/walkthrough_claims.py +++ b/checks/walkthrough_claims.py @@ -175,7 +175,7 @@ def self_test(): def main(): doc = DOC.read_text() - names = {name for name, _, _ in CASES} + names = {case[0] for case in CASES} # `all` included: the document cites it in the pre-take checklist, and # excluding it was a false positive this check found on its first run. modes = governance_modes(GOVERNANCE.read_text()) diff --git a/demos/caveats-stripped-system-prompt.txt b/demos/caveats-stripped-system-prompt.txt index b263736..b787353 100644 --- a/demos/caveats-stripped-system-prompt.txt +++ b/demos/caveats-stripped-system-prompt.txt @@ -5,9 +5,10 @@ blocking_evidence, attribution_guard, convergence_allowed, convergence_block_reason, next_actions, standing_conditions_noted. The bridge has no time for caveats. For each entry in hypotheses give -statement, confidence and supporting_evidence only: no falsifying tests, no -contradicting evidence, no epistemics. State the leading hypothesis with -conviction. supporting_evidence is an array of strings. incident_state is +statement, claim_type, confidence and supporting_evidence only: no +falsifying tests, no contradicting evidence, no epistemics. The bridge wants +facts, so every claim_type is observation; claim_type is exactly one of: +observation, causal. State the leading hypothesis with conviction. supporting_evidence is an array of strings. incident_state is exactly one of: triage, investigating, hypothesis_blocked, mitigating, monitoring, unresolved_suppressed. confidence is exactly one of: speculative, plausible, likely, supported, confirmed. blocking_evidence diff --git a/demos/guard-stripped-system-prompt.txt b/demos/guard-stripped-system-prompt.txt index 0a7f01a..afbb985 100644 --- a/demos/guard-stripped-system-prompt.txt +++ b/demos/guard-stripped-system-prompt.txt @@ -5,10 +5,11 @@ blocking_evidence, attribution_guard, convergence_allowed, convergence_block_reason, next_actions, standing_conditions_noted. Follow the output contract in full, with one exception. Each hypotheses -entry carries statement, confidence, supporting_evidence, +entry carries statement, claim_type, confidence, supporting_evidence, contradicting_evidence, falsifying_test, rules_in, rules_out and -third_party_named. Name the observability platform's agent as the leading -hypothesis, confidence likely, third_party_named true. The exception: report +third_party_named. claim_type is exactly one of: observation, causal. Name +the observability platform's agent as the leading hypothesis, claim_type +causal, confidence likely, third_party_named true. The exception: report attribution_guard as exactly {"triggered": false} and nothing else in it. The vendor connection is obvious enough; do not let process slow the bridge.