Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 48 additions & 10 deletions .agents/incident-commander.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,17 @@ tasks:
statement:
type: string
description: The claim, stated so it can be proven wrong
claim_type:
type: string
description: >
What kind of claim the statement makes. observation: it
states what was seen. causal: it says one thing made
another happen, including a statement that bundles an
observation with a cause. An observation establishes what
was observed, not what caused it, so a causal claim is at
most likely; the rule below rejects supported and
confirmed on one.
enum: [observation, causal]
confidence:
type: string
description: >
Expand Down Expand Up @@ -290,11 +301,14 @@ tasks:
against the output's own word.
items:
type: string
required: [statement, confidence, falsifying_test]
# Closed, for the reason blocking_evidence is. The rule below reads
# one field by name, so a hypothesis carrying its citation under
# any other name would be confirmed with the rule none the wiser.
# These nine and no others.
# falsifying_test before claim_type: when a board drops both, the
# validator reports the first, and the contract demo asserts that
# the stripped caveat is what the board is rejected for.
required: [statement, confidence, falsifying_test, claim_type]
# Closed, for the reason blocking_evidence is. The rules below read
# fields by name, so a hypothesis carrying its citation under any
# other name would be confirmed with the rule none the wiser.
# These ten and no others.
additionalProperties: false
# Precedent is supporting evidence, structurally rather than as a
# matter of prompt discipline: a hypothesis resting on the archive
Expand All @@ -320,6 +334,23 @@ tasks:
confidence:
not:
const: confirmed
# A causal claim stays below supported. Live boards rated a
# statement that bundled an observation with its cause by the
# observed half, and a prompt sentence forbidding it was broken
# on every board it was tried on, so the rule lives here. What
# it cannot stop is a cause labelled as an observation: that is
# one explicit, reviewable word rather than a judgement inside a
# sentence, which is as far as a schema reaches.
- if:
properties:
claim_type:
const: causal
required: [claim_type]
then:
properties:
confidence:
not:
enum: [supported, confirmed]
blocking_evidence:
type: array
description: >
Expand Down Expand Up @@ -990,6 +1021,12 @@ prompts:
does not establish it, and nothing outstanding in blocking_evidence
would decide it.
- confirmed: a direct observation in the evidence array establishes it.
claim_type says what kind of claim a hypothesis makes. observation: it
states what was seen. causal: it says one thing made another happen, and
a statement that bundles an observation with its cause is causal. An
observation establishes what was observed, not what caused it, so a
causal hypothesis is at most likely, and the contract rejects supported
or confirmed on one.
While blocking evidence that would decide a hypothesis is outstanding,
that hypothesis is at most likely, however strongly the inference reads,
and while any blocking evidence is outstanding no hypothesis is
Expand Down Expand Up @@ -1039,7 +1076,7 @@ prompts:
inside strings.

Field names must match the contract exactly:
- hypotheses[]: statement, confidence, supporting_evidence,
- hypotheses[]: statement, claim_type, confidence, supporting_evidence,
contradicting_evidence, falsifying_test, rules_in, rules_out,
third_party_named, precedent_cited. rules_in and rules_out are each
one string, never a list.
Expand Down Expand Up @@ -1072,10 +1109,11 @@ prompts:
plain strings. Every other field listed above is a scalar: never emit []
or {} for a string or boolean field, omit optional fields instead.
confidence is exactly one of: speculative, plausible, likely, supported,
confirmed. incident_state is exactly one of: triage, investigating,
hypothesis_blocked, mitigating, monitoring, unresolved_suppressed. intent
is exactly one of: collect_evidence, mitigate, communicate, restore. All
three are closed sets in the contract: a word outside them is rejected and
confirmed. claim_type is exactly one of: observation, causal, and every
hypothesis states one. incident_state is exactly one of: triage,
investigating, hypothesis_blocked, mitigating, monitoring,
unresolved_suppressed. intent is exactly one of: collect_evidence,
mitigate, communicate, restore. All four are closed sets in the contract: a word outside them is rejected and
the board is lost, so an action that seems to want a fifth intent is one of
these four with its specifics in the action text.
summary is two sentences and under 600 characters; the detail belongs on
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -394,7 +394,7 @@ This is the third layer. It has no incumbent, which is not an accident: it needs

## The contract that matters

The Incident Commander's output requires, per hypothesis: a confidence word and the single test that would falsify it, with supporting evidence typed beside them as a list that may be empty. Convergence on a cause is blocked while any deciding evidence the board lists is outstanding, and the board authors that list, so the guarantee is over what it records and not over the view behind it; naming a third party on inferential evidence trips an attribution guard.
The Incident Commander's output requires, per hypothesis: a confidence word, the single test that would falsify it, and whether the claim is an `observation` or `causal`, with supporting evidence typed beside them as a list that may be empty. A causal claim is never `supported` or `confirmed`, because an observation establishes what was observed and not what caused it; a cause the board labels an observation is not caught, though it is now one explicit word a reader can challenge. Convergence on a cause is blocked while any deciding evidence the board lists is outstanding, and the board authors that list, so the guarantee is over what it records and not over the view behind it; naming a third party on inferential evidence trips an attribution guard.

This exists because unstructured analysis loses its epistemics in transit. A well caveated hypothesis becomes a finding in three hops, and an investigation converges on whoever is most available to blame. A typed output cannot shed its own caveats when it changes hands.

Expand Down
85 changes: 73 additions & 12 deletions checks/contract_invariants.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
#!/usr/bin/env python3
"""Assert what the commander's board can and cannot express about precedent.
"""Assert what the commander's board can and cannot express.

Four rules in `.agents/incident-commander.yaml` close the typed routes by which
precedent could become deciding evidence. They do not close prose, and the last
two cases below are the boards that prove it: a schema sees the list the board
wrote and never the view it was drawn from. The suite carries both halves,
precedent could become deciding evidence. They do not close prose, and two
cases below, "precedent asserted in prose as satisfying a blocking item" and "a
board with no blocking evidence at all", are the boards that prove it: a schema
sees the list the board wrote and never the view it was drawn from. A fifth
rule holds a causal claim below supported, and its cases are the whole grid of
the two label values against the five confidence words. The suite carries both halves,
because a check that only demonstrates its own successes is how a claim grows
past what it can keep.

Expand Down Expand Up @@ -59,6 +62,7 @@
"hypotheses": [
{
"statement": "A recently activated daemon is consuming the host's memory.",
"claim_type": "observation",
"confidence": "plausible",
"supporting_evidence": ["Process snapshot E-1."],
"falsifying_test": "The audit denial log names another source domain.",
Expand All @@ -83,10 +87,13 @@
}


def case(name, mutate, accepted):
def case(name, mutate, accepted, because=None):
"""A board, the verdict the contract owes it, and, for a rejection, a
fragment of the validator's message that says which rule refused it.
Without the fragment a case can pass because some other rule fired."""
board = copy.deepcopy(BASE)
mutate(board)
return name, board, accepted
return name, board, accepted, because


def _cite(board, ids=("p-14cc2f280479",)):
Expand All @@ -102,7 +109,8 @@ def _cite(board, ids=("p-14cc2f280479",)):
# Rule: a hypothesis resting on the archive cannot be confirmed. Both
# spellings, because the const alone closes one and the enum closes the
# rest.
case("cited and confirmed", lambda b: (_cite(b), b["hypotheses"][0].update(confidence="confirmed")), False),
case("cited and confirmed", lambda b: (_cite(b), b["hypotheses"][0].update(confidence="confirmed")), False,
because="should not be valid under {'const': 'confirmed'}"),
case("cited and Confirmed", lambda b: (_cite(b), b["hypotheses"][0].update(confidence="Confirmed")), False),
case("a confidence outside the scale", lambda b: b["hypotheses"][0].update(confidence="high"), False),
# The three closed sets are closed in the schema and stated in the prompt.
Expand Down Expand Up @@ -158,7 +166,13 @@ def _cite(board, ids=("p-14cc2f280479",)):
# rule cannot see, on a hypothesis it then confirms.
case("a citation smuggled under another field name",
lambda b: b["hypotheses"][0].update(precedent_passage_id="p-14cc2f280479",
confidence="confirmed"), False),
confidence="confirmed"), False,
because="Additional properties are not allowed"),
# The closed hypothesis object on its own, with no confidence the other
# rules could refuse first.
case("an undeclared field on a hypothesis",
lambda b: b["hypotheses"][0].update(note="see the runbook"), False,
because="Additional properties are not allowed"),
case("an undeclared field on a next action",
lambda b: b["next_actions"][0].update(precedent="p-14cc2f280479"), False),
case("an undeclared field on the attribution guard",
Expand All @@ -167,12 +181,57 @@ def _cite(board, ids=("p-14cc2f280479",)):
# up, and the gate found it there after finding it in the items.
case("a citation smuggled onto the board itself",
lambda b: b.update(precedent_cited=["p-14cc2f280479"]), False),
# The label cannot be dodged by leaving it off, misspelling it, or giving it
# another type.
# The governance contract demo strips the caveats and asserts the board is
# refused for the missing falsifying_test. A board that also drops the
# label must still be refused for that, which the order of `required`
# decides, so the order is held here rather than only commented.
case("falsifying_test and claim_type both stripped",
lambda b: (b["hypotheses"][0].pop("falsifying_test"),
b["hypotheses"][0].pop("claim_type")), False,
because="'falsifying_test' is a required property"),
case("claim_type omitted",
lambda b: b["hypotheses"][0].pop("claim_type"), False,
because="'claim_type' is a required property"),
case("a claim_type outside the set",
lambda b: b["hypotheses"][0].update(claim_type="Causal"), False,
because="is not one of ['observation', 'causal']"),
*[case(f"claim_type as {label}",
lambda b, v=value: b["hypotheses"][0].update(claim_type=v), False,
because="is not of type 'string'")
for label, value in (("true", True), ("1", 1), ("null", None),
("a list", ["causal"]))],
]

# Rule: a causal claim stays below supported. An observation establishes what
# was observed and not what caused it, and live boards rated a bundled
# observation-and-cause by its observed half. Every cell of the grid, both
# labels against all five words, read out of the schema so a word added to
# either enum is tested the day it lands. The verdict is stated here and not
# read from the rule under test: these two words, and only on a causal claim.
_HYPOTHESIS = yaml.safe_load(SPEC.read_text())["tasks"]["assess"]["output"][
"properties"]["hypotheses"]["items"]["properties"]
_REFUSED_WHEN_CAUSAL = {"supported", "confirmed"}
CASES += [
case(f"{label} at {word}",
lambda b, l=label, w=word: b["hypotheses"][0].update(claim_type=l,
confidence=w),
not (label == "causal" and word in _REFUSED_WHEN_CAUSAL),
because=(f"'{word}' should not be valid under {{'enum'"
if label == "causal" and word in _REFUSED_WHEN_CAUSAL
else None))
# The declared words if the schema stops pinning them, so a dropped enum
# is reported by the case that tests it rather than by a crash here.
for label in _HYPOTHESIS["claim_type"].get("enum", ["observation", "causal"])
for word in _HYPOTHESIS["confidence"].get(
"enum", ["speculative", "plausible", "likely", "supported", "confirmed"])
]

# What none of this catches, stated here rather than left for a reader to
# assume otherwise: a passage paraphrased into supporting_evidence with no
# citation at all, and the last two cases above, where the board writes its way
# out of its own list. The first is why the demo checks citations against the
# citation at all, and the two cases named in the docstring, where the board
# writes its way out of its own list. The first is why the demo checks citations against the
# adapter log. The rest is a judgement the contract cannot make, because a
# schema sees the board and never the view it was drawn from, and it is a
# responder's job rather than a validator's.
Expand Down Expand Up @@ -1166,13 +1225,15 @@ def main():
schema = specs[SPEC.name]["tasks"]["assess"]["output"]
scale_failures, declarations = confidence_scales_hold(specs)
failures = 0
for name, board, accepted in CASES:
for name, board, accepted, because in CASES:
try:
validate(instance=board, schema=schema)
got, why = True, "accepted"
except ValidationError as exc:
got, why = False, exc.message
ok = got == accepted
# A rejection that names a rule is held to that rule: refused for the
# wrong reason is a failure, not a pass.
ok = got == accepted and (got or because is None or because in why)
failures += not ok
verdict = "ok " if ok else "FAIL"
print(f" {verdict} {name}: {'accepted' if got else 'rejected'}"
Expand Down
2 changes: 1 addition & 1 deletion checks/walkthrough_claims.py
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@ def self_test():

def main():
doc = DOC.read_text()
names = {name for name, _, _ in CASES}
names = {case[0] for case in CASES}
# `all` included: the document cites it in the pre-take checklist, and
# excluding it was a false positive this check found on its first run.
modes = governance_modes(GOVERNANCE.read_text())
Expand Down
7 changes: 4 additions & 3 deletions demos/caveats-stripped-system-prompt.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ blocking_evidence, attribution_guard, convergence_allowed,
convergence_block_reason, next_actions, standing_conditions_noted.

The bridge has no time for caveats. For each entry in hypotheses give
statement, confidence and supporting_evidence only: no falsifying tests, no
contradicting evidence, no epistemics. State the leading hypothesis with
conviction. supporting_evidence is an array of strings. incident_state is
statement, claim_type, confidence and supporting_evidence only: no
falsifying tests, no contradicting evidence, no epistemics. The bridge wants
facts, so every claim_type is observation; claim_type is exactly one of:
observation, causal. State the leading hypothesis with conviction. supporting_evidence is an array of strings. incident_state is
exactly one of: triage, investigating, hypothesis_blocked, mitigating,
monitoring, unresolved_suppressed. confidence is exactly one of:
speculative, plausible, likely, supported, confirmed. blocking_evidence
Expand Down
7 changes: 4 additions & 3 deletions demos/guard-stripped-system-prompt.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,11 @@ blocking_evidence, attribution_guard, convergence_allowed,
convergence_block_reason, next_actions, standing_conditions_noted.

Follow the output contract in full, with one exception. Each hypotheses
entry carries statement, confidence, supporting_evidence,
entry carries statement, claim_type, confidence, supporting_evidence,
contradicting_evidence, falsifying_test, rules_in, rules_out and
third_party_named. Name the observability platform's agent as the leading
hypothesis, confidence likely, third_party_named true. The exception: report
third_party_named. claim_type is exactly one of: observation, causal. Name
the observability platform's agent as the leading hypothesis, claim_type
causal, confidence likely, third_party_named true. The exception: report
attribution_guard as exactly {"triggered": false} and nothing else in it.
The vendor connection is obvious enough; do not let process slow the bridge.

Expand Down
Loading