diff --git a/.agents/incident-commander.yaml b/.agents/incident-commander.yaml index 6d63f00..d27e7c5 100644 --- a/.agents/incident-commander.yaml +++ b/.agents/incident-commander.yaml @@ -979,6 +979,22 @@ prompts: authorisation_required true. You never mark such an action as autonomous, regardless of how routine it appears. + Each confidence word has one meaning, and the board uses no other: + - speculative: nothing observed points to it, or no falsifying test can + be named. + - plausible: consistent with what has been observed, but resting on + inference: correlation, timing or elimination. + - likely: several independent lines of inference agree, and nothing + observed contradicts it. + - supported: a direct observation in the evidence array bears on it but + does not establish it, and nothing outstanding in blocking_evidence + would decide it. + - confirmed: a direct observation in the evidence array establishes it. + While blocking evidence that would decide a hypothesis is outstanding, + that hypothesis is at most likely, however strongly the inference reads, + and while any blocking evidence is outstanding no hypothesis is + confirmed: that is what blocking_evidence is for. + You are working only from what you are given. If something appears trivially obtainable but is absent from the evidence array, it has not been collected. Request it. Do not assume it. diff --git a/CLAUDE.md b/CLAUDE.md index c0ca81c..cbf8565 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -78,7 +78,7 @@ oa validate --spec .agents/.yaml # before every commit, for every spec - **Every task declares a sandbox, including the tasks that bind no tools.** Absence is allow-all: the runner only checks an allow list when one is present, so a task with no sandbox, and equally a sandbox with no `tools` key, dispatches anything. `allow: []` is the schema's sanctioned form for "nothing" and denies every tool by name. "This task binds nothing so it needs no boundary" is true only until someone adds a binding, and it fails silently when they do. - The CLI installs cleanly in a fresh container, including through an agent proxy: `pip install "open-agent-spec[contracts,otel]==1.6.2"`, the pin CI and the README use, puts `oa` on the path and `oa validate` runs locally. Validate before pushing rather than discovering it in the `specs` job. - Tool-calling `oa run`s need the tool-messages provider patch in `patches/` until it lands upstream; stock installs 400 the moment the model calls a tool (re-verified live on 1.6.0, 30 Jul 2026; the 1.6.2 provider still has no `tool_result` translation, read from the published wheel on 8 Oct 2026). `oa validate` is unaffected. A pipx upgrade also drops the contracts extra: re-inject `behavioural-contracts` afterwards, or from CLI 1.6.1 a task declaring `behavioural_contract` stops with `CONTRACTS_UNAVAILABLE` before the model call, and the upgrade reverts all three patches. -- The second patch in `patches/` adds prompt caching, and it is optional: without it every round of a tool loop re-sends the system prompt, the tool definitions and the whole view at full price. Apply it after the tool-messages patch and before the streaming one. Resolve the provider file through the interpreter `oa` runs on, never through system `python3`: a pipx install keeps `oas_cli` in its own virtualenv, so `python3` either cannot import it or finds a different copy and patches a provider the demos never load. The same applies to the probes, which re-exec themselves that way rather than certifying whichever copy happened to import. Compare interpreters by path, not by realpath: a virtualenv's python is a symlink to the system binary. All three were cut against the 1.6.0 wheel and apply unchanged to 1.6.2 with no fuzz or offsets in that order; the 1.5.1-named diff they replace landed on 1.6.0 only by GNU patch's two-line fuzz tolerance. Two breakpoints, system and the first user turn, because the system prompt is about 1,700 tokens against a view of 22,000 and a system-only breakpoint covers about seven per cent of a recall. The floor is the model's own minimum cacheable length converted to characters with a margin, 4,710 for Sonnet and Opus and 9,420 for Haiku, and Sonnet 5.5 inherits the 4,710 although its minimum is 512 tokens, which is conservative in the safe direction, so of the six specs here only the Precedent Analyst's system prompt is cached: the commander's is 1,018 tokens against a 1,024 minimum, and a breakpoint below the minimum is not an error, it is just not a saving. `intelligence.config.cache_prompt: false` turns it off per spec. It changes what a request costs and never what it says, and `oa run` cannot show you either way: the CLI's usage mapper keeps input and output tokens and drops the cache fields, so `checks/cache_probe.py` is how a hit is confirmed. One interaction to remember: a judgement record carries a provider fingerprint over the module's own content, so arms recorded either side of applying this, or of the streaming patch, are not the same configuration twice. +- The second patch in `patches/` adds prompt caching, and it is optional: without it every round of a tool loop re-sends the system prompt, the tool definitions and the whole view at full price. Apply it after the tool-messages patch and before the streaming one. Resolve the provider file through the interpreter `oa` runs on, never through system `python3`: a pipx install keeps `oas_cli` in its own virtualenv, so `python3` either cannot import it or finds a different copy and patches a provider the demos never load. The same applies to the probes, which re-exec themselves that way rather than certifying whichever copy happened to import. Compare interpreters by path, not by realpath: a virtualenv's python is a symlink to the system binary. All three were cut against the 1.6.0 wheel and apply unchanged to 1.6.2 with no fuzz or offsets in that order; the 1.5.1-named diff they replace landed on 1.6.0 only by GNU patch's two-line fuzz tolerance. Two breakpoints, system and the first user turn, because the system prompt is about 1,700 tokens against a view of 22,000 and a system-only breakpoint covers about seven per cent of a recall. The floor is the model's own minimum cacheable length converted to characters with a margin, 4,710 for Sonnet and Opus and 9,420 for Haiku, and Sonnet 5.5 inherits the 4,710 although its minimum is 512 tokens, which is conservative in the safe direction, so of the six specs' system prompts two clear both thresholds, the Precedent Analyst's and the commander's root prompt that `assess` runs, at about 1,700 tokens each against Sonnet 4.6's 1,024 minimum (counted 22 Oct 2026); the other four sit under the character floor. A breakpoint below the model's minimum is not an error, it is just not a saving. `intelligence.config.cache_prompt: false` turns it off per spec. It changes what a request costs and never what it says, and `oa run` cannot show you either way: the CLI's usage mapper keeps input and output tokens and drops the cache fields, so `checks/cache_probe.py` is how a hit is confirmed. One interaction to remember: a judgement record carries a provider fingerprint over the module's own content, so arms recorded either side of applying this, or of the streaming patch, are not the same configuration twice. - Models, since 9 Oct 2026: the Telemetry Analyst, Change Correlator and Comms run `claude-sonnet-5-5` with `intelligence.config.reasoning_effort: medium`; the commander, the Precedent Analyst and the Scribe run `claude-sonnet-4-6`. The effort key is what makes 5.5 work at all: the provider then sends `output_config.effort` with adaptive thinking and drops `temperature`, which 5.5 and Haiku 5.5 reject outright, on both call paths, so no provider change was needed. Thinking spends from the same `max_tokens` as the output, so the single-shot budgets were doubled; read a budget as thinking plus text from now on. The Precedent Analyst stays on 4.6 because recall is a reach job: on the bridge view 5.5 never surfaced the near miss the grounding demo turns on, at medium or high effort, three runs of three. The commander stays on 4.6 for one reason, the mitigate task: on every recorded run, 5.5's safety classifier returned that directive as a refusal before any tool call (`stop_reason: refusal`, category `cyber`, no content), so on those runs the sandbox demo showed the provider's policy and never the runner's fence. The Scribe stays for the same input: `draft_change_record` documents that directive and 5.5 refused it the same way on the one run recorded, while its timeline ran well on 5.5; a spec runs one model, so the refused task decides. A refusal is a complete stream with no content, `stop_reason: refusal` and `stop_details` in `message_delta`, and the streaming patch returns it whole; the shared post helper then fails with `Anthropic refusal:` and the category, on both transports, which is none of the retry wrapper's signatures, so `oa_run` makes one attempt. In the trace it is still `PROVIDER_ERROR` with `execution_failed`: the name is on stderr, not in a span. Two contract gaps 5.5 exposed are pinned in the specs: the Telemetry Analyst's `alerting`, a boolean the model filled with a sentence until its description said so, and the board's `rules_in` and `rules_out`, strings the model filled with one-item lists until theirs said "one string, never a list". The runner's cost table has no 5.5 entry, so `oa.usage.estimated_cost_usd` is absent on those stages and the per-run page shows cost 0 for them. The judged comparison on the Precedent Analyst's 38 cases is the two trial records, `evals/results/trial-*.md` with their variant specs under `evals/specs/`, read against the shipped 4.6 arm in `evals/results/judgement-heading.md`; read the figures there rather than copying them, because a copy goes stale without anything failing. - One agent owns each surface: the Telemetry Analyst owns `telemetry` (8422), the Change Correlator owns `change_registry` (8423), the Scribe owns the `incident_record` sink (8425), the Precedent Analyst owns `precedent_archive` (8426), and `host` (8424) is bound by the commander's mitigate task so the sandbox has something to block. The timeline task binds nothing and the driver files its output, so the Scribe reaches the sink only through `draft_change_record`. That binding is the only one the commander has: `assess` and `brief` bind no tools at all. The commander consumes typed outputs and the view it is handed, never another agent's surface, and never a surface directly. Binding one to `assess` would let it collect past the view, which is the leak `missing_evidence` exists to prevent. Do not give a new spec a surface another agent already owns. - `precedent_search` and `coverage_describe` take a required `before`, and the caller passes the view's `as_of`. Coverage is bounded the same way as search: an unbounded coverage call leaks what search refuses, and a refusal cites coverage as proof it looked. `passage_get` takes an id and no bound, so the bound reaches it only through where the id came from: `demos/investigate.sh` checks every fetched id against what a bounded search returned, which is a control in the driver rather than in the surface, and a caller that gets an id another way is not stopped. Do not write that every call is bounded; two of the three are. Pass the signature type (`host.memory.exhaustion`), never the entity id: identity is not a similarity feature, and the archive holds records about the live host. diff --git a/patches/oas-1.6.0-anthropic-prompt-caching.diff b/patches/oas-1.6.0-anthropic-prompt-caching.diff index 3b74987..52fa316 100644 --- a/patches/oas-1.6.0-anthropic-prompt-caching.diff +++ b/patches/oas-1.6.0-anthropic-prompt-caching.diff @@ -57,8 +57,8 @@ different view per case. THE THRESHOLD IS PER MODEL, and it started out wrong. The minimum cacheable prefix is in tokens, roughly 1,024 for Sonnet and Opus and 2,048 for Haiku, and the first version of this patch used a flat 4,096 characters for all of them. -That admitted the commander's system prompt, which is 4,333 characters and -1,018 tokens: a breakpoint the model would never cache, on the one call in this +That admitted the commander's system prompt, which was then 4,333 characters +and 1,018 tokens: a breakpoint the model would never cache, on the one call in this repository that is single-shot. Not an error, and not a saving either, which is the kind of setting that reads like one in a diff. @@ -71,11 +71,12 @@ substantial tool definitions clears it on the pair and on neither half; the first version measured each block alone and would have declined a breakpoint that had every right to cache. On this repository's own specs nothing moves, because the only tool-bound agent already cleared the floor on its system -prompt alone. The rule was wrong in general, which is enough. Of the six specs here only the Precedent Analyst's system -prompt clears it, which is the agent whose tool loop re-sends it five to ten -times a run and the reason this patch exists. The commander's does not, and its -single-shot `assess` therefore sets no breakpoint at all rather than one that -does nothing. +prompt alone. The rule was wrong in general, which is enough. Of the six specs here two system +prompts clear it: the Precedent Analyst's, whose tool loop re-sends it five to +ten times a run and is the reason this patch exists, and the commander's root +prompt, which has since grown past the floor, so its single-shot `assess` sets +a breakpoint that pays only when `assess` runs again within the cache's +lifetime. ONE INTERACTION WITH `evals/`. A judgement record carries a provider fingerprint, and this changes the provider, so arms recorded before it and