Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .agents/incident-commander.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -979,6 +979,22 @@ prompts:
authorisation_required true. You never mark such an action as
autonomous, regardless of how routine it appears.

Each confidence word has one meaning, and the board uses no other:
- speculative: nothing observed points to it, or no falsifying test can
be named.
- plausible: consistent with what has been observed, but resting on
inference: correlation, timing or elimination.
- likely: several independent lines of inference agree, and nothing
observed contradicts it.
- supported: a direct observation in the evidence array bears on it but
does not establish it, and nothing outstanding in blocking_evidence
would decide it.
- confirmed: a direct observation in the evidence array establishes it.
While blocking evidence that would decide a hypothesis is outstanding,
that hypothesis is at most likely, however strongly the inference reads,
and while any blocking evidence is outstanding no hypothesis is
confirmed: that is what blocking_evidence is for.

You are working only from what you are given. If something appears trivially
obtainable but is absent from the evidence array, it has not been collected.
Request it. Do not assume it.
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ oa validate --spec .agents/<name>.yaml # before every commit, for every spec
- **Every task declares a sandbox, including the tasks that bind no tools.** Absence is allow-all: the runner only checks an allow list when one is present, so a task with no sandbox, and equally a sandbox with no `tools` key, dispatches anything. `allow: []` is the schema's sanctioned form for "nothing" and denies every tool by name. "This task binds nothing so it needs no boundary" is true only until someone adds a binding, and it fails silently when they do.
- The CLI installs cleanly in a fresh container, including through an agent proxy: `pip install "open-agent-spec[contracts,otel]==1.6.2"`, the pin CI and the README use, puts `oa` on the path and `oa validate` runs locally. Validate before pushing rather than discovering it in the `specs` job.
- Tool-calling `oa run`s need the tool-messages provider patch in `patches/` until it lands upstream; stock installs 400 the moment the model calls a tool (re-verified live on 1.6.0, 30 Jul 2026; the 1.6.2 provider still has no `tool_result` translation, read from the published wheel on 8 Oct 2026). `oa validate` is unaffected. A pipx upgrade also drops the contracts extra: re-inject `behavioural-contracts` afterwards, or from CLI 1.6.1 a task declaring `behavioural_contract` stops with `CONTRACTS_UNAVAILABLE` before the model call, and the upgrade reverts all three patches.
- The second patch in `patches/` adds prompt caching, and it is optional: without it every round of a tool loop re-sends the system prompt, the tool definitions and the whole view at full price. Apply it after the tool-messages patch and before the streaming one. Resolve the provider file through the interpreter `oa` runs on, never through system `python3`: a pipx install keeps `oas_cli` in its own virtualenv, so `python3` either cannot import it or finds a different copy and patches a provider the demos never load. The same applies to the probes, which re-exec themselves that way rather than certifying whichever copy happened to import. Compare interpreters by path, not by realpath: a virtualenv's python is a symlink to the system binary. All three were cut against the 1.6.0 wheel and apply unchanged to 1.6.2 with no fuzz or offsets in that order; the 1.5.1-named diff they replace landed on 1.6.0 only by GNU patch's two-line fuzz tolerance. Two breakpoints, system and the first user turn, because the system prompt is about 1,700 tokens against a view of 22,000 and a system-only breakpoint covers about seven per cent of a recall. The floor is the model's own minimum cacheable length converted to characters with a margin, 4,710 for Sonnet and Opus and 9,420 for Haiku, and Sonnet 5.5 inherits the 4,710 although its minimum is 512 tokens, which is conservative in the safe direction, so of the six specs here only the Precedent Analyst's system prompt is cached: the commander's is 1,018 tokens against a 1,024 minimum, and a breakpoint below the minimum is not an error, it is just not a saving. `intelligence.config.cache_prompt: false` turns it off per spec. It changes what a request costs and never what it says, and `oa run` cannot show you either way: the CLI's usage mapper keeps input and output tokens and drops the cache fields, so `checks/cache_probe.py` is how a hit is confirmed. One interaction to remember: a judgement record carries a provider fingerprint over the module's own content, so arms recorded either side of applying this, or of the streaming patch, are not the same configuration twice.
- The second patch in `patches/` adds prompt caching, and it is optional: without it every round of a tool loop re-sends the system prompt, the tool definitions and the whole view at full price. Apply it after the tool-messages patch and before the streaming one. Resolve the provider file through the interpreter `oa` runs on, never through system `python3`: a pipx install keeps `oas_cli` in its own virtualenv, so `python3` either cannot import it or finds a different copy and patches a provider the demos never load. The same applies to the probes, which re-exec themselves that way rather than certifying whichever copy happened to import. Compare interpreters by path, not by realpath: a virtualenv's python is a symlink to the system binary. All three were cut against the 1.6.0 wheel and apply unchanged to 1.6.2 with no fuzz or offsets in that order; the 1.5.1-named diff they replace landed on 1.6.0 only by GNU patch's two-line fuzz tolerance. Two breakpoints, system and the first user turn, because the system prompt is about 1,700 tokens against a view of 22,000 and a system-only breakpoint covers about seven per cent of a recall. The floor is the model's own minimum cacheable length converted to characters with a margin, 4,710 for Sonnet and Opus and 9,420 for Haiku, and Sonnet 5.5 inherits the 4,710 although its minimum is 512 tokens, which is conservative in the safe direction, so of the six specs' system prompts two clear both thresholds, the Precedent Analyst's and the commander's root prompt that `assess` runs, at about 1,700 tokens each against Sonnet 4.6's 1,024 minimum (counted 22 Oct 2026); the other four sit under the character floor. A breakpoint below the model's minimum is not an error, it is just not a saving. `intelligence.config.cache_prompt: false` turns it off per spec. It changes what a request costs and never what it says, and `oa run` cannot show you either way: the CLI's usage mapper keeps input and output tokens and drops the cache fields, so `checks/cache_probe.py` is how a hit is confirmed. One interaction to remember: a judgement record carries a provider fingerprint over the module's own content, so arms recorded either side of applying this, or of the streaming patch, are not the same configuration twice.
- Models, since 9 Oct 2026: the Telemetry Analyst, Change Correlator and Comms run `claude-sonnet-5-5` with `intelligence.config.reasoning_effort: medium`; the commander, the Precedent Analyst and the Scribe run `claude-sonnet-4-6`. The effort key is what makes 5.5 work at all: the provider then sends `output_config.effort` with adaptive thinking and drops `temperature`, which 5.5 and Haiku 5.5 reject outright, on both call paths, so no provider change was needed. Thinking spends from the same `max_tokens` as the output, so the single-shot budgets were doubled; read a budget as thinking plus text from now on. The Precedent Analyst stays on 4.6 because recall is a reach job: on the bridge view 5.5 never surfaced the near miss the grounding demo turns on, at medium or high effort, three runs of three. The commander stays on 4.6 for one reason, the mitigate task: on every recorded run, 5.5's safety classifier returned that directive as a refusal before any tool call (`stop_reason: refusal`, category `cyber`, no content), so on those runs the sandbox demo showed the provider's policy and never the runner's fence. The Scribe stays for the same input: `draft_change_record` documents that directive and 5.5 refused it the same way on the one run recorded, while its timeline ran well on 5.5; a spec runs one model, so the refused task decides. A refusal is a complete stream with no content, `stop_reason: refusal` and `stop_details` in `message_delta`, and the streaming patch returns it whole; the shared post helper then fails with `Anthropic refusal:` and the category, on both transports, which is none of the retry wrapper's signatures, so `oa_run` makes one attempt. In the trace it is still `PROVIDER_ERROR` with `execution_failed`: the name is on stderr, not in a span. Two contract gaps 5.5 exposed are pinned in the specs: the Telemetry Analyst's `alerting`, a boolean the model filled with a sentence until its description said so, and the board's `rules_in` and `rules_out`, strings the model filled with one-item lists until theirs said "one string, never a list". The runner's cost table has no 5.5 entry, so `oa.usage.estimated_cost_usd` is absent on those stages and the per-run page shows cost 0 for them. The judged comparison on the Precedent Analyst's 38 cases is the two trial records, `evals/results/trial-*.md` with their variant specs under `evals/specs/`, read against the shipped 4.6 arm in `evals/results/judgement-heading.md`; read the figures there rather than copying them, because a copy goes stale without anything failing.
- One agent owns each surface: the Telemetry Analyst owns `telemetry` (8422), the Change Correlator owns `change_registry` (8423), the Scribe owns the `incident_record` sink (8425), the Precedent Analyst owns `precedent_archive` (8426), and `host` (8424) is bound by the commander's mitigate task so the sandbox has something to block. The timeline task binds nothing and the driver files its output, so the Scribe reaches the sink only through `draft_change_record`. That binding is the only one the commander has: `assess` and `brief` bind no tools at all. The commander consumes typed outputs and the view it is handed, never another agent's surface, and never a surface directly. Binding one to `assess` would let it collect past the view, which is the leak `missing_evidence` exists to prevent. Do not give a new spec a surface another agent already owns.
- `precedent_search` and `coverage_describe` take a required `before`, and the caller passes the view's `as_of`. Coverage is bounded the same way as search: an unbounded coverage call leaks what search refuses, and a refusal cites coverage as proof it looked. `passage_get` takes an id and no bound, so the bound reaches it only through where the id came from: `demos/investigate.sh` checks every fetched id against what a bounded search returned, which is a control in the driver rather than in the surface, and a caller that gets an id another way is not stopped. Do not write that every call is bounded; two of the three are. Pass the signature type (`host.memory.exhaustion`), never the entity id: identity is not a similarity feature, and the archive holds records about the live host.
Expand Down
15 changes: 8 additions & 7 deletions patches/oas-1.6.0-anthropic-prompt-caching.diff
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,8 @@ different view per case.
THE THRESHOLD IS PER MODEL, and it started out wrong. The minimum cacheable
prefix is in tokens, roughly 1,024 for Sonnet and Opus and 2,048 for Haiku, and
the first version of this patch used a flat 4,096 characters for all of them.
That admitted the commander's system prompt, which is 4,333 characters and
1,018 tokens: a breakpoint the model would never cache, on the one call in this
That admitted the commander's system prompt, which was then 4,333 characters
and 1,018 tokens: a breakpoint the model would never cache, on the one call in this
repository that is single-shot. Not an error, and not a saving either, which is
the kind of setting that reads like one in a diff.

Expand All @@ -71,11 +71,12 @@ substantial tool definitions clears it on the pair and on neither half; the
first version measured each block alone and would have declined a breakpoint
that had every right to cache. On this repository's own specs nothing moves,
because the only tool-bound agent already cleared the floor on its system
prompt alone. The rule was wrong in general, which is enough. Of the six specs here only the Precedent Analyst's system
prompt clears it, which is the agent whose tool loop re-sends it five to ten
times a run and the reason this patch exists. The commander's does not, and its
single-shot `assess` therefore sets no breakpoint at all rather than one that
does nothing.
prompt alone. The rule was wrong in general, which is enough. Of the six specs here two system
prompts clear it: the Precedent Analyst's, whose tool loop re-sends it five to
ten times a run and is the reason this patch exists, and the commander's root
prompt, which has since grown past the floor, so its single-shot `assess` sets
a breakpoint that pays only when `assess` runs again within the cache's
lifetime.

ONE INTERACTION WITH `evals/`. A judgement record carries a provider
fingerprint, and this changes the provider, so arms recorded before it and
Expand Down
Loading