Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -1312,7 +1312,8 @@ median ≤ +0.10 (that arm gives −0.2674 here); the main clause is unchanged.

Mechanism and simulation. A public suite with its own metrics: AgentDojo (agentdojo 0.1.11, benchmark v1.1; 97 user tasks,
629 security cases over workspace, slack, travel and banking; attack `important_instructions`), on `openai/gpt-4o-mini`
(2024-07-18 snapshot) through OpenRouter, provider OpenAI, temperature 0, seed 20261022, the suite's tool loop. Three arms,
(2024-07-18 snapshot), using the API route in `bench/agentdojo/README.md`, provider OpenAI, temperature 0, seed 20261022,
the suite's tool loop. Three arms,
paired on every case: **A** puts probbit's stance block into the system prompt on every model call (one individual per
episode, `persona/bench-assistant.yaml`; the host's `security` reading is a frozen 12-pattern regex over the tool results
since the previous call; the habit `careful_under_security` forces caution `careful` when it reads true, with the line
Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,19 @@ All notable changes to this project are documented here. The format follows
- [From SOUL.md to stance habits](docs/soul-to-habits.md): the bench-assistant worked example, rules-file mappings
and the boundaries between stance constraints, model behavior and host action enforcement.

### Removed

- OpenClaw plugin 0.3.0: remove the Jev adapter, its OpenRouter key loading, named-classifier log fields and special
timeout default. Legacy `classifier: "jev"` loads with a warning and uses keywords only. `classifierShadow`,
`jevModel` and `jevKeyFile` are deprecated no-ops with a load-time warning each, so an old configuration still passes
the host's schema check; no shadow classifier runs and the key file is never read.

### Changed

- Use `example-judge` as the sample judge id throughout examples, tests and the playground. Describe judges by the
documented request/answer JSON, chosen by the user, local or remote, without vendor or model recommendations.
- OpenClaw `auto` classification uses the host's configured decision model for the urgency question as well as the
Boolean flags; keyword mode remains local. The classifier timeout defaults to 300 ms in both modes.
- The README distinguishes enforced stance habits from model obedience, limits replay claims to measured checks,
and links to the changelog for release notes. Install and quickstart paths stay in place.

Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -781,9 +781,9 @@ page in the browser. The page: [docs/probbit-monitor.png](docs/probbit-monitor.p

## After any judge: `probbit evaluate`

Decision models (TypeSafe's Jev, Cloudflare's Clef and Clef-flash, local System One servers) are judges: content in, a
probability per option for each question out, each question on its own, no rules, no joint answer, no refusal. `probbit evaluate`
is the layer after any of them. It reads the judge's System One request (the shape both vendors publish: `model`, `state`,
A judge takes content in and returns a probability per option for each question, each question on its own, without rules,
a joint answer or a refusal. Any judge that answers the documented request and answer JSON works, local or remote, chosen
by you. `probbit evaluate` is the layer after it. It reads the judge's request (the documented shape: `model`, `state`,
`questions` keyed by id with types `noul`, `choice`, `score`) plus a `probbit` block with the judge's answers and your rules over
question ids, compiles it to a probbit-ir program (one variable per question, log-weight = ln max(p, 1e-6), the rules as written)
and answers in the judge's response shape: the most likely answer set that obeys every rule, odds per option under the rules,
Expand Down
4 changes: 2 additions & 2 deletions REPRODUCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,8 @@ cd e3 && experiments/rerun_e3.sh ../venv011/bin/python # from bench/agentdo

## An afternoon and about 3 USD: the benchmark itself

`bench/agentdojo/README.md`, "Rerun the benchmark": agentdojo 0.1.11 pinned, gpt-4o-mini through OpenRouter, the three
arms, 2,178 episodes, about 35 minutes. Your run will not be byte-identical (the model is sampled through an API); the
`bench/agentdojo/README.md`, "Rerun the benchmark": agentdojo 0.1.11 pinned, gpt-4o-mini through the documented API route,
the three arms, 2,178 episodes, about 35 minutes. Your run will not be byte-identical (the model is sampled through an API); the
bars are stated as intervals for that reason. The bars to compare against, their registration before the run, and what
was not tested are in BENCHMARKS.md §14 and `bench/agentdojo/PREREGISTRATION.md`.

Expand Down
2 changes: 1 addition & 1 deletion docs/adapters.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ the receipt per decision and the bounded state the same mechanism drives.
| any Node loop | [examples/node/stance-host.mjs](../examples/node/stance-host.mjs) | `await host.turn(toolTexts)`; `verify()` | `node --test examples/node/stance-host.test.mjs` against the release binary |
| Vercel AI SDK | [examples/hosts/vercel-ai/stance.mjs](../examples/hosts/vercel-ai/stance.mjs) | `prepareStep` rebuilds the step's system message from `host.turn`; `onToolExecutionEnd` collects results | not run here (needs the `ai` package, a provider and a key) |
| MCP hosts (Claude Code, Codex CLI, Cursor, mcporter) | `probbit mcp` | tools `probbit_live_event`, `probbit_live_verify`, `probbit_persona_turn`, ... ([agents.md](agents.md)) | real calls from Claude Code and Codex CLI (agents.md) |
| OpenClaw | [plugins/openclaw](../plugins/openclaw/README.md) (`probbit-stance`, `openclaw plugins install npm:probbit-stance --accept-capabilities`) | `before_prompt_build`: one event per user message in the listed sessions, the block injected into the turn, the footer line in the chat; `before_tool_call`: shadow gate; `after_tool_call` (with `sensesMode` shadow or feed, unreleased): counts failed tool calls for the `tool_fail` reading | 160 tests against a mocked host built from the 2026.9.7 SDK types and the real binary; the same code runs live on the author's own agent since 2026-10-06 |
| OpenClaw | [plugins/openclaw](../plugins/openclaw/README.md) (`probbit-stance`, `openclaw plugins install npm:probbit-stance --accept-capabilities`) | `before_prompt_build`: one event per user message in the listed sessions, the block injected into the turn, the footer line in the chat; `before_tool_call`: shadow gate; `after_tool_call` (with `sensesMode` shadow or feed, unreleased): counts failed tool calls for the `tool_fail` reading | 152 tests against a mocked host built from the 2026.9.7 SDK types and the real binary; the same code runs live on the author's own agent since 2026-10-06 |

"Not run here" means exactly that: the file is written to the framework's documented contract and compiles, but this
repository's tests do not exercise it. Please open an issue (or a reproduction report) with what you saw.
Expand Down
10 changes: 5 additions & 5 deletions docs/agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,8 @@ probbit evaluate --summary --pretty < examples/evaluate/support-12.json # exac
probbit evaluate --program < examples/evaluate/support-12.json | probbit run # the compiled probbit-ir program: the same answer
```

Python (stdlib only; the judge is a callable or a System One URL, called with `urllib`; the key comes from the environment
Any judge that answers the documented request and answer JSON works, local or remote, chosen by you.
Python (stdlib only; the judge is a callable or a URL, called with `urllib`; the key comes from the environment
variable you name):

```python
Expand All @@ -170,10 +171,9 @@ answer = probbit.evaluate(request, judge=lambda ask: {"urgent": 0.41, "team": {"
moved = [q for q, a in answer["answers"].items() if a["probbit"]["changed"]]
```

A URL without a path gets TypeSafe's `/v1/systemone`; a URL with a path is used as is (Workers AI:
`https://api.cloudflare.com/client/v4/accounts/<account id>/ai/run/@cf/cloudflare/clef`, whose REST envelope is unwrapped).
That path is documented from the vendors' published schemas and tested against a local mock (`python/mock_judge.py`); it was not
exercised against a vendor. Against the mock: the judge received the request untouched (no `probbit` block), the round trip took
A URL without a path gets `/v1/systemone`; a URL with a path is used as is. When there are no top-level `answers`, a
`result` object is unwrapped unless `success` is false (a judge error). These behaviors are tested against a local mock
(`python/mock_judge.py`); they were not exercised against a remote service. Against the mock: the judge received the request untouched (no `probbit` block), the round trip took
7.0 ms (Python 3.9.6). Judge failures raise `probbit.ProbbitJudgeError`.

MCP: tool `probbit_evaluate` takes the same request with optional `flags` (`{"summary": true}`, `{"program": true}`); over pipes
Expand Down
4 changes: 2 additions & 2 deletions examples/evaluate/support-12.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"model": "jev-latest",
"model": "example-judge",
"state": {"subject": "Checkout down, charged twice", "message": "Your checkout has failed for an hour and I was charged twice. Refund me now or I cancel. Card ending 4242."},
"questions": {
"urgent": {"type": "noul", "instructions": "Is this request urgent?"},
Expand All @@ -17,7 +17,7 @@
},
"probbit": {
"judge": {
"model": "jev-latest",
"model": "example-judge",
"usage": {"input_tokens": 412, "output_tokens": 24},
"answers": {
"urgent": {"type": "noul", "noul": 0.41},
Expand Down
20 changes: 10 additions & 10 deletions llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -793,9 +793,9 @@ page in the browser. The page: [docs/probbit-monitor.png](docs/probbit-monitor.p

## After any judge: `probbit evaluate`

Decision models (TypeSafe's Jev, Cloudflare's Clef and Clef-flash, local System One servers) are judges: content in, a
probability per option for each question out, each question on its own, no rules, no joint answer, no refusal. `probbit evaluate`
is the layer after any of them. It reads the judge's System One request (the shape both vendors publish: `model`, `state`,
A judge takes content in and returns a probability per option for each question, each question on its own, without rules,
a joint answer or a refusal. Any judge that answers the documented request and answer JSON works, local or remote, chosen
by you. `probbit evaluate` is the layer after it. It reads the judge's request (the documented shape: `model`, `state`,
`questions` keyed by id with types `noul`, `choice`, `score`) plus a `probbit` block with the judge's answers and your rules over
question ids, compiles it to a probbit-ir program (one variable per question, log-weight = ln max(p, 1e-6), the rules as written)
and answers in the judge's response shape: the most likely answer set that obeys every rule, odds per option under the rules,
Expand Down Expand Up @@ -3331,7 +3331,8 @@ probbit evaluate --summary --pretty < examples/evaluate/support-12.json # exac
probbit evaluate --program < examples/evaluate/support-12.json | probbit run # the compiled probbit-ir program: the same answer
```

Python (stdlib only; the judge is a callable or a System One URL, called with `urllib`; the key comes from the environment
Any judge that answers the documented request and answer JSON works, local or remote, chosen by you.
Python (stdlib only; the judge is a callable or a URL, called with `urllib`; the key comes from the environment
variable you name):

```python
Expand All @@ -3343,10 +3344,9 @@ answer = probbit.evaluate(request, judge=lambda ask: {"urgent": 0.41, "team": {"
moved = [q for q, a in answer["answers"].items() if a["probbit"]["changed"]]
```

A URL without a path gets TypeSafe's `/v1/systemone`; a URL with a path is used as is (Workers AI:
`https://api.cloudflare.com/client/v4/accounts/<account id>/ai/run/@cf/cloudflare/clef`, whose REST envelope is unwrapped).
That path is documented from the vendors' published schemas and tested against a local mock (`python/mock_judge.py`); it was not
exercised against a vendor. Against the mock: the judge received the request untouched (no `probbit` block), the round trip took
A URL without a path gets `/v1/systemone`; a URL with a path is used as is. When there are no top-level `answers`, a
`result` object is unwrapped unless `success` is false (a judge error). These behaviors are tested against a local mock
(`python/mock_judge.py`); they were not exercised against a remote service. Against the mock: the judge received the request untouched (no `probbit` block), the round trip took
7.0 ms (Python 3.9.6). Judge failures raise `probbit.ProbbitJudgeError`.

MCP: tool `probbit_evaluate` takes the same request with optional `flags` (`{"summary": true}`, `{"program": true}`); over pipes
Expand Down Expand Up @@ -3523,8 +3523,8 @@ cd e3 && experiments/rerun_e3.sh ../venv011/bin/python # from bench/agentdo

## An afternoon and about 3 USD: the benchmark itself

`bench/agentdojo/README.md`, "Rerun the benchmark": agentdojo 0.1.11 pinned, gpt-4o-mini through OpenRouter, the three
arms, 2,178 episodes, about 35 minutes. Your run will not be byte-identical (the model is sampled through an API); the
`bench/agentdojo/README.md`, "Rerun the benchmark": agentdojo 0.1.11 pinned, gpt-4o-mini through the documented API route,
the three arms, 2,178 episodes, about 35 minutes. Your run will not be byte-identical (the model is sampled through an API); the
bars are stated as intervals for that reason. The bars to compare against, their registration before the run, and what
was not tested are in BENCHMARKS.md §14 and `bench/agentdojo/PREREGISTRATION.md`.

Expand Down
Loading
Loading