diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 94ec3a1..d5a898d 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -1312,7 +1312,8 @@ median ≤ +0.10 (that arm gives −0.2674 here); the main clause is unchanged. Mechanism and simulation. A public suite with its own metrics: AgentDojo (agentdojo 0.1.11, benchmark v1.1; 97 user tasks, 629 security cases over workspace, slack, travel and banking; attack `important_instructions`), on `openai/gpt-4o-mini` -(2024-07-18 snapshot) through OpenRouter, provider OpenAI, temperature 0, seed 20261022, the suite's tool loop. Three arms, +(2024-07-18 snapshot), using the API route in `bench/agentdojo/README.md`, provider OpenAI, temperature 0, seed 20261022, +the suite's tool loop. Three arms, paired on every case: **A** puts probbit's stance block into the system prompt on every model call (one individual per episode, `persona/bench-assistant.yaml`; the host's `security` reading is a frozen 12-pattern regex over the tool results since the previous call; the habit `careful_under_security` forces caution `careful` when it reads true, with the line diff --git a/CHANGELOG.md b/CHANGELOG.md index 3933f56..1fe2898 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,8 +14,19 @@ All notable changes to this project are documented here. The format follows - [From SOUL.md to stance habits](docs/soul-to-habits.md): the bench-assistant worked example, rules-file mappings and the boundaries between stance constraints, model behavior and host action enforcement. +### Removed + +- OpenClaw plugin 0.3.0: remove the Jev adapter, its OpenRouter key loading, named-classifier log fields and special + timeout default. Legacy `classifier: "jev"` loads with a warning and uses keywords only. `classifierShadow`, + `jevModel` and `jevKeyFile` are deprecated no-ops with a load-time warning each, so an old configuration still passes + the host's schema check; no shadow classifier runs and the key file is never read. + ### Changed +- Use `example-judge` as the sample judge id throughout examples, tests and the playground. Describe judges by the + documented request/answer JSON, chosen by the user, local or remote, without vendor or model recommendations. +- OpenClaw `auto` classification uses the host's configured decision model for the urgency question as well as the + Boolean flags; keyword mode remains local. The classifier timeout defaults to 300 ms in both modes. - The README distinguishes enforced stance habits from model obedience, limits replay claims to measured checks, and links to the changelog for release notes. Install and quickstart paths stay in place. diff --git a/README.md b/README.md index b0571e2..73cb75a 100644 --- a/README.md +++ b/README.md @@ -781,9 +781,9 @@ page in the browser. The page: [docs/probbit-monitor.png](docs/probbit-monitor.p ## After any judge: `probbit evaluate` -Decision models (TypeSafe's Jev, Cloudflare's Clef and Clef-flash, local System One servers) are judges: content in, a -probability per option for each question out, each question on its own, no rules, no joint answer, no refusal. `probbit evaluate` -is the layer after any of them. It reads the judge's System One request (the shape both vendors publish: `model`, `state`, +A judge takes content in and returns a probability per option for each question, each question on its own, without rules, +a joint answer or a refusal. Any judge that answers the documented request and answer JSON works, local or remote, chosen +by you. `probbit evaluate` is the layer after it. It reads the judge's request (the documented shape: `model`, `state`, `questions` keyed by id with types `noul`, `choice`, `score`) plus a `probbit` block with the judge's answers and your rules over question ids, compiles it to a probbit-ir program (one variable per question, log-weight = ln max(p, 1e-6), the rules as written) and answers in the judge's response shape: the most likely answer set that obeys every rule, odds per option under the rules, diff --git a/REPRODUCE.md b/REPRODUCE.md index c232b97..e5e737e 100644 --- a/REPRODUCE.md +++ b/REPRODUCE.md @@ -80,8 +80,8 @@ cd e3 && experiments/rerun_e3.sh ../venv011/bin/python # from bench/agentdo ## An afternoon and about 3 USD: the benchmark itself -`bench/agentdojo/README.md`, "Rerun the benchmark": agentdojo 0.1.11 pinned, gpt-4o-mini through OpenRouter, the three -arms, 2,178 episodes, about 35 minutes. Your run will not be byte-identical (the model is sampled through an API); the +`bench/agentdojo/README.md`, "Rerun the benchmark": agentdojo 0.1.11 pinned, gpt-4o-mini through the documented API route, +the three arms, 2,178 episodes, about 35 minutes. Your run will not be byte-identical (the model is sampled through an API); the bars are stated as intervals for that reason. The bars to compare against, their registration before the run, and what was not tested are in BENCHMARKS.md §14 and `bench/agentdojo/PREREGISTRATION.md`. diff --git a/docs/adapters.md b/docs/adapters.md index cb8f84b..6026202 100644 --- a/docs/adapters.md +++ b/docs/adapters.md @@ -30,7 +30,7 @@ the receipt per decision and the bounded state the same mechanism drives. | any Node loop | [examples/node/stance-host.mjs](../examples/node/stance-host.mjs) | `await host.turn(toolTexts)`; `verify()` | `node --test examples/node/stance-host.test.mjs` against the release binary | | Vercel AI SDK | [examples/hosts/vercel-ai/stance.mjs](../examples/hosts/vercel-ai/stance.mjs) | `prepareStep` rebuilds the step's system message from `host.turn`; `onToolExecutionEnd` collects results | not run here (needs the `ai` package, a provider and a key) | | MCP hosts (Claude Code, Codex CLI, Cursor, mcporter) | `probbit mcp` | tools `probbit_live_event`, `probbit_live_verify`, `probbit_persona_turn`, ... ([agents.md](agents.md)) | real calls from Claude Code and Codex CLI (agents.md) | -| OpenClaw | [plugins/openclaw](../plugins/openclaw/README.md) (`probbit-stance`, `openclaw plugins install npm:probbit-stance --accept-capabilities`) | `before_prompt_build`: one event per user message in the listed sessions, the block injected into the turn, the footer line in the chat; `before_tool_call`: shadow gate; `after_tool_call` (with `sensesMode` shadow or feed, unreleased): counts failed tool calls for the `tool_fail` reading | 160 tests against a mocked host built from the 2026.9.7 SDK types and the real binary; the same code runs live on the author's own agent since 2026-10-06 | +| OpenClaw | [plugins/openclaw](../plugins/openclaw/README.md) (`probbit-stance`, `openclaw plugins install npm:probbit-stance --accept-capabilities`) | `before_prompt_build`: one event per user message in the listed sessions, the block injected into the turn, the footer line in the chat; `before_tool_call`: shadow gate; `after_tool_call` (with `sensesMode` shadow or feed, unreleased): counts failed tool calls for the `tool_fail` reading | 152 tests against a mocked host built from the 2026.9.7 SDK types and the real binary; the same code runs live on the author's own agent since 2026-10-06 | "Not run here" means exactly that: the file is written to the framework's documented contract and compiles, but this repository's tests do not exercise it. Please open an issue (or a reproduction report) with what you saw. diff --git a/docs/agents.md b/docs/agents.md index 16bfc6a..f048a4f 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -158,7 +158,8 @@ probbit evaluate --summary --pretty < examples/evaluate/support-12.json # exac probbit evaluate --program < examples/evaluate/support-12.json | probbit run # the compiled probbit-ir program: the same answer ``` -Python (stdlib only; the judge is a callable or a System One URL, called with `urllib`; the key comes from the environment +Any judge that answers the documented request and answer JSON works, local or remote, chosen by you. +Python (stdlib only; the judge is a callable or a URL, called with `urllib`; the key comes from the environment variable you name): ```python @@ -170,10 +171,9 @@ answer = probbit.evaluate(request, judge=lambda ask: {"urgent": 0.41, "team": {" moved = [q for q, a in answer["answers"].items() if a["probbit"]["changed"]] ``` -A URL without a path gets TypeSafe's `/v1/systemone`; a URL with a path is used as is (Workers AI: -`https://api.cloudflare.com/client/v4/accounts//ai/run/@cf/cloudflare/clef`, whose REST envelope is unwrapped). -That path is documented from the vendors' published schemas and tested against a local mock (`python/mock_judge.py`); it was not -exercised against a vendor. Against the mock: the judge received the request untouched (no `probbit` block), the round trip took +A URL without a path gets `/v1/systemone`; a URL with a path is used as is. When there are no top-level `answers`, a +`result` object is unwrapped unless `success` is false (a judge error). These behaviors are tested against a local mock +(`python/mock_judge.py`); they were not exercised against a remote service. Against the mock: the judge received the request untouched (no `probbit` block), the round trip took 7.0 ms (Python 3.9.6). Judge failures raise `probbit.ProbbitJudgeError`. MCP: tool `probbit_evaluate` takes the same request with optional `flags` (`{"summary": true}`, `{"program": true}`); over pipes diff --git a/examples/evaluate/support-12.json b/examples/evaluate/support-12.json index 35fa9e0..7479116 100644 --- a/examples/evaluate/support-12.json +++ b/examples/evaluate/support-12.json @@ -1,5 +1,5 @@ { - "model": "jev-latest", + "model": "example-judge", "state": {"subject": "Checkout down, charged twice", "message": "Your checkout has failed for an hour and I was charged twice. Refund me now or I cancel. Card ending 4242."}, "questions": { "urgent": {"type": "noul", "instructions": "Is this request urgent?"}, @@ -17,7 +17,7 @@ }, "probbit": { "judge": { - "model": "jev-latest", + "model": "example-judge", "usage": {"input_tokens": 412, "output_tokens": 24}, "answers": { "urgent": {"type": "noul", "noul": 0.41}, diff --git a/llms-full.txt b/llms-full.txt index 655a3c7..e6d7083 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -793,9 +793,9 @@ page in the browser. The page: [docs/probbit-monitor.png](docs/probbit-monitor.p ## After any judge: `probbit evaluate` -Decision models (TypeSafe's Jev, Cloudflare's Clef and Clef-flash, local System One servers) are judges: content in, a -probability per option for each question out, each question on its own, no rules, no joint answer, no refusal. `probbit evaluate` -is the layer after any of them. It reads the judge's System One request (the shape both vendors publish: `model`, `state`, +A judge takes content in and returns a probability per option for each question, each question on its own, without rules, +a joint answer or a refusal. Any judge that answers the documented request and answer JSON works, local or remote, chosen +by you. `probbit evaluate` is the layer after it. It reads the judge's request (the documented shape: `model`, `state`, `questions` keyed by id with types `noul`, `choice`, `score`) plus a `probbit` block with the judge's answers and your rules over question ids, compiles it to a probbit-ir program (one variable per question, log-weight = ln max(p, 1e-6), the rules as written) and answers in the judge's response shape: the most likely answer set that obeys every rule, odds per option under the rules, @@ -3331,7 +3331,8 @@ probbit evaluate --summary --pretty < examples/evaluate/support-12.json # exac probbit evaluate --program < examples/evaluate/support-12.json | probbit run # the compiled probbit-ir program: the same answer ``` -Python (stdlib only; the judge is a callable or a System One URL, called with `urllib`; the key comes from the environment +Any judge that answers the documented request and answer JSON works, local or remote, chosen by you. +Python (stdlib only; the judge is a callable or a URL, called with `urllib`; the key comes from the environment variable you name): ```python @@ -3343,10 +3344,9 @@ answer = probbit.evaluate(request, judge=lambda ask: {"urgent": 0.41, "team": {" moved = [q for q, a in answer["answers"].items() if a["probbit"]["changed"]] ``` -A URL without a path gets TypeSafe's `/v1/systemone`; a URL with a path is used as is (Workers AI: -`https://api.cloudflare.com/client/v4/accounts//ai/run/@cf/cloudflare/clef`, whose REST envelope is unwrapped). -That path is documented from the vendors' published schemas and tested against a local mock (`python/mock_judge.py`); it was not -exercised against a vendor. Against the mock: the judge received the request untouched (no `probbit` block), the round trip took +A URL without a path gets `/v1/systemone`; a URL with a path is used as is. When there are no top-level `answers`, a +`result` object is unwrapped unless `success` is false (a judge error). These behaviors are tested against a local mock +(`python/mock_judge.py`); they were not exercised against a remote service. Against the mock: the judge received the request untouched (no `probbit` block), the round trip took 7.0 ms (Python 3.9.6). Judge failures raise `probbit.ProbbitJudgeError`. MCP: tool `probbit_evaluate` takes the same request with optional `flags` (`{"summary": true}`, `{"program": true}`); over pipes @@ -3523,8 +3523,8 @@ cd e3 && experiments/rerun_e3.sh ../venv011/bin/python # from bench/agentdo ## An afternoon and about 3 USD: the benchmark itself -`bench/agentdojo/README.md`, "Rerun the benchmark": agentdojo 0.1.11 pinned, gpt-4o-mini through OpenRouter, the three -arms, 2,178 episodes, about 35 minutes. Your run will not be byte-identical (the model is sampled through an API); the +`bench/agentdojo/README.md`, "Rerun the benchmark": agentdojo 0.1.11 pinned, gpt-4o-mini through the documented API route, +the three arms, 2,178 episodes, about 35 minutes. Your run will not be byte-identical (the model is sampled through an API); the bars are stated as intervals for that reason. The bars to compare against, their registration before the run, and what was not tested are in BENCHMARKS.md §14 and `bench/agentdojo/PREREGISTRATION.md`. diff --git a/playground/index.html b/playground/index.html index 5d5ff3d..feb2812 100644 --- a/playground/index.html +++ b/playground/index.html @@ -62,7 +62,7 @@

Meet three individuals from one persona