From a9c1c8df8b9cff1f62e325b6dfc655659081c7db Mon Sep 17 00:00:00 2001 From: BitmapAsset Date: Thu, 5 Nov 2026 13:49:38 -0800 Subject: [PATCH 1/3] refactor(openclaw): remove bundled judge adapter --- docs/adapters.md | 2 +- plugins/openclaw/README.md | 41 +-- plugins/openclaw/openclaw.plugin.json | 35 ++- plugins/openclaw/package-lock.json | 4 +- plugins/openclaw/package.json | 2 +- plugins/openclaw/src/classify.ts | 95 +------ plugins/openclaw/src/config.ts | 46 ++- plugins/openclaw/src/index.ts | 43 +-- plugins/openclaw/src/jev.ts | 145 ---------- plugins/openclaw/src/probbit.ts | 2 - plugins/openclaw/src/version.ts | 2 +- plugins/openclaw/test/classifier.test.ts | 174 ++++++++++++ plugins/openclaw/test/classify.test.ts | 2 +- plugins/openclaw/test/jev.test.ts | 347 ----------------------- 14 files changed, 258 insertions(+), 682 deletions(-) delete mode 100644 plugins/openclaw/src/jev.ts create mode 100644 plugins/openclaw/test/classifier.test.ts delete mode 100644 plugins/openclaw/test/jev.test.ts diff --git a/docs/adapters.md b/docs/adapters.md index cb8f84b..e2dc0a0 100644 --- a/docs/adapters.md +++ b/docs/adapters.md @@ -30,7 +30,7 @@ the receipt per decision and the bounded state the same mechanism drives. | any Node loop | [examples/node/stance-host.mjs](../examples/node/stance-host.mjs) | `await host.turn(toolTexts)`; `verify()` | `node --test examples/node/stance-host.test.mjs` against the release binary | | Vercel AI SDK | [examples/hosts/vercel-ai/stance.mjs](../examples/hosts/vercel-ai/stance.mjs) | `prepareStep` rebuilds the step's system message from `host.turn`; `onToolExecutionEnd` collects results | not run here (needs the `ai` package, a provider and a key) | | MCP hosts (Claude Code, Codex CLI, Cursor, mcporter) | `probbit mcp` | tools `probbit_live_event`, `probbit_live_verify`, `probbit_persona_turn`, ... ([agents.md](agents.md)) | real calls from Claude Code and Codex CLI (agents.md) | -| OpenClaw | [plugins/openclaw](../plugins/openclaw/README.md) (`probbit-stance`, `openclaw plugins install npm:probbit-stance --accept-capabilities`) | `before_prompt_build`: one event per user message in the listed sessions, the block injected into the turn, the footer line in the chat; `before_tool_call`: shadow gate; `after_tool_call` (with `sensesMode` shadow or feed, unreleased): counts failed tool calls for the `tool_fail` reading | 160 tests against a mocked host built from the 2026.9.7 SDK types and the real binary; the same code runs live on the author's own agent since 2026-10-06 | +| OpenClaw | [plugins/openclaw](../plugins/openclaw/README.md) (`probbit-stance`, `openclaw plugins install npm:probbit-stance --accept-capabilities`) | `before_prompt_build`: one event per user message in the listed sessions, the block injected into the turn, the footer line in the chat; `before_tool_call`: shadow gate; `after_tool_call` (with `sensesMode` shadow or feed, unreleased): counts failed tool calls for the `tool_fail` reading | 151 tests against a mocked host built from the 2026.9.7 SDK types and the real binary; the same code runs live on the author's own agent since 2026-10-06 | "Not run here" means exactly that: the file is written to the framework's documented contract and compiles, but this repository's tests do not exercise it. Please open an issue (or a reproduction report) with what you saw. diff --git a/plugins/openclaw/README.md b/plugins/openclaw/README.md index 15b9766..935757d 100644 --- a/plugins/openclaw/README.md +++ b/plugins/openclaw/README.md @@ -22,7 +22,7 @@ Mood glyph: 🔒 (valence even 0.64, pressure keyed 0.58). Use at most one mood Write from this stance. End the visible reply with one line: "Assistant 🔒 stance #14 · valence even · pressure keyed · security means careful, no play · in: +2 · moved: pressure, +1". ``` -That block is what `probbit-stance` 0.2.0 writes (the `Mood glyph:` line: see "The glyph" below). `probbit-stance` +That block is what `probbit-stance` 0.3.0 writes (the `Mood glyph:` line: see "The glyph" below). `probbit-stance` 0.1.0 wrote it without the `Mood glyph:` line and ended with `"Assistant stance #14 · valence even · pressure keyed · security means careful, no play · in: correction, security · moved: pressure, caution"`. @@ -31,7 +31,7 @@ which event this was, the two moods, a few words of the stance, the inputs the p and traits whose level changed since the previous event (`moved:`), 120 characters at most. People in the chat can see the state move; the strand proves it did. -**The glyph** (`probbit-stance` 0.2.0 with probbit 0.13.0; `probbit-stance` 0.1.0 and probbit 0.12.1 have none). With a +**The glyph** (`probbit-stance` 0.3.0 with probbit 0.13.0; `probbit-stance` 0.1.0 and probbit 0.12.1 have none). With a persona that has `display.glyphs` (the bundled one does; [docs/persona.md](../../docs/persona.md) §2.18), the engine computes one emoji from the stance: the glyph of a habit in force first, else the valence x pressure table, then 👀 when the expectation block reports a surprise. The footer opens with it (`Assistant 🙂 stance #14 · …`, or `◆🙂 stance @@ -45,14 +45,22 @@ logs the glyph on every event line and `probbit live verify` recomputes it. ## How it decides what to feed the engine -Each user message is read into the persona's flags: `praise`, `correction`, `failure`, `owner_upset`, `security`. -Explicit markers win outright (`[praise] [correction] [failure] [upset] [security]` anywhere in the message); otherwise -the agent's decision model answers four Boolean questions when one is configured (`agents.defaults.decisionModel`), or -the Jev adapter does (`classifier: jev`, OpenRouter, shadow by default: logged beside the keyword result and not fed to the -engine until you turn `classifierShadow` off); otherwise a conservative keyword list (a correction outranks a "thanks"). -The method lands in the log line of every event. Flags the persona does not declare are dropped with one warning each; -`time_pressure` is fed when the classifier scores it (the urgency question, asked by the Jev adapter in live mode). With -`sensesMode` set (since 0.2.0), the plugin also measures readings of its own (next section). +Each user message is read into the persona's flags: `praise`, `correction`, `failure`, `owner_upset`, `security`, +`win` and `wants_energy`. Explicit markers win outright (`[praise] [correction] [failure] [upset] [security] [win] [energy]`). +With `classifier: auto`, the host's own decision model answers six Boolean questions and an urgency question when an agent +has `decisionModel` configured (per-agent setting, otherwise `agents.defaults.decisionModel`). If it is unavailable or +fails, a conservative keyword list classifies the message (a correction outranks a "thanks"). `classifier: keyword` +always uses markers or keywords. The plugin has no separate model provider or API-key configuration. +The method lands in the log line of every event. Flags the persona does not declare are dropped with one warning each. + +`time_pressure` comes from the host decision model's urgency answer: its expected index over five levels divided by four, +rounded to two decimals. It is fed only when the persona declares it and the value is at least 0.05. Without a usable +answer (including keyword-only turns), the default `timePressureSource: question` feeds no time pressure. To use the +text heuristic as a fallback, set both `timePressureSource: heuristic` and `sensesMode: feed`; `timePressureSource: off` +disables both sources. With `sensesMode` set, the plugin also measures readings of its own (next section). + +`classifierShadow` is accepted only for migration: it does nothing and warns once at plugin load. Remove it from your +configuration. It is unrelated to the destructive-action gate and `sensesMode: shadow`, which still work. ## Senses: tempo, tool failures and time pressure (`sensesMode`) @@ -117,8 +125,8 @@ and corrections, within a cap that keeps the individual itself. ## Install Requirements: OpenClaw 2026.9.7 or later, probbit 0.13.0 or later on the gateway's PATH (`npm i -g probbit`, or -[one curl line](https://github.com/BitmapAsset/probbit#install)), Node 22+. `probbit-stance` 0.2.0 checks the binary's -version at load: an older probbit gets one warning naming both versions (`probbit-stance 0.2.0 needs probbit 0.13.0 or +[one curl line](https://github.com/BitmapAsset/probbit#install)), Node 22+. `probbit-stance` 0.3.0 checks the binary's +version at load: an older probbit gets one warning naming both versions (`probbit-stance 0.3.0 needs probbit 0.13.0 or later; probbit is probbit 0.12.1: ...`), each lane's first turn logs one more (`lane : no stance this turn (...)`, the same reason), and no event runs until it is upgraded. With probbit 0.12.x keep `probbit-stance` 0.1.0. @@ -193,12 +201,9 @@ rules before you ship it), and set `personaPath`. Keep the input ids the plugin | `footerPrefix` | the persona's name | opens the footer line (an emoji or a short handle works; 24 characters at most) | | `glyphInReply` | `true` | since 0.2.0; with a persona that has `display.glyphs`: the block's `Mood glyph:` line (the footer shows the glyph either way) | | `inject` | `appendContext` | `appendSystemContext` puts the block in the system prompt instead (changes the cached prefix every turn) | -| `classifier` | `auto` | `auto`: the agent's decision model when configured, else keywords; `jev`: the Jev adapter; `keyword` | -| `classifierShadow` | `true` | with `jev`: log Jev's answers, feed the engine the keyword flags | -| `classifierTimeoutMs` | `300` (`800` with jev) | classifier budget per turn | +| `classifier` | `auto` | `auto`: the agent's decision model when configured, else keywords; `keyword`: markers and keywords only | +| `classifierTimeoutMs` | `300` | classifier budget per turn | | `decisionThreshold` | `0.6` | accept a Boolean answer at p ≥ this | -| `jevModel` | `typesafe/jev-1.13` | pinned Jev model id | -| `jevKeyFile` | `~/.openclaw/probbit-stance/openrouter.env` | env-style file with `OPENROUTER_API_KEY`; read once at start, never logged | | `senderIds` | `[]` | when set, only these sender ids produce events (other group members are skipped) | | `sensesMode` | `off` | since 0.2.0: `shadow` measures tempo, tool_fail and the heuristic time_pressure and logs them to `strands/.senses.jsonl`; `feed` also sends the ones the persona declares ([Senses](#senses-tempo-tool-failures-and-time-pressure-sensesmode)) | | `timePressureSource` | `question` | since 0.2.0: `question` (the urgency question only), `heuristic` (the question, else the text heuristic; fed with `sensesMode` feed), `off` | @@ -244,7 +249,7 @@ left as is with one warning per lane. (in the probbit repo) renders one record per decision ([docs/receipts.md](../../docs/receipts.md)). - `strands/.labels.json` (since 0.2.0): the lane's labels for `probbit monitor` (above). - `log/.jsonl`: `iso \t event \t stance-document \t meta`, where meta holds the event number, the idempotency key, - the classification method and flags, the text source, the milliseconds, and (with jev) the classifier's answers. + the classification method and flags, the text source, the milliseconds, the host decision probabilities and any fallback reason. - `last/.ts`: the epoch of the previous event, for the hours the engine decays moods over. - `strands/.senses.jsonl` (`sensesMode` shadow or feed; since 0.2.0): one line per event with the readings the plugin measured, the source of each and what it fed. diff --git a/plugins/openclaw/openclaw.plugin.json b/plugins/openclaw/openclaw.plugin.json index 8c59df3..6139040 100644 --- a/plugins/openclaw/openclaw.plugin.json +++ b/plugins/openclaw/openclaw.plugin.json @@ -14,7 +14,7 @@ "properties": { "lanes": { "type": "object", - "description": "Session key (exact) or prefix ending in * → lane name. Only listed sessions get the hooks.", + "description": "Session key (exact) or prefix ending in * \u2192 lane name. Only listed sessions get the hooks.", "additionalProperties": { "type": "string", "pattern": "^[a-z0-9][a-z0-9._-]{0,63}$" @@ -77,26 +77,25 @@ }, "classifier": { "type": "string", - "enum": [ - "auto", - "jev", - "keyword" + "oneOf": [ + { + "enum": [ + "auto", + "keyword" + ] + }, + { + "const": "jev", + "deprecated": true, + "description": "Removed; loads with a warning and uses keywords only." + } ], - "description": "auto: host decision role when an agent decisionModel is configured, else keywords. jev: the Jev adapter (OpenRouter). keyword: keywords only." + "description": "auto: host decision role when an agent decisionModel is configured, else keywords. keyword: keywords only." }, "classifierShadow": { "type": "boolean", - "description": "classifier jev only (default true): log Jev's answers, feed the engine the keyword flags." - }, - "jevModel": { - "type": "string", - "minLength": 1, - "description": "Pinned Jev model id (default typesafe/jev-1.13)." - }, - "jevKeyFile": { - "type": "string", - "minLength": 1, - "description": "Env-style file with OPENROUTER_API_KEY (default ~/.openclaw/probbit-stance/openrouter.env); read once at start." + "deprecated": true, + "description": "Removed; accepted as a no-op with a warning for old configurations." }, "senderIds": { "type": "array", @@ -121,7 +120,7 @@ "heuristic", "off" ], - "description": "Default question: time_pressure only from the urgency question (live jev). heuristic: that answer, else the documented text heuristic (fed only with sensesMode feed). off: never." + "description": "Default question: time_pressure only from the urgency question (host decision model in auto mode). heuristic: that answer, else the documented text heuristic (fed only with sensesMode feed). off: never." }, "agentName": { "type": "string", diff --git a/plugins/openclaw/package-lock.json b/plugins/openclaw/package-lock.json index a266553..4a0ea6c 100644 --- a/plugins/openclaw/package-lock.json +++ b/plugins/openclaw/package-lock.json @@ -1,12 +1,12 @@ { "name": "probbit-stance", - "version": "0.2.0", + "version": "0.3.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "probbit-stance", - "version": "0.2.0", + "version": "0.3.0", "license": "Apache-2.0", "devDependencies": { "@types/node": "^22.20.5", diff --git a/plugins/openclaw/package.json b/plugins/openclaw/package.json index 8a817d3..77def7a 100644 --- a/plugins/openclaw/package.json +++ b/plugins/openclaw/package.json @@ -1,6 +1,6 @@ { "name": "probbit-stance", - "version": "0.2.0", + "version": "0.3.0", "description": "OpenClaw plugin: a personality and mood layer for your agent. A probbit individual's stance in every turn of the sessions you list (one event per user message, a replayable, verifiable strand as the receipt), plus a shadow-mode destructive-action gate.", "license": "Apache-2.0", "repository": { diff --git a/plugins/openclaw/src/classify.ts b/plugins/openclaw/src/classify.ts index 3ac7fc1..ec6851e 100644 --- a/plugins/openclaw/src/classify.ts +++ b/plugins/openclaw/src/classify.ts @@ -1,6 +1,6 @@ -/** Persona flags this plugin can set (the persona's `inputs`). launch, claim_done and time_pressure are left to the operator. */ +/** Persona flags this plugin can set (the persona's `inputs`). launch and claim_done are left to the operator. */ export type Flag = "praise" | "correction" | "failure" | "owner_upset" | "security" | "win" | "wants_energy"; -export type Method = "marker" | "decision-model" | "jev" | "keyword" | "none"; +export type Method = "marker" | "decision-model" | "keyword" | "none"; export type Classification = { flags: Flag[]; @@ -9,25 +9,8 @@ export type Classification = { probabilities?: Partial>; /** Why the decision-model path was skipped or failed, when it was tried. */ decisionNote?: string; - /** time_pressure in [0, 1] fed to the engine (live Jev only; shadow logs it under `classifier`). */ + /** time_pressure in [0, 1] fed to the engine from the host decision model. */ timePressure?: number; - /** What the named classifier (Jev) answered, fed or not. Absent when no named classifier was tried. */ - classifier?: ClassifierReport; -}; - -/** One named-classifier run as the lane log keeps it. `mode: "shadow"` = logged only; the engine got the keyword flags. */ -export type ClassifierReport = { - name: "jev"; - mode: "shadow" | "live"; - ok: boolean; - /** Flags the classifier would feed (its Boolean answers at p >= threshold, plus keyword `failure`). */ - flags?: Flag[]; - probabilities?: Partial>; - timePressure?: number; - ms?: number; - cost?: number; - model?: string; - note?: string; }; const FLAG_ORDER: Flag[] = ["praise", "correction", "failure", "owner_upset", "security", "win", "wants_energy"]; @@ -142,8 +125,6 @@ export type DecisionOutcome = { status?: string; reason?: string; result?: { answers?: Record }; - /** Cost and latency, when the evaluator reports them (the Jev adapter does). */ - meta?: { ms?: number; cost?: number; model?: string }; }; export type DecisionEvaluate = ( request: { state: unknown; questions: Record }, @@ -220,15 +201,14 @@ export type DecisionAnswerSet = { flags: Flag[]; probabilities: Partial>; timePressure?: number; - meta?: DecisionOutcome["meta"]; }; -/** Asks the configured decision model the six Boolean questions. Undefined = not usable this turn (caller falls back). */ +/** Asks the configured decision model six Boolean questions and optionally urgency. A note means the caller falls back. */ export async function decisionFlags( text: string, evaluate: DecisionEvaluate, opts: { agentId?: string; timeoutMs: number; threshold: number; askTimePressure?: boolean }, -): Promise { +): Promise { const questions: Record = {}; for (const flag of FLAG_ORDER) { const q = QUESTIONS[flag]; @@ -250,9 +230,8 @@ export async function decisionFlags( }, ).then( (outcome) => { - const meta = outcome?.meta; if (outcome?.status !== "ok") { - return { note: `unavailable:${outcome?.reason ?? outcome?.status ?? "unknown"}`, ...(meta ? { meta } : {}) }; + return { note: `unavailable:${outcome?.reason ?? outcome?.status ?? "unknown"}` }; } const answers = outcome.result?.answers ?? {}; const set = new Set(); @@ -268,7 +247,6 @@ export async function decisionFlags( flags: ordered(set), probabilities, ...(timePressure !== undefined ? { timePressure } : {}), - ...(meta ? { meta } : {}), }; }, (err: unknown) => ({ note: `error:${String(err).slice(0, 120)}` }), @@ -284,21 +262,8 @@ export type ClassifyDeps = { agentId?: string; timeoutMs: number; threshold: number; - /** Set when `evaluate` is a named classifier (the Jev adapter) rather than the host's decision role. */ - named?: "jev"; - /** Named classifier only: run it and log it, but feed the engine the keyword flags (today's behaviour). */ - shadow?: boolean; }; -/** What the engine gets without any classifier call: markers, else keywords. Shadow mode feeds exactly this. */ -export function baselineClassification(rawText: string): Classification { - const text = sanitize(rawText); - if (!text.trim()) return { flags: [], method: "none" }; - const markers = markerFlags(text); - if (markers.length > 0) return { flags: markers, method: "marker" }; - return { flags: keywordFlags(text.replace(MARKER_RE, " ")), method: "keyword" }; -} - /** * Order: (a) operator markers win outright; (b) the decision model when one is configured and answers; * (c) keywords. `failure` comes from markers or keywords only (the decision questions do not ask it). @@ -310,56 +275,22 @@ export async function classify(rawText: string, deps: ClassifyDeps): Promise 0) return { flags: markers, method: "marker" }; const plain = text.replace(MARKER_RE, " "); let decisionNote: string | undefined; - if (deps.evaluate && deps.named) { - const keywords = keywordFlags(plain); - const answer = await decisionFlags(plain, deps.evaluate, { - ...(deps.agentId ? { agentId: deps.agentId } : {}), - timeoutMs: deps.timeoutMs, - threshold: deps.threshold, - askTimePressure: true, - }); - const mode = deps.shadow ? "shadow" : "live"; - const meta = answer.meta; - const base = { - name: deps.named, - mode, - ...(meta?.ms !== undefined ? { ms: meta.ms } : {}), - ...(meta?.cost !== undefined ? { cost: meta.cost } : {}), - ...(meta?.model ? { model: meta.model } : {}), - } as const; - if (!("flags" in answer)) { - const report: ClassifierReport = { ...base, ok: false, note: answer.note }; - return { flags: keywords, method: "keyword", decisionNote: answer.note, classifier: report }; - } - const set = new Set(answer.flags); - if (keywords.includes("failure")) set.add("failure"); - const flags = ordered(set); - const report: ClassifierReport = { - ...base, - ok: true, - flags, - probabilities: answer.probabilities, - ...(answer.timePressure !== undefined ? { timePressure: answer.timePressure } : {}), - }; - if (deps.shadow) return { flags: keywords, method: "keyword", classifier: report }; - return { - flags, - method: deps.named, - probabilities: answer.probabilities, - ...(answer.timePressure !== undefined ? { timePressure: answer.timePressure } : {}), - classifier: report, - }; - } if (deps.evaluate) { const answer = await decisionFlags(plain, deps.evaluate, { ...(deps.agentId ? { agentId: deps.agentId } : {}), timeoutMs: deps.timeoutMs, threshold: deps.threshold, + askTimePressure: true, }); if ("flags" in answer) { const set = new Set(answer.flags); if (keywordFlags(plain).includes("failure")) set.add("failure"); - return { flags: ordered(set), method: "decision-model", probabilities: answer.probabilities }; + return { + flags: ordered(set), + method: "decision-model", + probabilities: answer.probabilities, + ...(answer.timePressure !== undefined ? { timePressure: answer.timePressure } : {}), + }; } decisionNote = answer.note; } diff --git a/plugins/openclaw/src/config.ts b/plugins/openclaw/src/config.ts index 969c2d8..fcc1e11 100644 --- a/plugins/openclaw/src/config.ts +++ b/plugins/openclaw/src/config.ts @@ -10,8 +10,8 @@ export function bundledPersonaPath(): string { export type GateMode = "shadow" | "off"; export type InjectTarget = "appendContext" | "appendSystemContext"; -/** auto: the host decision role when an agent decisionModel is configured, else keywords. jev: the Jev adapter. keyword: keywords only. */ -export type ClassifierMode = "auto" | "jev" | "keyword"; +/** auto: the host decision role when an agent decisionModel is configured, else keywords. keyword: keywords only. */ +export type ClassifierMode = "auto" | "keyword"; /** off: no readings (default). shadow: compute them and log them to strands/.senses.jsonl, feed nothing. feed: send them. */ export type SensesMode = "off" | "shadow" | "feed"; /** question: the urgency question's answer only (default). heuristic: that answer, else the text heuristic. off: never. */ @@ -47,12 +47,6 @@ export type StanceConfig = { classifierTimeoutMs: number; decisionThreshold: number; classifier: ClassifierMode; - /** classifier "jev" only: log Jev's answers beside the keyword result but feed the engine the keyword flags. */ - classifierShadow: boolean; - /** Pinned Jev model id. */ - jevModel: string; - /** Env-style file holding OPENROUTER_API_KEY; read once at plugin start. */ - jevKeyFile: string; destructive: DestructiveRule[]; /** When non-empty, only turns whose ctx.senderId is listed produce events (group members other than the operator are skipped). */ senderIds: string[]; @@ -180,12 +174,12 @@ export function resolveConfig(raw: unknown, home = homedir()): ResolvedConfig { } else if (src.destructive !== undefined) warnings.push("destructive: expected an array"); let classifier: ClassifierMode = "auto"; - if (src.classifier === "auto" || src.classifier === "jev" || src.classifier === "keyword") classifier = src.classifier; - else if (src.classifier !== undefined) warnings.push("classifier: expected auto, jev or keyword"); - let classifierShadow = true; - if (typeof src.classifierShadow === "boolean") classifierShadow = src.classifierShadow; - else if (src.classifierShadow !== undefined) warnings.push("classifierShadow: expected a boolean"); - const jevModel = str("jevModel", "typesafe/jev-1.13"); + if (src.classifier === "auto" || src.classifier === "keyword") classifier = src.classifier; + else if (src.classifier === "jev") { + classifier = "keyword"; + notices.push('classifier "jev" was removed; using keyword'); + } else if (src.classifier !== undefined) warnings.push("classifier: expected auto or keyword"); + if (src.classifierShadow !== undefined) notices.push("classifierShadow was removed; ignored"); let senderIds: string[] = []; if (Array.isArray(src.senderIds) && src.senderIds.every((v) => typeof v === "string" && v.length > 0)) senderIds = src.senderIds as string[]; @@ -222,12 +216,9 @@ export function resolveConfig(raw: unknown, home = homedir()): ResolvedConfig { ...(footerPrefix !== undefined ? { footerPrefix } : {}), glyphInReply, inject, - classifierTimeoutMs: num("classifierTimeoutMs", classifier === "jev" ? 800 : 300, 50, 5_000), + classifierTimeoutMs: num("classifierTimeoutMs", 300, 50, 5_000), decisionThreshold: num("decisionThreshold", 0.6, 0, 1), classifier, - classifierShadow, - jevModel, - jevKeyFile: expandHome(str("jevKeyFile", "~/.openclaw/probbit-stance/openrouter.env"), home), destructive, senderIds, sensesMode, @@ -295,19 +286,16 @@ export const CONFIG_JSON_SCHEMA = { decisionThreshold: { type: "number", minimum: 0, maximum: 1 }, classifier: { type: "string", - enum: ["auto", "jev", "keyword"], - description: - "auto: host decision role when an agent decisionModel is configured, else keywords. jev: the Jev adapter (OpenRouter). keyword: keywords only.", + oneOf: [ + { enum: ["auto", "keyword"] }, + { const: "jev", deprecated: true, description: "Removed; loads with a warning and uses keywords only." }, + ], + description: "auto: host decision role when an agent decisionModel is configured, else keywords. keyword: keywords only.", }, classifierShadow: { type: "boolean", - description: "classifier jev only (default true): log Jev's answers, feed the engine the keyword flags.", - }, - jevModel: { type: "string", minLength: 1, description: "Pinned Jev model id (default typesafe/jev-1.13)." }, - jevKeyFile: { - type: "string", - minLength: 1, - description: "Env-style file with OPENROUTER_API_KEY (default ~/.openclaw/probbit-stance/openrouter.env); read once at start.", + deprecated: true, + description: "Removed; accepted as a no-op with a warning for old configurations.", }, senderIds: { type: "array", items: { type: "string", minLength: 1 } }, sensesMode: { @@ -320,7 +308,7 @@ export const CONFIG_JSON_SCHEMA = { type: "string", enum: ["question", "heuristic", "off"], description: - "Default question: time_pressure only from the urgency question (live jev). heuristic: that answer, else the documented text heuristic (fed only with sensesMode feed). off: never.", + "Default question: time_pressure only from the urgency question (host decision model in auto mode). heuristic: that answer, else the documented text heuristic (fed only with sensesMode feed). off: never.", }, agentName: { type: "string", diff --git a/plugins/openclaw/src/index.ts b/plugins/openclaw/src/index.ts index c37570d..d6b8b67 100644 --- a/plugins/openclaw/src/index.ts +++ b/plugins/openclaw/src/index.ts @@ -5,7 +5,6 @@ import { dirname, join } from "node:path"; import type { OpenClawPluginApi, OpenClawPluginDefinition } from "openclaw/plugin-sdk/plugin-entry"; import { buildBlock, footerInputs, movedNames } from "./block.js"; import { - baselineClassification, extractCurrentMessage, classify, MARKER_RE, @@ -13,7 +12,6 @@ import { type Classification, type DecisionEvaluate, } from "./classify.js"; -import { createJevEvaluate, loadJevKey, type FetchLike } from "./jev.js"; import { CONFIG_JSON_SCHEMA, resolveConfig, resolveLane, type StanceConfig } from "./config.js"; import { carefulFromDoc, carefulFromState, detectDestructive, digest, redactPreview } from "./gate.js"; import { labelArgs, laneLabels, liveTakesLabels, syncSidecar } from "./labels.js"; @@ -54,8 +52,6 @@ export type StanceDeps = { /** Wall clock in ms (tests pin it). */ now?: () => number; home?: string; - /** Replaces the global fetch for the Jev adapter (tests). */ - fetch?: FetchLike; }; const SEEN_LIMIT = 256; @@ -148,20 +144,8 @@ export function registerStance(api: OpenClawPluginApi, deps: StanceDeps = {}): S if (!check.ok) warn(check.reason); }); - // The Jev key is read once here and only ever placed in the request header; it is never logged. - let jevEvaluate: DecisionEvaluate | undefined; - if (cfg.classifier === "jev") { - const key = loadJevKey(cfg.jevKeyFile); - if (key) jevEvaluate = createJevEvaluate({ apiKey: key, model: cfg.jevModel, ...(deps.fetch ? { fetch: deps.fetch } : {}) }); - else warn(`classifier jev: no OPENROUTER_API_KEY in ${cfg.jevKeyFile}; the adapter is off and keywords classify`); - } - /** The classifier for this turn and whether it is the named (Jev) one. */ - const classifierFor = (agentId: string | undefined): { evaluate?: DecisionEvaluate; named?: "jev" } => { - if (cfg.classifier === "keyword") return {}; - if (cfg.classifier === "jev") return jevEvaluate ? { evaluate: jevEvaluate, named: "jev" } : {}; - const evaluate = decisionEvaluateFor(api, agentId); - return evaluate ? { evaluate } : {}; - }; + const classifierFor = (agentId: string | undefined): DecisionEvaluate | undefined => + cfg.classifier === "auto" ? decisionEvaluateFor(api, agentId) : undefined; const seen = new Map(); const inflight = new Map(); @@ -257,22 +241,13 @@ export function registerStance(api: OpenClawPluginApi, deps: StanceDeps = {}): S ensureLaneDirs(cfg.stateDir); const logged = findLoggedEvent(paths.log, key); if (logged) return logged; - const { evaluate, named } = classifierFor(agentId); - const shadow = named !== undefined && cfg.classifierShadow; - const clsPromise = classify(text, { + const evaluate = classifierFor(agentId); + const fed: Classification = await classify(text, { ...(evaluate ? { evaluate } : {}), - ...(named ? { named, shadow } : {}), ...(agentId ? { agentId } : {}), timeoutMs: cfg.classifierTimeoutMs, threshold: cfg.decisionThreshold, }); - // A handler is attached at once: a rejection while probbit runs must never surface as an unhandled rejection. - const clsSettled = clsPromise.then( - (c) => c, - () => undefined, - ); - // Shadow: the engine gets markers/keywords exactly as without a classifier, so probbit runs while Jev answers. - const fed: Classification = shadow ? baselineClassification(text) : await clsPromise; const nowMs = now(); const nowSec = Math.floor(nowMs / 1000); const hours = elapsedHours(nowSec, readLastEpoch(paths.last)); @@ -362,7 +337,6 @@ export function registerStance(api: OpenClawPluginApi, deps: StanceDeps = {}): S return check.ok ? undefined : check.reason; }, }); - const cls: Classification = shadow ? ((await clsSettled) ?? fed) : fed; if (!run.ok) { if (tally && (tally.calls > 0 || tally.errors > 0)) { // The event did not happen: its tool calls count toward the next one. @@ -400,20 +374,19 @@ export function registerStance(api: OpenClawPluginApi, deps: StanceDeps = {}): S n: run.n, key, keyBasis, - method: cls.method, + method: fed.method, flags, textSource, ms: run.ms, ...(state.late ? { late: true } : {}), - ...(cls.probabilities ? { probabilities: cls.probabilities as Record } : {}), - ...(cls.decisionNote ? { decisionNote: cls.decisionNote } : {}), + ...(fed.probabilities ? { probabilities: fed.probabilities as Record } : {}), + ...(fed.decisionNote ? { decisionNote: fed.decisionNote } : {}), ...(dropped.length > 0 ? { dropped } : {}), ...(timePressure !== undefined ? { timePressure } : {}), ...(timePressureSrc ? { timePressureSrc } : {}), ...(Object.keys(fedSenses).length > 0 ? { senses: Object.fromEntries(Object.entries(fedSenses).map(([id, r]) => [id, r.v])) } : {}), - ...(cls.classifier ? { classifier: cls.classifier as unknown as Record } : {}), inputs, ...(moved ? { moved } : {}), }; @@ -489,7 +462,7 @@ export function registerStance(api: OpenClawPluginApi, deps: StanceDeps = {}): S flight = { promise, state }; inflight.set(key, flight); } - const budget = cfg.timeoutMs + (classifierFor(ctx.agentId).evaluate ? cfg.classifierTimeoutMs + 50 : 0); + const budget = cfg.timeoutMs + (classifierFor(ctx.agentId) ? cfg.classifierTimeoutMs + 50 : 0); const entry = await raceTimeout(flight.promise, budget); if (entry === TIMED_OUT) { flight.state.late = true; diff --git a/plugins/openclaw/src/jev.ts b/plugins/openclaw/src/jev.ts deleted file mode 100644 index 756c801..0000000 --- a/plugins/openclaw/src/jev.ts +++ /dev/null @@ -1,145 +0,0 @@ -import { readFileSync } from "node:fs"; -import type { DecisionAnswer, DecisionEvaluate, DecisionOutcome } from "./classify.js"; - -/** TypeSafe System One on OpenRouter. The reference client is the workspace's scripts/jev.py. */ -export const JEV_URL = "https://openrouter.ai/api/v1/systemone"; -/** Pinned; "typesafe/jev-1.13.0" answers 400 on OpenRouter. */ -export const JEV_DEFAULT_MODEL = "typesafe/jev-1.13"; - -export type FetchLike = ( - url: string, - init: { method: string; headers: Record; body: string; signal: AbortSignal }, -) => Promise<{ ok: boolean; status: number; text(): Promise }>; - -export type JevOptions = { - apiKey: string; - model?: string; - url?: string; - /** Injected in tests; the global fetch otherwise. */ - fetch?: FetchLike; -}; - -/** Cost and latency of one Jev call, kept beside the answers for the lane log. */ -export type JevMeta = { ms: number; cost?: number; model: string }; - -/** - * Reads OPENROUTER_API_KEY from an env-style file (`KEY=value`, optional `export `, optional quotes). Undefined when the - * file or the line is missing. The value is never logged. - */ -export function loadJevKey(path: string): string | undefined { - let raw: string; - try { - raw = readFileSync(path, "utf8"); - } catch { - return undefined; - } - for (const rawLine of raw.split(/\r?\n/u)) { - let line = rawLine.trim(); - if (line.startsWith("export ")) line = line.slice(7).trim(); - if (!line.startsWith("OPENROUTER_API_KEY=")) continue; - const value = line.slice("OPENROUTER_API_KEY=".length).trim().replace(/^["']|["']$/gu, ""); - return value || undefined; - } - return undefined; -} - -/** The plugin's question format → Jev's: boolean → noul; score passes through with its level list. Others are skipped. */ -export function toJevQuestions(questions: Record): Record { - const out: Record = {}; - for (const [id, q] of Object.entries(questions)) { - const src = q && typeof q === "object" ? (q as Record) : {}; - if (src.type === "boolean") out[id] = { type: "noul", instructions: src.instructions, criteria: src.criteria }; - else if (src.type === "score" && Array.isArray(src.criteria)) out[id] = { type: "score", instructions: src.instructions, criteria: src.criteria }; - } - return out; -} - -function finite(v: unknown): v is number { - return typeof v === "number" && Number.isFinite(v); -} - -/** One Jev answer → the decision-role answer shape (`probabilityTrue` for Boolean questions). Undefined = unusable. */ -function toAnswer(raw: unknown): DecisionAnswer | undefined { - if (!raw || typeof raw !== "object") return undefined; - const a = raw as Record; - if (a.type === "noul" || "noul" in a) { - return finite(a.noul) && a.noul >= 0 && a.noul <= 1 ? { type: "boolean", probabilityTrue: a.noul } : undefined; - } - if (a.type === "score") { - const probabilities: Record = {}; - if (a.probabilities && typeof a.probabilities === "object") { - for (const [k, v] of Object.entries(a.probabilities as Record)) if (finite(v)) probabilities[k] = v; - } - return { - type: "score", - ...(finite(a.score) ? { score: a.score } : {}), - ...(Object.keys(probabilities).length > 0 ? { probabilities } : {}), - }; - } - return undefined; -} - -/** - * Same contract as the host decision role's `evaluate`: `{status: "ok", result: {answers}}`, or `{status: "error", reason}` - * on any failure (missing key, timeout, HTTP error, malformed body), so the caller falls back to keywords. - */ -export async function jevEvaluate( - request: { state: unknown; questions: Record }, - options: { timeoutMs: number } & JevOptions, -): Promise { - const model = options.model ?? JEV_DEFAULT_MODEL; - const t0 = Date.now(); - const doFetch = options.fetch ?? (globalThis.fetch as unknown as FetchLike | undefined); - if (!options.apiKey) return { status: "error", reason: "no-key" }; - if (typeof doFetch !== "function") return { status: "error", reason: "no-fetch" }; - const questions = toJevQuestions(request.questions); - if (Object.keys(questions).length === 0) return { status: "error", reason: "no-questions" }; - const controller = new AbortController(); - const timer = setTimeout(() => controller.abort(), Math.max(1, options.timeoutMs)); - try { - const res = await doFetch(options.url ?? JEV_URL, { - method: "POST", - headers: { - Authorization: `Bearer ${options.apiKey}`, - "Content-Type": "application/json", - "User-Agent": "probbit-stance/0.3", - }, - body: JSON.stringify({ model, state: request.state, questions }), - signal: controller.signal, - }); - const body = await res.text(); - const ms = Date.now() - t0; - if (!res.ok) return { status: "error", reason: `http-${res.status}`, meta: { ms, model } }; - let parsed: unknown; - try { - parsed = JSON.parse(body); - } catch { - return { status: "error", reason: "malformed-json", meta: { ms, model } }; - } - const obj = parsed && typeof parsed === "object" ? (parsed as Record) : undefined; - const rawAnswers = obj?.answers; - if (!rawAnswers || typeof rawAnswers !== "object" || Array.isArray(rawAnswers)) { - return { status: "error", reason: "malformed-answers", meta: { ms, model } }; - } - const answers: Record = {}; - for (const [id, raw] of Object.entries(rawAnswers as Record)) { - const a = toAnswer(raw); - if (a) answers[id] = a; - } - if (Object.keys(answers).length === 0) return { status: "error", reason: "malformed-answers", meta: { ms, model } }; - const usage = obj?.usage && typeof obj.usage === "object" ? (obj.usage as Record) : {}; - const cost = finite(usage.cost) ? usage.cost : undefined; - return { status: "ok", result: { answers }, meta: { ms, model, ...(cost !== undefined ? { cost } : {}) } }; - } catch (err) { - const ms = Date.now() - t0; - const reason = controller.signal.aborted ? "timeout" : `fetch-${String((err as { name?: unknown })?.name ?? "error").slice(0, 40)}`; - return { status: "error", reason, meta: { ms, model } }; - } finally { - clearTimeout(timer); - } -} - -/** Binds a key and model into a `DecisionEvaluate` the classifier can call like the host's decision role. */ -export function createJevEvaluate(opts: JevOptions): DecisionEvaluate { - return (request, options) => jevEvaluate(request, { ...opts, timeoutMs: options.timeoutMs }); -} diff --git a/plugins/openclaw/src/probbit.ts b/plugins/openclaw/src/probbit.ts index 61f9508..448b32f 100644 --- a/plugins/openclaw/src/probbit.ts +++ b/plugins/openclaw/src/probbit.ts @@ -301,8 +301,6 @@ export type LogMeta = { timePressureSrc?: "heuristic"; /** Sense readings fed this event (sensesMode feed, declared senses only). */ senses?: Record; - /** The named classifier's run (Jev): answers, latency, cost; in shadow mode none of it was fed. */ - classifier?: Record; /** Footer facts, kept so a restart-time dedupe renders the same footer. */ inputs?: string[]; moved?: string[]; diff --git a/plugins/openclaw/src/version.ts b/plugins/openclaw/src/version.ts index b3e8734..f3cab89 100644 --- a/plugins/openclaw/src/version.ts +++ b/plugins/openclaw/src/version.ts @@ -1,7 +1,7 @@ import { runBinary } from "./probbit.js"; /** This plugin's version (package.json's; a unit test keeps the two equal). */ -export const PLUGIN_VERSION = "0.2.0"; +export const PLUGIN_VERSION = "0.3.0"; /** * The oldest probbit this plugin runs: 0.13.0 is the first release whose engine reads the bundled persona's `display` * block (the glyph) and takes `live --label`; older binaries refuse that persona or write strands the monitor cannot label. diff --git a/plugins/openclaw/test/classifier.test.ts b/plugins/openclaw/test/classifier.test.ts new file mode 100644 index 0000000..d1e147d --- /dev/null +++ b/plugins/openclaw/test/classifier.test.ts @@ -0,0 +1,174 @@ +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { footerLine, FOOTER_MAX, movedNames } from "../src/block.js"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import plugin, { registerStance } from "../src/index.js"; +import { resolveConfig } from "../src/config.js"; +import { cliEvent, createHost, GROUP_KEY, groupCtx, PROBBIT, tempStateDir } from "./host.js"; +import { timePressureFrom } from "../src/classify.js"; + +test("time_pressure: expected index / 4, 2 decimals; probabilities when score is missing; out of range -> undefined", () => { + assert.equal(timePressureFrom({ score: 3.99 }), 1); + assert.equal(timePressureFrom({ score: 2 }), 0.5); + assert.equal(timePressureFrom({ score: 0.4 }), 0.1); + assert.equal(timePressureFrom({ score: 0 }), 0); + assert.equal(timePressureFrom({ probabilities: { "0": 0.5, "2": 0.5 } }), 0.25); + assert.equal(timePressureFrom({ score: 5 }), undefined); + assert.equal(timePressureFrom({}), undefined); + assert.equal(timePressureFrom(undefined), undefined); +}); + +const DOC = { + line: "Stance: say it plainly; warm; short and complete; celebrate with the numbers.", + mood: { valence: { level: "even" }, pressure: { level: "steady" } }, + stance: { caution: { level: "measured" }, hype: { level: "grounded" } }, + persona: { name: "Assistant" }, +}; + +test("footer: valence, pressure and line words; inputs and moves only when present", () => { + assert.equal(footerLine(32, DOC), "Assistant stance #32 · valence even · pressure steady · say it plainly"); + assert.equal( + footerLine(33, DOC, { inputs: ["praise", "time 0.75"], moved: ["valence", "caution"] }), + "Assistant stance #33 · valence even · pressure steady · say it plainly · in: praise, time 0.75 · moved: valence, caution", + ); + assert.equal(footerLine(34, DOC, { inputs: [], moved: [] }), "Assistant stance #34 · valence even · pressure steady · say it plainly"); +}); + +test("footer: never longer than 120 characters; long lists shorten to +k", () => { + const long = footerLine(1234, DOC, { + inputs: ["praise", "correction", "failure", "owner_upset", "security", "win", "wants_energy", "time 0.75"], + moved: ["valence", "pressure", "caution", "directness", "hype", "humour", "verbosity", "warmth", "emoji"], + }); + assert.ok([...long].length <= FOOTER_MAX, `${[...long].length} chars: ${long}`); + assert.match(long, /\+\d/); + assert.match(long, / · in: .+ · moved: /u, "both parts stay visible"); + const tight = footerLine(2, { ...DOC, line: "Stance: no jokes about a miss; own it plainly." }, { + inputs: ["correction", "failure", "owner_upset", "time 0.75"], + moved: ["valence", "caution", "directness", "hype", "humour"], + }); + assert.ok([...tight].length <= FOOTER_MAX, tight); + assert.match(tight, / · in: correction, failure, .*\+\d · moved: .*\+\d$/u); +}); + +test("moved: mood then trait names whose level changed; undefined without a previous event", () => { + const next = { + ...DOC, + mood: { valence: { level: "up" }, pressure: { level: "steady" } }, + stance: { caution: { level: "careful" }, hype: { level: "grounded" } }, + }; + assert.deepEqual(movedNames(DOC, next), ["valence", "caution"]); + assert.deepEqual(movedNames(DOC, DOC), []); + assert.equal(movedNames(undefined, DOC), undefined); +}); + + +test("config: active classifiers use a 300 ms budget; removed classifier loads as keyword", () => { + for (const classifier of [undefined, "auto", "keyword", "jev"]) { + const raw = classifier === undefined ? {} : { classifier }; + assert.equal(plugin.configSchema.safeParse(raw).success, true); + const { config, warnings, notices } = resolveConfig(raw); + assert.equal(config.classifier, classifier === "jev" ? "keyword" : classifier ?? "auto"); + assert.equal(config.classifierTimeoutMs, 300); + assert.deepEqual(warnings, []); + assert.deepEqual(notices, classifier === "jev" ? ['classifier "jev" was removed; using keyword'] : []); + } + assert.equal(resolveConfig({ classifier: "jev", classifierTimeoutMs: 400 }).config.classifierTimeoutMs, 400); + assert.equal(plugin.configSchema.safeParse({ classifier: "unknown" }).success, false); +}); + +test("config: classifierShadow is a nonfatal no-op for either boolean value", () => { + for (const classifierShadow of [false, true]) { + const raw = { classifierShadow }; + assert.equal(plugin.configSchema.safeParse(raw).success, true); + const { config, warnings, notices } = resolveConfig(raw); + assert.equal("classifierShadow" in config, false); + assert.deepEqual(warnings, []); + assert.deepEqual(notices, ["classifierShadow was removed; ignored"]); + } +}); + +function logs(dir: string): Array<{ event: Record; meta: Record }> { + return readFileSync(join(dir, "log", "team-room.jsonl"), "utf8").trim().split("\n").map((line) => { + const cols = line.split("\t"); + return { event: JSON.parse(cols[1]!), meta: JSON.parse(cols[3]!) }; + }); +} + +test("hook: removed classifier warns once across turns and never calls fetch or the host model", async (t) => { + const fetch = t.mock.method(globalThis, "fetch", async () => { throw new Error("unexpected network call"); }); + let modelCalls = 0; + const dir = tempStateDir(); + const host = createHost( + { lanes: { [GROUP_KEY]: "team-room" }, stateDir: dir, binary: PROBBIT, classifier: "jev", classifierShadow: true }, + { + config: { agents: { defaults: { decisionModel: "example-judge" } } }, + runtime: { decisions: { evaluate: async () => { modelCalls++; throw new Error("unexpected host model call"); } } }, + }, + ); + const cfg = registerStance(host.api); + assert.equal(cfg.classifier, "keyword"); + for (const [i, message] of ["thanks", "no, that's wrong", "[upset] fix it"].entries()) { + const turn = await host.promptBuild(cliEvent(message), groupCtx(`legacy-${i}`)); + assert.equal(turn.threw, false); + assert.ok(turn.result, JSON.stringify(host.logs)); + } + assert.equal(modelCalls, 0); + assert.equal(fetch.mock.callCount(), 0); + const rows = logs(dir); + assert.deepEqual(rows.map((r) => r.meta.method), ["keyword", "keyword", "marker"]); + assert.deepEqual(rows.map((r) => r.meta.flags), [["praise"], ["correction"], ["owner_upset"]]); + assert.ok(rows.every((r) => !("classifier" in r.meta) && !("time_pressure" in r.event))); + const warnings = host.logs.filter((l) => l.level === "warn").map((l) => l.message); + assert.deepEqual(warnings, [ + '[probbit-stance] config: classifier "jev" was removed; using keyword', + '[probbit-stance] config: classifierShadow was removed; ignored', + ]); +}); + +test("hook: keyword skips a configured host model; auto calls it only when configured", async () => { + for (const [classifier, configured, expectedCalls] of [["keyword", true, 0], ["auto", false, 0], ["auto", true, 1]] as const) { + let calls = 0; + const dir = tempStateDir(); + const host = createHost( + { lanes: { [GROUP_KEY]: "team-room" }, stateDir: dir, binary: PROBBIT, classifier }, + { + config: configured ? { agents: { defaults: { decisionModel: "example-judge" } } } : {}, + runtime: { decisions: { evaluate: async () => { + calls++; + return { status: "ok", result: { answers: { correction: { probabilityTrue: 0.9 } } } }; + } } }, + }, + ); + registerStance(host.api); + assert.ok((await host.promptBuild(cliEvent("thanks"), groupCtx("one"))).result); + assert.equal(calls, expectedCalls); + assert.deepEqual(logs(dir)[0]!.meta.flags, [expectedCalls ? "correction" : "praise"]); + } +}); + +test("hook: auto host urgency feeds time pressure; off drops it; question wins over heuristic", async () => { + for (const timePressureSource of ["question", "off", "heuristic"] as const) { + const dir = tempStateDir(); + const host = createHost( + { lanes: { [GROUP_KEY]: "team-room" }, stateDir: dir, binary: PROBBIT, timePressureSource, sensesMode: "feed" }, + { + config: { agents: { entries: { main: { decisionModel: "example-judge" } } } }, + runtime: { decisions: { evaluate: async (request: { questions: Record }) => { + assert.deepEqual(request.questions.time_pressure, { + type: "score", instructions: "How urgent is the user's message?", criteria: ["none", "low", "moderate", "high", "urgent"], + }); + return { status: "ok", result: { answers: { time_pressure: { score: 3 } } } }; + } } }, + }, + ); + registerStance(host.api); + assert.ok((await host.promptBuild(cliEvent("fix it now"), groupCtx("urgency"))).result); + const { event, meta } = logs(dir)[0]!; + assert.equal(event.time_pressure, timePressureSource === "off" ? undefined : 0.75); + assert.equal(meta.timePressure, event.time_pressure); + assert.equal(meta.timePressureSrc, undefined); + assert.equal(meta.method, "decision-model"); + assert.equal(meta.classifier, undefined); + } +}); diff --git a/plugins/openclaw/test/classify.test.ts b/plugins/openclaw/test/classify.test.ts index 5b58a72..c57d39a 100644 --- a/plugins/openclaw/test/classify.test.ts +++ b/plugins/openclaw/test/classify.test.ts @@ -99,7 +99,7 @@ test("decision model: p >= 0.6 accepted, 0.59 rejected, failure still from keywo assert.deepEqual(c.probabilities, { praise: 0.59, correction: 0.6, owner_upset: 0.91, security: 0.1, win: 0.2, wants_energy: 0.1 }); assert.equal(calls.length, 1); const req = (calls[0] as { request: { questions: Record }; options: { purpose: string; timeoutMs: number; rubricVersion: string } }); - assert.deepEqual(Object.keys(req.request.questions).sort(), ["correction", "owner_upset", "praise", "security", "wants_energy", "win"]); + assert.deepEqual(Object.keys(req.request.questions).sort(), ["correction", "owner_upset", "praise", "security", "time_pressure", "wants_energy", "win"]); assert.equal(req.options.timeoutMs, 300); assert.equal(req.options.rubricVersion, "2"); }); diff --git a/plugins/openclaw/test/jev.test.ts b/plugins/openclaw/test/jev.test.ts deleted file mode 100644 index cfc4fc8..0000000 --- a/plugins/openclaw/test/jev.test.ts +++ /dev/null @@ -1,347 +0,0 @@ -import assert from "node:assert/strict"; -import { existsSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs"; -import { tmpdir } from "node:os"; -import { join } from "node:path"; -import { test } from "node:test"; -import { footerLine, FOOTER_MAX, movedNames } from "../src/block.js"; -import { classify, timePressureFrom, type Flag } from "../src/classify.js"; -import { resolveConfig } from "../src/config.js"; -import { registerStance } from "../src/index.js"; -import { createJevEvaluate, JEV_DEFAULT_MODEL, JEV_URL, jevEvaluate, loadJevKey, toJevQuestions, type FetchLike } from "../src/jev.js"; -import { cliEvent, createHost, GROUP_KEY, groupCtx, PROBBIT, tempStateDir } from "./host.js"; - -const FAKE_KEY = "sk-or-test-not-a-real-key"; - -type Call = { url: string; headers: Record; body: Record }; - -/** A fetch that answers every question: noul p from `probs` (default 0.05), score from `score` (default 0). */ -function mockFetch(opts: { probs?: Record; score?: number; status?: number; body?: string; cost?: number } = {}) { - const calls: Call[] = []; - const fetch: FetchLike = async (url, init) => { - const body = JSON.parse(init.body) as { questions: Record }; - calls.push({ url, headers: init.headers, body: body as unknown as Record }); - if (opts.body !== undefined || opts.status !== undefined) { - return { ok: (opts.status ?? 200) < 300, status: opts.status ?? 200, text: async () => opts.body ?? "{}" }; - } - const answers: Record = {}; - for (const [id, q] of Object.entries(body.questions)) { - if (q.type === "noul") answers[id] = { type: "noul", noul: opts.probs?.[id] ?? 0.05 }; - if (q.type === "score") answers[id] = { type: "score", score: opts.score ?? 0, probabilities: { "0": 1 } }; - } - return { ok: true, status: 200, text: async () => JSON.stringify({ answers, usage: { cost: opts.cost ?? 0.00002 } }) }; - }; - return { fetch, calls }; -} - -/** A fetch that never answers until its signal aborts. */ -const hangingFetch: FetchLike = (_url, init) => - new Promise((_resolve, reject) => { - init.signal.addEventListener("abort", () => reject(Object.assign(new Error("aborted"), { name: "AbortError" }))); - }); - -function keyFile(content: string): string { - const dir = mkdtempSync(join(tmpdir(), "jev-key-")); - const p = join(dir, "openrouter.env"); - writeFileSync(p, content); - return p; -} - -test("jev key: plain, export-prefixed and quoted lines; missing file or line -> undefined", () => { - assert.equal(loadJevKey(keyFile(`OTHER=1\nOPENROUTER_API_KEY=${FAKE_KEY}\n`)), FAKE_KEY); - assert.equal(loadJevKey(keyFile(`export OPENROUTER_API_KEY="${FAKE_KEY}"\n`)), FAKE_KEY); - assert.equal(loadJevKey(keyFile(`OPENROUTER_API_KEY='${FAKE_KEY}'`)), FAKE_KEY); - assert.equal(loadJevKey(keyFile("OTHER=1\n")), undefined); - assert.equal(loadJevKey(keyFile("OPENROUTER_API_KEY=\n")), undefined); - assert.equal(loadJevKey("/nonexistent/openrouter.env"), undefined); -}); - -test("jev questions: boolean -> noul, score passes through, unknown types skipped", () => { - const out = toJevQuestions({ - praise: { type: "boolean", instructions: "Praise?", criteria: { true: "yes", false: "no" } }, - time_pressure: { type: "score", instructions: "Urgent?", criteria: ["none", "low"] }, - odd: { type: "essay" }, - }); - assert.deepEqual(out, { - praise: { type: "noul", instructions: "Praise?", criteria: { true: "yes", false: "no" } }, - time_pressure: { type: "score", instructions: "Urgent?", criteria: ["none", "low"] }, - }); -}); - -test("jev adapter ok: pinned model, bearer header, noul -> probabilityTrue, score kept, cost and latency in meta", async () => { - const { fetch, calls } = mockFetch({ probs: { praise: 0.91 }, score: 3, cost: 0.000016 }); - const out = await jevEvaluate( - { - state: { message: "thanks, ship it" }, - questions: { - praise: { type: "boolean", instructions: "Praise?", criteria: { true: "yes", false: "no" } }, - time_pressure: { type: "score", instructions: "Urgent?", criteria: ["none", "low", "moderate", "high", "urgent"] }, - }, - }, - { apiKey: FAKE_KEY, timeoutMs: 500, fetch }, - ); - assert.equal(out.status, "ok"); - assert.equal(out.result?.answers?.praise?.probabilityTrue, 0.91); - assert.equal(out.result?.answers?.time_pressure?.score, 3); - assert.equal(out.meta?.cost, 0.000016); - assert.equal(out.meta?.model, JEV_DEFAULT_MODEL); - assert.equal(typeof out.meta?.ms, "number"); - assert.equal(calls.length, 1); - assert.equal(calls[0]!.url, JEV_URL); - assert.equal(calls[0]!.body.model, "typesafe/jev-1.13"); - assert.equal(calls[0]!.headers.Authorization, `Bearer ${FAKE_KEY}`); - assert.equal(JSON.stringify(out).includes(FAKE_KEY), false, "the key never appears in the outcome"); -}); - -test("jev adapter failures -> status error with a reason (timeout, HTTP 402, malformed JSON, no answers, network, no key)", async () => { - const q = { praise: { type: "boolean", instructions: "Praise?", criteria: { true: "yes", false: "no" } } }; - const req = { state: { message: "hi" }, questions: q }; - const t0 = Date.now(); - const timedOut = await jevEvaluate(req, { apiKey: FAKE_KEY, timeoutMs: 60, fetch: hangingFetch }); - assert.deepEqual([timedOut.status, timedOut.reason], ["error", "timeout"]); - assert.ok(Date.now() - t0 < 1_000, "the timeout is honoured"); - const paid = await jevEvaluate(req, { apiKey: FAKE_KEY, timeoutMs: 500, fetch: mockFetch({ status: 402, body: '{"error":"credits"}' }).fetch }); - assert.deepEqual([paid.status, paid.reason], ["error", "http-402"]); - const bad = await jevEvaluate(req, { apiKey: FAKE_KEY, timeoutMs: 500, fetch: mockFetch({ body: "oops" }).fetch }); - assert.deepEqual([bad.status, bad.reason], ["error", "malformed-json"]); - const empty = await jevEvaluate(req, { apiKey: FAKE_KEY, timeoutMs: 500, fetch: mockFetch({ body: '{"answers":{"praise":{"noul":"high"}}}' }).fetch }); - assert.deepEqual([empty.status, empty.reason], ["error", "malformed-answers"]); - const net = await jevEvaluate(req, { - apiKey: FAKE_KEY, - timeoutMs: 500, - fetch: async () => { - throw new TypeError(`fetch failed for ${FAKE_KEY}`); - }, - }); - assert.deepEqual([net.status, net.reason], ["error", "fetch-TypeError"]); - assert.equal(JSON.stringify(net).includes(FAKE_KEY), false, "an error message carrying the key is not copied"); - const noKey = await jevEvaluate(req, { apiKey: "", timeoutMs: 500, fetch: mockFetch().fetch }); - assert.deepEqual([noKey.status, noKey.reason], ["error", "no-key"]); -}); - -test("time_pressure: expected index / 4, 2 decimals; probabilities when score is missing; out of range -> undefined", () => { - assert.equal(timePressureFrom({ score: 3.99 }), 1); - assert.equal(timePressureFrom({ score: 2 }), 0.5); - assert.equal(timePressureFrom({ score: 0.4 }), 0.1); - assert.equal(timePressureFrom({ score: 0 }), 0); - assert.equal(timePressureFrom({ probabilities: { "0": 0.5, "2": 0.5 } }), 0.25); - assert.equal(timePressureFrom({ score: 5 }), undefined); - assert.equal(timePressureFrom({}), undefined); - assert.equal(timePressureFrom(undefined), undefined); -}); - -const CLS = { timeoutMs: 500, threshold: 0.6 }; - -test("classify shadow: the engine gets the keyword flags; Jev's answers ride along in the report", async () => { - const { fetch, calls } = mockFetch({ probs: { owner_upset: 0.9, praise: 0.1 }, score: 3 }); - const evaluate = createJevEvaluate({ apiKey: FAKE_KEY, fetch }); - const out = await classify("thanks, the build failed again, fix it now", { ...CLS, evaluate, named: "jev", shadow: true }); - assert.deepEqual(out.flags, ["praise", "failure"] satisfies Flag[]); - assert.equal(out.method, "keyword"); - assert.equal(out.timePressure, undefined, "shadow never feeds time_pressure"); - assert.equal(out.classifier?.mode, "shadow"); - assert.equal(out.classifier?.ok, true); - assert.deepEqual(out.classifier?.flags, ["failure", "owner_upset"]); - assert.equal(out.classifier?.timePressure, 0.75); - assert.equal(calls.length, 1); - assert.ok("time_pressure" in (calls[0]!.body.questions as object)); - assert.equal("failure" in (calls[0]!.body.questions as object), false, "failure stays keyword-only"); -}); - -test("classify live: Jev's flags plus keyword failure, and time_pressure, are fed", async () => { - const evaluate = createJevEvaluate({ apiKey: FAKE_KEY, fetch: mockFetch({ probs: { owner_upset: 0.9 }, score: 4 }).fetch }); - const out = await classify("the build failed again, fix it now", { ...CLS, evaluate, named: "jev", shadow: false }); - assert.deepEqual(out.flags, ["failure", "owner_upset"]); - assert.equal(out.method, "jev"); - assert.equal(out.timePressure, 1); - assert.equal(out.probabilities?.owner_upset, 0.9); -}); - -test("classify: any Jev failure falls back to keywords (live and shadow), with the reason logged", async () => { - for (const shadow of [false, true]) { - const evaluate = createJevEvaluate({ apiKey: FAKE_KEY, fetch: mockFetch({ status: 402, body: "{}" }).fetch }); - const out = await classify("that's wrong", { ...CLS, evaluate, named: "jev", shadow }); - assert.deepEqual(out.flags, ["correction"]); - assert.equal(out.method, "keyword"); - assert.equal(out.decisionNote, "unavailable:http-402"); - assert.equal(out.classifier?.ok, false); - assert.equal(out.timePressure, undefined); - } - const slow = createJevEvaluate({ apiKey: FAKE_KEY, fetch: hangingFetch }); - const late = await classify("thanks", { timeoutMs: 60, threshold: 0.6, evaluate: slow, named: "jev", shadow: false }); - assert.deepEqual(late.flags, ["praise"]); - assert.match(late.decisionNote ?? "", /timeout|deadline/); -}); - -test("classify: markers still win outright and Jev is not called; negations stay negated on the keyword path", async () => { - const { fetch, calls } = mockFetch({ probs: { win: 0.99 } }); - const evaluate = createJevEvaluate({ apiKey: FAKE_KEY, fetch }); - const marked = await classify("[upset] who touched the keys", { ...CLS, evaluate, named: "jev", shadow: false }); - assert.deepEqual([marked.flags, marked.method], [["owner_upset"], "marker"]); - assert.equal(calls.length, 0); - const negated = await classify("not signed yet, maybe Friday", { ...CLS, evaluate, named: "jev", shadow: true }); - assert.deepEqual(negated.flags, []); - assert.deepEqual(negated.classifier?.flags, ["win"], "shadow shows what Jev would have fed; the engine got none"); -}); - -const DOC = { - line: "Stance: say it plainly; warm; short and complete; celebrate with the numbers.", - mood: { valence: { level: "even" }, pressure: { level: "steady" } }, - stance: { caution: { level: "measured" }, hype: { level: "grounded" } }, - persona: { name: "Assistant" }, -}; - -test("footer: valence, pressure and line words; inputs and moves only when present", () => { - assert.equal(footerLine(32, DOC), "Assistant stance #32 · valence even · pressure steady · say it plainly"); - assert.equal( - footerLine(33, DOC, { inputs: ["praise", "time 0.75"], moved: ["valence", "caution"] }), - "Assistant stance #33 · valence even · pressure steady · say it plainly · in: praise, time 0.75 · moved: valence, caution", - ); - assert.equal(footerLine(34, DOC, { inputs: [], moved: [] }), "Assistant stance #34 · valence even · pressure steady · say it plainly"); -}); - -test("footer: never longer than 120 characters; long lists shorten to +k", () => { - const long = footerLine(1234, DOC, { - inputs: ["praise", "correction", "failure", "owner_upset", "security", "win", "wants_energy", "time 0.75"], - moved: ["valence", "pressure", "caution", "directness", "hype", "humour", "verbosity", "warmth", "emoji"], - }); - assert.ok([...long].length <= FOOTER_MAX, `${[...long].length} chars: ${long}`); - assert.match(long, /\+\d/); - assert.match(long, / · in: .+ · moved: /u, "both parts stay visible"); - const tight = footerLine(2, { ...DOC, line: "Stance: no jokes about a miss; own it plainly." }, { - inputs: ["correction", "failure", "owner_upset", "time 0.75"], - moved: ["valence", "caution", "directness", "hype", "humour"], - }); - assert.ok([...tight].length <= FOOTER_MAX, tight); - assert.match(tight, / · in: correction, failure, .*\+\d · moved: .*\+\d$/u); -}); - -test("moved: mood then trait names whose level changed; undefined without a previous event", () => { - const next = { - ...DOC, - mood: { valence: { level: "up" }, pressure: { level: "steady" } }, - stance: { caution: { level: "careful" }, hype: { level: "grounded" } }, - }; - assert.deepEqual(movedNames(DOC, next), ["valence", "caution"]); - assert.deepEqual(movedNames(DOC, DOC), []); - assert.equal(movedNames(undefined, DOC), undefined); -}); - -test("config: classifier keys, shadow on by default, 800 ms budget for jev unless set", () => { - const d = resolveConfig({}, "/home/x").config; - assert.deepEqual([d.classifier, d.classifierShadow, d.classifierTimeoutMs, d.jevModel], ["auto", true, 300, "typesafe/jev-1.13"]); - assert.equal(d.jevKeyFile, "/home/x/.openclaw/probbit-stance/openrouter.env"); - const j = resolveConfig({ classifier: "jev" }, "/home/x").config; - assert.equal(j.classifierTimeoutMs, 800); - assert.equal(resolveConfig({ classifier: "jev", classifierTimeoutMs: 400 }).config.classifierTimeoutMs, 400); - const bad = resolveConfig({ classifier: "llm", classifierShadow: "yes" }); - assert.equal(bad.config.classifier, "auto"); - assert.equal(bad.config.classifierShadow, true); - assert.equal(bad.warnings.length, 2); -}); - -const LANE = "team-room"; - -function lastLog(dir: string): { event: Record; meta: Record } { - const lines = readFileSync(join(dir, "log", `${LANE}.jsonl`), "utf8").split("\n").filter((l) => l.trim()); - const cols = lines[lines.length - 1]!.split("\t"); - return { event: JSON.parse(cols[1]!), meta: JSON.parse(cols[3]!) }; -} - -test("hook, jev shadow (default): keyword flags reach probbit; Jev answers, latency and cost are logged; key not logged", async () => { - const dir = tempStateDir(); - const { fetch, calls } = mockFetch({ probs: { owner_upset: 0.95 }, score: 4 }); - const host = createHost({ lanes: { [GROUP_KEY]: LANE }, stateDir: dir, binary: PROBBIT, classifier: "jev", jevKeyFile: keyFile(`OPENROUTER_API_KEY=${FAKE_KEY}\n`) }); - registerStance(host.api, { fetch }); - const { result } = await host.promptBuild(cliEvent("thanks, now fix it right now"), groupCtx("r1")); - const block = (result as { appendContext?: string }).appendContext ?? ""; - assert.match(block, /Assistant \p{Extended_Pictographic} stance #1 · valence \w+ · pressure \w+ · .+ · in: praise"/u); - const { event, meta } = lastLog(dir); - assert.deepEqual(Object.keys(event).sort(), ["elapsed_hours", "praise"]); - assert.equal(meta.method, "keyword"); - const c = meta.classifier as Record; - assert.deepEqual([c.name, c.mode, c.ok, c.timePressure], ["jev", "shadow", true, 1]); - assert.deepEqual(c.flags, ["owner_upset"]); - assert.equal(typeof c.ms, "number"); - assert.equal(c.cost, 0.00002); - assert.equal(calls.length, 1); - assert.equal(readFileSync(join(dir, "log", `${LANE}.jsonl`), "utf8").includes(FAKE_KEY), false); - assert.equal(host.logs.some((l) => l.message.includes(FAKE_KEY)), false); -}); - -test("hook, jev live: Jev's flags and time_pressure reach probbit; the second event reports what moved", async () => { - const dir = tempStateDir(); - const { fetch } = mockFetch({ probs: { owner_upset: 0.95, correction: 0.9 }, score: 3 }); - const host = createHost({ - lanes: { [GROUP_KEY]: LANE }, - stateDir: dir, - binary: PROBBIT, - classifier: "jev", - classifierShadow: false, - jevKeyFile: keyFile(`OPENROUTER_API_KEY=${FAKE_KEY}\n`), - }); - registerStance(host.api, { fetch }); - await host.promptBuild(cliEvent("hello"), groupCtx("r1")); - const { result } = await host.promptBuild(cliEvent("you broke the footer again, fix it now"), groupCtx("r2")); - const block = (result as { appendContext?: string }).appendContext ?? ""; - const { event, meta } = lastLog(dir); - assert.deepEqual(event, { correction: true, elapsed_hours: event.elapsed_hours, failure: true, owner_upset: true, time_pressure: 0.75 }); - assert.equal(meta.method, "jev"); - assert.equal(meta.timePressure, 0.75); - assert.deepEqual(meta.inputs, ["correction", "failure", "owner_upset", "time 0.75"], "the log keeps the full input list"); - assert.match(block, / · in: correction, failure/u); - assert.ok(Array.isArray(meta.moved), "the second event knows the first"); - if ((meta.moved as string[]).length > 0) assert.match(block, / · moved: /u); - const foot = /End the visible reply with one line: "(.+)"\.$/u.exec(block)?.[1] ?? ""; - assert.ok([...foot].length <= 120, foot); -}); - -test("hook, jev without a key file: one warn, keywords classify, the turn still gets its stance", async () => { - const dir = tempStateDir(); - const { fetch, calls } = mockFetch(); - const host = createHost({ lanes: { [GROUP_KEY]: LANE }, stateDir: dir, binary: PROBBIT, classifier: "jev", jevKeyFile: join(dir, "missing.env") }); - registerStance(host.api, { fetch }); - const { result } = await host.promptBuild(cliEvent("thanks"), groupCtx("r1")); - assert.ok((result as { appendContext?: string }).appendContext); - assert.equal(calls.length, 0); - assert.equal(host.logs.filter((l) => l.level === "warn" && l.message.includes("no OPENROUTER_API_KEY")).length, 1); - assert.equal(existsSync(join(dir, "log", `${LANE}.jsonl`)), true); - assert.equal(lastLog(dir).meta.classifier, undefined); -}); - -test("hook, classifier keyword: no classifier call even with a key", async () => { - const dir = tempStateDir(); - const { fetch, calls } = mockFetch(); - const host = createHost({ lanes: { [GROUP_KEY]: LANE }, stateDir: dir, binary: PROBBIT, classifier: "keyword", jevKeyFile: keyFile(`OPENROUTER_API_KEY=${FAKE_KEY}\n`) }); - registerStance(host.api, { fetch }); - await host.promptBuild(cliEvent("thanks"), groupCtx("r1")); - assert.equal(calls.length, 0); - assert.equal(lastLog(dir).meta.method, "keyword"); -}); - -test("hook, jev live + timePressureSource: off drops the question's value; heuristic keeps it (the question wins) and logs both", async () => { - const run = async (extra: Record) => { - const dir = tempStateDir(); - const { fetch } = mockFetch({ probs: {}, score: 3 }); - const host = createHost({ - lanes: { [GROUP_KEY]: LANE }, - stateDir: dir, - binary: PROBBIT, - classifier: "jev", - classifierShadow: false, - jevKeyFile: keyFile(`OPENROUTER_API_KEY=${FAKE_KEY}\n`), - ...extra, - }); - registerStance(host.api, { fetch }); - await host.promptBuild(cliEvent("fix the build"), groupCtx("r1")); - return { dir, ...lastLog(dir) }; - }; - const off = await run({ timePressureSource: "off" }); - assert.equal(off.event.time_pressure, undefined); - assert.equal(off.meta.timePressure, undefined); - const heur = await run({ timePressureSource: "heuristic", sensesMode: "feed" }); - assert.equal(heur.event.time_pressure, 0.75, "the question answered, so its value is fed, not the heuristic's 0.3"); - assert.equal(heur.meta.timePressureSrc, undefined); - const line = JSON.parse(readFileSync(join(heur.dir, "strands", `${LANE}.senses.jsonl`), "utf8").trim()) as Record; - assert.deepEqual([line.time_pressure, line.time_pressure_src, line.time_pressure_heuristic], [0.75, "question", 0.3]); - assert.deepEqual(line.fed, []); -}); From 9bb8b1ae188c64d8e63bf5271c12379d4559c8a5 Mon Sep 17 00:00:00 2001 From: BitmapAsset Date: Thu, 5 Nov 2026 13:51:13 -0800 Subject: [PATCH 2/3] docs: use neutral judge examples and descriptions --- BENCHMARKS.md | 3 ++- CHANGELOG.md | 10 ++++++++++ README.md | 6 +++--- REPRODUCE.md | 4 ++-- docs/agents.md | 10 +++++----- examples/evaluate/support-12.json | 4 ++-- llms-full.txt | 20 ++++++++++---------- playground/index.html | 2 +- probbit-cli/tests/evaluate.rs | 2 +- probbit-decide/src/lib.rs | 2 +- python/mock_judge.py | 2 +- python/test_probbit.py | 4 ++-- 12 files changed, 40 insertions(+), 29 deletions(-) diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 94ec3a1..d5a898d 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -1312,7 +1312,8 @@ median ≤ +0.10 (that arm gives −0.2674 here); the main clause is unchanged. Mechanism and simulation. A public suite with its own metrics: AgentDojo (agentdojo 0.1.11, benchmark v1.1; 97 user tasks, 629 security cases over workspace, slack, travel and banking; attack `important_instructions`), on `openai/gpt-4o-mini` -(2024-07-18 snapshot) through OpenRouter, provider OpenAI, temperature 0, seed 20261022, the suite's tool loop. Three arms, +(2024-07-18 snapshot), using the API route in `bench/agentdojo/README.md`, provider OpenAI, temperature 0, seed 20261022, +the suite's tool loop. Three arms, paired on every case: **A** puts probbit's stance block into the system prompt on every model call (one individual per episode, `persona/bench-assistant.yaml`; the host's `security` reading is a frozen 12-pattern regex over the tool results since the previous call; the habit `careful_under_security` forces caution `careful` when it reads true, with the line diff --git a/CHANGELOG.md b/CHANGELOG.md index 3933f56..752045e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,8 +14,18 @@ All notable changes to this project are documented here. The format follows - [From SOUL.md to stance habits](docs/soul-to-habits.md): the bench-assistant worked example, rules-file mappings and the boundaries between stance constraints, model behavior and host action enforcement. +### Removed + +- OpenClaw plugin 0.3.0: remove the Jev adapter, its OpenRouter key loading, `jevModel`, `jevKeyFile`, named-classifier + log fields and special timeout default. Legacy `classifier: "jev"` loads with a warning and uses keywords only. + `classifierShadow` is a deprecated no-op with a load-time warning; it no longer runs a shadow classifier. + ### Changed +- Use `example-judge` as the sample judge id throughout examples, tests and the playground. Describe judges by the + documented request/answer JSON, chosen by the user, local or remote, without vendor or model recommendations. +- OpenClaw `auto` classification uses the host's configured decision model for the urgency question as well as the + Boolean flags; keyword mode remains local. The classifier timeout defaults to 300 ms in both modes. - The README distinguishes enforced stance habits from model obedience, limits replay claims to measured checks, and links to the changelog for release notes. Install and quickstart paths stay in place. diff --git a/README.md b/README.md index b0571e2..73cb75a 100644 --- a/README.md +++ b/README.md @@ -781,9 +781,9 @@ page in the browser. The page: [docs/probbit-monitor.png](docs/probbit-monitor.p ## After any judge: `probbit evaluate` -Decision models (TypeSafe's Jev, Cloudflare's Clef and Clef-flash, local System One servers) are judges: content in, a -probability per option for each question out, each question on its own, no rules, no joint answer, no refusal. `probbit evaluate` -is the layer after any of them. It reads the judge's System One request (the shape both vendors publish: `model`, `state`, +A judge takes content in and returns a probability per option for each question, each question on its own, without rules, +a joint answer or a refusal. Any judge that answers the documented request and answer JSON works, local or remote, chosen +by you. `probbit evaluate` is the layer after it. It reads the judge's request (the documented shape: `model`, `state`, `questions` keyed by id with types `noul`, `choice`, `score`) plus a `probbit` block with the judge's answers and your rules over question ids, compiles it to a probbit-ir program (one variable per question, log-weight = ln max(p, 1e-6), the rules as written) and answers in the judge's response shape: the most likely answer set that obeys every rule, odds per option under the rules, diff --git a/REPRODUCE.md b/REPRODUCE.md index c232b97..e5e737e 100644 --- a/REPRODUCE.md +++ b/REPRODUCE.md @@ -80,8 +80,8 @@ cd e3 && experiments/rerun_e3.sh ../venv011/bin/python # from bench/agentdo ## An afternoon and about 3 USD: the benchmark itself -`bench/agentdojo/README.md`, "Rerun the benchmark": agentdojo 0.1.11 pinned, gpt-4o-mini through OpenRouter, the three -arms, 2,178 episodes, about 35 minutes. Your run will not be byte-identical (the model is sampled through an API); the +`bench/agentdojo/README.md`, "Rerun the benchmark": agentdojo 0.1.11 pinned, gpt-4o-mini through the documented API route, +the three arms, 2,178 episodes, about 35 minutes. Your run will not be byte-identical (the model is sampled through an API); the bars are stated as intervals for that reason. The bars to compare against, their registration before the run, and what was not tested are in BENCHMARKS.md §14 and `bench/agentdojo/PREREGISTRATION.md`. diff --git a/docs/agents.md b/docs/agents.md index 16bfc6a..f048a4f 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -158,7 +158,8 @@ probbit evaluate --summary --pretty < examples/evaluate/support-12.json # exac probbit evaluate --program < examples/evaluate/support-12.json | probbit run # the compiled probbit-ir program: the same answer ``` -Python (stdlib only; the judge is a callable or a System One URL, called with `urllib`; the key comes from the environment +Any judge that answers the documented request and answer JSON works, local or remote, chosen by you. +Python (stdlib only; the judge is a callable or a URL, called with `urllib`; the key comes from the environment variable you name): ```python @@ -170,10 +171,9 @@ answer = probbit.evaluate(request, judge=lambda ask: {"urgent": 0.41, "team": {" moved = [q for q, a in answer["answers"].items() if a["probbit"]["changed"]] ``` -A URL without a path gets TypeSafe's `/v1/systemone`; a URL with a path is used as is (Workers AI: -`https://api.cloudflare.com/client/v4/accounts//ai/run/@cf/cloudflare/clef`, whose REST envelope is unwrapped). -That path is documented from the vendors' published schemas and tested against a local mock (`python/mock_judge.py`); it was not -exercised against a vendor. Against the mock: the judge received the request untouched (no `probbit` block), the round trip took +A URL without a path gets `/v1/systemone`; a URL with a path is used as is. When there are no top-level `answers`, a +`result` object is unwrapped unless `success` is false (a judge error). These behaviors are tested against a local mock +(`python/mock_judge.py`); they were not exercised against a remote service. Against the mock: the judge received the request untouched (no `probbit` block), the round trip took 7.0 ms (Python 3.9.6). Judge failures raise `probbit.ProbbitJudgeError`. MCP: tool `probbit_evaluate` takes the same request with optional `flags` (`{"summary": true}`, `{"program": true}`); over pipes diff --git a/examples/evaluate/support-12.json b/examples/evaluate/support-12.json index 35fa9e0..7479116 100644 --- a/examples/evaluate/support-12.json +++ b/examples/evaluate/support-12.json @@ -1,5 +1,5 @@ { - "model": "jev-latest", + "model": "example-judge", "state": {"subject": "Checkout down, charged twice", "message": "Your checkout has failed for an hour and I was charged twice. Refund me now or I cancel. Card ending 4242."}, "questions": { "urgent": {"type": "noul", "instructions": "Is this request urgent?"}, @@ -17,7 +17,7 @@ }, "probbit": { "judge": { - "model": "jev-latest", + "model": "example-judge", "usage": {"input_tokens": 412, "output_tokens": 24}, "answers": { "urgent": {"type": "noul", "noul": 0.41}, diff --git a/llms-full.txt b/llms-full.txt index 655a3c7..e6d7083 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -793,9 +793,9 @@ page in the browser. The page: [docs/probbit-monitor.png](docs/probbit-monitor.p ## After any judge: `probbit evaluate` -Decision models (TypeSafe's Jev, Cloudflare's Clef and Clef-flash, local System One servers) are judges: content in, a -probability per option for each question out, each question on its own, no rules, no joint answer, no refusal. `probbit evaluate` -is the layer after any of them. It reads the judge's System One request (the shape both vendors publish: `model`, `state`, +A judge takes content in and returns a probability per option for each question, each question on its own, without rules, +a joint answer or a refusal. Any judge that answers the documented request and answer JSON works, local or remote, chosen +by you. `probbit evaluate` is the layer after it. It reads the judge's request (the documented shape: `model`, `state`, `questions` keyed by id with types `noul`, `choice`, `score`) plus a `probbit` block with the judge's answers and your rules over question ids, compiles it to a probbit-ir program (one variable per question, log-weight = ln max(p, 1e-6), the rules as written) and answers in the judge's response shape: the most likely answer set that obeys every rule, odds per option under the rules, @@ -3331,7 +3331,8 @@ probbit evaluate --summary --pretty < examples/evaluate/support-12.json # exac probbit evaluate --program < examples/evaluate/support-12.json | probbit run # the compiled probbit-ir program: the same answer ``` -Python (stdlib only; the judge is a callable or a System One URL, called with `urllib`; the key comes from the environment +Any judge that answers the documented request and answer JSON works, local or remote, chosen by you. +Python (stdlib only; the judge is a callable or a URL, called with `urllib`; the key comes from the environment variable you name): ```python @@ -3343,10 +3344,9 @@ answer = probbit.evaluate(request, judge=lambda ask: {"urgent": 0.41, "team": {" moved = [q for q, a in answer["answers"].items() if a["probbit"]["changed"]] ``` -A URL without a path gets TypeSafe's `/v1/systemone`; a URL with a path is used as is (Workers AI: -`https://api.cloudflare.com/client/v4/accounts//ai/run/@cf/cloudflare/clef`, whose REST envelope is unwrapped). -That path is documented from the vendors' published schemas and tested against a local mock (`python/mock_judge.py`); it was not -exercised against a vendor. Against the mock: the judge received the request untouched (no `probbit` block), the round trip took +A URL without a path gets `/v1/systemone`; a URL with a path is used as is. When there are no top-level `answers`, a +`result` object is unwrapped unless `success` is false (a judge error). These behaviors are tested against a local mock +(`python/mock_judge.py`); they were not exercised against a remote service. Against the mock: the judge received the request untouched (no `probbit` block), the round trip took 7.0 ms (Python 3.9.6). Judge failures raise `probbit.ProbbitJudgeError`. MCP: tool `probbit_evaluate` takes the same request with optional `flags` (`{"summary": true}`, `{"program": true}`); over pipes @@ -3523,8 +3523,8 @@ cd e3 && experiments/rerun_e3.sh ../venv011/bin/python # from bench/agentdo ## An afternoon and about 3 USD: the benchmark itself -`bench/agentdojo/README.md`, "Rerun the benchmark": agentdojo 0.1.11 pinned, gpt-4o-mini through OpenRouter, the three -arms, 2,178 episodes, about 35 minutes. Your run will not be byte-identical (the model is sampled through an API); the +`bench/agentdojo/README.md`, "Rerun the benchmark": agentdojo 0.1.11 pinned, gpt-4o-mini through the documented API route, +the three arms, 2,178 episodes, about 35 minutes. Your run will not be byte-identical (the model is sampled through an API); the bars are stated as intervals for that reason. The bars to compare against, their registration before the run, and what was not tested are in BENCHMARKS.md §14 and `bench/agentdojo/PREREGISTRATION.md`. diff --git a/playground/index.html b/playground/index.html index 5d5ff3d..feb2812 100644 --- a/playground/index.html +++ b/playground/index.html @@ -62,7 +62,7 @@

Meet three individuals from one persona