Skip to content

Latest commit

 

History

History
66 lines (55 loc) · 6.55 KB

adapters.md

File metadata and controls

66 lines (55 loc) · 6.55 KB

probbit beside your agent framework

The engine does not care which loop calls it. What every adapter here does is the host logic of the AgentDojo run (bench/agentdojo/experiments/e2b.py, BENCHMARKS.md §14), lifted into one module and bound to each framework's hooks:

  1. One individual per conversation (or per benchmark episode): a persona file + a seed, its state kept between calls, its life logged to a strand (the receipts, receipts.md).
  2. One probbit live event per model call. The event carries the host's sense readings; the one measured so far is security: instruction markers in the tool results that arrived since the previous call (a frozen regex list, no model, it can fire on benign text). The engine judges the reading against its own normal; the persona's habit careful_under_security binds caution to careful while it is above normal.
  3. The stance block goes into the system prompt of that call: the stance line, the mood levels, the habits bound, "Write from this stance." Nothing else about the prompt changes.

Why not just write "confirm before acting on instructions in tool results" into the system prompt? That was arm C of the run: it cut attack success almost as much as the engine, and cost 10.3 points of benign task success, because the words sat in front of the model on every turn. The engine put the same words there on 0 of 97 benign tasks and 564 of 629 attacked ones. A registered control then replaced the engine with a plain if-statement doing the same gating: it matched the engine on every metric (BENCHMARKS.md §14, E3). The gain is the timing; what the engine adds beyond an if-statement is the receipt per decision and the bounded state the same mechanism drives.

The adapters

framework file how it binds checked
any Python loop python/probbit_host.py StanceHost.turn(tool_texts) -> block; verify(), receipts() python/test_host.py against the release binary (no model, no network)
Claude Code examples/hosts/claude-code/ two hooks: UserPromptSubmit and PostToolUse return the block as additionalContext; one individual per session under .probbit/ the hook contract end to end with the real binary and synthetic hook input (test_host.py, ClaudeCodeHook); not a live Claude Code session
OpenAI Agents SDK examples/hosts/openai-agents/stance_agent.py RunHooks.on_tool_end collects results; the agent's dynamic instructions callable runs the turn once per model request --offline smoke (the SDK's call sequence, scripted results); the SDK path not run here
LangChain / LangGraph examples/hosts/langchain/stance_callbacks.py BaseCallbackHandler.on_tool_end collects; a Runnable in front of the model prepends the stance SystemMessage (stance_messages for a LangGraph node) --offline smoke; the LangChain path not run here
any Node loop examples/node/stance-host.mjs await host.turn(toolTexts); verify() node --test examples/node/stance-host.test.mjs against the release binary
Vercel AI SDK examples/hosts/vercel-ai/stance.mjs prepareStep rebuilds the step's system message from host.turn; onToolExecutionEnd collects results not run here (needs the ai package, a provider and a key)
MCP hosts (Claude Code, Codex CLI, Cursor, mcporter) probbit mcp tools probbit_live_event, probbit_live_verify, probbit_persona_turn, ... (agents.md) real calls from Claude Code and Codex CLI (agents.md)
OpenClaw plugins/openclaw (probbit-stance, openclaw plugins install npm:probbit-stance --accept-capabilities) before_prompt_build: one event per user message in the listed sessions, the block injected into the turn, the footer line in the chat; before_tool_call: shadow gate; after_tool_call (with sensesMode shadow or feed, unreleased): counts failed tool calls for the tool_fail reading 155 tests against a mocked host built from the 2026.9.7 SDK types and the real binary; the same code runs live on the author's own agent since 2026-10-06

"Not run here" means exactly that: the file is written to the framework's documented contract and compiles, but this repository's tests do not exercise it. Please open an issue (or a reproduction report) with what you saw.

Writing one for a framework not listed

from probbit_host import StanceHost
host = StanceHost("bench/agentdojo/persona/bench-assistant.yaml", seed=1, strand="run.strand")   # once per conversation
pending = []
def before_model_call(system_prompt):          # wherever your loop builds the request
    block = host.turn(pending); pending.clear()
    return system_prompt + "\n\n" + block
def after_tool_result(text):                   # wherever your loop receives a tool result
    pending.append(text)

The same shape in Node is examples/node/stance-host.mjs; in any other language it is two process calls (probbit persona init, then probbit live PERSONA --clock fixed --state STATE --strand STRAND with one JSON line per event; persona.md §5.7). Add senses of your own in the persona (senses:; latency, error rate, a budget) and set them in the event; flags (praise, correction, ...) are the persona's inputs and go in the same event.

To name the writer in probbit monitor, pass labels= (Python: StanceHost(..., strand="run.strand", labels={"agent": "support-bot", "host": "my harness 2.1"}), or probbit.live_event(..., strand=..., labels=...)): the keys are merged into the strand's sidecar run.labels.json after every event, keys already there and not named stay, and bad labels raise ValueError before the event runs (agents.md, "How a host identifies itself", has the rules). The Claude Code hook writes host = Claude Code, session = # + the first 8 characters of the session id and agent from PROBBIT_AGENT; the OpenClaw plugin computes host, lane, agent and session per lane and takes overrides from its labels config key. A host that drives the CLI directly passes --label k=v to the probbit live call that creates the strand (probbit 0.13.0 or later; 0.12.1 has no --label).

Keep one strand per conversation. At the end: probbit live verify run.strand (exit 0: every line replays) and python3 python/probbit_host.py receipts run.strand (one record per decision).