Add bench: A/B harness for pi agents with and without xwebmcp tools

.github/workflows/validate.yml

@@ -25,3 +25,5 @@ jobs:
         run: node --test "tests/*.test.mjs"
       - name: Run extension tests
         run: node --test "extension/**/*.test.mjs"
+      - name: Run bench tests
+        run: node --test "bench/**/*.test.mjs"

.gitignore

@@ -1,2 +1,4 @@
 node_modules/
 .DS_Store
+bench/results/
+bench/.env

README.md

@@ -18,6 +18,7 @@ tests/                           # node --test "tests/*.test.mjs"
 registry.json                    # machine-readable index of every tool
 spec/                            # the rules this repo follows, and JSON Schemas
 scripts/validate-registry.mjs    # checks registry.json and domains/ against the spec
+bench/                           # A/B harness: an agent with vs without a domain's tools (bench/README.md)
 ```
 
 ## Specification

bench/.env.example

@@ -0,0 +1,6 @@
+# Copy to bench/.env and fill in; run.mjs loads it. Variables already set in the environment,
+# such as Hypeship secrets, take precedence over this file.
+KERNEL_API_KEY=
+# The API key for the provider in --model, for example:
+OPENAI_API_KEY=
+# ANTHROPIC_API_KEY=

bench/README.md

@@ -0,0 +1,216 @@
+# bench
+
+Measures, per user intent on a site, how much more often and how much faster an agent succeeds
+when the domain's read-only xwebmcp tools are registered in its Kernel browser.
+
+Each trial runs [pi](https://github.com/earendil-works/pi) (`pi -p --mode json`) against one
+Kernel browser. Both arms get the same model, prompt, kernel MCP server, tool allowlist and
+browser settings:
+
+- **control**: a fresh browser on the intent's start URL, with nothing registered;
+- **treatment**: the same, plus the domain's read-only tools at commits pinned for the run.
+
+Both arms get the same system prompt, which tells the agent to list the page's WebMCP tools first
+and fall back to Playwright or computer actions when none fits, as Kernel's docs tell customers
+to instruct their agents. It never mentions xwebmcp or whether tools are registered, so a control
+agent lists, finds nothing and falls back. A treatment trial that never invokes a tool still
+counts as treatment; adoption is reported next to every treatment number.
+
+## Layout
+
+```
+bench/
+  tasks/<domain>/<intent>.json    one intent per file; domain.json holds per-domain settings
+  schemas/                        JSON Schemas for answers
+  prompts/system.md               the system prompt both arms get
+  allowlist.txt                   the kernel MCP tools the agent gets, and no others
+  mcp.json                        the kernel MCP server; reads KERNEL_API_KEY
+  .env.example                    keys run.mjs loads from bench/.env
+  run.mjs  grade.mjs  report.mjs  lib/  test/
+  results/<run_id>/               gitignored: run.json, rows.jsonl, transcripts
+```
+
+## Running
+
+Needs the `kernel` and `pi` CLIs, Node 22.13+, git, `KERNEL_API_KEY` and the model provider's
+key (such as `OPENAI_API_KEY`), from the environment or `bench/.env`. Each agent gets its own
+`PI_CODING_AGENT_DIR`, so stored pi logins and MCP servers are not used.
+
+Run it as a Hypeship service under a hold that covers the whole run; holds cannot be renewed.
+This pilot is 105 trials and can take about 8 hours if every trial reaches its cap:
+
+```sh
+hypeship hold acquire --duration 12h
+hypeship services create bench-pilot-1 --cwd "$PWD" \
+  --command "node bench/run.mjs --run-id pilot-1 --model openai/gpt-5.4 --domain fandango.com --domain github.com --domain ipwho.is --adoption 5 --pairs 5 --slots 3"
+hypeship services start bench-pilot-1
+node bench/report.mjs bench/results/pilot-1 > pilot-1.md
+```
+
+| flag | meaning |
+|---|---|
+| `--run-id` | names `results/<run_id>/`; rerunning the same id resumes |
+| `--model`, `--thinking` | `provider/model` and thinking level |
+| `--domain`, `--intent` | select intents (repeatable); default is all |
+| `--pairs` | pairs per intent; default is the intent's `pairs` |
+| `--adoption` | treatment-only trials per domain before any pairs |
+| `--slots` | jobs run in parallel |
+| `--cap-sec` | per-trial time cap; default is the intent's `trial_timeout_sec` |
+| `--max-budget-usd` | per-trial spend cap, default 2 |
+| `--stealth` | stealth browsers for every trial |
+
+Once the smoke gate passes, a new run writes `run.json`: intents with fixtures resolved, schemas,
+tool commits, the MCP config, the pi and kernel CLI versions, and each pair's order (`AB` runs
+control first, `BA` treatment first, split evenly within each intent and shuffled). On resume
+`run.json` is authoritative, and the run refuses to continue if the allowlist, `mcp.json`, the
+system prompt or the pi version changed. A resume reruns aborted or missing treatment-only
+trials and every pair without a scored trial in both arms. One runner per run id.
+
+### Smoke gate
+
+Before any trial, stopping on the first failure:
+
+1. the hold covers the remaining jobs if every trial reaches its cap after 180s of setup (the
+   hold is checked again before each job, and no job starts that it cannot outlast);
+2. `KERNEL_API_KEY` is set, and on resume pi is the same version;
+3. a zero-tool pi run answers and reports a cost above zero (pi prices an unknown model at 0,
+   which would disable the spend cap);
+4. the tools pi declares to the model equal `allowlist.txt`;
+5. a treatment setup on the first pending intent works end to end;
+6. each signed-in domain passes its sign-in check.
+
+### Stages
+
+With `--adoption N`, each domain first gets N treatment-only trials, cycling through its non-gap
+intents. A domain where none of them invoked a tool, or all aborted, gets no pairs. Pairs follow,
+interleaved across intents; a pair's two trials run back to back in one slot.
+
+### A trial
+
+1. Create a browser on the start URL, tagged with the run, arm and intent. If the page is still
+   loading after 15s, navigate again; then wait up to 15s for `load`.
+2. Check the loaded URL matches the entry tool's URL patterns, in both arms.
+3. Treatment: register the batch, confirm the `entry_tool` is live on the tab, and on a signed-in
+   domain run the sign-in check. Control: on a signed-in domain, register the batch, run the same
+   check and remove the tools again; then fail if any custom tools are registered or live. Both
+   arms record the page's own WebMCP tools, which both can use.
+4. Wait until 10s after load in both arms, then launch the agent. The clock runs to
+   `agent_settled`; the runner stops the agent at the time, turn or spend cap.
+5. Grade the answer, delete the browser and append a row to `rows.jsonl`. Each attempt keeps its
+   own directory under `trials/<intent>/` (`p1-control`, then `p1-control-2` for a rerun), named in
+   the row's `trial_dir`.
+
+A setup failure gets up to three browsers before the trial is `aborted` (a signed-out profile
+gets one). A trial is also `aborted` when the agent started without exactly the allowlisted tools
+(pi waits only 10s for MCP servers), when pi gave up retrying a provider error, or on a runner
+error. An aborted trial reruns its whole pair once; a pair still incomplete is left out of both
+arms. The run stops after max(6, 2 × slots) aborts in a row. Ctrl-C or SIGTERM stops the agents
+and deletes every browser.
+
+## Intents
+
+```json
+{
+  "id": "fandango-theater-showtimes",
+  "domain": "fandango.com",
+  "stage": "inspect",
+  "prompt": "On fandango.com, list every film and every showtime at {{theater}} in {{city}} on {{date}}. Reply with JSON: ...",
+  "fixture": {"theater": "AMC 19th St. East 6", "city": "New York, NY", "date": {"days_from_start": 2}},
+  "start_url": "https://www.fandango.com/",
+  "tools": [{"repo": "kernel/xwebmcp", "ref": "hypeship/fandango-showtimes", "pr": 58, "names": ["search_theaters", "get_theater_showtimes"]}],
+  "entry_tool": "search_theaters",
+  "answer_schema": "schemas/theater-showtimes.json",
+  "rules": [{"path": "date", "equals": "{{date}}"}, {"path": "films[].title", "unique": true}],
+  "pairs": 5, "trial_timeout_sec": 600, "max_turns": 40
+}
+```
+
+- `stage` is `find`, `inspect`, `compare`, `prepare` or `gap`. A `gap` intent is one no tool
+  covers; it names no tools and serves as the placebo check.
+- `tools` lists kernel/xwebmcp refs (or full `sha`s) to register from. Treatment gets every tool
+  `registry.json` lists for the domain at those commits whose file declares
+  `readOnlyHint: true`; later sources replace earlier tools of the same name, so a PR to an
+  existing domain lists main first. Tool files are evaluated in a sandboxed child process.
+  `names` are the tools a treatment agent would use, and the run fails at start if any would not
+  be registered. A `pr` whose branch is gone resolves to `refs/pull/<pr>/head`, and the report
+  marks it.
+- `{{name}}` in `prompt` and `rules` comes from `fixture`. `{"days_from_start": n}` resolves to a
+  date when the run starts. A fixed date that has passed skips the intent, or stops the run if
+  the intent was selected by name; move dates tied to a film's run forward before each run.
+- `rules` apply to every value a path resolves to (`[]` steps into arrays): `equals` and `in`
+  (case- and whitespace-insensitive), `matches` (case-insensitive regex; fixture values match
+  literally), `range`, `count`, `time` (`HH:MM`), `date` (`YYYY-MM-DD`) and `unique`.
+- `agreement` (optional) compares the arms' answers within a pair. `{"path": "price", "better":
+  "lower", "tolerance": 1}` counts pairs where one arm missed the cheapest fare. `{"path":
+  "issues[]", "same": true}` is for one right answer that changes too often to pin in rules, and
+  counts both-correct pairs whose answers differ as sets. Leave it off intents with many right
+  answers.
+
+`tasks/<domain>/domain.json` holds per-domain settings:
+
+- `exclude`: read-only tools always left out, each with a reason;
+- `needs_sign_in`: read-only tools that need an account, left out unless the domain has a profile;
+- `profile` and `signed_in_check`: see below;
+- `instructions`: text added to every trial prompt on the domain, in both arms. The system prompt
+  forbids adding to a cart unless the task allows it, so a domain that needs carts says so here.
+  amctheatres.com uses it to name the step that creates a cart, since an AMC cart holds seats.
+
+### Signed-in trials
+
+A domain with a Kernel `profile` loads it in every browser, in both arms, without saving
+changes, and registers its `needs_sign_in` tools. Use a test account with no saved payment
+method, kept signed in by a managed auth connection:
+
+```sh
+kernel auth connections create --domain instacart.com --profile-name bench-instacart -o json
+kernel auth connections login <connection id>
+```
+
+`signed_in_check` names a read-only tool (and optional input) that returns an error when signed
+out. The smoke gate and every setup in both arms invoke it; a signed-out trial aborts. Control
+registers the batch for the check and removes it before the agent starts.
+Jobs on one profile run one at a time, but server-side state such as recent searches carries
+across trials and arms. A missing profile skips the domain, or stops the run if it was selected
+by name.
+
+## Grading
+
+1. Extract an answer from the final message: the last fenced JSON block, then a trailing object,
+   then `key: value` lines naming schema properties. Nothing extracted is `malformed`.
+2. A schema or rule failure is `wrong`; otherwise `correct`, which means plausible, not verified.
+
+A successful call outside the allowlist, a pi error or a missing cost is `crashed`; hitting a
+cap is `timeout`. `node bench/grade.mjs bench/results/<run_id>/trials/<intent>/<trial>` regrades
+one trial against its run's `run.json`.
+
+## Report
+
+`node bench/report.mjs bench/results/<run_id>` prints, per intent, over pairs with both trials
+scored:
+
+- the registered tools, drafts, and how answers are checked: `pinned` (a rule fixes a value),
+  `agreement`, or `shape only`;
+- per arm: correct with a Wilson interval, outcomes, fastest/median/slowest correct, median over
+  all, minutes per correct answer, and treatment tool use;
+- Δ correct with a Newcombe interval and an exact McNemar test over the pairs correct in one arm
+  only, and over both-correct pairs the speedup (ratio of medians) and an exact sign test; the
+  same over just the pairs whose treatment invoked a tool, when that is some but not all;
+- per arm, the median turns, input tokens and cost per trial;
+- a screening label: `no adoption`, `inconclusive`, `faster`, `slower`, `mixed` or `no signal`.
+  Thresholds are in `report.mjs`; at 5 pairs a label needs all 5 to go the same way. A faster or
+  slower arm is `mixed` when it was correct in 2 fewer pairs, gave the worse `better` answer in
+  2 more, or the arms' `same` answers differed in 2 pairs.
+
+A tool failures table lists, per tool, the treatment invocations that did not complete or whose
+tool returned `status: "error"`, with their most common error messages. Then the worst label per
+PR, and per run: spend over every attempt including smoke gates, the label histogram (noting
+labels from `shape only` intents), median speedup, each arm's agent-minutes, turns and spend
+over complete pairs, the pooled gap-intent
+placebo, an order check, a start delay check, a native tools check (control trials that started
+with the page's own WebMCP tools), cap calibration, and how many labels to expect
+from noise.
+
+## Not built yet
+
+Reference answers and spot checks, a no-agent baseline, telemetry-based outcome classes,
+scheduled runs, and the 30-pair verdict statistics.

bench/allowlist.txt

@@ -0,0 +1,4 @@
+mcp__kernel__browser_repl
+mcp__kernel__computer_action
+mcp__kernel__execute_playwright_code
+mcp__kernel__webmcp