[yummybomb]

Summary

Adds bench/, a harness that measures, per user intent on a site, how much more often and how much faster the same agent succeeds when the domain's read-only xwebmcp tools are registered in its Kernel browser. This is the pilot phase: the runner, the smoke gate, SHA-pinned registration, the agent adapter, grading by shape and rules, the report with labels and rollups, 9 pilot intents across ipwho.is (#69), fandango.com (#58) and github.com (#55), and 17 screening intents for the domains on main. doordash.com has no read-only tools and every purelymail.com tool needs a signed-in mailbox, so neither has an intent. instacart.com's catalog tools need a signed-in session, so its intent runs on a Kernel profile (see Signed-in trials).

Each trial runs pi (pi -p --mode json) against one Kernel browser. It works with any provider pi supports, for example --model openai/gpt-5.4 with OPENAI_API_KEY, or anthropic/... with ANTHROPIC_API_KEY. Both arms get the same model, prompt, kernel MCP server, tool allowlist and browser settings. Treatment additionally registers the domain's read-only tools before the clock starts. Both arms' system prompt tells the agent to list the page's WebMCP tools first and fall back to Playwright or computer actions when none fits, as Kernel's docs tell customers to instruct their agents; it never mentions xwebmcp or whether tools are registered.

• run.mjs: resolves each intent's tool refs to commits in a bare cache repo, so the working clone is never touched. It composes the batch in xwebmcp attach's format, keeping the tools with readOnlyHint: true minus per-domain excludes and, once the smoke gate passes, freezes everything in run.json, including the pi version and each pair's order, split evenly between AB and BA within each intent and shuffled.

  • Then it runs optional treatment-only adoption trials per domain, interleaved across domains so one broken domain cannot hit the abort limit alone, and pairs interleaved across intents over N slots. A domain where no adoption trial invoked a tool, or where every one aborted, gets no pairs.
  • Treatment trials abort if the intent's entry_tool is not live on the start tab. A pair with an aborted trial runs again whole, both trials in order, so its trials stay consecutive.
  • Rows go to rows.jsonl. Rerunning the same --run-id resumes, rerunning any pair that lacks a scored trial in either arm. One runner per run id.
  • Meant to run as a hypeship services service under a hold. The hold is sized per stage and checked again before each job.

• lib/agent.mjs runs pi isolated: --no-session --no-context-files --no-skills --no-prompt-templates --no-approve --no-extensions -e builtin:mcp --no-builtin-tools, the bench system prompt, and --tools set to allowlist.txt.

  • Each agent gets its own PI_CODING_AGENT_DIR holding only the MCP config frozen in run.json (the kernel server, exposure: direct), so the user's MCP servers, settings and stored logins never apply.
  • The runner enforces the time cap, the turn cap (by counting turn_start) and the spend cap (from the cost pi reports). pi has no flags for the last two.

• Smoke gate: checks KERNEL_API_KEY, the pi version on resume, that a zero-tool pi run answers and reports a cost above zero (which proves the model and its key, and that the spend cap can work; pi prices a model it has no prices for at 0), that the tools pi declares to the model exactly equal allowlist.txt, one treatment setup end to end on the first pending intent, and that the hold covers the remaining jobs.

• grade.mjs: extracts an answer leniently (fenced block, then trailing object, then key: value prose). Nothing extractable is malformed; a schema failure or an intent rule failure is wrong. A successful call outside the allowlist, a final message pi reports as an error, or a cost pi leaves out or reports as something other than a number is crashed, except that a provider error pi retried and gave up on, or an error in the runner itself, is aborted and the pair runs again; hitting the time, turn or spend cap is a timeout. An agent that started without exactly the allowlisted tools is aborted and retried, because pi waits only 10s for MCP servers.

• report.mjs: per intent, the registered tools and drafts; per arm, over pairs where both trials were scored, n and correct with a Wilson interval, outcome counts, speed over correct trials, median_all, minutes per correct answer, and treatment tool use. A Δ row gives a Newcombe interval, and over both-correct pairs the speedup (ratio of medians) and an exact sign test, plus a screening label. Cheapest-fare intents carry an agreement check: when both arms are correct but their fares differ by more than $1, the higher one missed the cheapest fare, and a faster arm that did so in 2 more pairs than the other is mixed. Intents with one right answer that changes too often to pin (recent bug issues with linked PRs, AMC and Fandango showtimes) carry agreement: {same: true}: both-correct pairs whose answers differ as sets are counted, and 2 of them make a faster or slower label mixed. A pair with an aborted or missing trial is left out of both arms and counted, with its aborts per arm, and so are correct answers given without a single tool call. It also prints per-PR rollups (worst label) and per-run rollups: spend over every attempt, an order check (control-first vs treatment-first pairs), label histogram, median speedup, the pooled gap-intent placebo check, a start delay check (pairs whose agents launched more than 1s apart after page load), cap calibration and the expected number of noise labels.

• Tool failures: each WebMCP invocation records the tool's own status and an error message, so a tool that returned {"status": "error"} no longer counts toward invoke ok, and a call the MCP server rejected, such as one with a stale tool_ref, is an error instead of unknown. The report adds a table per tool: calls, failures and the most common errors, and says how many invocations made through Playwright or the REPL have no recorded result.

node --test "bench/**/*.test.mjs" is added to validate.yml. These tests are unit-only; trials never run in CI.

Choices worth a look

• Cart policy: agents may not check out, pay, place an order or complete a booking, and may not add to a cart unless the task allows it; a domain whose intents need a cart opts in through its instructions, and no current intent does. Carts on live sites can hold real inventory: an AMC cart holds seats against other buyers, and AMC rate-limits the ticket flow after a few carts, so amctheatres.com/domain.json also names the step that creates one (Continue on the ticket page). Selecting seats and ticket types to see their prices is allowed, so AMC's read-only preview_tickets is registered. Registration is still limited to readOnlyHint: true tools.

• Allowlist: browser_repl, computer_action, execute_playwright_code and webmcp. pi's --tools drops every other kernel MCP tool, including any the server adds later, so there is no denylist. Which tools count as "driving the browser" is still open.

• Stealth is off by default (--stealth applies it to every browser in a run). With stealth on, github.com and github.com/explore often stayed in readyState: "loading"; without it, every page I tried loaded in 0–2s. The three pilot domains are low bot pressure.

• Both arms get the same setup. The loaded page must match the entry tool's URL patterns in both arms, and both launch the agent 10s after the page loaded (start_delay_ms is recorded per row), so registration does not give the treatment's page longer to settle.

• Setup handles stalled start pages. It waits up to 15s for the start page to be parsed, then navigates to the start URL once more. In stealth browsers the first navigation often stalls with the document still loading (49 of 149 trials in the main run needed the second navigation, and it parsed in 1–4s). After that it gives load up to 15s, and a failed setup gets up to three fresh browsers before the trial is aborted. GitHub intents start on /explore.

• The github prepare_* tools stay registered. They are read-only; they just have no intents.

• Excluded tools. A domain's domain.json lists exclude (always left out, such as a tool that does what the system prompt forbids; none currently) and needs_sign_in (reads that need a signed-in account, left out unless the domain signs in: amc, delta, etsy, github, kroger, resy, ubereats). Every name must exist.

• Signed-in trials. A domain can name a Kernel profile and a signed_in_check tool. Jobs on one profile run one at a time, so two trials never use the account at once, and the hold estimate counts them back to back. Both arms load the profile without saving changes, its needs_sign_in tools are registered, and the smoke gate invokes the check tool and stops the run if it reports an error such as not_logged_in. Every treatment setup invokes it again, since a session can lapse mid-run; a signed-out trial aborts without trying more browsers. A missing profile skips the domain, or stops the run when the domain was selected by name. instacart.com uses bench-instacart, which does not exist yet; the README shows how to create it with a managed auth connection on a test account.

• .env: bench/.env.example lists the keys; the runner loads bench/.env when present, and variables already set win.

• Tool files are evaluated in an empty vm context (no fetch, process or import) inside a child process with an empty environment and Node's permission model (no file system or child processes), and sources must come from kernel/xwebmcp. The definitions are reshaped and serialized inside the context, so only a JSON string comes back: handing the context any runner function, such as a .map callback, would give a tool file the runner's process. A vm context is not a security boundary, and the permission model denies the network only from Node 25, so on Node 22 a tool file that escaped could open connections but would find no keys or files. URL patterns are matched with the extension's own matcher.

• Fixtures were picked from live data. The fandango dates need moving forward before the pilot. There are no intents for Fandango's get_ticket_offers or prepare_checkout yet: a reserved-seat showtime shows no ticket prices until seats are selected. The system prompt now allows that, so they can be added once checked on live showtimes.

• Reading webmcp results: pi shortens tool results over 20 KB for the model, so adoption parsing reads the full text from the result's structuredContent. Tool names are taken from any listing an agent returns, including reshaped webmcp.listTools() output from Playwright or the REPL.

• Fixtures: {"days_from_start": n} fixture values resolve to dates at plan time and are frozen in run.json; the flight and Ticketmaster intents use them. The film intents keep fixed dates; once one has passed, a new run skips the intent unless it was selected by name, and a resume skips its remaining trials. A source with a pr whose branch was deleted resolves to refs/pull/<pr>/head; the runner logs it and the report marks the source (branch gone, PR head), since that is the PR's last head rather than main. Stable facts (ipwho 8.8.8.8, the merged Playwright PR, Playwright's branch and license, and the headless-chrome topic's top three repositories) are pinned in rules; the AMC answer must name the film, the release tag must look like v1.2.3, and fares are checked from $1. ipwho uses an IP the model cannot recall (41.203.18.200, ZA / AS37153), Playwright's star and release counts have floors at roughly today's values (95,000 and 160), and github-bug-issues-with-prs lists the 10 issues it judged, which must be distinct and numbered above 36,000.

• Robustness:

  • signals and fatal errors share one shutdown path, which kills agents, starts no new trials, and deletes browsers, including ones still being created; a second signal exits at once;
  • a browser whose delete fails stays tracked and is retried;
  • partially written rows are skipped;
  • the run stops once max(6, 2 × slots) trials in a row abort, since an outage or a runner bug would abort every one;
  • control setup fails if the browser already has custom tools;
  • flags, including --run-id and --thinking, are validated;
  • the hold check assumes every trial can run to its cap after a 180s setup;
  • each run fetches tool sources into its own cache, since concurrent runs would otherwise share FETCH_HEAD.

• Later review changes: the report labels each intent's answers pinned, agreement or shape only and names the faster or slower labels that come from shape-only intents; when treatment used the tools in some pairs but not all, it also gives the sign test and speedup over just those pairs, noted as not randomized. runQueue moved to lib/schedule.mjs with tests, and releases a job's profile even when the job throws. normalize now matches toolsource.Normalize when whitespace sits between trailing semicolons. run.json records the kernel CLI version.

Not in this PR

Reference answers and spot checks, the no-agent baseline, the telemetry-based outcome classes (blocked, busy, outcome_unknown, mutation_attempted, tampered), scheduled runs, and the 30-pair verdict statistics.

Testing

• node --test "bench/**/*.test.mjs": 70 pass on Node 26 in about 25s (the stub run tests set BENCH_START_DELAY_MS to 1s instead of waiting the real 10s) (the 59 before the tool failures change also passed on Node 22). These cover the plan's worked example (speedup 2.6×, Δ +20 pp [−16, +51], p = 0.016, faster), the label rules, pair-level scoring, answer extraction, rules, pi transcript parsing (including shortened results and tool redefinition), the outcome order, the hold estimate, the adoption gate, isolated tool evaluation (no fetch, process or import), composition order, job planning on resume (pending pairs, scored adoption trials, stale fixtures), pair retries (a complete pair always comes from one attempt), the paired speedup, fare agreement, the PR-ref fallback only on a missing branch, and failure capture when the call, the tool or the MCP server failed.

• The in-context evaluator gives the same output as the previous one for all 471 tool files on main, Add GitHub reads, content preparation and guarded stars #55, Add Fandango discovery, seat previews and ticket plans #58 and Add ipwho.is lookup_ip page tool #69, on Node 22 and 26, and a tool file's custom .map or .then can no longer reach the runner's process (a regression test covers both).

• run.mjs against stub pi, kernel and hypeship commands: with every browser create failing after the smoke gate, 5 pairs stopped after 6 aborts (before this change all 10 trials aborted and the run reported done), and a resume finished all 10 trials; a control browser that already has custom tools aborts its trial with that reason.

• With the vm evaluator, all 26 intents register the same tools and URL patterns as before, and every tool file on main and the Add GitHub reads, content preparation and guarded stars #55, Add Fandango discovery, seat previews and ticket plans #58 and Add ipwho.is lookup_ip page tool #69 branches (471) evaluates. A run with no hold resolved every intent and stopped at the hold check without starting the smoke gate; an unknown --intent stops with a one-line error.

• node scripts/validate-registry.mjs, node --test "tests/*.test.mjs" and the extension tests still pass.

• Stale fixtures and deleted branches, checked without trials: a deleted PR branch resolved to its refs/pull head; with one fixture date moved into the past, a run naming its domain stopped, a default run skipped it and planned the rest, and a resume skipped its pending pairs.

• A real run with pi and openai/gpt-5.4-mini, one pair each of ipwho-country-asn, github-repo-details and fandango-theater-showtimes:

  • The smoke gate passed, and all 6 trials graded correct, at about $0.25 total.
  • Only the fandango treatment agent used the tools: it listed them and invoked search_theaters (completed, 584 ms), then finished with Playwright, in 2:19 against control's 2:53.
  • The ipwho and github treatment agents never listed the tools, which is the case the pilot's adoption stage is designed to catch.
  • Resume skipped the finished trials, and no browsers were left behind.

• Earlier, all 22 trial setups (11 intents × 2 arms) passed browser creation, load, registration and the entry-tool check.

• A screening run on main (29da25c): 17 intents × 5 pairs, pi with openai/gpt-6-luna at medium thinking, stealth browsers, 8 slots.

  • All 170 trials were scored, with no aborts after the setup change, at $1.01 in total.
  • Control was correct in 78 of 85 trials and treatment in 80 of 85.
  • Treatment used 76 agent-minutes against control's 158, and 754 turns against 1,532.
  • Labels: 6 faster (aa, alaskaair, amctheatres, jetblue, southwest, united), 1 slower (resy), 1 no adoption (ubereats), 3 no signal, 6 inconclusive. Median speedup was 1.5×.
  • That run predates the excludes above and included the Instacart intent, whose treatment tools all returned not_logged_in.

• Runs after the latest fixes, pi with openai/gpt-6-luna, stealth browsers:

  • ipwho, AMC and Resy with 1 adoption trial and 1 pair each: all 9 correct, the smoke gate's treatment setup passed, AMC answers named the film, a second runner on the same id was refused, and a resume with nothing left skipped the smoke gate.
  • SIGTERM right after a pair's first trial: exit 143, the browser being created was deleted, and the resume reran that pair whole.
  • With Resy's entry tool broken in the frozen plan: its adoption trials aborted on both attempts and the domain was stopped before any control trial; in a run without adoption, the aborted pair reran once and the report left it out of both arms.
  • SIGINT during the smoke gate: exit 130 and no run.json. A wrong --model failed the smoke gate with the provider's error and left no run.json.
  • Signed-in trials: a missing bench-instacart profile is skipped in a default run and stops a run that names the intent; an empty throwaway profile on Instacart failed the smoke gate with not_logged_in; a throwaway profile on ipwho (whose lookup works signed out) passed the check, and both arms of two pairs ran on it, two browsers at a time, in orders BA and AB.
  • The changed GitHub and ipwho intents graded correct in both arms on live data.
  • No browsers were left behind by any of these runs.

• Stub runs after the latest changes (stub kernel, pi and hypeship):

  • With aa.com, github.com and ipwho.is, --adoption 3, one slot, and every github.com browser create failing: before, 6 github adoption aborts in a row stopped the whole run before ipwho.is ran. Now the adoption trials interleave, the run finished, and github.com was stopped by its own adoption verdict.
  • On instacart.com, with the sign-in check failing after two checks: pair 2's treatment aborted on its first browser with signed out: get_delivery_address returned not_logged_in, ran again once, and was left out; the report shows aborted: control 0, treatment 1.
  • With AMC, planning registers 12 tools, including preview_tickets.

• After review (stub pi, no trials):

  • With a stub pi, an assistant message without a cost, or with a null one, stops the agent as no cost reported; a normal cost does not, and $5 trips the spend cap.
  • runQueue's loop, driven with fake jobs over 3 slots, never ran two jobs on one profile at once, while other jobs filled the free slots.
  • The tool-file evaluator, after wrapping its long line, still evaluates all 215 tool files in registry.json to one named definition each.

Review fixes

• Control checks sign-in too. On a signed-in domain, control setup now registers the batch, runs the same signed_in_check as treatment and removes the tools before its agent starts, so a lapsed session aborts both arms instead of only failing control trials. Checked live: kernel browsers webmcp custom-tools remove clears both the registration and the page's live tools.
• run.mjs has an end-to-end test (bench/test/run.test.mjs) against stub pi, kernel and hypeship commands and a local repository standing in for kernel/xwebmcp: a signed-in pair checks sign-in in both arms and a resume has nothing to run; a signed-out control aborts its trial. Removing the control check or the tool removal fails it.
• Report: each intent gives an exact McNemar test over the pairs correct in one arm only, and each arm's median turns, input tokens and cost per trial; the run gives each arm's agent-minutes, turns and spend over complete pairs.

Later fixes

• Native tools check. Both arms record the start page's own WebMCP tools (native_tools), control fails setup if a custom tool is live (not only registered), and the report names control trials that started with the site's tools, since there control is not tool-free.
• Each attempt keeps its trial directory. A rerun wrote over trials/<intent>/<pair>-<arm> and deleted the transcript of the attempt it replaced; reruns now go to -2, -3 and so on, and each row records its trial_dir.
• fandango-theater-showtimes dates itself 2 days from the run's start instead of a fixed 2026-10-13. The two intents tied to a film's run keep fixed dates.
• Rebased onto main (delta.com: fix list_seat_map for current seat map markup #114, delta.com: fix search_flights for the current booking form #119).
• Tool failures reporting moved here from bench: checkout intents and per-tool failure reporting #115, which is closed: it corrects this PR's own invoke ok.