Add retail-cart-providers: retail-cart on competing providers' browsers

.github/workflows/_run.yml

@@ -65,6 +65,12 @@ jobs:
       CLAWBENCH_JUDGE_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
       CLAWBENCH_JUDGE_MODEL: deepseek/deepseek-v4-pro
       CLAWBENCH_JUDGE_API_TYPE: openai-completions
+      # Other browser providers, for remote-browser arms.
+      BROWSERBASE_API_KEY: ${{ secrets.BROWSERBASE_API_KEY }}
+      BROWSERBASE_PROJECT_ID: ${{ secrets.BROWSERBASE_PROJECT_ID }}
+      STEEL_API_KEY: ${{ secrets.STEEL_API_KEY }}
+      HYPERBROWSER_API_KEY: ${{ secrets.HYPERBROWSER_API_KEY }}
+      ANCHOR_API_KEY: ${{ secrets.ANCHOR_API_KEY }}
       BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }}
       BRAINTRUST_PROJECT: ${{ vars.BRAINTRUST_PROJECT }}
     steps:

.github/workflows/ci.yml

@@ -26,6 +26,8 @@ jobs:
         run: uv run bench prepare kernel-mcp-clawbench --arms codex-kernel-mcp,claude-kernel-mcp --tasks v2-1134-chapter-finder-redcross --out "$RUNNER_TEMP/prepared"
       - name: Prepare the static retail-cart arm
         run: uv run bench prepare retail-cart --arms codex-repl --out "$RUNNER_TEMP/prepared-retail-cart"
+      - name: Prepare the retail-cart provider arms
+        run: uv run bench prepare retail-cart-providers --out "$RUNNER_TEMP/prepared-retail-cart-providers"
 
   report:
     runs-on: ubuntu-latest

.github/workflows/retail-cart-providers.yml

@@ -0,0 +1,45 @@
+name: Retail cart flows on each provider's browser
+
+on:
+  workflow_dispatch:
+    inputs:
+      arms:
+        description: Comma-separated arm names (empty runs every arm in benchmarks/retail-cart-providers.toml)
+        required: false
+        default: ""
+        type: string
+      tasks:
+        description: Comma-separated task IDs, or all
+        required: false
+        default: all
+        type: string
+      attempts:
+        description: Trials per task and arm
+        required: false
+        default: "10"
+        type: string
+      concurrency:
+        description: Concurrent trials per arm. Every arm's task containers run on the one runner.
+        required: false
+        default: "4"
+        type: string
+      playwright_mcp_version:
+        description: "@playwright/mcp version to install (empty installs the toolset's pinned version)"
+        required: false
+        default: ""
+        type: string
+
+permissions:
+  contents: read
+
+jobs:
+  benchmark:
+    uses: ./.github/workflows/_run.yml
+    secrets: inherit
+    with:
+      benchmark: retail-cart-providers
+      arms: ${{ inputs.arms }}
+      tasks: ${{ inputs.tasks }}
+      attempts: ${{ inputs.attempts }}
+      concurrency: ${{ inputs.concurrency }}
+      toolset_ref: ${{ inputs.playwright_mcp_version }}

README.md

@@ -91,6 +91,7 @@ A run of two arms on two tasks (Red Cross and Indeed) produced this summary:
 | `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` | Codex and Claude Code arms; the `site-config` configurator's model |
 | `PURELY_MAIL_API_KEY`, `PURELY_MAIL_DOMAIN` | ClawBench creates a disposable [PurelyMail](https://purelymail.com) mailbox on your domain for every task, so agents can sign up and receive confirmation emails. |
 | `CLAWBENCH_JUDGE_BASE_URL`, `CLAWBENCH_JUDGE_API_KEY`, `CLAWBENCH_JUDGE_MODEL`, `CLAWBENCH_JUDGE_API_TYPE` | ClawBench's LLM judge, which checks the intercepted request against the task. Any OpenAI-compatible endpoint works with `CLAWBENCH_JUDGE_API_TYPE=openai-completions`; we use `deepseek/deepseek-v4-pro` through OpenRouter. See [ClawBench](https://github.com/TIGER-AI-Lab/ClawBench) for details. |
+| `BROWSERBASE_API_KEY`, `BROWSERBASE_PROJECT_ID`, `STEEL_API_KEY`, `HYPERBROWSER_API_KEY`, `ANCHOR_API_KEY` | Arms on those providers' browsers (`environment = "remote-browser"`); checked before a run that includes them |
 | `BRAINTRUST_API_KEY`, `BRAINTRUST_PROJECT` | Optional: publishing results |
 
 ## Running in GitHub Actions
@@ -139,6 +140,26 @@ Run several attempts per task to get a pass rate per site:
 uv run bench run retail-cart --attempts 10
 ```
 
+### `retail-cart-providers`
+
+The `retail-cart` tasks on each browser provider's browser. Every arm uses the same agent (Codex, `gpt-6-luna`), the same tools ([Playwright MCP](https://github.com/microsoft/playwright-mcp) attached over CDP) and the same task container on the runner's Docker, so arms differ only in the browser. Each provider gets its own anti-bot settings:
+
+| Arm | Browser |
+|---|---|
+| `codex-pw-kernel` | Kernel, `stealth: true` |
+| `codex-pw-browserbase` | Browserbase, Advanced Stealth (Scale plan), CAPTCHA solving, its residential proxies |
+| `codex-pw-steel` | Steel, residential proxy and CAPTCHA solving |
+| `codex-pw-hyperbrowser` | Hyperbrowser, stealth, proxy and CAPTCHA solving |
+| `codex-pw-anchor` | Anchor, its US proxy, extra stealth and CAPTCHA solving |
+
+The start page, the agent and the verifier each connect to the browser separately, so sessions are created to outlive a CDP disconnect (Browserbase's `keepAlive`, Anchor's idle timeout). Scoring is the same as `retail-cart`.
+
+```bash
+uv run bench run retail-cart-providers --attempts 10 --concurrency 4
+```
+
+All arms' task containers run on one machine, so keep `--concurrency` low on a GitHub runner.
+
 ## Extending
 
 ### Add an arm
@@ -162,6 +183,8 @@ steering = true
 
 `toolset_args` are passed to the toolset; `kernel-mcp-local` takes `toolsets`, the Kernel MCP toolsets to enable: `playwright` (the default), `repl`, `computer`, or several. `agent_kwargs` are passed to the Harbor agent.
 
+An arm with `environment = "remote-browser"` runs the task container on local Docker instead of in the Kernel browser's VM, and its `config` names a `provider` from `harness/providers.py` and that provider's session create `params`. The environment writes the browser's CDP URL to `/run/browser/cdp-url` in the container; the `playwright-mcp` toolset connects to it.
+
 `config` takes any [`POST /browsers`](https://kernel.sh/docs/api-reference/browsers/create-a-browser-session) parameter, such as `stealth`, `proxy`, `region`, `extensions`, or `viewport`. Browsers are always headful.
 
 ### Add a configurator

arms/codex-pw-anchor.toml

@@ -0,0 +1,24 @@
+# Codex driving an Anchor browser through Playwright MCP over CDP, with
+# Anchor's anti-bot settings: its US proxy, extra stealth and CAPTCHA solving
+# (both need the proxy).
+agent = "codex"
+model = "gpt-6-luna"
+agent_version = "0.163.0"
+toolset = "playwright-mcp"
+environment = "remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "anchor"
+
+[browser.config.params.session]
+proxy = { active = true, type = "anchor_proxy", country_code = "us" }
+# Minutes. The idle timeout covers the gaps between the start page, agent and
+# verifier connections.
+timeout = { max_duration = 45, idle_timeout = 30 }
+
+[browser.config.params.browser]
+extra_stealth = { active = true }
+captcha_solver = { active = true }

arms/codex-pw-browserbase.toml

@@ -0,0 +1,21 @@
+# Codex driving a Browserbase browser through Playwright MCP over CDP, with
+# Browserbase's anti-bot settings: Advanced Stealth (Scale plan), CAPTCHA
+# solving and its residential proxies. keepAlive keeps the session across the
+# start page, agent and verifier connections.
+agent = "codex"
+model = "gpt-6-luna"
+agent_version = "0.163.0"
+toolset = "playwright-mcp"
+environment = "remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "browserbase"
+
+[browser.config.params]
+keepAlive = true
+timeout = 2700
+proxies = true
+browserSettings = { advancedStealth = true, solveCaptchas = true }

arms/codex-pw-hyperbrowser.toml

@@ -0,0 +1,14 @@
+# Codex driving a Hyperbrowser browser through Playwright MCP over CDP, with
+# Hyperbrowser's anti-bot settings: stealth, proxy and CAPTCHA solving.
+agent = "codex"
+model = "gpt-6-luna"
+agent_version = "0.163.0"
+toolset = "playwright-mcp"
+environment = "remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "hyperbrowser"
+params = { useStealth = true, useProxy = true, solveCaptchas = true, timeoutMinutes = 45 }

arms/codex-pw-kernel.toml

@@ -0,0 +1,14 @@
+# Codex driving a Kernel stealth browser through Playwright MCP over CDP, the
+# same agent, tools and container as the other providers' codex-pw-* arms.
+agent = "codex"
+model = "gpt-6-luna"
+agent_version = "0.163.0"
+toolset = "playwright-mcp"
+environment = "remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "kernel"
+params = { stealth = true, timeout_seconds = 1800 }

arms/codex-pw-steel.toml

@@ -0,0 +1,15 @@
+# Codex driving a Steel browser through Playwright MCP over CDP, with Steel's
+# anti-bot settings: residential proxy and CAPTCHA solving.
+agent = "codex"
+model = "gpt-6-luna"
+agent_version = "0.163.0"
+toolset = "playwright-mcp"
+environment = "remote-browser"
+
+[agent_kwargs]
+disable_web_search = true
+
+[browser.config]
+provider = "steel"
+# timeout is in milliseconds.
+params = { useProxy = true, solveCaptcha = true, timeout = 2700000 }

benchmarks/retail-cart-providers.toml

@@ -0,0 +1,10 @@
+# The retail-cart tasks on each provider's browser, with the same agent,
+# Playwright MCP tools and task container, so arms differ only in the browser.
+suite = "retail-cart"
+arms = [
+  "codex-pw-kernel",
+  "codex-pw-browserbase",
+  "codex-pw-steel",
+  "codex-pw-hyperbrowser",
+  "codex-pw-anchor",
+]

harness/definitions.py

@@ -11,6 +11,11 @@
 from typing import Any
 
 ROOT = Path(__file__).resolve().parents[1]
+# Harbor environment each arm `environment` runs its trials in.
+ENVIRONMENTS = {
+    "kernel": "harness.environment:KernelTaskEnvironment",
+    "remote-browser": "harness.environment:RemoteBrowserEnvironment",
+}
 
 
 @dataclass(frozen=True)
@@ -26,6 +31,9 @@ class Arm:
     agent: str
     model: str
     toolset: str
+    # "kernel" runs the task container in the Kernel browser's VM;
+    # "remote-browser" runs it on local Docker and drives the browser over CDP.
+    environment: str = "kernel"
     agent_version: str | None = None
     agent_kwargs: dict[str, Any] = field(default_factory=dict)
     browser: Browser = field(default_factory=Browser)
@@ -59,7 +67,10 @@ def load_benchmark(name: str) -> Benchmark:
 def load_arm(name: str) -> Arm:
     data = _read(ROOT / "arms" / f"{name}.toml")
     browser = data.pop("browser", {})
-    return Arm(name=name, browser=Browser(**browser), **data)
+    arm = Arm(name=name, browser=Browser(**browser), **data)
+    if arm.environment not in ENVIRONMENTS:
+        raise SystemExit(f"Arm {name} has unknown environment {arm.environment!r}")
+    return arm
 
 
 def load_suite(name: str) -> ModuleType:

harness/environment.py

@@ -20,6 +20,11 @@
 the persona or WebGL renderer they ask for. `vm_setup` is a list of shell
 commands then run as root. A trial where either fails fails with
 `VmSetupError`.
+
+`RemoteBrowserEnvironment` instead runs the task container on local Docker
+and drives a browser from any provider in `harness.providers` over CDP, so
+providers are compared with the same agent, tools and container. Its config
+file names the `provider` and that provider's session create `params`.
 """
 
 from __future__ import annotations
@@ -33,14 +38,20 @@
 from pathlib import Path
 from typing import Any
 
+from harbor.environments.docker.docker import DockerEnvironment
 from harbor.environments.kernel import KernelEnvironment
 from kernel import AsyncKernel, Kernel
 
+from harness import providers
+
 CONFIG_FILE = "kernel-config.json"
 PROXY_CREATE = "proxy_create"
 CHROMIUM_ENV = "chromium_env"
 VM_SETUP = "vm_setup"
 RUN_ID_ENV = "BENCH_RUN_ID"
+# Where the task container finds the remote browser's CDP URL. Agents may not
+# pass their environment on to the MCP servers they launch, so it is a file.
+CDP_URL_FILE = "/run/browser/cdp-url"
 
 
 class ProxyCreationError(RuntimeError):
@@ -51,6 +62,10 @@ class VmSetupError(RuntimeError):
     """The browser's VM could not be set up for the trial before the first page load."""
 
 
+class BrowserCreationError(RuntimeError):
+    """The provider could not create the trial's remote browser."""
+
+
 SUPERVISOR_PROGRAM = """[program:chromium]
 command=/usr/bin/env {env} /usr/local/bin/chromium-launcher
 autostart=false
@@ -210,3 +225,49 @@ async def stop(self, delete: bool) -> None:
                     self._proxy_id = None
                 except Exception as error:
                     self.logger.error(f"Failed to delete Kernel proxy {self._proxy_id}: {error}")
+
+
+class RemoteBrowserEnvironment(DockerEnvironment):
+    def __init__(self, environment_dir: Path, *args: Any, **kwargs: Any) -> None:
+        config = json.loads((Path(environment_dir) / CONFIG_FILE).read_text())
+        self._provider_name: str = config["provider"]
+        self._provider = providers.get(self._provider_name)
+        self._create_params: dict[str, Any] = config.get("params") or {}
+        self._browser_session_id: str | None = None
+        super().__init__(environment_dir, *args, **kwargs)
+
+    async def start(self, force_build: bool) -> None:
+        await super().start(force_build)
+        try:
+            session = await self._provider.create(self._create_params)
+        except Exception as error:
+            raise BrowserCreationError(
+                f"Could not create a {self._provider_name} browser: {error}"
+            ) from error
+        self._browser_session_id = session.id
+        self.logger.info(f"Created {self._provider_name} browser {session.id}")
+        self._persistent_env.update(
+            {"BROWSER_PROVIDER": self._provider_name, "BROWSER_SESSION_ID": session.id}
+        )
+        directory = os.path.dirname(CDP_URL_FILE)
+        result = await self.exec(
+            f'mkdir -p {directory} && printf %s "$CDP_URL" > {CDP_URL_FILE} && chmod 0644 {CDP_URL_FILE}',
+            env={"CDP_URL": session.cdp_url},
+            user="root",
+        )
+        if result.return_code != 0:
+            raise BrowserCreationError(f"Could not write {CDP_URL_FILE}: {result.stderr}")
+
+    async def stop(self, delete: bool) -> None:
+        try:
+            await super().stop(delete)
+        finally:
+            if self._browser_session_id is not None:
+                try:
+                    await self._provider.release(self._browser_session_id)
+                    self._browser_session_id = None
+                except Exception as error:
+                    self.logger.error(
+                        f"Failed to release {self._provider_name} browser "
+                        f"{self._browser_session_id}: {error}"
+                    )