feat(evals): tool-selection eval for description routing regressions

.claude/hillclimb/tool-selection-openai/_state.json

@@ -0,0 +1,41 @@
+{
+  "flow": "tool-selection-openai",
+  "metrics": [
+    {
+      "id": "pass",
+      "label": "Pass",
+      "kind": "binary"
+    },
+    {
+      "id": "first_ok",
+      "label": "First tool ok",
+      "kind": "binary"
+    },
+    {
+      "id": "no_forbidden",
+      "label": "No forbidden",
+      "kind": "binary"
+    },
+    {
+      "id": "ref_ok",
+      "label": "Ref passed",
+      "kind": "binary"
+    },
+    {
+      "id": "no_token_ask",
+      "label": "No token ask",
+      "kind": "binary"
+    }
+  ],
+  "perf_fields": [
+    "tool_calls",
+    "tool_searches",
+    "latency_s",
+    "usage"
+  ],
+  "harness_paths": [
+    "evals/tool-selection/cases.json",
+    "evals/tool-selection/harness.ts"
+  ],
+  "harness_sha": "831b60ff5233625f5c7e02bc9a66498463a2ae1fd84231a879cff69b715a05e6"
+}

.claude/hillclimb/tool-selection/_state.json

@@ -0,0 +1,41 @@
+{
+  "flow": "tool-selection",
+  "metrics": [
+    {
+      "id": "pass",
+      "label": "Pass",
+      "kind": "binary"
+    },
+    {
+      "id": "first_ok",
+      "label": "First tool ok",
+      "kind": "binary"
+    },
+    {
+      "id": "no_forbidden",
+      "label": "No forbidden",
+      "kind": "binary"
+    },
+    {
+      "id": "ref_ok",
+      "label": "Ref passed",
+      "kind": "binary"
+    },
+    {
+      "id": "no_token_ask",
+      "label": "No token ask",
+      "kind": "binary"
+    }
+  ],
+  "perf_fields": [
+    "tool_calls",
+    "tool_searches",
+    "latency_s",
+    "usage"
+  ],
+  "harness_paths": [
+    "evals/tool-selection/cases.json",
+    "evals/tool-selection/harness.ts"
+  ],
+  "harness_sha": "831b60ff5233625f5c7e02bc9a66498463a2ae1fd84231a879cff69b715a05e6"
+}

.gitignore

@@ -155,3 +155,8 @@ test-results.xml
 # Build artifacts
 *.tsbuildinfo
 PR_DESCRIPTION.md
+
+# Tool-selection eval output (state and case files are committed)
+.claude/hillclimb/*/*/
+.claude/hillclimb/*/report.html
+.claude/hillclimb/*/trajectory/

CHANGELOG.md

@@ -1,5 +1,9 @@
 ## Unreleased
 
+### Other Features
+
+- **Tool-selection eval for catching routing regressions from tool descriptions (#274).** `npm run eval:tools` runs 19 prompts against Claude (Opus 5.5 by default) and grades which tools it calls against the expected ones: map only when asked, `static_map_image_tool` only for image files, `ground_location_tool` vs `reverse_geocode_tool`, brand vs category search, and whether web search stays available for place questions. The tool list and server instructions come from a real `McpServer` built the way the hosted deployment builds it and read back over MCP. Each prompt runs with all descriptions loaded and again with the tools behind the API's tool search, which approximates hosts like Claude Desktop that load tools on demand. Tool results are canned per case, so Mapbox APIs are never called. A grader check also confirms that every `render_map_tool` call passes the map ref unchanged, with a distinct ref per tool result so an invented or hand-built map is caught. Two site-building cases give the agent a `write_file` host tool and check that it neither asks the user for a Mapbox token nor leaves a placeholder token in the page; they expect `export_map_html_tool` (#275) and fail on servers without it. `EVAL_SERVER_ROOT` points the harness at another checkout, so two branches can be compared, and `npm run eval:tools:report` builds an HTML report with per-case transcripts. It calls the Claude API and costs about $8 per 114-row run, so it is not part of `npm test`. `EVAL_PROVIDER=openai` (`npm run eval:tools:openai -- --model <id>`) runs the same cases against an OpenAI model through the Responses API, including OpenAI's server-side tool search for the deferred mode, so description changes can be checked for ChatGPT-style regressions too. Adds `@anthropic-ai/sdk`, `openai` and `tsx` as dev dependencies. See `evals/tool-selection/README.md`.
+
 ## 0.15.1 - 2026-10-09
 
 ### New Features

evals/tool-selection/README.md

@@ -0,0 +1,87 @@
+# Tool-selection eval
+
+Measures which tools Claude picks for a prompt, given this server's tool descriptions and server instructions. Use it to catch routing regressions when you change a description, the server instructions, or the render hint that data tools append to their results.
+
+It is not part of `npm test`: it calls the Claude API and costs money. Mapbox APIs are never called, since every tool result is a canned stub.
+
+## How it works
+
+- **Tool surface:** `harness.ts` builds a real `McpServer` the same way the hosted deployment does (`publishJsonSchema2020`, `installTo`, `buildServerInstructions`) and reads `tools/list` and the server instructions back over an in-memory MCP connection.
+- **Host simulation:** two host tools sit next to the Mapbox tools. `web_search` and `places_map_display_v0` let us see whether our descriptions crowd out other sources. A case can add `write_file` through `host_tools`, for agents building a page or file.
+- **Two modes per case:**
+  - `eager`: every tool description is loaded up front.
+  - `deferred`: Mapbox tools are marked `defer_loading` behind the API's tool search. This approximates hosts like Claude Desktop that only load tools on demand, where the server instructions do most of the routing.
+- **Conversation:** up to 5 model calls per case. Tool calls get stub results, and data tools get the server's real render hint appended, with a distinct ref per result.
+- **Grading:** programmatic, against each case's `expect` in `cases.json`:
+  - `first`: acceptable first tool
+  - `must`: tools that must all be called
+  - `any_of`: at least one of these must be called
+  - `must_not`: none may be called; `*mapbox*` means any Mapbox tool
+- **Metrics:**
+  - `pass` (all checks hold), `first_ok` and `no_forbidden`
+  - `ref_ok`: every `render_map_tool` call passes refs it was actually given, unchanged, and doesn't hand-build a map when a ref was available
+  - `no_token_ask`: the agent neither asks the user for a Mapbox token nor writes a placeholder token (`YOUR_MAPBOX_...`, `pk...`) into a file
+
+## Running
+
+The runner reads `ANTHROPIC_API_KEY` from the environment.
+
+```bash
+# Baseline: the current checkout
+npm run eval:tools -- --variant baseline --reps 3
+
+# Compare another branch (e.g. a PR) from a worktree
+git worktree add --detach ../mcp-server-pr origin/<branch>
+ln -s "$PWD/node_modules" ../mcp-server-pr/node_modules
+EVAL_SERVER_ROOT=../mcp-server-pr npm run eval:tools -- --variant v1 --reps 3
+
+# Build the HTML report (baseline vs v1, per case)
+npm run eval:tools:report
+open .claude/hillclimb/tool-selection/report.html
+```
+
+- **Options:** `--model` (default `claude-opus-5-5`), `--reps`, `--concurrency` (default 8) and `--timeout-s` (default 600).
+- **Resume:** runs pick up where they left off. Each `(case, rep)` is skipped once it has a row in `results.jsonl`.
+- **Failures:** failed attempts go to `errors.jsonl` instead.
+- **Variants:** a v1 run should note what changed in `.claude/hillclimb/tool-selection/v1/change.md`, so the report can explain the difference.
+
+**First run:** the runner refuses to start until you approve the harness. Review `harness.ts` and `cases.json`, then add `--approve-harness` once. It records a hash of the runner, the lockfile and those two files in `_state.json`. Any later edit to them needs another approval.
+
+## OpenAI models
+
+The same cases, stubs and grader can run against an OpenAI model through the Responses API, to check that a description change doesn't regress routing for ChatGPT-style hosts. Results go to a separate flow, `.claude/hillclimb/tool-selection-openai`, so comparisons never mix providers.
+
+```bash
+npm run eval:tools:openai -- --variant baseline --reps 3 --model <OpenAI model id>
+EVAL_SERVER_ROOT=../mcp-server-pr npm run eval:tools:openai -- --variant v1 --reps 3 --model <same id>
+npm run eval:tools:report:openai
+```
+
+- **Key:** the runner reads `OPENAI_API_KEY`.
+- **`--model` is required:** the runner's default is a Claude model, so an OpenAI run without `--model` refuses to start.
+- **Which model:**
+  - `chat-latest` is the closest proxy for ChatGPT, but it only supports tools loaded up front (tool search returns a 400). Run it with `EVAL_MODES=eager`. It also reports itself as `chat-latest`, not a dated snapshot, so run the two branches back to back to keep the comparison on one model.
+  - `gpt-6.1-sol` supports both modes and stands in for API-based hosts. (Checked 2026-10-12.)
+- **Deferred mode:** uses OpenAI's server-executed tool search (`{type: "tool_search", execution: "server"}`) with `defer_loading` on the Mapbox functions.
+- **Turns:** chained with `previous_response_id`.
+- **Schemas:** passed with `strict: false`, so the 2020-12 schemas aren't rewritten for strict mode.
+- **Usage:** recorded in the same shape as Claude's. Cached tokens go to `cache_read_input_tokens`. The report has no OpenAI prices, so cost isn't computed.
+
+**What this does and doesn't measure.** It measures how an OpenAI model routes given our descriptions, server instructions and render hint. It doesn't reproduce ChatGPT itself. ChatGPT's system prompt, how it exposes connector tools, whether it passes MCP server instructions to the model, and its own built-in tools are not public. Treat a regression here as a strong signal, and a pass as "no regression at the model level".
+
+## Adding cases
+
+Add an entry to `cases.json` with a `why` that names the description or instruction it exercises.
+
+Give it `stubs` for every tool the prompt might reasonably call, written about the place in the prompt. A canned answer about the wrong place makes Claude retry or switch tools, and that gets scored as a routing failure. Calls without a stub return "No results" and are counted in the `unstubbed` column. A non-zero count means the case needs another stub, not that the model misrouted.
+
+The grader throws if a case names a tool that doesn't exist, so a renamed tool can't turn a `must_not` into a silent pass. A case can list tools in `needs_tools` instead: on a server without one of them, the case fails with "needs X, which this server doesn't have" rather than throwing. The site-building cases use this for `export_map_html_tool` (#275), so on main they show the baseline behavior instead of crashing the run.
+
+The stubs return canned output, so a stubbed `export_map_html_tool` result is a placeholder page. An agent that notices sometimes writes its own page or offers to rebuild it with a token, which `no_token_ask` flags. Read those transcripts before counting them as regressions. The noise floor for a pass rate is roughly `1/sqrt(rows x reps)`. At 34 rows x 3 reps that's about ±10 points overall, and much wider for any single case.
+
+## Files
+
+- `harness.ts`: tool surface, host tools, stubs, conversation loop, grader
+- `cases.json`: prompts and expected trajectories
+- `run-eval.mjs`: runner, vendored from the claude-api skill's eval scaffold (resume, backoff, wall-clock ceiling, served-model check, failure sidecar)
+- `build-report-lite.mjs`: static report builder, vendored from the same skill