feat(evals): tool-selection eval for description routing regressions

[mattpodwysocki]

Description

Adds an eval that checks which tools Claude (and, optionally, an OpenAI model) picks for a prompt, given our tool descriptions and server instructions. Nothing in the repo tested this, so changes like #273 (rewording descriptions for the Claude connector review) had no way to show they didn't break routing.

How it works

• Tool surface: built from a real McpServer, set up the way the hosted deployment does it (publishJsonSchema2020, installTo, buildServerInstructions), then read back over tools/list. The eval sees exactly what a host sees.
• Prompts: 19, each with expected tool calls. They cover:
  • map only when asked
  • static_map_image_tool only for image files
  • ground_location_tool vs reverse_geocode_tool
  • brand vs category search
  • web search staying available for place questions
  • building a page or file with a map: the agent gets a write_file host tool, and must not ask for a Mapbox token or leave a placeholder (these expect export_map_html_tool from feat(tools): add export_map_html_tool for maps in agent-built pages #275 and fail cleanly on servers without it)
• Two modes: each prompt runs with all descriptions loaded, and again with the Mapbox tools behind the API's tool search. The second roughly matches hosts like Claude Desktop that load tools on demand.
• Host tools: fake web_search and places_map_display_v0 tools sit alongside ours, so we can see if our descriptions crowd them out. That's the reviewer's concern.
• Stubbed results: tool results are canned per case and written about the place in the prompt, so no Mapbox calls are made. Data tools get the server's real render hint appended, so changes to the hint show up.
• Grading: checks the first tool, required tools and forbidden tools. Two extra metrics: ref_ok (every render_map_tool call passes refs it was actually given, unchanged; each stubbed result gets its own ref) and no_token_ask (no request for a token, no placeholder token written to a file).
• OpenAI: EVAL_PROVIDER=openai (npm run eval:tools:openai -- --model <id>) runs the same cases, stubs and grader through the Responses API, with OpenAI's server-side tool search for the deferred mode. Results go to a separate flow so providers never mix. chat-latest is the closest proxy for ChatGPT but only supports eager mode (EVAL_MODES=eager).
• Comparing branches: EVAL_SERVER_ROOT points at another checkout, so two branches can be compared. npm run eval:tools:report builds an HTML report with per-case transcripts.

The runner and report builder are vendored from the claude-api skill's eval scaffold. That gets us resume, backoff, a per-case timeout, a failure log and a check on which model actually answered. It calls the Claude API (~$8 per 114-row run on Opus 5.5), so it is not part of npm test.

First results (Opus 5.5, 3 reps each, on the original 17 prompts)

	main	#273	#273 + conditional render hint
Pass	75/102	80/102	89/102
Map drawn when none was asked for	27 rows	22 rows	13 rows
Maps still drawn when asked	24/24	24/24	24/24
Map ref passed unchanged	102/102	102/102	102/102

#273 doesn't regress anything. Most of the remaining unrequested maps come from the hint data tools append to their results. Making that hint conditional fixes most of them; I've suggested that change on #273.

ChatGPT proxy (chat-latest, eager only, 3 reps, on the original 17 prompts)

	main	#273	#273 + conditional render hint
Pass	35/51	45/51	46/51
Map drawn when none was asked for	13 rows	4 rows	3 rows

Same direction as Claude: #273 and the hint change both help, nothing regresses. Those runs used one shared ref across all stubs, which is why ref checks are now per result; they aren't rerun on the new harness yet.

Site-building cases (main vs #275, 3 reps)

	Claude, main	Claude, #275	chat-latest, main	chat-latest, #275
Site cases pass	0/12	12/12	0/6	6/6
No token ask or placeholder	1/12	11/12	1/6	5/6
Other cases pass	75/102	77/102	37/51	38/51
Map-requested cases pass	24/24	24/24	12/12	12/12

On main, both models hand-write GL JS and ask for a token or leave YOUR_MAPBOX_ACCESS_TOKEN in the file, which is what Codex did in the report. The one flagged row per provider on #275 comes from the stub returning a placeholder page, not from the tool.

---

Testing

• npm test: 980/980 pass. tsc --noEmit passes with evals/ included. The build only compiles src/ (tshy uses tsconfig.src.json).
• With no API calls, the grader passes every row when given the expected tool calls. Given no tool calls, it passes only the rows where calling nothing is correct. The ref check is tested against an invented ref and a hand-built map.
• Full runs against main, fix: rewrite tool descriptions flagged by Claude connector directory review #273 and fix: rewrite tool descriptions flagged by Claude connector directory review #273 + hint change, all 102/102 rows with no errors. On the current harness: main and feat(tools): add export_map_html_tool for maps in agent-built pages #275 on Claude (114/114 rows each) and chat-latest (57/57 each), no errors.

Notes

• @anthropic-ai/sdk, openai and tsx are new dev dependencies. The lockfile still resolves everything from registry.npmjs.org.
• Run output under .claude/hillclimb/*/ is gitignored. _state.json is committed: it lists the metrics, and the harness hash the runner checks before running.

Checklist

• Tests added or updated (npm test passes): no unit tests, since the eval is a separate paid run
• Lint passes (npm run lint)
• CHANGELOG.md updated under Unreleased
• Documentation updated if needed (evals/tool-selection/README.md)

🤖 Generated with Claude Code