From 330a083a1eee9882bc7c9de7a363b95fa396356e Mon Sep 17 00:00:00 2001 From: uipreliga Date: Thu, 23 Jul 2026 14:06:08 -0700 Subject: [PATCH 01/16] docs: agent guides, extending & report-schema references, and fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add missing documentation and fix defects found auditing docs against code. New docs: - docs/agents/ — move Codex guide to agents/CODEX.md; add CLAUDE_CODE.md (default agent) and ANTIGRAVITY.md (Gemini agent). - docs/EXTENDING.md — plugin SPI: custom agents, custom criteria, and register_pricing. - docs/REPORT_SCHEMA.md — field-level reference for run.json / task.json / variant.json / experiment.json / suite.json. USER_GUIDE: - document the `aggregate` command and `report --format`; complete the run flag table (--type/--repeats/--resume/--sample/--sample-per-stratum/ --include-skipped) and add plan --experiment, evaluate --run-dir. - add a Suite Thresholds & Classification Metrics section; add agent auth env vars (CODEX_*, GEMINI_API_KEY, ANTIGRAVITY_MODEL); clarify UIPATH_PLUGIN_MARKETPLACE_DIR vs PLUGIN_TOOLS_DIR. TASK_DEFINITION_GUIDE: - add the missing commands_efficiency and classification_match sections and complete the scoring-types summary (all 14 types). - fix llm_judge max_tokens default (1000 -> 2000), agent_judge default tools (read-only Bash/Read/Glob/Grep, not Write/Edit), and the llm_judge verdict wording (forced submit_verdict tool call). Other: - fix phantom experiments/example.yaml references in the agent guides. - regroup mkdocs nav (Agents / Running & scaling / Extending / Reference); align README/index/llms.txt doc indexes. - remove docs/IDEAS.md (internal backlog) and its mkdocs exclude. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 4 +- docs/EXTENDING.md | 239 +++++++++ docs/IDEAS.md | 459 ------------------ docs/REPORT_SCHEMA.md | 251 ++++++++++ docs/TASK_DEFINITION_GUIDE.md | 56 ++- docs/USER_GUIDE.md | 82 +++- docs/agents/ANTIGRAVITY.md | 203 ++++++++ docs/agents/CLAUDE_CODE.md | 193 ++++++++ .../{CODEX_AGENT_GUIDE.md => agents/CODEX.md} | 2 +- docs/index.md | 6 +- docs/llms.txt | 6 +- mkdocs.yml | 15 +- 12 files changed, 1035 insertions(+), 481 deletions(-) create mode 100644 docs/EXTENDING.md delete mode 100644 docs/IDEAS.md create mode 100644 docs/REPORT_SCHEMA.md create mode 100644 docs/agents/ANTIGRAVITY.md create mode 100644 docs/agents/CLAUDE_CODE.md rename docs/{CODEX_AGENT_GUIDE.md => agents/CODEX.md} (99%) diff --git a/README.md b/README.md index 4c63d01e..d3356651 100644 --- a/README.md +++ b/README.md @@ -183,8 +183,10 @@ alone. | [Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md) | The task-file schema — all criterion types, scoring, templates | | [A/B Experiments](docs/AB_EXPERIMENTS.md) | Compare models / tools / prompts across the same tasks | | [Bring Your Own Dataset](docs/BYOD.md) | Fan a single task out over a dataset | -| [Codex Agent Guide](docs/CODEX_AGENT_GUIDE.md) | Running the Codex agent | +| [Agent Guides](docs/agents/) | Running each agent — [Claude Code](docs/agents/CLAUDE_CODE.md), [Codex](docs/agents/CODEX.md), [Antigravity/Gemini](docs/agents/ANTIGRAVITY.md) | | [Docker Isolation](docs/DOCKER_ISOLATION.md) | The container sandbox driver | +| [Extending Coder Eval](docs/EXTENDING.md) | Author a custom agent, criterion, or model pricing via the plugin SPI | +| [Report Schema](docs/REPORT_SCHEMA.md) | Field-level reference for `run.json` / `variant.json` / `task.json` | | [CLAUDE.md](CLAUDE.md) | Architecture, key patterns, and extension points | | [CONTRIBUTING.md](CONTRIBUTING.md) | Dev setup, quality bar, and how to contribute | diff --git a/docs/EXTENDING.md b/docs/EXTENDING.md new file mode 100644 index 00000000..80d9320e --- /dev/null +++ b/docs/EXTENDING.md @@ -0,0 +1,239 @@ +--- +description: >- + Extend coder_eval with a custom agent, a custom success criterion, or model + pricing — the plugin SPI (coder_eval.plugins entry-point group), the Agent ABC + checklist, the @register_criterion decorator, and register_pricing. +--- + +# Extending Coder Eval + +coder_eval is extensible along three seams, all designed so a third party can add +capability **without editing the base package**: + +1. **Agents** — register a new `agent.type` via the plugin SPI. +2. **Criteria** — add a new success-criterion type via a decorator + auto-discovery. +3. **Pricing** — contribute USD rates for models your plugin runs. + +This guide covers all three. For the internal architecture notes see +[CLAUDE.md](https://github.com/UiPath/coder_eval/blob/main/CLAUDE.md). + +--- + +## 1. Custom agents (the plugin SPI) + +Agents register through the **`coder_eval.plugins` entry-point group** — there is no +closed enum or dispatch to edit. `agent.type` is an open string validated against +the `AgentRegistry`; built-in and third-party agents travel the exact same path. + +### Wire up the entry point + +In your plugin package's `pyproject.toml`: + +```toml +[project.entry-points."coder_eval.plugins"] +my_plugin = "my_plugin:register" +``` + +At CLI init, `load_plugins()` imports each entry point and calls it with the +`AgentRegistry` **class** (not an instance). A third-party hook that raises is logged +and skipped; only a failing *built-in* registration is fatal. + +### The `register` hook + +```python +from coder_eval.agents.registry import AgentRegistry + +def register(registry: type[AgentRegistry]) -> None: + # Bind type string → config class → agent class. + registry.register("my-agent", MyAgentConfig)(MyAgent) + # Optionally contribute pricing here too (see §3): + # register_pricing(MY_RATES) +``` + +`AgentRegistry.register(agent_kind, config_class)` returns a decorator, so the +decorator form works too: + +```python +@AgentRegistry.register("my-agent", MyAgentConfig) +class MyAgent(Agent[MyAgentConfig]): + ... +``` + +Registration is **anti-shadow**: re-registering the same `(agent_class, +config_class)` pair is a no-op, but claiming an existing `agent.type` with a +*different* implementation raises `ValueError`. Two plugins can never silently fight +over one type. + +### The config class + +Subclass `BaseAgentConfig` with your own `type` discriminator: + +```python +from typing import Literal +from coder_eval.models import BaseAgentConfig # importable from coder_eval.models + +class MyAgentConfig(BaseAgentConfig): + type: Literal["my-agent"] = "my-agent" + my_option: str = "default" +``` + +The factory `create_agent(kind, config, …)` raises `TypeError` if the passed config +isn't an instance of the registered `config_class`, so keep them paired. + +### The `Agent` ABC — implementation checklist + +Implement these three abstract methods: + +- [ ] `async def start(self, working_directory, *, env_path_prepend=None, plugin_tools_dir=None) -> None` +- [ ] `async def communicate(self, user_input, *, stream_callback=None, timeout=None, max_turns=None, should_stop=None) -> TurnRecord` +- [ ] `async def stop(self) -> None` + +Optional overrides (sensible defaults exist): `kill()`, `kill_sync()` (called from a +non-asyncio watchdog thread — must **not** await), `discard_pending_turn()`. + +Follow the shared turn lifecycle (do **not** hand-assemble a `TurnRecord`): + +- [ ] Call `self._begin_turn()` at the top of `communicate()`. +- [ ] Call `self._end_turn_ok()` on the success path. +- [ ] Call `self._mark_stopped()` in `stop()` after your own teardown. +- [ ] Before raising on a mid-turn failure, set `self.pending_turn` to a + `crashed=True` `TurnRecord` (built from an `EventCollector`), then raise + `AgentCrashError` / `TurnTimeoutError` (bare — no payload). The orchestrator + drains it and calls `discard_pending_turn()`. + +Emit the standardized event protocol (you are the **sole emitter**): one +`AgentStartEvent` at the top of `communicate()` and one matching `AgentEndEvent` on +**every** exit path (emit from `finally`), a `TurnStart`/`TurnEnd` pair per inner +turn, and `ToolStart`/`ToolEnd` per tool call (close orphaned tools with +`status=unresolved`). Fan events through an internal `EventCollector` — it builds the +returned `TurnRecord`, the single agent-agnostic capture path. + +Set `supports_cooperative_stop: ClassVar[bool] = True` only if your `communicate()` +actually honors `should_stop` (needed for `run_limits.stop_early`). Leaving it +`False` means early stop is rejected at resolution for your agent — which is correct +if you can't stop cooperatively. + +### Worked example + +The in-tree worked example is the BYOA test fixture at +[`tests/fixtures/byoa_demo_plugin/`](https://github.com/UiPath/coder_eval/tree/main/tests/fixtures/byoa_demo_plugin) +— a minimal package with a `register` hook and the `coder_eval.plugins` entry point. +(It subclasses `ClaudeCodeAgent` for brevity; a real third-party agent implements the +`Agent` ABC from scratch.) + +--- + +## 2. Custom success criteria + +Criteria are discovered automatically by `pkgutil` scan of the `coder_eval.criteria` +package — a new checker just needs the `@register_criterion` decorator. Adding one is +two steps: a Pydantic **model** (the YAML schema) and a **checker** (the logic). + +### Step 1 — the model + +In `models/criteria.py`, subclass `BaseSuccessCriterion` and set the discriminator as +a `Literal` default, then add it to the `SuccessCriterion` union: + +```python +class MyCriterion(BaseSuccessCriterion): + type: Literal["my_criterion"] = "my_criterion" + target: str + # Set requires_agent = True (ClassVar) if you read turn_records. + +# ...and add `| MyCriterion` to the SuccessCriterion discriminated union. +``` + +Union membership is required — a run validates that every union member's `type` has a +registered checker, and rejects unknown `type` tags in YAML. `BaseSuccessCriterion` +gives you `description`, `weight` (default 1.0; `0` = informational/non-gating), +`pass_threshold` (default 0.9), `stop_when`, and `suite_thresholds` for free, with +`extra="forbid"` so YAML typos are caught. + +### Step 2 — the checker + +Drop a file in `coder_eval/criteria/` with a `@register_criterion` class: + +```python +from coder_eval.criteria.base import BaseCriterion, register_criterion +from coder_eval.models import CriterionResult + +@register_criterion +class MyChecker(BaseCriterion[MyCriterion]): + criterion_type = "my_criterion" # must match the model discriminator + + def _check_impl(self, criterion, sandbox, reference_code=None, *, + turn_records=None, context=None) -> CriterionResult: + ok = ... # your logic + return CriterionResult( + criterion_type=self.criterion_type, + description=criterion.description, + score=1.0 if ok else 0.0, + details="...", + ) +``` + +Notes: + +- **Do not override `check()`** — it's final and wraps `_check_impl` with error + handling (an exception becomes a score-0.0 result with the error captured). +- Return `score` in `[0.0, 1.0]` — binary criteria use `0.0`/`1.0`; fractional ones + anything in between. +- For **suite-level metrics** on dataset-backed tasks, override + `aggregate(criterion, per_row_results)`; the base emits + `count/mean/median/std/min/max`, so your criterion is suite-thresholdable for free. + Classification-style criteria return a `ClassificationCriterionResult` and layer + accuracy / precision / recall / F1 / confusion on top. +- For **early stop**, implement `live_verdict(...)` and declare + `live_stop_polarities` — a lint rule keeps the two consistent. + +> A duplicate `criterion_type` **overwrites** the earlier checker with a warning (not +> a hard error, unlike agents) — keep type strings unique. + +--- + +## 3. Model pricing + +Plugins that run their own models contribute USD rates through `register_pricing` — +there is **no** separate entry-point group; call it from the same `register()` hook. + +```python +from coder_eval.pricing import ModelPricing, register_pricing + +# Rates are per MILLION tokens: (input, output, cache_write, cache_read) +MY_RATES = { + "my-model-v1": ModelPricing(3.0, 15.0, 3.75, 0.30), + "my-free-model": ModelPricing(0.0, 0.0, 0.0, 0.0), # a valid free-model entry +} + +def register(registry): + registry.register("my-agent", MyAgentConfig)(MyAgent) + register_pricing(MY_RATES) +``` + +Behavior: + +- **Keys** are the bare model id as it appears in `agent.model` (vendor/Bedrock + region prefixes like `eu.` / `anthropic.` are normalized off at lookup). +- **Idempotent** for identical rates; **raises `ValueError`** on a *conflicting* rate + for an existing key (built-in or another plugin). Registration is all-or-nothing — + a late conflict leaves nothing half-applied. +- **Zero rates are valid** (a genuinely free model) — the lookup uses "is a rate + present", not truthiness, so an all-zero entry prices to `0.0` rather than falling + through to the built-in table. +- The plugin overlay is consulted **before** the built-in table, so every consumer + (agents, reports, the cost simulator) prices your model transparently via + `calculate_cost(model, uncached_input, output, cache_creation=0, cache_read=0)`. + (Pass `uncached_input`, not the total input.) + +The base package ships **no** plugin rates; only the built-in table. + +--- + +## See also + +- [Claude Code](agents/CLAUDE_CODE.md) · [Codex](agents/CODEX.md) · + [Antigravity](agents/ANTIGRAVITY.md) — the built-in agents, each registered via + this same SPI +- [Task Definition Guide](TASK_DEFINITION_GUIDE.md) — the criterion catalogue +- [CLAUDE.md](https://github.com/UiPath/coder_eval/blob/main/CLAUDE.md) — architecture + and extension points in depth diff --git a/docs/IDEAS.md b/docs/IDEAS.md deleted file mode 100644 index a9e7a018..00000000 --- a/docs/IDEAS.md +++ /dev/null @@ -1,459 +0,0 @@ -# Ideas & Backlog - -A lightweight, stack-ranked list of improvements for `coder_eval` (the eval harness, -the dashboard/CI, and the evalboard UI). Anyone can add an idea — keep it short, then -we triage and work the list top-down. - -## How to stack rank - -**Rank lives in one place: row order in the table below. Top row = do first.** -To re-rank, just move a row up or down — nothing else changes. - -- Each idea has a **stable ID** (e.g. `expected-turns`) that never changes, even when - the rank does. Headings and cross-references use the ID, not a position number, so - reordering never means renumbering. -- To decide *where* a row goes, score it on **Impact** and **Effort** (1–5 each), then - let **Score** rank it: - - > **Score = Impact + (6 − Effort)** — range 2–10, higher floats up. Sort by Score; - > break ties with judgment (dependencies, momentum, who's free). - - - **Impact** = how much this helps the *skills people are building* or our - organization's *ability to improve* them. Bigger lever = higher. - - **Effort** = order-of-magnitude **tokens a coding agent burns to build it** - (best guess is fine — it's a log scale, so just pick the nearest power of ten). - -| Field | 1 | 2 | 3 | 4 | 5 | -|-------|---|---|---|---|---| -| **Impact** | trivial | minor | useful | important | game-changing | -| **Effort** (tokens) | ~100K | ~1M | ~10M | ~100M | ~1B | - -**Status:** 💡 idea · 🔍 scoping · 🚧 in progress · ✅ done · ❄️ parked - -## Backlog - -| ID | Idea | Impact | Effort | Score | Status | Owner | -|----|------|:------:|:------:|:-----:|:------:|-------| -| [`expected-turns`](#expected-turns) | Top-level Expected Turns score | 4 | 2 | 8 | 🚧 | Sherif | -| [`efficiency-columns`](#efficiency-columns) | Surface already-computed efficiency metrics on the board | 4 | 2 | 8 | 💡 | — | -| [`config-fingerprint`](#config-fingerprint) | Per-run config fingerprint + regression attribution | 4 | 2 | 8 | 💡 | — | -| [`judge-ensemble`](#judge-ensemble) | Multi-sample judge voting + verdict variance | 4 | 2 | 8 | 💡 | — | -| [`cost-preflight`](#cost-preflight) | Pre-flight budget validation + judge cost estimate | 3 | 1 | 8 | 💡 | — | -| [`long-failing`](#long-failing) | PR a fix for long-running failed tests | 4 | 3 | 7 | 💡 | — | -| [`failure-mode-rollup`](#failure-mode-rollup) | Failure-mode taxonomy headline | 4 | 3 | 7 | 💡 | — | -| [`judge-calibration`](#judge-calibration) | Golden-set judge calibration + drift detection | 4 | 3 | 7 | 💡 | — | -| [`result-cache`](#result-cache) | Skip-unchanged task result cache | 4 | 3 | 7 | 💡 | — | -| [`smart-judge-truncation`](#smart-judge-truncation) | Smarter judge context truncation | 3 | 2 | 7 | 💡 | — | -| [`per-criterion-telemetry`](#per-criterion-telemetry) | Per-criterion timing + token telemetry | 3 | 2 | 7 | 💡 | — | -| [`top-offenders`](#top-offenders) | Top Offenders page | 3 | 3 | 6 | 💡 | — | -| [`diff-criterion`](#diff-criterion) | Diff/patch validation criterion | 3 | 3 | 6 | 💡 | — | -| [`ux-revamp`](#ux-revamp) | UX revamp | 4 | 4 | 6 | 💡 | — | - ---- - -### expected-turns -**Top-level Expected Turns score** - -**What:** Roll today's *per-task* expected-turns signal up into a single headline -metric for a run (e.g. "% of tasks within expected turns", or mean turn overage), -surfaced on the dashboard and in the nightly Slack summary alongside pass-rate. - -**Why:** A task can pass but take far more turns than budgeted — a real efficiency -regression that the pass/fail headline hides. We already compute the per-task signal; -we just don't aggregate or surface it at the top. - -**What exists today:** -- `expected_turns_overage(result)` and `visible_turn_count(result)` — - [src/coder_eval/reports_stats.py:174](../src/coder_eval/reports_stats.py#L174) -- Per-task "expected_turns exceeded" badge in the HTML report — - [src/coder_eval/reports_html.py:341](../src/coder_eval/reports_html.py#L341) -- `run_limits.expected_turns` is the per-task budget source. - -**Rough approach:** aggregate overage across a run → add a field to the run summary → -render on the evalboard + add one line to the nightly Slack summary. - -**Open questions:** -- What's the right headline statistic — % within budget, mean overage, or count exceeded? -- Per-skill breakdown too, or run-level only to start? - ---- - -### long-failing -**PR a fix for long-running failed tests** - -**What:** Identify eval tasks/tests that consistently run long *and* fail (burning the -nightly's 4h budget for no signal), then open PRs to fix or quarantine them. - -**Why:** Long failing tasks eat wall-clock and cost on every nightly run and add noise. -Tightening or fixing them improves both run time and the trustworthiness of the headline. - -**Rough approach:** mine recent `runs/` for tasks with high duration + non-SUCCESS status → -rank worst offenders → per-task, either fix the underlying issue or adjust limits/quarantine → -PR each fix. - -**Open questions:** -- Threshold for "long-running" — absolute seconds, or top-N by duration? -- Fix vs. quarantine vs. limit-tuning — decide per task or set a default policy? -- One-off cleanup, or a recurring report we act on each cycle? - ---- - -### top-offenders -**Top Offenders page** - -**What:** A single page that ranks the things most worth addressing first — worst -tasks/skills by failure rate, turn overage, duration, and cost — so triage starts from -"here's what to fix" instead of scrolling a full run. - -**Why:** Today you read the whole board to find problems. A focused "top offenders" -view turns the metrics we collect (and the ones in `expected-turns` and `long-failing`) -into an actionable work queue, and gives this backlog a data-driven feed. - -**Rough approach:** aggregate per-task/per-skill stats across the latest run (reuse the -signals from [reports_stats.py](../src/coder_eval/reports_stats.py)) → rank by a few axes → -render as a sortable list on the evalboard, linking each row to its task detail. - -**Open questions:** -- Which axes and what default sort — failures first, or a blended "needs-attention" score? -- Latest-run only, or trend across recent runs to catch persistent offenders? -- Standalone page or a panel on the run overview? (overlaps with `ux-revamp`) - ---- - -### ux-revamp -**UX revamp** - -**What:** Refresh the evalboard UI ([evalboard/](../evalboard/), Next.js) — clearer run -overview, easier drill-down from suite → skill → task, better surfacing of the metrics -above (pass-rate, cost, turns). - -**Why:** The board is the daily touchpoint for reading nightly results; a clearer UX -makes regressions and outliers faster to spot. - -**Open questions (needs scoping before it's actionable):** -- What are the top 2–3 jobs-to-be-done on the board today that are painful? -- Incremental polish vs. a rethink of the information architecture? -- Any design input, or engineer's-judgment pass? - ---- - -### efficiency-columns -**Surface already-computed efficiency metrics on the board** - -**What:** We already compute per-task efficiency signals and write them into -`run.json`, but the evalboard never shows them. Add columns/badges for -`commands_efficiency`, a `max_turns_exhausted` status badge, total tokens, and a -derived cache-hit-rate (`cache_read / (cache_creation + cache_read)`) — and show -`expected_commands` next to actual so over-budget tasks are obvious at a glance. - -**Why:** Cost and efficiency regressions are invisible today. The data is *already -in the artifact* — this is pure surfacing, not new computation, so it's a cheap, -high-leverage win that complements [`expected-turns`](#expected-turns). - -**What exists today:** -- `eval_result_to_task_dict()` writes `commands_efficiency`, `max_turns_exhausted`, - `expected_commands`, and `total_tokens` into each task dict — - [src/coder_eval/reports_experiment.py:42](../src/coder_eval/reports_experiment.py#L42) -- The grid recomputes/displays only a subset (turns, cost, raw token buckets) — - [evalboard/app/runs/[id]/task-grid.tsx](../evalboard/app/runs/[id]/task-grid.tsx) -- Turn-budget coloring already exists as a model — [evalboard/lib/turns.ts](../evalboard/lib/turns.ts) - -**Rough approach:** widen the evalboard `RawTaskResult` type to read the existing -fields → add sortable columns + a `max_turns_exhausted` badge → reuse the -`turns.ts` color-ratio pattern for an efficiency/cache cell. - -**Open questions:** -- Which metrics earn a column vs. a hover/detail-panel stat (avoid grid sprawl)? -- Cache-hit-rate: compute in the loader, or add it to `run.json` once so the - Slack summary can use it too? - ---- - -### config-fingerprint -**Per-run config fingerprint + regression attribution** - -**What:** Stamp each run with a stable fingerprint of the inputs that actually drive -results (resolved task config + agent/model + skills/cli SHAs + framework version), -and add a board view that diffs two runs' fingerprints. When pass-rate moves, -answer "did the model change, the prompt change, or the task config change?" - -**Why:** Today a regression between nightlies is a whodunit — you can't separate a -model bump from a prompt rewrite from a config drift. The raw material (config -lineage, component SHAs) is already captured per task; nothing ties it into a -run-level, diffable fingerprint. - -**What exists today:** -- Per-task `task_config` + dotted-path config lineage — - [src/coder_eval/reports_experiment.py:42](../src/coder_eval/reports_experiment.py#L42) -- Component SHAs (coder_eval / skills / cli) already in the run summary and Slack - footer — the nightly Slack summary -- Single declarative merge resolver produces the resolved config — - [src/coder_eval/orchestration/config_merge.py](../src/coder_eval/orchestration/config_merge.py) - -**Rough approach:** hash the resolved-config + SHA bundle into `run.json` → add a -"compare runs" panel on the evalboard that renders the field-level diff → optionally -flag pass-rate deltas whose fingerprint is *identical* (i.e. pure agent nondeterminism). - -**Open questions:** -- Fingerprint at run level, per-skill, or per-task (tasks can differ within a run)? -- What's the canonical diff unit — resolved YAML, or the lineage entries? - ---- - -### judge-ensemble -**Multi-sample judge voting + verdict variance** - -**What:** Let `llm_judge` (and optionally `agent_judge`) run N independent samples -per row and aggregate — median/mean score plus a variance/disagreement signal — -instead of trusting a single one-shot verdict. Surface verdict spread so a -"flaky judge" is visible rather than silently noisy. - -**Why:** Subjective scoring underpins a large share of tasks, but every judged score -today is a single sample at temperature 0 with no measure of its own reliability. A -judge that would score the same artifact 0.4 / 0.8 / 0.6 looks authoritative. Voting -reduces variance; reporting spread tells us *which rubrics are unreliable*. - -**What exists today:** -- Single-sample verdict via forced `submit_verdict` tool call; temperature frozen at - parse time — [src/coder_eval/criteria/llm_judge.py](../src/coder_eval/criteria/llm_judge.py) -- `JudgeVerdict` (score / rationale / findings) with score clamped to [0,1] — - [src/coder_eval/models/judge.py](../src/coder_eval/models/judge.py) -- Per-criterion `aggregate()` already exists for suite rollups — - [src/coder_eval/criteria/base.py:192](../src/coder_eval/criteria/base.py#L192) - -**Rough approach:** add `samples: int` (+ aggregation rule) to the judge criteria → -run N calls (concurrently) → fold into one `JudgeCriterionResult` carrying mean + -stddev + raw scores → expose stddev as a suite-thresholdable metric. - -**Open questions:** -- Default N and temperature — fixed (e.g. 3 @ T=1.0) or per-criterion? -- Does verdict spread gate the suite, or is it report-only to start? -- Cost ceiling: cap samples on `agent_judge` (expensive) differently from `llm_judge`? - ---- - -### cost-preflight -**Pre-flight budget validation + judge cost estimate** - -**What:** Validate run-time budget config *before* a run starts, and estimate judge -token cost before each judge fires. Two concrete fixes: (1) error (or loudly warn) -when `max_usd` is set on a route that can't report cost, instead of silently -skipping the budget; (2) estimate the judge context's token footprint so a giant -file list / dialog can't blow out context unexpectedly. - -**Why:** `max_usd` silently no-ops on routes without per-turn cost data — you think -you're capped and you're not, and you only find out at result-finalization. Judges -also burn tokens with zero pre-flight check. Both are cheap guardrails against -surprise cost. - -**What exists today:** -- Cost budget skipped with a one-time warning when no per-turn cost is available — - [src/coder_eval/orchestrator.py:640](../src/coder_eval/orchestrator.py#L640) -- `cost_data_available` recorded *post-hoc* in environment info — - [src/coder_eval/orchestrator.py:516](../src/coder_eval/orchestrator.py#L516) -- Judge context assembled (files + reference + dialog) with no token estimate — - [src/coder_eval/evaluation/judge_context.py](../src/coder_eval/evaluation/judge_context.py) - -**Rough approach:** add a pre-run validator that cross-checks `run_limits.max_usd` -against the resolved route's cost capability (error or explicit warn) → add a rough -token estimate to `JudgeContextBuilder` and log/cap it before dispatch. - -**Open questions:** -- Hard error vs. warn for `max_usd` on a cost-blind route? -- Estimate-only, or auto-trim context to fit a configured ceiling? - ---- - -### failure-mode-rollup -**Failure-mode taxonomy headline** - -**What:** Roll the *reasons* tasks failed into a run-level headline — counts of -`max_turns_exhausted`, `TOKEN_BUDGET_EXCEEDED`, `COST_BUDGET_EXCEEDED`, crashes/ -timeouts, and the top review tags (e.g. `prompt-gap`) — surfaced in `run.json`, the -Slack summary, and the run page. Turn "62 failures" into "31 prompt-gap, 18 -max-turns, 9 crashes, 4 budget." - -**Why:** The pass/fail headline says *how many* failed, never *why*. The categories -already exist across the codebase (final-status enums, per-task flags, review tags) -but nothing aggregates them into one actionable breakdown — the input that makes -[`top-offenders`](#top-offenders) and triage useful. - -**What exists today:** -- `FinalStatus.TOKEN_BUDGET_EXCEEDED` / `COST_BUDGET_EXCEEDED` and `max_turns_exhausted` - per task — [src/coder_eval/reports_experiment.py:42](../src/coder_eval/reports_experiment.py#L42) -- Error categorization for crashes/timeouts — - [src/coder_eval/errors/categorization.py](../src/coder_eval/errors/categorization.py) -- Per-task review tags indexed in `review_index.json` — - the nightly Slack summary - -**Rough approach:** aggregate per-task statuses + flags + tags into a `failure_modes` -block on the run summary → add one section to the Slack summary → render as a -breakdown bar on the run page. - -**Open questions:** -- Status enum + flags only, or blend in the (looser) review-tag taxonomy? -- Fixed buckets, or data-driven top-N categories per run? - ---- - -### judge-calibration -**Golden-set judge calibration + drift detection** - -**What:** A small, version-controlled set of artifacts with hand-assigned "correct" -scores per rubric, plus a command that runs the judges against it and reports -agreement (e.g. MAE / rank correlation vs. the golden scores). Run it in CI so a -prompt or model change that silently shifts judge behavior trips a gate. - -**Why:** Judges have no ground truth to tune against — we can't tell a good rubric -from a lenient one, and a model upgrade can re-baseline every judged score without -anyone noticing. A golden set turns "trust me" into a measurable, regression-tested -property and complements [`judge-ensemble`](#judge-ensemble) (variance) with -*accuracy*. - -**What exists today:** -- LLM + agent judges produce structured `JudgeVerdict`s — - [src/coder_eval/evaluation/judge_verdict.py](../src/coder_eval/evaluation/judge_verdict.py) -- Suite-level aggregation + thresholds already gate on metrics — - [src/coder_eval/reports.py](../src/coder_eval/reports.py) -- No fixture of graded reference verdicts exists anywhere in the repo. - -**Rough approach:** curate ~10–20 graded artifacts per judge rubric under `tests/` -→ add a `coder-eval judge-calibrate` command that scores them and reports -agreement → wire a CI gate on a minimum agreement threshold. - -**Open questions:** -- Per-rubric golden sets, or one shared spread of easy/medium/hard cases? -- Agreement metric — MAE, Spearman rank, or pass/fail confusion at the threshold? - ---- - -### result-cache -**Skip-unchanged task result cache** - -**What:** Cache completed task results keyed by a fingerprint of everything that -determines the outcome (resolved task config + agent/model + skills/cli SHA + -template/sandbox inputs). On the next run, tasks whose fingerprint is unchanged -*and* deterministic can be reused instead of re-executed — dramatically cutting -nightly wall-clock and cost. - -**Why:** Every nightly re-runs every task from scratch even when nothing relevant -changed, burning the 4h budget and real money. A content-addressed cache lets the -nightly spend its budget on what actually changed. (Pairs naturally with -[`config-fingerprint`](#config-fingerprint), which produces the same key.) - -**What exists today:** -- Batch runner already supports *resume* — partition prior results, re-run only - failed/skipped — [src/coder_eval/orchestration/batch.py](../src/coder_eval/orchestration/batch.py) -- No inter-run result cache; runs are fully independent. -- Agent runs are stochastic, so caching must be opt-in / fingerprint-gated. - -**Rough approach:** reuse the [`config-fingerprint`](#config-fingerprint) key → -store last-good results in a content-addressed store → on run start, short-circuit -tasks with a cache hit (respecting a `--no-cache` / replicate-count override) → -clearly mark reused vs. fresh in `run.json`. - -**Open questions:** -- Does caching even make sense given agent nondeterminism — only for deterministic - criteria, or never reuse the *agent* run and only cache *grading*? -- Where does the cache live (local dir vs. blob) and how is it invalidated/expired? - ---- - -### smart-judge-truncation -**Smarter judge context truncation** - -**What:** Replace the naïve "drop trailing turns when over budget" dialog truncation -with a strategy that preserves high-signal context — keep first + last K turns, or -middle-ellipsis — so the judge never loses the final user question on a long -simulation. There's already a TODO in the code calling for exactly this. - -**Why:** On long dialogs, FIFO-dropping the *most recent* turns can discard the -decisive exchange the rubric needs, quietly degrading verdict quality. The fix is -local and well-scoped, and it directly improves judge reliability. - -**What exists today:** -- `JudgeContextBuilder` truncation with a standing TODO ("keep first+last K, or - middle-ellipsis ... the naïve cap is enough for the common case") — - [src/coder_eval/evaluation/judge_context.py](../src/coder_eval/evaluation/judge_context.py) -- Per-file cap + aggregate dialog cap (`max_dialog_chars`, default 80K), with - degradation already marked in the context notes. - -**Rough approach:** implement a first+last-K (or middle-ellipsis) selector behind -the existing truncation point → record which turns were elided in the degradation -note → keep the naïve cap as a fallback option. - -**Open questions:** -- First+last-K vs. middle-ellipsis vs. relevance-scored selection — start simple? -- Same strategy for per-file truncation, or dialog-only for now? - ---- - -### per-criterion-telemetry -**Per-criterion timing + token telemetry** - -**What:** Record wall-clock time and (for judges) token/cost spend per criterion, so -we can answer "which criterion is slow/expensive?" Today criteria — especially the -two judge types — are a black box inside the per-task total. - -**Why:** When a task is slow or pricey, there's no way to attribute it to a specific -criterion. Judges in particular can dominate cost, and we can't see it. This is the -instrumentation that makes [`long-failing`](#long-failing) and cost triage precise -rather than guesswork. - -**What exists today:** -- `CriterionResult` carries score/details but no timing or token attribution — - [src/coder_eval/models/results.py](../src/coder_eval/models/results.py) -- Judge token usage is captured at the *task* level (via the run's backend / SDK), - not per-criterion — [src/coder_eval/orchestrator.py](../src/coder_eval/orchestrator.py) -- `SuccessChecker` dispatches criteria sequentially — - [src/coder_eval/evaluation/checker.py](../src/coder_eval/evaluation/checker.py) - -**Rough approach:** wrap each criterion check in a timer in `SuccessChecker` → -add `duration_ms` (+ optional `tokens`/`cost_usd` for judges) to `CriterionResult` -→ surface a per-criterion mini-breakdown on the task detail page. - -**Open questions:** -- Token attribution for judges — exact (per-call delta) or best-effort? -- Worth running independent criteria concurrently while we're in here, or out of scope? - ---- - -### diff-criterion -**Diff/patch validation criterion** - -**What:** A first-class criterion that validates the *change* an agent made — expected -files touched/untouched, hunks present, lines added/removed within bounds — by -diffing the sandbox against its starting state. Built for incremental-edit tasks -where "did it modify the right things, and only those?" matters more than final -file content. - -**Why:** Coding agents are increasingly graded on edits, not greenfield files. Today -that's only checkable via brittle `file_contains` substrings or custom `run_command` -shell. A diff-aware criterion captures edit intent directly and discourages -over-broad changes (a real failure mode). - -**What exists today:** -- 17 criterion types, none diff-aware; closest is `file_contains` / `file_matches_regex` — - [src/coder_eval/models/criteria.py](../src/coder_eval/models/criteria.py) -- Sandbox starts from templates/starter files, so a baseline-vs-final diff is - available — [src/coder_eval/sandbox.py](../src/coder_eval/sandbox.py) -- Plugin registry makes adding a checker mechanical (`@register_criterion`) — - [src/coder_eval/criteria/__init__.py](../src/coder_eval/criteria/__init__.py) - -**Rough approach:** add a `diff_check` model to the criteria union → checker computes -the sandbox-vs-baseline diff and scores against expected changed-files / hunk -patterns / add-remove bounds → fractional score so partial edits get partial credit. - -**Open questions:** -- Compare against git baseline, a snapshot of starter files, or a reference patch? -- Score on file-set match, hunk match, or both (weighted)? - ---- - -## How to add an idea - -1. Pick a short, stable **ID** (kebab-case) and add a `### ` section using the - template above (What / Why / rough approach / open questions). -2. Score it on Impact and Effort (1–5), compute **Score = Impact + (6 − Effort)**. -3. Insert a row in the **Backlog** table at the position its Score implies — that - position *is* its rank. Done. diff --git a/docs/REPORT_SCHEMA.md b/docs/REPORT_SCHEMA.md new file mode 100644 index 00000000..905f9274 --- /dev/null +++ b/docs/REPORT_SCHEMA.md @@ -0,0 +1,251 @@ +--- +description: >- + Field-level reference for coder_eval's JSON outputs — run.json, task.json, + variant.json, experiment.json, and suite.json — plus the token/criterion + telemetry sub-models and FinalStatus values, for anyone consuming a run. +--- + +# Report Schema + +Coder Eval writes machine-readable JSON alongside every markdown/HTML report. This +page is the field-level reference for consumers (dashboards, CI parsers, evalboard +forks). For the on-disk directory tree see +[User Guide → Output Structure](USER_GUIDE.md#output-structure); for how to +re-generate these files see [`coder-eval report` / `aggregate`](USER_GUIDE.md#cli-commands). + +All JSON is Pydantic `model_dump_json` output — keys are the model field names +verbatim (no aliases, except `iterations` also accepts the legacy key `turns` on +read). Times are ISO-8601. + +## Who writes what + +| File | Model | When | +| --- | --- | --- | +| `run.json` / `run.md` | `RunSummary` | Every run (and rebuildable via `coder-eval aggregate`) | +| `///task.json` | `EvaluationResult` | One per replicate | +| `//suite.json` / `.md` | `SuiteRollup` | Dataset-backed suites only | +| `experiment.json` / `.md` | `ExperimentResult` | Every run (experiment layer) | +| `/variant.json` / `.md` | `VariantAggregate` | Per variant | + +`` is the zero-padded replicate index. Judge transcripts spill to sibling files +(e.g. `judge-0.yaml`) referenced by a `transcript_path`. + +--- + +## `run.json` — `RunSummary` + +**`run.json` is flat, not a `run → variant → task → replicate` tree.** It is the +run-level summary; full per-replicate detail lives in each `task.json`. + +| Key | Type | Meaning | +| --- | --- | --- | +| `run_id` | `str` | Timestamp id, e.g. `"2025-10-09_15-30-45"`. | +| `start_time` / `end_time` | `datetime` | Run window. | +| `total_duration_seconds` | `float` | Wall-clock. | +| `tasks_run` | `int` | Total replicates executed. | +| `tasks_succeeded` / `tasks_failed` / `tasks_error` | `int` | Category counts. **Invariant:** the three sum to `tasks_run`. | +| `tasks_token_budget_exceeded` / `tasks_cost_budget_exceeded` | `int` | Sub-counters of `tasks_failed` (not part of the invariant). | +| `skipped_tasks` | `list[{path, reason}]` | Load failures / `skip: true` opt-outs. | +| `max_parallel` | `int` | Concurrency used. | +| `task_results` | `list[dict]` | Flat per-replicate rows — see below. | +| `framework_version` | `str` | Coder Eval version chip. | +| `environment_info` | `dict` | Version/dependency info (may nest, e.g. `tool_plugins`). | + +### `task_results[]` — the flat per-task row + +Each entry is an **untyped dict** (a denormalization, not a Pydantic model) with keys +including: `task_id`, `replicate_index`, `variant_id`, `status` +([`FinalStatus`](#finalstatus)), `weighted_score`, `duration`, `iteration_count`, +`tags`, `task_path`, `model_used`, `reference_similarity`, the token buckets +(`input_tokens` = uncached input, `output_tokens`, `cache_creation_input_tokens`, +`cache_read_input_tokens`, `total_tokens`), `total_cost_usd`, `expected_commands`, +`actual_commands`, `commands_efficiency`, `agent_config`, `sdk_options`, +`installed_tools`, turn accounting (`total_turns`, `visible_turns`, `expected_turns`, +`max_turns_exhausted`, `has_final_reply`), and early-stop fields (`stopped_early`, +`early_stop_reason`, `turns_remaining_at_stop`). `iterations` here is a **reduced** +turn digest (`{iteration, duration_seconds, command_count, assistant_turn_count, +crashed, crash_reason}`) — the full transcript is in `task.json`. + +--- + +## `task.json` — `EvaluationResult` + +The authoritative per-replicate record. + +**Identity/metadata:** `task_id`, `task_description`, `variant_id` (default +`"default"`), `agent_type`, `model_used`, `started_at`, `completed_at`, +`duration_seconds`. + +**Results:** + +| Key | Type | Meaning | +| --- | --- | --- | +| `final_status` | [`FinalStatus`](#finalstatus) | Terminal status. | +| `weighted_score` | `float \| null` | Weighted average of criterion scores, 0.0–1.0. | +| `max_turns_exhausted` | `bool` | Ran out of turns. | +| `iteration_count` | `int` | Number of turns. | +| `success_criteria_results` | `list[CriterionResult]` | Per-criterion results — see [below](#criterionresult). | + +**Transcript:** `iterations: list[TurnRecord]` (accepts legacy alias `turns`) — see +[TurnRecord](#turnrecord). + +**Errors** (populated on failure): `error_message`, `error_details`, +`error_log_tail` (carries the Docker build-log tail for `BUILD_FAILED`). + +**Config/environment:** `environment_info`, `agent_config`, `sdk_options` (raw +`ClaudeAgentOptions` dump), `sandbox_path`, `task_config` +(`{resolved, source_yaml, source_file, lineage}` — `lineage` maps each field to +`{value, source, source_detail}` so you can trace which config layer set it). + +**Telemetry/totals:** `total_token_usage` ([TokenUsage](#tokenusage)), +`command_stats` (`CommandStatistics`), `total_assistant_turns`, `expected_commands` / +`actual_commands` / `commands_efficiency`, `pre_run_results` / `post_run_results` +(`{command, exit_code, stdout, stderr, duration_seconds, error}`), `simulation` +(dialog-mode telemetry, `null` in single-shot), and `early_stop` +([EarlyStopInfo](#earlystopinfo)). + +### CriterionResult + +A discriminated union on `result_kind` (`basic` / `judge` / `classification`); legacy +files without `result_kind` are inferred from `criterion_type`. Base fields +(`result_kind="basic"`): + +`criterion_type`, `description`, `score` (0.0–1.0), `details`, `error`, +`pass_threshold` (default 0.9), `gating` (default `true`; `false` = informational / +weight-0, excluded from the score and the pass/fail gate). The base allows extra +fields so subclass keys round-trip. + +- **`classification`** adds `observed_label`, `expected_label` (sentinels like + `(none)` / `(other)` allowed). Emitted by `classification_match`, `skill_triggered`. +- **`judge`** adds `findings`, `token_usage` (kept distinct from the agent total), and + `transcript_path` (a sibling `judge-N.yaml`). The full `transcript` is **stripped + from `task.json`** — read it from the referenced file. Emitted by `llm_judge`, + `agent_judge`. + +### TurnRecord + +`iteration`, `user_input`, `agent_output`, `commands` (`list[CommandTelemetry]`), +`timestamp`, `duration_seconds`, `token_usage`, `model_used`, `assistant_turn_count`, +`messages` (`list[TranscriptMessage]`, discriminated on `role`: +`user`/`assistant`/`reconciliation`), `num_turns`, `max_turns_exhausted`, +`result_summary` (`{is_error, subtype, stop_reason, result}`), `crashed`, +`crash_reason`. + +> **Token invariant.** Summing the four token buckets across `messages` +> (assistant + the synthetic `reconciliation` entry) equals `token_usage` exactly. +> The `reconciliation` message carries the residual the per-message stream +> under-reports; it has no cost and is excluded from turn/generation counts. See the +> [Claude Code guide](agents/CLAUDE_CODE.md#telemetry). + +### EarlyStopInfo + +Present (non-`null`) iff the run stopped early — there is no separate boolean. +Fields: `reason` (`criterion_passed` / `criterion_failed`), +`deciding_criterion_type`, `deciding_criterion_description`, `armed_criteria`, +`sdk_turn_index`, `tool_call_index` (1-based, includes the in-flight call), +`elapsed_seconds`, `turns_remaining_at_stop`. + +--- + +## `variant.json` — `VariantAggregate` + +A single aggregate (not wrapped): `variant_id`, `tasks_run`, `tasks_succeeded`, +`tasks_failed`, `tasks_error` (same sum-to-`tasks_run` invariant), `average_score`, +`average_duration`, `total_tokens`, `replicate_count`, `tasks_token_budget_exceeded`, +`tasks_cost_budget_exceeded`. + +## `experiment.json` — `ExperimentResult` + +The cross-variant summary: + +- `experiment_id`, `description`, `variant_ids`. +- `task_summaries: list[TaskExperimentSummary]` — each + `{task_id, variant_results, best_variant, is_tie, score_spread, replicate_count}`, + where each `VariantResult` carries `{variant_id, task_id, weighted_score, + final_status, duration_seconds, total_tokens, iteration_count, + total_assistant_turns, reference_similarity, replicate_index, replicate_count}`. +- `variant_aggregates: dict[str, VariantAggregate]` — keyed by variant id. +- `total_duration_seconds`. +- `per_replicate_scores: dict[variant_id -> dict[task_id -> list[float]]]`. + +> **Statistics are render-time only.** Bootstrap/Wilson confidence intervals and the +> Welch/paired mean-difference tests appear in `experiment.md` / HTML but are **not** +> serialized into `experiment.json`. A consumer that wants CIs must recompute them +> from `per_replicate_scores`. + +## `suite.json` — `SuiteRollup` + +Written for dataset-backed suites; its `passed` flag drives the CI exit code. + +| Key | Type | Meaning | +| --- | --- | --- | +| `suite_id` / `variant_id` | `str` | Identity. | +| `rows_total` / `rows_passed` / `rows_failed` / `rows_error` | `int` | Row counts. | +| `pass_rate` | `float` | `rows_passed / rows_total`. | +| `average_weighted_score` | `float \| null` | Mean row score. | +| `criterion_stats` | `list[{criterion_type, rows_evaluated, average_score, error_count}]` | Per-criterion summary. | +| `failed_samples` | `list[FailedRowSummary]` | Capped at 20 (`{row_id, task_id, final_status, weighted_score, failure_reasons, error_message, task_json_relpath, replicate_index}`). | +| `criterion_aggregates` | `list[CriterionAggregate]` | The thresholdable metrics — see below. | +| `passed` | `bool` | All aggregates met their thresholds. | + +### CriterionAggregate & ThresholdCheck + +`CriterionAggregate`: `criterion_type`, `description`, `rows_total`, `rows_excluded`, +`metrics` (a **flat** `dict[str, float]`, e.g. `accuracy`, `macro_f1`, +`precision.yes`, `recall.yes`, `f1.yes`), `threshold_checks`, `passed`, `details` +(untyped render extras — for classification: `labels`, `per_label`, `confusion`), +`error`. + +`ThresholdCheck`: `metric`, `min_value`, `actual_value` (`null` if the aggregator +didn't emit that metric), `passed` (`actual_value >= min_value`). These correspond to +the `suite_thresholds` you set on a criterion — see +[User Guide → Suite Thresholds](USER_GUIDE.md#suite-thresholds--classification-metrics). + +--- + +## TokenUsage + +Serialized fields: `uncached_input_tokens`, `output_tokens`, +`cache_creation_input_tokens`, `cache_read_input_tokens`, `total_cost_usd`, plus a +computed **`input_tokens`** (= sum of the three input buckets). Note: `total_tokens` +is a plain property and is **not** serialized. Cost bills `uncached_input_tokens`, +not `input_tokens`. Legacy files that only have `input_tokens` are adopted as +`uncached_input_tokens` on read. + +## FinalStatus + +String enum values and their reporting category: + +| Value | Category | Icon | +| --- | --- | --- | +| `SUCCESS` | succeeded | `+` | +| `FAILURE` | failed | `-` | +| `TIMEOUT` | failed | `T` | +| `MAX_TURNS_EXHAUSTED` | failed | `M` | +| `TOKEN_BUDGET_EXCEEDED` | failed | `#` | +| `COST_BUDGET_EXCEEDED` | failed | `$` | +| `ERROR` | error | `!` | +| `BUILD_FAILED` | error | `B` | + +> **Gotcha:** `BUILD_FAILED` (a failed Docker image build) categorizes as **error**, +> not failed — easy to miscount downstream. + +--- + +## Consumer gotchas at a glance + +- `run.json.task_results` is a flat, untyped denormalization — the typed source of + truth is each `task.json`. +- Experiment CIs / significance tests are **not** in `experiment.json` (render-time + only); recompute from `per_replicate_scores`. +- Judge `transcript` is stripped from `task.json`; follow `transcript_path`. +- `TokenUsage.total_tokens` is not serialized; sum the buckets (or use the computed + `input_tokens` + `output_tokens` + cache buckets). +- `EarlyStopInfo` presence is itself the "stopped early" signal. + +## See also + +- [User Guide → Output Structure](USER_GUIDE.md#output-structure) and the + [`aggregate`](USER_GUIDE.md#cli-commands) command +- [A/B Experiments → Reading the Report](AB_EXPERIMENTS.md#reading-the-report) +- [Task Definition Guide](TASK_DEFINITION_GUIDE.md) diff --git a/docs/TASK_DEFINITION_GUIDE.md b/docs/TASK_DEFINITION_GUIDE.md index d0baa887..dcf9806f 100644 --- a/docs/TASK_DEFINITION_GUIDE.md +++ b/docs/TASK_DEFINITION_GUIDE.md @@ -394,9 +394,9 @@ All criteria share these fields: | `stop_when` | `null` | Arms this criterion for early stop (`pass`/`fail`/`decided`); requires `run_limits.stop_early: true` and an observable criterion type (`skill_triggered`, `command_executed`). See [`stop_early`](#stop_early-opt-in-early-stop). | **Scoring types:** -- **Binary** (1.0 or 0.0): `file_exists`, `run_command`, `file_matches_regex` +- **Binary** (1.0 or 0.0): `file_exists`, `run_command`, `file_matches_regex`, `classification_match`, `skill_triggered` - **Fractional** (0.0–1.0): `file_contains`, `file_check`, `json_check`, `command_executed`, `uipath_eval` -- **Continuous** (0.0–1.0): `reference_comparison`, `llm_judge`, `agent_judge` +- **Continuous** (0.0–1.0): `reference_comparison`, `commands_efficiency`, `llm_judge`, `agent_judge` **Task success:** all *gating* criteria must score >= their `pass_threshold`. A criterion with `weight: 0` is informational — it is still checked, stored, and @@ -616,6 +616,22 @@ Checks whether the agent executed specific tools/commands during evaluation. Ins **Codex limitation.** Codex agents map `Read`, `Grep`, and `Glob` tools to `shell` commands (they execute via bash), so `tool_name: "Read"` on Codex returns no matches. Use `tool_name: "Bash"` or `tool_name: null` (any tool) for Codex-compatible checks. This criterion works correctly on Claude Code agents, which emit separate `Read`/`Grep`/`Glob` telemetry. +### `commands_efficiency` + +Scores how economically the agent worked, relative to a budget of expected tool calls. **Continuous scoring:** `score = expected_commands / max(actual_commands, expected_commands)` — so a run at or under budget scores `1.0`, and the score decays as the agent takes more calls than expected (e.g. twice the budget → `0.5`). + +```yaml +- type: "commands_efficiency" + expected_commands: 8 # budget of tool calls to complete the task (>= 1) + description: "Agent should solve this in ~8 tool calls" +``` + +| Field | Default | Description | +|-------|---------|-------------| +| `expected_commands` | *required* | Expected number of tool commands to complete the task (integer, `>= 1`). | + +This criterion requires an agent run (it reads `CommandTelemetry`). Pair it with a low `weight` if you want efficiency to *inform* the score without gating pass/fail on its own. + ### `uipath_eval` Evaluates a UiPath agent against a named evaluation set. **Fractional scoring:** metrics passed / total metrics. @@ -640,7 +656,7 @@ Evaluates a UiPath agent against a named evaluation set. **Fractional scoring:** ### `llm_judge` -Have an LLM grade the task against a rubric written in the task YAML. **Continuous scoring** from a JSON verdict `{"score": 0.0-1.0, "rationale": "..."}`; parse failure, non-numeric score, or LLM error all produce `score=0.0` with an `error` populated. +Have an LLM grade the task against a rubric written in the task YAML. **Continuous scoring** from a verdict the judge returns via a forced `submit_verdict` tool call (`{score: 0.0-1.0, rationale: "..."}`) — the model never returns free-form prose. A missing/malformed verdict, a non-numeric score, or an LLM error all produce `score=0.0` with an `error` populated. ```yaml - type: "llm_judge" @@ -657,7 +673,7 @@ Have an LLM grade the task against a rubric written in the task YAML. **Continuo include_dialog: false # Opt-in: include the full user<->agent conversation (recommended for simulation) model: "anthropic.claude-sonnet-4-6" temperature: 0.0 - max_tokens: 1000 + max_tokens: 2000 max_file_chars: 20000 # Per-file content truncation weight: 2.0 pass_threshold: 0.7 @@ -674,7 +690,7 @@ Have an LLM grade the task against a rubric written in the task YAML. **Continuo | `max_dialog_chars` | `80000` | Aggregate cap on dialog text rendered into the judge prompt (per-message cap is `max_file_chars`). When exceeded, trailing turns are dropped and a degraded note is recorded. | | `model` | `anthropic.claude-sonnet-4-6` | Judge model id (vendor-prefixed; auto-translated per backend) | | `temperature` | `0.0` | Sampling temperature (0.0 = deterministic) | -| `max_tokens` | `1000` | Maximum tokens in the judge's response | +| `max_tokens` | `2000` | Maximum tokens in the judge's response | | `max_file_chars` | `20000` | Per-file (and agent_output) truncation applied before building the prompt | **Transport selection.** The judge call is routed by the active `API_BACKEND`: @@ -694,14 +710,14 @@ The `direct`-mode transport is resolved once at startup, logged on the `API rout **Failure modes** — each sets `score=0.0` and populates `error`: -- Non-JSON response from the model (parse failure) -- `score` key missing from the JSON verdict +- The judge never emits the forced `submit_verdict` tool call (no verdict returned) +- `score` key missing from the verdict - `score` is not coercible to float - Judge backend unavailable / network error (handled by `@handle_criterion_errors`) ### `agent_judge` -Spawn a full Claude Code SDK agent as the judge. Unlike `llm_judge` (a single LLM call against a rubric), the judge agent has **tool access** — Bash, Read, Write, Glob, Grep, Edit by default — and runs in an isolated copy of the task sandbox. Use it when functional validation requires executing something (`uip rpa get-errors`, `xmllint`, a test suite) rather than just inspecting file content. +Spawn a full Claude Code SDK agent as the judge. Unlike `llm_judge` (a single LLM call against a rubric), the judge agent has **tool access** — a read-only toolkit of `Bash`, `Read`, `Glob`, `Grep` by default (no `Write`/`Edit`) — and runs in an isolated copy of the task sandbox. Use it when functional validation requires executing something (`uip rpa get-errors`, `xmllint`, a test suite) rather than just inspecting file content. ```yaml - type: "agent_judge" @@ -776,6 +792,30 @@ The judge runs with the evaluator's API credentials and can execute arbitrary Ba - `TurnTimeoutError` (judge exceeded `turn_timeout`) - SDK subprocess failure (e.g. `claude` CLI missing) +### `classification_match` + +Matches a single label the agent wrote to a file against ground truth — the file-based classifier. Reads the file, normalizes the content (strip, and lowercase unless `case_sensitive`), and compares it to `expected_label`. **Binary scoring:** `1.0` on a match, else `0.0`. + +The observed label is the canonical form from `allowed_labels` when the content matches; otherwise `(none)` when the file is missing/empty and `(other)` when the content isn't in the allowed set. Both sentinels are recorded so a suite rollup shows them as real failure classes in the confusion matrix. + +```yaml +- type: "classification_match" + path: "result.txt" # file (relative to sandbox) holding the agent's predicted label + expected_label: "positive" + allowed_labels: [positive, negative] + case_sensitive: false # default: case-insensitive + canonicalized + description: "Sentiment label matches ground truth" +``` + +| Field | Default | Description | +|-------|---------|-------------| +| `path` | *required* | File (relative to sandbox) containing the agent's predicted label. | +| `expected_label` | *required* | Ground-truth label for this row (drive it from `${row.…}` on a dataset-backed task). | +| `allowed_labels` | *required* | Canonical label set (≥1). Content not in this set becomes `(other)`. | +| `case_sensitive` | `false` | When `false`, matching is case-insensitive and labels are canonicalized. | + +Like `skill_triggered`, this criterion emits a `ClassificationCriterionResult`, so on a [dataset-backed task](#task-yaml-structure) the suite aggregator computes accuracy / precision / recall / F1 and a confusion matrix — gate them with `suite_thresholds`. Use `classification_match` when the agent writes its answer to a file (labeling/extraction tasks); use `skill_triggered` when the signal is whether a skill fired. + ### `skill_triggered` Binary classifier: **did the agent engage the target skill during the run?** Agent-agnostic — scans the run's `turn_records` for either signal: Claude's explicit `Skill` tool call whose `skill` parameter matches `skill_name` (namespace prefixes like `plugin:skill` are stripped, so `skill_name: uipath-agents` matches `Skill(skill="uipath-coded-agents:uipath-agents")`), or — for an agent with no `Skill` tool, e.g. Codex — a command that reads the skill's files off disk (a parameter contains `skills//`, matching both the repo path and the `.agents/skills/` symlink). diff --git a/docs/USER_GUIDE.md b/docs/USER_GUIDE.md index 844c040a..627d1107 100644 --- a/docs/USER_GUIDE.md +++ b/docs/USER_GUIDE.md @@ -16,6 +16,7 @@ start with the [tutorials](tutorials/README.md); for the task-file schema see th - [CLI Commands](#cli-commands) - [API Routing & Benchmarking](#api-routing--benchmarking) - [Output Structure](#output-structure) +- [Suite Thresholds & Classification Metrics](#suite-thresholds--classification-metrics) - [Environment Variables](#environment-variables) - [Troubleshooting](#troubleshooting) @@ -38,6 +39,12 @@ coder-eval run tasks/hello_date.yaml --stream full # live LLM output | `-D path=value` / `--set` | Override any resolved task-config field (`agent`/`run_limits`/`sandbox` roots), e.g. `-D run_limits.max_turns=30 -D agent.permission_mode=plan -D agent.sdk_options.effort=high`. Repeatable; schema-validated. This is the way to set permission mode, turn/timeout limits, tools, plugins, and SDK options. | | `--model, -m` | Shorthand alias for `-D agent.model=…` (e.g., `claude-sonnet-4-20250514`) | | `--driver` | Shorthand alias for `-D sandbox.driver=…` (`tempdir` or `docker`) | +| `--type, -T` | Override agent type for all tasks (`claude-code`, `codex`, `antigravity`, or a plugin kind). | +| `--repeats` | Run each `(task, variant)` N times (≥1); overrides experiment/variant `repeats:`. See [Replicates](#replicates). | +| `--resume` | Resume an interrupted run: skip tasks already finalized in `--run-dir` and run the rest, folding prior results into `run.json`. Requires `--run-dir`. A task with *any* final status (incl. FAILED/ERROR) counts as finalized, so resume does **not** retry failures — delete a task's `task.json` to force a re-run. A config mismatch is warned, not refused. | +| `--sample N` | For dataset-backed tasks, run a fixed-seed random N-row sample (reproducible; cheap smoke test). See [BYOD](BYOD.md). | +| `--sample-per-stratum N` | For dataset-backed tasks, keep up to N rows per stratum (`stratify_field`). Overridden by `--sample`. | +| `--include-skipped` | Also run tasks marked `skip: true` in their YAML (off by default so CI keeps excluding them). | | `--exclude-tags` | Skip tasks matching any of these tags (comma-separated) | | `--tags, -t` | Only run tasks matching any of these tags (comma-separated) | | `--experiment, -e` | Experiment definition YAML for multi-variant comparison (default: `experiments/default.yaml`) | @@ -55,6 +62,10 @@ coder-eval plan tasks/*.yaml # validate specific tasks Checks task syntax, required CLI tools, API keys, and schema validity without executing. +| Flag | Description | +| --- | --- | +| `--experiment, -e` | Experiment definition YAML to resolve variants against (default: `experiments/default.yaml`). | + ### `coder-eval evaluate` — test criteria without an agent ```bash @@ -69,15 +80,46 @@ already written. | Flag | Description | | --- | --- | | `--preserve / --no-preserve` | Preserve sandbox after evaluation (default: preserve) | +| `--run-dir` | Custom run directory (default: auto-generated timestamped dir in `runs/`). | | `--verbose, -v` | DEBUG-level logging | ### `coder-eval report` — view results ```bash -coder-eval report runs/latest # view latest run -coder-eval report runs/latest -o summary.md # export to file +coder-eval report runs/latest # view latest run (markdown to stdout) +coder-eval report runs/latest -o summary.md # export markdown to a file +coder-eval report runs/latest --format html # (re)render every task.json as task.html +``` + +The `run` command already writes reports during execution; `report` re-displays or +re-exports them later. For the on-disk layout and field-level schema of the JSON it +reads, see [Output Structure](#output-structure) and the +[Report Schema](REPORT_SCHEMA.md). + +| Flag | Description | +| --- | --- | +| `--output, -o` | Write to a file instead of stdout (markdown). | +| `--format, -f` | `md` (default) or `html`. `html` re-renders each `task.json` under the run dir to a `task.html` beside it (or to `-o` when exactly one task is found). | + +### `coder-eval aggregate` — rebuild `run.json` from task results + +```bash +coder-eval aggregate runs/2026-06-22_14-32-27 # rebuild the summary in place +coder-eval aggregate runs/combined -o runs/combined # aggregate a merged dir ``` +Re-derives the run-level `run.json` + `run.md` from the finalized `task.json` files +already on disk, using the same builder a live run uses. Use it when a run dir's +top-level summary is missing or stale — e.g. after recovering an interrupted run or +combining several run directories. It rebuilds the **run-level summary only**; +per-suite rollups (`suite.json`/`suite.md`) and experiment reports +(`experiment.json`/`experiment.md`) are *not* rebuilt, because the per-row +suite/variant grouping they need is not recoverable from `task.json` alone. + +| Flag | Description | +| --- | --- | +| `--output, -o` | Write `run.json`/`run.md` into this directory instead of the run dir (e.g. a merged output dir). | + ### Claude Code slash commands The project ships [Claude Code custom slash commands](https://docs.anthropic.com/en/docs/claude-code/slash-commands) @@ -133,6 +175,36 @@ Run the same (task, variant) N times via `repeats:` in an experiment YAML or reports aggregate them with bootstrap confidence intervals and (for 2-variant experiments) a paired mean-difference test. Defaults to 1 (no repetition). +For a field-level reference to `run.json`, `variant.json`, `task.json`, and the +suite/experiment rollups, see the [Report Schema](REPORT_SCHEMA.md). + + + +## Suite Thresholds & Classification Metrics + +Any criterion on a **dataset-backed** task (see [Bring Your Own Dataset](BYOD.md)) +can gate the whole suite, not just individual rows. Each criterion's `aggregate()` +emits `count / mean / median / std / min / max`, and classification-style criteria +(`classification_match`, `skill_triggered`) additionally emit accuracy, per-label +precision/recall/F1, and a confusion matrix. Add `suite_thresholds` to require a +minimum for any of those metrics — the run exits non-zero if any gate fails: + +```yaml +success_criteria: + - type: skill_triggered + skill_name: uipath-agents + expected_skill: "${row.expected_skill}" # "" for rows where it shouldn't fire + suite_thresholds: + recall.yes: 0.70 # fired on ≥70% of rows that needed it + precision.yes: 0.80 # ≤20% false activations +``` + +Available metric keys: `accuracy`, `macro_f1`, `weighted_f1`, `micro_f1`, and +per-label `precision.