diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
index 5423e735..fb49ef7d 100644
--- a/.github/workflows/docs.yml
+++ b/.github/workflows/docs.yml
@@ -10,12 +10,17 @@ name: Docs
# Source: "Deploy from a branch" → branch `gh-pages` / `/ (root)`. That's the same
# (legacy, branch-based) configuration uipath-python already runs on.
on:
- push:
- branches: [main]
- paths:
- - "docs/**"
- - "mkdocs.yml"
- - ".github/workflows/docs.yml"
+ # Auto-publish on push is DISABLED: GitHub Pages is not yet enabled for this
+ # repo on uipath.github.io (org owner must switch it on once — Settings → Pages
+ # → Deploy from a branch → gh-pages / root). The published docs currently serve
+ # from coder-eval.com/docs (synced separately by the website), so the gh-pages
+ # deploy is not needed yet. Re-enable by uncommenting the `push:` trigger below.
+ # push:
+ # branches: [main]
+ # paths:
+ # - "docs/**"
+ # - "mkdocs.yml"
+ # - ".github/workflows/docs.yml"
workflow_dispatch:
permissions:
diff --git a/CLAUDE.md b/CLAUDE.md
index 87b531a6..67b5b1eb 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -195,11 +195,17 @@ make format # ruff format
make check # ruff check (lint)
make typecheck # pyright
make test # pytest
-make lint # custom architectural lint rules (CE001–CE025)
+make lint # custom architectural lint rules (CE001+)
make verify # All of the above + coverage check (CI equivalent)
```
-When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001–CE025 pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested.
+When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE027–CE031 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name.)
+
+Adding a user-facing field to one of the models CE030 tracks (`TaskDefinition`, `RunLimits`, `Dataset`, `SimulationConfig` — see `tests/lint/doc_schema_parity.py`) means documenting it in its guide (mention the field name as inline code) or adding an `EXEMPT` entry with a reason it is not user-authored. `make lint` fails otherwise.
+
+**Docs index SSOT.** `nav:` plus `extra.docs_index` (blurbs) in `mkdocs.yml` are the single source of truth for the flat index surfaces — `README.md`'s Documentation table, `docs/index.md`'s "Where to go next" table, and the `## Docs` / `## Tutorials` sections of `docs/llms.txt`. Regenerate all three with `make docs-indexes`; **CE028** fails the build if any drifts, if a nav page lacks a blurb (or vice-versa), or if a `docs/*.md` page is missing from the nav. The website sidebar derives from the same `nav:`. When adding or renaming a docs page, edit `nav:` + `extra.docs_index` and run `make docs-indexes` — never hand-edit the generated tables (they sit between `` / `` markers).
+
+**Anchor slugger convention.** The docs are rendered by three sluggers (GitHub, Starlight/github-slugger on coder-eval.com, and python-markdown/mkdocs), which disagree on headings containing `&` or punctuation (`api-routing--benchmarking` vs `api-routing-benchmarking`). Prefer punctuation-free headings so all three agree; if a heading needs `&`, add a GitHub-form `
}` syntax used by [datasets](DATASETS.md). A `{name}` with no
+ matching entry in `variables` is left in the prompt verbatim rather than
+ raising, so a typo in a variable name fails silently at the *prompt* level.
+ Read the rendered prompt in the report to confirm.
+
+Every mutation model sets `extra="forbid"`, so a misspelled field (`text:`
+instead of `content:`) is a hard `ValidationError` at load — never a silently
+ignored no-op.
+
+When both `defaults.prompt_mutations` and a variant's `prompt_mutations` are set,
+they **compose** — the experiment-defaults mutations apply first, then the
+variant's, on the already-mutated prompt.
## Recipe: Smoke vs. e2e Flavors (Early Stop)
diff --git a/docs/BYOD.md b/docs/BYOD.md
deleted file mode 100644
index 5284d9cd..00000000
--- a/docs/BYOD.md
+++ /dev/null
@@ -1,210 +0,0 @@
----
-description: >-
- Use a custom Docker image with coder_eval — extend the base coder-eval-agent
- image with your own dependencies and tools, then point task configuration at
- it.
----
-
-# Bring Your Own Docker (BYOD)
-
-The BYOD feature allows customers to use custom Docker images that extend the base `coder-eval-agent` image, enabling them to add custom dependencies and tools while maintaining the latest coder-eval codebase.
-
-## Overview
-
-By default, coder-eval uses the official `coder-eval-agent` image built via `make docker-image`. With BYOD, you can:
-
-1. **Extend the base image** with custom dependencies
-2. **Override the image tag** in your task configuration
-3. **Run evaluations** with your custom image while leveraging all coder-eval features
-
-## How It Works
-
-### Architecture
-
-- **`DockerDriverConfig.image`** field now defaults to `DEFAULT_IMAGE_TAG` (e.g., `coder-eval-agent:0.1.0`)
-- **No environment variable override** needed—use task configuration instead
-- **Simple precedence**: Task config → Default image
-
-### Building a Custom Image
-
-Create a `Dockerfile` in your repo:
-
-```dockerfile
-FROM coder-eval-agent:0.1.0
-
-# Install custom dependencies
-RUN apt-get update && apt-get install -y \
- custom-tool \
- another-dependency \
- && rm -rf /var/lib/apt/lists/*
-
-# Add custom scripts or configuration
-COPY custom-script.sh /usr/local/bin/
-RUN chmod +x /usr/local/bin/custom-script.sh
-```
-
-Build the image:
-
-```bash
-docker build -t my-custom-image:0.1.0 .
-```
-
-### Using a Custom Image in Tasks
-
-Update your task YAML to reference the custom image:
-
-```yaml
-task_id: my_task
-description: Task using custom Docker image
-
-sandbox:
- driver: docker
- docker:
- image: my-custom-image:0.1.0 # Your custom image
- network: bridge
- limits:
- timeout: 300
-
-initial_prompt: |
- Use the custom tools installed in my-custom-image to complete this task.
-
-success_criteria:
- - type: file_exists
- path: output.txt
- description: Verify output file was created
-```
-
-## Example: BYOD Smoke Test
-
-The codebase includes a smoke test that demonstrates BYOD:
-
-**Task**: `tasks/byod_smoke_test.yaml`
-- Uses custom image `byod-custom-image:0.1.0`
-- Verifies the custom image was loaded via a marker file
-- Tagged `smoke-pass` for CI/CD pipeline
-
-**Template**: `templates/byod_smoke_test/Dockerfile`
-- Extends `coder-eval-agent:0.1.0`
-- Adds `/opt/byod_marker` to prove custom image was used
-
-Run locally:
-
-```bash
-# Build the custom image
-docker build -t byod-custom-image:0.1.0 templates/byod_smoke_test/
-
-# Run the smoke test
-set -a && source .env && set +a
-coder-eval run tasks/byod_smoke_test.yaml
-```
-
-## Configuration Details
-
-### DockerDriverConfig.image
-
-- **Type**: `str` (not optional)
-- **Default**: `DEFAULT_IMAGE_TAG` (e.g., `coder-eval-agent:0.1.0`)
-- **Override**: Set in task YAML via `sandbox.docker.image`
-
-### Example Configuration Layers
-
-```yaml
-# Default (no override)
-sandbox:
- driver: docker
- # image defaults to coder-eval-agent:0.1.0
-
-# Override at task level
-sandbox:
- driver: docker
- docker:
- image: my-team/image:latest
-
-# Override via CLI (future enhancement)
-# coder-eval run task.yaml --docker-image custom:v1
-```
-
-## Best Practices
-
-1. **Base image consistency**: Always extend from the same `coder-eval-agent` version as your host
- ```dockerfile
- FROM coder-eval-agent:0.1.0 # Match your host version
- ```
-
-2. **Keep images lean**: Add only necessary dependencies
- ```dockerfile
- # Good: Clean up after install
- RUN apt-get update && apt-get install -y tool && rm -rf /var/lib/apt/lists/*
-
- # Avoid: Large intermediate layers
- RUN apt-get update
- RUN apt-get install -y tool
- ```
-
-3. **Document custom tools**: Make it clear what you added
- ```dockerfile
- # Custom tools added for project X
- # - tool-a: used for step 1
- # - tool-b: used for step 2
- RUN apt-get install -y tool-a tool-b
- ```
-
-4. **Version your images**: Use semver tags
- ```bash
- docker build -t my-company/evaluation-agent:1.0.0 .
- docker build -t my-company/evaluation-agent:latest .
- ```
-
-5. **Test locally first**: Verify your image works before adding to CI
- ```bash
- docker build -t test-image:local .
- coder-eval run task.yaml # Ensure task.yaml points to test-image:local
- ```
-
-## Troubleshooting
-
-### Image Not Found
-
-**Error**: `docker: Error response from daemon: pull access denied`
-
-**Solution**: Ensure the image exists and the tag is correct
-```bash
-# List available images
-docker images | grep byod
-
-# Rebuild if missing
-docker build -t byod-custom-image:0.1.0 templates/byod_smoke_test/
-```
-
-### Version Mismatch
-
-**Warning**: `Image coder-eval-agent:0.1.0 != host 0.1.0`
-
-**Solution**: Rebuild the custom image after updating the base image
-```bash
-# Update base image
-make docker-image
-
-# Rebuild custom image
-docker build --no-cache -t my-image:0.1.0 .
-```
-
-## CI/CD Integration
-
-The BYOD feature is tested in the smoke-pass bucket of the e2e-smoke job:
-
-1. Base image is built: `make docker-image`
-2. BYOD template is built: `docker build templates/byod_smoke_test/`
-3. Smoke test runs: `coder-eval run tasks/*.yaml --tags smoke-pass`
-4. Results verified: `byod_smoke_test` should succeed
-
-See `.github/workflows/pr-checks.yml` for the full pipeline.
-
-## Related Files
-
-- **Implementation**: `src/coder_eval/models/sandbox.py` (DockerDriverConfig)
-- **Runner**: `src/coder_eval/isolation/docker_runner.py` (image selection)
-- **Tests**: `tests/test_byod_feature.py`
-- **Example Task**: `tasks/byod_smoke_test.yaml`
-- **Example Dockerfile**: `templates/byod_smoke_test/Dockerfile`
-- **CI/CD**: `.github/workflows/pr-checks.yml` (e2e-smoke, windows-smoke jobs)
diff --git a/docs/CI_GATE.md b/docs/CI_GATE.md
new file mode 100644
index 00000000..72791b69
--- /dev/null
+++ b/docs/CI_GATE.md
@@ -0,0 +1,147 @@
+---
+description: >-
+ Run Coder Eval as a CI gate — the packaged composite GitHub Action, JUnit XML
+ output for test-report ingestion, and an optional per-task score floor.
+---
+
+# CI Gate: GitHub Action & JUnit reports
+
+Coder Eval ships a **packaged CI gate**: a composite GitHub Action that installs
+the CLI, runs your tasks, emits a JUnit XML report, appends the run summary to the
+job summary, and fails the build on any task/gate failure. This page is the
+reference for the Action and the JUnit output. For a step-by-step walkthrough
+(including a hand-rolled workflow), see
+[Tutorial 02 — Running Coder Eval in CI](tutorials/02-ci-pipeline.md).
+
+## The GitHub Action
+
+A composite action lives at the repo root (`action.yml`), so you can reference it
+directly:
+
+```yaml
+- uses: UiPath/coder_eval@v0 # becomes @v1 once 1.0.0 ships; @vX.Y.Z pins exactly
+ with:
+ tasks: tests/tasks/**/*.yaml
+ model: claude-sonnet-5
+ env: |
+ ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
+```
+
+The action is **agent-agnostic** — it installs `coder-eval` but *not* any
+coding-agent runtime. Tasks using the default `claude-code` agent need the
+`claude` CLI on `PATH` (Node + `@anthropic-ai/claude-code`), provided by your job
+*before* this step runs.
+
+### Inputs
+
+| Input | Default | Purpose |
+| --- | --- | --- |
+| `tasks` | *(all `tasks/`)* | Task YAML path(s)/glob passed to `coder-eval run`. |
+| `tags` | — | Only run tasks matching these comma-separated tags (`--tags`). |
+| `model` | — | Override agent model for all tasks (`--model`). |
+| `extra-args` | — | Extra args appended verbatim to `coder-eval run` (`--experiment`, `-D …`, `--exclude-tags`, …). Trusted caller input. |
+| `version` | pinned release | `coder-eval` version to install from PyPI, or `local` to install from the action checkout. |
+| `run-dir` | `runs/ci` | Run directory (`--run-dir`). |
+| `junit-path` | `coder-eval-junit.xml` | Where to write the JUnit XML report. |
+| `step-summary` | `true` | Append `run.md` to the GitHub job summary. |
+| `env` | — | Credential/backend passthrough (see below). |
+| `minimum-task-score` | *(off)* | Optional strict per-task score floor (see below). |
+
+### Outputs
+
+| Output | Description |
+| --- | --- |
+| `run-dir` | The run directory containing `run.json` / `run.md`. |
+| `junit-path` | Path to the written JUnit XML report. |
+
+### Credentials via `env`
+
+**`env` is the sole channel for credentials and backend config.** It takes
+newline-separated `NAME=VALUE` pairs, exported for the `coder-eval` process
+**only** — scoped to the run step, never written to `$GITHUB_ENV`, so a forwarded
+secret can't bleed into later job steps. Names must match
+`^[A-Za-z_][A-Za-z0-9_]*$`; blank lines and `#` comments are ignored. Always wire
+values from repository secrets — never inline a secret literal.
+
+```yaml
+- uses: UiPath/coder_eval@v0
+ with:
+ tasks: tests/tasks/**/*.yaml
+ env: |
+ ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
+ API_BACKEND=direct
+```
+
+Set whatever the run needs — `ANTHROPIC_API_KEY`, `API_BACKEND`, Bedrock/model
+vars, `GEMINI_API_KEY` for Antigravity, `EVALBOARD_*`, plugin paths, etc. See the
+[User Guide → Environment Variables](USER_GUIDE.md#environment-variables) and the
+per-agent guides ([Claude Code](agents/CLAUDE_CODE.md) · [Codex](agents/CODEX.md) ·
+[Antigravity](agents/ANTIGRAVITY.md)) for what each backend needs.
+
+### The score floor (`minimum-task-score`)
+
+An **additional** gate on top of `coder-eval`'s own exit code. Set a float in
+`[0.0, 1.0]` and the step fails if **any** scored task, in any variant, has a
+`weighted_score` below it — *or* if `coder-eval` itself exits non-zero (both
+verdicts surface). It reads the always-written `run.json` spine
+(`task_results[*].weighted_score`), so it works for plain and experiment runs
+alike. Errored tasks (null score) are left to `coder-eval`'s exit code; a
+malformed/`NaN` score fails closed. Empty (the default) disables the floor.
+
+### Security
+
+Evaluated tasks execute agent-generated code. **Do not** run this action under
+`pull_request_target` with secrets exposed to untrusted fork PRs. For untrusted
+tasks, use the [Docker driver](DOCKER_ISOLATION.md) — the `tempdir` driver is not
+a security boundary.
+
+> The README's [Use as a GitHub Action](https://github.com/UiPath/coder_eval#use-as-a-github-action)
+> section carries the same reference alongside a copy-paste workflow.
+
+## JUnit XML output
+
+Any run can emit a JUnit XML report — the lingua franca CI platforms understand for
+per-test annotations, history, and flake tracking. Two entry points, one code path
+(so they can't drift):
+
+```bash
+# During a run
+coder-eval run tasks/*.yaml --junit-xml coder-eval-junit.xml
+
+# After the fact, from a finished run dir
+coder-eval report runs/latest -f junit # writes runs/latest/junit.xml
+coder-eval report runs/latest -f junit -o out.xml # custom path
+```
+
+The report is built **from the finalized run directory on disk** — the `run.json`
+spine (required), any `suite.json` gates (optional), and per-failed-row `task.json`
+for failure detail (best-effort). Each task result maps to a JUnit `testcase`;
+failures/errors carry a (capped) detail body, and statuses map through
+`FinalStatus.category` (`succeeded` / `failed` / `error`). See the
+[Report Schema](REPORT_SCHEMA.md) for the underlying fields.
+
+### Ingesting the report
+
+**GitHub Actions** — [`mikepenz/action-junit-report`](https://github.com/mikepenz/action-junit-report):
+
+```yaml
+- uses: mikepenz/action-junit-report@v5
+ if: always()
+ with:
+ report_paths: coder-eval-junit.xml
+```
+
+**Azure DevOps** — `PublishTestResults@2`:
+
+```yaml
+- task: PublishTestResults@2
+ inputs:
+ testResultsFormat: 'JUnit'
+ testResultsFiles: 'coder-eval-junit.xml'
+```
+
+## See also
+
+- [Tutorial 02 — Running Coder Eval in CI](tutorials/02-ci-pipeline.md) — the walkthrough
+- [Report Schema](REPORT_SCHEMA.md) — the JSON the JUnit report is built from
+- [User Guide](USER_GUIDE.md) — the `run` / `report` commands and environment variables
diff --git a/docs/DATASETS.md b/docs/DATASETS.md
new file mode 100644
index 00000000..498addab
--- /dev/null
+++ b/docs/DATASETS.md
@@ -0,0 +1,276 @@
+---
+description: >-
+ Fan one Coder Eval task out over a dataset — inline rows or JSONL files, row
+ substitution into prompts and criteria, sampling, and suite-level scoring
+ across every row.
+---
+
+# Bring Your Own Dataset
+
+A `dataset:` block turns **one** task file into **N** independent evaluation tasks — one per row.
+Write the prompt and the success criteria once, parameterize them with `${row.}`, and point
+the task at a list of rows. This is how Coder Eval runs benchmark-sized suites (classification sets,
+skill-activation corpora, regression fixtures) without duplicating YAML.
+
+The `dataset:` field is documented field-by-field in the
+[Task Definition Guide](TASK_DEFINITION_GUIDE.md#dataset). This page is the how-and-why.
+
+## How fan-out works
+
+Expansion happens in `orchestration/task_loader.expand_dataset` at load time, **before** experiment
+variant resolution. For each row:
+
+- `task_id` becomes `/` — e.g. `sentiment-classification/r1`.
+- `suite_id` is set to the original `task_id`, and `row_id` to the row's identifier. Both are set by
+ the expander; you never author them.
+- `dataset` is cleared on the expanded task, so nothing re-expands downstream.
+- Every row-task runs in its **own sandbox** and writes its **own run directory** and `task.json`,
+ exactly like a hand-written task.
+
+Because expansion runs before variant resolution, **an experiment variant cannot override the
+dataset** — every arm of an A/B sees the identical row set, which is what makes the comparison fair.
+
+Rows that share a `suite_id` roll up into a per-suite `suite.json` / `suite.md` alongside the normal
+run report. The [Report Schema](REPORT_SCHEMA.md) has the field-level contract in its `suite.json`
+section.
+
+## Two row sources
+
+Exactly one of `rows` or `paths` must be set. Both, or neither, raises at load time.
+
+### Inline rows
+
+Best for a handful of rows that belong with the task:
+
+```yaml
+dataset:
+ rows:
+ - id: alpha
+ expected: "alpha"
+ - id: beta
+ expected: "beta"
+```
+
+### JSONL files
+
+Best for real datasets. Each path is **relative to the task YAML's directory** (not your working
+directory), and multiple paths are concatenated into one row stream in declared order:
+
+```yaml
+dataset:
+ paths:
+ - "datasets/sentiment.jsonl"
+ - "datasets/sentiment_extra.jsonl"
+```
+
+Each line is one JSON object — the same shape as an inline row:
+
+```json
+{"id": "r1", "text": "This product is fantastic, I love it.", "expected": "positive"}
+```
+
+## Substituting row values
+
+`${row.}` placeholders are replaced with that row's value in exactly two places:
+
+1. the task's `initial_prompt`, and
+2. every **string leaf** of every `success_criteria` entry (at any nesting depth).
+
+Nothing else is substituted — not the task-level `description`, not `pre_run`, not `sandbox`. Keep
+row-varying data in the prompt and the criteria. (A *criterion's* own `description` is a string leaf
+of `success_criteria`, so it **is** substituted — see the example below.)
+
+```yaml
+initial_prompt: "Classify this text: ${row.text}"
+
+success_criteria:
+ - type: "classification_match"
+ path: "result.txt"
+ expected_label: "${row.expected}"
+ allowed_labels: [positive, negative]
+ description: "Predicted label matches expected for row ${row.id}"
+```
+
+Two failure modes, both raised at load time rather than producing a silently wrong prompt:
+
+- a placeholder naming a field the row doesn't have → `KeyError: ${row.foo}: key not found
+ (available: [...])`;
+- a placeholder resolving to a list or dict → `TypeError: ${row.foo}: value must be a scalar`.
+
+A field whose value is `null` substitutes as the empty string. That is the shape a classification
+suite wants for a negative row — "this row has no expected label" — though such rows are more often
+written with an explicit `""`.
+
+## Row identifiers
+
+`id_field` (default `id`) names the row field used as the row's identity. That value becomes a
+directory name under the run dir, so it is validated: it must match
+`^[A-Za-z0-9_][A-Za-z0-9_.\-]*$` — letters, digits, underscore, hyphen, dot, not starting with a
+dot or hyphen — and must be unique across the dataset.
+
+```yaml
+dataset:
+ id_field: "case_id" # rows carry `case_id` instead of `id`
+ paths: ["datasets/cases.jsonl"]
+```
+
+Load-time errors, with their message shapes:
+
+| Situation | Message |
+| --- | --- |
+| No rows at all | `Dataset for task '' is empty` |
+| Row lacks the id field | `Dataset row for task '' missing id_field '': ` |
+| Id has illegal characters | `Dataset row id '' must match ^[A-Za-z0-9_][A-Za-z0-9_.\-]*$ (letters, digits, underscore, hyphen, dot)` |
+| Two rows share an id | `Duplicate dataset row id for task '': ''` |
+| Both/neither `rows` and `paths` | `Dataset must specify either 'paths' or 'rows'` / `... only one of ...` |
+| `paths: []` | `Dataset.paths must be a non-empty list` |
+
+## Sampling a subset
+
+A full dataset is expensive. Two independent mechanisms cut it down, and **`--sample` wins whenever
+both apply**:
+
+1. **`--sample N` (CLI only)** — a flat uniform-random N rows over the whole dataset. Fixed seed, so
+ the same N rows come back every run: a reproducible, cheap smoke flavor of a big suite. Unlike a
+ first-N slice it is unbiased across concatenated `paths`. If `N` is greater than or equal to the
+ dataset size, the whole dataset runs — not an error.
+2. **Stratified: `--sample-per-stratum N` (CLI) or `dataset.sample_per_stratum: N` (YAML)** — keep up
+ to N rows per stratum, where a row's stratum is the value of `stratify_field` (default
+ `expected_skill`). Strata with N rows or fewer are taken whole. The CLI flag overrides the YAML
+ value, so a runner can cap a dataset without editing the task. Rows missing `stratify_field` fall
+ into the `""` stratum. This is the right sampler for classification suites, where a uniform draw
+ would starve rare strata.
+
+!!! warning "Stratified sampling is nondeterministic by default"
+
+ The stratified draw is seeded **only** by `dataset.sample_seed`. When `sample_seed` is unset the
+ sample is deliberately **re-drawn on every run** — whether the per-stratum count came from the
+ YAML `sample_per_stratum` or the CLI `--sample-per-stratum` flag. That is intentional: the
+ nightly activation suite relies on it to broaden coverage across runs rather than re-measuring
+ the same rows forever.
+
+ Set `sample_seed` to an integer to pin a reproducible sample — do that when you are comparing
+ two runs and need the row set held constant (a regression bisect, an A/B where you want the
+ identical rows in both arms on separate invocations). Leave it unset for recurring suites where
+ coverage over time matters more than run-to-run comparability. Note the contrast with
+ `--sample N`, which is fixed-seed and reproducible by default.
+
+## Suite-level scoring
+
+Per-row pass/fail is rarely the number you care about on a dataset — the suite metric is. Every
+criterion's `aggregate()` emits `count / mean / median / std / min / max`, and classification-style
+criteria (`classification_match`, `skill_triggered`) additionally emit accuracy, per-label
+precision / recall / F1, and a confusion matrix.
+
+Add `suite_thresholds` to a criterion to gate the whole suite on those aggregated metrics; the run
+exits non-zero if any gate fails:
+
+```yaml
+success_criteria:
+ - type: "classification_match"
+ path: "result.txt"
+ expected_label: "${row.expected}"
+ allowed_labels: [positive, negative]
+ description: "Predicted label matches expected"
+ suite_thresholds:
+ accuracy: 0.8
+ macro_f1: 0.75
+```
+
+Full metric-key list and the rollup outputs:
+[User Guide → Suite Thresholds](USER_GUIDE.md#suite-thresholds--classification-metrics) and
+[Task Definition Guide → `skill_triggered`](TASK_DEFINITION_GUIDE.md#skill_triggered).
+
+## Worked example: inline rows
+
+`tasks/dataset_example.yaml` — two rows, one criterion, expands to
+`dataset-example/alpha` and `dataset-example/beta`:
+
+```yaml
+task_id: "dataset-example"
+description: "Demonstrates dataset fan-out: one task file, N sub-tasks, one per row."
+initial_prompt: "Write the string '${row.expected}' into out.txt. Say nothing else."
+tags: [smoke, smoke-pass, dataset]
+
+agent:
+ type: "claude-code"
+ permission_mode: "acceptEdits"
+ allowed_tools: ["Write"]
+
+dataset:
+ rows:
+ - id: alpha
+ expected: "alpha"
+ - id: beta
+ expected: "beta"
+
+success_criteria:
+ - type: "file_check"
+ path: "out.txt"
+ includes: ["${row.expected}"]
+ description: "out.txt contains '${row.expected}'"
+```
+
+## Worked example: JSONL plus suite thresholds
+
+`tasks/sentiment_classification.yaml` reads its rows from a JSONL file and gates the suite on
+aggregated classification metrics:
+
+```yaml
+task_id: "sentiment-classification"
+description: "Binary sentiment classification over a 3-row dataset with one intentionally mislabeled ground truth, to exercise non-trivial P/R/F1."
+tags: [classification]
+
+agent:
+ type: "claude-code"
+ permission_mode: "acceptEdits"
+ allowed_tools: ["Write"]
+
+# Row 3 is intentionally mislabeled (text is clearly positive but expected=negative).
+# A correct model produces: accuracy 2/3, recall(positive)=1/2, recall(negative)=1/1,
+# precision(positive)=1/1, precision(negative)=1/2, f1(positive)=f1(negative)=2/3.
+dataset:
+ paths:
+ - "datasets/sentiment.jsonl"
+
+initial_prompt: |
+ Classify the sentiment of the following text as exactly one word: 'positive' or 'negative'.
+ Write only that single word (lowercase, no punctuation, no other text) to a file named result.txt.
+
+ Text: "${row.text}"
+
+success_criteria:
+ - type: "classification_match"
+ path: "result.txt"
+ expected_label: "${row.expected}"
+ allowed_labels: [positive, negative]
+ description: "Predicted label matches expected for row ${row.id}"
+ # Suite-level thresholds: evaluated against the aggregated metrics.
+ # With the one intentionally mislabeled row the expected numbers are
+ # accuracy=0.667, macro_f1=0.667, recall.positive=1.0.
+ suite_thresholds:
+ accuracy: 0.6
+ macro_f1: 0.5
+ recall.positive: 0.9
+```
+
+Its rows live in `tasks/datasets/sentiment.jsonl`:
+
+```json
+{"id": "r1", "text": "This product is fantastic, I love it.", "expected": "positive"}
+{"id": "r2", "text": "Terrible service, would not recommend.", "expected": "negative"}
+{"id": "r3", "text": "Amazing quality and excellent value for money.", "expected": "negative"}
+```
+
+Run it, or a cheap sample of it:
+
+```bash
+coder-eval run tasks/sentiment_classification.yaml # every row
+coder-eval run tasks/sentiment_classification.yaml --sample 2 # a reproducible 2-row sample
+```
+
+## See also
+
+- [Task Definition Guide](TASK_DEFINITION_GUIDE.md#dataset) — the `dataset:` field reference.
+- [Report Schema](REPORT_SCHEMA.md) — the `suite.json` field contract.
+- [A/B Experiments](AB_EXPERIMENTS.md) — running a dataset across multiple variants.
diff --git a/docs/DIALOG_MODE.md b/docs/DIALOG_MODE.md
new file mode 100644
index 00000000..e9697a82
--- /dev/null
+++ b/docs/DIALOG_MODE.md
@@ -0,0 +1,237 @@
+---
+description: >-
+ Evaluate coding agents in conversation, not one-shot — Coder Eval's dialog
+ mode drives a simulated user with a persona and a goal, so you can measure
+ clarifying questions, requirement discovery, and multi-turn recovery.
+---
+
+# Dialog Mode: multi-turn user simulation
+
+Most evaluation harnesses hand an agent one perfect prompt and grade the artifact. Real users don't
+write perfect prompts. They ask for something vague, answer questions, change their mind, and hold
+back requirements until asked.
+
+Dialog mode replaces the single-shot loop with a conversation between the coding agent and a
+**simulated user** — a second LLM driven by a persona and a goal. It is the only way to measure the
+behaviors that only exist in conversation.
+
+## When to use it
+
+Reach for dialog mode when the thing you want to score is conversational:
+
+- **Does the agent ask?** A good agent asks a clarifying question instead of guessing when the
+ request is underspecified. Give the simulator a goal that withholds a requirement and see whether
+ the agent surfaces it.
+- **Requirement discovery.** Score whether the agent eventually extracts everything it needs across
+ several turns, not whether it one-shots a fully-specified prompt.
+- **Multi-turn recovery.** The agent goes down the wrong path, the user corrects it — does it
+ recover, or double down?
+- **Prompt robustness.** A persona that is terse, impatient, or non-technical is a much harsher test
+ than your hand-tuned `initial_prompt`.
+
+Stay in single-shot mode when the task is genuinely fire-and-forget. Dialog mode multiplies cost and
+adds variance; don't pay for it when the extra turns measure nothing.
+
+## How it works
+
+Add an enabled `simulation:` block to any task and the orchestrator swaps its iteration loop for a
+dialog driver:
+
+```yaml
+simulation:
+ enabled: true
+ persona: |
+ You are a user with one specific request. You give the request only when
+ asked, and then you wait for the agent to do it.
+ goal: |
+ Get the agent to echo your string back verbatim. Reveal the string only
+ when the agent asks what you want.
+ max_turns: 4
+```
+
+The mechanics:
+
+- The task's `initial_prompt` is the user's **opening message**. The simulator takes over from turn
+ two, seeing the agent's last reply and producing the next user utterance.
+- `initial_prompt` is **optional** in dialog mode. Omit it and the simulator writes the opener too,
+ from its persona and goal — closer to a cold-start user.
+- Each exchange is one user message plus the agent's full response (the agent may make many tool
+ calls inside a single exchange).
+- The simulator is a **tools-disabled Claude Code agent** with `allowed_tools: []`, an explicit
+ deny-list, and no plugins or settings sources. It is pure text-in / text-out, and it **cannot see
+ the sandbox** — no files, no terminal, no agent reasoning. Only what the agent writes in the chat.
+- The simulator shares the coding agent's resolved `ApiRoute`, so backend and model come from the
+ run's routing (`--backend direct` / `--backend bedrock`) rather than from the `simulation:` block.
+ There is no model field here to set.
+
+**Agent-kind constraint.** The *subject* agent can be any registered kind — the dialog driver only
+calls the agent's `communicate()`, so Codex and plugin agents work. The *simulator*, however, is
+always a Claude Code agent. A dialog run therefore needs the `claude` CLI on `PATH` and working
+Anthropic or Bedrock credentials **even when the agent under test is not Claude**.
+
+**Early stop is not available here.** [`run_limits.stop_early`](TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop)
+is single-shot-only; arming it alongside `simulation.enabled` is a hard error at resolution time, not
+a silent no-op. Use `stop_on_criteria_pass` (below) for the dialog equivalent.
+
+Every field's default and constraint lives in one place — the
+[Task Definition Guide's simulation section](TASK_DEFINITION_GUIDE.md#simulation-multi-turn-user-dialog).
+This page deliberately does not restate them.
+
+## Designing the persona and goal
+
+`persona` is *who is talking*; `goal` is *what they want*. Keep them separate — a persona that
+smuggles in the requirements defeats the point.
+
+The most useful pattern is **withholding**. State the full goal for the simulator, then instruct it
+not to volunteer part of it:
+
+```yaml
+persona: |
+ A non-technical business analyst who knows the outcome they want but not how
+ automation works. Mildly impatient, answers questions directly.
+goal: |
+ Build a flow that reads invoice PDFs from an Outlook folder, extracts
+ vendor/amount/date, and posts the result to Google Sheets.
+ Do NOT volunteer the Google Sheets requirement unless the agent asks where
+ the output should go.
+constraints:
+ - "Do not paste code — you cannot read code."
+ - "If the agent goes silent for two turns, ask 'are you still there?'."
+```
+
+Now the task measures something a single-shot prompt cannot: whether the agent asks about the output
+destination at all. Pair it with a criterion that checks the Sheets integration exists and you have a
+clean signal.
+
+`constraints` are behavioral rules, not more requirements. Good ones bound *how* the simulator
+speaks ("no code", "one question at a time", "reveal X only if asked"). Resist writing constraints
+that quietly hand the agent the answer.
+
+## How a dialog ends
+
+After each exchange the driver evaluates the stop conditions **in this order**, first match wins:
+
+1. **`run_limits` breach** (`run_limit_exceeded`) — a subject-agent budget cap tripped. The
+ end-of-dialog criteria still run for partial credit, then the task **aborts**.
+2. **`stop_on_criteria_pass`** (`criteria_passed`) — every success criterion passes. Requires
+ per-turn checking (`check_criteria: every_turn` or `both`); pairing it with the default
+ `end_of_dialog` is rejected at load time, since there would be nothing to check against.
+3. **`max_turns`** (`max_turns`) — the hard cap on exchanges. The agent exhausting its *own* inner
+ `max_turns` mid-exchange ends the dialog with the same reason.
+4. **`max_total_tokens`** (`budget`) — the dialog-wide budget across simulator **and** agent. The
+ dialog ends and the task is **still scored** — unlike
+ [`run_limits.max_total_tokens`](TASK_DEFINITION_GUIDE.md#run-limits), which covers the subject
+ agent only and aborts.
+5. **`stop_token`** (`stop_token`) — only if none of the above fired is the simulator asked for
+ another message; the sentinel token in *that fresh utterance* ends the dialog. This is the
+ workhorse in practice — the simulator decides, in character, that it got what it wanted — but it
+ is evaluated **last**, so a turn that trips `max_turns` or the budget never gets the chance to
+ produce it.
+
+A simulator call that raises ends the dialog with `error` (and increments
+`simulation.simulator_failures`). The reason is recorded as `simulation.stop_reason` on the result,
+so you can tell a conversation that finished from one that merely ran out of turns.
+
+> **`check_criteria: every_turn` multiplies criterion cost.** Criteria are re-evaluated after every
+> exchange, so an `llm_judge` or `agent_judge` criterion in a 12-turn dialog runs twelve times.
+> That is often more expensive than the dialog itself. Use `end_of_dialog` (the default) unless you
+> genuinely need early stop or a per-turn record; `both` gives you the per-turn trace *and* an
+> authoritative final check.
+
+## Trials and variance
+
+A single dialog is one sample from a stochastic process — the simulator improvises as much as the
+agent does. `n_trials` runs the conversation N independent times per (task, variant):
+
+```yaml
+simulation:
+ enabled: true
+ n_trials: 5
+```
+
+Each trial is fanned out as its own unit of work with its own sandbox, its own run directory
+(`runs/////`, zero-padded index), and its own `task.json`. On the result,
+`simulation.replicate_index` and `simulation.n_trials` identify the trial, and reports render a
+"Trial *k* of *N*" row. Trials reuse the replicate machinery, so when a simulation task sets
+`n_trials > 1` it takes precedence over experiment-level `repeats`.
+
+Trials are ordinary batch units: they run concurrently exactly as far as `--max-parallel` allows —
+there is no per-task trial-concurrency switch; scheduling is entirely `--max-parallel`'s job.
+
+## Grading a dialog
+
+Set `include_dialog: true` on every `llm_judge` / `agent_judge` criterion in a simulated task. This
+is not optional polish — without it the judge sees only the agent's outputs and no user side, so a
+requirement the user stated in turn 3 looks like something the agent invented. Judges routinely
+score that as a hallucination.
+
+```yaml
+success_criteria:
+ - type: llm_judge
+ description: "Agent satisfied the requirement the user gave mid-conversation"
+ prompt: |
+ Read the DIALOG block. Grade the agent on whether it satisfied the
+ requirement the user actually stated — not what you assume the task is.
+ include_dialog: true
+ pass_threshold: 0.9
+```
+
+The rendered dialog is wrapped as `UNTRUSTED DATA` with a rubric guard telling the judge that claims
+made only by the *simulated user* may be invented, and not to penalize the agent for going along
+with them. That guard is what keeps a chatty simulator from poisoning the grade.
+
+## What it costs
+
+Budget roughly `max_turns × n_trials` agent turns per (task, variant) — the worst case, since the
+dialog usually stops on the stop token first. On top of that:
+
+- **Simulator tokens**, one generation per turn. Small next to the agent's, but not free, and they
+ are reported separately as `simulation.simulator_input_tokens` / `simulator_output_tokens`.
+- **Criterion cost**, multiplied by turn count under `check_criteria: every_turn` or `both`.
+
+Cap it with [`run_limits`](TASK_DEFINITION_GUIDE.md#run-limits) — but note the accounting boundary:
+`run_limits` budgets count the **subject agent only**, so simulator spend is invisible to
+`max_total_tokens` and `max_usd`. To bound the conversation as a whole, use
+`simulation.max_total_tokens`, which counts both sides.
+
+## Dialog mode as an experiment axis
+
+Everything on the `simulation:` block is overridable per variant, which makes the conversation itself
+the thing under test. A variant sets a partial `simulation:` block that shallow-merges onto the
+task's:
+
+```yaml
+variants:
+ - variant_id: patient-user
+ simulation:
+ persona: "A patient engineer who answers questions precisely."
+ - variant_id: terse-user
+ simulation:
+ persona: "A terse, impatient user who gives one-line answers."
+```
+
+Productive axes: persona (terse vs. chatty, technical vs. not), how much the goal withholds, and
+`n_trials` as a cost/confidence trade. See [A/B Experiments](AB_EXPERIMENTS.md) for the full
+experiment layer.
+
+## Security posture
+
+The task's `reference` solution is **never** passed to the simulator, exactly as it is never passed
+to the coding agent. A simulator that could read the reference would leak it into the dialog and the
+agent would be graded against an answer it was handed.
+
+What the simulator *does* see, all inlined into its system prompt: `persona`, `goal`, `constraints`,
+the task's **`description`**, and the pinned `initial_prompt` when there is one. The `description` is
+easy to forget — it reads like evaluator-private metadata, but the simulator gets it verbatim. If
+your persona withholds a requirement, don't spell that requirement out in `description`, or the
+simulator may volunteer what you meant it to keep back.
+
+The simulator also runs with no tools, no plugins, and no settings sources, so it cannot read or
+write the sandbox even if its persona instructs it to.
+
+## See also
+
+- [Task Definition Guide → Simulation](TASK_DEFINITION_GUIDE.md#simulation-multi-turn-user-dialog) —
+ every field, default, and constraint.
+- [A/B Experiments](AB_EXPERIMENTS.md) — running personas as variants.
+- [Run Limits](TASK_DEFINITION_GUIDE.md#run-limits) — the caps that bound a dialog's cost.
diff --git a/docs/DOCKER_ISOLATION.md b/docs/DOCKER_ISOLATION.md
index 6deba4a5..d3076229 100644
--- a/docs/DOCKER_ISOLATION.md
+++ b/docs/DOCKER_ISOLATION.md
@@ -1,6 +1,6 @@
---
description: >-
- Run each coder_eval task in its own fresh Docker container — strong host
+ Run each Coder Eval task in its own fresh Docker container — strong host
isolation, a pinned reproducible agent runtime, and custom images for
task-specific dependencies.
---
@@ -51,6 +51,63 @@ sandbox:
image: my-custom:tag # override the default image
```
+## Using a pre-built custom image
+
+When your tasks need extra tools or dependencies, extend the framework image once and point tasks at
+the result — no per-task build. The custom image must extend `coder-eval-agent:` so it
+inherits the coder-eval runtime, the entrypoint script, and the `org.coder-eval.version` label the
+host's preflight check reads. (For a task whose Dockerfile can't be rebased onto Debian, use
+[the runtime kit](#tasks-that-bring-their-own-base-image-the-runtime-kit-coder-eval-runtime)
+instead. To have coder-eval build the image per task rather than pre-building it, see
+[Building the image from a task Dockerfile](#building-the-image-from-a-task-dockerfile).)
+
+```dockerfile
+FROM coder-eval-agent: # match the version your host runs
+RUN apt-get update && apt-get install -y --no-install-recommends custom-tool \
+ && rm -rf /var/lib/apt/lists/*
+```
+
+```bash
+docker build -t my-team/image:latest .
+```
+
+Select it either in the task YAML:
+
+```yaml
+sandbox:
+ driver: docker
+ docker:
+ image: my-team/image:latest
+```
+
+…or from the CLI, which overrides whatever the YAML says:
+
+```bash
+coder-eval run task.yaml -D sandbox.docker.image=my-team/image:latest
+```
+
+The default when you set nothing is `coder-eval-agent:`.
+
+A worked example ships in-tree: `tasks/byod_smoke_test.yaml` runs against
+`templates/byod_smoke_test/Dockerfile`, which extends the framework image and drops a marker file
+that the task's success criterion then asserts — proving the custom image was actually used. (The
+`byod_*` names here mean "Bring Your Own **Docker**"; they are unrelated to the
+[Bring Your Own Dataset](DATASETS.md) guide, which is about fanning one task out over data rows.)
+
+```bash
+make docker-image # base image first
+docker build -t byod-custom-image:0.1.0 templates/byod_smoke_test/ # then the derived one
+coder-eval run tasks/byod_smoke_test.yaml
+```
+
+### Troubleshooting custom images
+
+| Symptom | Cause and fix |
+| --- | --- |
+| `docker: Error response from daemon: pull access denied` | The image isn't built locally and isn't pullable. Check `docker images`, then rebuild it. Docker treats an unknown local tag as a remote reference, which is why the error mentions a pull. |
+| `Image coder_eval != host ` | The custom image carries an `org.coder-eval.version` label inherited from a stale framework base. Rebuild the base with `make docker-image`, then rebuild your derived image with `docker build --no-cache`. |
+| `Image has no org.coder-eval.version label` | The image doesn't descend from `coder-eval-agent` (or predates the label). Rebase it on the framework image, or use the runtime kit. |
+
## Building the image from a task Dockerfile
Instead of pointing at a pre-built `image`, a task can ship its own `Dockerfile`
diff --git a/docs/EXTENDING.md b/docs/EXTENDING.md
new file mode 100644
index 00000000..c0eb1693
--- /dev/null
+++ b/docs/EXTENDING.md
@@ -0,0 +1,239 @@
+---
+description: >-
+ Extend Coder Eval with a custom agent, a custom success criterion, or model
+ pricing — the plugin SPI (coder_eval.plugins entry-point group), the Agent ABC
+ checklist, the @register_criterion decorator, and register_pricing.
+---
+
+# Extending Coder Eval
+
+Coder Eval is extensible along three seams, all designed so a third party can add
+capability **without editing the base package**:
+
+1. **Agents** — register a new `agent.type` via the plugin SPI.
+2. **Criteria** — add a new success-criterion type via a decorator + auto-discovery.
+3. **Pricing** — contribute USD rates for models your plugin runs.
+
+This guide covers all three. For the internal architecture notes see
+[CLAUDE.md](https://github.com/UiPath/coder_eval/blob/main/CLAUDE.md).
+
+---
+
+## 1. Custom agents (the plugin SPI)
+
+Agents register through the **`coder_eval.plugins` entry-point group** — there is no
+closed enum or dispatch to edit. `agent.type` is an open string validated against
+the `AgentRegistry`; built-in and third-party agents travel the exact same path.
+
+### Wire up the entry point
+
+In your plugin package's `pyproject.toml`:
+
+```toml
+[project.entry-points."coder_eval.plugins"]
+my_plugin = "my_plugin:register"
+```
+
+At CLI init, `load_plugins()` imports each entry point and calls it with the
+`AgentRegistry` **class** (not an instance). A third-party hook that raises is logged
+and skipped; only a failing *built-in* registration is fatal.
+
+### The `register` hook
+
+```python
+from coder_eval.agents.registry import AgentRegistry
+
+def register(registry: type[AgentRegistry]) -> None:
+ # Bind type string → config class → agent class.
+ registry.register("my-agent", MyAgentConfig)(MyAgent)
+ # Optionally contribute pricing here too (see §3):
+ # register_pricing(MY_RATES)
+```
+
+`AgentRegistry.register(agent_kind, config_class)` returns a decorator, so the
+decorator form works too:
+
+```python
+@AgentRegistry.register("my-agent", MyAgentConfig)
+class MyAgent(Agent[MyAgentConfig]):
+ ...
+```
+
+Registration is **anti-shadow**: re-registering the same `(agent_class,
+config_class)` pair is a no-op, but claiming an existing `agent.type` with a
+*different* implementation raises `ValueError`. Two plugins can never silently fight
+over one type.
+
+### The config class
+
+Subclass `BaseAgentConfig` with your own `type` discriminator:
+
+```python
+from typing import Literal
+from coder_eval.models import BaseAgentConfig # importable from coder_eval.models
+
+class MyAgentConfig(BaseAgentConfig):
+ type: Literal["my-agent"] = "my-agent"
+ my_option: str = "default"
+```
+
+The factory `create_agent(kind, config, …)` raises `TypeError` if the passed config
+isn't an instance of the registered `config_class`, so keep them paired.
+
+### The `Agent` ABC — implementation checklist
+
+Implement these three abstract methods:
+
+- [ ] `async def start(self, working_directory, *, env_path_prepend=None, plugin_tools_dir=None) -> None`
+- [ ] `async def communicate(self, user_input, *, stream_callback=None, timeout=None, max_turns=None, should_stop=None) -> TurnRecord`
+- [ ] `async def stop(self) -> None`
+
+Optional overrides (sensible defaults exist): `kill()`, `kill_sync()` (called from a
+non-asyncio watchdog thread — must **not** await), `discard_pending_turn()`.
+
+Follow the shared turn lifecycle (do **not** hand-assemble a `TurnRecord`):
+
+- [ ] Call `self._begin_turn()` at the top of `communicate()`.
+- [ ] Call `self._end_turn_ok()` on the success path.
+- [ ] Call `self._mark_stopped()` in `stop()` after your own teardown.
+- [ ] Before raising on a mid-turn failure, set `self.pending_turn` to a
+ `crashed=True` `TurnRecord` (built from an `EventCollector`), then raise
+ `AgentCrashError` / `TurnTimeoutError` (bare — no payload). The orchestrator
+ drains it and calls `discard_pending_turn()`.
+
+Emit the standardized event protocol (you are the **sole emitter**): one
+`AgentStartEvent` at the top of `communicate()` and one matching `AgentEndEvent` on
+**every** exit path (emit from `finally`), a `TurnStart`/`TurnEnd` pair per inner
+turn, and `ToolStart`/`ToolEnd` per tool call (close orphaned tools with
+`status=unresolved`). Fan events through an internal `EventCollector` — it builds the
+returned `TurnRecord`, the single agent-agnostic capture path.
+
+Set `supports_cooperative_stop: ClassVar[bool] = True` only if your `communicate()`
+actually honors `should_stop` (needed for `run_limits.stop_early`). Leaving it
+`False` means early stop is rejected at resolution for your agent — which is correct
+if you can't stop cooperatively.
+
+### Worked example
+
+The in-tree worked example is the BYOA test fixture at
+[`tests/fixtures/byoa_demo_plugin/`](https://github.com/UiPath/coder_eval/tree/main/tests/fixtures/byoa_demo_plugin)
+— a minimal package with a `register` hook and the `coder_eval.plugins` entry point.
+(It subclasses `ClaudeCodeAgent` for brevity; a real third-party agent implements the
+`Agent` ABC from scratch.)
+
+---
+
+## 2. Custom success criteria
+
+Criteria are discovered automatically by `pkgutil` scan of the `coder_eval.criteria`
+package — a new checker just needs the `@register_criterion` decorator. Adding one is
+two steps: a Pydantic **model** (the YAML schema) and a **checker** (the logic).
+
+### Step 1 — the model
+
+In `models/criteria.py`, subclass `BaseSuccessCriterion` and set the discriminator as
+a `Literal` default, then add it to the `SuccessCriterion` union:
+
+```python
+class MyCriterion(BaseSuccessCriterion):
+ type: Literal["my_criterion"] = "my_criterion"
+ target: str
+ # Set requires_agent = True (ClassVar) if you read turn_records.
+
+# ...and add `| MyCriterion` to the SuccessCriterion discriminated union.
+```
+
+Union membership is required — a run validates that every union member's `type` has a
+registered checker, and rejects unknown `type` tags in YAML. `BaseSuccessCriterion`
+gives you `description`, `weight` (default 1.0; `0` = informational/non-gating),
+`pass_threshold` (default 0.9), `stop_when`, and `suite_thresholds` for free, with
+`extra="forbid"` so YAML typos are caught.
+
+### Step 2 — the checker
+
+Drop a file in `coder_eval/criteria/` with a `@register_criterion` class:
+
+```python
+from coder_eval.criteria.base import BaseCriterion, register_criterion
+from coder_eval.models import CriterionResult
+
+@register_criterion
+class MyChecker(BaseCriterion[MyCriterion]):
+ criterion_type = "my_criterion" # must match the model discriminator
+
+ def _check_impl(self, criterion, sandbox, reference_code=None, *,
+ turn_records=None, context=None) -> CriterionResult:
+ ok = ... # your logic
+ return CriterionResult(
+ criterion_type=self.criterion_type,
+ description=criterion.description,
+ score=1.0 if ok else 0.0,
+ details="...",
+ )
+```
+
+Notes:
+
+- **Do not override `check()`** — it's final and wraps `_check_impl` with error
+ handling (an exception becomes a score-0.0 result with the error captured).
+- Return `score` in `[0.0, 1.0]` — binary criteria use `0.0`/`1.0`; fractional ones
+ anything in between.
+- For **suite-level metrics** on dataset-backed tasks, override
+ `aggregate(criterion, per_row_results)`; the base emits
+ `count/mean/median/std/min/max`, so your criterion is suite-thresholdable for free.
+ Classification-style criteria return a `ClassificationCriterionResult` and layer
+ accuracy / precision / recall / F1 / confusion on top.
+- For **early stop**, implement `live_verdict(...)` and declare
+ `live_stop_polarities` — a lint rule keeps the two consistent.
+
+> A duplicate `criterion_type` **overwrites** the earlier checker with a warning (not
+> a hard error, unlike agents) — keep type strings unique.
+
+---
+
+## 3. Model pricing
+
+Plugins that run their own models contribute USD rates through `register_pricing` —
+there is **no** separate entry-point group; call it from the same `register()` hook.
+
+```python
+from coder_eval.pricing import ModelPricing, register_pricing
+
+# Rates are per MILLION tokens: (input, output, cache_write, cache_read)
+MY_RATES = {
+ "my-model-v1": ModelPricing(3.0, 15.0, 3.75, 0.30),
+ "my-free-model": ModelPricing(0.0, 0.0, 0.0, 0.0), # a valid free-model entry
+}
+
+def register(registry):
+ registry.register("my-agent", MyAgentConfig)(MyAgent)
+ register_pricing(MY_RATES)
+```
+
+Behavior:
+
+- **Keys** are the bare model id as it appears in `agent.model` (vendor/Bedrock
+ region prefixes like `eu.` / `anthropic.` are normalized off at lookup).
+- **Idempotent** for identical rates; **raises `ValueError`** on a *conflicting* rate
+ for an existing key (built-in or another plugin). Registration is all-or-nothing —
+ a late conflict leaves nothing half-applied.
+- **Zero rates are valid** (a genuinely free model) — the lookup uses "is a rate
+ present", not truthiness, so an all-zero entry prices to `0.0` rather than falling
+ through to the built-in table.
+- The plugin overlay is consulted **before** the built-in table, so every consumer
+ (agents, reports, the cost simulator) prices your model transparently via
+ `calculate_cost(model, uncached_input, output, cache_creation=0, cache_read=0)`.
+ (Pass `uncached_input`, not the total input.)
+
+The base package ships **no** plugin rates; only the built-in table.
+
+---
+
+## See also
+
+- [Claude Code](agents/CLAUDE_CODE.md) · [Codex](agents/CODEX.md) ·
+ [Antigravity](agents/ANTIGRAVITY.md) — the built-in agents, each registered via
+ this same SPI
+- [Task Definition Guide](TASK_DEFINITION_GUIDE.md) — the criterion catalogue
+- [CLAUDE.md](https://github.com/UiPath/coder_eval/blob/main/CLAUDE.md) — architecture
+ and extension points in depth
diff --git a/docs/IDEAS.md b/docs/IDEAS.md
deleted file mode 100644
index a9e7a018..00000000
--- a/docs/IDEAS.md
+++ /dev/null
@@ -1,459 +0,0 @@
-# Ideas & Backlog
-
-A lightweight, stack-ranked list of improvements for `coder_eval` (the eval harness,
-the dashboard/CI, and the evalboard UI). Anyone can add an idea — keep it short, then
-we triage and work the list top-down.
-
-## How to stack rank
-
-**Rank lives in one place: row order in the table below. Top row = do first.**
-To re-rank, just move a row up or down — nothing else changes.
-
-- Each idea has a **stable ID** (e.g. `expected-turns`) that never changes, even when
- the rank does. Headings and cross-references use the ID, not a position number, so
- reordering never means renumbering.
-- To decide *where* a row goes, score it on **Impact** and **Effort** (1–5 each), then
- let **Score** rank it:
-
- > **Score = Impact + (6 − Effort)** — range 2–10, higher floats up. Sort by Score;
- > break ties with judgment (dependencies, momentum, who's free).
-
- - **Impact** = how much this helps the *skills people are building* or our
- organization's *ability to improve* them. Bigger lever = higher.
- - **Effort** = order-of-magnitude **tokens a coding agent burns to build it**
- (best guess is fine — it's a log scale, so just pick the nearest power of ten).
-
-| Field | 1 | 2 | 3 | 4 | 5 |
-|-------|---|---|---|---|---|
-| **Impact** | trivial | minor | useful | important | game-changing |
-| **Effort** (tokens) | ~100K | ~1M | ~10M | ~100M | ~1B |
-
-**Status:** 💡 idea · 🔍 scoping · 🚧 in progress · ✅ done · ❄️ parked
-
-## Backlog
-
-| ID | Idea | Impact | Effort | Score | Status | Owner |
-|----|------|:------:|:------:|:-----:|:------:|-------|
-| [`expected-turns`](#expected-turns) | Top-level Expected Turns score | 4 | 2 | 8 | 🚧 | Sherif |
-| [`efficiency-columns`](#efficiency-columns) | Surface already-computed efficiency metrics on the board | 4 | 2 | 8 | 💡 | — |
-| [`config-fingerprint`](#config-fingerprint) | Per-run config fingerprint + regression attribution | 4 | 2 | 8 | 💡 | — |
-| [`judge-ensemble`](#judge-ensemble) | Multi-sample judge voting + verdict variance | 4 | 2 | 8 | 💡 | — |
-| [`cost-preflight`](#cost-preflight) | Pre-flight budget validation + judge cost estimate | 3 | 1 | 8 | 💡 | — |
-| [`long-failing`](#long-failing) | PR a fix for long-running failed tests | 4 | 3 | 7 | 💡 | — |
-| [`failure-mode-rollup`](#failure-mode-rollup) | Failure-mode taxonomy headline | 4 | 3 | 7 | 💡 | — |
-| [`judge-calibration`](#judge-calibration) | Golden-set judge calibration + drift detection | 4 | 3 | 7 | 💡 | — |
-| [`result-cache`](#result-cache) | Skip-unchanged task result cache | 4 | 3 | 7 | 💡 | — |
-| [`smart-judge-truncation`](#smart-judge-truncation) | Smarter judge context truncation | 3 | 2 | 7 | 💡 | — |
-| [`per-criterion-telemetry`](#per-criterion-telemetry) | Per-criterion timing + token telemetry | 3 | 2 | 7 | 💡 | — |
-| [`top-offenders`](#top-offenders) | Top Offenders page | 3 | 3 | 6 | 💡 | — |
-| [`diff-criterion`](#diff-criterion) | Diff/patch validation criterion | 3 | 3 | 6 | 💡 | — |
-| [`ux-revamp`](#ux-revamp) | UX revamp | 4 | 4 | 6 | 💡 | — |
-
----
-
-### expected-turns
-**Top-level Expected Turns score**
-
-**What:** Roll today's *per-task* expected-turns signal up into a single headline
-metric for a run (e.g. "% of tasks within expected turns", or mean turn overage),
-surfaced on the dashboard and in the nightly Slack summary alongside pass-rate.
-
-**Why:** A task can pass but take far more turns than budgeted — a real efficiency
-regression that the pass/fail headline hides. We already compute the per-task signal;
-we just don't aggregate or surface it at the top.
-
-**What exists today:**
-- `expected_turns_overage(result)` and `visible_turn_count(result)` —
- [src/coder_eval/reports_stats.py:174](../src/coder_eval/reports_stats.py#L174)
-- Per-task "expected_turns exceeded" badge in the HTML report —
- [src/coder_eval/reports_html.py:341](../src/coder_eval/reports_html.py#L341)
-- `run_limits.expected_turns` is the per-task budget source.
-
-**Rough approach:** aggregate overage across a run → add a field to the run summary →
-render on the evalboard + add one line to the nightly Slack summary.
-
-**Open questions:**
-- What's the right headline statistic — % within budget, mean overage, or count exceeded?
-- Per-skill breakdown too, or run-level only to start?
-
----
-
-### long-failing
-**PR a fix for long-running failed tests**
-
-**What:** Identify eval tasks/tests that consistently run long *and* fail (burning the
-nightly's 4h budget for no signal), then open PRs to fix or quarantine them.
-
-**Why:** Long failing tasks eat wall-clock and cost on every nightly run and add noise.
-Tightening or fixing them improves both run time and the trustworthiness of the headline.
-
-**Rough approach:** mine recent `runs/` for tasks with high duration + non-SUCCESS status →
-rank worst offenders → per-task, either fix the underlying issue or adjust limits/quarantine →
-PR each fix.
-
-**Open questions:**
-- Threshold for "long-running" — absolute seconds, or top-N by duration?
-- Fix vs. quarantine vs. limit-tuning — decide per task or set a default policy?
-- One-off cleanup, or a recurring report we act on each cycle?
-
----
-
-### top-offenders
-**Top Offenders page**
-
-**What:** A single page that ranks the things most worth addressing first — worst
-tasks/skills by failure rate, turn overage, duration, and cost — so triage starts from
-"here's what to fix" instead of scrolling a full run.
-
-**Why:** Today you read the whole board to find problems. A focused "top offenders"
-view turns the metrics we collect (and the ones in `expected-turns` and `long-failing`)
-into an actionable work queue, and gives this backlog a data-driven feed.
-
-**Rough approach:** aggregate per-task/per-skill stats across the latest run (reuse the
-signals from [reports_stats.py](../src/coder_eval/reports_stats.py)) → rank by a few axes →
-render as a sortable list on the evalboard, linking each row to its task detail.
-
-**Open questions:**
-- Which axes and what default sort — failures first, or a blended "needs-attention" score?
-- Latest-run only, or trend across recent runs to catch persistent offenders?
-- Standalone page or a panel on the run overview? (overlaps with `ux-revamp`)
-
----
-
-### ux-revamp
-**UX revamp**
-
-**What:** Refresh the evalboard UI ([evalboard/](../evalboard/), Next.js) — clearer run
-overview, easier drill-down from suite → skill → task, better surfacing of the metrics
-above (pass-rate, cost, turns).
-
-**Why:** The board is the daily touchpoint for reading nightly results; a clearer UX
-makes regressions and outliers faster to spot.
-
-**Open questions (needs scoping before it's actionable):**
-- What are the top 2–3 jobs-to-be-done on the board today that are painful?
-- Incremental polish vs. a rethink of the information architecture?
-- Any design input, or engineer's-judgment pass?
-
----
-
-### efficiency-columns
-**Surface already-computed efficiency metrics on the board**
-
-**What:** We already compute per-task efficiency signals and write them into
-`run.json`, but the evalboard never shows them. Add columns/badges for
-`commands_efficiency`, a `max_turns_exhausted` status badge, total tokens, and a
-derived cache-hit-rate (`cache_read / (cache_creation + cache_read)`) — and show
-`expected_commands` next to actual so over-budget tasks are obvious at a glance.
-
-**Why:** Cost and efficiency regressions are invisible today. The data is *already
-in the artifact* — this is pure surfacing, not new computation, so it's a cheap,
-high-leverage win that complements [`expected-turns`](#expected-turns).
-
-**What exists today:**
-- `eval_result_to_task_dict()` writes `commands_efficiency`, `max_turns_exhausted`,
- `expected_commands`, and `total_tokens` into each task dict —
- [src/coder_eval/reports_experiment.py:42](../src/coder_eval/reports_experiment.py#L42)
-- The grid recomputes/displays only a subset (turns, cost, raw token buckets) —
- [evalboard/app/runs/[id]/task-grid.tsx](../evalboard/app/runs/[id]/task-grid.tsx)
-- Turn-budget coloring already exists as a model — [evalboard/lib/turns.ts](../evalboard/lib/turns.ts)
-
-**Rough approach:** widen the evalboard `RawTaskResult` type to read the existing
-fields → add sortable columns + a `max_turns_exhausted` badge → reuse the
-`turns.ts` color-ratio pattern for an efficiency/cache cell.
-
-**Open questions:**
-- Which metrics earn a column vs. a hover/detail-panel stat (avoid grid sprawl)?
-- Cache-hit-rate: compute in the loader, or add it to `run.json` once so the
- Slack summary can use it too?
-
----
-
-### config-fingerprint
-**Per-run config fingerprint + regression attribution**
-
-**What:** Stamp each run with a stable fingerprint of the inputs that actually drive
-results (resolved task config + agent/model + skills/cli SHAs + framework version),
-and add a board view that diffs two runs' fingerprints. When pass-rate moves,
-answer "did the model change, the prompt change, or the task config change?"
-
-**Why:** Today a regression between nightlies is a whodunit — you can't separate a
-model bump from a prompt rewrite from a config drift. The raw material (config
-lineage, component SHAs) is already captured per task; nothing ties it into a
-run-level, diffable fingerprint.
-
-**What exists today:**
-- Per-task `task_config` + dotted-path config lineage —
- [src/coder_eval/reports_experiment.py:42](../src/coder_eval/reports_experiment.py#L42)
-- Component SHAs (coder_eval / skills / cli) already in the run summary and Slack
- footer — the nightly Slack summary
-- Single declarative merge resolver produces the resolved config —
- [src/coder_eval/orchestration/config_merge.py](../src/coder_eval/orchestration/config_merge.py)
-
-**Rough approach:** hash the resolved-config + SHA bundle into `run.json` → add a
-"compare runs" panel on the evalboard that renders the field-level diff → optionally
-flag pass-rate deltas whose fingerprint is *identical* (i.e. pure agent nondeterminism).
-
-**Open questions:**
-- Fingerprint at run level, per-skill, or per-task (tasks can differ within a run)?
-- What's the canonical diff unit — resolved YAML, or the lineage entries?
-
----
-
-### judge-ensemble
-**Multi-sample judge voting + verdict variance**
-
-**What:** Let `llm_judge` (and optionally `agent_judge`) run N independent samples
-per row and aggregate — median/mean score plus a variance/disagreement signal —
-instead of trusting a single one-shot verdict. Surface verdict spread so a
-"flaky judge" is visible rather than silently noisy.
-
-**Why:** Subjective scoring underpins a large share of tasks, but every judged score
-today is a single sample at temperature 0 with no measure of its own reliability. A
-judge that would score the same artifact 0.4 / 0.8 / 0.6 looks authoritative. Voting
-reduces variance; reporting spread tells us *which rubrics are unreliable*.
-
-**What exists today:**
-- Single-sample verdict via forced `submit_verdict` tool call; temperature frozen at
- parse time — [src/coder_eval/criteria/llm_judge.py](../src/coder_eval/criteria/llm_judge.py)
-- `JudgeVerdict` (score / rationale / findings) with score clamped to [0,1] —
- [src/coder_eval/models/judge.py](../src/coder_eval/models/judge.py)
-- Per-criterion `aggregate()` already exists for suite rollups —
- [src/coder_eval/criteria/base.py:192](../src/coder_eval/criteria/base.py#L192)
-
-**Rough approach:** add `samples: int` (+ aggregation rule) to the judge criteria →
-run N calls (concurrently) → fold into one `JudgeCriterionResult` carrying mean +
-stddev + raw scores → expose stddev as a suite-thresholdable metric.
-
-**Open questions:**
-- Default N and temperature — fixed (e.g. 3 @ T=1.0) or per-criterion?
-- Does verdict spread gate the suite, or is it report-only to start?
-- Cost ceiling: cap samples on `agent_judge` (expensive) differently from `llm_judge`?
-
----
-
-### cost-preflight
-**Pre-flight budget validation + judge cost estimate**
-
-**What:** Validate run-time budget config *before* a run starts, and estimate judge
-token cost before each judge fires. Two concrete fixes: (1) error (or loudly warn)
-when `max_usd` is set on a route that can't report cost, instead of silently
-skipping the budget; (2) estimate the judge context's token footprint so a giant
-file list / dialog can't blow out context unexpectedly.
-
-**Why:** `max_usd` silently no-ops on routes without per-turn cost data — you think
-you're capped and you're not, and you only find out at result-finalization. Judges
-also burn tokens with zero pre-flight check. Both are cheap guardrails against
-surprise cost.
-
-**What exists today:**
-- Cost budget skipped with a one-time warning when no per-turn cost is available —
- [src/coder_eval/orchestrator.py:640](../src/coder_eval/orchestrator.py#L640)
-- `cost_data_available` recorded *post-hoc* in environment info —
- [src/coder_eval/orchestrator.py:516](../src/coder_eval/orchestrator.py#L516)
-- Judge context assembled (files + reference + dialog) with no token estimate —
- [src/coder_eval/evaluation/judge_context.py](../src/coder_eval/evaluation/judge_context.py)
-
-**Rough approach:** add a pre-run validator that cross-checks `run_limits.max_usd`
-against the resolved route's cost capability (error or explicit warn) → add a rough
-token estimate to `JudgeContextBuilder` and log/cap it before dispatch.
-
-**Open questions:**
-- Hard error vs. warn for `max_usd` on a cost-blind route?
-- Estimate-only, or auto-trim context to fit a configured ceiling?
-
----
-
-### failure-mode-rollup
-**Failure-mode taxonomy headline**
-
-**What:** Roll the *reasons* tasks failed into a run-level headline — counts of
-`max_turns_exhausted`, `TOKEN_BUDGET_EXCEEDED`, `COST_BUDGET_EXCEEDED`, crashes/
-timeouts, and the top review tags (e.g. `prompt-gap`) — surfaced in `run.json`, the
-Slack summary, and the run page. Turn "62 failures" into "31 prompt-gap, 18
-max-turns, 9 crashes, 4 budget."
-
-**Why:** The pass/fail headline says *how many* failed, never *why*. The categories
-already exist across the codebase (final-status enums, per-task flags, review tags)
-but nothing aggregates them into one actionable breakdown — the input that makes
-[`top-offenders`](#top-offenders) and triage useful.
-
-**What exists today:**
-- `FinalStatus.TOKEN_BUDGET_EXCEEDED` / `COST_BUDGET_EXCEEDED` and `max_turns_exhausted`
- per task — [src/coder_eval/reports_experiment.py:42](../src/coder_eval/reports_experiment.py#L42)
-- Error categorization for crashes/timeouts —
- [src/coder_eval/errors/categorization.py](../src/coder_eval/errors/categorization.py)
-- Per-task review tags indexed in `review_index.json` —
- the nightly Slack summary
-
-**Rough approach:** aggregate per-task statuses + flags + tags into a `failure_modes`
-block on the run summary → add one section to the Slack summary → render as a
-breakdown bar on the run page.
-
-**Open questions:**
-- Status enum + flags only, or blend in the (looser) review-tag taxonomy?
-- Fixed buckets, or data-driven top-N categories per run?
-
----
-
-### judge-calibration
-**Golden-set judge calibration + drift detection**
-
-**What:** A small, version-controlled set of artifacts with hand-assigned "correct"
-scores per rubric, plus a command that runs the judges against it and reports
-agreement (e.g. MAE / rank correlation vs. the golden scores). Run it in CI so a
-prompt or model change that silently shifts judge behavior trips a gate.
-
-**Why:** Judges have no ground truth to tune against — we can't tell a good rubric
-from a lenient one, and a model upgrade can re-baseline every judged score without
-anyone noticing. A golden set turns "trust me" into a measurable, regression-tested
-property and complements [`judge-ensemble`](#judge-ensemble) (variance) with
-*accuracy*.
-
-**What exists today:**
-- LLM + agent judges produce structured `JudgeVerdict`s —
- [src/coder_eval/evaluation/judge_verdict.py](../src/coder_eval/evaluation/judge_verdict.py)
-- Suite-level aggregation + thresholds already gate on metrics —
- [src/coder_eval/reports.py](../src/coder_eval/reports.py)
-- No fixture of graded reference verdicts exists anywhere in the repo.
-
-**Rough approach:** curate ~10–20 graded artifacts per judge rubric under `tests/`
-→ add a `coder-eval judge-calibrate` command that scores them and reports
-agreement → wire a CI gate on a minimum agreement threshold.
-
-**Open questions:**
-- Per-rubric golden sets, or one shared spread of easy/medium/hard cases?
-- Agreement metric — MAE, Spearman rank, or pass/fail confusion at the threshold?
-
----
-
-### result-cache
-**Skip-unchanged task result cache**
-
-**What:** Cache completed task results keyed by a fingerprint of everything that
-determines the outcome (resolved task config + agent/model + skills/cli SHA +
-template/sandbox inputs). On the next run, tasks whose fingerprint is unchanged
-*and* deterministic can be reused instead of re-executed — dramatically cutting
-nightly wall-clock and cost.
-
-**Why:** Every nightly re-runs every task from scratch even when nothing relevant
-changed, burning the 4h budget and real money. A content-addressed cache lets the
-nightly spend its budget on what actually changed. (Pairs naturally with
-[`config-fingerprint`](#config-fingerprint), which produces the same key.)
-
-**What exists today:**
-- Batch runner already supports *resume* — partition prior results, re-run only
- failed/skipped — [src/coder_eval/orchestration/batch.py](../src/coder_eval/orchestration/batch.py)
-- No inter-run result cache; runs are fully independent.
-- Agent runs are stochastic, so caching must be opt-in / fingerprint-gated.
-
-**Rough approach:** reuse the [`config-fingerprint`](#config-fingerprint) key →
-store last-good results in a content-addressed store → on run start, short-circuit
-tasks with a cache hit (respecting a `--no-cache` / replicate-count override) →
-clearly mark reused vs. fresh in `run.json`.
-
-**Open questions:**
-- Does caching even make sense given agent nondeterminism — only for deterministic
- criteria, or never reuse the *agent* run and only cache *grading*?
-- Where does the cache live (local dir vs. blob) and how is it invalidated/expired?
-
----
-
-### smart-judge-truncation
-**Smarter judge context truncation**
-
-**What:** Replace the naïve "drop trailing turns when over budget" dialog truncation
-with a strategy that preserves high-signal context — keep first + last K turns, or
-middle-ellipsis — so the judge never loses the final user question on a long
-simulation. There's already a TODO in the code calling for exactly this.
-
-**Why:** On long dialogs, FIFO-dropping the *most recent* turns can discard the
-decisive exchange the rubric needs, quietly degrading verdict quality. The fix is
-local and well-scoped, and it directly improves judge reliability.
-
-**What exists today:**
-- `JudgeContextBuilder` truncation with a standing TODO ("keep first+last K, or
- middle-ellipsis ... the naïve cap is enough for the common case") —
- [src/coder_eval/evaluation/judge_context.py](../src/coder_eval/evaluation/judge_context.py)
-- Per-file cap + aggregate dialog cap (`max_dialog_chars`, default 80K), with
- degradation already marked in the context notes.
-
-**Rough approach:** implement a first+last-K (or middle-ellipsis) selector behind
-the existing truncation point → record which turns were elided in the degradation
-note → keep the naïve cap as a fallback option.
-
-**Open questions:**
-- First+last-K vs. middle-ellipsis vs. relevance-scored selection — start simple?
-- Same strategy for per-file truncation, or dialog-only for now?
-
----
-
-### per-criterion-telemetry
-**Per-criterion timing + token telemetry**
-
-**What:** Record wall-clock time and (for judges) token/cost spend per criterion, so
-we can answer "which criterion is slow/expensive?" Today criteria — especially the
-two judge types — are a black box inside the per-task total.
-
-**Why:** When a task is slow or pricey, there's no way to attribute it to a specific
-criterion. Judges in particular can dominate cost, and we can't see it. This is the
-instrumentation that makes [`long-failing`](#long-failing) and cost triage precise
-rather than guesswork.
-
-**What exists today:**
-- `CriterionResult` carries score/details but no timing or token attribution —
- [src/coder_eval/models/results.py](../src/coder_eval/models/results.py)
-- Judge token usage is captured at the *task* level (via the run's backend / SDK),
- not per-criterion — [src/coder_eval/orchestrator.py](../src/coder_eval/orchestrator.py)
-- `SuccessChecker` dispatches criteria sequentially —
- [src/coder_eval/evaluation/checker.py](../src/coder_eval/evaluation/checker.py)
-
-**Rough approach:** wrap each criterion check in a timer in `SuccessChecker` →
-add `duration_ms` (+ optional `tokens`/`cost_usd` for judges) to `CriterionResult`
-→ surface a per-criterion mini-breakdown on the task detail page.
-
-**Open questions:**
-- Token attribution for judges — exact (per-call delta) or best-effort?
-- Worth running independent criteria concurrently while we're in here, or out of scope?
-
----
-
-### diff-criterion
-**Diff/patch validation criterion**
-
-**What:** A first-class criterion that validates the *change* an agent made — expected
-files touched/untouched, hunks present, lines added/removed within bounds — by
-diffing the sandbox against its starting state. Built for incremental-edit tasks
-where "did it modify the right things, and only those?" matters more than final
-file content.
-
-**Why:** Coding agents are increasingly graded on edits, not greenfield files. Today
-that's only checkable via brittle `file_contains` substrings or custom `run_command`
-shell. A diff-aware criterion captures edit intent directly and discourages
-over-broad changes (a real failure mode).
-
-**What exists today:**
-- 17 criterion types, none diff-aware; closest is `file_contains` / `file_matches_regex` —
- [src/coder_eval/models/criteria.py](../src/coder_eval/models/criteria.py)
-- Sandbox starts from templates/starter files, so a baseline-vs-final diff is
- available — [src/coder_eval/sandbox.py](../src/coder_eval/sandbox.py)
-- Plugin registry makes adding a checker mechanical (`@register_criterion`) —
- [src/coder_eval/criteria/__init__.py](../src/coder_eval/criteria/__init__.py)
-
-**Rough approach:** add a `diff_check` model to the criteria union → checker computes
-the sandbox-vs-baseline diff and scores against expected changed-files / hunk
-patterns / add-remove bounds → fractional score so partial edits get partial credit.
-
-**Open questions:**
-- Compare against git baseline, a snapshot of starter files, or a reference patch?
-- Score on file-set match, hunk match, or both (weighted)?
-
----
-
-## How to add an idea
-
-1. Pick a short, stable **ID** (kebab-case) and add a `### ` section using the
- template above (What / Why / rough approach / open questions).
-2. Score it on Impact and Effort (1–5), compute **Score = Impact + (6 − Effort)**.
-3. Insert a row in the **Backlog** table at the position its Score implies — that
- position *is* its rank. Done.
diff --git a/docs/REPORT_SCHEMA.md b/docs/REPORT_SCHEMA.md
new file mode 100644
index 00000000..3aceab87
--- /dev/null
+++ b/docs/REPORT_SCHEMA.md
@@ -0,0 +1,256 @@
+---
+description: >-
+ Field-level reference for Coder Eval's JSON outputs — run.json, task.json,
+ variant.json, experiment.json, and suite.json — plus the token/criterion
+ telemetry sub-models and FinalStatus values, for anyone consuming a run.
+---
+
+# Report Schema
+
+Coder Eval writes machine-readable JSON alongside every markdown/HTML report. This
+page is the field-level reference for consumers (dashboards, CI parsers, evalboard
+forks). For the on-disk directory tree see
+[User Guide → Output Structure](USER_GUIDE.md#output-structure); for how to
+re-generate these files see [`coder-eval report` / `aggregate`](USER_GUIDE.md#cli-commands).
+
+All JSON is Pydantic `model_dump_json` output — keys are the model field names
+verbatim (no aliases, except `iterations` also accepts the legacy key `turns` on
+read). Times are ISO-8601.
+
+## Who writes what
+
+| File | Model | When |
+| --- | --- | --- |
+| `run.json` / `run.md` | `RunSummary` | Every run (and rebuildable via `coder-eval aggregate`) |
+| `///task.json` | `EvaluationResult` | One per replicate |
+| `//suite.json` / `.md` | `SuiteRollup` | Dataset-backed suites only |
+| `experiment.json` / `.md` | `ExperimentResult` | Every run (experiment layer) |
+| `/variant.json` / `.md` | `VariantAggregate` | Per variant |
+
+`` is the zero-padded replicate index. Judge transcripts spill to sibling files
+(e.g. `judge-0.yaml`) referenced by a `transcript_path`.
+
+---
+
+## `run.json` — `RunSummary`
+
+**`run.json` is flat, not a `run → variant → task → replicate` tree.** It is the
+run-level summary; full per-replicate detail lives in each `task.json`.
+
+| Key | Type | Meaning |
+| --- | --- | --- |
+| `run_id` | `str` | Timestamp id, e.g. `"2025-10-09_15-30-45"`. |
+| `start_time` / `end_time` | `datetime` | Run window. |
+| `total_duration_seconds` | `float` | Wall-clock. |
+| `tasks_run` | `int` | Total replicates executed. |
+| `tasks_succeeded` / `tasks_failed` / `tasks_error` | `int` | Category counts. **Invariant:** the three sum to `tasks_run`. |
+| `tasks_token_budget_exceeded` / `tasks_cost_budget_exceeded` | `int` | Sub-counters of `tasks_failed` (not part of the invariant). |
+| `skipped_tasks` | `list[{path, reason}]` | Load failures / `skip: true` opt-outs. |
+| `max_parallel` | `int` | Concurrency used. |
+| `task_results` | `list[dict]` | Flat per-replicate rows — see below. |
+| `framework_version` | `str` | Coder Eval version chip. |
+| `environment_info` | `dict` | Version/dependency info (may nest, e.g. `tool_plugins`). |
+
+### `task_results[]` — the flat per-task row
+
+Each entry is an **untyped dict** (a denormalization, not a Pydantic model) with keys
+including: `task_id`, `replicate_index`, `variant_id`, `status`
+([`FinalStatus`](#finalstatus)), `weighted_score`, `duration`, `iteration_count`,
+`tags`, `task_path`, `model_used`, `reference_similarity`, the token buckets
+(`input_tokens` = uncached input, `output_tokens`, `cache_creation_input_tokens`,
+`cache_read_input_tokens`, `total_tokens`), `total_cost_usd`, `expected_commands`,
+`actual_commands`, `commands_efficiency`, `agent_config`, `sdk_options`,
+`installed_tools`, turn accounting (`total_turns`, `visible_turns`, `expected_turns`,
+`max_turns_exhausted`, `has_final_reply`), and early-stop fields (`stopped_early`,
+`early_stop_reason`, `turns_remaining_at_stop`). `iterations` here is a **reduced**
+turn digest (`{iteration, duration_seconds, command_count, assistant_turn_count,
+crashed, crash_reason}`) — the full transcript is in `task.json`.
+
+---
+
+## `task.json` — `EvaluationResult`
+
+The authoritative per-replicate record.
+
+**Identity/metadata:** `task_id`, `task_description`, `variant_id` (default
+`"default"`), `agent_type`, `model_used`, `started_at`, `completed_at`,
+`duration_seconds`.
+
+**Results:**
+
+| Key | Type | Meaning |
+| --- | --- | --- |
+| `final_status` | [`FinalStatus`](#finalstatus) | Terminal status. |
+| `weighted_score` | `float \| null` | Weighted average of criterion scores, 0.0–1.0. |
+| `max_turns_exhausted` | `bool` | Ran out of turns. |
+| `iteration_count` | `int` | Number of turns. |
+| `success_criteria_results` | `list[CriterionResult]` | Per-criterion results — see [below](#criterionresult). |
+
+**Transcript:** `iterations: list[TurnRecord]` (accepts legacy alias `turns`) — see
+[TurnRecord](#turnrecord).
+
+**Errors** (populated on failure): `error_message`, `error_details`,
+`error_log_tail` (carries the Docker build-log tail for `BUILD_FAILED`).
+
+**Config/environment:** `environment_info`, `agent_config`, `sdk_options` (raw
+`ClaudeAgentOptions` dump), `sandbox_path`, `task_config`
+(`{resolved, source_yaml, source_file, lineage}` — `lineage` maps each field to
+`{value, source, source_detail}` so you can trace which config layer set it).
+
+**Telemetry/totals:** `total_token_usage` ([TokenUsage](#tokenusage)),
+`command_stats` (`CommandStatistics`), `total_assistant_turns`, `expected_commands` /
+`actual_commands` / `commands_efficiency`, `pre_run_results` / `post_run_results`
+(`{command, exit_code, stdout, stderr, duration_seconds, error}`), `simulation`
+(dialog-mode telemetry, `null` in single-shot), and `early_stop`
+([EarlyStopInfo](#earlystopinfo)).
+
+### CriterionResult
+
+A discriminated union on `result_kind` (`basic` / `judge` / `classification`); legacy
+files without `result_kind` are inferred from `criterion_type`. Base fields
+(`result_kind="basic"`):
+
+`criterion_type`, `description`, `score` (0.0–1.0), `details`, `error`,
+`pass_threshold` (default 0.9), `gating` (default `true`; `false` = informational /
+weight-0, excluded from the score and the pass/fail gate). The base allows extra
+fields so subclass keys round-trip.
+
+- **`classification`** adds `observed_label`, `expected_label` (sentinels like
+ `(none)` / `(other)` allowed). Emitted by `classification_match`, `skill_triggered`.
+- **`judge`** adds `findings`, `token_usage` (kept distinct from the agent total), and
+ `transcript_path` (a sibling `judge-N.yaml`). The full `transcript` is **stripped
+ from `task.json`** — read it from the referenced file. Emitted by `llm_judge`,
+ `agent_judge`.
+
+### TurnRecord
+
+`iteration`, `user_input`, `agent_output`, `commands` (`list[CommandTelemetry]`),
+`timestamp`, `duration_seconds`, `token_usage`, `model_used`, `assistant_turn_count`,
+`messages` (`list[TranscriptMessage]`, discriminated on `role`:
+`user`/`assistant`/`reconciliation`), `num_turns`, `max_turns_exhausted`,
+`result_summary` (`{is_error, subtype, stop_reason, result}`), `crashed`,
+`crash_reason`.
+
+> **Token invariant.** Summing the four token buckets across `messages`
+> (assistant + the synthetic `reconciliation` entry) equals `token_usage` exactly.
+> The `reconciliation` message carries the residual the per-message stream
+> under-reports; it has no cost and is excluded from turn/generation counts. See the
+> [Claude Code guide](agents/CLAUDE_CODE.md#telemetry).
+
+### EarlyStopInfo
+
+Present (non-`null`) iff the run stopped early — there is no separate boolean.
+Fields: `reason` (`criterion_passed` / `criterion_failed`),
+`deciding_criterion_type`, `deciding_criterion_description`, `armed_criteria`,
+`sdk_turn_index`, `tool_call_index` (1-based, includes the in-flight call),
+`elapsed_seconds`, `turns_remaining_at_stop`.
+
+---
+
+## `variant.json` — `VariantAggregate`
+
+A single aggregate (not wrapped): `variant_id`, `tasks_run`, `tasks_succeeded`,
+`tasks_failed`, `tasks_error` (same sum-to-`tasks_run` invariant), `average_score`,
+`average_duration`, `total_tokens`, `replicate_count`, `tasks_token_budget_exceeded`,
+`tasks_cost_budget_exceeded`.
+
+## `experiment.json` — `ExperimentResult`
+
+The cross-variant summary:
+
+- `experiment_id`, `description`, `variant_ids`.
+- `task_summaries: list[TaskExperimentSummary]` — each
+ `{task_id, variant_results, best_variant, is_tie, score_spread, replicate_count}`,
+ where each `VariantResult` carries `{variant_id, task_id, weighted_score,
+ final_status, duration_seconds, total_tokens, iteration_count,
+ total_assistant_turns, reference_similarity, replicate_index, replicate_count}`.
+- `variant_aggregates: dict[str, VariantAggregate]` — keyed by variant id.
+- `total_duration_seconds`.
+- `per_replicate_scores: dict[variant_id -> dict[task_id -> list[float]]]`.
+
+> **Statistics are render-time only.** Bootstrap/Wilson confidence intervals and the
+> Welch/paired mean-difference tests appear in `experiment.md` / HTML but are **not**
+> serialized into `experiment.json`. A consumer that wants CIs must recompute them
+> from `per_replicate_scores`.
+
+## `suite.json` — `SuiteRollup`
+
+Written for dataset-backed suites; its `passed` flag drives the CI exit code.
+
+| Key | Type | Meaning |
+| --- | --- | --- |
+| `suite_id` / `variant_id` | `str` | Identity. |
+| `rows_total` / `rows_passed` / `rows_failed` / `rows_error` | `int` | Row counts. |
+| `pass_rate` | `float` | `rows_passed / rows_total`. |
+| `average_weighted_score` | `float \| null` | Mean row score. |
+| `criterion_stats` | `list[{criterion_type, rows_evaluated, average_score, error_count}]` | Per-criterion summary. |
+| `failed_samples` | `list[FailedRowSummary]` | Capped at 20 (`{row_id, task_id, final_status, weighted_score, failure_reasons, error_message, task_json_relpath, replicate_index}`). |
+| `criterion_aggregates` | `list[CriterionAggregate]` | The thresholdable metrics — see below. |
+| `passed` | `bool` | All aggregates met their thresholds. |
+
+### CriterionAggregate & ThresholdCheck
+
+`CriterionAggregate`: `criterion_type`, `description`, `rows_total`, `rows_excluded`,
+`metrics` (a **flat** `dict[str, float]`, e.g. `accuracy`, `macro_f1`,
+`precision.yes`, `recall.yes`, `f1.yes`), `threshold_checks`, `passed`, `details`
+(untyped render extras — for classification: `labels`, `per_label`, `confusion`),
+`error`.
+
+`ThresholdCheck`: `metric`, `min_value`, `actual_value` (`null` if the aggregator
+didn't emit that metric), `passed` (`actual_value >= min_value`). These correspond to
+the `suite_thresholds` you set on a criterion — see
+[User Guide → Suite Thresholds](USER_GUIDE.md#suite-thresholds--classification-metrics).
+
+---
+
+## TokenUsage
+
+Serialized fields: `uncached_input_tokens`, `output_tokens`,
+`cache_creation_input_tokens`, `cache_read_input_tokens`, `total_cost_usd`, plus a
+computed **`input_tokens`** (= sum of the three input buckets). Note: `total_tokens`
+is a plain property and is **not** serialized. Cost bills `uncached_input_tokens`,
+not `input_tokens`. Legacy files that only have `input_tokens` are adopted as
+`uncached_input_tokens` on read.
+
+## FinalStatus
+
+String enum values and their reporting category:
+
+| Value | Category | Icon |
+| --- | --- | --- |
+| `SUCCESS` | succeeded | `+` |
+| `FAILURE` | failed | `-` |
+| `TIMEOUT` | failed | `T` |
+| `MAX_TURNS_EXHAUSTED` | failed | `M` |
+| `TOKEN_BUDGET_EXCEEDED` | failed | `#` |
+| `COST_BUDGET_EXCEEDED` | failed | `$` |
+| `ERROR` | error | `!` |
+| `BUILD_FAILED` | error | `B` |
+
+> **Gotcha:** `BUILD_FAILED` (a failed Docker image build) categorizes as **error**,
+> not failed — easy to miscount downstream.
+
+`TOKEN_BUDGET_EXCEEDED` and `COST_BUDGET_EXCEEDED` are produced by the cumulative budget caps under
+`run_limits:` (`max_input_tokens` / `max_output_tokens` / `max_total_tokens`, and `max_usd`
+respectively), checked after each completed agent turn — see
+[Task Definition Guide → Run Limits](TASK_DEFINITION_GUIDE.md#run-limits).
+
+---
+
+## Consumer gotchas at a glance
+
+- `run.json.task_results` is a flat, untyped denormalization — the typed source of
+ truth is each `task.json`.
+- Experiment CIs / significance tests are **not** in `experiment.json` (render-time
+ only); recompute from `per_replicate_scores`.
+- Judge `transcript` is stripped from `task.json`; follow `transcript_path`.
+- `TokenUsage.total_tokens` is not serialized; sum the buckets (or use the computed
+ `input_tokens` + `output_tokens` + cache buckets).
+- `EarlyStopInfo` presence is itself the "stopped early" signal.
+
+## See also
+
+- [User Guide → Output Structure](USER_GUIDE.md#output-structure) and the
+ [`aggregate`](USER_GUIDE.md#cli-commands) command
+- [A/B Experiments → Reading the Report](AB_EXPERIMENTS.md#reading-the-report)
+- [Task Definition Guide](TASK_DEFINITION_GUIDE.md)
diff --git a/docs/TASK_DEFINITION_GUIDE.md b/docs/TASK_DEFINITION_GUIDE.md
index d0baa887..0e75f25b 100644
--- a/docs/TASK_DEFINITION_GUIDE.md
+++ b/docs/TASK_DEFINITION_GUIDE.md
@@ -1,18 +1,21 @@
---
description: >-
- Full schema reference for coder_eval task YAML — agent config, sandboxes, run
+ Full schema reference for Coder Eval task YAML — agent config, sandboxes, run
limits, dataset fan-out, and all 14 success criterion types with weighted
0.0–1.0 scoring.
---
# Task Definition Guide
-Complete reference for defining evaluation tasks in coder_eval.
+Complete reference for defining evaluation tasks in Coder Eval.
## Table of Contents
- [Task YAML Structure](#task-yaml-structure)
+ - [dataset](#dataset)
+ - [skip](#skip)
- [Agent Configuration](#agent-configuration)
+- [Run Limits](#run-limits)
- [Sandbox Configuration](#sandbox-configuration)
- [Template Sources](#template-sources)
- [Success Criteria](#success-criteria)
@@ -46,18 +49,72 @@ description: "What this task tests" # Human-readable description (required)
initial_prompt: "Instructions..." # Prompt sent to the agent (required)
tags: [smoke, golden, pure-python] # Optional tags for filtering (kebab-case)
+skip: false # Optional: quarantine this task (see below)
+
agent: { ... } # Agent configuration (optional, resolved from experiment)
sandbox: { ... } # Sandbox configuration (optional, defaults to tempdir)
+run_limits: { ... } # Optional run-time caps (turns, wall-clock, tokens, USD)
success_criteria: [ ... ] # List of criteria (required, at least 1)
reference: { ... } # Optional reference solution
pre_run: [ ... ] # Optional pre-run commands (before agent starts)
post_run: [ ... ] # Optional post-run commands
+dataset: { ... } # Optional dataset fan-out (one task -> N row-tasks)
```
+### `dataset`
+
+An optional `dataset:` block fans this single task out into **one sub-task per row**. Each row-task
+gets its own sandbox, run directory, and `task.json`; its `task_id` becomes `/`.
+Row values substitute into `initial_prompt` and into the string leaves of `success_criteria` via
+`${row.}`. Expansion happens at load time, **before** experiment-variant resolution, so a
+variant can never override the dataset.
+
+```yaml
+dataset:
+ rows: # inline rows — mutually exclusive with `paths`
+ - id: alpha
+ expected: "alpha"
+ # paths: ["datasets/rows.jsonl"] # or JSONL files, relative to this task YAML
+ id_field: "id" # which row field is the row identifier
+ sample_per_stratum: 5 # optional: keep up to N rows per stratum
+ stratify_field: "expected_skill" # which row field defines the stratum
+ sample_seed: 1234 # optional: pin the stratified draw
+```
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `rows` | `null` | Inline list of row dicts. Mutually exclusive with `paths`; exactly one is required. |
+| `paths` | `null` | JSONL file paths **relative to the task YAML**, concatenated in declared order. |
+| `id_field` | `"id"` | Row field used as the row identifier. Must be present, unique, and match `^[A-Za-z0-9_][A-Za-z0-9_.\-]*$` (it becomes a directory name). |
+| `sample_per_stratum` | `null` | Stratified random sample: keep up to N rows per stratum. Overridden by CLI `--sample`. |
+| `stratify_field` | `"expected_skill"` | Row field whose value defines the stratum for `sample_per_stratum`. |
+| `sample_seed` | `null` | Seed for the stratified draw. Unset means the sample is **re-drawn every run**; set an integer to pin it. CLI `--sample` is separately fixed-seed and always reproducible. |
+
+Full guide — row sources, substitution rules, sampling precedence, suite-level scoring, and worked
+examples: **[Bring Your Own Dataset](DATASETS.md)**.
+
+### `skip`
+
+`skip: true` quarantines a task. The runner records it in `RunSummary.skipped_tasks` at resolution
+time and it **never reaches the orchestrator** — no dataset fan-out, no variant resolution, no
+sandbox, no API call. Use it to park a task that is blocked on something outside your control
+(an upstream bug, a missing service) without deleting the YAML and losing its history.
+
+```yaml
+task_id: "codex_disallowed_tools_test"
+# Blocked: the Codex SDK doesn't enforce disallowed_tools via config. Re-enable
+# once upstream ships the fix.
+skip: true
+```
+
+Pair it with a comment naming the blocker — a ticket link, an upstream issue — so the next reader
+knows what has to change before it can come back. Run quarantined tasks on demand with
+`coder-eval run --include-skipped`; CI leaves the flag off, so they stay excluded there.
+
## Tags
-Tags categorize tasks for selective execution. Each tag is lowercase kebab-case and may
+The `tags` list categorizes tasks for selective execution. Each tag is lowercase kebab-case and may
optionally be namespaced as `key:value` where both sides are kebab-case.
```yaml
@@ -95,14 +152,10 @@ coder-eval run tasks/*.yaml --exclude-tags example # Skip example tasks
## Agent Configuration
-```yaml
-# Run-time caps (turns, wall-clock, tokens, USD) live under run_limits
-run_limits:
- max_turns: 20 # Optional: hard cap on inner-loop turns per iteration
- expected_turns: 8 # Optional: SOFT efficiency budget (visible turns) — not a cap
- turn_timeout: 300 # Optional: per-communicate() timeout in seconds
- task_timeout: 600 # Optional: wall-clock cap across all iterations
+Run-time caps (turns, wall-clock, tokens, USD) are **not** part of this block — they live under
+[`run_limits`](#run-limits).
+```yaml
agent:
type: "claude-code" # Agent type — optional if supplied via experiment / --type
permission_mode: "acceptEdits" # Permission mode (see below)
@@ -116,7 +169,7 @@ agent:
```
**`sdk_options`** is a typed pass-through dict for Claude Code SDK
-`ClaudeAgentOptions` fields that coder_eval doesn't own directly. Keys are
+`ClaudeAgentOptions` fields that Coder Eval doesn't own directly. Keys are
validated against the SDK's dataclass at YAML load; framework-managed keys
(`model`, `allowed_tools`, `permission_mode`, `hooks`, `mcp_servers`, …)
are rejected. Deep-merged across the 5-layer config chain. Override via
@@ -130,7 +183,7 @@ an error.
- `plan` — Agent proposes changes, waits for approval
- `bypassPermissions` — No permission checks (use with caution)
-> **Codex note:** `permission_mode` confines the **`claude-code`** agent only. The **`codex`** agent always runs full-access regardless of the mode — its in-process OS sandbox is redundant given coder_eval's docker/tempdir isolation and unusable on our CI hosts (and on Windows). Run adversarial or untrusted Codex evals under the **docker driver**, which is the OS-level write boundary; the tempdir/host driver is a working directory, not a confinement boundary.
+> **Codex note:** `permission_mode` confines the **`claude-code`** agent only. The **`codex`** agent always runs full-access regardless of the mode — its in-process OS sandbox is redundant given Coder Eval's docker/tempdir isolation and unusable on our CI hosts (and on Windows). Run adversarial or untrusted Codex evals under the **docker driver**, which is the OS-level write boundary; the tempdir/host driver is a working directory, not a confinement boundary.
**Agent Types:**
- `claude-code` (default) — Claude Code SDK agent. Supports `sdk_options`, `claude_settings`, and all permission modes.
@@ -172,11 +225,78 @@ trajectory (`command_executed`, `skill_triggered`, `reference_comparison`,
`commands_efficiency`) are rejected. A worked example lives at
[`tasks/agentless_smoke_test.yaml`](https://github.com/UiPath/coder_eval/blob/main/tasks/agentless_smoke_test.yaml).
-### `max_turns`, `task_timeout`, `turn_timeout` location
+## Run Limits
-These live under `run_limits:` on the task — alongside the token / USD
-budget caps. They are scenario constraints; the agent identity (type,
-model, etc.) is conceptually separate.
+`run_limits:` is the single namespace for every run-time cap: the **structural** caps that bound how
+long a task may run, and the **budget** caps that bound what it may spend. Any subset of fields is
+valid and an empty block is legal — every field defaults to "no limit".
+
+```yaml
+run_limits:
+ # Structural caps
+ max_turns: 20 # hard cap on agent inner-loop turns per iteration
+ expected_turns: 8 # SOFT efficiency budget (visible turns) — never aborts
+ task_timeout: 600 # wall-clock cap across all iterations, seconds
+ turn_timeout: 300 # per-communicate() timeout, seconds
+
+ # Budget caps
+ max_total_tokens: 200000 # cumulative input + output
+ max_usd: 2.50 # cumulative cost
+
+ # Early stop
+ stop_early: true # end once the armed criteria are decided
+```
+
+| Field | Default | Constraint | Description |
+|-------|---------|------------|-------------|
+| `max_turns` | *unset* | `> 0` | Hard cap on agent inner-loop turns per iteration. Unset uses the SDK default. |
+| `expected_turns` | *unset* | `>= 1` | **Soft** target for cumulative visible turns. Exceeding it warns and badges the report; it never aborts. See [`expected_turns`](#expected_turns-soft-efficiency-budget). |
+| `task_timeout` | *unset* | `>= 30` | Max seconds for the whole evaluation loop (all iterations). |
+| `turn_timeout` | *unset* | `>= 10` | Max seconds for a single agent `communicate()` call. |
+| `max_input_tokens` | *unset* | `>= 1` | Max cumulative input (prompt) tokens. |
+| `max_output_tokens` | *unset* | `>= 1` | Max cumulative output (completion) tokens. |
+| `max_total_tokens` | *unset* | `>= 1` | Max cumulative input + output tokens. Distinct from [`simulation.max_total_tokens`](#simulation-multi-turn-user-dialog) — see the note below. |
+| `max_usd` | *unset* | `> 0.0` | Max cumulative cost in USD. Requires per-turn SDK cost reporting. |
+| `count_cached_input` | `false` | — | Count `cache_read_input_tokens` toward the input/total budgets. Off by default — cached reads are typically free. |
+| `count_cache_creation` | `false` | — | Count `cache_creation_input_tokens` toward the input/total budgets. Off by default. |
+| `stop_early` | `false` | — | Opt-in master switch for early-stop-on-criterion. See [`stop_early`](#stop_early-opt-in-early-stop). |
+
+The authoritative source is `src/coder_eval/models/limits.py`. A lint rule (CE030) fails the build if
+a field defined there goes undocumented in this guide, so the table can't quietly fall behind the
+model.
+
+**Budget-cap semantics:**
+
+- **Checked after each completed agent turn**, and **cumulative** across all of the task's turns.
+ There is no mid-turn enforcement, so a single runaway turn can overshoot the cap before the
+ between-turns check sees it. Size caps with headroom for one turn.
+- **Subject agent only.** Judge (`llm_judge` / `agent_judge`) and user-simulator token spend are
+ **not** counted against these caps.
+- A breach aborts the task with `FinalStatus.TOKEN_BUDGET_EXCEEDED` (any of the three token caps) or
+ `FinalStatus.COST_BUDGET_EXCEEDED` (`max_usd`). Both categorize as `failed` — see
+ [Report Schema](REPORT_SCHEMA.md).
+- **`max_usd` needs per-turn cost from the SDK.** If no turn reports a cost, the check is **skipped
+ with a one-shot warning per task**, not failed. A run can therefore blow past `max_usd` silently
+ on a backend that doesn't report cost — don't rely on it as your only guardrail.
+- **Cached-read and cache-creation tokens are excluded by default.** `count_cache_creation: true` is
+ what makes an input-token budget meaningful for **Codex**, which buckets its fresh (full-price)
+ prompt slice into `cache_creation`; with the default `false`, a Codex token budget effectively
+ caps output only.
+
+**Setting caps from the CLI** — every field is reachable through `-D`, which merges per-key into
+`run_limits` without disturbing the task's other caps:
+
+```bash
+coder-eval run task.yaml -D run_limits.max_turns=30 -D run_limits.task_timeout=900
+coder-eval run task.yaml -D run_limits.max_usd=2.50 -D run_limits.max_total_tokens=200000
+```
+
+> **`run_limits.max_total_tokens` vs. `simulation.max_total_tokens`.** They are different budgets.
+> `run_limits.max_total_tokens` caps the **subject agent's** cumulative tokens and **aborts the task**
+> with `TOKEN_BUDGET_EXCEEDED`. [`simulation.max_total_tokens`](#simulation-multi-turn-user-dialog)
+> caps the **whole dialog** — simulator *plus* agent — and ends the dialog gracefully with
+> `stop_reason='budget'`, leaving the task to be scored normally. Set both if you want a hard
+> ceiling on a simulated task.
> **No longer supported:** `max_turns` / `turn_timeout` (and top-level
> `task_timeout`) under `agent:` or at the task top level are rejected —
@@ -394,9 +514,9 @@ All criteria share these fields:
| `stop_when` | `null` | Arms this criterion for early stop (`pass`/`fail`/`decided`); requires `run_limits.stop_early: true` and an observable criterion type (`skill_triggered`, `command_executed`). See [`stop_early`](#stop_early-opt-in-early-stop). |
**Scoring types:**
-- **Binary** (1.0 or 0.0): `file_exists`, `run_command`, `file_matches_regex`
+- **Binary** (1.0 or 0.0): `file_exists`, `run_command`, `file_matches_regex`, `classification_match`, `skill_triggered`
- **Fractional** (0.0–1.0): `file_contains`, `file_check`, `json_check`, `command_executed`, `uipath_eval`
-- **Continuous** (0.0–1.0): `reference_comparison`, `llm_judge`, `agent_judge`
+- **Continuous** (0.0–1.0): `reference_comparison`, `commands_efficiency`, `llm_judge`, `agent_judge`
**Task success:** all *gating* criteria must score >= their `pass_threshold`. A
criterion with `weight: 0` is informational — it is still checked, stored, and
@@ -616,6 +736,22 @@ Checks whether the agent executed specific tools/commands during evaluation. Ins
**Codex limitation.** Codex agents map `Read`, `Grep`, and `Glob` tools to `shell` commands (they execute via bash), so `tool_name: "Read"` on Codex returns no matches. Use `tool_name: "Bash"` or `tool_name: null` (any tool) for Codex-compatible checks. This criterion works correctly on Claude Code agents, which emit separate `Read`/`Grep`/`Glob` telemetry.
+### `commands_efficiency`
+
+Scores how economically the agent worked, relative to a budget of expected tool calls. **Continuous scoring:** `score = expected_commands / max(actual_commands, expected_commands)` — so a run at or under budget scores `1.0`, and the score decays as the agent takes more calls than expected (e.g. twice the budget → `0.5`).
+
+```yaml
+- type: "commands_efficiency"
+ expected_commands: 8 # budget of tool calls to complete the task (>= 1)
+ description: "Agent should solve this in ~8 tool calls"
+```
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `expected_commands` | *required* | Expected number of tool commands to complete the task (integer, `>= 1`). |
+
+This criterion requires an agent run (it reads `CommandTelemetry`). Pair it with a low `weight` if you want efficiency to *inform* the score without gating pass/fail on its own.
+
### `uipath_eval`
Evaluates a UiPath agent against a named evaluation set. **Fractional scoring:** metrics passed / total metrics.
@@ -640,7 +776,7 @@ Evaluates a UiPath agent against a named evaluation set. **Fractional scoring:**
### `llm_judge`
-Have an LLM grade the task against a rubric written in the task YAML. **Continuous scoring** from a JSON verdict `{"score": 0.0-1.0, "rationale": "..."}`; parse failure, non-numeric score, or LLM error all produce `score=0.0` with an `error` populated.
+Have an LLM grade the task against a rubric written in the task YAML. **Continuous scoring** from a verdict the judge returns via a forced `submit_verdict` tool call (`{score: 0.0-1.0, rationale: "..."}`) — the model never returns free-form prose. A missing/malformed verdict, a non-numeric score, or an LLM error all produce `score=0.0` with an `error` populated.
```yaml
- type: "llm_judge"
@@ -657,7 +793,7 @@ Have an LLM grade the task against a rubric written in the task YAML. **Continuo
include_dialog: false # Opt-in: include the full user<->agent conversation (recommended for simulation)
model: "anthropic.claude-sonnet-4-6"
temperature: 0.0
- max_tokens: 1000
+ max_tokens: 2000
max_file_chars: 20000 # Per-file content truncation
weight: 2.0
pass_threshold: 0.7
@@ -674,7 +810,7 @@ Have an LLM grade the task against a rubric written in the task YAML. **Continuo
| `max_dialog_chars` | `80000` | Aggregate cap on dialog text rendered into the judge prompt (per-message cap is `max_file_chars`). When exceeded, trailing turns are dropped and a degraded note is recorded. |
| `model` | `anthropic.claude-sonnet-4-6` | Judge model id (vendor-prefixed; auto-translated per backend) |
| `temperature` | `0.0` | Sampling temperature (0.0 = deterministic) |
-| `max_tokens` | `1000` | Maximum tokens in the judge's response |
+| `max_tokens` | `2000` | Maximum tokens in the judge's response |
| `max_file_chars` | `20000` | Per-file (and agent_output) truncation applied before building the prompt |
**Transport selection.** The judge call is routed by the active `API_BACKEND`:
@@ -694,14 +830,14 @@ The `direct`-mode transport is resolved once at startup, logged on the `API rout
**Failure modes** — each sets `score=0.0` and populates `error`:
-- Non-JSON response from the model (parse failure)
-- `score` key missing from the JSON verdict
+- The judge never emits the forced `submit_verdict` tool call (no verdict returned)
+- `score` key missing from the verdict
- `score` is not coercible to float
- Judge backend unavailable / network error (handled by `@handle_criterion_errors`)
### `agent_judge`
-Spawn a full Claude Code SDK agent as the judge. Unlike `llm_judge` (a single LLM call against a rubric), the judge agent has **tool access** — Bash, Read, Write, Glob, Grep, Edit by default — and runs in an isolated copy of the task sandbox. Use it when functional validation requires executing something (`uip rpa get-errors`, `xmllint`, a test suite) rather than just inspecting file content.
+Spawn a full Claude Code SDK agent as the judge. Unlike `llm_judge` (a single LLM call against a rubric), the judge agent has **tool access** — a read-only toolkit of `Bash`, `Read`, `Glob`, `Grep` by default (no `Write`/`Edit`) — and runs in an isolated copy of the task sandbox. Use it when functional validation requires executing something (`uip rpa get-errors`, `xmllint`, a test suite) rather than just inspecting file content.
```yaml
- type: "agent_judge"
@@ -776,6 +912,30 @@ The judge runs with the evaluator's API credentials and can execute arbitrary Ba
- `TurnTimeoutError` (judge exceeded `turn_timeout`)
- SDK subprocess failure (e.g. `claude` CLI missing)
+### `classification_match`
+
+Matches a single label the agent wrote to a file against ground truth — the file-based classifier. Reads the file, normalizes the content (strip, and lowercase unless `case_sensitive`), and compares it to `expected_label`. **Binary scoring:** `1.0` on a match, else `0.0`.
+
+The observed label is the canonical form from `allowed_labels` when the content matches; otherwise `(none)` when the file is missing/empty and `(other)` when the content isn't in the allowed set. Both sentinels are recorded so a suite rollup shows them as real failure classes in the confusion matrix.
+
+```yaml
+- type: "classification_match"
+ path: "result.txt" # file (relative to sandbox) holding the agent's predicted label
+ expected_label: "positive"
+ allowed_labels: [positive, negative]
+ case_sensitive: false # default: case-insensitive + canonicalized
+ description: "Sentiment label matches ground truth"
+```
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `path` | *required* | File (relative to sandbox) containing the agent's predicted label. |
+| `expected_label` | *required* | Ground-truth label for this row (drive it from `${row.…}` on a dataset-backed task). |
+| `allowed_labels` | *required* | Canonical label set (≥1). Content not in this set becomes `(other)`. |
+| `case_sensitive` | `false` | When `false`, matching is case-insensitive and labels are canonicalized. |
+
+Like `skill_triggered`, this criterion emits a `ClassificationCriterionResult`, so on a [dataset-backed task](#dataset) the suite aggregator computes accuracy / precision / recall / F1 and a confusion matrix — gate them with `suite_thresholds`. Use `classification_match` when the agent writes its answer to a file (labeling/extraction tasks); use `skill_triggered` when the signal is whether a skill fired.
+
### `skill_triggered`
Binary classifier: **did the agent engage the target skill during the run?** Agent-agnostic — scans the run's `turn_records` for either signal: Claude's explicit `Skill` tool call whose `skill` parameter matches `skill_name` (namespace prefixes like `plugin:skill` are stripped, so `skill_name: uipath-agents` matches `Skill(skill="uipath-coded-agents:uipath-agents")`), or — for an agent with no `Skill` tool, e.g. Codex — a command that reads the skill's files off disk (a parameter contains `skills//`, matching both the repo path and the `.agents/skills/` symlink).
@@ -799,7 +959,7 @@ Observed label is `"yes"` when either signal is found, else `"no"`. Expected lab
**Requires agent telemetry.** This criterion reads `turn_records`, so it only works against a real agent run (not a static check). With no turn records it reports `score=0.0` and an `error`.
-**Classification metrics.** `skill_triggered` returns a `ClassificationCriterionResult`, so on a [dataset-backed task](#task-yaml-structure) the suite aggregator computes accuracy / precision / recall / F1 / confusion matrix across all rows. Gate the suite with `suite_thresholds` using any of: `accuracy`, `macro_f1`, `weighted_f1`, `micro_f1`, or per-label `precision.` / `recall.` / `f1.` (labels are `yes` / `no`). The run exits non-zero if any listed metric falls below its minimum.
+**Classification metrics.** `skill_triggered` returns a `ClassificationCriterionResult`, so on a [dataset-backed task](#dataset) the suite aggregator computes accuracy / precision / recall / F1 / confusion matrix across all rows. Gate the suite with `suite_thresholds` using any of: `accuracy`, `macro_f1`, `weighted_f1`, `micro_f1`, or per-label `precision.` / `recall.` / `f1.` (labels are `yes` / `no`). The run exits non-zero if any listed metric falls below its minimum.
**Typical pattern.** Label each dataset row with its true skill (`expected_skill`, `""` for negatives) and stack one `skill_triggered` criterion per skill against the same dataset — each gets its own confusion matrix from the same agent traces. This is the natural companion to a skill A/B experiment (skill plugin on vs. off); see the [A/B Experiment Guide](AB_EXPERIMENTS.md#recipe-ab-a-skill).
@@ -913,6 +1073,9 @@ The shipped `experiments/*.yaml` use this to drop sandbox-scoped npm dirs (intro
Optional `simulation` block. When present and enabled, the orchestrator replaces the single-shot iteration loop with a multi-turn dialog between the coding agent and a simulated user (a second LLM with a persona and goal). Use this for tasks where the real usage pattern is conversational — clarifying questions, incremental requirements, mid-task corrections — rather than a single fire-and-forget prompt.
+> This section is the field reference. For when to use dialog mode, how to design a persona and
+> goal, trials and variance, grading, and cost, see **[Dialog Mode](DIALOG_MODE.md)**.
+
```yaml
simulation:
enabled: true # Master switch; when false, simulation is skipped entirely.
@@ -937,7 +1100,6 @@ simulation:
# Sampling (variance analysis).
n_trials: 3 # Run N independent dialogs per (task, variant).
- parallel_trials: true # Trials run concurrently (subject to batch max_parallel).
# Criteria timing.
check_criteria: every_turn # One of: end_of_dialog | every_turn | both.
@@ -954,9 +1116,8 @@ simulation:
| `max_turns` | `8` | Hard cap on user↔agent exchanges (1–100). |
| `stop_token` | `"<<>>"` | Sentinel the simulator emits to end the dialog. |
| `stop_on_criteria_pass` | `false` | End when all criteria pass (requires per-turn checking). |
-| `max_total_tokens` | *unset* | Optional dialog-wide token budget. |
+| `max_total_tokens` | *unset* | Optional dialog-wide token budget (simulator **plus** agent). Distinct from [`run_limits.max_total_tokens`](#run-limits) — see below. |
| `n_trials` | `1` | Independent dialog trajectories per (task, variant). |
-| `parallel_trials` | `true` | Run trials concurrently within the batch. |
| `check_criteria` | `end_of_dialog` | `end_of_dialog`, `every_turn`, or `both`. |
The simulator runs as a tools-disabled Claude Code agent sharing the coding agent's `ApiRoute` — model/temperature/sampling are resolved at the route level (same `-b` flag as the coding agent), so they are not configured on this block.
@@ -966,9 +1127,19 @@ The simulator runs as a tools-disabled Claude Code agent sharing the coding agen
- The task's `initial_prompt` is the user's *opening* message; the simulator picks up from turn 2.
- `max_turns` is the intra-dialog cap (the worst-case agent call budget per trial). Use `n_trials` for variance sampling.
- The `reference` solution, if present, is hidden from the simulator (same security posture as for the coding agent).
-- When `n_trials > 1`, each trial becomes its own `ResolvedTask` with `task_id` suffix `/trial-N` (0-indexed), its own run directory, and its own `task.json`. Trial-level metadata appears under `simulation.trial_id` / `simulation.n_trials` on the `EvaluationResult`.
-
-**Termination precedence:** `stop_on_criteria_pass` → `stop_token` → `max_turns` → `max_total_tokens`.
+- When `n_trials > 1`, each trial becomes its own `ResolvedTask` with its own zero-padded replicate directory (`runs/////`) and its own `task.json` — the same fan-out mechanism as experiment `repeats`, which `n_trials` takes precedence over when simulation is enabled. Trial-level metadata appears under `simulation.replicate_index` / `simulation.n_trials` on the `EvaluationResult`.
+
+**Termination precedence** (evaluated after each exchange, first match wins): `run_limits` breach →
+`stop_on_criteria_pass` → `max_turns` → `max_total_tokens` → `stop_token`. The stop token is checked
+**last**, on the simulator's next utterance, which is only solicited if nothing above fired — so a
+turn that hits `max_turns` or the token budget ends the dialog before the simulator can emit it.
+A raising simulator call ends the dialog with `stop_reason: error`.
+
+> **`simulation.max_total_tokens` vs. `run_limits.max_total_tokens`.** This one covers the **whole
+> dialog** (simulator + agent) and ends the conversation gracefully with `stop_reason='budget'`, so
+> the task is still scored. [`run_limits.max_total_tokens`](#run-limits) covers the **subject agent
+> only** and **aborts** the task with `FinalStatus.TOKEN_BUDGET_EXCEEDED`. They compose — set both
+> for a hard ceiling on a simulated task.
**Grading simulated dialogs.** When a task uses `simulation:`, set `include_dialog: true` on any `llm_judge` / `agent_judge` criterion. Without it, the judge sees only the agent's outputs and may flag a fabricated-but-conceded premise as a hallucination by the agent. The dialog block is rendered with a rubric guard telling the judge to treat any claim made only by the simulated user as possibly invented, and not to penalize the agent for going along with it unless the grading prompt contradicts it.
diff --git a/docs/USER_GUIDE.md b/docs/USER_GUIDE.md
index 844c040a..f5467617 100644
--- a/docs/USER_GUIDE.md
+++ b/docs/USER_GUIDE.md
@@ -1,11 +1,11 @@
---
description: >-
- Complete coder_eval reference — every CLI command and flag, configuration
+ Complete Coder Eval reference — every CLI command and flag, configuration
layers and -D overrides, environment variables, run outputs, and reports for
evaluating AI coding agents.
---
-# coder_eval User Guide
+# Coder Eval User Guide
The full command, configuration, and output reference. For a gentle introduction
start with the [tutorials](tutorials/README.md); for the task-file schema see the
@@ -16,6 +16,7 @@ start with the [tutorials](tutorials/README.md); for the task-file schema see th
- [CLI Commands](#cli-commands)
- [API Routing & Benchmarking](#api-routing--benchmarking)
- [Output Structure](#output-structure)
+- [Suite Thresholds & Classification Metrics](#suite-thresholds--classification-metrics)
- [Environment Variables](#environment-variables)
- [Troubleshooting](#troubleshooting)
@@ -35,9 +36,15 @@ coder-eval run tasks/hello_date.yaml --stream full # live LLM output
| `--max-parallel, -j` | Concurrent tasks (default: 1) |
| `--preservation-mode` | Sandbox persistence: `NONE` / `MOVE_ON_WRITE` / `DIRECT_WRITE`. Default is driver-derived (docker → `DIRECT_WRITE`, else `MOVE_ON_WRITE`); explicit value always wins. |
| `--run-dir` | Custom run directory (default: timestamped in `runs/`) |
-| `-D path=value` / `--set` | Override any resolved task-config field (`agent`/`run_limits`/`sandbox` roots), e.g. `-D run_limits.max_turns=30 -D agent.permission_mode=plan -D agent.sdk_options.effort=high`. Repeatable; schema-validated. This is the way to set permission mode, turn/timeout limits, tools, plugins, and SDK options. |
+| `-D path=value` / `--set` | Override any resolved task-config field (`agent`/`run_limits`/`sandbox` roots), e.g. `-D run_limits.max_turns=30 -D agent.permission_mode=plan -D agent.sdk_options.effort=high`. Repeatable; schema-validated. This is the way to set permission mode, turn/timeout limits, token/USD budget caps, tools, plugins, and SDK options. |
| `--model, -m` | Shorthand alias for `-D agent.model=…` (e.g., `claude-sonnet-4-20250514`) |
| `--driver` | Shorthand alias for `-D sandbox.driver=…` (`tempdir` or `docker`) |
+| `--type, -T` | Override agent type for all tasks (`claude-code`, `codex`, `antigravity`, or a plugin kind). |
+| `--repeats` | Run each `(task, variant)` N times (≥1); overrides experiment/variant `repeats:`. See [Replicates](#replicates). |
+| `--resume` | Resume an interrupted run: skip tasks already finalized in `--run-dir` and run the rest, folding prior results into `run.json`. Requires `--run-dir`. A task with *any* final status (incl. FAILED/ERROR) counts as finalized, so resume does **not** retry failures — delete a task's `task.json` to force a re-run. A config mismatch is warned, not refused. |
+| `--sample N` | For dataset-backed tasks, run a fixed-seed random N-row sample (reproducible; cheap smoke test). See [Bring Your Own Dataset](DATASETS.md). |
+| `--sample-per-stratum N` | For dataset-backed tasks, keep up to N rows per stratum (`stratify_field`). Overridden by `--sample`. Nondeterministic unless `dataset.sample_seed` is set — see [Bring Your Own Dataset](DATASETS.md). |
+| `--include-skipped` | Also run tasks marked `skip: true` in their YAML (off by default so CI keeps excluding them). |
| `--exclude-tags` | Skip tasks matching any of these tags (comma-separated) |
| `--tags, -t` | Only run tasks matching any of these tags (comma-separated) |
| `--experiment, -e` | Experiment definition YAML for multi-variant comparison (default: `experiments/default.yaml`) |
@@ -46,6 +53,11 @@ coder-eval run tasks/hello_date.yaml --stream full # live LLM output
| `--stream, -s` | Stream LLM events to terminal: `full` or `minimal` (disables progress bar) |
| `--verbose, -v` | DEBUG-level logging |
+**Run-time caps** — turns, wall-clock timeouts, and cumulative token / USD budgets — are not CLI
+flags of their own. They live under `run_limits:` in the task YAML, or on the command line as
+`-D run_limits.=` (e.g. `-D run_limits.max_usd=2.50`). The complete field reference is
+in the [Task Definition Guide](TASK_DEFINITION_GUIDE.md#run-limits).
+
### `coder-eval plan` — validate tasks
```bash
@@ -55,6 +67,10 @@ coder-eval plan tasks/*.yaml # validate specific tasks
Checks task syntax, required CLI tools, API keys, and schema validity without executing.
+| Flag | Description |
+| --- | --- |
+| `--experiment, -e` | Experiment definition YAML to resolve variants against (default: `experiments/default.yaml`). |
+
### `coder-eval evaluate` — test criteria without an agent
```bash
@@ -69,15 +85,46 @@ already written.
| Flag | Description |
| --- | --- |
| `--preserve / --no-preserve` | Preserve sandbox after evaluation (default: preserve) |
+| `--run-dir` | Custom run directory (default: auto-generated timestamped dir in `runs/`). |
| `--verbose, -v` | DEBUG-level logging |
### `coder-eval report` — view results
```bash
-coder-eval report runs/latest # view latest run
-coder-eval report runs/latest -o summary.md # export to file
+coder-eval report runs/latest # view latest run (markdown to stdout)
+coder-eval report runs/latest -o summary.md # export markdown to a file
+coder-eval report runs/latest --format html # (re)render every task.json as task.html
+```
+
+The `run` command already writes reports during execution; `report` re-displays or
+re-exports them later. For the on-disk layout and field-level schema of the JSON it
+reads, see [Output Structure](#output-structure) and the
+[Report Schema](REPORT_SCHEMA.md).
+
+| Flag | Description |
+| --- | --- |
+| `--output, -o` | Write to a file instead of stdout (markdown). |
+| `--format, -f` | `md` (default) or `html`. `html` re-renders each `task.json` under the run dir to a `task.html` beside it (or to `-o` when exactly one task is found). |
+
+### `coder-eval aggregate` — rebuild `run.json` from task results
+
+```bash
+coder-eval aggregate runs/2026-06-22_14-32-27 # rebuild the summary in place
+coder-eval aggregate runs/combined -o runs/combined # aggregate a merged dir
```
+Re-derives the run-level `run.json` + `run.md` from the finalized `task.json` files
+already on disk, using the same builder a live run uses. Use it when a run dir's
+top-level summary is missing or stale — e.g. after recovering an interrupted run or
+combining several run directories. It rebuilds the **run-level summary only**;
+per-suite rollups (`suite.json`/`suite.md`) and experiment reports
+(`experiment.json`/`experiment.md`) are *not* rebuilt, because the per-row
+suite/variant grouping they need is not recoverable from `task.json` alone.
+
+| Flag | Description |
+| --- | --- |
+| `--output, -o` | Write `run.json`/`run.md` into this directory instead of the run dir (e.g. a merged output dir). |
+
### Claude Code slash commands
The project ships [Claude Code custom slash commands](https://docs.anthropic.com/en/docs/claude-code/slash-commands)
@@ -133,6 +180,36 @@ Run the same (task, variant) N times via `repeats:` in an experiment YAML or
reports aggregate them with bootstrap confidence intervals and (for 2-variant
experiments) a paired mean-difference test. Defaults to 1 (no repetition).
+For a field-level reference to `run.json`, `variant.json`, `task.json`, and the
+suite/experiment rollups, see the [Report Schema](REPORT_SCHEMA.md).
+
+` / `recall.` / `f1.`. This makes
+Coder Eval usable as a SkillsBench-style activation harness. Full details and the
+suite-rollup outputs live in [A/B Experiments](AB_EXPERIMENTS.md#measuring-the-difference)
+and the [Task Definition Guide](TASK_DEFINITION_GUIDE.md).
+
## Environment Variables
Set these in `.env` (copy from `.env.example`).
@@ -145,8 +222,10 @@ Set these in `.env` (copy from `.env.example`).
| `AWS_REGION` | For Bedrock | AWS region for Bedrock endpoint (e.g., `eu-north-1`) |
| `BEDROCK_MODEL` | No | Cross-region Bedrock model ID (e.g., `eu.anthropic.claude-sonnet-4-5-20250929-v1:0`) |
| `BEDROCK_SMALL_MODEL` | No | Cross-region Bedrock small/fast model ID |
-| `UIPATH_PLUGIN_MARKETPLACE_DIR` | No | Base directory for Claude Code plugins (substitutes `$UIPATH_PLUGIN_MARKETPLACE_DIR` in plugin paths) |
-| `PLUGIN_TOOLS_DIR` | No | Canonical `node_modules/@uipath` to pin UiPath CLI plugin discovery. When unset, the sandbox auto-derives it from the resolved `uip` binary. |
+| `CODEX_API_KEY` / `CODEX_BASE_URL` / `CODEX_MODEL` / `CODEX_API_VERSION` | For Codex | Codex agent auth & endpoint routing — see [Codex Agent Guide](agents/CODEX.md#endpoint-routing). |
+| `GEMINI_API_KEY` / `ANTIGRAVITY_MODEL` | For Antigravity | Antigravity (Gemini) agent auth & model — see [Antigravity Agent Guide](agents/ANTIGRAVITY.md#setup). |
+| `UIPATH_PLUGIN_MARKETPLACE_DIR` | No | Conventional base directory for local plugins. Not special-cased by the framework: **any** `$VAR` / `${VAR}` referenced in a plugin `path` is expanded from the environment, so a plugin path like `$UIPATH_PLUGIN_MARKETPLACE_DIR/my-plugin` resolves against this variable. (An undefined variable in a plugin path logs a warning.) |
+| `PLUGIN_TOOLS_DIR` | No | A *separate* mechanism from the above: the canonical `node_modules/@uipath` directory used to pin UiPath CLI plugin discovery (not path substitution). When unset, the sandbox auto-derives it from the resolved `uip` binary. |
| `CODER_EVAL_REMEDIATE_HOME_PLUGINS` | No | **DESTRUCTIVE.** Truthy deletes `$HOME/node_modules/@uipath` at sandbox setup to clear sibling-task pollution on dedicated eval hosts. Off by default; do **not** enable on developer workstations. |
| `LOG_LEVEL` | No | Logging level (default: INFO) |
| `LOG_TO_FILE` | No | Enable file logging (default: false) |
diff --git a/docs/agents/ANTIGRAVITY.md b/docs/agents/ANTIGRAVITY.md
new file mode 100644
index 00000000..a7cee48f
--- /dev/null
+++ b/docs/agents/ANTIGRAVITY.md
@@ -0,0 +1,203 @@
+---
+description: >-
+ Run Google Antigravity (Gemini) as the agent under evaluation in Coder Eval —
+ installation, authentication, model and skill configuration, and how its
+ telemetry maps to sandboxed, weighted scoring.
+---
+
+# Running Google Antigravity (Gemini) in Coder Eval
+
+## Overview
+
+Coder Eval can run Google's **Antigravity** agent — powered by **Gemini** — as the
+agent under evaluation. Set `agent.type: antigravity` in a task and the rest of the
+framework (sandbox, scoring, telemetry, reports) works unchanged.
+
+Under the hood `AntigravityAgent` drives the Antigravity SDK's **local harness**
+(the `localharness` binary bundled in the wheel) rather than the branded `agy` CLI
+or the remote Interactions API. The local harness is the only surface that can (a)
+authenticate headlessly with an API key and (b) make edits land in the sandbox
+working directory — both required for an unattended eval.
+
+## Setup
+
+### 1. Install the Antigravity SDK
+
+```bash
+pip install 'coder-eval[antigravity]'
+```
+
+This pulls in `google-antigravity` (pinned to `0.1.7`), whose wheel bundles the
+platform `localharness` binary. As with the other agents the SDK is imported lazily
+— a base install without the extra still runs end-to-end; Antigravity tasks fail at
+dispatch with a clear hint to install the extra.
+
+### 2. Authentication
+
+Antigravity authenticates against the **Gemini Developer API** with a single
+credential:
+
+```bash
+GEMINI_API_KEY=
+```
+
+Set it in `.env` (or the environment). That is the *only* credential Antigravity
+reads — there is **no base-URL, project, or region override** for this agent
+(unlike Codex's `CODEX_BASE_URL` / Azure routing). If `GEMINI_API_KEY` is unset the
+SDK raises a clear error on first use.
+
+> The `--backend` flag (`direct` / `bedrock`) governs *Claude* routing only; it has
+> no effect on the Antigravity agent.
+
+## Usage
+
+### Command line
+
+```bash
+coder-eval run tasks/agents/antigravity_hello_world.yaml --type antigravity
+```
+
+Or override the agent type for every task in an experiment:
+
+```bash
+coder-eval run experiments/model-comparison.yaml --type antigravity
+```
+
+### Task definition (YAML)
+
+```yaml
+agent:
+ type: antigravity
+ model: gemini-3.1-pro-preview # optional; see "Model selection" below
+ thinking_level: medium # minimal | low | medium | high (default: medium)
+ plugins:
+ - type: local
+ path: "$SKILLS_PLUGIN_PATH" # a directory of skills (SKILL.md), env-expanded
+
+run_limits:
+ max_turns: 5
+ task_timeout: 360
+ turn_timeout: 300
+
+success_criteria:
+ - type: file_exists
+ path: "hello.py"
+ description: "hello.py must be created"
+```
+
+Two runnable examples ship in the repo:
+
+- `tasks/agents/antigravity_hello_world.yaml` — `tempdir` driver
+- `tasks/agents/antigravity_hello_world_docker.yaml` — `docker` driver
+
+### Model selection
+
+The resolved model is the first of:
+
+1. `agent.model` in the task YAML
+2. `ANTIGRAVITY_MODEL` in the environment
+3. the built-in default, **`gemini-3.5-flash`**
+
+> **Note:** the shipped example tasks pin `gemini-3.1-pro-preview`. Pin `agent.model`
+> explicitly in any task you care about reproducing — don't rely on the built-in
+> default, which tracks the current recommended coding model and may change.
+
+### `thinking_level`
+
+Antigravity exposes a `thinking_level` field (`minimal` / `low` / `medium` /
+`high`, default `medium`) that maps to Gemini's thinking budget. This is
+Antigravity-specific — Claude Code and Codex don't take this field. Thinking tokens
+are billed as **output** tokens (see [Telemetry](#telemetry)).
+
+### Skills (SKILL.md)
+
+Antigravity supports [Agent Skills](https://agentskills.io/specification)
+(`SKILL.md`) natively. Skill directories are discovered from:
+
+1. `agent.plugins` entries with `type: local` and a `path` (env vars in the path are
+ expanded at runtime), and
+2. the runtime plugin directory passed to `start()`.
+
+For each source, the harness is handed whichever of `/skills` or ``
+directly contains subdirectories with a `SKILL.md`. Unlike Codex — which symlinks
+skills into `.agents/skills/` — Antigravity is given the search-path roots directly
+via the SDK's `skills_paths`, and those roots are also added to the harness
+`workspaces` allowlist (otherwise the agent would find a skill but be denied the
+`SKILL.md` read as out-of-workspace). The agent logs a loud warning if a skills path
+can't be resolved or if zero skills are discovered.
+
+## Permissions & tools — important differences
+
+**Antigravity ignores `permission_mode`, `allowed_tools`, and `disallowed_tools`.**
+The local harness runs in a single unconditional mode: every tool call (including
+`run_command`) is approved via an allow-all policy, and file tools are restricted to
+the configured `workspaces` (the sandbox working directory plus any skill roots).
+
+The trust boundary for an Antigravity run is therefore the **sandbox**, not the
+agent config. Run untrusted tasks under the [Docker driver](../DOCKER_ISOLATION.md);
+the `tempdir` driver is not a security boundary. This mirrors the reality that the
+`bypassPermissions`-equivalent behavior is always on for this backend.
+
+Those inherited fields still exist on the config for schema uniformity but have no
+runtime effect here — don't rely on them to gate Antigravity.
+
+## Telemetry
+
+Each `communicate()` call is one logical turn; the standard `TurnRecord` is built by
+the shared `EventCollector`, so Antigravity runs report the same per-turn structure
+as every other agent.
+
+- **Tool-name normalization.** Antigravity's built-in tool names are mapped to the
+ canonical Claude-style names so cross-agent criteria work: `run_command` → `Bash`,
+ `create_file` → `Write`, `edit_file` → `Edit`, `view_file` → `Read`,
+ `search_directory` → `Grep`, `find_file` → `Glob`, `list_directory` → `LS`,
+ `start_subagent` → `Task`, `search_web` → `WebSearch`. Argument keys are also
+ normalized (e.g. `command_line` → `command`) so `command_executed` criteria key on
+ the same params across agents.
+- **Tokens.** Gemini usage maps to Coder Eval's four buckets: uncached input, cache
+ read (`cached_content_token_count`), output, and **`cache_creation` is always 0**
+ (Gemini bills no separate cache-write). Thinking tokens are folded into **output**.
+ Per-generation usage is cut into `AssistantMessage`s that sum exactly to the turn
+ total, so there is typically no reconciliation residual.
+- **Commands.** Each tool call is captured as `CommandTelemetry` with status, an
+ untruncated `result_summary`, error message, and duration. Harness-appended result
+ fields (`exit_code`, `combined_output`, `diff_block`, `stdout`/`stderr`, …) are
+ stripped from the recorded `parameters` so tool *output* never leaks into the
+ recorded call — which also keeps it from false-positiving a `skill_triggered`
+ substring match.
+
+## Known limitations
+
+1. **No cooperative early stop.** Antigravity does not support the `should_stop`
+ seam, so `run_limits.stop_early` is unsupported for this agent (it errors at
+ resolution, as it does for any non-Claude agent).
+2. **No endpoint routing.** Only `GEMINI_API_KEY` + `ANTIGRAVITY_MODEL` are read —
+ there is no base-URL, project, region, or gateway override.
+3. **Default-model drift.** The runtime fallback (`gemini-3.5-flash`) may differ from
+ what a given release's docs or example tasks pin; always set `agent.model`
+ explicitly for reproducible runs.
+4. **`kill_sync()` is best-effort.** The SDK's cancel/disconnect are async-only, so
+ the watchdog's synchronous kill only flips agent state to `ERROR`; real teardown
+ happens on the subsequent async `stop()`.
+5. **Process-global spawn lock.** The SDK spawns `localharness` via a subprocess with
+ no env-injection seam, so the agent transiently mutates `PATH` across the spawn
+ under a process-wide lock. This serializes harness startup across concurrent
+ tasks (it does not serialize the turns themselves).
+
+## Running in Docker
+
+The `docker` driver works the same as for other agents (see
+[Docker Isolation](../DOCKER_ISOLATION.md)). The `localharness` binary ships inside
+the image via the `[antigravity]` extra, and `GEMINI_API_KEY` + `ANTIGRAVITY_MODEL`
+are on the container env passthrough allowlist, so they are forwarded automatically.
+
+```bash
+coder-eval run tasks/agents/antigravity_hello_world_docker.yaml
+```
+
+## References
+
+- [Agent Skills specification](https://agentskills.io/specification)
+- [Codex Agent Guide](CODEX.md) — the sibling third-party-agent guide
+- [Claude Code Agent](CLAUDE_CODE.md) — the default agent
+- [Extending Coder Eval](../EXTENDING.md) — how agents register via the plugin SPI
diff --git a/docs/agents/CLAUDE_CODE.md b/docs/agents/CLAUDE_CODE.md
new file mode 100644
index 00000000..d8e6075c
--- /dev/null
+++ b/docs/agents/CLAUDE_CODE.md
@@ -0,0 +1,194 @@
+---
+description: >-
+ Configure and run the default Claude Code agent in Coder Eval — the full
+ agent-config surface, direct vs. Bedrock authentication, permission modes,
+ sandbox isolation, skills/plugins, early stop, and token telemetry.
+---
+
+# Running Claude Code in Coder Eval
+
+## Overview
+
+**Claude Code is the default agent** in Coder Eval — `agent.type: claude-code`. It
+ships in the base install (no extra needed; the `claude-agent-sdk` is a core
+dependency), and most tasks and tutorials assume it. This guide is the reference
+for its config surface, authentication, and telemetry; for the agent-agnostic task
+schema see the [Task Definition Guide](../TASK_DEFINITION_GUIDE.md).
+
+Because `agent.type` defaults to `claude-code` in `experiments/default.yaml`, you can
+omit the `type` field entirely.
+
+## Setup
+
+Nothing beyond the base install:
+
+```bash
+uv tool install coder-eval # or: pip install coder-eval
+```
+
+Provide model credentials (see [Authentication](#authentication)) and you're ready:
+
+```bash
+coder-eval run tasks/hello_date.yaml # claude-code is the default
+```
+
+## Authentication
+
+Claude Code routing follows the `--backend` flag (or `API_BACKEND` env var):
+
+### Direct (default)
+
+`--backend direct` calls the Anthropic API. The SDK picks its own credential, in
+order:
+
+1. `ANTHROPIC_API_KEY` in the environment / `.env`, or
+2. a cached `claude login` (subscription) session.
+
+No key is validated at startup — the SDK fails with a clear error if none is found.
+The `llm_judge` / `agent_judge` criteria also need `ANTHROPIC_API_KEY` under the
+direct backend (they call `api.anthropic.com`).
+
+### AWS Bedrock
+
+`--backend bedrock` routes through Bedrock with bearer-token auth. Set in `.env`:
+
+| Variable | Purpose |
+| --- | --- |
+| `AWS_BEARER_TOKEN_BEDROCK` | Bedrock bearer token (required) |
+| `AWS_REGION` | Bedrock region, e.g. `eu-north-1` (required) |
+| `BEDROCK_MODEL` | Cross-region model id, e.g. `eu.anthropic.claude-sonnet-4-5-20250929-v1:0` (required) |
+| `BEDROCK_SMALL_MODEL` | Small/fast model id (falls back to the main model) |
+
+The agent sets `CLAUDE_CODE_USE_BEDROCK=1` and forwards these into the SDK
+subprocess. `BEDROCK_MODEL` is the route-level default; an explicit `--model` /
+`-D agent.model=…` / task `agent.model` always wins over it, and bare aliases are
+auto-qualified to a regional inference profile (`eu.` / `us.` / `apac.` / `global.`).
+
+> **For official benchmarking use the direct API** — the SDK reports accurate
+> token/cost there. See [User Guide → API Routing](../USER_GUIDE.md#api-routing--benchmarking).
+
+## Agent config surface
+
+All fields live under `agent:` in a task (or an experiment variant). Only `type` is
+required; everything else has a default.
+
+```yaml
+agent:
+ type: claude-code
+ model: claude-sonnet-4-5-20250929 # optional; omit to use the route default
+ permission_mode: acceptEdits # default | acceptEdits | plan | bypassPermissions
+ allowed_tools: ["Read", "Write", "Bash"]
+ disallowed_tools: ["WebSearch"]
+ setting_sources: [] # [] = fully isolated (recommended, see below)
+ plugins:
+ - type: local
+ path: "$SKILLS_PLUGIN_PATH"
+ system_prompt: "You are a careful engineer." # or system_prompt_file
+ claude_settings:
+ permissions:
+ deny: ["Read(/etc/**)"]
+ sdk_options:
+ effort: high
+```
+
+| Field | Type / default | Meaning |
+| --- | --- | --- |
+| `type` | `"claude-code"` | Agent kind (the default). |
+| `model` | `str \| null` | Specific model id. Omit to use the backend/route default. |
+| `permission_mode` | default **`acceptEdits`** | `default` / `acceptEdits` / `plan` / `bypassPermissions` — semantics come from the Claude Code SDK. `plan` is read-only; `bypassPermissions` grants full autonomy. |
+| `allowed_tools` | `list[str] \| null` | Tool allowlist. Unset ⇒ all tools allowed. |
+| `disallowed_tools` | `list[str] \| null` | Tool denylist. (`ToolSearch` is always appended for Bedrock parity.) |
+| `plugins` | `list[{type: local, path}]` | Local plugin/skill directories; `$VAR` in `path` is expanded and resolved to an absolute path. |
+| `system_prompt` | `str \| null` | **Replaces** the default system prompt (there is no *append* seam). Mutually exclusive with `system_prompt_file`. |
+| `system_prompt_file` | `str \| null` | Path (relative to the task YAML) loaded into `system_prompt` at resolution. |
+| `setting_sources` | `list["user"\|"project"\|"local"] \| null` | Which host setting sources the SDK reads. Default resolves to `["project"]`. See [Sandbox isolation](#sandbox-isolation). |
+| `claude_settings` | `str \| dict \| null` | Passed to the SDK `--settings`. A dict is JSON-serialized; a str is a settings file path. Use `permissions.deny` to block tools/paths. |
+| `sdk_options` | `dict` (default `{}`) | Pass-through for `ClaudeAgentOptions` fields Coder Eval doesn't own (e.g. `effort`). Validated at load — an unknown or framework-owned key is a hard error. |
+| `ignore_patterns` | `list[str] \| null` | Gitignore-style overrides for the workspace copy used by judge sub-agents (supports `!` negation). |
+
+> `sdk_options` is a deliberate escape hatch. Framework-owned keys (`model`,
+> `permission_mode`, `allowed_tools`, `mcp_servers`, `resume`, `max_turns`,
+> `setting_sources`, `include_partial_messages`, …) are rejected there — set those
+> through their typed fields or `-D run_limits.*`. MCP servers are not a YAML field.
+
+### Setting fields from the CLI
+
+Any of these merge-resolve through `-D` / `--set` (see
+[User Guide → CLI overrides](../USER_GUIDE.md#cli-commands)):
+
+```bash
+coder-eval run tasks/hello_date.yaml \
+ -D agent.model=claude-opus-4-8 \
+ -D agent.permission_mode=plan \
+ -D agent.sdk_options.effort=high
+```
+
+## Sandbox isolation
+
+`setting_sources` controls whether the host project's settings leak into the run.
+The default (`["project"]`) lets the SDK discover `.mcp.json` — but it also pulls in
+the host project's `CLAUDE.md`, settings, and hooks, which for this repo is 20 KB+
+injected into every API call (inflating cache-creation tokens and cost).
+
+**For tasks that don't need host MCP servers, set `setting_sources: []`** for full
+isolation from the host `CLAUDE.md`/settings/hooks. (Judge sub-agents and the user
+simulator force `[]` for the same reason.)
+
+## Skills & plugins
+
+- **Skills** are engaged via Claude's explicit `Skill` tool call; the
+ [`skill_triggered`](../TASK_DEFINITION_GUIDE.md#skill_triggered) criterion detects
+ that call and scores skill-activation suites.
+- **Plugins** are supplied as `plugins: [{type: local, path: …}]`; the path is
+ env-expanded and resolved to an absolute path before being handed to the SDK.
+- **`PLUGIN_TOOLS_DIR`** pins the canonical `node_modules/@uipath` directory for
+ UiPath CLI plugin discovery; when unset the sandbox derives it from the resolved
+ `uip` binary. See [User Guide → Environment Variables](../USER_GUIDE.md#environment-variables).
+
+## Early stop (Claude-only)
+
+Claude Code is the only agent that supports the cooperative early-stop seam. With
+`run_limits.stop_early: true`, a single-shot run ends cleanly at the next
+tool-call boundary once its **armed** criteria (`stop_when: pass|fail|decided`) are
+decided — so a raised `max_turns` isn't wasted on a smoke run. Early stop errors at
+resolution for any other agent. See the
+[Task Definition Guide](../TASK_DEFINITION_GUIDE.md) for the full contract.
+
+## Telemetry
+
+Claude Code produces the richest telemetry of the agents:
+
+- **Authoritative billing** comes from the SDK's cumulative `model_usage` on the
+ terminal result message and reconciles to `total_cost_usd` — this already includes
+ sub-agent consumption that the per-message stream under-reports.
+- **Sub-agent accounting** is derived by grouping `parent_tool_use_id`-tagged
+ assistant messages; there is no separate per-sub-agent field. The terminal
+ sub-agent generation (delivered as the Agent tool result, never streamed) is
+ synthesized into one message.
+- **Reconciliation message.** Because the per-message stream slightly under-reports
+ the turn total (a fixed ~512-token input slice is billed on no streamed message,
+ and sub-agent input/cache only partially bubbles up), each turn carries one
+ synthetic `role="reconciliation"` entry holding the per-bucket residual. The
+ invariant: summing the token buckets across `TurnRecord.messages` equals the turn
+ total exactly. It carries no cost and is excluded from turn/generation counts.
+
+Set the `CODER_EVAL_RAW_SDK_LOG` environment variable to `1` to dump every raw SDK
+event to the task log.
+
+## Lifecycle & robustness
+
+- **Session resume** across turns via the SDK `resume` option; the session id only
+ advances on clean (non-error) turns.
+- **Timeouts** are enforced by a `ThreadedWatchdog` (an OS-thread timer immune to
+ event-loop stalls) plus an in-loop wall-clock guard; on breach it SIGKILLs the CLI
+ subprocess and raises `TurnTimeoutError` with a partial `TurnRecord` preserved.
+- **Crashes** raise `AgentCrashError` with assembled stderr; the orchestrator drains
+ the partial turn and rolls back. Cost is backfilled from the rate card when a run
+ is killed before a terminal result message arrives.
+
+## References
+
+- [Task Definition Guide](../TASK_DEFINITION_GUIDE.md) — the full task/criterion schema
+- [User Guide](../USER_GUIDE.md) — CLI, backends, environment variables
+- [Codex Agent Guide](CODEX.md) · [Antigravity (Gemini) Agent Guide](ANTIGRAVITY.md)
+- [Extending Coder Eval](../EXTENDING.md) — how agents register via the plugin SPI
diff --git a/docs/CODEX_AGENT_GUIDE.md b/docs/agents/CODEX.md
similarity index 96%
rename from docs/CODEX_AGENT_GUIDE.md
rename to docs/agents/CODEX.md
index eb2cc160..ce84b727 100644
--- a/docs/CODEX_AGENT_GUIDE.md
+++ b/docs/agents/CODEX.md
@@ -1,15 +1,15 @@
---
description: >-
- Run OpenAI Codex as the agent under evaluation in coder_eval — installation,
+ Run OpenAI Codex as the agent under evaluation in Coder Eval — installation,
authentication, task configuration, and how Codex telemetry maps to sandboxed,
weighted scoring.
---
-# Running OpenAI Codex in coder_eval
+# Running OpenAI Codex in Coder Eval
## Overview
-coder_eval can run OpenAI's Codex as the agent under evaluation, via the official Codex SDK. The `CodexAgent` mirrors the structure of `ClaudeCodeAgent` and plugs into the same sandbox, scoring, and telemetry pipeline — set `agent.type: codex` in a task and the rest of the framework works unchanged.
+Coder Eval can run OpenAI's Codex as the agent under evaluation, via the official Codex SDK. The `CodexAgent` mirrors the structure of `ClaudeCodeAgent` and plugs into the same sandbox, scoring, and telemetry pipeline — set `agent.type: codex` in a task and the rest of the framework works unchanged.
## Setup
@@ -75,7 +75,7 @@ coder-eval run tasks/agents/codex_hello_world.yaml --type codex
Or override agent type for all tasks in an experiment:
```bash
-coder-eval run experiments/example.yaml --type codex
+coder-eval run experiments/model-comparison.yaml --type codex
```
### Task Definition (YAML)
@@ -188,7 +188,7 @@ The agent maps `permission_mode` to the Codex SDK's `Sandbox`. The approval mode
| `default` | `workspace-write` | `deny_all` |
| `plan` | `read-only` | `deny_all` |
-`deny_all` means *run autonomously, never prompt, no server-side reviewer*: in-sandbox operations execute directly and only escalations beyond the sandbox are refused. `coder_eval` uses it for every mode because the alternative (`auto_review`) adds a server-side reviewer that can spuriously return `declined` under gateway load.
+`deny_all` means *run autonomously, never prompt, no server-side reviewer*: in-sandbox operations execute directly and only escalations beyond the sandbox are refused. Coder Eval uses it for every mode because the alternative (`auto_review`) adds a server-side reviewer that can spuriously return `declined` under gateway load.
`allowed_tools` / `disallowed_tools` are normalized (`Bash` → `shell`, `Write`/`Edit` → `apply_patch`, etc.) and passed as `enabled_tools` / `disabled_tools` in the thread `config`. **Note:** the Codex SDK does not currently enforce `disabled_tools`; do not rely on it as a security boundary (the agent logs a warning when it is set).
@@ -215,7 +215,7 @@ The Codex SDK is synchronous. The agent uses `_run_async()` helper to detect and
| **Model Selection** | Direct via `--model` or config | `agent.model` pinned into `thread_start` |
| **Session Resume** | `--resume {session_id}` | Via thread ID |
| **Permissions** | `permission_mode` + `allowed_tools` | `permission_mode` → sandbox/approval + `allowed_tools`/`disallowed_tools` → thread config |
-| **Tool Enforcement** | Not enforced by coder_eval wrapper | `enabled_tools` honored; `disabled_tools` NOT enforced by the SDK |
+| **Tool Enforcement** | Not enforced by Coder Eval wrapper | `enabled_tools` honored; `disabled_tools` NOT enforced by the SDK |
## Known Limitations
diff --git a/docs/comparison.md b/docs/comparison.md
index f46b39cc..6dad1f4b 100644
--- a/docs/comparison.md
+++ b/docs/comparison.md
@@ -45,7 +45,7 @@ primary use is a shared leaderboard on a canonical dataset.
Coder Eval is a framework for authoring tasks in YAML, with continuous 0.0–1.0
scoring and cost telemetry rather than a single pass/fail. A fixed dataset can be
-wrapped as tasks via [Bring Your Own Dataset](BYOD.md).
+wrapped as tasks via [Bring Your Own Dataset](DATASETS.md).
**Choose SWE-bench** for a standardized model leaderboard on patch-and-test tasks.
**Choose Coder Eval** to evaluate your own tasks, agents, and skills and to gate CI
@@ -129,9 +129,9 @@ focus on tasks and criteria rather than harness infrastructure.
## Next steps
- [Tutorial 01 — Your First Evaluation](tutorials/01-first-evaluation.md)
-- [Tutorial 02 — Running coder_eval in CI](tutorials/02-ci-pipeline.md)
+- [Tutorial 02 — Running Coder Eval in CI](tutorials/02-ci-pipeline.md)
- [A/B Experiments](AB_EXPERIMENTS.md)
-- [Bring Your Own Dataset](BYOD.md)
+- [Bring Your Own Dataset](DATASETS.md)
## Sources
diff --git a/docs/index.md b/docs/index.md
index 5eca53e6..09656e2a 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -72,23 +72,31 @@ New here? Start with **[Tutorial 01 — Your First Evaluation](tutorials/01-firs
## Where to go next
+
| Guide | What's in it |
| --- | --- |
| [Tutorials](tutorials/README.md) | Step-by-step walkthroughs — start here |
| [User Guide](USER_GUIDE.md) | Full CLI, configuration, output, and environment-variable reference |
| [Task Definition Guide](TASK_DEFINITION_GUIDE.md) | The task-file schema — all criterion types, scoring, templates |
+| [Claude Code](agents/CLAUDE_CODE.md) | Configuring and running the default Claude Code agent |
+| [Codex](agents/CODEX.md) | Running the OpenAI Codex agent |
+| [Antigravity (Gemini)](agents/ANTIGRAVITY.md) | Running the Google Antigravity / Gemini agent |
| [A/B Experiments](AB_EXPERIMENTS.md) | Compare models / tools / prompts across the same tasks |
-| [Bring Your Own Dataset](BYOD.md) | Fan a single task out over a dataset |
-| [Codex Agent Guide](CODEX_AGENT_GUIDE.md) | Running the Codex agent |
-| [Docker Isolation](DOCKER_ISOLATION.md) | The container sandbox driver |
-| [How it compares](comparison.md) | Coder Eval vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, and hand-rolled harnesses |
+| [Bring Your Own Dataset](DATASETS.md) | Fan a single task out over a dataset |
+| [Dialog Mode](DIALOG_MODE.md) | Evaluate agents in multi-turn conversation via a simulated user |
+| [Docker Isolation](DOCKER_ISOLATION.md) | The container sandbox driver, with custom images |
+| [CI Gate & GitHub Action](CI_GATE.md) | Run Coder Eval as a CI gate — the packaged Action, JUnit output, score floor |
+| [Extending Coder Eval](EXTENDING.md) | Author a custom agent, criterion, or model pricing via the plugin SPI |
+| [Report Schema](REPORT_SCHEMA.md) | Field-level reference for run.json / variant.json / task.json |
+| [How It Compares](comparison.md) | vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts |
+
## How Coder Eval compares
- **vs. SWE-bench and fixed benchmarks** — SWE-bench is a fixed dataset; Coder Eval
is a *framework* for authoring your own tasks in YAML, so you evaluate the skills
and workflows you care about (and can still wrap a fixed dataset via
- [Bring Your Own Dataset](BYOD.md)).
+ [Bring Your Own Dataset](DATASETS.md)).
- **vs. other agent-eval frameworks (e.g. Harbor) and LLM-eval tools (OpenAI Evals)** —
OpenAI Evals grades model text; Harbor targets large-scale agent eval and RL
optimization. Coder Eval is purpose-built for coding-agent/skill suites — weighted
diff --git a/docs/llms.txt b/docs/llms.txt
index 11835263..ae33be28 100644
--- a/docs/llms.txt
+++ b/docs/llms.txt
@@ -21,23 +21,33 @@ and A/B plumbing.
## Docs
-- [Home / overview](https://uipath.github.io/coder_eval/): what Coder Eval is, features, quick start
-- [How it compares](https://uipath.github.io/coder_eval/comparison/): vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts
-- [User Guide](https://uipath.github.io/coder_eval/USER_GUIDE/): full CLI, configuration, output, environment variables
-- [Task Definition Guide](https://uipath.github.io/coder_eval/TASK_DEFINITION_GUIDE/): task-file schema, all criterion types, scoring, templates
-- [A/B Experiments](https://uipath.github.io/coder_eval/AB_EXPERIMENTS/): compare models/tools/prompts across the same tasks
-- [Bring Your Own Dataset](https://uipath.github.io/coder_eval/BYOD/): fan a single task out over a dataset
-- [Codex Agent Guide](https://uipath.github.io/coder_eval/CODEX_AGENT_GUIDE/): running the Codex agent
-- [Docker Isolation](https://uipath.github.io/coder_eval/DOCKER_ISOLATION/): the container sandbox driver, custom images
+
+- [Home](https://coder-eval.com/docs): What Coder Eval is, its features, and a quick start
+- [User Guide](https://coder-eval.com/docs/user-guide): Full CLI, configuration, output, and environment-variable reference
+- [Task Definition Guide](https://coder-eval.com/docs/task-definition-guide): The task-file schema — all criterion types, scoring, templates
+- [Claude Code](https://coder-eval.com/docs/agents/claude-code): Configuring and running the default Claude Code agent
+- [Codex](https://coder-eval.com/docs/agents/codex): Running the OpenAI Codex agent
+- [Antigravity (Gemini)](https://coder-eval.com/docs/agents/antigravity): Running the Google Antigravity / Gemini agent
+- [A/B Experiments](https://coder-eval.com/docs/ab-experiments): Compare models / tools / prompts across the same tasks
+- [Bring Your Own Dataset](https://coder-eval.com/docs/datasets): Fan a single task out over a dataset
+- [Dialog Mode](https://coder-eval.com/docs/dialog-mode): Evaluate agents in multi-turn conversation via a simulated user
+- [Docker Isolation](https://coder-eval.com/docs/docker-isolation): The container sandbox driver, with custom images
+- [CI Gate & GitHub Action](https://coder-eval.com/docs/ci-gate): Run Coder Eval as a CI gate — the packaged Action, JUnit output, score floor
+- [Extending Coder Eval](https://coder-eval.com/docs/extending): Author a custom agent, criterion, or model pricing via the plugin SPI
+- [Report Schema](https://coder-eval.com/docs/report-schema): Field-level reference for run.json / variant.json / task.json
+- [How It Compares](https://coder-eval.com/docs/comparison): vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts
+
## Tutorials
-- [01 · Your first evaluation](https://uipath.github.io/coder_eval/tutorials/01-first-evaluation/)
-- [02 · Running coder_eval in CI](https://uipath.github.io/coder_eval/tutorials/02-ci-pipeline/)
-- [03 · Browsing results with evalboard](https://uipath.github.io/coder_eval/tutorials/03-evalboard-local/)
-- [04 · Writing a task](https://uipath.github.io/coder_eval/tutorials/04-writing-a-task/)
-- [05 · Comparing two models](https://uipath.github.io/coder_eval/tutorials/05-comparing-models/)
-- [06 · Docker isolation](https://uipath.github.io/coder_eval/tutorials/06-use-docker-isolation/)
+
+- [01 · Your first evaluation](https://coder-eval.com/docs/tutorials/01-first-evaluation)
+- [02 · Running Coder Eval in CI](https://coder-eval.com/docs/tutorials/02-ci-pipeline)
+- [03 · Browsing results with evalboard](https://coder-eval.com/docs/tutorials/03-evalboard-local)
+- [04 · Writing a task](https://coder-eval.com/docs/tutorials/04-writing-a-task)
+- [05 · Comparing two models](https://coder-eval.com/docs/tutorials/05-comparing-models)
+- [06 · Docker isolation](https://coder-eval.com/docs/tutorials/06-use-docker-isolation)
+
## Source
diff --git a/docs/tutorials/02-ci-pipeline.md b/docs/tutorials/02-ci-pipeline.md
index 6943ae79..ee960a36 100644
--- a/docs/tutorials/02-ci-pipeline.md
+++ b/docs/tutorials/02-ci-pipeline.md
@@ -1,11 +1,11 @@
---
description: >-
- Run coder_eval in GitHub Actions — install the CLI, execute your coding-agent
+ Run Coder Eval in GitHub Actions — install the CLI, execute your coding-agent
evaluation suite on demand or on a schedule, and gate merges on a pass/fail
verdict.
---
-# Tutorial 02 — Running coder_eval in CI
+# Tutorial 02 — Running Coder Eval in CI
Run your evaluation suite automatically in GitHub Actions: on demand, on a
schedule, or as a merge gate. By the end you'll have a workflow that installs
@@ -18,7 +18,7 @@ drives its task tree under
directory checked into the repo, a pinned `coder-eval` version, and a
`task.json`-based verdict — distilled to the parts that transfer to any project.
-## The shape of a coder_eval CI job
+## The shape of a Coder Eval CI job
Every CI setup, however elaborate, is the same five steps:
@@ -160,7 +160,7 @@ jobs:
## Shortcut: the packaged action
-The five steps above spell out the mechanics, but coder_eval also ships a
+The five steps above spell out the mechanics, but Coder Eval also ships a
composite action at the repo root that bundles install + run + JUnit report +
job-summary + fail-on-failure into one step:
@@ -179,10 +179,10 @@ job-summary + fail-on-failure into one step:
The action is agent-agnostic — it installs `coder-eval` but not the agent
runtime, so provide the `claude` CLI (or your agent's runtime) in the job first.
-See the [README "Use as a GitHub Action"](../../README.md#use-as-a-github-action)
-section for the full inputs table and the fork/`pull_request_target` security
-caveat. The rest of this tutorial's hand-rolled workflow is still useful when you
-need finer control than the action's inputs expose.
+See the [CI Gate reference](../CI_GATE.md) for the full inputs table, JUnit
+output, the score-floor gate, and the fork/`pull_request_target` security caveat.
+The rest of this tutorial's hand-rolled workflow is still useful when you need
+finer control than the action's inputs expose.
## Publishing test results (JUnit)
diff --git a/docs/tutorials/03-evalboard-local.md b/docs/tutorials/03-evalboard-local.md
index 9392e997..e9a83a4f 100644
--- a/docs/tutorials/03-evalboard-local.md
+++ b/docs/tutorials/03-evalboard-local.md
@@ -1,6 +1,6 @@
---
description: >-
- Browse coder_eval runs in evalboard, a local web UI — pass rates, per-task
+ Browse Coder Eval runs in evalboard, a local web UI — pass rates, per-task
detail, tool and message timelines, token costs, and artifact downloads,
straight from your filesystem.
---
@@ -69,10 +69,10 @@ Then open **http://localhost:3030**.
- **OSS edition by default.** evalboard runs in its open-source edition with no
configuration; UiPath-internal surfaces are opt-in via `EVALBOARD_EDITION=internal`.
- **No database.** Run listings and detail pages are rendered on the fly from the
- files coder_eval already wrote.
+ files Coder Eval already wrote.
## Next steps
- **Produce more runs to compare** → [A/B Experiments](../AB_EXPERIMENTS.md)
- **Automate runs in CI, then download the artifact and open it here** →
- [Running coder_eval in CI](02-ci-pipeline.md)
+ [Running Coder Eval in CI](02-ci-pipeline.md)
diff --git a/docs/tutorials/04-writing-a-task.md b/docs/tutorials/04-writing-a-task.md
index cdbfc514..dfb8a296 100644
--- a/docs/tutorials/04-writing-a-task.md
+++ b/docs/tutorials/04-writing-a-task.md
@@ -1,6 +1,6 @@
---
description: >-
- Write a coder_eval task YAML from scratch — prompt, isolated sandbox, turn
+ Write a Coder Eval task YAML from scratch — prompt, isolated sandbox, turn
budget, and three success criteria — validate it without spending tokens, then
run and score it.
---
@@ -103,6 +103,6 @@ created are preserved under `runs/latest////` (`task.json`,
- **All 14 criterion types, weights, thresholds** → [Task Definition Guide](../TASK_DEFINITION_GUIDE.md)
- **Stop a run early once the key criteria are decided** (opt-in `run_limits.stop_early` + `stop_when` on a criterion) → [Task Definition Guide → `stop_early`](../TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop)
-- **Fan one task out over a dataset of rows** → [Bring Your Own Data](../BYOD.md)
+- **Fan one task out over a dataset of rows** → [Bring Your Own Dataset](../DATASETS.md)
- **Full CLI & config reference** → [User Guide](../USER_GUIDE.md)
- **Compare two configurations on this task** → [Tutorial 05](05-comparing-models.md)
diff --git a/docs/tutorials/05-comparing-models.md b/docs/tutorials/05-comparing-models.md
index ec3539c0..a6ff5297 100644
--- a/docs/tutorials/05-comparing-models.md
+++ b/docs/tutorials/05-comparing-models.md
@@ -1,6 +1,6 @@
---
description: >-
- A/B-test two Claude models on the same task with coder_eval's experiment layer
+ A/B-test two Claude models on the same task with Coder Eval's experiment layer
and read the cross-variant report — a fair, like-for-like coding-agent model
comparison.
---
diff --git a/docs/tutorials/06-use-docker-isolation.md b/docs/tutorials/06-use-docker-isolation.md
index 9f0931e0..2e6b4d25 100644
--- a/docs/tutorials/06-use-docker-isolation.md
+++ b/docs/tutorials/06-use-docker-isolation.md
@@ -1,6 +1,6 @@
---
description: >-
- Switch a coder_eval task from the temp-dir sandbox to a fresh Docker container
+ Switch a Coder Eval task from the temp-dir sandbox to a fresh Docker container
per task — strong host isolation, a pinned toolchain, and task-specific
dependencies.
---
diff --git a/docs/tutorials/README.md b/docs/tutorials/README.md
index 22d95a43..91fb924d 100644
--- a/docs/tutorials/README.md
+++ b/docs/tutorials/README.md
@@ -1,6 +1,6 @@
---
description: >-
- Step-by-step coder_eval tutorials — run your first AI coding-agent evaluation,
+ Step-by-step Coder Eval tutorials — run your first AI coding-agent evaluation,
wire it into CI, browse results, write tasks, compare models, and isolate runs
in Docker.
---
@@ -14,7 +14,7 @@ the task-file schema see the [Task Definition Guide](../TASK_DEFINITION_GUIDE.md
| # | Tutorial | You'll learn |
| - | --- | --- |
| 01 | [Your first evaluation](01-first-evaluation.md) | Install, configure a key, and run + read your first eval |
-| 02 | [Running coder_eval in CI](02-ci-pipeline.md) | Wire the suite into GitHub Actions with a pass/fail gate |
+| 02 | [Running Coder Eval in CI](02-ci-pipeline.md) | Wire the suite into GitHub Actions with a pass/fail gate |
| 03 | [Browsing results with evalboard](03-evalboard-local.md) | Explore runs in a local web UI — pass rates, timelines, artifacts |
| 04 | [Writing a task](04-writing-a-task.md) | Author a task YAML with success criteria from scratch |
| 05 | [Comparing two models](05-comparing-models.md) | Use the experiment layer to A/B two configurations |
diff --git a/mkdocs.yml b/mkdocs.yml
index 9952af4c..bf2f2e07 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -3,7 +3,7 @@ site_description: >-
Evaluate & benchmark AI coding agents and Claude Code skills — a sandboxed,
reproducible framework with declarative YAML eval suites for Claude Code,
Codex & Gemini, A/B experiments, weighted scoring, and CI gates.
-site_url: https://uipath.github.io/coder_eval/
+site_url: https://coder-eval.com/docs/
repo_url: https://github.com/UiPath/coder_eval
repo_name: UiPath/coder_eval
edit_uri: edit/main/docs/
@@ -38,11 +38,6 @@ theme:
icon:
repo: fontawesome/brands/github
-# IDEAS.md is internal engineering backlog (links to source files) — not for the
-# public docs site.
-exclude_docs: |
- IDEAS.md
-
# The social plugin generates per-page Open Graph / Twitter card images (and
# the matching og:/twitter: meta tags) so shared links unfurl properly. It
# needs Cairo system libraries, so it is enabled only in CI (the Docs workflow
@@ -73,13 +68,36 @@ extra:
link: https://github.com/UiPath/coder_eval
- icon: fontawesome/brands/python
link: https://pypi.org/project/coder-eval/
+ # docs_index: one-line blurb per nav page (docs-root-relative path -> blurb),
+ # for every page EXCEPT tutorial leaves (tutorials/0*.md). This map plus `nav`
+ # are the single source of truth for the flat index surfaces — README.md's
+ # Documentation table, docs/index.md's "Where to go next" table, and the
+ # ## Docs / ## Tutorials sections of docs/llms.txt. Regenerate them all with
+ # `make docs-indexes`; CE028 fails the build if any of the three drifts, if a
+ # nav page has no blurb (or vice-versa), or if a docs/ page is missing from nav.
+ docs_index:
+ index.md: "What Coder Eval is, its features, and a quick start"
+ tutorials/README.md: "Step-by-step walkthroughs — start here"
+ USER_GUIDE.md: "Full CLI, configuration, output, and environment-variable reference"
+ TASK_DEFINITION_GUIDE.md: "The task-file schema — all criterion types, scoring, templates"
+ agents/CLAUDE_CODE.md: "Configuring and running the default Claude Code agent"
+ agents/CODEX.md: "Running the OpenAI Codex agent"
+ agents/ANTIGRAVITY.md: "Running the Google Antigravity / Gemini agent"
+ AB_EXPERIMENTS.md: "Compare models / tools / prompts across the same tasks"
+ DATASETS.md: "Fan a single task out over a dataset"
+ DIALOG_MODE.md: "Evaluate agents in multi-turn conversation via a simulated user"
+ DOCKER_ISOLATION.md: "The container sandbox driver, with custom images"
+ CI_GATE.md: "Run Coder Eval as a CI gate — the packaged Action, JUnit output, score floor"
+ EXTENDING.md: "Author a custom agent, criterion, or model pricing via the plugin SPI"
+ REPORT_SCHEMA.md: "Field-level reference for run.json / variant.json / task.json"
+ comparison.md: "vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts"
nav:
- Home: index.md
- Tutorials:
- Overview: tutorials/README.md
- 01 · Your first evaluation: tutorials/01-first-evaluation.md
- - 02 · Running coder_eval in CI: tutorials/02-ci-pipeline.md
+ - 02 · Running Coder Eval in CI: tutorials/02-ci-pipeline.md
- 03 · Browsing results with evalboard: tutorials/03-evalboard-local.md
- 04 · Writing a task: tutorials/04-writing-a-task.md
- 05 · Comparing two models: tutorials/05-comparing-models.md
@@ -87,8 +105,16 @@ nav:
- Guides:
- User Guide: USER_GUIDE.md
- Task Definition Guide: TASK_DEFINITION_GUIDE.md
+ - Agents:
+ - Claude Code: agents/CLAUDE_CODE.md
+ - Codex: agents/CODEX.md
+ - Antigravity (Gemini): agents/ANTIGRAVITY.md
+ - Advanced:
- A/B Experiments: AB_EXPERIMENTS.md
- - Bring Your Own Dataset: BYOD.md
- - Codex Agent Guide: CODEX_AGENT_GUIDE.md
+ - Bring Your Own Dataset: DATASETS.md
+ - Dialog Mode: DIALOG_MODE.md
- Docker Isolation: DOCKER_ISOLATION.md
- - How it compares: comparison.md
+ - CI Gate & GitHub Action: CI_GATE.md
+ - Extending Coder Eval: EXTENDING.md
+ - Report Schema: REPORT_SCHEMA.md
+ - How It Compares: comparison.md
diff --git a/pyproject.toml b/pyproject.toml
index 03c1de2a..120c90b3 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -53,7 +53,7 @@ dependencies = [
[project.urls]
Homepage = "https://coder-eval.com"
Repository = "https://github.com/UiPath/coder_eval"
-Documentation = "https://github.com/UiPath/coder_eval/tree/main/docs"
+Documentation = "https://coder-eval.com/docs"
Issues = "https://github.com/UiPath/coder_eval/issues"
Changelog = "https://github.com/UiPath/coder_eval/blob/main/CHANGELOG.md"
diff --git a/src/coder_eval/models/tasks.py b/src/coder_eval/models/tasks.py
index c566d73d..77b9169b 100644
--- a/src/coder_eval/models/tasks.py
+++ b/src/coder_eval/models/tasks.py
@@ -115,10 +115,6 @@ class SimulationConfig(BaseModel):
le=100,
description="Number of independent dialog trajectories to run per (task, variant).",
)
- parallel_trials: bool = Field(
- default=True,
- description="When True, trials run concurrently (subject to batch max_parallel).",
- )
# Criteria timing.
check_criteria: CriteriaCheckTiming = Field(
@@ -238,10 +234,11 @@ class Dataset(BaseModel):
sample_seed: int | None = Field(
default=None,
description=(
- "(re-draws every night; broadens coverage for the nightly suites) — this holds whether "
- "the count comes from YAML or the CLI --sample-per-stratum flag. NOTE: unlike CLI --sample "
- "(fixed-seed, reproducible by default), the stratified sample_per_stratum draw is "
- "NONDETERMINISTIC by default; set an integer here to pin a reproducible sample."
+ "Seed for the stratified 'sample_per_stratum' draw. Unlike CLI --sample (fixed-seed, "
+ "reproducible by default), that draw is NONDETERMINISTIC when this is None: it re-draws "
+ "every run, which broadens coverage for the nightly suites. This holds whether the "
+ "per-stratum count comes from YAML or the CLI --sample-per-stratum flag. Set an integer "
+ "here to pin a reproducible sample."
),
)
diff --git a/tasks/python_cli_simulated_judged/echo_simulated_judged.yaml b/tasks/python_cli_simulated_judged/echo_simulated_judged.yaml
index 24be765c..f6a6578d 100644
--- a/tasks/python_cli_simulated_judged/echo_simulated_judged.yaml
+++ b/tasks/python_cli_simulated_judged/echo_simulated_judged.yaml
@@ -54,7 +54,6 @@ simulation:
stop_on_criteria_pass: false
check_criteria: end_of_dialog
n_trials: 1
- parallel_trials: true
success_criteria:
- type: llm_judge
diff --git a/tests/lint/dead_config_fields.py b/tests/lint/dead_config_fields.py
new file mode 100644
index 00000000..5c5ddb6f
--- /dev/null
+++ b/tests/lint/dead_config_fields.py
@@ -0,0 +1,83 @@
+"""CE031 — behavior-driving config fields must be consumed somewhere in ``src/``.
+
+A Pydantic field on a config model that users set in a task YAML but that no code
+ever reads is *dead config*: it silently does nothing, and the author has no way
+to know. This is what ``SimulationConfig.parallel_trials`` was — documented, set
+in a shipped task YAML, defaulting to ``True``, and read nowhere (trial
+concurrency is entirely ``--max-parallel``'s job). CE031 makes that class
+impossible to reintroduce for a small, explicit registry of models.
+
+"Consumed" here means the field name appears as an **attribute access**
+(``x.field``) anywhere under ``src/`` — the consumption contract for a
+*behavior-driving* config: the orchestrator/validators must read the field by
+name for it to have any effect. A field read only via ``model_dump()`` /
+serialization is NOT caught by this definition, which is exactly why the registry
+is restricted to behavior models (``SimulationConfig``, ``RunLimits``,
+``Dataset``) and does **not** include serialization/telemetry models or the
+sprawling ``TaskDefinition`` (whose fields are largely round-tripped through
+``model_dump`` in the dataset expander).
+
+Known floor (documented, accepted): attribute names collide across models — if
+``RunLimits`` and ``SimulationConfig`` both declare ``max_turns`` and only one is
+read by name, both count as consumed. Collisions cause **false negatives** (a dead
+field masked by a same-named live one elsewhere), never false positives, so the
+rule can never wrongly break the build. An ``EXEMPT`` map covers any field that is
+legitimately consumed only via serialization, with a reason.
+
+Like CE027 through CE030, this is not a ``BaseRule`` in the AST runner (that runner walks
+one file at a time and reports line-level violations; this rule reasons over the
+*whole* ``src/`` tree at once). It is wired as
+``tests/test_custom_lint.py::TestCE031DeadConfigFields``.
+"""
+
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+from pydantic import BaseModel
+
+from coder_eval.models import Dataset, RunLimits, SimulationConfig
+
+
+# Behavior-driving config models whose fields MUST be read by name to do anything.
+# Deliberately excludes serialization/telemetry models and TaskDefinition (fields
+# round-tripped through model_dump would false-positive under the attribute rule).
+CONSUMED_MODELS: list[type[BaseModel]] = [
+ SimulationConfig,
+ RunLimits,
+ Dataset,
+]
+
+# Fields legitimately consumed only via serialization (not a by-name attribute
+# read), with the reason. Empty today; an entry here is a promise the field IS
+# used, just not through attribute access. Every entry must name a real field.
+EXEMPT: dict[str, dict[str, str]] = {}
+
+
+def consumed_attr_names(src_root: Path) -> set[str]:
+ """Every attribute name read anywhere under ``src/`` (``x.attr`` -> ``attr``)."""
+ names: set[str] = set()
+ for py in src_root.rglob("*.py"):
+ tree = ast.parse(py.read_text(encoding="utf-8"))
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Attribute):
+ names.add(node.attr)
+ return names
+
+
+def dead_config_fields(model: type[BaseModel], consumed: set[str], exempt: dict[str, str]) -> list[str]:
+ """Fields of ``model`` neither read as an attribute in ``src/`` nor exempted."""
+ return [name for name in model.model_fields if name not in exempt and name not in consumed]
+
+
+def find_dead_config_fields(src_root: Path) -> dict[str, list[str]]:
+ """Map ``Model`` name to its dead (unconsumed) fields, for every registered model."""
+ consumed = consumed_attr_names(src_root)
+ findings: dict[str, list[str]] = {}
+ for model in CONSUMED_MODELS:
+ exempt = EXEMPT.get(model.__name__, {})
+ dead = dead_config_fields(model, consumed, exempt)
+ if dead:
+ findings[model.__name__] = dead
+ return findings
diff --git a/tests/lint/doc_examples.py b/tests/lint/doc_examples.py
new file mode 100644
index 00000000..ff807365
--- /dev/null
+++ b/tests/lint/doc_examples.py
@@ -0,0 +1,211 @@
+"""CE029 — self-contained YAML examples in the docs must validate against their models.
+
+A published example that does not parse is worse than no example: readers copy it,
+hit a ``ValidationError``, and conclude the feature is broken. This rule caught
+exactly that — the ``prompt_mutations`` recipe in ``docs/AB_EXPERIMENTS.md`` used
+``text:`` where the field is ``content:``, and every mutation model declares
+``extra="forbid"``, so the published snippet raised
+``variants.1.prompt_mutations.0.suffix.content Field required``.
+
+Scope is deliberately narrow — the rule only validates blocks it can *prove* are
+whole documents, because a false positive on an illustrative fragment would make
+``make lint`` a nuisance and get the rule deleted:
+
+* a block is a **task** when it has ``task_id`` + ``initial_prompt`` +
+ ``success_criteria``, and an **experiment** when it has ``experiment_id`` +
+ ``variants``. Anything else is a fragment and is skipped — including a bare
+ ``success_criteria:`` list, which is the single most common doc shape.
+* **schematic** blocks are skipped: a doc that writes ``agent: { ... }`` to mean
+ "and so on" parses to the literal key ``"..."``. The task guide's overview block
+ uses that form deliberately.
+* a block that is not valid YAML at all is skipped rather than reported — it
+ cannot be classified, so the rule has no basis for claiming it is a broken
+ *example* as opposed to deliberately-invalid illustrative text.
+* escape hatch: a block preceded by ```` is skipped,
+ for a future example that is intentionally partial in a way the heuristic
+ cannot see.
+
+Like CE027, this is intentionally NOT a ``BaseRule`` registered in
+``tests/lint/runner.py``: that runner is AST-only and walks ``.py`` files, whereas
+this rule reasons over Markdown. It is wired as a dedicated test in
+``tests/test_custom_lint.py::TestCE029DocYamlExamples``.
+"""
+
+from __future__ import annotations
+
+import warnings
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+import yaml
+
+
+SKIP_MARKER = ""
+
+_YAML_INFO_STRINGS = frozenset({"yaml", "yml"})
+
+# Keys that make a block a whole document rather than an illustrative fragment.
+_TASK_KEYS = frozenset({"task_id", "initial_prompt", "success_criteria"})
+_EXPERIMENT_KEYS = frozenset({"experiment_id", "variants"})
+
+# The "and so on" placeholder docs use inside a flow mapping (``agent: { ... }``),
+# which YAML parses as the literal key ``...``.
+_SCHEMATIC = "..."
+
+
+@dataclass(frozen=True)
+class YamlBlock:
+ """One fenced ```yaml block lifted out of a Markdown file."""
+
+ path: Path
+ line: int # 1-based line number of the opening fence
+ text: str
+ skip_marked: bool
+
+
+def extract_yaml_blocks(path: Path, text: str) -> list[YamlBlock]:
+ """Pull every fenced ```yaml / ```yml block out of a Markdown document.
+
+ Tracks the opening fence's exact backtick run so a nested fence (pymdownx
+ superfences allows ````) closes against the right marker rather than the
+ first ``` it meets.
+ """
+ blocks: list[YamlBlock] = []
+ lines = text.splitlines()
+ i = 0
+ while i < len(lines):
+ stripped = lines[i].strip()
+ if not stripped.startswith("```"):
+ i += 1
+ continue
+ ticks = len(stripped) - len(stripped.lstrip("`"))
+ # First info-string token only — mkdocs-material allows attributes
+ # (```yaml title="x"), and skipping those would silently drop a block.
+ info = stripped[ticks:].strip().lower().split()
+ if not info or info[0] not in _YAML_INFO_STRINGS:
+ i += 1
+ continue
+ fence = "`" * ticks
+ start = i
+ body: list[str] = []
+ i += 1
+ while i < len(lines):
+ closing = lines[i].strip()
+ if closing.startswith(fence) and not closing[ticks:].strip():
+ break
+ body.append(lines[i])
+ i += 1
+ blocks.append(
+ YamlBlock(
+ path=path,
+ line=start + 1,
+ text="\n".join(body),
+ skip_marked=_has_skip_marker(lines, start),
+ )
+ )
+ i += 1
+ return blocks
+
+
+def _has_skip_marker(lines: list[str], fence_index: int) -> bool:
+ """True when ```` precedes the fence (blank lines allowed)."""
+ j = fence_index - 1
+ while j >= 0 and not lines[j].strip():
+ j -= 1
+ return j >= 0 and lines[j].strip() == SKIP_MARKER
+
+
+def is_schematic(data: Any) -> bool:
+ """True when the parsed block uses the literal ``...`` placeholder anywhere.
+
+ ``agent: { ... }`` means "and other keys" to a reader and parses to the key
+ ``"..."``. Such a block documents *shape*, not a runnable document.
+ """
+ if isinstance(data, dict):
+ return any(k == _SCHEMATIC or is_schematic(v) for k, v in data.items())
+ if isinstance(data, list):
+ return any(is_schematic(v) for v in data)
+ return data == _SCHEMATIC
+
+
+def classify(data: Any) -> str | None:
+ """``"task"``, ``"experiment"``, or None when the block is a fragment."""
+ if not isinstance(data, dict):
+ return None
+ keys = set(data)
+ if keys >= _TASK_KEYS:
+ return "task"
+ if keys >= _EXPERIMENT_KEYS:
+ return "experiment"
+ return None
+
+
+def validate_block(block: YamlBlock) -> str | None:
+ """Validate one block; return an error string, or None when it passes or is skipped."""
+ from coder_eval.models import ExperimentDefinition, TaskDefinition
+
+ if block.skip_marked:
+ return None
+ try:
+ data = yaml.safe_load(block.text)
+ except yaml.YAMLError:
+ return None # unclassifiable — see the module docstring
+ if is_schematic(data):
+ return None
+ kind = classify(data)
+ if kind is None:
+ return None
+
+ model = TaskDefinition if kind == "task" else ExperimentDefinition
+ try:
+ with warnings.catch_warnings():
+ # Unknown top-level task fields warn rather than raise (soft-launch
+ # policy); that is CE009's territory, not this rule's.
+ warnings.simplefilter("ignore")
+ model.model_validate(data)
+ except Exception as exc: # any validation failure is the finding
+ return f"line {block.line} ({kind}): {_first_error(exc)}"
+ return None
+
+
+def _first_error(exc: Exception) -> str:
+ """The first concrete validation error, not Pydantic's "N validation errors" header.
+
+ The header alone ("2 validation errors for ExperimentDefinition") tells an
+ author nothing about which key is wrong, which is the whole point of the rule.
+ """
+ errors = getattr(exc, "errors", None)
+ if callable(errors):
+ try:
+ first = errors()[0]
+ except (IndexError, TypeError):
+ return str(exc).strip().splitlines()[0]
+ loc = ".".join(str(p) for p in first.get("loc", ()))
+ return f"{loc}: {first.get('msg', '')}".strip(": ")
+ return str(exc).strip().splitlines()[0]
+
+
+def find_invalid_doc_examples(doc_paths: list[Path]) -> dict[str, list[str]]:
+ """Map each doc path to the self-contained YAML examples in it that don't validate."""
+ findings: dict[str, list[str]] = {}
+ for path in doc_paths:
+ if not path.is_file():
+ continue
+ errors = [
+ err
+ for block in extract_yaml_blocks(path, path.read_text(encoding="utf-8"))
+ if (err := validate_block(block)) is not None
+ ]
+ if errors:
+ findings[str(path)] = errors
+ return findings
+
+
+def default_doc_paths(repo_root: Path) -> list[Path]:
+ """The Markdown surfaces CE029 scans: README plus every page under docs/."""
+ paths = [repo_root / "README.md"]
+ docs = repo_root / "docs"
+ if docs.is_dir():
+ paths.extend(sorted(docs.rglob("*.md")))
+ return paths
diff --git a/tests/lint/doc_indexes.py b/tests/lint/doc_indexes.py
new file mode 100644
index 00000000..1a29825e
--- /dev/null
+++ b/tests/lint/doc_indexes.py
@@ -0,0 +1,283 @@
+"""CE028 — the flat doc-index surfaces are generated from the mkdocs nav.
+
+The docs overhaul's root cause was doc/code drift; the index surfaces
+(``README.md``'s Documentation table, ``docs/index.md``'s "Where to go next"
+table, and the ``## Docs`` / ``## Tutorials`` sections of ``docs/llms.txt``) drift
+the same way — a page is added to the nav and forgotten in the three flat lists,
+or a page is deleted and left dangling in them. This module makes ``nav`` (plus
+the ``extra.docs_index`` blurb map) in ``mkdocs.yml`` the single source of truth:
+``write()`` renders all three surfaces from it, ``make docs-indexes`` calls
+``write()``, and CE028 (``check()``) re-renders and diffs against disk. There is
+deliberately **no ``--check`` mode and no arg parser** — CE028 *is* the checker; a
+second entry point would be untested duplication.
+
+CE028 also enforces the invariants the render depends on: every nav page has a
+blurb and every blurb has a nav page (bijection, tutorial leaves exempted), every
+published ``docs/*.md`` is in the nav (the check that would have caught this whole
+overhaul's bug class), and the hand-written ``docs/tutorials/README.md`` table
+stays in parity with the nav's tutorial pages (that table is **checked, not
+generated** — generating it would need a second per-page field).
+
+Like CE027/CE029/CE030 this is not a ``BaseRule`` in the AST runner; it reasons
+over Markdown/YAML and is wired as
+``tests/test_custom_lint.py::TestCE028DocIndexParity``.
+"""
+
+from __future__ import annotations
+
+import difflib
+import re
+from pathlib import Path
+from typing import NamedTuple
+
+import yaml
+
+
+# --- Markers (already present in the target files; write() fills between them) ---
+_START = ""
+_END = ""
+_START_DOCS = ""
+_END_DOCS = ""
+_START_TUT = ""
+_END_TUT = ""
+
+_SITE = "https://coder-eval.com"
+_NAV_EXCLUDE_FROM_NAV_CHECK = {"IDEAS.md"} # published outside the nav on purpose
+_TUTORIALS_GROUP = "Tutorials"
+_TUTORIAL_README = "tutorials/README.md"
+
+
+class NavPage(NamedTuple):
+ """One resolved nav leaf, in nav order.
+
+ ``group`` is the parent group label (e.g. ``"Advanced"``) or None for a
+ top-level page; ``doc`` is the docs-root-relative path (``agents/CODEX.md``).
+ """
+
+ group: str | None
+ label: str
+ doc: str
+
+
+def _is_tutorial_leaf(doc: str) -> bool:
+ """A numbered tutorial page (``tutorials/01-...md``) — no blurb, own llms section."""
+ return doc.startswith("tutorials/0")
+
+
+def _load_mkdocs(mkdocs_path: Path) -> dict:
+ """Parse ``mkdocs.yml`` with a private ``SafeLoader`` subclass.
+
+ ``mkdocs.yml`` carries the custom tag ``!ENV [CI, false]``, which
+ ``yaml.safe_load`` rejects. A private ``SafeLoader`` **subclass** tolerates
+ local ``!``-prefixed tags without touching global YAML state — mutating the
+ shared ``SafeLoader`` would disable unknown-tag rejection for every other
+ ``yaml.safe_load`` in the same pytest worker under ``-n auto``. ``!!python/...``
+ tags expand to the ``tag:yaml.org,2002:`` handle, which this multi-constructor
+ does NOT match, so the subclass still rejects them — this stays
+ safe_load-equivalent for object construction (proven in the CE028 tests).
+ """
+
+ class _NavLoader(yaml.SafeLoader):
+ """Private loader: tolerates mkdocs custom tags (!ENV) without touching global state."""
+
+ _NavLoader.add_multi_constructor("!", lambda loader, suffix, node: None)
+ # yaml.load with a private SafeLoader subclass — safe_load-equivalent (see above), not unsafe_load.
+ return yaml.load(mkdocs_path.read_text(encoding="utf-8"), Loader=_NavLoader)
+
+
+def load_nav(mkdocs_path: Path) -> list[NavPage]:
+ """Parse ``nav:`` into an ordered list of ``NavPage`` (nav order preserved)."""
+ data = _load_mkdocs(mkdocs_path)
+ pages: list[NavPage] = []
+ for entry in data["nav"]:
+ ((label, value),) = entry.items()
+ if isinstance(value, str):
+ pages.append(NavPage(group=None, label=label, doc=value))
+ elif isinstance(value, list):
+ for sub in value:
+ ((sub_label, sub_doc),) = sub.items()
+ pages.append(NavPage(group=label, label=sub_label, doc=sub_doc))
+ return pages
+
+
+def load_blurbs(mkdocs_path: Path) -> dict[str, str]:
+ """The ``extra.docs_index`` map: docs-root-relative path -> one-line blurb."""
+ data = _load_mkdocs(mkdocs_path)
+ return dict(data.get("extra", {}).get("docs_index", {}))
+
+
+def route_for(doc: str) -> str:
+ """The published site route for a docs-root-relative path.
+
+ ``index.md`` and any ``README.md`` map to their directory landing route;
+ other segments are kebab-cased (``_``/space -> ``-``, lowered). Examples:
+ ``index.md`` -> ``/docs``; ``USER_GUIDE.md`` -> ``/docs/user-guide``;
+ ``agents/CLAUDE_CODE.md`` -> ``/docs/agents/claude-code``;
+ ``tutorials/README.md`` -> ``/docs/tutorials``.
+ """
+ stem = doc[:-3] if doc.endswith(".md") else doc
+ parts = stem.split("/")
+ if parts[-1] in ("index", "README"):
+ parts = parts[:-1]
+ suffix = "/".join(seg.replace("_", "-").replace(" ", "-").lower() for seg in parts)
+ return "/docs" + (f"/{suffix}" if suffix else "")
+
+
+def _table_rows(nav: list[NavPage]) -> list[tuple[str, str]]:
+ """(label, doc) rows for the README/index tables: nav order, ``index.md``
+ skipped, the whole Tutorials group collapsed to one ``Tutorials`` row."""
+ rows: list[tuple[str, str]] = []
+ tutorials_done = False
+ for page in nav:
+ if page.doc == "index.md":
+ continue
+ if page.group == _TUTORIALS_GROUP:
+ if not tutorials_done:
+ rows.append((_TUTORIALS_GROUP, _TUTORIAL_README))
+ tutorials_done = True
+ continue
+ rows.append((page.label, page.doc))
+ return rows
+
+
+def render_readme_table(nav: list[NavPage], blurbs: dict[str, str]) -> str:
+ """README Documentation table — links are repo-relative (``docs/...``)."""
+ lines = ["| Guide | What's in it |", "| --- | --- |"]
+ lines += [f"| [{label}](docs/{doc}) | {blurbs[doc]} |" for label, doc in _table_rows(nav)]
+ return "\n".join(lines)
+
+
+def render_index_table(nav: list[NavPage], blurbs: dict[str, str]) -> str:
+ """docs/index.md table — links are docs-relative (page lives under ``docs/``)."""
+ lines = ["| Guide | What's in it |", "| --- | --- |"]
+ lines += [f"| [{label}]({doc}) | {blurbs[doc]} |" for label, doc in _table_rows(nav)]
+ return "\n".join(lines)
+
+
+def render_llms_docs(nav: list[NavPage], blurbs: dict[str, str]) -> str:
+ """llms.txt ``## Docs`` — every non-tutorial page (incl. index.md), absolute links."""
+ lines = []
+ for page in nav:
+ if page.group == _TUTORIALS_GROUP:
+ continue
+ lines.append(f"- [{page.label}]({_SITE}{route_for(page.doc)}): {blurbs[page.doc]}")
+ return "\n".join(lines)
+
+
+def render_llms_tutorials(nav: list[NavPage]) -> str:
+ """llms.txt ``## Tutorials`` — the numbered tutorial leaves, absolute links, no blurb."""
+ lines = []
+ for page in nav:
+ if page.group == _TUTORIALS_GROUP and _is_tutorial_leaf(page.doc):
+ lines.append(f"- [{page.label}]({_SITE}{route_for(page.doc)})")
+ return "\n".join(lines)
+
+
+# --- bijection / coverage checks --------------------------------------------
+
+
+def _blurb_required_docs(nav: list[NavPage]) -> list[str]:
+ """Docs that must carry a blurb: every nav page except tutorial leaves."""
+ return [p.doc for p in nav if not _is_tutorial_leaf(p.doc)]
+
+
+def missing_blurbs(nav: list[NavPage], blurbs: dict[str, str]) -> list[str]:
+ """Nav pages (tutorial leaves exempt) that have no ``extra.docs_index`` blurb."""
+ return [doc for doc in _blurb_required_docs(nav) if doc not in blurbs]
+
+
+def orphan_blurbs(nav: list[NavPage], blurbs: dict[str, str]) -> list[str]:
+ """Blurb entries with no matching nav page (renamed/removed page left behind)."""
+ required = set(_blurb_required_docs(nav))
+ return sorted(k for k in blurbs if k not in required)
+
+
+def docs_missing_from_nav(repo_root: Path, nav: list[NavPage]) -> list[str]:
+ """Published ``docs/*.md`` pages absent from the nav (the caught-nothing bug class)."""
+ nav_docs = {p.doc for p in nav}
+ missing: list[str] = []
+ docs_dir = repo_root / "docs"
+ for md in sorted(docs_dir.rglob("*.md")):
+ rel = md.relative_to(docs_dir).as_posix()
+ if rel in _NAV_EXCLUDE_FROM_NAV_CHECK or rel in nav_docs:
+ continue
+ missing.append(rel)
+ return missing
+
+
+def tutorials_table_drift(repo_root: Path, nav: list[NavPage]) -> str | None:
+ """Assert docs/tutorials/README.md's table links equal the nav's tutorial leaves.
+
+ Checked, not generated — that table carries a "you'll learn" column the nav
+ doesn't, so regenerating it would reintroduce a dual source of truth.
+ """
+ table = repo_root / "docs/tutorials/README.md"
+ linked = re.findall(r"\]\((\d\d-[\w-]+\.md)\)", table.read_text(encoding="utf-8"))
+ nav_leaves = [p.doc.split("/")[-1] for p in nav if p.group == _TUTORIALS_GROUP and _is_tutorial_leaf(p.doc)]
+ if linked != nav_leaves:
+ return f"docs/tutorials/README.md links {linked} but nav has tutorial leaves {nav_leaves}"
+ return None
+
+
+# --- rendering to disk -------------------------------------------------------
+
+
+def _replace_between(text: str, start: str, end: str, body: str) -> str:
+ """Replace the content between the ``start`` and ``end`` marker lines with ``body``."""
+ pattern = re.compile(re.escape(start) + r"\n.*?\n" + re.escape(end), re.DOTALL)
+ if not pattern.search(text):
+ raise ValueError(f"marker pair {start!r} .. {end!r} not found in the target file")
+ return pattern.sub(lambda _m: f"{start}\n{body}\n{end}", text)
+
+
+def _rendered_files(repo_root: Path) -> dict[Path, str]:
+ """The full intended content of each generated file, keyed by path."""
+ mkdocs = repo_root / "mkdocs.yml"
+ nav = load_nav(mkdocs)
+ blurbs = load_blurbs(mkdocs)
+
+ readme = repo_root / "README.md"
+ index = repo_root / "docs/index.md"
+ llms = repo_root / "docs/llms.txt"
+
+ out: dict[Path, str] = {}
+ out[readme] = _replace_between(readme.read_text(encoding="utf-8"), _START, _END, render_readme_table(nav, blurbs))
+ out[index] = _replace_between(index.read_text(encoding="utf-8"), _START, _END, render_index_table(nav, blurbs))
+ llms_text = llms.read_text(encoding="utf-8")
+ llms_text = _replace_between(llms_text, _START_DOCS, _END_DOCS, render_llms_docs(nav, blurbs))
+ llms_text = _replace_between(llms_text, _START_TUT, _END_TUT, render_llms_tutorials(nav))
+ out[llms] = llms_text
+ return out
+
+
+def write(repo_root: Path) -> list[Path]:
+ """Regenerate all three index surfaces in place. Returns the files touched."""
+ written: list[Path] = []
+ for path, text in _rendered_files(repo_root).items():
+ if path.read_text(encoding="utf-8") != text:
+ path.write_text(text, encoding="utf-8")
+ written.append(path)
+ return written
+
+
+def check(repo_root: Path) -> dict[str, str]:
+ """Unified diff per file whose generated content differs from disk (empty = clean)."""
+ findings: dict[str, str] = {}
+ for path, text in _rendered_files(repo_root).items():
+ current = path.read_text(encoding="utf-8")
+ if current != text:
+ diff = difflib.unified_diff(
+ current.splitlines(),
+ text.splitlines(),
+ fromfile=f"{path} (on disk)",
+ tofile=f"{path} (generated)",
+ lineterm="",
+ )
+ findings[str(path)] = "\n".join(diff)
+ return findings
+
+
+if __name__ == "__main__":
+ root = Path(__file__).resolve().parents[2]
+ for p in write(root):
+ print(f"wrote {p}")
diff --git a/tests/lint/doc_schema_parity.py b/tests/lint/doc_schema_parity.py
new file mode 100644
index 00000000..5ded814d
--- /dev/null
+++ b/tests/lint/doc_schema_parity.py
@@ -0,0 +1,85 @@
+"""CE030 — models the project commits to documenting must have no undocumented fields.
+
+Every defect this docs overhaul fixed was the same failure: a doc claim that no
+longer matched (or never matched) the code. P0 and P1 were literally "a Pydantic
+field the user must set, documented nowhere." CE030 is the sensor that makes that
+class impossible to reintroduce: for a small, explicit registry of user-facing
+models, every field must appear in the model's doc page as inline code, or be
+listed in ``EXEMPT`` with a reason it is not user-authored.
+
+Design choices, each load-bearing:
+
+* **Allowlist, not denylist.** A new field on a registered model that is neither
+ documented nor exempted *fails* — which is the point. Adding a user-facing field
+ now forces a doc update or a reasoned exemption in the same change.
+* **Explicit registry, no recursion.** Only the four registered models are
+ checked; nested models (``AgentConfig``, ``SandboxConfig``, criteria, …) are NOT
+ walked. Walking them would silently expand the documentation commitment to
+ dozens of models nobody signed up for.
+* **Inline-code match, deliberately simple.** A field counts as documented when
+ its bare name appears wrapped in Markdown inline-code backticks anywhere in the
+ doc. This is a floor, not a proof — a field name that appears in an unrelated
+ context (e.g. a
+ common word like ``rows``) can pass spuriously. Accepted: the rule exists to
+ catch *entirely undocumented* fields, and a fuzzier "documented in the right
+ section" rule invites false passes that erode trust in the gate.
+
+Like CE027/CE029, this is intentionally NOT a ``BaseRule`` registered in
+``tests/lint/runner.py`` (that runner is AST-only over ``.py`` files); it reasons
+over Markdown and is wired as ``tests/test_custom_lint.py::TestCE030DocSchemaParity``.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from pydantic import BaseModel
+
+from coder_eval.models import Dataset, RunLimits, SimulationConfig, TaskDefinition
+
+
+# Models the project commits to documenting, paired with the doc page that owns
+# their field reference. Keep this list SHORT and explicit — every entry is a
+# standing documentation obligation.
+DOCUMENTED_MODELS: list[tuple[type[BaseModel], str]] = [
+ (TaskDefinition, "docs/TASK_DEFINITION_GUIDE.md"),
+ (RunLimits, "docs/TASK_DEFINITION_GUIDE.md"),
+ (Dataset, "docs/TASK_DEFINITION_GUIDE.md"),
+ (SimulationConfig, "docs/TASK_DEFINITION_GUIDE.md"),
+]
+
+# Fields deliberately absent from the user docs, with the reason each is not
+# user-authored. An entry here is a promise: this field is set by the framework,
+# not by a task author, so it needs no doc. Every entry must name a real field on
+# its model (see test_exemptions_reference_real_fields).
+EXEMPT: dict[str, dict[str, str]] = {
+ "TaskDefinition": {
+ "suite_id": "set by the dataset expander on expanded row-tasks; not user-authored",
+ "row_id": "set by the dataset expander from Dataset.id_field; not user-authored",
+ },
+}
+
+
+def undocumented_fields(model: type[BaseModel], doc_text: str, exempt: dict[str, str]) -> list[str]:
+ """Field names of ``model`` that appear neither as inline code in ``doc_text`` nor in ``exempt``."""
+ missing: list[str] = []
+ for name in model.model_fields:
+ if name in exempt:
+ continue
+ if f"`{name}`" in doc_text:
+ continue
+ missing.append(name)
+ return missing
+
+
+def find_undocumented_fields(repo_root: Path) -> dict[str, list[str]]:
+ """Map ``"Model (doc_path)"`` to its undocumented field names, for every registered model."""
+ findings: dict[str, list[str]] = {}
+ for model, doc_rel in DOCUMENTED_MODELS:
+ doc_path = repo_root / doc_rel
+ doc_text = doc_path.read_text(encoding="utf-8") if doc_path.is_file() else ""
+ exempt = EXEMPT.get(model.__name__, {})
+ missing = undocumented_fields(model, doc_text, exempt)
+ if missing:
+ findings[f"{model.__name__} ({doc_rel})"] = missing
+ return findings
diff --git a/tests/test_custom_lint.py b/tests/test_custom_lint.py
index 09d7c86e..c4b0623b 100644
--- a/tests/test_custom_lint.py
+++ b/tests/test_custom_lint.py
@@ -707,3 +707,461 @@ def test_src_scan_requires_a_real_consumer_not_any_literal(self, tmp_path: Path)
names = src_env_literals(tmp_path)
assert "CODER_EVAL_REAL" in names # a genuine os.getenv read is backed
assert "CODER_EVAL_BOGUS" not in names # a bare constant is not
+
+
+@pytest.mark.lint
+class TestCE029DocYamlExamples:
+ """CE029 — self-contained YAML examples in the docs must validate.
+
+ A published snippet that raises when copy-pasted reads as a broken feature.
+ The motivating bug: the `prompt_mutations` recipe used `text:` where the
+ field is `content:`, and every mutation model sets `extra="forbid"`. Scans
+ real Markdown, so it lives here rather than in the AST-only runner.
+ """
+
+ REPO_ROOT = Path(__file__).parent.parent
+
+ @staticmethod
+ def _check(text: str) -> str | None:
+ from tests.lint.doc_examples import extract_yaml_blocks, validate_block
+
+ blocks = extract_yaml_blocks(Path("doc.md"), text)
+ assert len(blocks) == 1, f"expected exactly one yaml block, got {len(blocks)}"
+ return validate_block(blocks[0])
+
+ _VALID_TASK = """```yaml
+task_id: "demo"
+description: "A demo task"
+initial_prompt: "Write hello.py"
+success_criteria:
+ - type: "file_exists"
+ path: "hello.py"
+ description: "hello.py exists"
+```
+"""
+
+ def test_repo_doc_examples_validate(self):
+ from tests.lint.doc_examples import default_doc_paths, find_invalid_doc_examples
+
+ findings = find_invalid_doc_examples(default_doc_paths(self.REPO_ROOT))
+ assert not findings, (
+ "\nSelf-contained YAML example(s) in the docs that do not validate against their "
+ "model — a reader who copy-pastes this hits a ValidationError. Fix the example, or "
+ f"mark the block with `{''}` if it is intentionally partial:\n\n"
+ + "\n".join(f" {path}:\n " + "\n ".join(errs) for path, errs in sorted(findings.items()))
+ )
+
+ def test_valid_task_block_passes(self):
+ assert self._check(self._VALID_TASK) is None
+
+ def test_catches_the_prompt_mutations_regression(self):
+ # The exact historical bug this rule exists for: `text:` where the
+ # PromptSuffix field is `content:` (every mutation model forbids extras).
+ finding = self._check(
+ """```yaml
+experiment_id: prompt-phrasing
+description: "Terse vs. detailed"
+variants:
+ - variant_id: terse
+ - variant_id: detailed
+ prompt_mutations:
+ - type: suffix
+ text: "Think step by step."
+```
+"""
+ )
+ assert finding is not None
+ assert "content" in finding
+
+ def test_schematic_placeholder_blocks_are_skipped(self):
+ # The task guide's overview block deliberately writes `agent: { ... }`.
+ assert (
+ self._check(
+ """```yaml
+task_id: "my_task"
+description: "What this task tests"
+initial_prompt: "Instructions..."
+agent: { ... }
+success_criteria: [ ... ]
+```
+"""
+ )
+ is None
+ )
+
+ def test_fragment_blocks_are_not_validated(self):
+ # A bare criteria list is illustrative, not a document. Validating it
+ # would flag most blocks in the docs and get the rule deleted.
+ assert (
+ self._check(
+ """```yaml
+success_criteria:
+ - type: "file_exists"
+ path: "app.py"
+```
+"""
+ )
+ is None
+ )
+
+ def test_lint_skip_marker_is_honored(self):
+ from tests.lint.doc_examples import SKIP_MARKER
+
+ broken = self._VALID_TASK.replace(' - type: "file_exists"', " - type: no_such_criterion")
+ assert self._check(broken) is not None, "control: the block must fail without the marker"
+ assert self._check(f"{SKIP_MARKER}\n\n{broken}") is None
+
+ def test_non_yaml_fences_are_ignored(self):
+ from tests.lint.doc_examples import extract_yaml_blocks
+
+ text = '```json\n{"task_id": 1}\n```\n\n```bash\ncoder-eval run\n```\n'
+ assert extract_yaml_blocks(Path("doc.md"), text) == []
+
+ def test_info_string_attributes_are_still_captured(self):
+ # mkdocs-material allows ```yaml title="x"; such a block must not be
+ # silently skipped, or a broken example there escapes the rule.
+ from tests.lint.doc_examples import extract_yaml_blocks
+
+ text = '```yaml title="task.yaml"\ntask_id: 1\n```\n'
+ blocks = extract_yaml_blocks(Path("doc.md"), text)
+ assert len(blocks) == 1
+
+ def test_valid_experiment_block_passes(self):
+ assert (
+ self._check(
+ """```yaml
+experiment_id: demo-experiment
+description: "A demo experiment"
+variants:
+ - variant_id: baseline
+ - variant_id: treatment
+ agent:
+ model: claude-sonnet-4-6
+```
+"""
+ )
+ is None
+ )
+
+
+@pytest.mark.lint
+class TestCE030DocSchemaParity:
+ """CE030 — a registered model's fields must be documented or exempted with a reason.
+
+ Every P0/P1 defect in the docs overhaul was an undocumented user-facing
+ field. This gate makes that recurrence impossible for a small, explicit
+ registry of models: adding a field forces a doc update or a reasoned
+ EXEMPT entry. Scans real Markdown, so it lives here, not in the AST runner.
+ """
+
+ REPO_ROOT = Path(__file__).parent.parent
+
+ def test_registered_models_are_fully_documented(self):
+ from tests.lint.doc_schema_parity import find_undocumented_fields
+
+ findings = find_undocumented_fields(self.REPO_ROOT)
+ assert not findings, (
+ "\nRegistered model field(s) documented nowhere in their doc page (and not EXEMPT) — "
+ "document them (mention the field name as `inline code`) or add an EXEMPT entry with a "
+ "reason in tests/lint/doc_schema_parity.py:\n\n"
+ + "\n".join(f" {model}: {', '.join(fields)}" for model, fields in sorted(findings.items()))
+ )
+
+ def test_exempt_fields_carry_a_reason(self):
+ from tests.lint.doc_schema_parity import EXEMPT
+
+ for model_name, fields in EXEMPT.items():
+ for field_name, reason in fields.items():
+ assert reason and reason.strip(), f"EXEMPT[{model_name}][{field_name}] has an empty reason"
+
+ def test_exemptions_reference_real_fields(self):
+ # An exemption left behind after a field rename would silently mask a
+ # real undocumented field. Pin every exempt name to a real model field.
+ from tests.lint.doc_schema_parity import DOCUMENTED_MODELS, EXEMPT
+
+ by_name = {m.__name__: m for m, _ in DOCUMENTED_MODELS}
+ for model_name, fields in EXEMPT.items():
+ assert model_name in by_name, f"EXEMPT names unknown model {model_name!r}"
+ real = set(by_name[model_name].model_fields)
+ for field_name in fields:
+ assert field_name in real, f"EXEMPT[{model_name}] names non-field {field_name!r}"
+
+ def test_detects_an_undocumented_field(self):
+ from pydantic import BaseModel, Field
+
+ from tests.lint.doc_schema_parity import undocumented_fields
+
+ class Synthetic(BaseModel):
+ documented: str = Field(default="")
+ undocumented: str = Field(default="")
+
+ doc = "The `documented` field is described here."
+ assert undocumented_fields(Synthetic, doc, {}) == ["undocumented"]
+
+ def test_inline_code_match_only(self):
+ # A field mentioned in prose without backticks does NOT count as documented.
+ from pydantic import BaseModel, Field
+
+ from tests.lint.doc_schema_parity import undocumented_fields
+
+ class Synthetic(BaseModel):
+ widget: str = Field(default="")
+
+ assert undocumented_fields(Synthetic, "The widget setting is great.", {}) == ["widget"]
+ assert undocumented_fields(Synthetic, "The `widget` setting is great.", {}) == []
+
+ def test_exempt_field_is_not_reported(self):
+ from pydantic import BaseModel, Field
+
+ from tests.lint.doc_schema_parity import undocumented_fields
+
+ class Synthetic(BaseModel):
+ internal: str = Field(default="")
+
+ assert undocumented_fields(Synthetic, "", {"internal": "set by the framework"}) == []
+
+ def test_missing_doc_file_fails_loudly(self):
+ # A moved/renamed doc must fail the gate, not vacuously pass. The
+ # integration wrapper maps a missing file to "" → every field reported.
+ from tests.lint.doc_schema_parity import find_undocumented_fields
+
+ findings = find_undocumented_fields(self.REPO_ROOT / "no_such_dir")
+ assert findings, "a missing doc file must surface every field, not return empty"
+ assert any("RunLimits" in key for key in findings)
+
+
+@pytest.mark.lint
+class TestCE028DocIndexParity:
+ """CE028 — the flat index surfaces (README, docs/index.md, docs/llms.txt) are
+ generated from the mkdocs nav + extra.docs_index; disk must match.
+
+ Also enforces the invariants the render depends on: nav<->blurb bijection,
+ every published docs/ page is in the nav, and the hand-written tutorials
+ table stays in parity with the nav. Reasons over Markdown/YAML, so it lives
+ here rather than in the AST runner.
+ """
+
+ REPO_ROOT = Path(__file__).parent.parent
+
+ @property
+ def _nav(self):
+ from tests.lint.doc_indexes import load_nav
+
+ return load_nav(self.REPO_ROOT / "mkdocs.yml")
+
+ @property
+ def _blurbs(self):
+ from tests.lint.doc_indexes import load_blurbs
+
+ return load_blurbs(self.REPO_ROOT / "mkdocs.yml")
+
+ def test_repo_indexes_match_generated_output(self):
+ from tests.lint.doc_indexes import check
+
+ findings = check(self.REPO_ROOT)
+ assert not findings, (
+ "\nGenerated index surface(s) drifted from the mkdocs nav — run `make docs-indexes` "
+ "to regenerate:\n\n" + "\n\n".join(f"{path}:\n{diff}" for path, diff in sorted(findings.items()))
+ )
+
+ def test_every_nav_page_has_a_blurb(self):
+ from tests.lint.doc_indexes import missing_blurbs
+
+ missing = missing_blurbs(self._nav, self._blurbs)
+ assert not missing, f"nav pages without an extra.docs_index blurb (tutorial leaves exempt): {missing}"
+
+ def test_no_orphan_blurbs(self):
+ from tests.lint.doc_indexes import orphan_blurbs
+
+ orphans = orphan_blurbs(self._nav, self._blurbs)
+ assert not orphans, f"extra.docs_index blurbs with no matching nav page: {orphans}"
+
+ def test_every_published_doc_is_in_the_nav(self):
+ from tests.lint.doc_indexes import docs_missing_from_nav
+
+ missing = docs_missing_from_nav(self.REPO_ROOT, self._nav)
+ assert not missing, f"published docs/ pages absent from the nav (add to nav or the exclusion set): {missing}"
+
+ def test_tutorials_table_matches_nav(self):
+ from tests.lint.doc_indexes import tutorials_table_drift
+
+ drift = tutorials_table_drift(self.REPO_ROOT, self._nav)
+ assert drift is None, drift
+
+ def test_real_mkdocs_yaml_parses_despite_env_tag(self):
+ # The real mkdocs.yml carries `!ENV [CI, false]`; load_nav must tolerate it.
+ nav = self._nav
+ assert any(p.doc == "index.md" for p in nav)
+ assert any(p.doc == "DIALOG_MODE.md" for p in nav)
+
+ def test_nav_loader_does_not_mutate_global_safeloader(self):
+ import yaml
+
+ from tests.lint.doc_indexes import load_nav
+
+ load_nav(self.REPO_ROOT / "mkdocs.yml")
+ with pytest.raises(yaml.YAMLError):
+ yaml.safe_load("x: !ENV [A, b]")
+
+ def test_python_object_tags_are_still_rejected(self):
+ import yaml
+
+ from tests.lint.doc_indexes import load_nav
+
+ # Prove the private loader stays safe_load-equivalent for object construction.
+ load_nav(self.REPO_ROOT / "mkdocs.yml")
+
+ class _Probe(yaml.SafeLoader):
+ pass
+
+ _Probe.add_multi_constructor("!", lambda loader, suffix, node: None)
+ with pytest.raises(yaml.constructor.ConstructorError):
+ yaml.load("x: !!python/object/apply:os.system ['echo hi']", Loader=_Probe)
+
+ @pytest.mark.parametrize(
+ ("doc", "route"),
+ [
+ ("index.md", "/docs"),
+ ("USER_GUIDE.md", "/docs/user-guide"),
+ ("agents/CLAUDE_CODE.md", "/docs/agents/claude-code"),
+ ("tutorials/README.md", "/docs/tutorials"),
+ ],
+ )
+ def test_route_for_known_shapes(self, doc: str, route: str):
+ from tests.lint.doc_indexes import route_for
+
+ assert route_for(doc) == route
+
+ def test_write_is_idempotent(self, tmp_path: Path):
+ import shutil
+
+ from tests.lint.doc_indexes import check, write
+
+ # Copy the pieces write() touches into a scratch tree so the live repo is untouched.
+ (tmp_path / "docs").mkdir()
+ shutil.copy(self.REPO_ROOT / "mkdocs.yml", tmp_path / "mkdocs.yml")
+ for rel in ("README.md", "docs/index.md", "docs/llms.txt", "docs/tutorials/README.md"):
+ dest = tmp_path / rel
+ dest.parent.mkdir(parents=True, exist_ok=True)
+ shutil.copy(self.REPO_ROOT / rel, dest)
+
+ write(tmp_path)
+ first = (tmp_path / "README.md").read_text(encoding="utf-8")
+ write(tmp_path)
+ assert (tmp_path / "README.md").read_text(encoding="utf-8") == first
+ assert check(tmp_path) == {}
+
+ def test_missing_marker_raises_clear_error(self):
+ from tests.lint.doc_indexes import _replace_between
+
+ with pytest.raises(ValueError, match="marker pair"):
+ _replace_between("no markers here", "", "", "body")
+
+ def test_drift_is_detected(self, tmp_path: Path):
+ # A hand-edit between the markers must be reported by check().
+ import shutil
+
+ from tests.lint.doc_indexes import check
+
+ (tmp_path / "docs").mkdir()
+ shutil.copy(self.REPO_ROOT / "mkdocs.yml", tmp_path / "mkdocs.yml")
+ for rel in ("README.md", "docs/index.md", "docs/llms.txt", "docs/tutorials/README.md"):
+ dest = tmp_path / rel
+ dest.parent.mkdir(parents=True, exist_ok=True)
+ shutil.copy(self.REPO_ROOT / rel, dest)
+
+ readme = tmp_path / "README.md"
+ text = readme.read_text(encoding="utf-8")
+ readme.write_text(text.replace("| [User Guide]", "| [Tampered Guide]"), encoding="utf-8")
+ findings = check(tmp_path)
+ assert str(readme) in findings
+
+
+@pytest.mark.lint
+class TestCE031DeadConfigFields:
+ """CE031 — a behavior-driving config field must be read somewhere in src/.
+
+ Guards the dead-config class that SimulationConfig.parallel_trials was: a
+ field users set in a task YAML that no code reads by name, so it silently
+ does nothing. Scans the whole src/ tree for attribute reads, so it lives
+ here rather than in the per-file AST runner.
+ """
+
+ REPO_ROOT = Path(__file__).parent.parent
+ SRC = REPO_ROOT / "src"
+
+ def test_no_dead_config_fields(self):
+ from tests.lint.dead_config_fields import find_dead_config_fields
+
+ findings = find_dead_config_fields(self.SRC)
+ assert not findings, (
+ "\nConfig field(s) that no code in src/ reads by name — dead config a user could set "
+ "with no effect. Wire the field to real behavior, remove it, or (if it is consumed only "
+ "via serialization) add an EXEMPT entry with a reason in tests/lint/dead_config_fields.py:\n\n"
+ + "\n".join(f" {model}: {', '.join(fields)}" for model, fields in sorted(findings.items()))
+ )
+
+ def test_detects_a_dead_field(self):
+ from pydantic import BaseModel, Field
+
+ from tests.lint.dead_config_fields import dead_config_fields
+
+ class Synthetic(BaseModel):
+ wired: str = Field(default="")
+ orphan: str = Field(default="")
+
+ # `wired` is read as an attribute somewhere; `orphan` is not.
+ assert dead_config_fields(Synthetic, consumed={"wired"}, exempt={}) == ["orphan"]
+
+ def test_consumed_field_not_flagged(self):
+ from pydantic import BaseModel, Field
+
+ from tests.lint.dead_config_fields import dead_config_fields
+
+ class Synthetic(BaseModel):
+ live: str = Field(default="")
+
+ assert dead_config_fields(Synthetic, consumed={"live"}, exempt={}) == []
+
+ def test_exempt_field_not_flagged(self):
+ from pydantic import BaseModel, Field
+
+ from tests.lint.dead_config_fields import dead_config_fields
+
+ class Synthetic(BaseModel):
+ serialized_only: str = Field(default="")
+
+ assert dead_config_fields(Synthetic, consumed=set(), exempt={"serialized_only": "read via model_dump"}) == []
+
+ def test_would_catch_parallel_trials_shape(self):
+ # A field named like the removed parallel_trials, absent from the consumed
+ # set, must be reported — the exact regression this rule exists for.
+ from pydantic import BaseModel, Field
+
+ from tests.lint.dead_config_fields import dead_config_fields
+
+ class Synthetic(BaseModel):
+ parallel_trials: bool = Field(default=True)
+
+ assert dead_config_fields(Synthetic, consumed={"n_trials", "max_turns"}, exempt={}) == ["parallel_trials"]
+
+ def test_exemptions_reference_real_fields(self):
+ # A stale EXEMPT entry (field renamed/removed) would silently mask a dead field.
+ from tests.lint.dead_config_fields import CONSUMED_MODELS, EXEMPT
+
+ by_name = {m.__name__: m for m in CONSUMED_MODELS}
+ for model_name, fields in EXEMPT.items():
+ assert model_name in by_name, f"EXEMPT names unregistered model {model_name!r}"
+ real = set(by_name[model_name].model_fields)
+ for field_name, reason in fields.items():
+ assert field_name in real, f"EXEMPT[{model_name}] names non-field {field_name!r}"
+ assert reason and reason.strip(), f"EXEMPT[{model_name}][{field_name}] has an empty reason"
+
+ def test_registered_fields_are_actually_consumed_on_the_real_tree(self):
+ # Belt: prove the attribute scan really finds known-live fields, so a
+ # broken scanner (returning everything or nothing) can't pass silently.
+ from tests.lint.dead_config_fields import consumed_attr_names
+
+ consumed = consumed_attr_names(self.SRC)
+ for name in ("n_trials", "max_usd", "stratify_field"):
+ assert name in consumed, f"expected {name!r} to be read as an attribute in src/"
diff --git a/tests/test_simulation_config.py b/tests/test_simulation_config.py
index 953b21d4..7d8c9d16 100644
--- a/tests/test_simulation_config.py
+++ b/tests/test_simulation_config.py
@@ -34,7 +34,6 @@ def test_sensible_defaults(self):
assert cfg.max_turns == 8
assert cfg.stop_token == DEFAULT_SIMULATION_STOP_TOKEN
assert cfg.n_trials == 1
- assert cfg.parallel_trials is True
assert cfg.check_criteria == "end_of_dialog"
# Default stop_on_criteria_pass=False so that the default
# check_criteria=end_of_dialog pairing is internally consistent.
@@ -49,7 +48,7 @@ telemetry. See [How it compares](https://uipath.github.io/coder_eval/comparison/
> **Keeping skills fresh?** Run Coder Eval as a scheduled GitHub Actions job so your
> skills are continuously re-evaluated against the latest model — a skill that quietly
> stops triggering surfaces as a failing criterion before your users hit it. See
-> **[Tutorial 02 — Running coder_eval in CI](docs/tutorials/02-ci-pipeline.md)**.
+> **[Tutorial 02 — Running Coder Eval in CI](docs/tutorials/02-ci-pipeline.md)**.
## Quick Start
@@ -93,7 +92,7 @@ To add it as a project dependency instead: `uv add coder-eval` or
`pip install coder-eval`. In a real CI gate, pin to a specific released version
so a harness upgrade can't silently move your results. (The example `tasks/`
live in this repo — clone it or point the CLI at your own task files.) See
-[Tutorial 02 — Running coder_eval in CI](docs/tutorials/02-ci-pipeline.md) for
+[Tutorial 02 — Running Coder Eval in CI](docs/tutorials/02-ci-pipeline.md) for
the full setup.
## Use as a GitHub Action
@@ -176,15 +175,27 @@ alone.
## Documentation
+
| Guide | What's in it |
| --- | --- |
| [Tutorials](docs/tutorials/README.md) | Step-by-step walkthroughs — start here |
| [User Guide](docs/USER_GUIDE.md) | Full CLI, configuration, output, and environment-variable reference |
| [Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md) | The task-file schema — all criterion types, scoring, templates |
+| [Claude Code](docs/agents/CLAUDE_CODE.md) | Configuring and running the default Claude Code agent |
+| [Codex](docs/agents/CODEX.md) | Running the OpenAI Codex agent |
+| [Antigravity (Gemini)](docs/agents/ANTIGRAVITY.md) | Running the Google Antigravity / Gemini agent |
| [A/B Experiments](docs/AB_EXPERIMENTS.md) | Compare models / tools / prompts across the same tasks |
-| [Bring Your Own Dataset](docs/BYOD.md) | Fan a single task out over a dataset |
-| [Codex Agent Guide](docs/CODEX_AGENT_GUIDE.md) | Running the Codex agent |
-| [Docker Isolation](docs/DOCKER_ISOLATION.md) | The container sandbox driver |
+| [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset |
+| [Dialog Mode](docs/DIALOG_MODE.md) | Evaluate agents in multi-turn conversation via a simulated user |
+| [Docker Isolation](docs/DOCKER_ISOLATION.md) | The container sandbox driver, with custom images |
+| [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate — the packaged Action, JUnit output, score floor |
+| [Extending Coder Eval](docs/EXTENDING.md) | Author a custom agent, criterion, or model pricing via the plugin SPI |
+| [Report Schema](docs/REPORT_SCHEMA.md) | Field-level reference for run.json / variant.json / task.json |
+| [How It Compares](docs/comparison.md) | vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts |
+
+
+| Repo doc | What's in it |
+| --- | --- |
| [CLAUDE.md](CLAUDE.md) | Architecture, key patterns, and extension points |
| [CONTRIBUTING.md](CONTRIBUTING.md) | Dev setup, quality bar, and how to contribute |
@@ -192,7 +203,7 @@ alone.
- **vs. fixed benchmarks (SWE-bench, SkillsBench)** — they score a canonical dataset;
Coder Eval scores *your* tasks with continuous 0.0–1.0 weighted criteria (and can
- still wrap a fixed dataset via [Bring Your Own Dataset](docs/BYOD.md)).
+ still wrap a fixed dataset via [Bring Your Own Dataset](docs/DATASETS.md)).
- **vs. large-scale / RL harnesses (Harbor)** — Harbor targets scale and RL rollouts;
Coder Eval targets weighted, skill-aware suites gated in CI.
- **vs. model-output eval tools (OpenAI Evals)** — they grade model text; Coder Eval
@@ -200,7 +211,7 @@ alone.
- **vs. hand-rolled scripts** — reproducible sandboxes, weighted criteria,
cost/token telemetry, A/B experiments, and CI-ready pass/fail gates out of the box.
-See the full [comparison — with sources](https://uipath.github.io/coder_eval/comparison/).
+See the full [comparison — with sources](https://coder-eval.com/docs/comparison).
## Task Definition
diff --git a/docs/AB_EXPERIMENTS.md b/docs/AB_EXPERIMENTS.md
index 0f437bc3..629b93ae 100644
--- a/docs/AB_EXPERIMENTS.md
+++ b/docs/AB_EXPERIMENTS.md
@@ -1,6 +1,6 @@
---
description: >-
- A/B-test AI coding agents with coder_eval's experiment layer — Claude Code vs.
+ A/B-test AI coding agents with Coder Eval's experiment layer — Claude Code vs.
Codex vs. Gemini, model vs. model, skill on vs. off, prompt vs. prompt — on
identical tasks.
---
@@ -122,7 +122,7 @@ From `ExperimentVariant` (`coder_eval/models/experiment.py`):
| `variant_id` | str | Unique arm identifier (required) |
| `description` | str | Human-readable label shown in reports |
| `agent` | dict | Partial `AgentConfig` overrides (model, plugins, tools, system_prompt, sdk_options, …) |
-| `simulation` | dict | Partial `SimulationConfig` overrides (persona/model/temperature per arm) |
+| `simulation` | dict | Partial `SimulationConfig` overrides (persona, goal, `n_trials`, … per arm) — see [Dialog Mode](DIALOG_MODE.md) |
| `repeats` | int | Replicate count for this arm (overrides experiment default) |
| `template_sources` | list | Extra templates appended after the task base (e.g. a docs overlay) |
| `prompt_mutations` | list | Ordered mutations applied to `initial_prompt` |
@@ -212,8 +212,9 @@ variants:
## Recipe: A/B a Prompt
-Use `prompt_mutations` (transform the task prompt) or `initial_prompt` (replace
-it wholesale). They are mutually exclusive per variant.
+Use `prompt_mutations` (transform the task prompt) or `initial_prompt` /
+`initial_prompt_file` (replace it wholesale). All three are mutually exclusive
+per variant — setting two raises at load.
```yaml
experiment_id: prompt-phrasing
@@ -224,11 +225,45 @@ variants:
- variant_id: detailed
prompt_mutations:
- type: suffix
- text: "\n\nThink step by step and validate your work before finishing."
+ content: "Think step by step and validate your work before finishing."
```
-The full mutation catalog (prefix / suffix / replace / template) is defined in
-`coder_eval/models/mutations.py`.
+### The mutation catalog
+
+Four mutation types, applied to the base `initial_prompt` at **variant
+resolution time** — before the task ever reaches the orchestrator:
+
+| `type` | Fields | Defaults |
+| ---------- | ------------------------------------------------- | --------------------- |
+| `prefix` | `content` (required), `separator` | `separator: "\n\n"` |
+| `suffix` | `content` (required), `separator` | `separator: "\n\n"` |
+| `replace` | `pattern`, `replacement` (both required), `regex` | `regex: false` |
+| `template` | `variables` (mapping, required) | — |
+
+- **Ordered.** `prompt_mutations` is a list and each mutation operates on the
+ result of the one before it. A `replace` listed after a `suffix` will rewrite
+ the appended text too, which is occasionally what you want and more often a
+ surprise — keep the list short and ordered deliberately.
+- **`prefix` / `suffix`** join with `separator`, default `"\n\n"` (write it
+ quoted in YAML so the escape is interpreted).
+- **`replace`** is a literal `str.replace` by default. With `regex: true` it
+ becomes `re.sub`, so `pattern` is a Python regular expression and
+ `replacement` follows `re.sub` semantics — `\1` backreferences work, and an
+ unanchored pattern can match far more than you intended.
+- **`template`** substitutes literal `{name}` occurrences with the mapped value
+ via plain string replacement — **not** `str.format`, and unrelated to the
+ `${row.