docs: agent guides, extending & report-schema references, and coder-eval.com/docs migration

.github/workflows/docs.yml

@@ -10,12 +10,17 @@ name: Docs
 # Source: "Deploy from a branch" → branch `gh-pages` / `/ (root)`. That's the same
 # (legacy, branch-based) configuration uipath-python already runs on.
 on:
-  push:
-    branches: [main]
-    paths:
-      - "docs/**"
-      - "mkdocs.yml"
-      - ".github/workflows/docs.yml"
+  # Auto-publish on push is DISABLED: GitHub Pages is not yet enabled for this
+  # repo on uipath.github.io (org owner must switch it on once — Settings → Pages
+  # → Deploy from a branch → gh-pages / root). The published docs currently serve
+  # from coder-eval.com/docs (synced separately by the website), so the gh-pages
+  # deploy is not needed yet. Re-enable by uncommenting the `push:` trigger below.
+  # push:
+  #   branches: [main]
+  #   paths:
+  #     - "docs/**"
+  #     - "mkdocs.yml"
+  #     - ".github/workflows/docs.yml"
   workflow_dispatch:
 
 permissions:

CLAUDE.md

@@ -195,11 +195,17 @@ make format      # ruff format
 make check       # ruff check (lint)
 make typecheck   # pyright
 make test        # pytest
-make lint        # custom architectural lint rules (CE001–CE025)
+make lint        # custom architectural lint rules (CE001+)
 make verify      # All of the above + coverage check (CI equivalent)
 ```
 
-When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001–CE025 pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested.
+When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE027–CE031 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name.)
+
+Adding a user-facing field to one of the models CE030 tracks (`TaskDefinition`, `RunLimits`, `Dataset`, `SimulationConfig` — see `tests/lint/doc_schema_parity.py`) means documenting it in its guide (mention the field name as inline code) or adding an `EXEMPT` entry with a reason it is not user-authored. `make lint` fails otherwise.
+
+**Docs index SSOT.** `nav:` plus `extra.docs_index` (blurbs) in `mkdocs.yml` are the single source of truth for the flat index surfaces — `README.md`'s Documentation table, `docs/index.md`'s "Where to go next" table, and the `## Docs` / `## Tutorials` sections of `docs/llms.txt`. Regenerate all three with `make docs-indexes`; **CE028** fails the build if any drifts, if a nav page lacks a blurb (or vice-versa), or if a `docs/*.md` page is missing from the nav. The website sidebar derives from the same `nav:`. When adding or renaming a docs page, edit `nav:` + `extra.docs_index` and run `make docs-indexes` — never hand-edit the generated tables (they sit between `<!-- docs-index:start -->` / `<!-- docs-index:end -->` markers).
+
+**Anchor slugger convention.** The docs are rendered by three sluggers (GitHub, Starlight/github-slugger on coder-eval.com, and python-markdown/mkdocs), which disagree on headings containing `&` or punctuation (`api-routing--benchmarking` vs `api-routing-benchmarking`). Prefer punctuation-free headings so all three agree; if a heading needs `&`, add a GitHub-form `<a id="…"></a>` shim above it and link that form. Verify a new intra-doc anchor link resolves in the built HTML (`mkdocs build`), not by eye.
 
 ## Configuration
 

Makefile

@@ -1,4 +1,4 @@
-.PHONY: help install format check typecheck test test-live test-smoke verify verify-noextra clean run lint docker-image docker-image-full coder-eval-runtime docker-images
+.PHONY: help install format check typecheck test test-live test-smoke verify verify-noextra clean run lint docs-indexes docker-image docker-image-full coder-eval-runtime docker-images
 
 # Single source of the installed coder-eval version (used to tag the docker
 # images). Referenced lazily inside the docker recipes, so it doesn't run on
@@ -24,6 +24,9 @@ check:  ## Run linting checks
 lint:  ## Run custom architectural lint rules (CE001+)
 	uv run pytest tests/test_custom_lint.py -v --tb=short --no-header -p no:warnings
 
+docs-indexes:  ## Regenerate README/docs indexes from the mkdocs nav (SSOT)
+	uv run python -m tests.lint.doc_indexes
+
 typecheck:  ## Run type checking with pyright
 	uv run pyright
 

README.md

@@ -2,11 +2,10 @@
 
 [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
 [![Website](https://img.shields.io/badge/website-coder--eval.com-1f6feb.svg)](https://coder-eval.com)
-[![Docs](https://img.shields.io/badge/docs-uipath.github.io%2Fcoder__eval-1f6feb.svg)](https://uipath.github.io/coder_eval/)
+[![Docs](https://img.shields.io/badge/docs-coder--eval.com%2Fdocs-1f6feb.svg)](https://coder-eval.com/docs)
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
 [![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/)
 [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)
-[![Code style: Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 
 **Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an open-source framework for
 **evaluating and benchmarking AI coding agents and their skills** — built for CLI
@@ -22,8 +21,8 @@ prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-benc
 SkillsBench) that rank models on a shared leaderboard, Coder Eval evaluates the
 tasks, skills, and workflows *you* ship — with weighted 0.0–1.0 criteria, a
 `skill_triggered` activation check, an A/B experiment layer, and per-tool cost
-telemetry. See [How it compares](https://uipath.github.io/coder_eval/comparison/).
-📚 **Full docs:** **[uipath.github.io/coder_eval](https://uipath.github.io/coder_eval/)**.
+telemetry. See [How it compares](https://coder-eval.com/docs/comparison).
+📚 **Full docs:** **[coder-eval.com/docs](https://coder-eval.com/docs)**.
 
 <p align="center">
   <img src="docs/assets/hero.gif" alt="Coder Eval running the hello_date task: a sandboxed agent writes and runs a script from a YAML task, then the scored result is browsed in evalboard" width="100%">
@@ -49,7 +48,7 @@ telemetry. See [How it compares](https://uipath.github.io/coder_eval/comparison/
 > **Keeping skills fresh?** Run Coder Eval as a scheduled GitHub Actions job so your
 > skills are continuously re-evaluated against the latest model — a skill that quietly
 > stops triggering surfaces as a failing criterion before your users hit it. See
-> **[Tutorial 02 — Running coder_eval in CI](docs/tutorials/02-ci-pipeline.md)**.
+> **[Tutorial 02 — Running Coder Eval in CI](docs/tutorials/02-ci-pipeline.md)**.
 
 ## Quick Start
 
@@ -93,7 +92,7 @@ To add it as a project dependency instead: `uv add coder-eval` or
 `pip install coder-eval`. In a real CI gate, pin to a specific released version
 so a harness upgrade can't silently move your results. (The example `tasks/`
 live in this repo — clone it or point the CLI at your own task files.) See
-[Tutorial 02 — Running coder_eval in CI](docs/tutorials/02-ci-pipeline.md) for
+[Tutorial 02 — Running Coder Eval in CI](docs/tutorials/02-ci-pipeline.md) for
 the full setup.
 
 ## Use as a GitHub Action
@@ -176,31 +175,43 @@ alone.
 
 ## Documentation
 
+<!-- docs-index:start -->
 | Guide | What's in it |
 | --- | --- |
 | [Tutorials](docs/tutorials/README.md) | Step-by-step walkthroughs — start here |
 | [User Guide](docs/USER_GUIDE.md) | Full CLI, configuration, output, and environment-variable reference |
 | [Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md) | The task-file schema — all criterion types, scoring, templates |
+| [Claude Code](docs/agents/CLAUDE_CODE.md) | Configuring and running the default Claude Code agent |
+| [Codex](docs/agents/CODEX.md) | Running the OpenAI Codex agent |
+| [Antigravity (Gemini)](docs/agents/ANTIGRAVITY.md) | Running the Google Antigravity / Gemini agent |
 | [A/B Experiments](docs/AB_EXPERIMENTS.md) | Compare models / tools / prompts across the same tasks |
-| [Bring Your Own Dataset](docs/BYOD.md) | Fan a single task out over a dataset |
-| [Codex Agent Guide](docs/CODEX_AGENT_GUIDE.md) | Running the Codex agent |
-| [Docker Isolation](docs/DOCKER_ISOLATION.md) | The container sandbox driver |
+| [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset |
+| [Dialog Mode](docs/DIALOG_MODE.md) | Evaluate agents in multi-turn conversation via a simulated user |
+| [Docker Isolation](docs/DOCKER_ISOLATION.md) | The container sandbox driver, with custom images |
+| [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate — the packaged Action, JUnit output, score floor |
+| [Extending Coder Eval](docs/EXTENDING.md) | Author a custom agent, criterion, or model pricing via the plugin SPI |
+| [Report Schema](docs/REPORT_SCHEMA.md) | Field-level reference for run.json / variant.json / task.json |
+| [How It Compares](docs/comparison.md) | vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts |
+<!-- docs-index:end -->
+
+| Repo doc | What's in it |
+| --- | --- |
 | [CLAUDE.md](CLAUDE.md) | Architecture, key patterns, and extension points |
 | [CONTRIBUTING.md](CONTRIBUTING.md) | Dev setup, quality bar, and how to contribute |
 
 ## How it compares
 
 - **vs. fixed benchmarks (SWE-bench, SkillsBench)** — they score a canonical dataset;
   Coder Eval scores *your* tasks with continuous 0.0–1.0 weighted criteria (and can
-  still wrap a fixed dataset via [Bring Your Own Dataset](docs/BYOD.md)).
+  still wrap a fixed dataset via [Bring Your Own Dataset](docs/DATASETS.md)).
 - **vs. large-scale / RL harnesses (Harbor)** — Harbor targets scale and RL rollouts;
   Coder Eval targets weighted, skill-aware suites gated in CI.
 - **vs. model-output eval tools (OpenAI Evals)** — they grade model text; Coder Eval
   runs a full agent in a sandbox and scores the files and commands it produced.
 - **vs. hand-rolled scripts** — reproducible sandboxes, weighted criteria,
   cost/token telemetry, A/B experiments, and CI-ready pass/fail gates out of the box.
 
-See the full [comparison — with sources](https://uipath.github.io/coder_eval/comparison/).
+See the full [comparison — with sources](https://coder-eval.com/docs/comparison).
 
 ## Task Definition
 

docs/AB_EXPERIMENTS.md

@@ -1,6 +1,6 @@
 ---
 description: >-
-  A/B-test AI coding agents with coder_eval's experiment layer — Claude Code vs.
+  A/B-test AI coding agents with Coder Eval's experiment layer — Claude Code vs.
   Codex vs. Gemini, model vs. model, skill on vs. off, prompt vs. prompt — on
   identical tasks.
 ---
@@ -122,7 +122,7 @@ From `ExperimentVariant` (`coder_eval/models/experiment.py`):
 | `variant_id`          | str                | Unique arm identifier (required)                                                       |
 | `description`         | str                | Human-readable label shown in reports                                                  |
 | `agent`               | dict               | Partial `AgentConfig` overrides (model, plugins, tools, system_prompt, sdk_options, …) |
-| `simulation`          | dict               | Partial `SimulationConfig` overrides (persona/model/temperature per arm)               |
+| `simulation`          | dict               | Partial `SimulationConfig` overrides (persona, goal, `n_trials`, … per arm) — see [Dialog Mode](DIALOG_MODE.md) |
 | `repeats`             | int                | Replicate count for this arm (overrides experiment default)                            |
 | `template_sources`    | list               | Extra templates appended after the task base (e.g. a docs overlay)                     |
 | `prompt_mutations`    | list               | Ordered mutations applied to `initial_prompt`                                          |
@@ -212,8 +212,9 @@ variants:
 
 ## Recipe: A/B a Prompt
 
-Use `prompt_mutations` (transform the task prompt) or `initial_prompt` (replace
-it wholesale). They are mutually exclusive per variant.
+Use `prompt_mutations` (transform the task prompt) or `initial_prompt` /
+`initial_prompt_file` (replace it wholesale). All three are mutually exclusive
+per variant — setting two raises at load.
 
 ```yaml
 experiment_id: prompt-phrasing
@@ -224,11 +225,45 @@ variants:
   - variant_id: detailed
     prompt_mutations:
       - type: suffix
-        text: "\n\nThink step by step and validate your work before finishing."
+        content: "Think step by step and validate your work before finishing."
 ```
 
-The full mutation catalog (prefix / suffix / replace / template) is defined in
-`coder_eval/models/mutations.py`.
+### The mutation catalog
+
+Four mutation types, applied to the base `initial_prompt` at **variant
+resolution time** — before the task ever reaches the orchestrator:
+
+| `type`     | Fields                                            | Defaults              |
+| ---------- | ------------------------------------------------- | --------------------- |
+| `prefix`   | `content` (required), `separator`                 | `separator: "\n\n"`   |
+| `suffix`   | `content` (required), `separator`                 | `separator: "\n\n"`   |
+| `replace`  | `pattern`, `replacement` (both required), `regex`  | `regex: false`        |
+| `template` | `variables` (mapping, required)                   | —                     |
+
+- **Ordered.** `prompt_mutations` is a list and each mutation operates on the
+  result of the one before it. A `replace` listed after a `suffix` will rewrite
+  the appended text too, which is occasionally what you want and more often a
+  surprise — keep the list short and ordered deliberately.
+- **`prefix` / `suffix`** join with `separator`, default `"\n\n"` (write it
+  quoted in YAML so the escape is interpreted).
+- **`replace`** is a literal `str.replace` by default. With `regex: true` it
+  becomes `re.sub`, so `pattern` is a Python regular expression and
+  `replacement` follows `re.sub` semantics — `\1` backreferences work, and an
+  unanchored pattern can match far more than you intended.
+- **`template`** substitutes literal `{name}` occurrences with the mapped value
+  via plain string replacement — **not** `str.format`, and unrelated to the
+  `${row.<field>}` syntax used by [datasets](DATASETS.md). A `{name}` with no
+  matching entry in `variables` is left in the prompt verbatim rather than
+  raising, so a typo in a variable name fails silently at the *prompt* level.
+  Read the rendered prompt in the report to confirm.
+
+Every mutation model sets `extra="forbid"`, so a misspelled field (`text:`
+instead of `content:`) is a hard `ValidationError` at load — never a silently
+ignored no-op.
+
+When both `defaults.prompt_mutations` and a variant's `prompt_mutations` are set,
+they **compose** — the experiment-defaults mutations apply first, then the
+variant's, on the already-mutated prompt.
 
 ## Recipe: Smoke vs. e2e Flavors (Early Stop)