| description | Complete Coder Eval reference — every CLI command and flag, configuration layers and -D overrides, environment variables, run outputs, and reports for evaluating AI coding agents. |
|---|
The full command, configuration, and output reference. For a gentle introduction start with the tutorials; for the task-file schema see the Task Definition Guide.
- CLI Commands
- API Routing & Benchmarking
- Output Structure
- Suite Thresholds & Classification Metrics
- Environment Variables
- Troubleshooting
coder-eval run # all tasks (discovers tasks/ recursively)
coder-eval run tasks/hello_date.yaml # a single task
coder-eval run tasks/*.yaml --max-parallel 3 # multiple, 3 concurrent
coder-eval run tasks/hello_date.yaml --stream full # live LLM output| Flag | Description |
|---|---|
--max-parallel, -j |
Concurrent tasks (default: 1) |
--preservation-mode |
Sandbox persistence: NONE / MOVE_ON_WRITE / DIRECT_WRITE. Default is driver-derived (docker → DIRECT_WRITE, else MOVE_ON_WRITE); explicit value always wins. |
--run-dir |
Custom run directory (default: timestamped in runs/) |
-D path=value / --set |
Override any resolved task-config field (agent/run_limits/sandbox roots), e.g. -D run_limits.max_turns=30 -D agent.permission_mode=plan -D agent.sdk_options.effort=high. Repeatable; schema-validated. This is the way to set permission mode, turn/timeout limits, token/USD budget caps, tools, plugins, and SDK options. |
--model, -m |
Shorthand alias for -D agent.model=… (e.g., claude-sonnet-4-20250514) |
--driver |
Shorthand alias for -D sandbox.driver=… (tempdir or docker) |
--type, -T |
Override agent type for all tasks (claude-code, codex, antigravity, or a plugin kind). |
--repeats |
Run each (task, variant) N times (≥1); overrides experiment/variant repeats:. See Replicates. |
--resume |
Resume an interrupted run: skip tasks already finalized in --run-dir and run the rest, folding prior results into run.json. Requires --run-dir. A task with any final status (incl. FAILED/ERROR) counts as finalized, so resume does not retry failures — delete a task's task.json to force a re-run. A config mismatch is warned, not refused. |
--sample N |
For dataset-backed tasks, run a fixed-seed random N-row sample (reproducible; cheap smoke test). See Bring Your Own Dataset. |
--sample-per-stratum N |
For dataset-backed tasks, keep up to N rows per stratum (stratify_field). Overridden by --sample. Nondeterministic unless dataset.sample_seed is set — see Bring Your Own Dataset. |
--include-skipped |
Also run tasks marked skip: true in their YAML (off by default so CI keeps excluding them). |
--exclude-tags |
Skip tasks matching any of these tags (comma-separated) |
--tags, -t |
Only run tasks matching any of these tags (comma-separated) |
--experiment, -e |
Experiment definition YAML for multi-variant comparison (default: experiments/default.yaml) |
--log-file |
Write logs to file |
--backend, -b |
API backend: direct or bedrock (default: from API_BACKEND env var) |
--stream, -s |
Stream LLM events to terminal: full or minimal (disables progress bar) |
--verbose, -v |
DEBUG-level logging |
Run-time caps — turns, wall-clock timeouts, and cumulative token / USD budgets — are not CLI
flags of their own. They live under run_limits: in the task YAML, or on the command line as
-D run_limits.<field>=<value> (e.g. -D run_limits.max_usd=2.50). The complete field reference is
in the Task Definition Guide.
coder-eval plan # validate all tasks
coder-eval plan tasks/*.yaml # validate specific tasksChecks task syntax, required CLI tools, API keys, and schema validity without executing.
| Flag | Description |
|---|---|
--experiment, -e |
Experiment definition YAML to resolve variants against (default: experiments/default.yaml). |
coder-eval evaluate tasks/hello_date.yaml ./my_solution # evaluate a directory
coder-eval evaluate tasks/hello_date.yaml ./my_solution --preserve # keep the sandboxRuns a task's success criteria against a directory without an agent — useful for testing criterion definitions, validating task configs, or scoring code that was already written.
| Flag | Description |
|---|---|
--preserve / --no-preserve |
Preserve sandbox after evaluation (default: preserve) |
--run-dir |
Custom run directory (default: auto-generated timestamped dir in runs/). |
--verbose, -v |
DEBUG-level logging |
coder-eval report runs/latest # view latest run (markdown to stdout)
coder-eval report runs/latest -o summary.md # export markdown to a file
coder-eval report runs/latest --format html # (re)render every task.json as task.htmlThe run command already writes reports during execution; report re-displays or
re-exports them later. For the on-disk layout and field-level schema of the JSON it
reads, see Output Structure and the
Report Schema.
| Flag | Description |
|---|---|
--output, -o |
Write to a file instead of stdout (markdown). |
--format, -f |
md (default) or html. html re-renders each task.json under the run dir to a task.html beside it (or to -o when exactly one task is found). |
coder-eval aggregate runs/2026-06-22_14-32-27 # rebuild the summary in place
coder-eval aggregate runs/combined -o runs/combined # aggregate a merged dirRe-derives the run-level run.json + run.md from the finalized task.json files
already on disk, using the same builder a live run uses. Use it when a run dir's
top-level summary is missing or stale — e.g. after recovering an interrupted run or
combining several run directories. It rebuilds the run-level summary only;
per-suite rollups (suite.json/suite.md) and experiment reports
(experiment.json/experiment.md) are not rebuilt, because the per-row
suite/variant grouping they need is not recoverable from task.json alone.
| Flag | Description |
|---|---|
--output, -o |
Write run.json/run.md into this directory instead of the run dir (e.g. a merged output dir). |
The project ships Claude Code custom slash commands
in .claude/commands/, available when using Claude Code in this repository:
| Command | Description |
|---|---|
/coder-eval-run-analysis <path> |
Analyze evaluation runs and suggest improvements to tasks, config, and prompts. Works at task, variant, or run scope. |
/coder-eval-task-create |
Create evaluation task YAML files from a natural language description. |
coder-eval supports two API routing modes, selected via --backend or the
API_BACKEND env var:
- Direct API (
--backend direct, default) — calls the Anthropic API directly using yourANTHROPIC_API_KEY. Accurate token/cost reporting from the SDK. - AWS Bedrock (
--backend bedrock) — routes through AWS Bedrock with bearer token auth. Useful for cross-region model access and org-managed AWS deployments.
coder-eval run tasks/hello_date.yaml # direct (default)
coder-eval run tasks/hello_date.yaml --backend bedrock # via Bedrock (set BEDROCK_* in .env)For official benchmarking, use the direct API (
--backend direct) for accurate token/cost reporting.
runs/
├── 2026-02-26_14-30-00/ # Timestamped run directory
│ ├── run.json # Run-level summary (tasks, durations, tokens)
│ ├── run.md # Run-level markdown report
│ ├── experiment.md / .json / .log # Cross-variant comparison + aggregated log
│ ├── <variant_id>/ # Per-variant directory
│ │ ├── variant.md / variant.json # Variant aggregate report + data
│ │ └── <task_id>/
│ │ └── 00/ # Replicate index — one dir per replicate
│ │ ├── task.json # Evaluation result
│ │ ├── task.log # Execution log
│ │ └── artifacts/ # Preserved sandbox (unless --preservation-mode NONE)
│ └── ...
└── latest -> 2026-02-26_14-30-00/ # Symlink to most recent run
Run the same (task, variant) N times via repeats: in an experiment YAML or
--repeats N on the CLI. Per-replicate results live in separate NN/ directories;
reports aggregate them with bootstrap confidence intervals and (for 2-variant
experiments) a paired mean-difference test. Defaults to 1 (no repetition).
For a field-level reference to run.json, variant.json, task.json, and the
suite/experiment rollups, see the Report Schema.
Any criterion on a dataset-backed task (see Bring Your Own Dataset)
can gate the whole suite, not just individual rows. Each criterion's aggregate()
emits count / mean / median / std / min / max, and classification-style criteria
(classification_match, skill_triggered) additionally emit accuracy, per-label
precision/recall/F1, and a confusion matrix. Add suite_thresholds to require a
minimum for any of those metrics — the run exits non-zero if any gate fails:
success_criteria:
- type: skill_triggered
skill_name: uipath-agents
expected_skill: "${row.expected_skill}" # "" for rows where it shouldn't fire
suite_thresholds:
recall.yes: 0.70 # fired on ≥70% of rows that needed it
precision.yes: 0.80 # ≤20% false activationsAvailable metric keys: accuracy, macro_f1, weighted_f1, micro_f1, and
per-label precision.<label> / recall.<label> / f1.<label>. This makes
Coder Eval usable as a SkillsBench-style activation harness. Full details and the
suite-rollup outputs live in A/B Experiments
and the Task Definition Guide.
Set these in .env (copy from .env.example).
| Variable | Required | Description |
|---|---|---|
ANTHROPIC_API_KEY |
Yes (for Claude Code) | Anthropic API key |
API_BACKEND |
No | API backend: direct or bedrock (default: direct). Overridden by --backend. |
AWS_BEARER_TOKEN_BEDROCK |
For Bedrock | AWS Bedrock bearer token for authentication |
AWS_REGION |
For Bedrock | AWS region for Bedrock endpoint (e.g., eu-north-1) |
BEDROCK_MODEL |
No | Cross-region Bedrock model ID (e.g., eu.anthropic.claude-sonnet-4-5-20250929-v1:0) |
BEDROCK_SMALL_MODEL |
No | Cross-region Bedrock small/fast model ID |
CODEX_API_KEY / CODEX_BASE_URL / CODEX_MODEL / CODEX_API_VERSION |
For Codex | Codex agent auth & endpoint routing — see Codex Agent Guide. |
GEMINI_API_KEY / ANTIGRAVITY_MODEL |
For Antigravity | Antigravity (Gemini) agent auth & model — see Antigravity Agent Guide. |
UIPATH_PLUGIN_MARKETPLACE_DIR |
No | Conventional base directory for local plugins. Not special-cased by the framework: any $VAR / ${VAR} referenced in a plugin path is expanded from the environment, so a plugin path like $UIPATH_PLUGIN_MARKETPLACE_DIR/my-plugin resolves against this variable. (An undefined variable in a plugin path logs a warning.) |
PLUGIN_TOOLS_DIR |
No | A separate mechanism from the above: the canonical node_modules/@uipath directory used to pin UiPath CLI plugin discovery (not path substitution). When unset, the sandbox auto-derives it from the resolved uip binary. |
CODER_EVAL_REMEDIATE_HOME_PLUGINS |
No | DESTRUCTIVE. Truthy deletes $HOME/node_modules/@uipath at sandbox setup to clear sibling-task pollution on dedicated eval hosts. Off by default; do not enable on developer workstations. |
LOG_LEVEL |
No | Logging level (default: INFO) |
LOG_TO_FILE |
No | Enable file logging (default: false) |
TELEMETRY_ENABLED |
No | Anonymous usage telemetry, on by default. Set false to disable entirely — see Usage Telemetry. |
TELEMETRY_CONNECTION_STRING |
No | Route telemetry to your own App Insights resource instead of the shared default (aliases: APPLICATIONINSIGHTS_CONNECTION_STRING, UIPATH_AI_CONNECTION_STRING) |
TELEMETRY_SOURCE |
No | Origin stamp emitted as the Source dimension (default: coder-eval) |
coder-eval collects anonymous usage telemetry (command names, outcomes, counts,
durations, an anonymous per-install id, and platform info) to help improve the tool.
It never captures prompts, file contents, or repo paths. Telemetry is on by
default and the first run prints a one-time notice to stderr disclosing this.
- Disable it entirely: set
TELEMETRY_ENABLED=false(in.envor the environment). - Send it to your own resource: set
TELEMETRY_CONNECTION_STRINGto your Azure Application Insights connection string.
| Problem | Solution |
|---|---|
ANTHROPIC_API_KEY is required |
Create .env from .env.example and add your key |
claude command not found |
brew install claude |
uv command not found |
brew install uv or pip install uv |
| Tests failing | source .venv/bin/activate && uv pip install -e ".[dev]" |
| Pre-commit hooks failing | pre-commit autoupdate && pre-commit run --all-files |