feat: packaged CI gate — JUnit XML output + composite GitHub Action

[uipreliga]

Makes coder-eval consumable as a CI gate: a JUnit XML report generated from any run directory, plus a composite GitHub Action published from the repo root.

What's in it

1. src/coder_eval/reports_junit.py (new) — generate_junit_xml(run_dir) reads a finalized run directory and returns JUnit XML; write_junit_xml is the thin persist wrapper (mirrors the build_run_summary/write_run_summary seam).

• Reads the run.json spine via the RunSummary model (SSOT, not hand-parsed), optional */*/suite.json gates via SuiteRollup, and per-failed-row task.json as a plain dict (best-effort, so schema skew in blob-pulled/older runs degrades to a status-only body instead of aborting).
• One <testsuite> per variant, one <testcase> per task row ([NN] suffix for replicates); skipped and suite-gates synthetic suites. Count attributes are derived from the emitted children, so they can't drift.
• Status classification goes through FinalStatus(value).category — an explicit allowlist with an explicit unknown-value fallback, never a denylist (CE018).
• No new runtime dependencies. Production code builds and serializes an element tree and never parses XML, so XXE / billion-laughs have no surface; defusedxml is added dev-only for test-side round-trips.

2. CLI wiring — output-only flags; neither enters the 5-layer config merge.

• coder-eval run --junit-xml PATH — written after the run summary is persisted and before the failure exit-code gate, so a red run still emits a report. A write error propagates (a silently missing file would make a downstream reporter fail confusingly).
• coder-eval report <run-dir> -f junit [-o PATH] — regenerates from any existing run dir (defaults to <run-dir>/junit.xml).
• Both entry points call the same writer, so they can't drift.

3. action.yml + release automation + dogfood

• Composite action: pinned-PyPI install with a version: local escape hatch, always emits run-dir/junit-path outputs and appends run.md to $GITHUB_STEP_SUMMARY, then exits with coder-eval's real exit code. Every input crosses into bash via env: — zero ${{ }} interpolation into script bodies.
• release.yml bumps the action's pinned version: default inside the amended release commit (anchored on a # <-- kept in sync comment, with a grep guard that fails the release loudly rather than shipping a stale pin) and force-moves the v<major> tag.
• pr-checks.yml gains a fork-gated action-dogfood job — the sensor for this surface, since it isn't reachable from pytest.

Design note (deviation from the original plan)

The action is deliberately agent-agnostic: it installs coder-eval but no coding-agent runtime. claude-agent-sdk does not bundle the claude binary, so callers provide it — the dogfood job installs Node + @anthropic-ai/claude-code itself, and the README/tutorial document this. Keeps the action usable for codex/antigravity consumers instead of forcing a Claude CLI install on everyone.

Review fixes

A multi-model review (gpt-5.6, gemini-3.1-pro, gpt-5.3-codex) found several robustness gaps around the untyped task_results row dicts, all fixed with regression tests that fail without the fix:

• Path containment — a crafted variant_id/task_id could steer the task.json lookup outside the run dir (an absolute value discards run_dir; .. walks up). A test confirmed a real leak before the fix.
• UnicodeDecodeError is a ValueError but not a json.JSONDecodeError, so undecodable bytes aborted the whole report despite the documented graceful degrade.
• Ambiguous replicate — with replicate_index absent and several replicate dirs, replicate 00's failure detail was misattributed to the row; now degrades instead.
• bool is an int subclass — a bool replicate_index rendered as [01]; NaN/inf durations emitted an invalid time="nan".
• Non-str variant_id/task_path crashed the dict key / Path(); same-basename skipped tasks collapsed into one JUnit identity.

Testing

• make verify green: 3454 passed, 2 skipped (both pre-existing environment skips), 90.98% coverage (threshold 80%); reports_junit.py ~98%.
• All new tests are hermetic — run dirs built by hand under tmp_path, no agents, no API.
• Phase 3 has no pytest surface by design; validated by a YAML syntax gate, a locally rehearsed release sed (proven to rewrite only the version line and preserve the trailing comment), an empirical simulation of the action's argv building under set -uo pipefail, and actionlint (clean on all additions).

What can't be verified until this PR runs

• action-dogfood — this PR both adds the job and is its first subject. Composite-action quirks (setup-uv PATH propagation, $GITHUB_ACTION_PATH install, output plumbing) are only provable in CI.
• Release automation — the version bump and moving major tag need a real release dispatch. Both have loud-failure guards.

🤖 Generated with Claude Code

[github-actions]

Claude Code is working…

I'll analyze this and get back to you.

View job run

[uipreliga]

Review: coder_eval — pr:37

Scope: pr:37 · branch feat/ci-gate-junit-action · 3b00991 · 2026-07-23T02:44Z · workflow variant

Change class: complex — new JUnit XML report generator (path-containment, FinalStatus classification, best-effort graceful-degrade parsing) plus CLI wiring and a composite GitHub Action with release automation

coder_eval is in excellent health (9.5/10) with clean security, type safety, and architecture across the board; the real risks are concentrated in the new JUnit/CI-action surface — a documented-but-nonexistent env var that silently drops the Bedrock backend, plus several JUnit-detail defects that misinform CI readers without ever changing a task's score or final_status — so the bottom line is a strong PR that needs targeted fixes to its CI-facing edges before merge.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	9.5 / 10	0	0	1	0	Three new reports_junit.py functions land in the CC 10-20 band
2. Type Safety	9.9 / 10	0	0	0	1	_load_task_json declares dict[str, Any]
3. Test Health	9.4 / 10	0	0	1	1	JUnit writer's key-coupling to eval_result_to_task_dict is untyped (RunSummary.task_results is list[dict[str, Any]]) and has no parity test — all JUnit tests use the synthetic _row() helper
4. Security	10 / 10	0	0	0	0	—
5. Architecture & Design	9.8 / 10	0	0	0	2	Per-task score gate implemented as inline Python in action.yml duplicates the package's scored-row interpretation and is CI-vendor-locked
6. Error Handling & Resilience	9.5 / 10	0	0	1	0	Root time attribute not guarded against NaN/inf, unlike per-testcase time — emits silently-invalid JUnit XML
7. API Surface & Maintainability	9 / 10	0	1	0	0	Action/README document a non-existent env var CODER_EVAL_API_BACKEND — the Bedrock-via-action config path is silently ignored
8. Evaluation Harness Quality	9 / 10	0	0	2	0	JUnit failure body mislabels informational (non-gating) criteria as [FAIL], diverging from reports.py

Overall Score: 9.5 / 10 · Weakest Axis: API Surface & Maintainability at 9 / 10
Totals: 🔴 0 · 🟠 1 · 🟡 5 · 🔵 4 across 8 axes.

Blockers

1. [Axis 7] Action/README document a non-existent env var CODER_EVAL_API_BACKEND — the Bedrock-via-action config path is silently ignored (README.md:148) — The Settings class (src/coder_eval/config.py:84) is a pydantic-settings BaseSettings with model_config at line 147 that sets NO env_prefix; field api_backend (config.py:98) therefore reads the env var API_BACKEND (case-insensitive), and extra="ignore" means any unrecognized env var is silently dropped. Both new doc surfaces tell users to set CODER_EVAL_API_BACKEND instead:

• README.md:148 (copy-pasteable action example): CODER_EVAL_API_BACKEND=bedrock
• action.yml:55 (the env input description): ... set ANTHROPIC_API_KEY, CODER_EVAL_API_BACKEND, model vars ...
  Because CODER_EVAL_API_BACKEND is not a Settings field and has no AliasChoices (only the telemetry fields do), it is ignored — the run silently falls back to ApiBackend.DIRECT. A user following the README's Bedrock example thus runs on Direct Anthropic (wrong backend/model, or a confusing missing-ANTHROPIC_API_KEY auth error) with no indication their backend selection was dropped. This is the ONLY documented way to select the backend through the composite action (there is no backend input; only model, extra-args, env). By contrast run_command.py:224 and docs/tutorials/02-ci-pipeline.md:240 correctly reference API_BACKEND / --backend. Fix both to API_BACKEND=bedrock (README.md:148 and action.yml:55). Optionally add a CE lint / alias so the framework's canonical env vars are documented from one source.

Non-blocking, but please consider before merge

1. [Axis 1] Three new reports_junit.py functions land in the CC 10-20 band (src/coder_eval/reports_junit.py:183) — Per the Axis-1 anchor, cyclomatic complexity 10-20 outside a hot module is Medium. radon confirms three new functions in this band: _task_case (line 183) C(17), _criteria_body (line 145) C(14), and _suite_gate_suite (line 271) C(11). _task_case in particular interleaves variant/task_id resolution, replicate-index formatting (has_replicate = isinstance(replicate_index, int) and not isinstance(replicate_index, bool)), classname derivation, finite/non-negative duration validation (valid_duration = ... and math.isfinite(duration) and duration >= 0), a properties-emission loop, and the status->category->failure/error branching in one body. The complexity is driven by defensive isinstance guards over untyped dict rows, so it is not a correctness defect, but it exceeds the readability bar for a single function. Consider extracting the duration-validation and properties-emission blocks (e.g. _time_attr(row) and _case_properties(case, row)) so each function stays comfortably under 10; this also isolates the well-tested edge-case logic. Low real-world impact given the code is well-tested and localized.
2. [Axis 3] JUnit writer's key-coupling to eval_result_to_task_dict is untyped (RunSummary.task_results is list[dict[str, Any]]) and has no parity test — all JUnit tests use the synthetic _row() helper (tests/test_reports_junit.py:39) — Every JUnit test builds its input via the local _row(...) helper (line 39) and the write_run_json fixture, both of which hand-craft dicts whose keys (status, variant_id, task_path, replicate_index, weighted_score, model_used, total_tokens, total_cost_usd, visible_turns, duration) are copied from what the writer reads — not from the real producer. generate_junit_xml consumes RunSummary.task_results, which in production is built by reports_experiment.eval_result_to_task_dict (batch.py:628). No test feeds real eval_result_to_task_dict output through generate_junit_xml, so the writer's key-coupling to the producer is unverified. A rename in the producer (e.g. visible_turns, total_cost_usd, or task_path) would silently drop the property/classname; a rename of status would route _status_of to "" and mis-bucket every task as an error, all with zero failing JUnit assertions. Add one integration test that constructs an EvaluationResult, runs it through eval_result_to_task_dict (or build_run_summary) into a real run.json, and asserts the resulting JUnit tree (grouping, classname, properties, status classification) — a full-field parity check rather than synthetic rows.
3. [Axis 6] Root time attribute not guarded against NaN/inf, unlike per-testcase time — emits silently-invalid JUnit XML (src/coder_eval/reports_junit.py:335) — In generate_junit_xml the root time is set unconditionally: root.set("time", f"{summary.total_duration_seconds:.3f}") (line 335). RunSummary.total_duration_seconds is a plain float with no allow_inf_nan=False/finite validator, and both json.loads and pydantic v2 accept a JSON NaN/Infinity literal, so a blob-pulled/older/corrupt run.json survives RunSummary validation and produces <testsuites name="r1" time="nan" ...> (confirmed by direct reproduction) — an invalid document many JUnit ingesters reject wholesale. This contradicts the module's own contract: _task_case (lines 199-205) deliberately guards the same value per row (valid_duration = isinstance(duration, int|float) and not isinstance(duration, bool) and math.isfinite(duration) and duration >= 0 ... time_str = f"{duration:.3f}" if valid_duration else "0.000") and is covered by test_malformed_row_types_degrade_gracefully, but the root time path was never given the same guard. Apply the identical finite/non-negative guard to summary.total_duration_seconds before setting the root time (fall back to "0.000"), and add a test asserting the root time is finite for a NaN/inf total_duration_seconds run.json. Severity Medium (not High/Critical): does not affect any task score/final_status and is only reachable via a corrupt/foreign run.json since coder-eval itself always computes a finite duration; matches the Medium anchor 'inconsistent error context (some paths wrap, some don't)'.
4. [Axis 8] JUnit failure body mislabels informational (non-gating) criteria as [FAIL], diverging from reports.py (src/coder_eval/reports_junit.py:165) — In _criteria_body, line 165 computes passed = isinstance(score, int | float) and isinstance(threshold, int | float) and score >= threshold from score>=threshold alone and, on the else branch, emits [FAIL] {ctype}: .... It never consults the gating field on the CriterionResult. Per the documented contract on CriterionResult.gating (models/results.py: 'False marks an informational criterion ... excluded from the score and the pass/fail gate, so every display surface must render it as informational rather than failed'), a non-gating criterion below its threshold must NOT be rendered as failed. The parallel path in reports.py:_compute_suite_rollup explicitly guards this — if not cr.gating: continue (with the comment 'Informational (weight: 0) criteria cannot fail a row, so they must not supply the reason the row is reported as failed'). The JUnit body diverges: on an already-failed testcase it will surface an informational criterion as [FAIL] ... < threshold, misleading a CI reader into thinking an informational metric caused the failure. This does NOT change the testcase's pass/fail (that comes from FinalStatus via _category_of) nor any score, so it is a display-contract defect, not a scoring-correctness one. Fix: read crit.get('gating', True) and render non-gating criteria as [INFO] (or skip them from the failure cause), mirroring reports.py.
5. [Axis 8] Dataset-row tasks (task_id contains '/') never load task.json, degrading JUnit detail for the whole dataset/activation suite (src/coder_eval/reports_junit.py:106) — _is_safe_component (line 106) returns False for any value containing '/': return bool(value) and value not in {'.', '..'} and '/' not in value and '\\' not in value, and _load_task_json (line 119) bails to None when not _is_safe_component(task_id). But dataset expansion sets task_id = f'{task.task_id}/{row_id}' (orchestration/task_loader.py:433), and the on-disk layout is run_dir/variant_id/task_id/NN/task.json (path_utils.build_task_run_dir:37) — i.e. the '/' in a dataset task_id is a legitimate structural path separator, and run_dir / variant / task_id already resolves to the correct nested dir. As written, EVERY dataset-derived row (the flagship nightly activation suite is dataset-backed per CLAUDE.md) fails the '/' check, so _criteria_body always falls back to the status-only body (status=... weighted_score=...) and never emits per-criterion detail — even though task.json exists and the belt-and-braces candidate.resolve().is_relative_to(run_dir.resolve()) check (line 137) already prevents traversal escapes. The testcase is still classified correctly (no scoring impact), so this is a detail-degradation, not a correctness bug. Fix: stop treating an internal '/' as unsafe — reject only absolute paths and '..' components (e.g. check '..' not in PurePosixPath(value).parts and not value.startswith('/')) and rely on the existing resolve-containment guard, so nested dataset task.json paths load.

Nits

1. [Axis 2] _load_task_json declares dict[str, Any] | None return but can return any JSON type (src/coder_eval/reports_junit.py:109) — Line 109 declares def _load_task_json(...) -> dict[str, Any] | None:, but line 140 returns json.loads(candidate.read_text(encoding="utf-8")), which is typed Any and at runtime yields whatever the file's top-level JSON is (a list, int, or str for a corrupted/blob-pulled task.json whose root is not an object). Pyright does not flag this because Any is assignable to dict. The annotation is therefore narrower than reality. There is no runtime bug: the sole caller _criteria_body (line 153-154) re-narrows via criteria = data.get(...) if isinstance(data, dict) else None, so a non-dict falls through safely. Tighten the annotation to -> dict[str, Any] | list[Any] | ... | None or return json.loads(...) if isinstance(<parsed>, dict) else None inside the helper so the declared type is honest and the isinstance guard cannot be dropped by a future caller without a type error.
2. [Axis 3] Symlink-escape containment branch (reports_junit.py:137) is untested; the test that claims to cover escape only exercises the earlier name guard (tests/test_reports_junit.py:520) — test_task_json_lookup_cannot_escape_run_dir (line 520) uses variant_id="../outside" and task_id=".." (line 542). Both are rejected by _is_safe_component (contains / / equals ..), so _load_task_json returns None at the early guard and the candidate.resolve().is_relative_to(run_dir.resolve()) symlink-containment check at reports_junit.py:137 (a coverage-missing line) never executes. The actual scenario line 137 guards — a legitimately-named <variant>/<task_id>/NN/task.json that is a symlink pointing outside run_dir — is uncovered. Add a case that creates such a symlink and asserts the outside content is not read (status-only fallback body).
3. [Axis 5] Per-task score gate implemented as inline Python in action.yml duplicates the package's scored-row interpretation and is CI-vendor-locked (action.yml:155) — The new minimum-task-score gate lives entirely in the composite action as embedded Python: for r in data.get("task_results", []): s = r.get("weighted_score") ... if isinstance(s, (int, float)) and not isinstance(s, bool) and math.isfinite(s) (action.yml ~line 181). This re-implements the same scored-row filtering the package already owns (reports_junit._task_case / reports_stats scored-row aggregation apply the identical isinstance(... int|float) and not bool and math.isfinite interpretation of weighted_score). Because it lives in action.yml rather than as a coder-eval run capability, (a) it is not reachable from the Azure DevOps path the JUnit output explicitly targets (docstring names PublishTestResults@2), and (b) it is a second interpretation of run.json's scoring semantics maintained outside the SSOT models, which can silently drift if weighted_score/task_results semantics change. Consider hoisting the floor into the CLI (e.g. coder-eval run --min-task-score) so the action becomes a thin caller and the interpretation stays in one tested place. Low: a new self-contained feature rather than a worsening of an existing path, and shell/YAML cannot import the Python helper directly.
4. [Axis 5] reports_junit.py couples the reports layer to evaluation/judge_context for a trivial string helper while a sibling renderer keeps its own (src/coder_eval/reports_junit.py:30) — Line 30 is from .evaluation.judge_context import truncate — the new reports module reaches into the judge-specific evaluation/judge_context.py for a generic string truncation helper, introducing a reports->evaluation dependency. Meanwhile the parallel renderer reports_html.py defines its own local _truncate (reports_html.py:250 def _truncate(text, limit)). This is pattern drift between sibling renderers and pulls a general utility out of a judge-context module. No import cycle exists (judge_context does not import reports), so this is a Low layering/DRY nit: move truncate to a shared util (e.g. formatting.py) or have both renderers share one helper.

What's Missing

Tests:

• 🟠 The new minimum-task-score gate (inline Python in action.yml, ~lines 155-205) has zero test coverage. This is the branch that turns a per-task score into a CI pass/fail verdict, yet nothing exercises it: no test trips the floor (a BELOW task failing the step), no test covers the [0.0,1.0]/non-numeric validation branches, the missing-run.json fail-closed branch, the NaN/inf skip (fail-closed against a blob-pulled run.json), or the 'no scored tasks' branch. Because the logic lives as embedded Python in YAML it is invisible to pytest/coverage. Add a test that runs the gate script against fixture run.json files (below-floor, at-floor, empty, malformed) and asserts exit status. This is new public scoring behavior shipped untested — the most common reviewer block per Agent 3 ('the gate that turns a gap into a score'). (trigger: action.yml)
• 🟡 No integration/parity test feeds real eval_result_to_task_dict output through generate_junit_xml. Every JUnit test builds input via the synthetic _row() helper (tests/test_reports_junit.py), which hand-copies the keys the writer reads, so the untyped producer->consumer key-coupling (RunSummary.task_results is list[dict[str,Any]]) is unverified — a producer rename of status/task_path/total_cost_usd/model_used/total_tokens/weighted_score would silently drop a property/classname or mis-bucket every row, with no failing assertion. Add one test that runs a real EvaluationResult -> eval_result_to_task_dict -> run.json -> generate_junit_xml and asserts grouping, classname, properties, and status classification. (trigger: tests/test_reports_junit.py) (restates: Axis 3: JUnit writer key-coupling untested)
• 🟡 The root <testsuites time=...> NaN/inf guard is both un-added and untested. _task_case guards the per-testcase time (valid_duration finite/non-negative check) and is covered by test_malformed_row_types_degrade_gracefully, but generate_junit_xml sets the root time unconditionally from summary.total_duration_seconds (which has no allow_inf_nan=False validator), so a corrupt/blob-pulled run.json emits time="nan" — invalid JUnit — with no test asserting root-time finiteness. Add the same finite/non-negative guard plus a regression test. (trigger: src/coder_eval/reports_junit.py) (restates: Axis 6: root testsuites time NaN/inf unguarded)
• 🔵 The symlink-escape containment branch (reports_junit.py:136, candidate.resolve().is_relative_to(run_dir.resolve())) is untested; test_task_json_lookup_cannot_escape_run_dir uses ../outside/.. which are rejected by the earlier _is_safe_component name guard, so line 136 never executes. Add a case with a legitimately-named <variant>/<task_id>/NN/task.json symlink pointing outside run_dir and assert the outside content is not read. (trigger: tests/test_reports_junit.py) (restates: Axis 3: symlink-escape branch untested)
• 🔵 The generic env-passthrough parser in action.yml (NAME=VALUE loop: CRLF trimming, left-trim, comment/blank skipping, name-regex validation, secret-safe positional error reporting, and the deliberate non-write to $GITHUB_ENV) is entirely new, security-relevant shell logic with no test. Nothing verifies a missing = is rejected, an invalid name is rejected, a secret VALUE is never echoed, or that forwarded vars don't bleed into later steps. Add a workflow-level or bats-style test over the parsing loop. (trigger: action.yml)

Parallel paths:

• 🟡 reports_junit.py _criteria_body computes pass/fail from score>=threshold alone and renders sub-threshold criteria as [FAIL], diverging from the parallel renderer path reports.py _compute_suite_rollup, which guards if not cr.gating: continue. A non-gating/informational criterion below threshold is thus mislabeled [FAIL] in the JUnit failure body — a fifth renderer that didn't adopt the gating contract the other renderers honor. Read crit.get('gating', True) and render non-gating as [INFO] (or skip). Does not change the testcase verdict (that comes from FinalStatus via _category_of), so it is a display-contract divergence, not a scoring bug. (trigger: src/coder_eval/reports_junit.py) (restates: Axis 8: informational criteria mislabeled [FAIL])
• 🔵 truncate was pulled from evaluation/judge_context into the new reports-layer module (reports_junit.py:30), introducing a reports->evaluation dependency, while the sibling renderer reports_html.py keeps its own local _truncate. The generic helper was not hoisted to a shared util, so the parallel renderers now drift on how they truncate. Move truncate to a shared formatting util used by both. (trigger: src/coder_eval/reports_junit.py) (restates: Axis 5: reports_junit couples to judge_context truncate)

Downstream consumers:

• 🔵 The action.yml score gate re-implements run.json's scored-row weighted_score interpretation (isinstance int|float, not bool, math.isfinite) outside the SSOT models — a second interpretation of task_results scoring semantics maintained in embedded Python, which can silently drift from reports_junit._task_case / reports_stats if those semantics change. It is also unreachable from the Azure DevOps PublishTestResults@2 path the JUnit output explicitly targets. Consider hoisting the floor into the CLI (coder-eval run --min-task-score) so the interpretation stays in one tested place and the action becomes a thin caller. (trigger: action.yml) (restates: Axis 5: per-task score gate duplicates scored-row logic)

Daily/nightly:

• 🟠 The action's only documented way to select a backend — CODER_EVAL_API_BACKEND (README.md:148, action.yml:55 env description) — is not a Settings field (Settings has no env_prefix and reads API_BACKEND; extra='ignore' silently drops the unknown var). So the nightly/CI --backend bedrock judge route selected through the composite action silently falls back to ApiBackend.DIRECT. A user following the README Bedrock example runs on Direct Anthropic (wrong backend/model, or a confusing missing-ANTHROPIC_API_KEY error) with no signal. Fix both surfaces to API_BACKEND=bedrock. (trigger: README.md) (restates: Axis 7: CODER_EVAL_API_BACKEND env var doesn't exist)
• 🟡 _is_safe_component rejects any value containing '/', and dataset expansion sets task_id = '/<row_id>' (task_loader.py:433) with a matching nested on-disk layout (path_utils.py:37). So every dataset-derived row in the flagship dataset-backed nightly activation suite fails the '/' check and degrades to a status-only JUnit body (no per-criterion detail), even though task.json exists and the resolve-containment guard already prevents traversal. The PR does not state this blast radius on the nightly. Reject only absolute paths and '..' components (rely on the existing resolve-containment check) so nested dataset task.json paths load; classification is unaffected (detail-degradation only). (trigger: src/coder_eval/reports_junit.py) (restates: Axis 8: dataset-row tasks never load task.json)

Harness & Lint Improvements

Static checks (lint / type):

• [ruff] Enable ruff's mccabe gate: add "C901" to the select list in pyproject.toml and set [tool.ruff.lint.mccabe] max-complexity = 10. reports_junit._task_case C(17), _criteria_body C(14), and _suite_gate_suite C(11) would all trip the gate in make check before review. (ruff already runs PLR0912/PLR0915 but neither is a cyclomatic-complexity gate.) Prevents: Finding #1 (Axis 1): three new reports_junit.py functions land in the CC 10-20 band.
• [ce-lint] New rule CE026 (layering, AST ImportFrom check mirroring CE004/no_cli_imports_in_core.py): the reports layer (reports.py, reports_junit.py, reports_html.py, reports_experiment.py, reports_stats.py) must not import from coder_eval.evaluation.*. Slot the rule in tests/lint/rules/ce026_no_reports_import_evaluation.py and register it in tests/lint/runner.py. Would have flagged from .evaluation.judge_context import truncate at reports_junit.py:30 and forced the shared string helper into a util module. Prevents: Finding #6 (Axis 5): reports layer couples to evaluation/judge_context for a trivial string helper.
• [ce-lint] New rule CE027 (doc/config env-var parity): scan README.md, action.yml, and docs/ for env-var tokens (uppercase NAME= assignments and CODER_EVAL_*/API_* references) and cross-check each against the field env names + AliasChoices of coder_eval.config.Settings (the rule imports Settings directly). Flag any documented env var with no backing Settings field, since extra="ignore" silently drops it at runtime. Would have caught the non-existent CODER_EVAL_API_BACKEND in README.md:148 and action.yml:55 (real field is API_BACKEND). Prevents: Finding #8 (Axis 7, High): action/README document a non-existent env var CODER_EVAL_API_BACKEND.
• [ce-lint] New rule CE028 (finite-float on run.json-sourced models): any float field on the pydantic models that parse an external/blob-pulled run.json (RunSummary and the task-row models) must declare allow_inf_nan=False (or an equivalent finite validator), so a JSON NaN/Infinity literal fails validation instead of surviving into the writer. This fixes the class at the parse seam and covers BOTH the guarded per-row time and the unguarded root time in one place. Prevents: Finding #7 (Axis 6): root time attribute unguarded against NaN/inf emits invalid JUnit XML.

Harness improvements (not statically reachable):

• Add a producer→consumer integration/parity test: build a real EvaluationResult, run it through reports_experiment.eval_result_to_task_dict (or build_run_summary) into a real run.json, feed that through generate_junit_xml, and assert grouping / classname / properties / status classification on the resulting tree. Replaces the synthetic _row() helper as the source of the writer's key contract. Why not static: RunSummary.task_results is list[dict[str, Any]] (models/results.py:880), so the writer's dependence on producer key names (status, task_path, total_cost_usd, model_used, total_tokens, weighted_score) is untyped — pyright/pydantic cannot see a rename; only running real producer output through the consumer surfaces the drift. Prevents: Finding #3 (Axis 3): JUnit writer's key-coupling to eval_result_to_task_dict has no parity test.
• Add a JUnit test that creates a validly-named <variant>/<task_id>/NN/task.json which is a symlink pointing OUTSIDE run_dir, and assert the outside content is not read (status-only fallback body), exercising the candidate.resolve().is_relative_to(run_dir.resolve()) containment branch at reports_junit.py:136. Why not static: The guarded branch only executes for a name that passes _is_safe_component yet resolves outside run_dir — that requires a real on-disk symlink and pathlib resolve() at runtime; no static check can reach a symlink-containment outcome. Prevents: Finding #4 (Axis 3): symlink-escape containment branch is untested (existing test only hits the name guard).
• Add a JUnit test with a dataset-style nested task_id (containing '/', e.g. suite/row_0) laid out on disk as run_dir/variant/suite/row_0/NN/task.json, asserting _load_task_json loads it and per-criterion detail is emitted (not the status-only fallback). This also pins the intended fix (reject only absolute paths and '..' components, rely on resolve-containment). Why not static: Correctness depends on the '/' in a dataset task_id being a real nested-directory separator matching path_utils.build_task_run_dir — verifiable only against an actual run-dir layout at runtime, not by grep/AST. Prevents: Finding #10 (Axis 8): dataset-row tasks (task_id contains '/') never load task.json, degrading JUnit detail for the whole activation suite.
• Add a shared/parity test (or extract one shared criterion-render helper used by both reports.py suite-rollup and reports_junit._criteria_body) that feeds a CriterionResult with gating=False below threshold through both surfaces and asserts neither renders it as the failure cause — reports_junit should emit [INFO], matching reports.py's if not cr.gating: continue. Why not static: The contract is semantic — 'a non-gating criterion below threshold must render as informational, not failed' — which requires evaluating a populated CriterionResult through the rendering logic and comparing the two surfaces; a lint grep for [FAIL] without a gating reference is too false-positive-prone to be the gate. Prevents: Finding #9 (Axis 8): JUnit failure body mislabels informational (non-gating) criteria as [FAIL].
• Hoist the minimum-task-score floor into the package/CLI (e.g. coder-eval run --min-task-score) so the composite action becomes a thin caller, and back it with a unit test on the CLI. The action.yml then stops re-implementing the scored-row interpretation (isinstance(...int|float) and not bool and math.isfinite). Why not static: The duplication lives as Python embedded inside action.yml shell/YAML, which no ruff/pyright/CE rule can parse or tie back to the package helper; it is an architecture/design decision to keep run.json scoring semantics in one tested place rather than a mechanically-detectable code pattern. Prevents: Finding #5 (Axis 5): per-task score gate implemented as inline Python in action.yml duplicates the package's scored-row interpretation.

Top 5 Priority Actions

1. Fix the non-existent env var CODER_EVAL_API_BACKEND at README.md:148 and action.yml:55 to API_BACKEND (or add an AliasChoices) — it is currently the only action-level way to select Bedrock and is silently dropped by Settings' extra="ignore", so users get a Direct-Anthropic run with wrong backend/model or a confusing auth error.
2. Stop treating an internal '/' as unsafe in _is_safe_component/_load_task_json (reports_junit.py:106,119) and rely on the existing resolve-containment guard, so dataset/activation-suite rows (task_id = suite/row_id) load task.json instead of degrading every row to status-only JUnit detail.
3. Consult the CriterionResult.gating field in _criteria_body (reports_junit.py:165) and render non-gating criteria as [INFO] rather than [FAIL], mirroring reports.py's if not cr.gating: continue, so informational metrics below threshold no longer appear to CI readers as the failure cause.
4. Apply the same finite/non-negative guard used per-testcase (reports_junit.py:199-205) to the root time at reports_junit.py:335, so a corrupt/blob-pulled run.json with NaN/inf total_duration_seconds cannot emit time="nan" and produce JUnit XML that ingesters reject wholesale.
5. Add an integration parity test that runs a real EvaluationResult through eval_result_to_task_dict → run.json → generate_junit_xml (tests/test_reports_junit.py currently only uses the synthetic _row helper), asserting grouping/classname/properties/status so a producer-side rename of status/task_path/total_cost_usd/etc. can no longer silently mis-bucket or drop JUnit fields.

---

Stats: 0 🔴 · 1 🟠 · 5 🟡 · 4 🔵 across 8 axes reviewed.

[uipreliga]

Review responses — addressed in aa8ef40

Thanks for the thorough review. All blocker + non-blocking findings are fixed; nits triaged below. make verify green (3541 passed, 91% cov), rebased on latest main.

Blocker

• ✅ [Axis 7] CODER_EVAL_API_BACKEND non-existent env var — fixed to API_BACKEND in both README.md:148 and action.yml. To prevent recurrence I added CE027 (tests/lint/doc_env_parity.py), a lint rule that scans README/action.yml/docs for framework-prefixed env-var assignments not backed by a Settings field/alias or a real src/ consumer — exactly the "silently dropped by extra="ignore"" class. Verified it catches the reintroduced bug with zero false positives on the current corpus.

Non-blocking

• ✅ [Axis 3] Producer→consumer parity test — added test_parity_real_producer_output_through_writer: builds a real EvaluationResult, runs it through eval_result_to_task_dict → run.json → generate_junit_xml, and asserts grouping/classname/properties/status against the produced dict (not hand-copied literals), so a producer-side rename now fails an assertion.
• ✅ [Axis 6] Root <testsuites> time NaN/inf — extracted a shared _time_attr helper now applied to both per-testcase and root time; regression test injects a raw NaN literal. (While here, also hardened it against a pathological huge-int duration that overflows float()/.3f — math.isfinite(10**400) raises OverflowError, which would have crashed the whole report; now degrades to 0.000.)
• ✅ [Axis 8] Informational criteria mislabeled [FAIL] — _criteria_body now renders non-gating criteria as [INFO] regardless of pass/fail, mirroring reports.py. Fail-safe: only an explicit JSON false is informational (null/non-bool → gating), matching CriterionResult.gating's default. Tests cover failing, passing, and null-gating cases.
• ✅ [Axis 8] Dataset rows never loaded task.json — replaced the _is_safe_component /-reject with _is_safe_relpath, which accepts a nested <suite>/<row_id> task_id but still rejects absolute/backslash/../Windows-drive values; the resolve()-containment check remains the backstop. Tests: nested load, nested .. rejection, drive-letter rejection.

Nits

• ⏭️ [Axis 1] reports_junit.py function complexity — partially reduced (_time_attr extraction pulled the duration-guard out of _task_case). Full CC-band refactor deferred; low real-world impact given coverage.
• 💬 [Axis 2] _load_task_json return type — left as-is. json.loads returns Any; the sole caller re-narrows via isinstance(data, dict), so there's no runtime gap. Low value vs. churn.
• ⏭️ [Axis 3] Symlink-escape containment branch untested — added nested-.. and drive-letter degradation tests; the specific "validly-named symlink pointing outside run_dir" case is not yet added. Deferred (🔵).
• ⏭️ [Axis 5] Per-task score gate hoisted to CLI (--min-task-score) — deferred to its own change. It's net-new surface (and closes the Azure DevOps reachability gap + gains pytest coverage), so it deserves a dedicated PR + tests rather than riding this one.
• 💬 [Axis 5] truncate coupling reports_junit→evaluation — left as-is. judge_context.truncate (-> str) and reports_html._truncate (-> tuple[str, bool]) have different contracts, so it's not a clean dedup; a shared-util move is cosmetic. Skipped per YAGNI.

CodeQL

• 💬 Alert 43 "Comparison of identical values" — false positive; details in the inline reply. The flagged test already uses math.isfinite, which is CodeQL's own recommendation; there is no self-comparison.