feat(metric-benchmark): specify and validate experiment results by mstaats-nvidia · Pull Request #105 · NVIDIA-NeMo/eval-author

mstaats-nvidia · 2026-10-21T21:10:08Z

Specify the results that benchmark execution must produce, and add validation code that checks those results against both the upstream benchmark definition and the results-format constraints.

results-format.md defines the directory layout, experiment and case JSON indexes, completeness and attempt rules, and native Harbor evidence requirements. A mock results directory illustrates one baseline and two mutants across three tasks. validation-report.md describes the validator’s output and includes the corresponding example report.

The validator checks that configured cases, agents and tasks are accounted for and that recorded outcomes agree with the prepared inputs and native evidence. A thin CLI calls the Python library, validating all cases by default or one case selected with --case, using the same experiment output root. It writes a report only after all selected cases pass; missing observations require explicit reasons rather than being treated as zero scores.

Tests cover the example, CLI/library agreement, invalid evidence, relocation, and case selection. Validation completed on the final revision: full local suite 2,596 passed and 25 skipped; the focused configuration/signing/validation run passed all 82 tests. Ruff lint/format, type checks for Python 3.12 and 3.13, copyright, dependency licenses, whitespace, and commit sign-off checks passed. Hosted CI passed, including both Python test jobs. Local skips cover unavailable Node rendering, optional external/live runtimes, and Windows-specific behavior.

This follows the merged benchmark-definition PR #94. The execution runner and analysis stages remain in downstream draft PRs #106, #95, #96, #97, and #99; their callers need to follow the revised results schema when the stack is updated.

Defines an experimental benchmark for evaluating how well our coverage metric predicts evaluation effectiveness through mutation testing. The concise [root README](https://github.com/NVIDIA-NeMo/eval-author/blob/docs/metric-benchmark-design/labs/metric-benchmark/README.md) describes the project, its work-in-progress status, and the expectation that case examples will grow, change, or be dropped as we build in plain sight. Prep is external to this repository; runs consume prepared case examples, fixed subset manifests, and a YAML benchmark definition. The [stage descriptions](https://github.com/NVIDIA-NeMo/eval-author/blob/docs/metric-benchmark-design/labs/metric-benchmark/docs/stages.md) focus on four executable stages: run mutants × eval tasks, compute mutant scores, compute coverage, and correlate the score pairs. The first execution script will be a basic local Harbor loop; NeMo Helix support is planned afterward. The [architecture and both SVG figures](https://github.com/NVIDIA-NeMo/eval-author/blob/docs/metric-benchmark-design/labs/metric-benchmark/docs/architecture.md) explain the inputs, component boundaries, shared results directory, and statistical comparison. This is the documentation-only base of the stack. Implementation PRs #94 → #105 → #106 → #95 → #96 → #97 → #99 inherit the overview and design unchanged. Their detailed CLI guide lives separately in `docs/tools.md`. Validation: the five files in this PR are Markdown/SVG only. Local links and anchors, Markdown fences, headers, SVG XML, and whitespace checks pass; the updated figure was rendered and visually inspected. Restacking preserves all source and test files byte-for-byte. No live evaluations or model calls were made.