Skip to content

docs(metric-benchmark): define benchmark architecture - #93

Merged
mstaats-nvidia merged 5 commits into
mainfrom
docs/metric-benchmark-design
Oct 23, 2026
Merged

mstaats-nvidia merged 5 commits into
mainfrom
docs/metric-benchmark-design

Conversation

@mstaats-nvidia

@mstaats-nvidia mstaats-nvidia commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Defines an experimental benchmark for evaluating how well our coverage metric predicts evaluation effectiveness through mutation testing.

The concise root README describes the project, its work-in-progress status, and the expectation that case examples will grow, change, or be dropped as we build in plain sight. Prep is external to this repository; runs consume prepared case examples, fixed subset manifests, and a YAML benchmark definition.

The stage descriptions focus on four executable stages: run mutants × eval tasks, compute mutant scores, compute coverage, and correlate the score pairs. The first execution script will be a basic local Harbor loop; NeMo Helix support is planned afterward. The architecture and both SVG figures explain the inputs, component boundaries, shared results directory, and statistical comparison.

This is the documentation-only base of the stack. Implementation PRs #94 → #105 → #106 → #95 → #96 → #97 → #99 inherit the overview and design unchanged. Their detailed CLI guide lives separately in docs/tools.md.

Validation: the five files in this PR are Markdown/SVG only. Local links and anchors, Markdown fences, headers, SVG XML, and whitespace checks pass; the updated figure was rendered and visually inspected. Restacking preserves all source and test files byte-for-byte. No live evaluations or model calls were made.

Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
@mstaats-nvidia mstaats-nvidia changed the title docs: draft external metric benchmark architecture docs(metric-benchmark): define benchmark architecture Oct 21, 2026
Signed-off-by: mstaats <mstaats@nvidia.com>
@mstaats-nvidia
mstaats-nvidia marked this pull request as ready for review October 21, 2026 20:29
@mstaats-nvidia
mstaats-nvidia requested a review from a team as a code owner October 21, 2026 20:29
Comment thread labs/metric-benchmark/docs/stages.md Resolved
Comment thread labs/metric-benchmark/docs/stages.md Resolved
Signed-off-by: mstaats <mstaats@nvidia.com>
@mstaats-nvidia
mstaats-nvidia merged commit 500e5f5 into main Oct 23, 2026
16 of 18 checks passed
@mstaats-nvidia
mstaats-nvidia deleted the docs/metric-benchmark-design branch October 23, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants