Repository navigation
docs(metric-benchmark): define benchmark architecture - #93
Merged
Merged
Conversation
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
This was referenced Oct 20, 2026
Signed-off-by: mstaats <mstaats@nvidia.com>
This was referenced Oct 21, 2026
Signed-off-by: mstaats <mstaats@nvidia.com>
mstaats-nvidia
marked this pull request as ready for review
October 21, 2026 20:29
bcleary-nv
reviewed
Oct 21, 2026
This was referenced Oct 21, 2026
bcleary-nv
reviewed
Oct 22, 2026
bcleary-nv
reviewed
Oct 22, 2026
bcleary-nv
approved these changes
Oct 22, 2026
bcleary-nv
reviewed
Oct 22, 2026
bcleary-nv
reviewed
Oct 22, 2026
bcleary-nv
reviewed
Oct 22, 2026
Signed-off-by: mstaats <mstaats@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Defines an experimental benchmark for evaluating how well our coverage metric predicts evaluation effectiveness through mutation testing.
The concise root README describes the project, its work-in-progress status, and the expectation that case examples will grow, change, or be dropped as we build in plain sight. Prep is external to this repository; runs consume prepared case examples, fixed subset manifests, and a YAML benchmark definition.
The stage descriptions focus on four executable stages: run mutants × eval tasks, compute mutant scores, compute coverage, and correlate the score pairs. The first execution script will be a basic local Harbor loop; NeMo Helix support is planned afterward. The architecture and both SVG figures explain the inputs, component boundaries, shared results directory, and statistical comparison.
This is the documentation-only base of the stack. Implementation PRs #94 → #105 → #106 → #95 → #96 → #97 → #99 inherit the overview and design unchanged. Their detailed CLI guide lives separately in
docs/tools.md.Validation: the five files in this PR are Markdown/SVG only. Local links and anchors, Markdown fences, headers, SVG XML, and whitespace checks pass; the updated figure was rendered and visually inspected. Restacking preserves all source and test files byte-for-byte. No live evaluations or model calls were made.