Repository navigation
feat(metric-benchmark): define benchmark format, reader, and verification - #94
Merged
Merged
Conversation
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-subsets
branch
3 times, most recently
from
October 21, 2026 21:08
95ea55e to
a4f467e
Compare
mstaats-nvidia
changed the title
This was referenced Oct 21, 2026
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-subsets
branch
from
October 23, 2026 14:33
a4f467e to
37fa19f
Compare
mstaats-nvidia
added a commit
that referenced
this pull request
Oct 23, 2026
Defines an experimental benchmark for evaluating how well our coverage metric predicts evaluation effectiveness through mutation testing. The concise [root README](https://github.com/NVIDIA-NeMo/eval-author/blob/docs/metric-benchmark-design/labs/metric-benchmark/README.md) describes the project, its work-in-progress status, and the expectation that case examples will grow, change, or be dropped as we build in plain sight. Prep is external to this repository; runs consume prepared case examples, fixed subset manifests, and a YAML benchmark definition. The [stage descriptions](https://github.com/NVIDIA-NeMo/eval-author/blob/docs/metric-benchmark-design/labs/metric-benchmark/docs/stages.md) focus on four executable stages: run mutants × eval tasks, compute mutant scores, compute coverage, and correlate the score pairs. The first execution script will be a basic local Harbor loop; NeMo Helix support is planned afterward. The [architecture and both SVG figures](https://github.com/NVIDIA-NeMo/eval-author/blob/docs/metric-benchmark-design/labs/metric-benchmark/docs/architecture.md) explain the inputs, component boundaries, shared results directory, and statistical comparison. This is the documentation-only base of the stack. Implementation PRs #94 → #105 → #106 → #95 → #96 → #97 → #99 inherit the overview and design unchanged. Their detailed CLI guide lives separately in `docs/tools.md`. Validation: the five files in this PR are Markdown/SVG only. Local links and anchors, Markdown fences, headers, SVG XML, and whitespace checks pass; the updated figure was rendered and visually inspected. Restacking preserves all source and test files byte-for-byte. No live evaluations or model calls were made.
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-subsets
branch
4 times, most recently
from
October 26, 2026 15:21
c0def9f to
4f511c7
Compare
mstaats-nvidia
changed the title
mstaats-nvidia
added a commit
that referenced
this pull request
Oct 27, 2026
…braries Signed-off-by: mstaats <mstaats@nvidia.com> Validation deferred at user request while iterating on manifests. [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com> Validation deferred at user request while iterating on manifests. [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com> Validation deferred at user request while iterating on manifests. [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com> Propagate explicit unsigned loading and rebase onto current #94. Validation remains deferred during review. [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com>
mstaats-nvidia
added a commit
that referenced
this pull request
Oct 27, 2026
…braries Signed-off-by: mstaats <mstaats@nvidia.com> Validation deferred at user request while iterating on manifests. [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com> Validation deferred at user request while iterating on manifests. [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com> Validation deferred at user request while iterating on manifests. [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com> Propagate explicit unsigned loading and rebase onto current #94. Validation remains deferred during review. [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com> Rebase onto comment-preserving SHA signing in #94. [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com>
mstaats-nvidia
changed the title
mstaats-nvidia
marked this pull request as ready for review
October 27, 2026 16:18
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-subsets
branch
from
October 27, 2026 16:27
cdffdda to
1961059
Compare
bcleary-nv
approved these changes
Oct 27, 2026
bcleary-nv
left a comment
bcleary-nv
left a comment
Contributor
There was a problem hiding this comment.
Unsigned iteration / experiment digest — Might be worth a mention in benchmark-format.md or tools.md that benchmark_sha256 doesn't mean "bundle unchanged" (e.g. when iterating on tasks with allow_unsigned) until strict load / sha-sign --check.
File extensions — experiment.yml / agent.yml / tasks.yml vs subsets.json / description.json — can we state the convention once (what’s YAML vs JSON and why)?
Generation (my POC) — Planning to match this format, not a parallel schema. Likely need a defined slot for generated tasks (tasks.generated.yml or generated_tasks: in tasks.yml); happy to follow whatever you prefer when generation hooks in.
Signed-off-by: mstaats <mstaats@nvidia.com>
Rely on typed configuration validation for non-JSON YAML values. Report missing pinned inputs by relative path and attach case/artifact context once on both standalone and complete loads. Preserve task/agent context and other filesystem errors. Remove premature tool documentation and clarify model and test names. Signed-off-by: mstaats <mstaats@nvidia.com>
Move the prepared benchmark definition into a checked-in YAML fixture. Tests create its referenced inputs and verify the recorded content pins. Signed-off-by: mstaats <mstaats@nvidia.com>
Drop schema_version from benchmark configuration and shared records. Use the strict model directly and update the prepared subset pin. Signed-off-by: mstaats <mstaats@nvidia.com>
…nt YAML Signed-off-by: mstaats <mstaats@nvidia.com>
Validation deferred at user request while iterating on the configuration format. [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com>
Update the existing example and its references; validation remains deferred at user request. [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com>
Keep task revisions in the task manifest and resolve named subsets from task ID lists. Remove inherited task-collection benchmark metadata. Validation is deferred at user request. [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com>
[skip ci] Signed-off-by: mstaats <mstaats@nvidia.com>
[skip ci] Signed-off-by: mstaats <mstaats@nvidia.com>
… membership [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com>
Collapse task inventory and source declarations into one task list. Add the sha-sign CLI and library with preflight validation and explicit force replacement of existing pins. [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com>
…rary APIs Move argument parsing into sha-sign and add mutually exclusive --check and --force modes. Share content hashing and verification across signing, checking, and benchmark reads. [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com>
Allow unsigned inputs only via explicit allow_unsigned=True. Route check_benchmark directly through the normal strict loader. [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com>
Edit only SHA scalar spans or insert SHA fields using YAML parser locations. Rename --force and the library keyword to --resign/resign, and document the fixed-depth signing invariant. [skip ci] Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-subsets
branch
from
October 27, 2026 20:50
9325f08 to
22ef4c8
Compare
Collaborator
|
nit: If not there, perhaps worth updating AGENTS.md to ensure benchmark/labs context shouldn't be used in the eval-authoring context (basically guardrailing)? |
asutermo
approved these changes
Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR defines the prepared benchmark format, provides the Python library for reading it, and adds input validation and SHA signing/verification. It establishes the input contract used by the downstream execution and analysis stages.
experiment.ymldefines cases inline, referencing reusable agent, task and subset manifests. Each task is listed once with its identity, revision and source path. All input paths resolve relative to the bundle root.benchmark-format.mdspecifies the recommended layout and fields, with example YAML/JSON files undertests/fixtures/metric_benchmark/. It explains the YAML/JSON convention, unsigned configuration identity, and how generators produce ordinary task manifests.load_benchmark()loads the experiment and its prepared inputs.read_benchmark()reads the configuration and referenced manifests without requiring execution files. Signed inputs are required by default;allow_unsigned=Truepermits missing pins during iteration while still verifying supplied SHAs.sha-signCLI delegates to library functions: signing adds pins,--resignupdates existing pins while preserving comments and unrelated formatting, and--checkuses the same strictload_benchmark()path without writing files.Review the format specification and examples first, then
config.pyfor reading,models.pyfor validation, andsigning.py/io.pyfor SHA handling.tools.mddocuments library and CLI usage. The example agents are synthetic Oracle/NOP stand-ins, not a runnable real-agent benchmark.Validation: rebased onto
main(027ff01). The full local suite passed 2,583 tests with 9 optional/platform skips. Fresh CI passed on Python 3.12 and 3.13, including NeMo Gym and Linux/macOS/Windows compatibility jobs. All 53 configuration/signing tests, lint, formatting, both Python type-check targets, copyright, licenses and whitespace checks passed. The downstream benchmark suite passed 127 tests (one opt-in Docker skip), including the latest #105 fixtures. The unchanged execution implementation previously passed nine real Docker trials with resume and relocation checks.Stack: #94 → #105 → #106 → #95 → #96 → #97 → #99. Architecture #93 is merged. This PR is approved; downstream implementation PRs remain drafts.