Skip to content

test(metric-benchmark): revalidate sealed jobs after validation failure - #163

Open
mstaats-nvidia wants to merge 1 commit into
mainfrom
test/metric-benchmark-revalidate-sealed-jobs
Open

mstaats-nvidia wants to merge 1 commit into
mainfrom
test/metric-benchmark-revalidate-sealed-jobs

Conversation

@mstaats-nvidia

Copy link
Copy Markdown
Contributor

A job receipt is written before case validation. Add a regression test proving that a validation failure does not let sealed jobs bypass validation on resume.

The fake Harbor runner emits a trial with an incorrect task checksum before sealing. The test checks that both attempts fail real validation, all three job receipts exist, resume never invokes Harbor, no results index or experiment manifest is published, and saved files remain byte-identical. Production code is unchanged.

Follow-up to the receipt-ordering review concern on #106.

Validation:

  • Full repository suite: 3,012 passed, 29 skipped (unavailable/opt-in Gym, Node/browser, Docker, Ratchet, Windows and live-model checks).
  • Execution suite: 21 passed, 1 opt-in Docker skip.
  • Deliberately skipping validation on resume makes the new test fail.
  • The same test passes on feat(metric-benchmark): separate cases, execution, and analysis configuration #157 at 12c9718 with its load_execution(cases.yml, execution.yml) fixture-loading adjustment; that branch was not modified.
  • Ruff/format, type checks for Python 3.12/3.13, copyright, license, and whitespace checks passed.

@mstaats-nvidia
mstaats-nvidia requested a review from a team as a code owner October 30, 2026 09:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants