[mstaats-nvidia]

Separate the experiment into three referenced files: cases.yml, execution.yml, and analysis.yml. Cases are defined once; stage CLIs receive only the files they consume. Execution receipts use an execution-only identity, and indexes retain all native verifier metrics so analysis can change without rerunning Harbor.

The signing tool, validation contract, documentation, fixtures and examples are updated. Saved evidence also records the case-definition digest, allowing downstream analysis to check agent/task identities without opening executable bundles.

Earlier full-suite validation (before the signing follow-up): 3,082 tests, 29 skipped, plus Ruff/format, type checks for Python 3.12/3.13, copyright, license and whitespace checks.

Real pressure test: maintained Rho Agent baseline + an action-budget mutant × three Terminal Bench 2.0 tasks (regex-log, log-summary-date-ranges, polyglot-c-py). All six trials completed without runtime errors after pinning the common runtime to Python 3.12 and a dependency cutoff matching Rho’s lockfile era. The initial unpinned run exposed upstream installation failures and was preserved separately. Both agents passed 2/3 tasks. Mutant activation was recorded in every mutant trace.

A model authored a tool-only audit; the real Eval Author generation, measurement and reporting tools computed coverage for four fixed subsets (0.25, 0.50, 0.50, 0.50). All mutant scores were zero, so correlation correctly returned unavailable (constant score ranks). This exercises the pipeline, not scientific metric validity or semantic-judgment coverage.

Verified native validation, completed execution resume with no new Harbor calls, byte-identical cached analysis, standalone CLI/coordinator parity, and changed analysis with executable agent bundles and task environments removed. Saved execution evidence remained byte-identical. Raw evidence and credentials remain outside Git.

Stack: main → #157 configuration separation → #95 mutant scores → #96 coverage → #97 correlation → #99 coordinator. All five PRs are ready for review; GitHub stack #158.

Signing transition follow-up

Case identity now uses the same canonical content digests as execution identity. Adding pins to unchanged inputs preserves resume, validation and scoring; changed agent/task content still fails. Signed metadata can analyze the saved run after executable inputs are removed. Missing execution rows must contain an explicitly unavailable metric.

Current follow-up validation: 178 benchmark tests passed, 1 skipped; Ruff/format, type checks on Python 3.12/3.13, copyright, license and whitespace checks passed. Regression coverage exercises a genuinely unsigned execution followed by signing, unchanged evidence on resume, rejection of modified agent/task contents, standalone scoring and full analysis without executable inputs. Hashing optimization is deferred. No new live model or Docker trials were needed for this follow-up. Downstream PRs are rebased; all are ready for review.

Review cleanup: unknown case IDs now use the existing lookup in both analysis loaders; removed the unused stored definitions and signing/execution parameters; corrected the architecture description of execution versus analysis settings. Current #157 head 607c95f passed the full local suite: 3,029 passed, 29 skipped, plus the focused 120-test benchmark suite, Ruff/format, Python 3.12/3.13 type checks, copyright, license, whitespace, and DCO checks. The dependent metric stack also passed 178 tests with one optional Docker skip.

[asutermo]

nit: I know we talked licenses - is rho-agent fine here?

[mstaats-nvidia]

Here we're ok for now, no code merged. Codex thinks it's also fine but it sounds like we will need some passes through some approvers based on convos I had with folks.

[aleckhoury]

Reviewed as part of stack #158. The three-way config split is clean: no single-file paths are left over. Execution identity covers only case payloads plus execution settings, so analysis-only changes reuse jobs. Signing unsigned inputs keeps resume working, and changed agent/task content refuses reuse. At this head, pytest -k metric_benchmark gives 120 passed, 1 skipped.

The inline notes are minor cleanup and docs fixes. None of them block merge.