[mstaats-nvidia]

Correlate coverage and mutant scores by exact subset and experiment context, using average ranks for ties. Preserve paired observations, explicit exclusions, and input digests. Constant ranks and fewer than two paired subsets produce an unavailable coefficient with a reason.

correlate.py is a thin CLI over correlate; it consumes the two score artifacts and the explicit missing-pair policy. It needs neither execution settings nor executable inputs. The coordinator passes the policy from analysis.yml.

Usage and library API: labs/metric-benchmark/docs/correlate.md.

Earlier full-suite validation (before the signing follow-up): 3,082 tests, 29 skipped, plus Ruff/format, type checks for Python 3.12/3.13, copyright, license and whitespace checks.

Real pressure test: maintained Rho Agent baseline + an action-budget mutant × three Terminal Bench 2.0 tasks (regex-log, log-summary-date-ranges, polyglot-c-py). All six trials completed without runtime errors after pinning the common runtime to Python 3.12 and a dependency cutoff matching Rho’s lockfile era. The initial unpinned run exposed upstream installation failures and was preserved separately. Both agents passed 2/3 tasks. Mutant activation was recorded in every mutant trace.

A model authored a tool-only audit; the real Eval Author generation, measurement and reporting tools computed coverage for four fixed subsets (0.25, 0.50, 0.50, 0.50). All mutant scores were zero, so correlation correctly returned unavailable (constant score ranks). This exercises the pipeline, not scientific metric validity or semantic-judgment coverage.

Verified native validation, completed execution resume with no new Harbor calls, byte-identical cached analysis, standalone CLI/coordinator parity, and changed analysis with executable agent bundles and task environments removed. Saved execution evidence remained byte-identical. Raw evidence and credentials remain outside Git.

Stack: main (includes merged #157 configuration separation) → #95 mutant scores → #96 coverage → #97 correlation → #99 coordinator. These four PRs are ready for review; GitHub stack #158.

Signing transition follow-up

Case identity now uses the same canonical content digests as execution identity. Adding pins to unchanged inputs preserves resume, validation and scoring; changed agent/task content still fails. Signed metadata can analyze the saved run after executable inputs are removed. Missing execution rows must contain an explicitly unavailable metric.

Current follow-up validation: 178 benchmark tests passed, 1 skipped; Ruff/format, type checks on Python 3.12/3.13, copyright, license and whitespace checks passed. Regression coverage exercises a genuinely unsigned execution followed by signing, unchanged evidence on resume, rejection of modified agent/task contents, standalone scoring and full analysis without executable inputs. Hashing optimization is deferred. No new live model or Docker trials were needed for this follow-up. Downstream PRs are rebased; all are ready for review.

Rebased after #157 merged: 178 benchmark tests passed, 1 optional test skipped; Ruff/format and type checks for Python 3.12/3.13 passed at the coordinator tip. Incremental PR patches are unchanged.

[aleckhoury]

Correlation looks right.

• Compared spearman with scipy.stats.spearmanr over 20,000 random cases (n = 0..12, both continuous and heavily tied inputs). Maximum absolute difference was 3.3e-16, and the two agreed on every case where no coefficient can be computed.
• Matching on exact subset ID plus an identical Context is a safe join, because both inputs build Context with scoring.context().
• missing_policy values and the default match config.py and the CLI.
• The caveat that n = 2 always gives ±1 is documented.

At this head: 163 passed, 1 skipped.