[mstaats-nvidia]

Adds an exploratory Helix execution path alongside the local Harbor runner: one CLI submits already-prepared cases; a second checks completion and downloads the complete per-task evidence. Both are thin wrappers over Python library functions.

Stacked on #157. This is a separate implementation draft; #159 remains unchanged.

Current behavior

• Takes the existing cases file plus separate endpoint/scheduling settings and prepared-input bindings. Examples use https://helix.example.com; authentication comes from an existing Helix client context.
• Submits one native taskset run per case/agent, with fixed attempt policy and recoverable submission using native idempotency keys.
• Checks all runs and the complete task matrix before downloading. Returns exit code 2 while execution or packaging is pending, and exit code 1 for failures or incomplete terminal output.
• Preserves complete native task artifacts and publishes the collection only after successful download and validation of its membership.

Deliberate draft boundary

The download currently produces an evidence collection with analysis_ready: false, not the benchmark's execution.json contract. Helix image binding changes the native task checksum; resolved task and agent identities must be verified against prepared inputs before this can feed scoring. Native evidence is not rewritten and the existing validator is not bypassed.

The official Helix client is an optional, separately installed runtime for this prototype. No dependency lock changes, upload/preparation workflow, new service, database, or UI are included. Deployed submission/download remains unverified; the next step is a small live test on an authorized instance with prepared inputs.

Validation

• Full repository suite: 3,047 passed, 30 skipped. Final focused suite after small hardening changes: 131 passed, 2 skipped.
• Prototype tested in the actual Helix source environment: 21 passed, including native request-schema validation. Network transport was mocked.
• Three actual local Docker trials exercised Oracle, NOP, and an image-bound task. SDK-packaged evidence was collected with native result bytes preserved. The unchanged benchmark validator accepted the original samples and reproduced the checksum rejection after image binding.
• Ruff lint/format, targeted type checks, copyright, license/OSV, whitespace and sign-off checks passed. Optional browser/runtime/platform/live-model tests account for skips; no hosted evaluation or model calls were made.

Suggested review order

1. labs/metric-benchmark/docs/submit-helix-evals.md
2. labs/metric-benchmark/docs/download-helix-results.md
3. labs/metric-benchmark/metric_benchmark/helix.py
4. labs/metric-benchmark/metric_benchmark/helix_client.py
5. labs/metric-benchmark/metric_benchmark/helix_cli.py and the two CLI shims
6. tests/test_metric_benchmark_helix.py

Rebased onto main after #157 merged. No prototype changes. Current benchmark suite: 140 passed, 2 skipped; targeted Ruff/format and type checks passed. No hosted jobs were run.

[asutermo]

I would link the Helix repo if not done already (either here or somewhere higher level). Maybe longer-term a link to compass/eval-author inside helix, etc