[ronniegeraghty]

What changed

Define the chosen starter model roster in one native Vally experiment: Claude Sonnet 5.5 (claude-sonnet-5.5) and GPT-6.1 Terra (gpt-6.1-terra).

• evals/standard-models.experiment.yaml uses Vally's model matrix, keeping measurement inputs fixed and running one worker. It selects the retained skill and independent-agent evals, not fixtures or generated results.
• npm run eval:models runs the experiment; npm run eval:models -- --dry-run previews it without execution.
• README setup instructions tell template adopters where and how to choose their own roster and baseline. The existing eval guide explains model access, costs, native output handling and the distinction from single-model commands/Actions.
• Ignore the explicit experiment results directory. This is the same ignore needed by the separate results-path PR, not a dependency on its report changes.

The experiment is the single source for the roster. .vally.yaml does not support a project-global model list, and the current Action does not support experiment execution; neither is given a duplicated or invented configuration field. Existing individual eval and Action defaults remain unchanged, as requested.

Evidence and limits

The installed Vally 0.19.0 CLI resolves 14 plans: seven evals under each of the two models. Each pair retains the same eval hash; only its declared model configuration differs. Skill/eval lint and diff checks pass.

This is native planning evidence, not a paid two-model run or a claim that either model passed or is available to every account. No model request, judge, installation, runtime upgrade, workflow dispatch or repository setting change was made.

Vally 0.19 experiments use a separate output flag, not project paths.results; the npm command explicitly selects evals/results/standard-models/, and documentation explains how to customize it.

Native contract: Vally 0.19 experiment schema. Sandbox defaults and #13's grader fixes remain separate work.

[Copilot]

🟡 Changes recommended

The new eval:models script is missing from the validation test’s expected scripts map, causing the Validate workflow to fail.

1 open issue

• Update validation test expectations for the new eval:models script · New

What changed in this PR

Adds a native Vally experiment for evaluating retained skill and agent evals across a configurable starter model roster, without changing existing single-model defaults.

Changes:

• Defines a two-model matrix with Claude Sonnet 5.5 as the baseline and one worker.
• Adds eval:models with an explicit results directory and dry-run support.
• Documents roster customization, model access, costs, and output handling.

File	Description
README.md	Introduces the standard-model command and setup guidance.
package.json	Adds the native experiment command.
evals/​standard-models.experiment.yaml	Defines eval selection, model variants, baseline, and worker count.
docs/​checks-and-evals.md	Explains customization, execution costs, previews, and results storage.
.gitignore	Excludes generated results under evals/results/.

_{Review effort: Balanced (auto)}

[Copilot]

🟢 Approval recommended

No active review issues are recorded for the native multi-model evaluation experiment.

No open issues 🎉

1 resolved since last review

• Update validation test expectations for the new eval:models script

_{Review effort: Balanced (auto)}