[ronniegeraghty]

What changed

Upgrade the pinned Vally CLI to 0.21.0 and default repository evals to its native Docker backend in .vally.yaml, with offline deterministic grading (grade-network: none). Native CLI/npm runs and the manual Actions launcher share the configuration. Paid evals remain manual-only; no custom runner or repository Dockerfile is added.

Document Linux Docker/Compose and authentication requirements, native backend precedence, separate task/grade roles, and failure/cleanup limits. The pinned Evals plugin Run reader rejects this project default before execution: it is currently unsupported here, not a local fallback. Raise the repository Node minimum to 22.15.0, matching Vally (24+ recommended).

Keep the lockfile registry-neutral using npm's documented omit-lockfile-registry-resolved option. Copilot review identified that the earlier dependency upgrade had already downgraded 109 unchanged-version packages from SHA-512 to SHA-1 before URL removal. Restore their exact base SHA-512 values: all 218 dependency entries now have SHA-512 pins, with no version or other dependency-entry metadata changes and no registry-specific URLs. The configured mirror's metadata omits dist.integrity for a checked affected package and supplies the matching SHA-1 dist.shasum; URL stripping itself changed no other fields. Document this fallback explicitly.

Version and changelog

No plugin source or plugin version changes. Remove only the failed, unpublished pr-description 0.1.1 candidate; preserve published 0.1.0 source and changelog history. Behavioral failures remain for future PRs. The host Copilot 1.0.98 pin and marketplace remote pins are unchanged.

Checks

• npm run lint: six skills and seven eval definitions passed.
• Five focused model-free Docker-default tests passed. They cover native backend selection, missing Docker fail-closed behavior, retained/removed plugin coverage, and rejection of all shared-launcher backend overrides in both argument forms. No Docker or model is started.
• Local catalog consistency, git diff --check, and a full 218-entry integrity/metadata audit passed. The full retained model-free script suite also passed on the preceding head.
• All 19 cases select Docker/offline grading in model-free planning, including the independent reviewer's three cases.
• GitHub CI and actual Copilot reviews are checked per pushed head; earlier green checks are not claimed for a later head. Cycle-one's three inline findings were fixed, replied to and resolved in 438fea4; its overview-only integrity warning was also fixed and answered. Cycle two is requested, not yet reported as complete.
• Earlier clean dependency installation used the configured development registry, not proof of public npm availability. No paid workflow was dispatched for these updates.

Previously approved native full-suite execution

Run native-docker-21-full-default-20261112T020833014Z completed once: 7 evals / 19 cases; 12 PASS, 7 behavioral FAIL; zero execution errors, zero grading errors, zero retries, zero exclusions. Every selected case completed and was graded. A behavioral FAIL is valid evidence, not an infrastructure error; the manual workflow's existing --require-pass still requires behavioral passes.

These results belong to the recorded pre-removal snapshot, including the failed unpublished routing-frontmatter candidate, not a paid rerun of the final PR head. Eval bodies, prompts, graders and thresholds were unchanged. Raw failed candidate evidence remains retained outside the contribution; this is not a passing skill evaluation.

• Source-manifest SHA256: d4c7babd2bb63367c66cd6b5e6df8307ccc46a0463e093fb9499257513704c32.
• Corrected derived handoff SHA256: 474875a6c7b150c630fa0dff20b6963eff0cd22f9a7b02042d301e7df5c47b2c.
• Native report: outputs/2026-11-12T02-31-32-422Z/eval-results.md, with raw results.jsonl in that run folder.
• Runtime: Vally 0.21.0, actual container Copilot 1.0.109, model claude-haiku-5.5. The host CLI pin is distinct. Sixteen skill cases exposed 15 skill names (12 catalog and three built-ins); three independent reviewer cases exposed the three built-ins and actual custom agent.
• Immutable custom validation image: sha256:654c029a370cb5cb0bce6cea326fc47cdeda3eaa67e89f20ff3f02566f98723d, built from the official 0.21 recipe with a credential-free npm feed build-argument workaround. Explicit --backend docker and repeated offline-grade policy were required. This proves native Docker full-suite execution, not bare-default live image bootstrap.
• Cleanup telemetry was balanced and observed owner container/volume inventories were empty. Live grading inspection was partial; complete live-isolation inspection is not claimed. Earlier corrected synthetic task/grade probing passed 1/1; its invalid predecessor remains retained. No paid evals were rerun for these fixes.

Risk

• Canonical automatic bootstrap remains unverified/environment-blocked. Public npm TLS fails before certificate exchange in observed host/container/proxy paths; the root cause is not established. The working configured feed is not inherited by Vally's native builder, which exposes no registry override. The custom validation image was not tagged as the canonical cache image or published. No company registry, private Dockerfile, cache workaround or TLS weakening is committed.
• The host runs Vally orchestration and trusted build/setup tooling. Docker task/grade roles use separate workspaces; tasks receive model credentials and normal network access for Copilot and pinned GitHub skill fetching. Offline deterministic grading is configured policy, not proof from complete live inspection. Root initialization/evidence helpers and shared-kernel risks remain.
• Workspaces, captured evidence, reports and model inputs remain real data/retention surfaces. Review eval setup, network effects and forwarded credentials before each paid run; interrupted cleanup may leave resources. No new credential storage or publishing service is added.
• Vally CLI is an upgraded MIT-licensed development dependency. Existing remote sources, immutable pins and ownership are unchanged.
• Repository defaults and minimum Node compatibility change; pinned plugin Run is currently unsupported. Use native Vally CLI until upstream support exists. Missing Docker/image preparation fails closed, not into host execution. Reverting the Docker-default commit restores previous backend selection without machine-wide settings changes.

No merges or auto-merge changes are requested.

[Copilot]

🟡 Changes recommended

The lockfile weakens dependency integrity by dropping SHA-512 pins, while unresolved Node-version and regression-test issues affect supported installs, plugin customizations, and launcher safeguards.

3 open issues

• Gate reviewer eval assertion on the plugin being retained · New
• Reject backend overrides in launcher guard checks · New
• Raise the Node minimum to Vally’s required version · New

What changed in this PR

Moves repository eval tooling to Vally 0.21.0 and defaults execution to native Docker, while keeping paid runs manual-only. Plugin source and versions remain unchanged.

Changes:

• Configures Docker execution with offline deterministic grading.
• Adds Docker preflight checks and three model-free backend-selection and fail-closed regressions.
• Documents authentication, backend precedence, isolation limits, cleanup, and the pinned Evals plugin’s unsupported Run entrypoint.

File	Description
README.md	Introduces Docker requirements, defaults, and native CLI guidance.
package.json	Pins Vally CLI to 0.21.0.
package-lock.json	Updates Vally and transitive dependencies; removes registry-specific tarball URLs.
docs/​checks-and-evals.md	Details Docker setup, authentication, overrides, execution boundaries, and lockfile maintenance.
.vally.yaml	Selects Docker by default and disables grading network access.
.github/​workflows/​evals.yml	Adds Docker/Compose preflight checks and removes host executor settings.
.github/​scripts/​docker-defaults.test.mjs	Adds model-free checks for Docker inheritance and failure without Docker.

_{Review effort: Max (auto)}

[ronniegeraghty]

Addressed the integrity warning in review PRR_kwDOVRlbx88AAAABXe7pXA in 438fea4, including the overview-only finding. Comparing the PR base, the saved pre-URL-removal lockfile, and the reviewed head showed 109 unchanged-version packages had already changed from SHA-512 to SHA-1 during the dependency upgrade. The native omit-lockfile-registry-resolved step itself changed no other package fields. The configured registry currently omits dist.integrity for @types/debug@4.1.13 and supplies dist.shasum, which exactly matches the reviewed SHA-1 value. I restored all 109 SHA-512 values from the base for those exact unchanged versions. All 218 dependency entries now have SHA-512 integrity; no package version or other dependency-entry metadata changed, and zero registry-specific resolved URLs were reintroduced. The earlier preservation statement was too narrow: URL stripping preserved the initial upgrade, but that upgrade had already weakened these hashes. Documentation now calls out this mirror fallback. The three inline findings were also fixed, replied to, and resolved. Model-free lint, five focused tests, catalog consistency, diff checks, and the checksum audit pass. New-head GitHub CI and the next completed Copilot review are still pending; no paid evals or Docker operations were run.

[Copilot]

🟡 Changes recommended

The documentation’s token examples leave existing GitHub credentials exposed to agent-controlled tasks instead of isolating the intended token.

1 open issue

• Clear existing GitHub tokens before exposing the selected credential · New

3 resolved since last review

• Raise the Node minimum to Vally’s required version
• Reject backend overrides in launcher guard checks
• Gate reviewer eval assertion on the plugin being retained

_{Review effort: Max (auto)}