[Azayzel]

Summary

• Add one bounded evidence-resolution round per inconclusive pair, using selected freshly verified read-only tools and one adjudication. Preserve original judgments, audit evidence, usage, and failures without rerunning generation.
• Offer best-supported recommendations with explicit qualitative confidence while preserving genuine ties, strict mode, required-phrase gates, and correctness blockers. Keep legacy draft settings compatible.
• Add optional Docker-isolated checks for single-file Node/CommonJS answers with a trusted user-selected node:test checker, explicit approval, resource limits, no network or workspace mounts, and result provenance.
• Wire settings, confidence, executable results, and resolution audits through the UI, history, imports/exports, Markdown, HTML, and aggregate decisions.
• Include configurable request deadlines, phase-level diagnostics, optional simultaneous generation, branding/documentation improvements, and Pokemon and async-scheduler demos.

Validation

• COMPARELAB_DOCKER_TESTS=1 npm run package -- 0.2.4 --no-update-package-json --no-git-tag-version: 273 tests passed, zero failures or skips.
• Real scheduler answer passed all 11/11 isolated checks; an intentionally incorrect implementation failed 10/11.
• Real Docker checks exercised failures, timeout cleanup, early-exit false-pass prevention, and isolation restrictions; no test containers remained.
• Browser validation at 390px confirmed settings persistence, checker selection/removal, and no horizontal overflow.
• Verified packaged identity and final source contents; git diff --check passed.

Limits and rollout

• Docker is optional for normal comparisons. Executable checks currently support dependency-free, single-file Node/CommonJS only; containers reduce risk but are not a perfect sandbox, and imported results are not signed attestations.
• Resolution may add up to five model requests per reviewed pair. Confidence is qualitative, not a calibrated probability. Genuine ties remain valid.
• Live model adjudication has not yet been smoke-tested; its orchestration is covered by mocked tests.
• This does not claim to fix provider latency or intermittent GPT timeouts. Reasoning settings remain unchanged; there are no automatic retries.
• Existing drafts retain their prior policy; enable Best-supported pick and Resolve inconclusive reviews in Advanced settings to opt in.
• The v0.2.4 VSIX was built locally only. No release was published, and the source manifest version remains unchanged.

[Copilot]

🟡 Changes recommended

Unresolved defects can distort comparison decisions and confidence, break verification under restrictive permissions, and discard completed verification evidence or assessments.

7 open issues

• Render aggregate confidence in the repeated-test report · New
• Scope acceptance failures to the affected side · New
• Require a preferred decision before claiming prompt improvement · New
• Preserve verification results when evaluation fails · New
• Make input files readable by the container user · New
• Use each pair’s decision status for response badges · New
• Enforce the byte limit on terminal error messages · New

What changed in this PR

This PR extends CompareLab’s evaluation pipeline with bounded evidence resolution, confidence-aware recommendations, and optional Docker-isolated correctness checks, while preserving acceptance gates and legacy settings.

Changes:

• Add evidence adjudication, executable checks, and recommendation gates with retained audit data.
• Support configurable VS Code deadlines, simultaneous generation, and phase-level diagnostics.
• Update the UI, exports, documentation, demos, and regression coverage.

File	Description
test/​verification.test.js	Covers checker validation, Docker isolation, cleanup, and cancellation.
test/​usage.test.js	Requires complete paired timing for latency ratios.
test/​tool-agent.test.js	Tests evidence-resolution request and retention budgets.
test/​review-resolution.test.js	Covers adjudication, confidence, checker provenance, and exports.
test/​prompt-review.test.js	Tests acceptance blockers in prompt recommendations.
test/​orchestration.test.js	Covers concurrent generation and execution settings.
test/​model-request.test.js	Tests deadlines, cancellation, and stalled request phases.
test/​html.test.js	Verifies embedded offline branding.
test/​extension.test.js	Exercises controls, diagnostics, checks, resolution, and persistence.
test/​decision.test.js	Covers acceptance gates and recommendation eligibility.
test/​core.test.js	Updates report expectations for unsupported recommendations.
src/​webview/​panel.js	Adds review policies, checker controls, and execution settings.
src/​tools/​agent.js	Adds optional evidence-resolution budgets.
src/​reporting/​html.js	Renders gated decisions, confidence, and audit details.
src/​providers/​vscode/​review.js	Extends review identity and adjudication instructions.
src/​providers/​vscode/​request.js	Adds cancellable model-request deadlines.
src/​providers/​vscode/​limits.js	Defines and validates configurable timeouts.
src/​providers/​vscode/​generation.js	Applies deadlines and reports request phases.
src/​providers/​verification.js	Runs restricted Docker checks and captures results.
src/​metrics/​usage.js	Gates timing ratios on complete observations.
src/​extension.js	Integrates verification, resolution, diagnostics, and concurrency.
src/​comparison/​verification.js	Validates checkers, extracts modules, and binds results.
src/​comparison/​review-resolution.js	Coordinates verification and one evidence-resolution round.
src/​comparison/​prompt-review.js	Adds acceptance gates to prompt-adoption decisions.
src/​comparison/​orchestration.js	Preserves execution settings and aggregates gated decisions.
src/​comparison/​decision.js	Centralizes recommendation eligibility and confidence.
src/​comparison/​core.js	Adds review evidence and decision details to reports.
README.md	Refreshes onboarding, feature summaries, and demo links.
package.json	Updates the extension description.
media/​runners.css	Adds shared branding styles.
media/​app.js	Persists controls and displays confidence and audits.
examples/​pokemon-picnic.json	Adds a ready-made tool-backed comparison.
examples/​pokemon-picnic.agent.md	Defines evidence-based picnic selection instructions.
examples/​pokemon-mcp-demo.md	Documents the Pokemon tool workflow.
examples/​check-async-scheduler.cjs	Adds scheduler conformance checks.
examples/​async-scheduler.json	Provides a scheduler comparison preset and contract.
examples/​async-scheduler.agent.md	Defines scheduler implementation instructions.
examples/​async-scheduler-demo.md	Explains comparison and external checker execution.
docs/​guide.md	Documents policies, diagnostics, execution options, and limits.
DECISIONS.md	Records the new evaluation and execution tradeoffs.
.vscode/​mcp.json	Configures the Pokemon demo endpoint.

_{Review effort: Balanced}

[Azayzel]

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply the suggestion exactly.

Do not make changes beyond what is described in the linked review thread.

[Copilot]

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply the...

Addressed all seven review items in commit 0b9b513.

[Copilot]

🟡 Changes recommended

Permissive staging-directory and file permissions expose potentially private generated answers and trusted checker code to other local users on Linux hosts with shared /tmp.

3 open issues

• Keep shared staging files private during execution · New
• Render verification results when evaluation fails · New
• Apply the record-size guard to retained checker observations · New

3 previously missed issues

In code that hasn't changed since last review

Retain failed collection attempts in the resolution audit

src/​extension.js:118

When collection recovers from an invalid-arguments failure and succeeds, freezeEvidence excludes the failed invocation and retains only collection metadata, so that failure disappears from the saved resolution audit. Its call count and usage remain, but the activity explaining them is lost. Preserve the complete bounded toolEvidence in the audit collection while continuing to send only successful observations as evidence.

Propagate resolution-tool approval cancellation to the review

src/​extension.js:344

Declining or closing a resolution tool-approval dialog throws vscode.CancellationError, but this path never cancels the shared comparison token. orchestrateTest therefore marks the review as failed and starts generating the next pair, potentially consuming more model usage after cancellation. Propagate this cancellation to reviewCancellation, as the generation-tool path already does.

Preserve checker results when the review-stage timeout fires

src/​extension.js:344

If the checker finishes but the evaluator reaches the review-stage deadline, withModelTimeout rejects with its own error, without the partialVerification attached inside resolveReview. The saved pair therefore loses both completed checker observations. Capture verification independently of the raced review promise and preserve it on timeout or cancellation without marking the review successful.

7 resolved since last review

• Enforce the byte limit on terminal error messages
• Use each pair’s decision status for response badges
• Make input files readable by the container user
• Preserve verification results when evaluation fails
• Require a preferred decision before claiming prompt improvement
• Scope acceptance failures to the affected side
• Render aggregate confidence in the repeated-test report

_{Review effort: Balanced}

[Azayzel]

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply the suggestion exactly.

Do not make changes beyond what is described in the linked review thread.

[Copilot]

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply the...

Addressed all three review comments in 2d071a6.

[Copilot]

🔵 Needs a closer look

Recommendation paths bypass investigation-limit and verification-binding safeguards, allowing budget-limited or evidence-truncated outputs and candidates with stale checks to be recommended despite pair-level blocks.

No open issues 🎉

6 previously missed issues

In code that hasn't changed since last review

Enforce limit state in reopened and standalone quality reviews

src/​comparison/​decision.js:103

For a reopened budget-limited or evidence-truncated investigation, pairDecision blocks recommendation, but this helper omits limited, allowing the main quality review and Markdown export to recommend the same output. Derive the limit state from the selected retained pair before calling assessmentDecision. Also preserve this metadata in compare()'s normalized variants so standalone reviews enforce the same gate.

Validate checker bindings before accepting a candidate majority

src/​comparison/​orchestration.js:331

With two valid candidate wins and one stale observation still labeled passed, a three-pair imported record can recommend the candidate even though pairDecision blocks the mismatched pair. Reopening does not prevent this because validateTestRecord does not validate evaluation.verification. Check the output and checker bindings here as well, so a majority cannot hide mismatched executable checks.

Retain collection metadata before rejecting oversized evidence

src/​comparison/​review-resolution.js:45

A collection can finish within the raw tool-evidence budget but exceed 128 KiB after freezeEvidence formats the snapshot. This check then throws before retaining collected.collection, so the failure path omits the completed collection's request count and token usage from the audit and totals. Retain the collection metadata before validating the evidence size; the oversized evidence can still be rejected.

Keep failed tool requests in the resolution audit

src/​extension.js:118

When a model corrects a rejected tool request and completes collection, freezeEvidence() removes the failed invocation and retains only successful observations plus summary metadata. The resolution audit therefore loses the rejected request and its error. Retain the complete bounded tool evidence in the audit while continuing to send only successful observations to adjudication.

Preserve completed checker observations across timeout and cancellation

src/​extension.js:344

If the review deadline expires after executable checks finish, withModelTimeout rejects with its own error before resolveReview can attach partialVerification, so the saved pair loses completed checker observations. Checkpoint those observations outside the raced operation and propagate them through timeout and cancellation failures. The same timeout wrapper is used for direct reviews at src/extension.js:500–504.

Preserve partial verification when direct review evaluation fails

src/​extension.js:515

For Review existing responses, an evaluator failure after completed checks carries error.partialVerification, but this handler discards it and leaves evaluation null. Users therefore cannot inspect, save, or export those observations without executing the answers again. Preserve completed verification independently of the direct assessment and pass it through the display, history, and export paths.

3 resolved since last review

• Keep shared staging files private during execution
• Apply the record-size guard to retained checker observations
• Render verification results when evaluation fails

_{Review effort: Balanced}