Add evidence-backed recommendations and isolated correctness checks

.vscode/mcp.json

@@ -0,0 +1,8 @@
+{
+  "servers": {
+    "pokemon": {
+      "type": "http",
+      "url": "https://gateway.pipeworx.io/pokemon/mcp"
+    }
+  }
+}

DECISIONS.md

@@ -18,24 +18,29 @@ See [module ownership](CONTRIBUTING.md#code-boundaries) for the source layout an
 | **Optional `max_completion_tokens` / `max_tokens`** | Accommodates different compatible servers | User must know the selected endpoint's dialect |
 | **Explicit model-facing messages** | Preserve task and incoming-event context across providers | HTTP providers support system/user roles; VS Code receives instructions as user context, not a privileged system message |
 | **Shared prompt when override is blank** | Invisible whitespace must not mask a populated task | Review, runs, reports, and cache identity share resolution; meaningful prompt text is preserved exactly |
-| **Sequential generation** | Reduces local GPU contention | Generate both (no tools) skips tools and review; timing is not a concurrent throughput benchmark |
+| **Sequential by default, opt-in simultaneous models** | Avoids local GPU contention unless the user chooses overlap | Both generation actions honor the checkbox; paired reviews wait for both successful outputs. Concurrency and non-default timeouts survive history/export; timing is not an isolated throughput benchmark |
 | **Controlled paired orchestration** | Snapshot configuration, alternate generation order, and reuse one evaluator across 1-5 pairs | Live investigations can gather different evidence; repetitions of one task are not a representative benchmark |
 | **Native VS Code tool discovery** | Reuse configured MCP and extension tools without a vendor allowlist or separate client | Chat selections are not inherited; provider setup and authentication remain in VS Code |
 | **Fresh read-only verification and per-call approval** | Keep tool availability separate from permission to invoke it | Unmarked tools are excluded; saved selections grant no approval; declarations are not sandboxing |
 | **Frozen evidence for prompt edits by default** | Separate instruction effects from differences in discovered evidence | Collect once or reuse supplied evidence, then run both prompts without live tools; results remain untrusted data |
-| **No artificial investigation budget** | Let agents gather the evidence needed for the task | Provider limits, per-request timeouts, record-size limits, and cancellation still apply; call counts and costs can vary |
+| **No artificial generation-investigation budget** | Let agents gather the evidence needed for the task | Provider limits, per-request timeouts, record-size limits, and cancellation still apply; review-only resolution has a separate explicit budget |
 | **CodeMind web-event normalization** | Mirror `MappedAlertMapper` and the web `AgentDefinitionRunner` message roles | Does not reproduce backend safety/report additions, Kafka filtering, trusted hashes, mobile scope, or live tools |
 | **Bounded retained test records** | Export original instructions, trigger, frozen evidence, all retained results and measurements | 8 MiB cap; failures/cancellation remain explicit; imported artifacts are not authenticated attestations |
 | **Explicit context-window provenance** | Configured maximum does not prove effective input capacity | Unknown capacities and actual usage are surfaced, not guessed |
 | **Record provider usage when available** | More faithful than estimated token counts | Tokenizers and hidden reasoning differ across models |
 | **SecretStorage keyed by endpoint** | API keys kept outside Webview and exports | Keys are managed separately for distinct URLs |
 | **HTTPS for remote, HTTP for loopback** | Avoid inadvertent plaintext transport to remote endpoints | Plain HTTP compatible servers on a LAN are not supported |
 | **Reject redirect; bound body/time** | Limit accidental credential forwarding and excessive work | Some proxy architectures and long generations may fail |
-| **Deterministic literal phrase checks** | Easy to reproduce and audit | Not semantic accuracy, JSON-schema validation or fact checking |
+| **Required phrases are acceptance gates** | Recompute against visible outputs; a failing output cannot be recommended, even if the judge prefers it | Case-insensitive substrings, not semantic accuracy or fact checking; failure never automatically awards the other side |
+| **Separate evaluator judgment from decision readiness** | Retain the audit trail without calling ties, unsupported judgments, or metadata-only prompt edits improvements | Reopened records may lose previously displayed recommendations under stricter gates |
+| **Complete paired timing for ratios** | Avoid speed claims from a selectively recorded subset | Partial medians stay visible, but their ratios are unavailable |
 | **No overall model-quality score** | Avoid unjustified claims from isolated outputs | Teams must choose their rubric and representative samples |
 | **Optional Copilot Language Model API judge** | Uses the user's available Copilot models without another key | Consent, policy and quota apply; no automatic Agent tools or chat history |
 | **VS Code model discovery and generation by default** | Reuse installed model providers and existing access for baseline/candidate | Input tokens are estimates; actual usage/settings and a privileged system role are not exposed by this integration |
-| **Fixed evidence and six-criterion rubric** | Prioritizes supported claims and safe actions over prose style | No live investigation or independent truth verification during in-panel evaluation |
+| **Frozen outputs and six-criterion rubric** | Prioritizes supported claims and safe actions over prose style | Optional post-generation review evidence is separately attributed; it never changes the inputs either model saw |
+| **One bounded evidence-resolution round** | Investigate concrete disputed claims rather than abstaining immediately or voting indefinitely | At most three collection requests and six approved read-only tool calls, then one adjudication; errors and missing capabilities stay explicit |
+| **Separate practical picks from confidence** | A supported task-specific ranking can remain useful despite judge uncertainty | Best-supported policy permits provisional picks, not invented wins; strict policy and acceptance gates remain available |
+| **Opt-in isolated Node correctness checks** | Obtain executable evidence without running generated code in the extension host | Requires a local Docker engine and preinstalled image, explicit approval and a trusted checker; no network/workspace mounts, strict resource limits, cleanup, and hash-bound observations |
 | **Anonymized labels and reversed-order check** | Reduces identity and position bias | Default doubles requests; agreement does not establish reliability |
 | **Source citation validation and abstention** | Accept whitespace-only differences while retaining the source passage; reject altered claims | Valid quotations can still be misinterpreted; human calibration remains necessary |
 | **One citation correction request per evaluation** | Recover from model quotation errors without unbounded retries | Extra usage is recorded; persistent failures stop the evaluation, with no relaxed semantic matching |

README.md

@@ -1,3 +1,5 @@
+<img src="media/comparelab-icon.png" alt="CompareLab logo" width="64" height="64">
+
 # CompareLab
 
 Compare two AI models on the same task, or test whether a prompt edit helps.
@@ -9,19 +11,15 @@ or paste existing responses. No hosted CompareLab service or runtime npm depende
 
 ## Quick start
 
-To run CompareLab from source:
-
-1. Clone this repository and open it in VS Code.
-2. Press **F5** and select **Run CompareLab Extension**.
-3. In the new **Extension Development Host** window, click the **CompareLab**
-   Activity Bar icon, then **Open CompareLab**.
-4. Choose **Model comparison** or **Prompt comparison** and follow the steps below.
+1. Install [CompareLab from the VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=azayzel-comparelab.comparelab).
+2. Click the **CompareLab** Activity Bar icon, then **Open CompareLab**.
+3. Choose **Model comparison** or **Prompt comparison** and follow the steps below.
 
-Requires Node.js 20+ and VS Code 1.90+. Live tools require VS Code 1.95+,
+Requires VS Code 1.90+. Live tools require VS Code 1.95+,
 a tool-capable model, and a working tool provider in the same window.
 MCP support also depends on your VS Code version and server setup.
 
-Prefer a local install? See [build and installation](docs/guide.md#build-and-package).
+For source development or a local build, see [build and installation](docs/guide.md#build-and-package).
 The **CompareLab: Compare Model Outputs** command remains available in the Command Palette.
 
 ## Compare models
@@ -34,7 +32,7 @@ The **CompareLab: Compare Model Outputs** command remains available in the Comma
 4. Click **Run comparison**. It runs both sides and reviews their outputs.
 
 Read the reports and cited assessment side by side. The visible run summary shows
-how many pairs and reviews are planned; adjust them under **Advanced settings**.
+how many pairs and reviews are planned; **Advanced settings** also offers simultaneous models and a configurable VS Code timeout (10 minutes by default).
 The default is three pairs with reversed-order reviews.
 
 For pasted responses, open each **Paste / edit raw response** editor, then use
@@ -52,8 +50,10 @@ By default, both prompts receive the same evidence without live tools.
 The review links the instruction change to quotes from both outputs.
 It can recommend keeping the original; a different answer is not automatically better.
 
-Try the [GitHub demo](examples/github-tools-demo.md) for ready-to-use prompts,
-a [triage agent](examples/github-triage.agent.md), and a screenshot walkthrough.
+Try the [Pokemon MCP picnic](examples/pokemon-mcp-demo.md) for a free, no-key tool demo,
+or [GitHub triage](examples/github-tools-demo.md). Both include an agent file,
+copy-paste prompts, and screenshot tips.
+For code, try the [async scheduler challenge](examples/async-scheduler-demo.md) with executable checks.
 
 ## Keep and share results
 
@@ -64,6 +64,7 @@ without rerunning its models.
 ## Know the limits
 
 - Reviews are model judgments, not verified correctness or a model leaderboard.
+- Required phrase checks block recommendations when they fail; passing is not proof of correctness.
 - Live investigations can gather different evidence. Use a shared snapshot to isolate prompt changes.
 - Every live test needs fresh read-only verification, and every tool call needs approval.
   Verification is not a sandbox; use trusted tools and read-only server permissions.