+
# CompareLab
Compare two AI models on the same task, or test whether a prompt edit helps.
@@ -9,19 +11,15 @@ or paste existing responses. No hosted CompareLab service or runtime npm depende
## Quick start
-To run CompareLab from source:
-
-1. Clone this repository and open it in VS Code.
-2. Press **F5** and select **Run CompareLab Extension**.
-3. In the new **Extension Development Host** window, click the **CompareLab**
- Activity Bar icon, then **Open CompareLab**.
-4. Choose **Model comparison** or **Prompt comparison** and follow the steps below.
+1. Install [CompareLab from the VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=azayzel-comparelab.comparelab).
+2. Click the **CompareLab** Activity Bar icon, then **Open CompareLab**.
+3. Choose **Model comparison** or **Prompt comparison** and follow the steps below.
-Requires Node.js 20+ and VS Code 1.90+. Live tools require VS Code 1.95+,
+Requires VS Code 1.90+. Live tools require VS Code 1.95+,
a tool-capable model, and a working tool provider in the same window.
MCP support also depends on your VS Code version and server setup.
-Prefer a local install? See [build and installation](docs/guide.md#build-and-package).
+For source development or a local build, see [build and installation](docs/guide.md#build-and-package).
The **CompareLab: Compare Model Outputs** command remains available in the Command Palette.
## Compare models
@@ -34,7 +32,7 @@ The **CompareLab: Compare Model Outputs** command remains available in the Comma
4. Click **Run comparison**. It runs both sides and reviews their outputs.
Read the reports and cited assessment side by side. The visible run summary shows
-how many pairs and reviews are planned; adjust them under **Advanced settings**.
+how many pairs and reviews are planned; **Advanced settings** also offers simultaneous models and a configurable VS Code timeout (10 minutes by default).
The default is three pairs with reversed-order reviews.
For pasted responses, open each **Paste / edit raw response** editor, then use
@@ -52,8 +50,10 @@ By default, both prompts receive the same evidence without live tools.
The review links the instruction change to quotes from both outputs.
It can recommend keeping the original; a different answer is not automatically better.
-Try the [GitHub demo](examples/github-tools-demo.md) for ready-to-use prompts,
-a [triage agent](examples/github-triage.agent.md), and a screenshot walkthrough.
+Try the [Pokemon MCP picnic](examples/pokemon-mcp-demo.md) for a free, no-key tool demo,
+or [GitHub triage](examples/github-tools-demo.md). Both include an agent file,
+copy-paste prompts, and screenshot tips.
+For code, try the [async scheduler challenge](examples/async-scheduler-demo.md) with executable checks.
## Keep and share results
@@ -64,6 +64,7 @@ without rerunning its models.
## Know the limits
- Reviews are model judgments, not verified correctness or a model leaderboard.
+- Required phrase checks block recommendations when they fail; passing is not proof of correctness.
- Live investigations can gather different evidence. Use a shared snapshot to isolate prompt changes.
- Every live test needs fresh read-only verification, and every tool call needs approval.
Verification is not a sandbox; use trusted tools and read-only server permissions.
diff --git a/docs/guide.md b/docs/guide.md
index a44217a..b51a031 100644
--- a/docs/guide.md
+++ b/docs/guide.md
@@ -116,6 +116,22 @@ Known VS Code-only dictation-cleanup models are excluded from both generation an
**Debug Console reports `TypeError: Missing dataLength in event`?** This matches a Node network-inspector bug that can abort HTTP responses when debugging ([nodejs/node#66551](https://github.com/nodejs/node/issues/66551)). The workspace disables the experimental JavaScript Network view in `.vscode/settings.json`; stop and restart the **Run CompareLab Extension** debug session after pulling this change. If launching from another workspace, set `debug.javascript.enableNetworkView` to `false` there too. This disables debugger network inspection, not tool access or HTTP requests.
+**A model reports no internet connection, or seems to run past its timeout?** Check **Output → GitHub Copilot Chat** for the underlying error and compare its timestamp with system sleep/wake events. `net::ERR_NETWORK_IO_SUSPENDED` can accompany system sleep; it does not by itself prove that your connection or the model is unavailable. A later model can fail while an earlier one finishes before idle sleep begins. Parallel execution does not prevent this.
+
+**Output → CompareLab** identifies the loaded extension path/version and records generation request IDs, model IDs, stages, elapsed time, and configured deadlines. Stages distinguish input token counting, waiting for the provider, streaming, and output token counting. Timeouts identify the stalled stage; cancellation and provider failures are recorded separately. These local diagnostics do not log prompts, responses, tool arguments, API keys, or raw provider errors. Check the runtime path when comparing an installed release with a development checkout; a window title alone does not establish which extension registered the command.
+
+JavaScript deadlines cannot execute while the extension host is suspended, so a ten-minute timeout can be reported later in wall-clock time after waking. The timeout is per generation request or review stage, not for the entire multi-pair comparison.
+
+On macOS, inspect sleep/wake events with `pmset -g log`. To test without idle system sleep, run this in a terminal:
+
+```sh
+caffeinate -i -t 1800
+```
+
+This prevents idle system sleep for 30 minutes without changing permanent power settings; it may use more battery. It does not prevent manual sleep or closing the lid. Cancel the interrupted run and start a fresh test while awake; a suspended provider connection may not recover. Save the original comparison rather than silently replacing failed benchmark attempts with retries.
+
+Do not treat switching Copilot's Electron/Node transport as a fix for system sleep: it may change the error into a stalled request. If an awake test still fails, preserve the provider error, timing, execution settings, and model identity for diagnosis before changing more settings.
+
1. Load or paste the agent instructions and supply the incoming trigger. Keep differing per-side prompt overrides empty for a controlled model test.
2. Choose **End-to-end agent** to let each model independently use selected VS Code tools. **Source evidence / tool results** is optional in this mode. Configure/start MCP servers or enable tool-provider extensions in the same VS Code window/profile first. CompareLab uses `vscode.lm.tools`, not tools, selections, or results inherited from Chat. Tool calling requires VS Code 1.95+ and a tool-capable model; MCP availability also depends on your VS Code version and server setup. Alternatively, choose **Model / prompt only** and supply the same frozen evidence both models will receive.
3. Use **Select model** on each side. Optionally choose Ollama or an OpenAI-compatible endpoint instead; match sampling settings and output budgets for those providers where possible.
@@ -148,7 +164,7 @@ Each newly completed model request retains its input/output token counts, includ
- **Not billed cost:** Copilot subscriptions, hidden/unreported reasoning, caching discounts and cache-write charges, tools, taxes, negotiated discounts, PTUs, Batch, Flex, Priority and other deployment types are not modeled. All input is priced as uncached. Failed/cancelled requests can consume usage that is not retained. Unknown model IDs, deployment aliases, or missing token counts remain unpriced; there is no fuzzy name matching or silent zero-cost fallback.
- **Historical estimates:** Reports recompute estimates using the bundled dated snapshot, not historical invoices. Update that snapshot only after verifying the exact meters, units (1K versus 1M), deployment type and effective dates. Check current prices before making purchasing decisions.
-Other existing safety limits still apply: the shared generation task/evidence request is limited to 150,000 characters, model/evaluator responses to 2 MiB, individual model requests and each paired review to 180 seconds, and the test record to 8 MiB. Direct evaluator reviews retain all supplied evidence and use the selected evaluator's reported input-token limit, not a fixed character cap. Exceeding a limit fails explicitly rather than silently trimming evidence or manufacturing a final answer. Provider rate limits still apply.
+Other safety limits still apply: the shared generation task/evidence request is limited to 150,000 characters, model/evaluator responses to 2 MiB, and the test record to 8 MiB. **Advanced settings -> VS Code timeout (seconds)** defaults to **600** and accepts whole seconds from **30 to 3600**. It applies to each VS Code generation request (including each tool round and prompt-draft generation) and each complete review stage, including reversed-order checks and citation repair. It is a total deadline, not an idle-stream timer. HTTP providers retain their own request behavior. Cancellation remains available; CompareLab releases the UI even if a VS Code provider ignores cancellation, but cannot guarantee the remote service stops processing or billing. Direct evaluator reviews retain all supplied evidence and use the selected evaluator's reported input-token limit, not a fixed character cap. Exceeding a limit fails explicitly rather than silently trimming evidence or manufacturing a final answer. Provider rate limits still apply.
### Selecting VS Code tools
@@ -168,11 +184,11 @@ VS Code models declaring that they cannot call tools are rejected before paired
Live requests may differ in scope, observation time, or side effects. End-to-end results compare **agents**, not isolated model/prompt quality. A task-specific preference can still be shown when majority and evidence checks pass; it is not proof of causality. The Copilot evaluator reviews recorded results without making live calls. Historical Datadog exports remain supported through compatibility-only instruction reconstruction; they do not enable any tools.
-Runs are sequential, alternating baseline-first and candidate-first to reduce resource contention and ordering effects. In frozen-evidence mode, three pairs make six generation requests and six review requests by default, plus at most three citation corrections. End-to-end mode makes as many generation requests and tool calls as needed, with the same review budget. There is no automatic generation retry. Cancellation stops remaining calls and retains a partial record; failed or incomplete pairs never reuse pasted responses. Test records are bounded to 8 MiB; hitting the limit stops further work and labels the unretained result.
+Runs are sequential by default, alternating baseline-first and candidate-first to reduce resource contention and ordering effects. Enable **Advanced settings -> Run models simultaneously** to start both sides of each pair together. Reviews wait for both sides to succeed; pairs and review orders remain sequential. Simultaneous runs can compete for provider quotas or local compute, so their timings are not isolated speed measurements. The choice and any non-default timeout are preserved in saved comparisons. In frozen-evidence mode, three pairs make six generation requests and six review requests by default, plus at most three citation corrections. End-to-end mode makes as many generation requests and tool calls as needed, with the same review budget. There is no automatic generation retry. Cancellation stops remaining calls and retains a partial record; a failed side does not discard its successful peer, and failed or incomplete pairs never reuse pasted responses. Test records are bounded to 8 MiB; hitting the limit stops further work and labels the unretained result.
VS Code generation records the selected model/vendor/version, request messages, wall-clock duration, and model-counted input/output token estimates. Input estimates count each request's messages and tool definitions; output estimates count visible text and tool-call output, not hidden reasoning. Repeated conversation context is counted again on each turn. These are **not billed usage**: actual usage remains null, and estimates are retained separately in history and exports. Providers such as Ollama/OpenAI-compatible endpoints retain actual returned usage when available. Output caps, temperature and effective context size are not exposed as controlled settings for VS Code models. The Language Model API receives the instructions as user context (not a privileged system role) and does not inherit Agent-mode tools or chat history. End-to-end orchestration explicitly supplies the selected VS Code tools and executes requested calls. Exact request roles are retained in the test record.
-Supported preferences require consistent evaluator order checks, grounded substantive criteria, and no contradictory safety/grounding judgment or severe run warning. Default baseline retention is not counted as a win. A strict majority can produce a **task-specific preference** only when every pair completed and recorded configuration warnings do not confound the result. Other outcomes remain inconclusive. Median latency uses pairs where both generations succeeded; missing usage remains unknown, with known subtotals retained. These are repeated samples of **one task**, not statistical significance, a representative benchmark, verified correctness, or a model leaderboard.
+New drafts offer **Best-supported pick with confidence**: a cited substantive advantage can produce a provisional recommendation despite incomplete grounding or an order disagreement. Required checks, failed executable checks, contradictory grounding/actionability findings, severe comparability warnings, and unchanged effective prompts still block adoption. Genuine ties and unsupported votes do not become wins. **Strict** policy retains the previous requirement for consistent, grounded support; older saved drafts retain their policy. A strict majority can produce a **task-specific preference** only when every pair completed and recorded configuration warnings do not confound the result. Median latency uses pairs where both generations succeeded; missing usage remains unknown. These are repeated samples of **one task**, not statistical significance or a model leaderboard. Confidence labels are qualitative, not calibrated probabilities.
Test JSON includes the instructions, trigger, exact requests, frozen evidence, settings, timestamps, responses, measurements, and assessments. Importing a matching test record or exporting a report makes no model calls. Editing its inputs or responses invalidates the attached record. Imported records are user-controlled artifacts, not authenticated execution attestations.
@@ -208,13 +224,13 @@ History begins with new saves after this feature is installed. Save the comparis
Provide the original task, both responses, and a frozen evidence snapshot: logs, metrics, tool results, reference material, or expected outcomes. Open **Other actions → Review existing responses**, then select an available model. Use a current VS Code release with Copilot access; availability depends on sign-in, organization policy, consent, and quota. No separate provider API key is needed for this workflow.
-- The overall result is **Recommend Baseline** or **Recommend Candidate**: a practical recommendation, not proof of superiority. The summary requests confidence, decisive differences, tradeoffs, and evidence that could change the choice. Limited evidence produces a provisional, low-confidence recommendation; ties retain the baseline. Individual criteria can still be **Tie** or **Insufficient evidence**.
+- **Recommend Baseline/Candidate** identifies the best-supported eligible pick. **Provisional pick** and **low confidence** distinguish uncertain rankings from the stricter **Preferred** outcome. Strict policy abstains on inconsistent or insufficient grounding. Acceptance failures still block recommendations under either policy. The original evaluator judgments remain visible; baseline retention is not counted as a win. Grounding disagreements are labeled as disagreements rather than falsely claiming evidence was absent.
- The evaluator receives anonymized A/B response labels, their tasks, and supplied evidence, without provider metadata or tools. Identities mentioned inside response text are not automatically redacted.
- **Check reversed order** is enabled by default and runs **two sequential reviews** with the same selected model. Disagreement produces a provisional recommendation to retain the baseline, not a claim that it is better; both judgments and disputed criteria remain visible. Turning it off runs one review and explicitly records that order was not checked. Invalid citations can trigger **one extra correction request total**, so evaluation uses up to three requests (two without order checking). All requests consume Copilot usage and appear in the run details/report.
-- Citations must match supplied text, allowing only whitespace differences and retaining the original source passage. Wording, numbers, punctuation, and Markdown must match. This validates quotation provenance, not whether the passage supports the judgment. Missing evidence prevents verification of factual correctness, but does not block a provisional recommendation based on the tasks and responses; no model assessment is verified ground truth.
+- Citations must match supplied text, allowing only whitespace differences and retaining the original source passage. Wording, numbers, punctuation, and Markdown must match. This validates quotation provenance, not whether the passage supports the judgment. Missing evidence may still yield an evaluator opinion, but unsupported grounding cannot establish a preference; no model assessment is verified ground truth.
- Direct reviews preserve both complete responses, their tasks, and all supplied evidence. Before each request (including reversed order and citation corrections), CompareLab counts the full messages with the selected model's token counter and checks its reported input-token limit. There is no fixed 150,000-character cap on direct review packets. Missing/invalid token metadata fails explicitly rather than assuming the packet fits.
- If the packet exceeds the evaluator's input limit, the error shows the counted tokens, evaluator ID, and reported limit. Choose a larger-context evaluator; CompareLab does not silently excerpt or summarize evidence to make it fit. Larger reviews can take longer and consume more tokens.
-- Cancel from the panel or progress notification. Reviews still enforce a 180-second evaluation timeout and a 2 MiB response limit per request.
+- Cancel from the panel or progress notification. Reviews use the configured VS Code timeout (600 seconds by default) for the complete review stage and a 2 MiB response limit per request.
**Saved review failed with “Review exceeds the 150,000-character limit”?** Older versions applied that fixed cap even after full investigations began retaining larger evidence sets. Updated direct reviews use the evaluator's token limit instead. Open the saved comparison and click **Review existing responses** to review the displayed response pair without rerunning its investigations; the original historical failed attempts remain unchanged.
@@ -293,9 +309,9 @@ Remote endpoints must use **HTTPS**; HTTP is permitted only for local loopback a
### Generate both without tools
-Configure providers and model IDs on both sides, then open **Other actions → Generate both (no tools)**. This skips tools and evaluation regardless of comparison mode. Runs execute **sequentially**, not concurrently, to reduce resource contention on a shared local GPU. Each successful run updates its response, reported model ID, token usage and finish reason when available. If one side fails, the other is still attempted.
+Configure providers and model IDs on both sides, then open **Other actions → Generate both (no tools)**. This skips tools and evaluation regardless of comparison mode. Runs execute **sequentially** by default to reduce resource contention on a shared local GPU; the **Run models simultaneously** checkbox also applies to this action. Each successful run updates its response, reported model ID, token usage and finish reason when available. If one side fails, the other is still attempted.
-Elapsed duration is recorded in experiment JSON and Markdown reports when available. These sequential wall-clock measurements are not a controlled performance benchmark; missing measurements stay unknown.
+Elapsed duration is recorded in experiment JSON and Markdown reports when available. These wall-clock measurements are not a controlled performance benchmark, especially with concurrent requests competing for resources; missing measurements stay unknown.
## Understanding the results
@@ -337,6 +353,62 @@ Optional top-level `evidence` contains the source snapshot and `review.checkOrde
**Exports can contain sensitive prompts, evidence, and responses.** Check them before committing or sharing. API keys are never included.
+## Acceptance and decision readiness
+
+- **Required phrases** are case-insensitive substring checks on the visible response, excluding recognized provider-envelope metadata. A judged winner that misses a required phrase cannot be recommended. This does not automatically award the other side a win.
+- Repeated tests show pass counts across successful responses. A side that fails a required check in any successful response cannot receive the aggregate recommendation. Errors remain errors, not acceptance passes.
+- Acceptance is recomputed from retained outputs and configured checks when reopening records; saved pass/fail booleans are not trusted. UI, Markdown, and HTML preserve the original evaluator judgment separately from recommendation readiness.
+- Budget-limited investigations and truncated tool evidence block recommendations in individual, reopened, and standalone reviews. Repeated recommendations also require matching output/checker bindings for every pair; a majority cannot hide stale executable observations.
+- Prompt edits that produce identical effective model instructions cannot be labeled **Better** or **Worse**, even if metadata and sampled outputs differ.
+- Latency ratios require valid paired timing for every planned run. Partial timing samples remain visible but are not presented as a complete speed comparison.
+
+Literal checks can be gamed by repeating the required text. They are not semantic tests, code execution, or proof of correctness. Use independent tests and representative inputs before adoption.
+
+## Build a representative test suite
+
+CompareLab runs repeated pairs for **one task at a time**, not dataset batches. Optional isolated Node.js checks can execute single-file answers and gate recommendations. External test observations can be supplied as source evidence, but are not authenticated executable acceptance results. Required phrases remain literal substring checks.
+
+### Resolving inconclusive reviews
+
+Enable **Advanced settings -> Resolve inconclusive reviews** for one follow-up round per reviewed pair. The reviewer identifies disputed criteria, then uses the selected, freshly verified read-only tools to gather relevant observations for both frozen outputs. Each tool call still requires approval. Collection is limited to three model requests, six tool calls, and 128 KiB of retained supplemental evidence; exhausting a limit is an explicit failure, not silently truncated success. A single adjudication follows, with at most one citation repair: up to five extra model requests per pair. The complete review-stage timeout includes checking, initial reviews and resolution.
+
+If no suitable tools are selected, the adjudicator re-examines the supplied contract, outputs and checker observations and explicitly reports that no new tool evidence was obtained. It cannot manufacture facts. Original reviews, new observations, failures, usage and the adjudication remain in the audit details and exports. Later evidence is never retroactively inserted into generation inputs. An adjudicated recommendation does not erase the original order disagreement; confidence remains low when that matters. Prompt adoption still distinguishes a practical pick from an established instruction improvement.
+
+The collection audit retains rejected tool requests and their errors, even when a later corrected request succeeds; only successful observations go to adjudication. Completed collection requests and token usage remain counted if the formatted evidence exceeds the size limit and is rejected.
+
+### Optional isolated executable checks
+
+1. Prepare a trusted, dependency-free CommonJS checker using `node:test`. It receives the generated module path in `process.argv[2]`; see [the scheduler checker](../examples/check-async-scheduler.cjs). Answers must contain exactly one JavaScript/CommonJS fenced code block.
+2. Start a **local** Linux Docker engine (Docker Desktop on macOS/Windows) and obtain the image with `docker pull node:22-alpine`. CompareLab does not start Docker, pull images, or use remote Docker engines automatically.
+3. Use **Advanced settings -> Select executable checker**. The checker is snapshotted into the experiment, including its source; review it before sharing exports. The default per-answer deadline is 30 seconds; imported configurations may specify 5-120 seconds.
+4. Approve execution for each comparison/review. Each answer runs in a separate container using the resolved image ID, with no network, read-only root and input mounts, no workspace/credential/Docker-socket mounts, a non-root user, dropped capabilities, and CPU/memory/process/output limits. Containers and temporary files are removed afterward, including after cancellation or timeouts.
+5. Inspect **Executable check observations**. Both the checker and exact visible answers are hash-bound to results. Exit zero alone is not a pass: the Node test runner must report passing tests. Failed, timed-out, missing or mismatched checks block recommendation of that answer. Passing limited tests is not proof of general correctness.
+
+Completed checker observations survive later evaluator errors, review timeouts, and cancellation without rerunning the answers. This applies to both paired tests and **Review existing responses**: observations remain visible and can be saved to history or exported as JSON, Markdown, or HTML even without a quality assessment. Standalone observations use the optional top-level `verificationResults` field; changed answers or checker source cannot reuse those results.
+
+Docker is optional for everything else. Container isolation is not a guarantee against deliberately hostile code, and code sharing a process with a checker can attempt to interfere with tests. Use a disposable Docker environment for adversarial outputs; review checkers and avoid secrets in them. Results in imported JSON are user-supplied records, not signed attestations. No generated code is ever silently run on the host as a fallback.
+
+Start with 10-20 sanitized tasks from the work you actually do, including typical requests, edge cases, and cases where the correct answer is to report missing information. For example:
+
+| Task family | Independent correctness check |
+| --- | --- |
+| Implement a small API | Unit tests for its contract, including invalid input |
+| Fix a bug | Reproduction test plus existing regression tests |
+| Extract structured data | JSON parsing, schema validation, and exact expected values |
+| Answer from supplied documents | Expected facts and source references; human review where ambiguous |
+| Investigate an incident | Known observations, allowed actions, and explicit unknowns |
+
+1. Define the expected behavior and tests **before** generating responses. Keep several tasks held out while tuning prompts; do not rewrite expectations to favor the latest output.
+2. Save an experiment JSON for each task. Freeze its input, evidence, and acceptance criteria. For a model comparison, change only the model; for a prompt comparison, keep the model/settings fixed and change only the instructions.
+3. Import each task, run the same number of paired repetitions, and save/export every attempt. Keep execution mode, timeout, model versions when available, and evaluator consistent. Use shared evidence rather than live MCP responses to isolate the change being tested.
+4. Validate **each** response independently. For coding, the [scheduler demo](../examples/async-scheduler-demo.md#check-the-actual-code) already has an executable checker with 11 cases. Review generated code first; run untrusted code only in an externally isolated, disposable environment with no secrets and an enforced process timeout. CompareLab does not provide that isolation.
+5. Record a row per task and attempt: baseline pass/fail, candidate pass/fail, failed checks, generation errors/timeouts, latency, and usage. Keep execution failures in the planned-attempt denominator rather than dropping or selectively retrying them. Report task-family results too, so many easy tasks cannot hide a critical regression.
+6. Compare **tasks passing their required checks**, not the number of assertions passed. A task with 100 easy assertions should not outweigh a task with one critical requirement. Use model-judge feedback for qualitative differences, not to override failed correctness checks.
+
+For example, predeclare: "All critical tasks must pass; no regression on the held-out tasks; compare cost and latency only among acceptable variants." The appropriate thresholds depend on your workload. A small repeated suite is a regression signal, not statistical proof of general model superiority.
+
+Until batch execution and external-check ingestion exist, keep this results table alongside the exported comparisons; do not assume CompareLab's **Preferred** label incorporates external tests.
+
## Current capabilities
- Optional cited Copilot evaluation and fresh Copilot Chat draft handoff (unreleased additions).
@@ -344,7 +416,7 @@ Optional top-level `evidence` contains the source snapshot and `review.checkOrde
- Verdict-first reports, readable triage responses, order-bias checking, and missing-evidence abstention.
- Two-variant word-level inline diff (large inputs use a coarse replacement fallback).
- Per-side manual entry or opt-in non-streaming Ollama/OpenAI-compatible model runs.
-- Deterministic required-phrase checks and descriptive comparability warnings.
+- Deterministic required-phrase acceptance gates and descriptive comparability warnings.
- JSON import/export, Markdown reports, standalone CLI and Node built-in tests.
- API keys in VS Code SecretStorage; no account, backend, or hosted telemetry.
diff --git a/examples/async-scheduler-demo.md b/examples/async-scheduler-demo.md
new file mode 100644
index 0000000..8ae8ca4
--- /dev/null
+++ b/examples/async-scheduler-demo.md
@@ -0,0 +1,63 @@
+# Hard coding demo: cancellable async scheduler
+
+Compare two models on a small API with difficult concurrency semantics.
+Unlike the Pokemon demo, this needs **no MCP server**: both models get the
+same precise contract, and you check their code separately.
+
+## Run the comparison
+
+1. Open CompareLab and **Import** [async-scheduler.json](async-scheduler.json).
+2. Choose two models. The preset uses **Model / prompt only**, one paired run,
+ reversed-order reviews, and no tools.
+3. Click **Run comparison** and choose an evaluator.
+
+For slow reasoning models, **Advanced settings** offers a VS Code timeout
+(600 seconds by default, up to 3600) and **Run models simultaneously**.
+Parallel runs reduce waiting between models but may hit shared provider quotas.
+
+The preset embeds [async-scheduler.agent.md](async-scheduler.agent.md) and the
+full contract in **Source evidence / tool results**. Reload the agent if you
+edit it; the embedded copy does not update automatically.
+
+Each model must implement `runTasks(tasks, { concurrency, signal })`: bounded
+concurrency, input-ordered settled results, eager validation, thenable support,
+and cancellation that skips queued work but waits for running tasks.
+
+The traps are deliberate:
+
+- A later invalid task must prevent *all* tasks from starting.
+- A task can abort synchronously before the next slot is filled.
+- A rejected task must free a slot without failing the whole run.
+- Cancellation must not overwrite completed results or abandon active work.
+- A thenable can throw while its `then` property is being read.
+- Serial execution does not satisfy a concurrency limit of two.
+
+## Check the actual code
+
+Review each response before executing it. Save **only its JavaScript code
+block**, without fences, as `baseline.cjs` or `candidate.cjs` outside the
+repository. These are executable third-party programs: neither CompareLab
+nor the checker sandboxes them. Use a disposable environment without secrets,
+private files, or network access for untrusted output.
+
+With Node.js 22+ in that environment:
+
+```sh
+node examples/check-async-scheduler.cjs /absolute/path/to/baseline.cjs
+node examples/check-async-scheduler.cjs /absolute/path/to/candidate.cjs
+```
+
+The checker loads each module and runs 11 cases with Node's built-in test
+runner. Failing tests produce a nonzero exit status. Deferred promises control
+completion order; the per-test timeout is a hang guard, not a speed benchmark.
+It cannot interrupt synchronous infinite loops or malicious code: enforce
+process timeouts and isolation externally.
+
+Capture the code side by side, CompareLab's assessment, and each test result.
+Use test failures as concrete evidence; the model evaluator cannot execute
+these tests and its preference is not proof of correctness.
+
+This is a useful conformance check, not exhaustive certification. Manually
+review linear scheduling complexity, abort-listener cleanup, and prohibited
+APIs as well. Model/evaluator usage may incur charges; the checker needs no
+packages, accounts, or API keys.
diff --git a/examples/async-scheduler.agent.md b/examples/async-scheduler.agent.md
new file mode 100644
index 0000000..1d7ebf3
--- /dev/null
+++ b/examples/async-scheduler.agent.md
@@ -0,0 +1,18 @@
+---
+name: "Async Scheduler Engineer"
+description: "Use when implementing a bounded JavaScript task scheduler against an explicit concurrency and cancellation contract."
+tools: []
+---
+
+Implement the supplied contract precisely in dependency-free CommonJS JavaScript
+for Node.js 22+. Treat the contract as authoritative; do not silently simplify
+validation, cancellation, or failure semantics.
+
+Reason about queue ownership, exactly-once invocation, promise assimilation,
+result ordering, and cancellation races before writing the implementation.
+Do not use tools, execute code, or claim tests passed without execution evidence.
+
+Return exactly one JavaScript code block containing the complete module, followed
+by at most 180 words explaining invariants, complexity, and remaining risks.
+Export only `runTasks` using `module.exports = { runTasks }`. Do not include
+example invocations, dependencies, test code, or a CLI inside the module.
diff --git a/examples/async-scheduler.json b/examples/async-scheduler.json
new file mode 100644
index 0000000..fcd9dff
--- /dev/null
+++ b/examples/async-scheduler.json
@@ -0,0 +1,13 @@
+{
+ "comparisonKind": "model",
+ "comparisonMode": "model-prompt-only",
+ "prompt": "Implement the supplied contract precisely in dependency-free CommonJS JavaScript\nfor Node.js 22+. Treat the contract as authoritative; do not silently simplify\nvalidation, cancellation, or failure semantics.\n\nReason about queue ownership, exactly-once invocation, promise assimilation,\nresult ordering, and cancellation races before writing the implementation.\nDo not use tools, execute code, or claim tests passed without execution evidence.\n\nReturn exactly one JavaScript code block containing the complete module, followed\nby at most 180 words explaining invariants, complexity, and remaining risks.\nExport only `runTasks` using `module.exports = { runTasks }`. Do not include\nexample invocations, dependencies, test code, or a CLI inside the module.",
+ "incomingMessage": "Implement runTasks(tasks, options) according to the shared contract. This is a production-style concurrency problem, not Promise.all with a wrapper. Carefully handle a task that aborts its own signal synchronously, a later invalid task, and promises that finish out of order. Explain why your implementation preserves the required invariants.",
+ "evidence": "AUTHORITATIVE CONTRACT (not execution evidence)\n\nAPI: runTasks(tasks, { concurrency = 2, signal } = {}) returns a Promise. Export it as module.exports = { runTasks } from one .cjs file. No dependencies, network, file access, timers, eval, or code execution APIs.\n\nVALIDATION: tasks must be a dense Array of functions. options, if supplied, must be a non-null, non-array object. concurrency must be a positive safe integer; do not coerce strings. signal, if supplied, must be an AbortSignal from the current realm. Reject invalid input with TypeError through the returned Promise, never a synchronous throw. Validate all input before invoking any task, even with an already-aborted signal or an empty array. Do not mutate tasks or options.\n\nSCHEDULING: Invoke tasks in ascending index order, each at most once, as task(signal). With no signal, pass undefined. Keep at most concurrency unsettled tasks active. Fill available slots without waiting for slower active tasks; a serial implementation is invalid when concurrency > 1. A task can return a plain value, a promise, or a thenable, or throw synchronously. Assimilate thenables using normal Promise semantics, including a throwing then getter. Ordinary failures must not stop other tasks.\n\nRESULTS: Resolve an array in INPUT order with exactly one entry per task: { status: 'fulfilled', value } or { status: 'rejected', reason }. Preserve value and reason identity. Task failures do not reject the outer promise. The outer promise settles only after every started task settles. Empty valid input resolves [].\n\nCANCELLATION: Never start another task after signal.aborted becomes true, including when a running task calls abort() synchronously. Every task not yet started gets { status: 'rejected', reason: signal.reason }. Already-started tasks retain their actual outcomes; pass them the original signal so they may cooperate. Aborting does NOT settle the outer promise early, forcibly cancel active work, or replace a completed result. If a started task ignores abort and never settles, the outer promise remains pending. Already-aborted input invokes no tasks. With no explicit abort reason, use the signal's default AbortError reason unchanged.\n\nUse linear total scheduling work (apart from the tasks themselves), no polling or repeated whole-array scans. If you install abort listeners, remove them when the run finishes.",
+ "toolSelection": [],
+ "baseline": { "provider": "vscode", "model": "select-baseline-in-vscode" },
+ "candidate": { "provider": "vscode", "model": "select-candidate-in-vscode" },
+ "review": { "checkOrder": true, "decisionPolicy": "best-supported", "resolveEvidence": true },
+ "test": { "repetitions": 1 },
+ "checks": { "requiredPhrases": [] }
+}
diff --git a/examples/check-async-scheduler.cjs b/examples/check-async-scheduler.cjs
new file mode 100644
index 0000000..570ec68
--- /dev/null
+++ b/examples/check-async-scheduler.cjs
@@ -0,0 +1,196 @@
+'use strict';
+const assert = require('node:assert/strict');
+const { test } = require('node:test');
+const path = require('node:path');
+
+if (process.argv.length !== 3) {
+ console.error('Usage: node examples/check-async-scheduler.cjs /absolute/path/to/solution.cjs');
+ process.exit(1);
+}
+const { runTasks } = require(path.resolve(process.argv[2]));
+assert.equal(typeof runTasks, 'function', 'Export module.exports = { runTasks }');
+const check = (name, fn) => test(name, { timeout: 2000 }, fn);
+const fulfilled = value => ({ status: 'fulfilled', value });
+const rejected = reason => ({ status: 'rejected', reason });
+const tick = () => new Promise(resolve => setImmediate(resolve));
+function deferred() {
+ let resolve, reject;
+ const promise = new Promise((yes, no) => { resolve = yes; reject = no; });
+ return { promise, resolve, reject };
+}
+
+check('validation rejects through a Promise before starting ANY task', async () => {
+ let calls = 0;
+ const task = () => { calls++; };
+ const aborted = new AbortController();
+ aborted.abort();
+ const inputs = [
+ [null], [{}], [[task, null]], [[task, , task]],
+ [[task], null], [[task], []], [[task], 'options'],
+ ...[0, -1, 1.5, NaN, Infinity, '2', Number.MAX_SAFE_INTEGER + 1]
+ .map(concurrency => [[task], { concurrency }]),
+ [[task], { signal: {} }], [[], { concurrency: 0 }],
+ [[task, null], { signal: aborted.signal }]
+ ];
+ for (const args of inputs) {
+ let result;
+ assert.doesNotThrow(() => { result = runTasks(...args); });
+ assert.ok(result instanceof Promise, 'Invalid input must still return a Promise');
+ await assert.rejects(result, TypeError);
+ }
+ assert.equal(calls, 0);
+});
+
+check('empty input resolves with no results', async () => {
+ assert.deepEqual(await runTasks([]), []);
+});
+
+check('fills two slots by default and refills a free slot out of order', async () => {
+ const gates = Array.from({ length: 4 }, deferred);
+ const started = [];
+ let active = 0, peak = 0;
+ const pending = runTasks(gates.map((gate, index) => async signal => {
+ assert.equal(signal, undefined);
+ started.push(index);
+ peak = Math.max(peak, ++active);
+ try { return await gate.promise; } finally { active--; }
+ }));
+ await tick();
+ assert.deepEqual(started, [0, 1]);
+ gates[1].resolve('one');
+ await tick();
+ assert.deepEqual(started, [0, 1, 2], 'Task 2 must start before task 0 finishes');
+ gates[2].resolve('two');
+ await tick();
+ assert.deepEqual(started, [0, 1, 2, 3]);
+ gates[3].resolve('three');
+ gates[0].resolve('zero');
+ assert.deepEqual(await pending, ['zero', 'one', 'two', 'three'].map(fulfilled));
+ assert.equal(peak, 2);
+});
+
+check('honors explicit concurrency one and above the roster length', async () => {
+ for (const concurrency of [1, 10]) {
+ const gates = Array.from({ length: 3 }, deferred);
+ const started = [];
+ const pending = runTasks(gates.map((gate, i) => () => {
+ started.push(i);
+ return gate.promise;
+ }), { concurrency });
+ await tick();
+ assert.deepEqual(started, concurrency === 1 ? [0] : [0, 1, 2]);
+ for (let i = 0; i < gates.length; i++) {
+ gates[i].resolve(i);
+ await tick();
+ if (concurrency === 1) assert.deepEqual(started, [0, 1, 2].slice(0, i + 2));
+ }
+ assert.deepEqual(await pending, [0, 1, 2].map(fulfilled));
+ }
+});
+
+check('handles throws, rejections, values, and hostile thenables without stopping', async () => {
+ const reason = new Error('task failure');
+ const value = { ok: true };
+ const badThen = Object.defineProperty({}, 'then', { get() { throw reason; } });
+ const result = await runTasks([
+ () => { throw reason; },
+ () => Promise.reject(reason),
+ () => value,
+ () => ({ then(resolve, reject) { resolve('first'); reject(reason); resolve('last'); } }),
+ () => badThen,
+ () => undefined
+ ], { concurrency: 2 });
+ assert.deepEqual(result, [rejected(reason), rejected(reason), fulfilled(value),
+ fulfilled('first'), rejected(reason), fulfilled(undefined)]);
+ assert.equal(result[0].reason, reason);
+ assert.equal(result[2].value, value);
+});
+
+check('pre-aborted input skips all tasks and preserves the exact reason', async () => {
+ for (const reason of [undefined, { code: 'cancelled' }]) {
+ const controller = new AbortController();
+ controller.abort(reason);
+ let calls = 0;
+ const result = await runTasks([() => calls++, () => calls++], { signal: controller.signal });
+ assert.equal(calls, 0);
+ assert.deepEqual(result, [rejected(controller.signal.reason), rejected(controller.signal.reason)]);
+ assert.equal(result[0].reason, controller.signal.reason);
+ if (reason === undefined) assert.equal(result[0].reason.name, 'AbortError');
+ assert.deepEqual(await runTasks([], { signal: controller.signal }), []);
+ }
+});
+
+check('synchronous abort inside a task prevents even the next initial launch', async () => {
+ const controller = new AbortController();
+ const reason = { stop: true };
+ let skippedCalls = 0;
+ const value = { completed: true };
+ const result = await runTasks([
+ signal => { assert.equal(signal, controller.signal); controller.abort(reason); return value; },
+ () => skippedCalls++,
+ () => skippedCalls++
+ ], { concurrency: 3, signal: controller.signal });
+ assert.equal(skippedCalls, 0);
+ assert.deepEqual(result, [fulfilled(value), rejected(reason), rejected(reason)]);
+});
+
+check('abort waits for active work and keeps its actual success and failure', async () => {
+ const controller = new AbortController();
+ const first = deferred(), second = deferred();
+ const reason = new Error('cancel queued work');
+ const taskFailure = new Error('active task failed');
+ const started = [];
+ const pending = runTasks([first, second, deferred()].map((gate, i) => signal => {
+ assert.equal(signal, controller.signal);
+ started.push(i);
+ return gate.promise;
+ }), { concurrency: 2, signal: controller.signal });
+ let settled = false;
+ pending.then(() => { settled = true; }, () => { settled = true; });
+ await tick();
+ assert.deepEqual(started, [0, 1]);
+ controller.abort(reason);
+ await tick();
+ assert.equal(settled, false);
+ first.resolve('finished despite abort');
+ await tick();
+ assert.equal(settled, false);
+ second.reject(taskFailure);
+ assert.deepEqual(await pending, [fulfilled('finished despite abort'),
+ rejected(taskFailure), rejected(reason)]);
+ assert.deepEqual(started, [0, 1]);
+});
+
+check('abort after a completed task preserves its result and skips queued work', async () => {
+ const controller = new AbortController();
+ const gate = deferred();
+ let skippedCalls = 0;
+ const pending = runTasks([() => 'done', () => gate.promise, () => skippedCalls++],
+ { concurrency: 1, signal: controller.signal });
+ await tick();
+ controller.abort('stop');
+ gate.resolve('also done');
+ assert.deepEqual(await pending, [fulfilled('done'), fulfilled('also done'), rejected('stop')]);
+ assert.equal(skippedCalls, 0);
+});
+
+check('does not mutate frozen input and supports repeated independent runs', async () => {
+ const tasks = Object.freeze([() => 1, () => 2]);
+ const options = Object.freeze({ concurrency: 1 });
+ const result = await Promise.all([runTasks(tasks, options), runTasks(tasks, options)]);
+ assert.deepEqual(result, [[fulfilled(1), fulfilled(2)], [fulfilled(1), fulfilled(2)]]);
+});
+
+check('large synchronous batches invoke each task once without stack overflow', async () => {
+ const size = 20000;
+ const calls = new Uint8Array(size);
+ const result = await runTasks(Array.from({ length: size }, (_, i) => () => {
+ calls[i]++;
+ return i;
+ }), { concurrency: 4 });
+ assert.equal(result.length, size);
+ for (let i = 0; i < size; i++) {
+ assert.equal(calls[i], 1);
+ assert.deepEqual(result[i], fulfilled(i));
+ }
+});
diff --git a/examples/pokemon-mcp-demo.md b/examples/pokemon-mcp-demo.md
new file mode 100644
index 0000000..206c5fa
--- /dev/null
+++ b/examples/pokemon-mcp-demo.md
@@ -0,0 +1,142 @@
+# Pokemon picnic: a no-key MCP comparison
+
+Give two models the same four Pokemon and ask them to draft a tiny picnic
+team. They must call a real MCP tool, add the returned stats, and obey a
+budget instead of guessing from familiar characters.
+
+## Free server, no installation
+
+Use the [Pipeworx Pokemon MCP endpoint](https://github.com/pipeworx-io/mcp-pokemon):
+
+```text
+https://gateway.pipeworx.io/pokemon/mcp
+```
+
+No account, API key, npm package, or local server is needed for this small
+demo. The provider currently offers **50 anonymous calls per day**; a live
+lookup confirmed that limit on 2026-10-31 UTC. Availability, quotas, and terms
+can change. **Model and evaluator usage is separate and may cost money.**
+
+1. In VS Code, run **MCP: Add Server**, choose HTTP, enter the URL above, and
+ name the server `pokemon`. Alternatively, merge this entry into your
+ workspace's `.vscode/mcp.json`; do not replace existing servers:
+
+ ```json
+ {
+ "servers": {
+ "pokemon": {
+ "type": "http",
+ "url": "https://gateway.pipeworx.io/pokemon/mcp"
+ }
+ }
+ }
+ ```
+
+2. Review the third-party server before trusting it. Start it through
+ **MCP: List Servers**. Use a current VS Code release with MCP support.
+3. Open CompareLab in that same VS Code window/profile.
+
+If the server says Running but CompareLab cannot see its tools, use
+**MCP: List Servers → Show locally configured servers… → pokemon → Start Server**,
+then reopen **Select tools**. Some VS Code versions have separate Copilot
+agent-host and local MCP runtimes; CompareLab uses the local language-model tools.
+
+The endpoint also advertises unrelated gateway tools, including memory and
+routing operations. **Select only `get_pokemon`**, not the whole server.
+Its input is `{"name":"pikachu"}` and its output includes `types` and `stats`.
+The advertised read-only annotation is not a sandbox or a safety guarantee.
+Send only public Pokemon names, never private code, credentials, or logs.
+
+## Two models, one challenge
+
+### Import the ready-made setup
+
+This workspace includes the [Pokemon server configuration](../.vscode/mcp.json).
+Use **MCP: List Servers** to start `pokemon` and review any VS Code trust prompt.
+Then open CompareLab and use **Import** to load
+[pokemon-picnic.json](pokemon-picnic.json).
+
+The preset includes the agent instructions, task, end-to-end mode, one paired
+run, and reversed-order reviews. Choose both models with **Select model**,
+then select **only `get_pokemon`** in **Select tools** and run the comparison.
+Model names are deliberate placeholders: available models and prefixed tool
+names depend on your VS Code profile. No permissions or approvals are saved.
+The preset embeds the current agent text; after editing the agent, reload it
+with **Load agent file** or update the preset before using it again.
+
+### Or configure it manually
+
+1. Choose **Model comparison** and load
+ [pokemon-picnic.agent.md](pokemon-picnic.agent.md) with **Load agent file**.
+2. Select two tool-capable VS Code models and **End-to-end agent** mode.
+ Leave source evidence empty and clear any saved prompt overrides.
+3. In **Select tools**, select only the Pokemon server's `get_pokemon` tool.
+ VS Code may prefix its name; use the actual picker entry.
+4. Set **Advanced settings -> Paired runs** to **1** for a small first test.
+5. Paste this into **Incoming message / trigger**:
+
+ ```text
+ Draft a two-Pokemon team for a lakeside picnic.
+
+ Candidates: pikachu, eevee, bulbasaur, squirtle.
+ Retrieve all four with the selected Pokemon MCP tool.
+
+ Rules:
+ - Choose exactly two distinct candidates.
+ - At least one must have "water" in its returned types.
+ - Their combined base-stat total must be at most 637.
+ - Among valid pairs, maximize that combined total.
+
+ Give the pair a playful nickname. Show your arithmetic and explain why
+ each other Water-containing pair loses or breaks the budget.
+ Use only returned base stats and types, not battle knowledge.
+ ```
+
+6. Click **Run comparison**, choose an available evaluator, and freshly verify
+ only the selected tool as read-only. Inspect and approve each lookup.
+ Expect four successful lookups per model if it follows the prompt; this
+ is an instruction, not an enforced call budget.
+
+The agent file's `tools: []` intentionally grants nothing if opened in Chat.
+CompareLab strips its frontmatter and loads its instructions; the tool picker
+and per-call confirmations are what authorize access.
+
+## What makes a good screenshot?
+
+Capture one approval with the Pokemon name visible, then the side-by-side
+roster tables and winning teams. Include **Usage and tool details** to show
+that the models actually made MCP calls.
+
+Check tool use, evidence coverage, arithmetic, and constraint-following.
+Do not pick a winner just because its nickname or prose is more entertaining.
+
+${escape(value?.model || result[side].model || 'Unspecified model')}
${winner === side ? preferred : ''} + const response = (side, value, status, winner, error, badge = preferredBadge(decision)) => `${escape(value?.model || result[side].model || 'Unspecified model')}
${winner === side ? badge : ''} ${escape(status)}${label(side)}${result.comparisonKind === 'prompt' ? ' prompt' : ''}
${tokens(value)}${value?.response ? formatted(value.response) : `
${value?.response ? details('Raw response', raw(value.response)) : ''}
${value?.toolEvidence ? details('Tool requests and full returned results', raw(JSON.stringify(value.toolEvidence, null, 2))) : ''}
${value?.investigationBudget?.limited ? `${escape(error || 'No response available.')}
`}Historical budget-limited investigation: ${escape(value.investigationBudget.reasons.join(' '))}
` : ''}${escape(result[side].model || 'Unspecified model')}
${winner === side ? preferred : ''}${escape(result[side].model || 'Unspecified model')}
${winner === side ? preferredBadge( + test ? { preference: winner } : decision, true) : ''}${label(side)}${result.comparisonKind === 'prompt' ? ' prompt' : ''}
${metric(test ? 'Responses' : 'Response', test ? metrics.succeeded + ' / ' + test.plan.repetitions
: result[side].response ? 'Available' : 'Missing')}
+ ${test && summary?.requiredCheckCount ? metric('Responses passing required checks',
+ summary.acceptance[side].passed + ' / ' + summary.acceptance[side].total, 'Across successful responses; failures block recommendation.')
+ : !test && result.checks?.length ? metric('Required checks', result.checks.filter(check => check[side]).length + ' / ' + result.checks.length) : ''}
${metric(test ? 'Median paired time' : 'Elapsed time', Number.isFinite(latency) ? (latency / 1000).toFixed(1) + ' s' : 'Unknown')}
${tokens(metrics, test ? 'Total tokens in / out' : 'Tokens in / out')}
${test ? tokens(metrics.tokensPerRun, 'Avg tokens in / out per run', 'Across ' + metrics.succeeded + ' successful runs; ~ means estimated') +
@@ -100,15 +116,21 @@ function asHtml(result, { css, markdownScript }) {
`;
const warnings = [...(summary?.warnings || []), ...(result.fairness?.warnings || []).map(item => item.message)];
const prompt = result.comparisonKind === 'prompt' ? promptDecision(result) : null;
const promptReport = prompt ? `Preferred reflects the quality review, not lower token usage, cost, or latency. Tokens accumulate across turns as evidence is sent again.
${summary.efficiency.map(item => metric(item.label, item.value, item.hint)).join('')}
${details('Evaluator usage (separate from investigations)', usage(summary.evaluatorUsage))}` : '';
- const runs = test ? test.pairs.map(pair => `
+ const runs = test ? test.pairs.map(pair => {
+ const decision = pairDecision(test.plan, pair);
+ return `
- `).join('')
- : `
+ Run ${pair.number}
${escape(pair.evaluationStatus)}Generation order: ${escape(pair.order.join(' then '))}
+${test.plan.parallel ? 'Simultaneous generation (launch order)' : 'Generation order'}: ${escape(pair.order.join(' then '))}
${['baseline', 'candidate'].map(side => response(side, pair[side].result, pair[side].status,
- pair.evaluationStatus === 'succeeded' ? pair.evaluation?.verdict : null, pair[side].error)).join('')}
- ${assessment(pair.evaluation)}${pair.evaluationError ? `
${escape(pair.evaluationError)}
` : ''}Quality review
${assessment(result.evaluation)}Model reports
+ decision.preference, pair[side].error, preferredBadge(decision))).join('')}${assessment(pair.evaluation, decision, pair.verification)}
+ ${decision.checks.map(check => `
${escape(check.phrase)}: baseline ${check.baseline ? 'Pass' : 'Fail'}, candidate ${check.candidate ? 'Pass' : 'Fail'}
`).join('')} + ${pair.evaluationError ? `${escape(pair.evaluationError)}
` : ''}Quality review
${assessment(result.evaluation, decision, result.verificationResults)}Model reports
${['baseline', 'candidate'].map(side =>
- response(side, result[side], result[side].response ? 'Received' : 'Missing', winner)).join('')}
PROMPT EDIT REVIEW
@@ -125,7 +147,7 @@ function asHtml(result, { css, markdownScript }) { ${details('Output changes', changesOnly(result.spans, result.outputPassages, 'output'))}` : ''; const data = JSON.stringify(markdown).replaceAll('<', '\\u003c').replaceAll('\u2028', '\\u2028').replaceAll('\u2029', '\\u2029'); return ` - +⇄ CompareLab
${kind} · Offline assessment report
+ ${logoPng ? `.toString('base64')})
` : ''}
+
${escape(test?.status || (result.evaluation ? 'Reviewed' : 'Not evaluated'))}` : ''}
+ CompareLab
${kind} · Offline assessment report
${kind.toUpperCase()}
${title}
${test ? escape(summary?.note) : 'A practical, task-specific comparison. Recommendations and cited evidence still require human verification.'}
diff --git a/src/tools/agent.js b/src/tools/agent.js index 20badf2..7335c54 100644 --- a/src/tools/agent.js +++ b/src/tools/agent.js @@ -84,7 +84,12 @@ async function verifyReadOnlyTools(vscode, tools) { return verified; } -async function runToolAgent(vscode, input, tools, runTurn, token, progress = () => {}) { +async function runToolAgent(vscode, input, tools, runTurn, token, progress = () => {}, limits) { + if (limits && (!Number.isSafeInteger(limits.modelRequests) || limits.modelRequests < 1 || + !Number.isSafeInteger(limits.toolCalls) || limits.toolCalls < 1 || + limits.evidenceBytes !== undefined && (!Number.isSafeInteger(limits.evidenceBytes) || limits.evidenceBytes < 1))) { + throw new Error('Invalid evidence-resolution limits.'); + } const verified = readOnlyCatalogs.get(tools); if (!verified || !isDeepStrictEqual(verified, toolCatalog(tools))) { throw new Error('Tools must be freshly verified read-only for this test. Saved selections and per-call approval cannot bypass this requirement.'); @@ -104,9 +109,23 @@ async function runToolAgent(vscode, input, tools, runTurn, token, progress = () }; const catalog = validateToolCatalog(toolCatalog(tools)); const allowed = new Map(catalog.map(tool => [tool.name, tool])); + const policy = limits ? AGENT_PROMPT.replace('CompareLab imposes no tool-call, model-turn, investigation-time, or tool-result truncation budget.', + 'This evidence-resolution round allows at most ' + limits.modelRequests + ' model requests and ' + limits.toolCalls + + ' tool calls. Collect the most relevant observations and finish within that budget; missing evidence must remain explicit.') : AGENT_PROMPT; const conversation = [...(input.systemPrompt ? [{ role: 'system', content: input.systemPrompt }] : []), - { role: 'user', content: input.prompt }, { role: 'user', content: AGENT_PROMPT }]; + { role: 'user', content: input.prompt }, { role: 'user', content: policy }]; const toolEvidence = []; + const exceedsEvidenceLimit = entry => limits?.evidenceBytes && + Buffer.byteLength(JSON.stringify([...toolEvidence, entry])) > limits.evidenceBytes; + const retainFailure = entry => { + if (!exceedsEvidenceLimit(entry)) { + toolEvidence.push(entry); + return; + } + const marker = { tool: String(entry.tool || '').slice(0, 200), collectedAt: entry.collectedAt, isError: true, + content: 'Tool failure exceeded the evidence-resolution byte limit; details were not retained.' }; + if (!exceedsEvidenceLimit(marker)) toolEvidence.push(marker); + }; const usageTurns = []; const started = Date.now(); const totals = { inputTokens: 0, outputTokens: 0, estimatedInputTokens: 0, estimatedOutputTokens: 0 }; @@ -114,6 +133,7 @@ async function runToolAgent(vscode, input, tools, runTurn, token, progress = () try { while (true) { check(); + if (limits && modelRequests >= limits.modelRequests) throw new Error('Evidence resolution exhausted its model-request budget.'); progress('Tool-enabled run: model request ' + (++modelRequests) + ' | Tool calls: ' + toolEvidence.length); const result = await runCancellable(() => runTurn(conversation, catalog, cancellation.token)); check(); @@ -141,6 +161,7 @@ async function runToolAgent(vscode, input, tools, runTurn, token, progress = () conversation.push({ role: 'assistant', content: result.response || '', toolCalls: calls }); for (const call of calls) { check(); + if (limits && toolEvidence.length >= limits.toolCalls) throw new Error('Evidence resolution exhausted its tool-call budget.'); const approval = await runCancellable(() => vscode.window.showWarningMessage('CompareLab: review this tool request?', { modal: true, detail: (input.runLabel || 'Tool-enabled run') + ' | Model: ' + (input.model || '(selected model)') + '\n\n' + call.name + '\n' + JSON.stringify(call.input, null, 2) + @@ -172,9 +193,9 @@ async function runToolAgent(vscode, input, tools, runTurn, token, progress = () } catch (error) { const repeated = previousRejections.some(entry => entry.tool === call.name && isDeepStrictEqual(entry.input, call.input)); if (cancellation.token.isCancellationRequested || error instanceof vscode.CancellationError || !isInvalidArguments(error) || repeated) { - toolEvidence.push({ tool: call.name, input: call.input, collectedAt: new Date().toISOString(), - content: String(error.message || error || 'Tool invocation interrupted.') + '\nThe invocation failed or was interrupted; side effects may have occurred.', - isError: true }); + retainFailure({ tool: call.name, input: call.input, collectedAt: new Date().toISOString(), + content: String(error.message || error || 'Tool invocation interrupted.') + + '\nThe invocation failed or was interrupted; side effects may have occurred.', isError: true }); check(); if (!repeated || !isInvalidArguments(error)) throw error; throw new Error('Tool ' + call.name + ' rejected the same arguments again after correction feedback: ' + error.message); @@ -185,6 +206,11 @@ async function runToolAgent(vscode, input, tools, runTurn, token, progress = () } const entry = { tool: call.name, input: call.input, collectedAt: new Date().toISOString(), content, ...(isError ? { isError: true, inputSchema: allowed.get(call.name).inputSchema } : {}) }; + if (exceedsEvidenceLimit(entry)) { + retainFailure({ tool: call.name, collectedAt: entry.collectedAt, isError: true, + content: 'Tool result exceeded the evidence-resolution byte limit and was not retained or sent to the model.' }); + throw new Error('Evidence resolution exceeded its retained-evidence limit. No oversized result was used as evidence.'); + } toolEvidence.push(entry); conversation.push({ role: 'tool', toolCallId: call.callId, content: JSON.stringify(entry) }); } diff --git a/src/webview/panel.js b/src/webview/panel.js index 216c12a..2a59504 100644 --- a/src/webview/panel.js +++ b/src/webview/panel.js @@ -52,10 +52,11 @@ function panelHtml(vscode, panel, context) { const media = (...parts) => panel.webview.asWebviewUri(vscode.Uri.joinPath(context.extensionUri, 'media', ...parts)); return ` - +⇄ CompareLab
Visual, context-aware model comparisons.
}){kind=icon}
+ CompareLab
Compare models. Test prompt changes.