Cross-adapter conformance suite, and the bugs it found

[vishnuv688]

Refs #300

Every adapter is supposed to tell the dashboard the same story about the same test, and nothing checked that. Each adapter's own tests consider its own output correct, so a capture fix that lands in one adapter and not another, or a field one adapter formats differently, passes CI and surfaces after release. This adds a suite that runs one scenario through each adapter and judges what it actually sends, two ways.

How it works

• A fake W3C WebDriver (packages/conformance/src/fake-webdriver.ts) models two pages and records ground truth: which documents were born and when, which navigations failed and with what message, and when it completed each navigation, click and keystroke. A minimal BiDi endpoint lets a registered collector preload put the collector in every later document from birth, as in Chrome; it recognises the collector by its own source, not by any adapter's way of injecting it, and an unrelated preload registers nothing.
• A recording backend stands in for the dashboard, records every worker-socket frame, and serves the real collector bundle as the real backend does.
• The harness runs each adapter's scenario as its own process, reached through the existing rerun reuse handshake, so no adapter gains a test-only code path. Adapters are registered once in src/adapters.ts; one test runs every registered adapter with and without BiDi. A scenario that hangs is killed and reported with the frames recorded up to then.
• The rules (src/checker.ts, 12 of them, each traced to a bug the repo has hit): a row ends no earlier than the browser completed its action, one DOM anchor per document carrying its exact birth time, every navigate, click or fill row replays the document the browser was on when that action completed, a navigation row names the page the browser loaded, one row per browser action, a failed command carries the browser's own message, replacements target a sent row, advertised run controls have a command, and so on. Replacements are applied with the same rule the dashboard uses.
• Cross-adapter agreement (src/normalize.ts, src/agreement.ts): within each mode, every run is reduced to what a dashboard user sees (each step's shared action name, target, typed text, outcome and the page it replays, the pages snapshotted, the run controls) and compared with the others. Any difference fails as an adapters-agree-on-* rule, with no per-field rule needed. The rules catch every adapter being wrong in the same way; the comparison catches a difference no rule anticipated.
• Known gaps live in an allowlist with a mandatory reason. An entry that stops matching fails the run, and a gap excuses a cross-adapter difference only where its holder's own run shows the gap on that same step or document and the other adapter's run is clean there, so one adapter's limitation cannot hide a second bug in the other. One gap exists: the Selenium JS adapter without BiDi, whose documented <script> fallback cannot anchor a page left within tens of ms.
• A report: every run writes its stream and verdict to packages/conformance/out/, plus an HTML page with an agreement table per mode (a column per adapter, a row per step, differing rows highlighted) and, per run, a timeline with each violation on the event it concerns and the adapter responsibility the broken rule tests, linked to the guide. CI uploads it when the job fails.

Covers selenium-devtools and selenium-devtools-py; the WDIO service and Nightwatch have no scenario yet. Console, network and assertion capture are not checked yet.

Bugs it found, fixed here

Adapter	Bug
selenium-devtools	A plain script that never called DevTools.startTest captured nothing: only a start call created the TestManager.
selenium-devtools	A plain script never exited: the open worker socket stopped beforeExit, where teardown runs, from firing. This includes every rerun of one.
selenium-devtools-py	A failed command's row carried str(WebDriverException), so the Errors tab showed Message: and the driver's stacktrace around the browser's text. It now carries msg, as the JS adapters do.

Also in this PR

• replaceCommand (find a row by id, else the last row with the old timestamp) moves into shared; the app, the backend and the checker use one copy.
• RunCapabilities and RunnerOptions move into shared, since they describe Metadata.options on the wire.
• docs/ADAPTERS.md: a guide to writing an adapter in another language, written around the suite. ARCHITECTURE.md gains a section on the suite and corrects its claim that the Python adapter writes no trace archive.
• CI's build job installs the Python adapter before pnpm test, so both languages run there. The adapter tests skip locally when a prerequisite is missing, and fail rather than skip in CI.
• The lockfile merge with main consolidates some peer-dependency contexts onto webdriverio@9.32.0, which was already in the lockfile; no installed version changes.

Review feedback

The two latest comments are addressed: a navigation row must again name the page the browser loaded (its own rule, beside the generalised page rule), and a row without command, args[] and timestamp is a clear checker error that keeps the frames and leaves every other run's report intact.

The three comments before those are addressed too: a fill's typed text is compared across adapters; a gap is scoped to the step or document it occurs on, with the page rule generalised to every action row so a fill stamped after a later navigation is a finding in its own right; and a checker that cannot read a stream keeps the recorded run and records its error beside it.

The first seven review comments are addressed in earlier commits: rows are held to the browser's recorded completion times (not only startTime <= timestamp), anchors must match the exact birth time, an empty preload no longer counts as the collector, a second injection starts an unanchored collector so a duplicate anchor can be modelled, the report is written once after all runs, a hung scenario keeps its frames, and the guide now states that replacements match by id first.

Verification

• pnpm build, pnpm test (2564 tests), pnpm lint, pnpm typecheck: green. Python unit tests (709): green.
• Each rule was checked by breaking real adapter code and confirming the suite fails with that rule, then restoring it: removing Python's forced anchor in the drain script, stamping JS rows at invocation, removing the JS launchCommand getter, and storing Python's str() error. The cross-adapter comparison was checked the same way with the str() error, which it flags with no error-specific rule.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no blocking issue remains.

Summary

Adds a shared conformance suite that compares JavaScript and Python adapter output against browser ground truth and against each other.

• The JavaScript and Python adapters now face the same browser scenario and shared checks.
• Plain JavaScript scripts now capture commands and Python rows keep the browser’s error message.
• The app and backend now use one shared rule for replacing command rows.
• The lockfile consolidates repeated peer-dependency contexts.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Scenario[Adapter scenario] --> Browser[Fake WebDriver]
  Scenario --> Backend[Recording backend]
  Browser --> Truth[Browser ground truth]
  Backend --> Frames[Recorded frames]
  Truth --> Checker[Shared rules]
  Frames --> Checker
  Frames --> Agreement[Compare adapters]
  Checker --> Report[JSON and HTML report]
  Agreement --> Report

Loading

_{Reviews (4) · Last reviewed commit: "test(conformance): refuse a row without ..." · Reviewed by Greptile}