Cross-adapter conformance suite, and the bugs it found

.changeset/selenium-plain-script-capture.md

@@ -0,0 +1,5 @@
+---
+'@wdio/selenium-devtools': patch
+---
+
+A plain `node script.js` run with no test runner is now captured without calling `DevTools.startTest`: its commands land under the "Selenium Session" row the README describes, where before every command was dropped. Such a script also exits once its work is done, where before the open dashboard connection kept the process alive indefinitely, which also left a rerun of a plain script running after it finished.

.github/workflows/ci.yml

@@ -43,8 +43,22 @@ jobs:
       # because it is fast and its failures are cheaper to read.
       - name: 🔎 Typecheck
         run: pnpm typecheck
+      # The conformance suite runs the Python adapter beside the JS ones, so a
+      # capture fix missing from one language fails this job.
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: '3.13'
+      - name: Install the Python adapter
+        run: pip install -e packages/selenium-devtools-py
       - name: 🧪 Run Tests
         run: pnpm test
+      - name: Upload conformance report
+        if: failure()
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
+        with:
+          name: conformance-report
+          path: packages/conformance/out/
+          if-no-files-found: ignore
 
   # Separate job because these are the only tests that need a real browser, and
   # they parallelise with `build` instead of extending it. No bundle step: the

.gitignore

@@ -54,6 +54,8 @@ examples/**/trace-*/
 
 # test results
 examples/**/test-results*/
+packages/conformance/**/test-results*/
+packages/conformance/out/
 
 # vitest --coverage output
 coverage/

ARCHITECTURE.md

@@ -135,7 +135,7 @@ Contains:
 - `pytest_plugin.py` — the suite tree, keyed by pytest nodeid; auto-discovered, opt-in by env var.
 - `rerun.py`, `run_id.py`, `performance.py`, `transport.py` — run controls, run identity, navigation timings, and a stdlib-only WebSocket client.
 
-Live mode only: it does not write a trace archive. That work, and the question of whether the ~3,100 lines of trace transforms in `core` should move behind the wire rather than be reimplemented per language, is tracked in [#330](https://github.com/webdriverio/devtools/issues/330) and [#278](https://github.com/webdriverio/devtools/issues/278).
+It writes trace archives without running the transforms itself: `trace_export.py` streams the action snapshots and filmstrip frames, then sends a `traceExport` request and the backend builds the archive from the run it accumulated, answering with `traceExported`. `export_trace()` and teardown (`disable()`, the pytest plugin's session finish) trigger it. Writing an adapter in another language is covered in [docs/ADAPTERS.md](./docs/ADAPTERS.md).
 
 Imports from: `selenium` (required, 4.44+). Does not import: `core`, `shared`, any other package.
 
@@ -188,6 +188,18 @@ Imports from: `core`. The script bodies, serializers, types, and locator generat
 
 The DOM-walking scripts run in the page via `browser.execute`, so — like `script` — they avoid Node-only APIs.
 
+### `packages/conformance`: cross-adapter conformance suite
+
+Test-only and never published or bundled. Each adapter runs one scenario (open `/login`, type, submit to `/secure`, log out, then a navigation that fails) as its own process, against a fake W3C WebDriver server that models two pages and records what really happened in the browser: which documents were born and when, and which navigations failed. The adapter reports into a recording backend that stands in for the dashboard, reached through the same reuse handshake a rerun uses, so no adapter gains a code path for being tested. `src/checker.ts` then judges the recorded frame stream against behavioural rules, for example "a row replays the page the browser was on when its action completed" and "every document gets exactly one anchor at its birth time". Every adapter is checked by the same rules, so a capture fix that lands in one language and not another fails CI.
+
+Each scenario runs twice, with and without BiDi. The fake answers the BiDi commands the adapters send, and a registered preload puts the collector into every later document from birth, as in Chrome. A difference an adapter is allowed to show lives in `src/allowlist.ts` with a mandatory reason, and an entry that stops matching fails the run, so a closed gap forces its own removal.
+
+Every run, passing or failing, dumps its frames, ground truth, violations and scenario output to `packages/conformance/out/<adapter>-<bidi|no-bidi>.json` and rebuilds `out/index.html`, a self-contained page laying each run out as a timeline of documents, command rows and DOM anchors with every violation beside the event it concerns. CI uploads `out/` as the `conformance-report` artifact when the `build` job fails.
+
+Beside the rules, the adapters are compared with each other: each run is reduced to what a dashboard user sees (steps, outcomes, the page each step replays, snapshots, run controls) and any difference between adapters in the same mode fails unless a declared gap explains it. The rules catch every adapter being wrong in the same way; the comparison catches a difference no rule anticipated. Adapters are registered once in `src/adapters.ts`.
+
+Covers `selenium-devtools` and `selenium-devtools-py`. The WDIO service and Nightwatch have no scenario yet. The fake emits no BiDi events, so console and network capture are not checked.
+
 ### `examples/`
 
 Per-adapter demo projects used for manual verification. One directory per adapter, and inside it one directory per test runner, named for the runner itself: