diff --git a/.changeset/selenium-plain-script-capture.md b/.changeset/selenium-plain-script-capture.md new file mode 100644 index 00000000..0a9d8c65 --- /dev/null +++ b/.changeset/selenium-plain-script-capture.md @@ -0,0 +1,5 @@ +--- +'@wdio/selenium-devtools': patch +--- + +A plain `node script.js` run with no test runner is now captured without calling `DevTools.startTest`: its commands land under the "Selenium Session" row the README describes, where before every command was dropped. Such a script also exits once its work is done, where before the open dashboard connection kept the process alive indefinitely, which also left a rerun of a plain script running after it finished. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6cd81815..c70815a4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -43,8 +43,22 @@ jobs: # because it is fast and its failures are cheaper to read. - name: ๐Ÿ”Ž Typecheck run: pnpm typecheck + # The conformance suite runs the Python adapter beside the JS ones, so a + # capture fix missing from one language fails this job. + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: '3.13' + - name: Install the Python adapter + run: pip install -e packages/selenium-devtools-py - name: ๐Ÿงช Run Tests run: pnpm test + - name: Upload conformance report + if: failure() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: conformance-report + path: packages/conformance/out/ + if-no-files-found: ignore # Separate job because these are the only tests that need a real browser, and # they parallelise with `build` instead of extending it. No bundle step: the diff --git a/.gitignore b/.gitignore index ad7b9c30..1ca4ecec 100644 --- a/.gitignore +++ b/.gitignore @@ -54,6 +54,8 @@ examples/**/trace-*/ # test results examples/**/test-results*/ +packages/conformance/**/test-results*/ +packages/conformance/out/ # vitest --coverage output coverage/ diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 6cee2380..fd718d02 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -135,7 +135,7 @@ Contains: - `pytest_plugin.py` โ€” the suite tree, keyed by pytest nodeid; auto-discovered, opt-in by env var. - `rerun.py`, `run_id.py`, `performance.py`, `transport.py` โ€” run controls, run identity, navigation timings, and a stdlib-only WebSocket client. -Live mode only: it does not write a trace archive. That work, and the question of whether the ~3,100 lines of trace transforms in `core` should move behind the wire rather than be reimplemented per language, is tracked in [#330](https://github.com/webdriverio/devtools/issues/330) and [#278](https://github.com/webdriverio/devtools/issues/278). +It writes trace archives without running the transforms itself: `trace_export.py` streams the action snapshots and filmstrip frames, then sends a `traceExport` request and the backend builds the archive from the run it accumulated, answering with `traceExported`. `export_trace()` and teardown (`disable()`, the pytest plugin's session finish) trigger it. Writing an adapter in another language is covered in [docs/ADAPTERS.md](./docs/ADAPTERS.md). Imports from: `selenium` (required, 4.44+). Does not import: `core`, `shared`, any other package. @@ -188,6 +188,18 @@ Imports from: `core`. The script bodies, serializers, types, and locator generat The DOM-walking scripts run in the page via `browser.execute`, so โ€” like `script` โ€” they avoid Node-only APIs. +### `packages/conformance`: cross-adapter conformance suite + +Test-only and never published or bundled. Each adapter runs one scenario (open `/login`, type, submit to `/secure`, log out, then a navigation that fails) as its own process, against a fake W3C WebDriver server that models two pages and records what really happened in the browser: which documents were born and when, and which navigations failed. The adapter reports into a recording backend that stands in for the dashboard, reached through the same reuse handshake a rerun uses, so no adapter gains a code path for being tested. `src/checker.ts` then judges the recorded frame stream against behavioural rules, for example "a row replays the page the browser was on when its action completed" and "every document gets exactly one anchor at its birth time". Every adapter is checked by the same rules, so a capture fix that lands in one language and not another fails CI. + +Each scenario runs twice, with and without BiDi. The fake answers the BiDi commands the adapters send, and a registered preload puts the collector into every later document from birth, as in Chrome. A difference an adapter is allowed to show lives in `src/allowlist.ts` with a mandatory reason, and an entry that stops matching fails the run, so a closed gap forces its own removal. + +Every run, passing or failing, dumps its frames, ground truth, violations and scenario output to `packages/conformance/out/-.json` and rebuilds `out/index.html`, a self-contained page laying each run out as a timeline of documents, command rows and DOM anchors with every violation beside the event it concerns. CI uploads `out/` as the `conformance-report` artifact when the `build` job fails. + +Beside the rules, the adapters are compared with each other: each run is reduced to what a dashboard user sees (steps, outcomes, the page each step replays, snapshots, run controls) and any difference between adapters in the same mode fails unless a declared gap explains it. The rules catch every adapter being wrong in the same way; the comparison catches a difference no rule anticipated. Adapters are registered once in `src/adapters.ts`. + +Covers `selenium-devtools` and `selenium-devtools-py`. The WDIO service and Nightwatch have no scenario yet. The fake emits no BiDi events, so console and network capture are not checked. + ### `examples/` Per-adapter demo projects used for manual verification. One directory per adapter, and inside it one directory per test runner, named for the runner itself: diff --git a/docs/ADAPTERS.md b/docs/ADAPTERS.md new file mode 100644 index 00000000..ed3110d7 --- /dev/null +++ b/docs/ADAPTERS.md @@ -0,0 +1,284 @@ +# Writing an adapter for a new language + +This guide is for someone adding devtools support to a language or framework that cannot import the TypeScript packages in this repo: Java + JUnit, C# + NUnit, Ruby + RSpec, and so on. It assumes you know WebDriver and your target framework, and that you have read the package map in [ARCHITECTURE.md](../ARCHITECTURE.md). + +The worked example throughout is the Python adapter, `packages/selenium-devtools-py`. It shares no code with the JavaScript adapters, only the wire format, so everything it does is something your adapter has to do too. + +--- + +## 1. What an adapter is, and what it is not + +An adapter is a process-local library that watches a test run and streams `{scope, data}` JSON frames over one WebSocket to the devtools backend. That is the whole interface. It does not render anything, store anything, or build trace archives itself. + +What you reuse, and how you reach it from another language: + +| You get | Provided by | How a non-JS adapter reaches it | +|---|---|---| +| The dashboard UI (Actions, Console, Network, DOM replay, Source, video) | `packages/app`, served by the backend | Nothing to do: it renders whatever frames you send. | +| WS gateway, broadcast to dashboard tabs, run accumulation, Preserve & Rerun | `packages/backend` | Connect to `/worker` and send frames. | +| The page-side DOM collector (`window.wdioTraceCollector`) | `packages/script`, served by the backend | `GET /api/collector` (`COLLECTOR_API.get` in `shared/src/collector.ts`). Raw JS source. | +| Element / accessibility-tree scripts for the trace's A11y tab | `shared/src/element-scripts.ts`, served by the backend | `GET /api/element-scripts?runner=` (`ELEMENT_SCRIPTS_API`). JSON `{accessibilityTree, elements}`. | +| Trace archive (`trace.zip`) building, retention policy, per-test slicing | `packages/trace`, run by the backend | Send a `traceExport` frame on the worker socket; the answer comes back as `traceExported` (`shared/src/trace-export.ts`). | +| Rerun / Run-all from the dashboard | backend `runner.ts` | Publish `launchCommand` / `rerunCommand` in your `metadata` frame and honour the reuse handshake (section 2d). | +| The backend itself | `@wdio/devtools-backend` on npm (`dist/server.js`, bin `devtools-backend`) | Spawn it with Node, or attach to one already running. | + +Serving the collector and element scripts over HTTP means their version always matches the backend you are talking to. Do not vendor a copy. + +What an adapter must not do: re-implement the trace transforms, invent scope names, or branch on anything the backend did not tell you. If you need a new concept on the wire, it starts as a change to `packages/shared`. + +## 2. The four responsibilities + +### 2a. Driver command interception + +**Contract.** Every user-visible WebDriver command produces one row in a `commands` frame (a `CommandLog`, see `wire-schema.json`): + +- `startTime` is the wall clock (epoch ms) at **invocation**. +- `timestamp` is the wall clock at **completion**, after the driver has answered. `timestamp >= startTime`. +- `command` uses a name the trace vocabulary knows. `ACTION_MAP` in `shared/src/trace-actions.ts` already maps raw W3C protocol names (`get`, `clickElement`, `sendKeysToElement`, `clearElement`, `getElementText`, `goBack`, ...) because a protocol-level adapter reports those. An unmapped command still shows live but is dropped from a trace. +- A failed command still produces a row, with `error: {name, message}`, and the original exception is rethrown unchanged. +- Optional fields are omitted, never sent as `null` (the schema rejects `null` for an optional field). +- Your own scripts (collector injection, drains, screenshots) must not appear as rows. + +To enrich a row after the fact (navigation timings, for example), send `replaceCommand` with `{oldTimestamp, command}`. Preserve the original row's `id`: the backend and app match a numeric `id` first, then fall back to the last row with `oldTimestamp`. If neither matches, they append the replacement as a new row. + +**Pitfall already hit.** Two JS adapters stamped `timestamp` with the invocation clock while the page's mutation stream runs on real time, so a fill row replayed an empty field and a navigation row replayed the page it had just left (CLAUDE.md Known debt: "A command row is stamped at COMPLETION"). + +**Python.** `instrumentation.py` `install()` patches `WebDriver.execute`, the single chokepoint every driver and element command flows through (`patched_execute`). It records `start = now_ms()` before calling through, and `SessionCapturer.capture_command` in `capturer.py` stamps `timestamp=now_ms()` after the call returns. `frames.command_log` in `frames.py` builds the row and omits unset fields. Internal scripts run through `_guarded_execute_script`, which sets a flag that `patched_execute` checks so they bypass capture. `SKIP_COMMANDS` and `NAVIGATION_COMMANDS` live in `constants.py`; `_attach_performance` sends the enriched navigation row via `send_replace_command`. + +Find the equivalent chokepoint in your binding (Java's `CommandExecutor`, .NET's `ICommandExecutor`, Ruby's bridge `execute`) rather than wrapping each public method. + +### 2b. Collector injection and draining + +**Contract.** The collector lives in the page and buffers DOM mutations. A **DOM anchor** is a `childList` mutation carrying `url`; the collector stamps it with the document's own `performance.timeOrigin`, i.e. the document's birth time. Your adapter must: + +1. **Get the source** from `GET /api/collector`. +2. **Install it at document start when BiDi is available.** Request `webSocketUrl: true` in the new-session capabilities, then call BiDi `script.addPreloadScript` with **no browsing-context id** (global, so later contexts are covered) and the source wrapped as a function declaration: `async () => { }`. Register it before the session's first navigation. +3. **Fall back to `' + const html = renderReport([ + run({ + frames: [ + { + scope: 'commands', + data: [ + { + command: 'sendKeys', + args: [hostile], + startTime: 1000, + timestamp: 1010 + } + ] + } + ], + output: hostile + }) + ]) + expect(html).not.toContain('