Conformance suite: run the Nightwatch adapter, and the bug it found

.changeset/nightwatch-failed-command-error.md

@@ -0,0 +1,5 @@
+---
+'@wdio/nightwatch-devtools': patch
+---
+
+A command that failed at the protocol level, such as a navigation the browser refused, now carries its error on its row, so it shows red in the actions list and in the Errors tab. Nightwatch reports that failure as `{status: -1, value: null, error}`, and the row was read from `value`, so it rendered as a success with no result.

ARCHITECTURE.md

@@ -198,7 +198,7 @@ Every run, passing or failing, dumps its frames, ground truth, violations and sc
 
 Beside the rules, the adapters are compared with each other: each run is reduced to what a dashboard user sees (steps, outcomes, the page each step replays, snapshots, run controls) and any difference between adapters in the same mode fails unless a declared gap explains it. The rules catch every adapter being wrong in the same way; the comparison catches a difference no rule anticipated. Adapters are registered once in `src/adapters.ts`.
 
-Covers `selenium-devtools` and `selenium-devtools-py`. The WDIO service and Nightwatch have no scenario yet. The fake emits no BiDi events, so console and network capture are not checked.
+Covers `selenium-devtools`, `selenium-devtools-py` and `nightwatch-devtools`; the WDIO service has no scenario yet. Nightwatch runs through its own CLI against a config under `scenarios/nightwatch/`, which is the shape a runner-based adapter takes. The fake emits no BiDi events, so console and network capture are not checked.
 
 ### `examples/`
 

docs/ADAPTERS.md

@@ -229,8 +229,8 @@ The browser therefore creates exactly three documents.
 ### Adding your language
 
 1. Add your id to `AdapterId` in `src/types.ts`.
-2. Add `scenarios/<lang>_scenario.<ext>` performing the steps above. Read the three variables by name (the Python scenario repeats them, since it cannot import `SCENARIO_ENV`).
-3. Add one entry to `ADAPTERS` in `src/adapters.ts`: `ready()` (your toolchain and build are present) and `prepare()` (the command, args and env that launch your scenario, plus an optional `cleanup`). `tests/conformance.test.ts` then runs your adapter in both modes, holds it to every rule, and compares it with every other adapter. `ready()` may only skip a local run; in CI a missing prerequisite fails.
+2. Add `scenarios/<lang>_scenario.<ext>` performing the steps above, or, for an adapter that only runs under its framework's CLI, a folder with a config and a spec (`scenarios/nightwatch/` is the model: the config reads the fake's origin from `CONFORMANCE_WEBDRIVER` and points the runner at it with `start_process: false`). Read the three variables by name (the Python scenario repeats them, since it cannot import `SCENARIO_ENV`).
+3. Add one entry to `ADAPTERS` in `src/adapters.ts`: `ready()` (your toolchain and build are present) and `prepare()` (the command, args and env that launch your scenario, plus an optional `cleanup`). `tests/conformance.test.ts` then runs your adapter in both modes, holds it to every rule, and compares it with every other adapter. Expect the first run to be red: classify each line as a suite assumption to fix, a difference to declare with a reason, or an adapter bug to fix, and merge only when it is green. `ready()` may only skip a local run; in CI a missing prerequisite fails.
 4. Make sure CI installs your toolchain before `pnpm test` in `.github/workflows/ci.yml` (Python's install step is there).
 
 Run it locally with `pnpm test packages/conformance`. Vitest only collects `packages/**/tests/**/*.test.ts`.
@@ -251,7 +251,7 @@ Defined in `src/checker.ts`, names in `RULES` (`src/constants.ts`):
 | `failed-command-has-error-row` | Every navigation that failed in the browser has a row carrying `error`. |
 | `replace-targets-a-sent-row` | Every `replaceCommand.oldTimestamp` names a row already sent. |
 | `one-row-per-browser-action` | The number of `navigate`, `click` and `fill` rows equals what the fake browser received, after applying `replaceCommand` the way the dashboard does (shared `replaceCommand`: by `id`, then the last row with the old timestamp). |
-| `run-controls-have-a-command` | If `metadata.options.runCapabilities` advertises any control, the metadata carries `launchCommand` or `rerunCommand`. The backend falls back from either to the other and only to the WebdriverIO binary when both are missing, so a control with neither runs the wrong tool. |
+| `run-controls-are-served` | If `metadata.options.runCapabilities` advertises any control, the metadata carries `launchCommand` or `rerunCommand` (the backend's shell path, falling back from either to the other) or `configFile` (its framework path, which reruns a WDIO or Nightwatch config). With none, the backend falls back to the WebdriverIO binary, so the control runs the wrong tool. |
 
 ### Cross-adapter agreement
 
@@ -265,7 +265,7 @@ A failing test prints the `unexplained` violations, each `{rule, message, subjec
 
 ### Allowlist entries
 
-`src/allowlist.ts` `KNOWN_GAPS` lists differences an adapter is allowed to show: `{adapter, bidi, rules, reason, timingDependent?}`. An entry is legitimate only for a documented limitation that cannot be fixed in the adapter (for example, the `<script>` fallback without BiDi racing a short-lived document), and `reason` is required: say what happens and point at the Known debt entry. An entry that stops matching is reported as `stale` and fails the run, so a fixed gap forces its own removal; `timingDependent: true` exempts a race that a run can happen to win. Do not allowlist a rule your adapter breaks because it is unfinished.
+`src/allowlist.ts` `KNOWN_GAPS` lists differences an adapter is allowed to show: `{adapter, bidi, rules, reason, timingDependent?, lacksRunControls?}`. An entry is legitimate only for a documented limitation that cannot be fixed in the adapter (for example, the `<script>` fallback without BiDi racing a short-lived document), and `reason` is required: say what happens and point at the Known debt entry. A gap on `adapters-agree-on-run-controls` names the controls the adapter does not offer in `lacksRunControls` and explains their absence only: a control lost for any other reason is still a difference. An entry that stops matching is reported as `stale` and fails the run (the report shows a run's stale gaps in its own section and a comparison's in the agreement section), so a fixed gap forces its own removal; `timingDependent: true` exempts a race that a run can happen to win. Do not allowlist a rule your adapter breaks because it is unfinished.
 
 ## 5. Checklist
 

packages/conformance/package.json

@@ -19,7 +19,9 @@
     "@types/selenium-webdriver": "^4.35.6",
     "@types/ws": "^8.18.2",
     "@wdio/devtools-shared": "workspace:^",
+    "@wdio/nightwatch-devtools": "workspace:^",
     "@wdio/selenium-devtools": "workspace:^",
+    "nightwatch": "^3.16.0",
     "selenium-webdriver": "^4.50.0",
     "ws": "^8.22.0"
   }

packages/conformance/scenarios/nightwatch/nightwatch.conf.cjs

@@ -0,0 +1,25 @@
+// Nightwatch config for the conformance scenario: the harness supplies the fake
+// WebDriver and reaches the recording backend through the reuse handshake.
+const path = require('node:path')
+const nightwatchDevtools = require('@wdio/nightwatch-devtools').default
+
+const webdriver = process.env.CONFORMANCE_WEBDRIVER
+if (!webdriver) {
+  throw new Error(
+    'CONFORMANCE_WEBDRIVER is not set; run this through the conformance harness'
+  )
+}
+const { hostname, port } = new URL(webdriver)
+
+module.exports = {
+  src_folders: [path.resolve(__dirname, 'specs')],
+  output_folder: false,
+  webdriver: { start_process: false, host: hostname, port: Number(port) },
+  test_settings: {
+    default: {
+      skip_testcases_on_fail: false,
+      desiredCapabilities: { browserName: 'chrome', webSocketUrl: true },
+      globals: nightwatchDevtools({ mode: 'live', bidi: true })
+    }
+  }
+}

packages/conformance/scenarios/nightwatch/specs/scenario.js

@@ -0,0 +1,26 @@
+// The conformance scenario, driven through @wdio/nightwatch-devtools: the same
+// steps as every other adapter's scenario, in Nightwatch's own idiom.
+const env = (name) => {
+  const value = process.env[name]
+  if (!value) {
+    throw new Error(
+      `${name} is not set; run this through the conformance harness`
+    )
+  }
+  return value
+}
+
+describe('conformance', function () {
+  it('drives the fake pages', async function (browser) {
+    await browser.url(env('CONFORMANCE_LOGIN_URL'))
+    await browser.setValue('#username', 'tom')
+    await browser.setValue('#password', 'secret')
+    await browser.click('button[type="submit"]')
+    await browser.click('#logout')
+    try {
+      await browser.url(env('CONFORMANCE_BROKEN_URL'))
+    } catch {
+      // The failed navigation is part of the scenario.
+    }
+  })
+})

packages/conformance/src/adapters.ts

@@ -10,6 +10,11 @@ const PYTHON = process.env.CONFORMANCE_PYTHON ?? 'python3'
 const PYTHON_SRC = path.join(REPO_ROOT, 'packages/selenium-devtools-py/src')
 const scenario = (file: string) =>
   path.join(REPO_ROOT, 'packages/conformance/scenarios', file)
+// This package's own bin: nightwatch is a dev dependency here, not of the root.
+const NIGHTWATCH_BIN = path.join(
+  REPO_ROOT,
+  'packages/conformance/node_modules/.bin/nightwatch'
+)
 
 /** Every adapter the suite runs. Adding one is an entry here plus its scenario script. */
 export const ADAPTERS: AdapterScenario[] = [
@@ -25,6 +30,19 @@ export const ADAPTERS: AdapterScenario[] = [
       args: [scenario('selenium.ts')]
     })
   },
+  {
+    id: 'nightwatch-devtools',
+    ready: () =>
+      [
+        path.join(REPO_ROOT, 'packages/nightwatch-devtools/dist/index.js'),
+        NIGHTWATCH_BIN,
+        COLLECTOR_BUNDLE
+      ].every((f) => existsSync(f)),
+    prepare: () => ({
+      command: NIGHTWATCH_BIN,
+      args: ['-c', scenario('nightwatch/nightwatch.conf.cjs')]
+    })
+  },
   {
     id: 'selenium-devtools-py',
     ready: () =>

packages/conformance/src/agreement.ts

@@ -1,12 +1,14 @@
-import { gapsFor } from './allowlist.js'
+import { explainingGap, gapsFor, isAgreementRule } from './allowlist.js'
 import { RULES } from './constants.js'
-import { normalizeRun, pathOf } from './normalize.js'
+import { describeRunControls, normalizeRun, pathOf } from './normalize.js'
 import type {
   AdapterId,
   Divergence,
+  KnownGap,
   NormalizedRun,
   NormalizedStep,
   RuleId,
+  RunControls,
   RunReport,
   Violation
 } from './types.js'
@@ -64,8 +66,11 @@ function compareTwo(
       `snapshots [${a.snapshots.join(', ')}] vs [${b.snapshots.join(', ')}]`
     )
   }
-  if (a.runControls !== b.runControls) {
-    add(RULES.agreeOnRunControls, `${a.runControls} vs ${b.runControls}`)
+  if (!sameRunControls(a.runControls, b.runControls)) {
+    add(
+      RULES.agreeOnRunControls,
+      `${describeRunControls(a.runControls)} vs ${describeRunControls(b.runControls)}`
+    )
   }
   return out
 }
@@ -106,11 +111,11 @@ function describeAt(i: number, a: string[], b: string[]): string {
   return `step ${i + 1} is ${a[i] ?? '(nothing)'} vs ${b[i] ?? '(nothing)'} (${a.length} vs ${b.length} steps)`
 }
 
-/** A divergence is explained only by a gap occurring on that very subject, with the other adapter clean there. */
+/** A divergence is explained only by a gap occurring on that very subject, unless the other side declares its own. */
 export function judgeDivergences(
   divergences: Divergence[],
   runs: RunReport[]
-): { unexplained: Divergence[]; explained: Divergence[] } {
+): { unexplained: Divergence[]; explained: Divergence[]; stale: KnownGap[] } {
   const readable = runs.filter((r) => !r.checkerError)
   const normalized = new Map(
     readable.map((r) => [r.adapter, normalizeRun(r.frames)] as const)
@@ -123,9 +128,23 @@ export function judgeDivergences(
       )
       return run && other ? gapExplains(d, run, other, normalized) : false
     })
+  // A gap whose comparison rules matched no difference no longer holds, which judge() cannot see.
+  const stale = readable.flatMap((run) =>
+    gapsFor(run.adapter, run.bidi).filter((g) => {
+      const own = g.rules.filter(isAgreementRule)
+      return (
+        !g.timingDependent &&
+        own.length > 0 &&
+        !divergences.some(
+          (d) => d.adapters.includes(run.adapter) && own.includes(d.rule)
+        )
+      )
+    })
+  )
   return {
     unexplained: divergences.filter((d) => !explainedBy(d)),
-    explained: divergences.filter(explainedBy)
+    explained: divergences.filter(explainedBy),
+    stale
   }
 }
 
@@ -136,42 +155,72 @@ function gapExplains(
   other: RunReport,
   normalized: Map<AdapterId, NormalizedRun>
 ): boolean {
-  const gapRules = new Set(
-    gapsFor(holder.adapter, holder.bidi)
-      .filter((g) => g.rules.includes(d.rule))
-      .flatMap((g) => g.rules)
+  const gaps = gapsFor(holder.adapter, holder.bidi).filter((g) =>
+    g.rules.includes(d.rule)
   )
-  const shown = holder.violations.filter((v) => gapRules.has(v.rule))
-  if (!shown.length) {
+  if (!gaps.length) {
     return false
   }
+  const gapRules = new Set(gaps.flatMap((g) => g.rules))
+  const shown = holder.violations.filter((v) => gapRules.has(v.rule))
   const step = d.step
   if (d.rule === RULES.agreeOnPages && step !== undefined) {
     const at = (run: RunReport) => normalized.get(run.adapter)?.steps[step]?.at
     return (
       shown.some((v) => onRow(v, at(holder))) &&
-      !other.violations.some((v) => onRow(v, at(other)))
+      !other.violations.some((v) => onRow(v, at(other)) && !declared(other, v))
     )
   }
   if (d.rule === RULES.agreeOnSnapshots) {
-    const unanchored = shown
-      .filter((v) => v.rule === RULES.documentAnchored)
-      .flatMap((v) => (v.subject?.url ? [pathOf(v.subject.url)] : []))
-    const holderPages = [
-      ...(normalized.get(holder.adapter)?.snapshots ?? []),
-      ...unanchored
-    ].sort()
-    const otherPages = [
-      ...(normalized.get(other.adapter)?.snapshots ?? [])
-    ].sort()
+    // Each side's snapshots plus the documents its own declared gap left unanchored must agree.
+    const pages = (run: RunReport, violations: Violation[]) =>
+      [
+        ...(normalized.get(run.adapter)?.snapshots ?? []),
+        ...violations
+          .filter((v) => v.rule === RULES.documentAnchored && v.subject?.url)
+          .map((v) => pathOf(v.subject!.url!))
+      ].sort()
+    const otherDeclared = other.violations.filter((v) => declared(other, v))
     return (
-      sameList(holderPages, otherPages) &&
-      !other.violations.some((v) => v.subject?.kind === 'document')
+      sameList(pages(holder, shown), pages(other, otherDeclared)) &&
+      !other.violations.some(
+        (v) => v.subject?.kind === 'document' && !declared(other, v)
+      )
     )
   }
+  if (d.rule === RULES.agreeOnRunControls) {
+    return runControlsExplained(
+      gaps,
+      normalized.get(holder.adapter),
+      normalized.get(other.adapter)
+    )
+  }
+  // A rule with no per-run counterpart (actions, errors): the difference is the gap.
   return true
 }
 
+// The gap explains the absence of the controls it declares and nothing else, so a control lost since is a finding.
+function runControlsExplained(
+  gaps: KnownGap[],
+  holder: NormalizedRun | undefined,
+  other: NormalizedRun | undefined
+): boolean {
+  if (!holder || !other) {
+    return false
+  }
+  const lacks = new Set<string>(gaps.flatMap((g) => g.lacksRunControls ?? []))
+  return sameRunControls(holder.runControls, {
+    ...other.runControls,
+    advertised: other.runControls.advertised.filter((c) => !lacks.has(c))
+  })
+}
+
+const sameRunControls = (a: RunControls, b: RunControls) =>
+  a.served === b.served && sameList(a.advertised, b.advertised)
+
+const declared = (run: RunReport, v: Violation) =>
+  Boolean(explainingGap(gapsFor(run.adapter, run.bidi), v))
+
 const onRow = (v: Violation, at: number | undefined) =>
   at !== undefined && v.subject?.kind === 'row' && v.subject.timestamp === at
 

packages/conformance/src/allowlist.ts

@@ -1,4 +1,11 @@
-import type { AdapterId, Judgement, KnownGap, Violation } from './types.js'
+import { RULES } from './constants.js'
+import type {
+  AdapterId,
+  Judgement,
+  KnownGap,
+  RuleId,
+  Violation
+} from './types.js'
 
 export const KNOWN_GAPS: KnownGap[] = [
   {
@@ -13,9 +20,36 @@ export const KNOWN_GAPS: KnownGap[] = [
     reason:
       'Without BiDi the collector is appended as a <script> after a readiness poll, so a document the test leaves within tens of ms is never anchored. CLAUDE.md Known debt: "Document-start injection is what removes the whole race class".',
     timingDependent: true
-  }
+  },
+  {
+    adapter: 'nightwatch-devtools',
+    bidi: false,
+    rules: [
+      'every-document-anchored',
+      'row-replays-its-page',
+      'adapters-agree-on-pages',
+      'adapters-agree-on-snapshots'
+    ],
+    reason:
+      'Without BiDi the collector is appended as a <script> and the destination is anchored by polling for the old collector to disappear, so a document the test leaves within tens of ms is never anchored. CLAUDE.md Known debt: "Document-start injection is what removes the whole race class".',
+    timingDependent: true
+  },
+  ...[true, false].map((bidi): KnownGap => ({
+    adapter: 'nightwatch-devtools',
+    bidi,
+    rules: [RULES.agreeOnRunControls],
+    lacksRunControls: ['canRunAll'],
+    reason:
+      'Nightwatch advertises no Run-all: the backend reruns a Nightwatch config through its framework path, which selects suites and tests but has no whole-run entry. Documented on the capability matrix.'
+  }))
 ]
 
+/** An `adapters-agree-*` rule is judged by the comparison; every other rule by the run itself. */
+export function isAgreementRule(rule: RuleId): boolean {
+  return rule.startsWith('adapters-agree-')
+}
+
+/** Judges one run's own violations: a gap counts as stale only through the rules this run can show. */
 export function judge(
   adapter: AdapterId,
   bidi: boolean,
@@ -24,10 +58,14 @@ export function judge(
   const gaps = gapsFor(adapter, bidi)
   return {
     unexplained: violations.filter((v) => !explainingGap(gaps, v)),
-    stale: gaps.filter(
-      (g) =>
-        !g.timingDependent && !violations.some((v) => g.rules.includes(v.rule))
-    )
+    stale: gaps.filter((g) => {
+      const own = g.rules.filter((r) => !isAgreementRule(r))
+      return (
+        !g.timingDependent &&
+        own.length > 0 &&
+        !violations.some((v) => own.includes(v.rule))
+      )
+    })
   }
 }