Repository navigation
A plain-script run always reports passed #309
Description
Part of #277
A plain-script run reports passed whatever happened. _send_default_suite is
called with a hardcoded state at
packages/selenium-devtools-py/src/selenium_devtools/instrumentation.py:356:
_send_default_suite(capturer, "passed") # the whole script run is doneand the adapter has no failure path at all. The dashboard shows a green 1/1 passed for a run that raised, which is the one signal a user reads at a glance.
Reproduced on examples/selenium/python-test/login.py with a deliberately
broken assertion: the tree shows 1/1 passed while the Actions list stops dead
at the getElementText that preceded the failing line, with the logout click
and final getCurrentUrl absent. The command stream tells the truth; the tree
contradicts it.
pytest runs are unaffected. _external_suites suppresses the synthetic
suite entirely, and pytest_plugin derives each state from
report.passed/failed/skipped. This is the plain-script path only.
Why it is not a one-line change
The "passed" is sent from driver.quit(), which runs inside the user's
finally before the exception reaches top level — so at that moment the
adapter genuinely does not know the outcome yet. The final state has to move out
of the quit path and into teardown:
- install a
sys.excepthookwrapper inenable()that records the failure, and
send the final suite state fromdisable()/ theatexithandler, which run
after the hook sys.exc_info()inside the quit path is the tempting shortcut — it does
return the in-flight exception during afinally— but it also returns one
when the caller is merely inside an unrelatedexceptblock, so it would
report false failures
Also worth deciding here: a plain script has no test boundaries, so "the run
failed" is the most the adapter can honestly say.
Acceptance criteria
A plain script that raises reports failed for its synthetic suite and test; a
clean run still reports passed; an unrelated except block in the user's code
does not make a passing run report failed. pytest runs are unchanged. Unit tests
cover all three.
Activity
vishnuv688 commentedon Aug 17, 2026
Follow-on found while verifying this: commands carry no test identity
Chasing why the screencast picker offers "Recording 1/2/3" with no way to know
which belongs to the selected test turned up a larger gap underneath it.
frames.command_logemitscommand,args,result,timestamp,startTime,callSource,id— and nothing else.grep -rn "testUid" packages/selenium-devtools-py/src/returns nothing. Shared'sCommandLoghas
testUid?: string, and the Nightwatch and WDIO adapters stamp it; thePython adapter never has.
Consequence, visible in the dashboard today: selecting a test does not narrow
the Actions list. It cannot — every command in the run is one undifferentiated
stream, so the middle panel shows all of them whatever you click. On the pytest
example that is three tests' commands under every selection.
So there are two missing links, not one:
testUid)The second is cheap once the first exists: the app's
#videoslist incomponents/browser/snapshot.tsis already keyed bysessionId, andsend_screencastalready files each video under the session it came from. Whatis missing is knowing which session a given test ran in.
One wrinkle specific to a function-scoped pytest fixture (the shape the
example uses, and the common one): the driver is created during test setup, so
at
pytest_runtest_logstartno session exists yet. Attribution has to happenwhen the session starts —
instrumentationknows the driver and its session id— against the test currently running, which only the plugin knows. That means
the two modules need a shared notion of "current test", which is exactly what
the JS adapters carry.
Not this issue's outcome fix, and bigger than it. Recording here because it was
found on this branch and the two are adjacent in the same surface.