Skip to content

test: three modules pass alone but fail together, and pytest reports it green #981

Description

@JarryShaw

Three test modules pass individually and fail when run in one process. pytest does not report the
failure at all
, which is what makes it worth filing rather than just fixing.

Reproduced on main (c49500438), repo venv, PYTHONPATH at the repo root:

tests.test_base_class_contract alone
  -> Ran 7 tests in 1.218s ... OK

tests.protocols.application.test_http_unit + tests.test_base_class_contract
  + tests.const.test_const_str_payload_870_unit, one process, plain unittest
  -> Ran 73 tests in 72.481s ... FAILED (failures=1, errors=3)

pytest over the same three files
  -> 73 passed, 603 subtests passed, exit 0

The failing assertion is RegistrationGateTests.test_user_style_subclass_registers_when_it_opts_in,
at suite='dumpers':

AssertionError: Items in the second set but not the first:
'user_opt_in_dumpers'

So a dumper the test registers under its opt-in keyword is absent from the registry by the time the
assertion runs, once the other two modules have imported first. Four subTests fail in total —
engines, reassembly, traceflow, dumpers.

Two things make this more than a flaky ordering nuisance.

The file's own .. note:: predicts cross-module registry interference, so the mechanism is known; what
is not recorded is that it currently fires. And pytest reports the run fully green — a parent node
reads passed when only its subTests failed, so CI cannot see this. Any suite-wide ordering regression
of this shape is invisible to the checks we run.

Not caused by any open change: I reproduced it at the merge base before the three prose PRs in flight.

Worth deciding separately whether the fix is isolating the registry per module, making the assertion
order-independent, or adding a plain-unittest leg to CI so the class of defect stops being invisible.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugtestPull requests that add or correct tests (test: subject prefix)

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions