twelve tests/project/ tests fail with `TypeError: type 'ProtocolBase' is not subscriptable` depending on what ran first: the fake-module helpers purge on entry and never restore on exit

[JarryShaw]

Twelve tests in tests/project/ fail with TypeError: type 'ProtocolBase' is not subscriptable depending only on what ran before them. The same three files pass when reordered.

The cause is that tests/_support.py's fake-module helpers achieve isolation by purging on entry instead of restoring on exit. A test that installs a stand-in ProtocolBase into sys.modules never removes it, so the next test that imports pcapkit without purging first inherits a pcapkit.protocols.protocol module whose ProtocolBase is not Generic — and every class X(Protocol[...]) in the package raises.

Measured on 375e9d411, CPython 3.14.7, pytest 9.1.1. No ordering plugin is installed (pytest-randomly / pytest-xdist absent), so this is default collection order, not randomisation.

Reproduction

Failing — exit code 1, 12 failed, 5 passed in 0.89s, twelve not subscriptable:

$ python -m pytest -p no:cacheprovider -q \
    tests/corekit/test_protochain.py \
    tests/project/test_public_api.py tests/project/test_documentation_claims.py
...
12 failed, 5 passed, 1 warning, 2 subtests passed in 0.89s

Passing control 1 — drop the polluting file. Exit code 0:

$ python -m pytest -p no:cacheprovider -q \
    tests/project/test_public_api.py tests/project/test_documentation_claims.py
13 passed, 1 warning, 432 subtests passed in 0.67s

Passing control 2, the decisive one — the same three files, reversed. Exit code 0:

$ python -m pytest -p no:cacheprovider -q \
    tests/project/test_public_api.py tests/project/test_documentation_claims.py \
    tests/corekit/test_protochain.py
17 passed, 1 warning, 432 subtests passed in 0.70s

Identical file set, identical environment, opposite result. The only variable is order.

The twelve

tests/project/test_public_api.py::PublicAPISurfaceTests::test_aggregators_export_every_public_attribute
tests/project/test_public_api.py::PublicAPISurfaceTests::test_all_entries_are_strings
tests/project/test_public_api.py::PublicAPISurfaceTests::test_every_all_entry_resolves
tests/project/test_public_api.py::PublicAPISurfaceTests::test_every_public_package_declares_all
tests/project/test_public_api.py::PublicAPISurfaceTests::test_every_public_package_exports_every_public_attribute
tests/project/test_public_api.py::PublicAPISurfaceTests::test_no_duplicate_all_entries
tests/project/test_public_api.py::PublicAPISurfaceTests::test_no_misspelled_all
tests/project/test_public_api.py::PublicAPISurfaceTests::test_public_modules_were_actually_found
tests/project/test_public_api.py::PublicAPISurfaceTests::test_star_import_of_every_public_subpackage
tests/project/test_public_api.py::PublicAPISurfaceTests::test_the_non_export_allowlist_is_tight
tests/project/test_documentation_claims.py::TestNoEOFDocumentedOnce::test_documented_sense_matches_the_code
tests/project/test_documentation_claims.py::TestNoEOFDocumentedOnce::test_extractor_and_extract_agree

Ten of PublicAPISurfaceTests's methods and two of TestNoEOFDocumentedOnce's three. The third, test_assigned_flags_are_declared, only ast.parses source text (tests/project/test_documentation_claims.py:117) and never imports the package, which is why it survives — a useful confirmation that the discriminator is "does this test import pcapkit at runtime".

Traceback

______________ PublicAPISurfaceTests.test_all_entries_are_strings ______________

    def test_all_entries_are_strings(self) -> None:
>       for name, _ in public_modules():
                       ^^^^^^^^^^^^^^^^

tests/project/test_public_api.py:344:
tests/project/test_public_api.py:269: in public_modules
    walk(pcapkit, 'pcapkit.')
tests/project/test_public_api.py:267: in walk
    walk(importlib.import_module(info.name), info.name + '.')
...
pcapkit/dumpkit/__init__.py:13: in <module>
    from pcapkit.dumpkit.pcap import PCAPIO
pcapkit/dumpkit/pcap.py:18: in <module>
    from pcapkit.protocols.misc.pcap.header import Header
pcapkit/protocols/misc/__init__.py:16: in <module>
    from pcapkit.protocols.misc.pcap import *
pcapkit/protocols/misc/pcap/__init__.py:14: in <module>
    from pcapkit.protocols.misc.pcap.frame import Frame

>   class Frame(Protocol[Data_Frame, Schema_Frame],
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                schema=Schema_Frame, data=Data_Frame):
E               TypeError: type 'ProtocolBase' is not subscriptable

pcapkit/protocols/misc/pcap/frame.py:59: TypeError

Mechanism, confirmed directly rather than inferred

The real base is generic — pcapkit/protocols/protocol.py:77:

class ProtocolBase(Generic[_PT, _ST], metaclass=ProtocolMeta):

and nine concrete classes subscript it at class-definition time, through the conventional as Protocol alias. pcapkit/protocols/misc/pcap/frame.py:59 is the one in the traceback above; the others are pcapkit/protocols/misc/raw.py:31, misc/null.py:30, misc/pcap/header.py:62, misc/pcapng.py:295, transport/transport.py:41, link/link.py:37, application/application.py:31, internet/internet.py:34.

The stand-in is not generic — tests/_support.py:205-229:

def install_fake_protocol_module() -> type:
    ensure_package('pcapkit.protocols', ROOT / 'pcapkit' / 'protocols')

    protocol_module = types.ModuleType('pcapkit.protocols.protocol')

    class ProtocolBase:
        alias = 'PROTOCOL'
        ...

    protocol_module.ProtocolBase = ProtocolBase
    sys.modules['pcapkit.protocols.protocol'] = protocol_module
    return ProtocolBase

It shares the real class's __name__, which is why the error message names ProtocolBase and reads as though the library itself were broken.

purge_modules only pops — it does not save or restore — tests/_support.py:271-275:

def purge_modules(prefixes: Iterable[str]) -> None:
    for name in list(sys.modules):
        if any(name == prefix or name.startswith(prefix + '.') for prefix in prefixes):
            sys.modules.pop(name, None)
    _reset_abc_caches()

And the caller installs it in setUp with no tearDown — tests/corekit/test_protochain.py:9-11:

    def setUp(self) -> None:
        purge_modules(['pcapkit'])
        self.ProtocolBase = install_fake_protocol_module()

grep -c 'def tearDown\|addCleanup' on that file returns 0. So each test method protects itself on the way in and leaves the fake in sys.modules on the way out.

Confirmed without pytest at all, three cases in one interpreter:

--- case A: purge, install fake protocol module ONLY, then import pcapkit ---
  TypeError: type 'ProtocolBase' is not subscriptable

--- case B: same, then a SECOND fresh import (is the failure sticky?) ---
  TypeError: type 'ProtocolBase' is not subscriptable

--- case C: purge first (what a well-behaved setUp does), then import ---
  imported OK -> .../pcapkit/__init__.py

Case A is the defect. Case B matters because a failed import is not cached, so every later import pcapkit re-raises — which is why twelve tests fail rather than one. Case C is why the bug is invisible most of the time: a later setUp that purges heals the state for everything after it.

The victim side never purges. tests/project/test_public_api.py has no setUp and no tearDown; it reaches the package through public_modules(), tests/project/test_public_api.py:244-257:

@functools.lru_cache(maxsize=1)
def public_modules() -> 'tuple[tuple[str, bool], ...]':
    ...
    import pcapkit

functools.lru_cache does not cache exceptions, so all ten methods re-trigger the failing import independently. tests/project/test_documentation_claims.py:62-63 and :81 do from pcapkit.foundation.extraction import Extractor / from pcapkit.interface.core import extract, both of which pull the same chain.

There is a second, inline copy of the same idiom at tests/interface/test_core.py:114-124, which installs its own non-generic ProtocolBase into sys.modules; that file also has no tearDown (setUp at :73 and :218, both purging).

Why it looks intermittent, and why CI does not catch it

Because healing is accidental, and test_protochain.py sorts last in its own directory.

tests/corekit/ contains, in collection order, test_fields_*.py, test_infoclass.py, test_io.py, test_module.py, test_multidict.py, test_protochain.py — the polluter is alphabetically last, so no sibling runs after it to heal the state. A whole-directory selection is therefore enough:

$ python -m pytest -p no:cacheprovider -q tests/corekit/ tests/project/test_public_api.py
...
10 failed, 159 passed, 5 warnings, 374 subtests passed in 128.16s (0:02:08)

Exit code 1, ten not subscriptable. No hand-picked file list required.

What saves the wider runs is an intervening directory whose tests purge. Measured clean, none showing this failure:

selection	result
tests/project/ alone	no ProtocolBase failure
tests/interface/ tests/project/	no ProtocolBase failure
tests/cli/ tests/const/ tests/corekit/ tests/dumpkit/ tests/foundation/ tests/interface/ tests/project/	no ProtocolBase failure
tests/protocols/ tests/project/	732 passed, no ProtocolBase failure
--ignore=tests/integration tests/ — the exact scope of the "Unit Tests" CI job	1420 passed, no ProtocolBase failure

Note row 3 against the tests/corekit/ run above: the same tests/corekit/ is in both, and the only difference is that dumpkit, foundation and interface run between it and tests/project/. That is the whole trigger.

So the condition is: tests/corekit/test_protochain.py is the last thing to touch sys.modules['pcapkit.protocols.protocol'] before a tests/project/ test imports pcapkit. Anything in between that purges hides it.

The consequence is that CI is green on this and will stay green — its unit job happens to include the healing directories — while a developer running a narrower selection, which is the normal way to work, hits twelve failures pointing at library code that is fine. The error names ProtocolBase and the traceback runs through pcapkit/protocols/misc/pcap/frame.py, so it reads as a library defect rather than as test pollution, and the first instinct is to go debug protocol.py.

Notes

• Pre-existing, and not introduced by any recent merge. tests/_support.py was last modified by corekit: stop the option and list loops spinning forever on a truncated area (#431) #432; none of docs(readme): trim to a landing page and convert to Markdown #619, docs: adopt Contributor Covenant 3.0 for the Code of Conduct #624, feat(util): have bump_version.py keep CITATION.cff in step with the bump #625, ci: wire the existing linters into CI as advisory checks #626, test(tcp): expect the short-read padding on the tail, repairing main (#604) (#621) #627, test(tcp): retarget the last assertion pinning the old padding side (#604) #628, fix(license): start the copyright term at 2017, and drop the end year #630, fix(packaging): ship CITATION.cff in the source distribution #631, docs(github): review the non-workflow metadata under .github/ #637 or docs(changelog): correct the LICENSE history in the #630 entry #638 touched it.
• Earlier reports of this put the count at twelve against a pristine 6c3d1b0d9, and a reviewer on fix(foundation): give no_eof a way to stop, so extract() returns (#620) #639 did not see it at all. Both are consistent with the above: same twelve tests, and the reviewer's selection happened to be one of the healing ones.
• The fix is a teardown rather than a change to any of the twelve tests. addCleanup capturing and restoring the previous sys.modules entries — or a purge_modules in tearDown — would make the order irrelevant. Restoring is better than purging, since purging on the way out just moves the cost to whoever imported legitimately before.
• Worth considering whether the stand-in should stop being named ProtocolBase, or should subclass Generic. Either would make the failure message say what actually happened.
• tests/integration/ uses the same idiom (tests/integration/_helpers.py, test_engine_runtime.py, test_frame_iteration.py, test_module_loading.py) and was not exercised here — CI's unit job excludes it. Untested, so unknown rather than clean.
• Observed but not investigated, and not part of this issue: three unrelated pre-existing failures showed up in the wide runs — tests/project/test_changelog_md.py::RepositoryStateTests::test_committed_changelog_is_in_step_with_the_newest_entry, and two subtest failures in tests/protocols/test_option_coverage_runtime.py::OptionCoverageCaptureTests::test_option_captures_are_what_the_generator_says_they_are (options-ipv6.pcap, options-tcp.pcap). The latter two may be artefacts of regenerating examples/captures during this session rather than genuine defects; flagged only so they are not mistaken for part of this one.