Repository navigation
fix(corekit): copy SeekableReader's octets and feed MultiDict a dict, and gate GraalPy's limits (#1963) - #1974
Conversation
… and gate GraalPy's limits (#1963) - SeekableReader copies buffered octets out of its view: GraalPy's BytesIO read/read1/readline from an exported buffer's start can return bytes sharing its memory, which the next buffer slide overwrote - MultiDict hands dict.__init__ and dict.update a dict, not pairs: GraalPy stores pairs through the overridden __setitem__, nesting each value list - new test_graalpy_semantics_1963_unit reproduces both on CPython with stand-ins for GraalPy's behaviour - gate in the tests: pickle protocols 0/1 (typing.Generic is not a heap type), the >sys.maxsize PCAP claim, resource.setrlimit and _abc_impl; bound the huge BytesField length by sys.maxsize Refs #1963
JarryShaw
added
fix
|
NEEDS CHANGES at Question. Fixes:
|
JarryShaw
added
review: needs-changes
#1963) - drop the OrderedMultiDict copy and the plain-dict state cases, which passed without the fixes - say why the stand-in check stays: it guards the stand-in, not the fix - drop the count from the module docstring - correction to the first commit's review: werkzeug 2.3.8's MultiDict.__init__ and __setstate__ match pcapkit's apart from the annotations and the l -> v rename Refs #1963
|
Correction to my last comment. Neither this PR's description nor its commits claim the |
|
Coverage: 90.07% (unit tier, Python 3.15,
Per-file detail: the |
|
Not sure i followed. But |
|
Thanks, that settles it. My question was whether to keep the werkzeug 2.3.8 copy verbatim or hand-patch it. Following your ruling, a worker is upgrading the |
JarryShaw
removed
the
needs: decision
- port MultiDict, OrderedMultiDict and iter_multi_items from werkzeug 3.1.9 rather than 2.3.8: dict reached through super(), any Mapping and set values accepted, | and |= added, setdefault/setlistdefault simplified - keep the local adaptations: dict base without TypeConversionDict, MissingKeyError/UnsupportedCall, get() and getlist() without type=, OrderedMultiDict.deepcopy order fix, and no DeprecationWarning on OrderedMultiDict - werkzeug 3.1.9 still feeds pairs to dict.__init__ and update, which GraalPy routes through __setitem__; keep the two dict-building lines as the only behavioural divergence, noted in place - the GraalPy stand-in now sits in the MRO behind MultiDict, since super() no longer reads a module-global dict - test the new Mapping/set handling and the | operators tests/corekit and every test module importing the multidict pass on CPython; the #1963 and multidict tests pass on GraalPy 25.4.4. Refs #1963
|
Pushed
A cross-review is starting. |
JarryShaw
added
review: running
|
GOOD TO GO at |
JarryShaw
added
review: good-to-go
|
dont add versionchanged in MultiDict, it's a port from the existing source code. aim to be vertibam as possible to the original source. |
|
Understood. The worker is removing the |
JarryShaw
added
review: needs-changes
- drop the versionchanged notes and explanatory prose this PR added - restore werkzeug 3.1.9's docstrings, comments, overloads, annotations (as strings) and code layout for the ported classes - keep pcapkit's adaptations: dict + Generic base, MissingKeyError and UnsupportedCall, no type= on getlist, local get() and _missing, the #1295 OrderedMultiDict.deepcopy, poplist's [] default, and no OrderedMultiDict deprecation warning - keep the two GraalPy dict-building lines, with shorter comments tests/corekit passes on CPython; the multidict and #1963 modules pass on CPython and GraalPy 25.4.4. Refs #1963
JarryShaw
added
review: good-to-go
|
NEEDS CHANGES at |
JarryShaw
added
review: needs-changes
- add useless-parent-delegation (W0246) to KNOWN_MESSAGES: the pragma on MultiDict.__iter__ keeps upstream's CPython 87412 fast-path workaround Refs #1963
JarryShaw
added
review: running
|
GOOD TO GO at |
JarryShaw
added
review: good-to-go
|
Pushed |
JarryShaw
added
review: running
|
GOOD TO GO at |
JarryShaw
added
review: good-to-go
…#1958) - Opt in the modules the review found on LATEST despite the criterion: pickling and copying, the warning machinery and exception hooks, struct and int boundary tests, and the modules #1974 gates on GraalPy: fallback cycle, MH timestamp and the two #1541 overrun tests that compare against sys.maxsize. - Opt Scapy's engine and toolkit modules into ALL like the other engines, and PyShark's toolkit and non-packet modules with them. - State each demoted vendor module's LATEST tier and reason. - branches_on_interpreter() also sees sys.flags, sys._is_gil_enabled, sys.maxsize comparisons, sysconfig.get_config_var() and hasattr/getattr probes of sys; its docstring and the guard's message say the check is syntactic and best-effort. - _tiers.py and workflows.rst "Test Tiers" give the Scapy and vendor reasons. Tier guard, #1732, #1954, #1958 pins and every touched module pass on CPython 3.14.
JarryShaw
removed
the
review: good-to-go
* ci(unit-tests): run the whole suite on latest CPython (#1958) - test, integration: a step on the latest leg alone installs the whole suite's extras (Scapy, PyShark, vendor, docs, cli on top of the base); every other leg keeps its install line and runs the ALL tier. - project-tests, gate: install the same set. - tests/_dependency_gates.py: read a latest-only install step as that leg's extras; drop the exclusions it closes (HAS_SCAPY, HAS_PYSHARK, HAS_RUNTIME, HAS_VENDOR_DEPS, HAS_SPHINX, HAS_DOCUTILS, HAS_FURO). - tests/project/test_whole_suite_on_latest_1958.py pins the lines, the leg, the gates closed and the docs; tier-guard, #1732 and #1954 pins follow the new reading. - workflows.rst describes it; unit-tests.yml line citations re-anchored. Selection steps and tests/_tiers.py are unchanged. * ci(unit-tests): say where the latest leg gets cli, crypto and NGAP (#1958) - test, integration: the install comments said cli, and crypto and NGAP, are not installed; the latest leg's whole-suite step installs them. - workflows.rst, releasing.rst: unit-tests.yml line citations re-anchored. * test(tiers): make ALL a curated opt-in tier (#1958) - tests/_tiers.py: module_tier() is the declared tier, else DEFAULT_TIER (LATEST); rule_tier() goes. all_tier_paths() lists ALL, and branches_on_interpreter() flags a module that compares the interpreter itself. The docstring states the curation criterion (#1963). - Curated modules opt in with `__tier__ = 'ALL'` and a reason: version gates, GC/weakref, int/struct limits, pickling and copying, metaclass and enum machinery, IO, threading and hooks, engines whose releases vary by interpreter, the hand-picked round trips, and a smoke slice per protocol family. Six vendor markers with no such reason are gone. - tests/test_tier_guard.py: ALL is opt-in, every ALL module declares it, and a module that branches on the interpreter must state its tier; the mixed-directory split test no longer assumes tests/integration is all ALL. test_sweeps_on_latest_only_1732 compares what the ALL fixture selection drops with the LATEST modules it reaches. - tests/project/test_curated_all_tier_1958.py pins the opt-in rule, the strict subset, the ruling-B round trips and the docs. - workflows.rst "Test Tiers" and the `test` job's `Run unit tests` comment describe it. Tier guard, #1732, #1954 and #1958 pins pass on CPython 3.14. * test(tiers): opt the remaining interpreter-sensitive modules into ALL (#1958) - Opt in the modules the review found on LATEST despite the criterion: pickling and copying, the warning machinery and exception hooks, struct and int boundary tests, and the modules #1974 gates on GraalPy: fallback cycle, MH timestamp and the two #1541 overrun tests that compare against sys.maxsize. - Opt Scapy's engine and toolkit modules into ALL like the other engines, and PyShark's toolkit and non-packet modules with them. - State each demoted vendor module's LATEST tier and reason. - branches_on_interpreter() also sees sys.flags, sys._is_gil_enabled, sys.maxsize comparisons, sysconfig.get_config_var() and hasattr/getattr probes of sys; its docstring and the guard's message say the check is syntactic and best-effort. - _tiers.py and workflows.rst "Test Tiers" give the Scapy and vendor reasons. Tier guard, #1732, #1954, #1958 pins and every touched module pass on CPython 3.14.
…#1975) (#1979) * fix(protocols): handle declared lengths above sys.maxsize as overruns (#1975) Where Py_ssize_t is 32 bits wide (GraalPy, 32-bit CPython) a read or seek past sys.maxsize raises OverflowError instead of coming back short, so a PCAP incl_len or PCAP-NG length reaching that far crashed the reader before the overrun handling could keep the record as captured. - Schema.unpack bounds its payload and field reads at sys.maxsize. - Frame.read bounds the record read, and a next-record offset past sys.maxsize becomes the end of the octets read. - PCAPNG.read and _read_block_cut bound the block read. - SeekableReader._fill bounds a READ_CHUNK of -1 at sys.maxsize. - Lift #1974's sys.maxsize skips from the #1541 tests, and add a test that runs the narrow build's limit on any interpreter. Claims that fit read exactly as before. Closes #1975 * fix(protocols): cite #1975 as #NNN in plain comments (#1975) * fix(protocols): advance past a record longer than sys.maxsize instead of truncating (#1975) A record or block the file really holds past sys.maxsize was read only to the limit, and the next one was then parsed from inside it. - corekit.io.advance moves a stream forward any distance: relative seeks of at most sys.maxsize, or chunked read-and-discard on a SeekableReader or a non-seekable stream. - Frame.read and PCAPNG.read call it when the bounded read came back whole, so the next record starts at the declared end; a short read is unchanged. - The PCAP and PCAP-NG engines cap the read-ahead at sys.maxsize, and the PCAP frame schema reads no more than the header and payload can hold in one read, so a SeekableReader keeps the record start in its window. - SeekableReader._fill bounds any READ_CHUNK at sys.maxsize. - Tests cover a sparse file holding a record past a lowered limit, seekable and through a pipe, for PCAP and PCAP-NG. Closes #1975
make pylint,make mypy,make isort)make testpasses, and a test case covers the changeWhat is the purpose of your pull request?
fix— corrects a defectDescription of your pull request and other information
Sorts #1963's GraalPy-only failures into pcapkit defects and interpreter gates. Every failure was reproduced on a real GraalPy 25.4.4 before the change.
SeekableReader. On GraalPy,BytesIOreads alias an exported buffer, so a later buffer slide rewrote octets already returned. Reads now copy out of the view.MultiDict. On GraalPy,dict.__init__/dict.updatestore pairs through the overridden__setitem__, which nested each value list.__init__and__setstate__now pass a dict.tests/corekit/test_graalpy_semantics_1963_unit.pyreproduces both behaviours on CPython, and fails without the fixes.sys.maxsize,resource.setrlimitand_abc_impl. Each gated test still runs on CPython.Left open on #1963: GraalPy's speed, and a read clamp for
incl_lenclaims abovesys.maxsizeon 32-bit-Py_ssize_tbuilds.Refs #1963