Repository navigation
fix(dumpkit): JSON output of DPKT/Scapy frames grows without bound and fills the disk - #1381
Conversation
|
Cross-review verdict on The Everything else held:
The re-import pollution that sends real |
…d fills the disk - _append_fallback writes a class, module or routine as its qualified name instead of expanding it through vars(), which since #1305 turned a class's mappingproxy into a dict and walked the whole type graph. - An object already on the current expansion path is written as a <circular reference: ...> marker, so a cycle terminates; a shared object is still expanded in full each time it is met. - A top-level fallback object expanding into more than FALLBACK_LIMIT (10000) objects raises UnsupportedCall instead of writing on. - test_runtime_engines: expect the version= keyword #1305 added to PCAP's _open_output call. Closes #1372
eabeaab to
b89db14
Compare
|
Round-2 verdict on
Nit, worth a follow-up: |
|
Coverage: 88.98% (unit tier, Python 3.14,
Per-file detail: the |
…1803) Entries: - add #1791 (breaking: pack() leaves its instance unchanged), #1778 (Protocol in protocol.__all__), #1781 (PortEnumField miss with a nameless proto), #1779 (vendor.ipx _path), #1780 (examples READMEs to Markdown, build leftovers) and #1790 (36 ambiguous cross-references) - fold #1776 into the RPL 0x23 entry and #1381 into the dump-schema one Accuracy: - the benchmark and legacy_smoke READMEs are README.md since #1780 - EngineBase stays out of __all__, as #1016 settled - the #575 peek calls get() only once the value resolves - the #647 and #764 entries point at #775/#860, under which nothing mints - the #861 entry names #869 and #874 for the EnumRegistry move Figures: - preamble: 685 citations, #340 to #1791 - process.rst pin: 298 entries; CHANGELOG.md regenerated Refs #1103
make pylint,make mypy,make isort): mypy and isort are clean oncommon.py; pylint adds nothing new except the existingtoo-many-statementsonmake_dumper, now 85 statements against 65 beforemake testpasses, and a test case covers the change: see below for the modules and legs actually runWhat is the purpose of your pull request?
fix— corrects a defectCloses #1372
First bad commit:
9093061b3(#1305). Found withgit bisect runfrom13aa677be(bad) back to11ae68ebf(good), running thefoundationleg with a 50 MB file cap. #1335 and #1336 are not the cause.Root cause. In the plain-unittest leg, the
defaultrun attest_extraction_record_header_unit.py:63ends up with an Ethernet info object (pcapkit.protocols.data.link.ethernet.Ethernet) whose class does not match the dumper'sInfo. That is cross-import pollution. Soobject_hooknever matches it and the object goes to_append_fallback. Itsvars()include__next_type__, which is a protocol class.Before #1305,
vars(cls)was amappingproxythat was written out withstr(). #1305 changed this tonew_value.update(vars(value)), so every class became a real dict and was expanded. That walks typing generics, schemas, fields, enum types,intand the descriptors of each.Nothing is memoised, so every shared node is expanded again on every path to it. Output therefore grows exponentially while the stack stays shallow. At 3,000 fallback calls the stack was 64 frames deep against a recursion limit of 1,000, and
wrapper_descriptorhad been expanded 931 times. That is why noRecursionErrorwas raised.Fix (
pcapkit/dumpkit/common.py).<circular reference: T>, so a cycle terminates. A shared object that is not part of a cycle is still written in full every time it appears, so no data is dropped.FALLBACK_LIMIT = 10_000objects, the dumper raisesUnsupportedCallinstead of writing a marker. Every fixture stays far below that: across thefoundation,dumpkitandinterfacelegs no fallback object expands into more than 7, so the budget never fires on a fixture.These rules depend on no particular engine. Schema change: a class, module or function in fallback output is now written as its name.
Second issue.
test_runtime_engineswas wrong, not the code. #1305 passesversion=on purpose (#1255), so the test now expectsversion=VersionInfo(2, 4).Evidence (all runs under
ulimit -f 204800):tests/dumpkit/test_fallback_cycle_unit.py: 4 passed. Withorigin/main'scommon.pyrestored, 3 fail (cycle and class:RecursionError; budget: nothing raised). The shared-object test passes there too, as it should: it pins that nothing is lost.util/run_unittest_leg.py foundation:483 test(s), 0 failure(s), 0 error(s). Onmainit was2 failure(s), 2 error(s),Errno 27. Within the leg,out.jsonis now at most 25,767 bytes.util/run_unittest_leg.py dumpkit:220 test(s), 0 failure(s), 0 error(s).test_runtime_engines5 passed,test_extraction_record_header_unit10 passed,test_common_unit14 passed,test_dumper_roundtrip_unit7 passed,tests/project389 passed and 1 skipped.