Skip to content

fix(dumpkit): JSON output of DPKT/Scapy frames grows without bound and fills the disk #1372

Closed

Description

@JarryShaw

main (13aa677be) has a test that writes JSON without bound until the disk fills. On this host, four concurrent runs of util/run_unittest_leg.py foundation each grew a /tmp/tmp*/out.json to 5–7 GB within minutes (22 GB in total, and free disk fell from 56 GB to 42 GB) before I killed them. It recurred on the next four runs.

Reproduced on a clean origin/main worktree under a 400 MB file-size cap (bash -c 'ulimit -f 409600; …'):

  • tests/foundation/test_extraction_record_header_unit.py: DPKTHeaderStepTests and ScapyHeaderStepTests ::test_pcap_output_opens_with_the_global_header both end in OSError: [Errno 27] File too large.
  • The traceback cycles between pcapkit/dumpkit/common.py:558 (_append_fallback) and dictdumper/json.py:231 (_append_object). The call is extract(fin=sample_path('in.pcap'), fout=…/out, engine=…, format='json', nofile=False) at :42.

_append_fallback itself is old. The new ingredient is what reaches it from the dpkt and scapy frames. That points at today's merges, most likely #1335 (a buffer attached to each dpkt packet) or #1336 (DecodedString). Neither is confirmed: this needs bisecting.

The same leg also fails tests/foundation/engines/test_runtime_engines.py::PCAPEngineTests::test_pcap_run_writes_global_header_for_file_and_stream_modes (both subtests). The test expects _open_output without the version= keyword that #1305 added. That is a second, separate red mark on main.

Workaround until fixed: never run the foundation leg without a file-size cap.

Activity

  1. added
    fixPull requests that fix a defect (fix: subject prefix)
    wipWork in flight - a covering PR is open or an agent is actively on it
    on Oct 7, 2026
  2. added a commit that references this issue on Oct 7, 2026
    b89db14
  3. added a commit that references this issue on Oct 8, 2026
    8c8ce87
  4. removed
    wipWork in flight - a covering PR is open or an agent is actively on it
    on Oct 8, 2026
  5. added this to the PyPCAPKit v1.5 milestone on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    fixPull requests that fix a defect (fix: subject prefix)

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions