design: late duplicate IP fragments, and dpkt/scapy trace_format=pcap fallback

Two behaviours found by the #1202 foundation slice that may be intended. Decision needed on each. Both reproduce on main (00aa1fc90).

A late duplicate IP fragment opens a new PARTIAL datagram. Feed IPv4() reassembly fragment 0 (8 octets, MF set), then the last fragment, then the last fragment again. It yields [('PARTIAL', (3,), (b'BBBB',)), ('COMPLETE', (1, 2), b'AAAAAAAABBBB')]. The buffer is popped on completion (pcapkit/foundation/reassembly/ip.py:193), and the duplicate opens a fresh one (:121). RFC 791 also frees the buffer on completion. Options:
- (a) Keep it, recorded as a known behaviour.
- (b) Remember completed buffer IDs for the reassembly timeout, and drop late duplicates.
dpkt and scapy write JSON when asked for trace_format='pcap'. Their trace adapters build frames with packet2dict (toolkit/dpkt.py:600, toolkit/scapy.py:363), and extraction.py:1367 falls back to JSON with a FormatWarning. The default engine writes PCAP. Options:
- (a) Keep the warned fallback.
- (b) Have those adapters write PCAP from the raw frame bytes.

Recommendation: 1(a) and 2(b). 1 follows the RFC. 2 is a byte-exact gap: the trace output differs by engine.

1 - check RFC and industry standards.
2 - if dpkt and scapy cannot and/or does not provide the frame bytes payload, meaning, they cannot write pcap files, then we should either keep the warn or raise from there.

Recorded, as the owner's direction on both items:

Late duplicate fragment: decide from the RFCs and industry practice. A research pass is starting: RFC 791, 815 and 8200, RFC 5722 on overlapping IPv6 fragments, and how Linux, BSD, Wireshark, Suricata, Snort and Zeek treat a fragment that arrives after its datagram has completed. It will report the evidence here before any change.
dpkt/scapy trace_format='pcap': this turns on whether those engines expose each frame's raw bytes.
- If they do, the adapters should write PCAP from those bytes.
- If they cannot, keep the warning, or raise from the adapter.
A worker is checking what each engine actually provides, and will act on what it finds.

Labelled wip.

Item 1 evidence, for the owner's decision. The research covered the RFCs and six implementations. Every source either does what PyPCAPKit does, or does not address the case.

Source	A fragment arriving after its datagram completed
RFC 791 §3.2	The buffer is freed on completion, so a straggler opens a new one that times out.
RFC 8200 §4.5	An implementation "may" drop exact duplicates; it is not required to.
RFC 5722, RFC 4963	Silent on post-completion duplicates. RFC 4963 warns that stale state can mis-associate after ID wraparound, which argues against remembering completed IDs.
Linux, FreeBSD	The queue is freed; a late fragment starts a fresh queue that expires (Linux counts `REASMTIMEOUT`).
Suricata, Zeek	The tracker is removed on completion; a late fragment gets a new one that times out, with no event raised.
Wireshark (tested with tshark)	The late fragment is shown as a separate, unreassembled fragment, with no overlap or retransmission flag.
scapy `defragment()`	The group is deleted on completion; a late fragment comes back as a bad fragment.

Recommendation: (a), keep today's behaviour and document it, since an offline analyser should match Wireshark and Zeek. If fewer spurious PARTIALs are wanted, (b) could be an opt-in suppression keyed on (src, dst, proto, ID) and expiring after the reassembly timeout, measured on capture time. It should flag the duplicate rather than drop it.

Snort 3 is unverified; its source could not be fetched. Item 2 is still being worked.

okay, take (a) then

Recorded: the owner chose (a) for item 1. A fragment that arrives after its datagram has completed opens a new buffer, and that buffer surfaces as an incomplete datagram. This matches RFC 791 and the behaviour of Linux, FreeBSD, Suricata, Zeek and Wireshark.

Follow-ups:

Document it: the IP reassembly docs will describe this behaviour (pcapkit/foundation/reassembly/ip.py).
test(foundation): reassembly, traceflow, engine and registry round trips (#1202) #1508: its reassembly test pins this as the expected outcome rather than as a Gap against design: late duplicate IP fragments, and dpkt/scapy trace_format=pcap fallback #1507.

Item 2, whether dpkt and scapy can write PCAP traces, is still being worked, so this stays wip.

docs(reassembly): a fragment after its datagram completes starts a ne…

test(foundation): reassembly, traceflow, engine and registry round tr…

docs(reassembly): a fragment after its datagram completes starts a ne…

test(foundation): reassembly, traceflow, engine and registry round tr…

update on item 2?

fix(engines): write real PCAP traces for dpkt and scapy (#1507)

Item 2 status: #1511, head d8d631ef8, round 3 in cross-review. Per your ruling, dpkt and scapy can supply the frame bytes, so they now write real PCAP traces instead of falling back to JSON. pyshark and pypcapfile keep the JSON fallback and its warning.

Fixed since round 1:

PCAP traces from dpkt and scapy are byte-identical to the default engine's on every real capture and on synthetic µs/ms/ns PCAP and PCAP-NG, including ns timestamps that are exact µs multiples. Round 1 found dpkt writing 775 where 774789 belongs on ns PCAP-NG.
dpkt now writes the true original length for snapped frames, on PCAP-NG and classic PCAP alike, so the provisional "unknown original length" warning is gone.
JSON, plist and tree traces from both engines are byte-identical to main again; 30 of 30 hashes match.

Under review now: the dpkt classic-PCAP reader reaches two private dpkt attributes to keep PktHdr.len. The reviewer is checking it against upstream's loop and against a dpkt that renames them.

Found along the way and filed separately, each reproduced on main:

fix(toolkit): scapy engine raises MissingKeyError('IP') on a PCAP-NG interface with raw-IP link type #1517: scapy fails on a raw-IP PCAP-NG interface.
fix(engines): TCP trace flows disagree between engines on snapped frames and on IPv4 total_length=0 #1518: the engines group TCP flows differently on snapped or TSO captures.
fix(engines): scapy engine keeps the first section's timestamp resolution after a second PCAP-NG section #1522: scapy's timestamp resolution goes stale after a second PCAP-NG section.

Interaction: whichever of #1508 and #1511 merges second deletes #1508's dpkt/scapy Gap(1507) rows.

test(foundation): reassembly, traceflow, engine and registry round tr…

fix(engines): write real PCAP traces for dpkt and scapy (#1507)

fix(toolkit): accept tshark's True/False TCP flags in the pyshark tra…

fix(engines): write real PCAP traces for dpkt and scapy (#1507)

fix(engines): write real PCAP traces for dpkt and scapy (#1507) (#1511)

Both items are done, so closing. Item 1 (the owner chose (a)): the late-duplicate behaviour is documented at pcapkit/foundation/reassembly/ip.py:93 and docs/source/pcapkit/foundation/reassembly/ip/index.rst:94. Item 2: #1511 merged as 2d088a745, so dpkt and scapy now write real PCAP traces, while pyshark and pypcapfile keep the warned JSON fallback.

test: remove the cross-engine agreement tests from the shared modules (…

Navigation Menu

Repository navigation

design: late duplicate IP fragments, and dpkt/scapy trace_format=pcap fallback #1507

3 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

design: late duplicate IP fragments, and dpkt/scapy trace_format=pcap fallback #1507

Description

Activity

JarryShaw commented on Oct 9, 2026

JarryShaw commented on Oct 9, 2026

JarryShaw commented on Oct 9, 2026

JarryShaw commented on Oct 9, 2026

JarryShaw commented on Oct 9, 2026

3 remaining items

JarryShaw commented on Oct 9, 2026

JarryShaw commented on Oct 9, 2026

JarryShaw commented on Oct 10, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions