Skip to content

fix(engines): dpkt float timestamps, Simple Packet Blocks and raw-IP tracing (#1501, #1520, #1548) #1550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JarryShaw merged 1 commit into from
Oct 10, 2026
Merged

fix(engines): dpkt float timestamps, Simple Packet Blocks and raw-IP tracing (#1501, #1520, #1548) #1550

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 6 additions & 20 deletions docs/source/pcapkit/foundation/engines/3rdparty.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,26 +64,6 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.

.. _DPKT: https://dpkt.readthedocs.io

.. important::

On PCAP-NG, `DPKT`_ yields **Enhanced Packet Blocks and (obsolete) Packet
Blocks only**. ``dpkt.pcapng.Reader.__iter__`` skips every other block type
without a warning (``# just ignore other blocks``, as of ``dpkt`` 1.9.8), and
:class:`~pcapkit.foundation.engines.dpkt.PCAPNGReader`, which this engine
reads PCAP-NG with, keeps that block set. It includes the **Simple Packet
Block**, which carries a packet just as an Enhanced Packet Block does. Such packets are silently missing from this
engine's output, and the frames after them are numbered as if they had never
been captured.

Reading :file:`examples/captures/test.pcapng` -- four Enhanced Packet Blocks
and one Simple Packet Block, across two sections -- this engine gives four
frames where the default engine and
:class:`~pcapkit.foundation.engines.scapy.Scapy` give five; the missing one is
frame 4, the Simple Packet Block. The second Section Header Block is not the
cause: the Enhanced Packet Block after it is read. This is a limitation of
`DPKT`_, not of :mod:`pcapkit`, so read a capture that may contain Simple
Packet Blocks with the default engine instead.

.. note::

``dpkt.pcapng.Reader`` reads only the **first** Interface Description Block
Expand All @@ -93,6 +73,12 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
resolves each packet's link type, ``if_tsresol`` and ``if_tsoffset`` from
the interface it names, in its own section (:issue:`1379`).

``dpkt.pcapng.Reader`` also skips Simple Packet Blocks, which `DPKT`_ has no
type for; :class:`~pcapkit.foundation.engines.dpkt.PCAPNGReader` reads them
through :mod:`pcapkit`'s own :class:`~pcapkit.protocols.misc.pcapng.PCAPNG`
and, since the block carries no timestamp, dates them 0, as the default
engine does (:issue:`1520`).

.. autoclass:: pcapkit.foundation.engines.dpkt.DPKT
:no-members:
:show-inheritance:
Expand Down
58 changes: 52 additions & 6 deletions pcapkit/foundation/engines/dpkt.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,11 @@

from pcapkit.foundation.engines.pcapng import Context
from pcapkit.foundation.extraction import Extractor
from pcapkit.protocols.data.misc.pcapng import IF_TSResolOption as Data_IF_TSResolOption
from pcapkit.protocols.data.misc.pcapng import \
InterfaceDescriptionBlock as Data_InterfaceDescriptionBlock
from pcapkit.protocols.data.misc.pcapng import SectionHeaderBlock as Data_SectionHeaderBlock
from pcapkit.protocols.data.misc.pcapng import SimplePacketBlock as Data_SimplePacketBlock

Reader = Union[PCAPReader, 'PCAPNGReader']

Expand All @@ -55,8 +57,11 @@ class PCAPNGReader:
``if_tsresol`` and ``if_tsoffset`` from the interface it names. Packet
blocks are still decoded by `DPKT`_.

As :class:`dpkt.pcapng.Reader` does, it yields Enhanced Packet Blocks and
(obsolete) Packet Blocks only, and skips every other block.
It yields Enhanced Packet Blocks, (obsolete) Packet Blocks and Simple Packet
Blocks, and skips every other block. :class:`dpkt.pcapng.Reader` skips the
Simple Packet Block too, and `DPKT`_ has no type for one, so that block is
parsed with :class:`~pcapkit.protocols.misc.pcapng.PCAPNG` as well
(:issue:`1520`).

.. _DPKT: https://dpkt.readthedocs.io

Expand Down Expand Up @@ -123,6 +128,8 @@ def _read(self) -> 'Iterator[tuple[float, bytes]]':
self._ctx.interfaces.append(cast('Data_InterfaceDescriptionBlock', idb.info))
elif block_type in (Enum_BlockType.Enhanced_Packet_Block, Enum_BlockType.Packet_Block):
yield self._read_packet(buf, block_type, byteorder == 'little')
elif block_type == Enum_BlockType.Simple_Packet_Block:
yield self._read_simple(buf)

def _read_packet(self, buf: 'bytes', block_type: 'int',
little: 'bool') -> 'tuple[float, bytes]':
Expand Down Expand Up @@ -177,6 +184,46 @@ def _read_packet(self, buf: 'bytes', block_type: 'int',
return (Timestamp(float(timestamp), timestamp, resolution, pkt.pkt_len), # pylint: disable=no-member
pkt.pkt_data)

def _read_simple(self, buf: 'bytes') -> 'tuple[float, bytes]':
"""Decode a Simple Packet Block, against the section's first interface.

`DPKT`_ has no type for this block, so it is parsed by
:class:`~pcapkit.protocols.misc.pcapng.PCAPNG`, as for the default
engine: the packet data is ``min(original_len, snaplen)`` octets, with
the snaplen of interface 0.

Args:
buf: Whole block, as read from the file.

Returns:
Timestamp and packet data, as :meth:`_read_packet` returns them. A
Simple Packet Block carries no timestamp, so the packet is dated at
the UNIX epoch, as the default engine dates it; the original length
is the block's own.

Raises:
FormatError: If the section describes no interface. A Simple Packet
Block names none, so it refers to the first (Section 4.4 of the
PCAP-NG specification).

"""
from pcapkit.protocols.misc.pcapng import PCAPNG as P_PCAPNG
from pcapkit.toolkit.dpkt import Timestamp # isort:skip

if not self._ctx.interfaces:
raise FormatError('PCAP-NG: [SPB] section has no interface description block')
interface = self._ctx.interfaces[0]
block = P_PCAPNG(buf, num=0, sct=self._sect, ctx=self._ctx)
spb = cast('Data_SimplePacketBlock', block.info)

tsresol = cast('Optional[Data_IF_TSResolOption]',
interface.options.get(Enum_OptionType.if_tsresol))
resolution = 1_000_000 if tsresol is None else tsresol.resolution

self._linktype = interface.linktype
return (Timestamp(0.0, decimal.Decimal(0), resolution, spb.original_len),
bytes(spb.packet))


def _pcap_reader() -> 'Type[PCAPReader]':
"""A :class:`dpkt.pcap.Reader` that keeps each record's original length.
Expand Down Expand Up @@ -258,10 +305,9 @@ class DPKT(EngineBase['DPKTPacket']):
:class:`dpkt.pcapng.Reader`, so each packet takes the link type and the
timestamp resolution and offset of the interface it was captured on.

Like `DPKT`_'s reader, it yields Enhanced Packet Blocks and (obsolete)
Packet Blocks only, and skips every other block without a warning -- the
Simple Packet Block included. A packet carried in a Simple Packet Block is
therefore missing from this engine's output; the default engine reads it.
It reads every packet block the default engine reads -- the Enhanced, the
Simple and the (obsolete) Packet Block -- and skips every other block
without a warning. `DPKT`_'s own reader skips the Simple Packet Block too.

.. _DPKT: https://dpkt.readthedocs.io

Expand Down
69 changes: 49 additions & 20 deletions pcapkit/toolkit/dpkt.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,9 @@ class RecordTimestamp:
resolution and its original length would be lost between the reader and
:func:`packet2frame`. The engine's readers yield a :class:`Timestamp` or a
:class:`DecimalTimestamp` instead, which *is* the value `DPKT`_ yields --
so reassembly and flow labels see exactly what they did before -- and also
carries those three.
so a stored frame's timestamp is exactly what it was before -- and also
carries those three. Reassembly and flow tracing are handed a plain
:obj:`float` of it, as by every other engine (:issue:`1501`).

.. _DPKT: https://dpkt.readthedocs.io

Expand Down Expand Up @@ -395,6 +396,30 @@ def _layer2bytes(packet: 'Packet', layer: 'Packet') -> 'tuple[bytes, bool]':
return packet2bytes(layer), False


def _network(packet: 'Packet') -> 'Optional[IP | IP6]':
"""Fetch the network layer of a DPKT packet.

Args:
packet: DPKT packet, i.e. the outermost layer.

Returns:
The packet itself if it is an IPv4 or IPv6 packet, as `DPKT`_ reads every
frame of a raw-IP link type (:issue:`1548`); otherwise the IPv4 or IPv6
payload of its link layer, which `DPKT`_ names ``ip`` or ``ip6`` -- a
bare IP packet names neither -- or :data:`None` if it has none.

.. _DPKT: https://dpkt.readthedocs.io

"""
import dpkt # isort:skip

if isinstance(packet, (dpkt.ip.IP, dpkt.ip6.IP6)):
return packet
if hasattr(packet, 'ip'):
return cast('IP', packet.ip)
return getattr(packet, 'ip6', None)


def ipv6_hdr_len(ipv6: 'IP6') -> 'int':
"""Calculate length of headers before IPv6 Fragment header.

Expand Down Expand Up @@ -482,7 +507,9 @@ def ipv4_reassembly(packet: 'Packet', timestamp: 'float', *,
than on the packet, so it is passed in -- as :func:`tcp_traceflow`
does. A caller holding only a frame can read it back with
:func:`packet2timestamp`, which is where
:class:`~pcapkit.foundation.engines.dpkt.DPKT` leaves it.
:class:`~pcapkit.foundation.engines.dpkt.DPKT` leaves it. Handed on
as a :obj:`float`, as every engine's adapters do, though `DPKT`_ reads
a nanosecond PCAP's as a :class:`~decimal.Decimal` (:issue:`1501`).
count: Packet index. If not provided, default to ``-1``.

Returns:
Expand All @@ -507,7 +534,9 @@ def ipv4_reassembly(packet: 'Packet', timestamp: 'float', *,
:class:`pcapkit.foundation.reassembly.ipv4.IPv4`

"""
ipv4 = getattr(packet, 'ip', None) # type: Optional[IP]
network = _network(packet)
# an IPv6 layer is the one with extension headers, as in ``_wire_hdr_len``
ipv4 = None if hasattr(network, 'extension_hdrs') else network
if ipv4 is not None:
if ipv4.df: # dismiss not fragmented packet
return None
Expand Down Expand Up @@ -537,7 +566,7 @@ def ipv4_reassembly(packet: 'Packet', timestamp: 'float', *,
tl=ipv4.len, # total length, header includes
header=wire[:ihl], # raw bytes type header
payload=bytearray(wire[ihl:]), # raw bytearray type payload
timestamp=timestamp, # capture timestamp
timestamp=float(timestamp), # capture timestamp
)
return data
return None
Expand All @@ -551,6 +580,7 @@ def ipv6_reassembly(packet: 'Packet', timestamp: 'float', *,
packet: DPKT packet.
timestamp: Capture timestamp of the packet, which drives the reassembly
timeout; :func:`packet2timestamp` reads it back off a stored frame.
Handed on as a :obj:`float`, as for :func:`ipv4_reassembly`.
count: Packet index. If not provided, default to ``-1``.

Returns:
Expand All @@ -575,7 +605,8 @@ def ipv6_reassembly(packet: 'Packet', timestamp: 'float', *,
:class:`pcapkit.foundation.reassembly.ipv6.IPv6`

"""
ipv6 = getattr(packet, 'ip6', None) # type: Optional[IP6]
network = _network(packet)
ipv6 = cast('IP6', network) if hasattr(network, 'extension_hdrs') else None
if ipv6 is not None:
ipv6_frag = ipv6.extension_hdrs.get(44) # type: Optional[IP6FragmentHeader]
if ipv6_frag is None: # dismiss not fragmented packet
Expand Down Expand Up @@ -618,7 +649,7 @@ def ipv6_reassembly(packet: 'Packet', timestamp: 'float', *,
tl=hdr_len + len(payload), # total length, header includes
header=wire[:hdr_len], # raw bytes type header before IPv6-Frag
payload=bytearray(payload), # raw bytearray type payload after IPv6-Frag
timestamp=timestamp, # capture timestamp
timestamp=float(timestamp), # capture timestamp
)
return data
return None
Expand All @@ -632,6 +663,7 @@ def tcp_reassembly(packet: 'Packet', timestamp: 'float', *,
packet: DPKT packet.
timestamp: Capture timestamp of the packet, which drives the reassembly
timeout; :func:`packet2timestamp` reads it back off a stored frame.
Handed on as a :obj:`float`, as for :func:`ipv4_reassembly`.
count: Packet index. If not provided, default to ``-1``.

Returns:
Expand All @@ -655,11 +687,8 @@ def tcp_reassembly(packet: 'Packet', timestamp: 'float', *,
:class:`pcapkit.foundation.reassembly.tcp.TCP`

"""
if hasattr(packet, 'ip'):
ip = cast('IP', packet.ip)
elif hasattr(packet, 'ip6'):
ip = cast('IP6', packet.ip6)
else:
ip = _network(packet)
if ip is None:
return None

tcp = getattr(ip, 'tcp', None) # type: Optional[TCP]
Expand Down Expand Up @@ -688,7 +717,7 @@ def tcp_reassembly(packet: 'Packet', timestamp: 'float', *,
first=tcp.seq, # first sequence number of payload
last=tcp.seq + raw_len - 1, # last sequence number of payload
len=raw_len, # payload length, header excludes
timestamp=timestamp, # capture timestamp
timestamp=float(timestamp), # capture timestamp
)
return data
return None
Expand All @@ -700,7 +729,10 @@ def tcp_traceflow(packet: 'Packet', timestamp: 'float', *,

Args:
packet: DPKT packet.
timestamp: Timestamp of the packet.
timestamp: Timestamp of the packet, handed on as a :obj:`float`, as for
:func:`ipv4_reassembly`. The PCAP frame record is built from the
exact timestamp attached by :func:`attach_timestamp`, if any; see
:func:`packet2frame`.
data_link: Data link layer protocol (from global header).
count: Packet index. If not provided, default to ``-1``.

Expand All @@ -725,11 +757,8 @@ def tcp_traceflow(packet: 'Packet', timestamp: 'float', *,
:class:`pcapkit.foundation.traceflow.tcp.TCP`

"""
if hasattr(packet, 'ip'):
ip = cast('IP', packet.ip)
elif hasattr(packet, 'ip6'):
ip = cast('IP6', packet.ip6)
else:
ip = _network(packet)
if ip is None:
return None

tcp = getattr(ip, 'tcp', None) # type: Optional[TCP]
Expand All @@ -750,7 +779,7 @@ def tcp_traceflow(packet: 'Packet', timestamp: 'float', *,
dst=ipaddress.ip_address(ip.dst), # destination IP
srcport=tcp.sport, # TCP source port
dstport=tcp.dport, # TCP destination port
timestamp=timestamp, # timestamp
timestamp=float(timestamp), # timestamp
seq=tcp.seq, # TCP sequence number
ack=tcp.ack, # TCP acknowledgement number
header=wire[:tcp.off * 4], # raw bytes type header
Expand Down
Loading
Loading