Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 16 additions & 20 deletions pcapkit/foundation/engines/dpkt.py
Original file line number Diff line number Diff line change
Expand Up @@ -189,18 +189,16 @@ def _read_simple(self, buf: 'bytes') -> 'tuple[float, bytes]':
"""Decode a Simple Packet Block, against the section's first interface.

`DPKT`_ has no type for this block, so it is parsed by
:class:`~pcapkit.protocols.misc.pcapng.PCAPNG`, as for the default
engine: the packet data is ``min(original_len, snaplen)`` octets, with
the snaplen of interface 0.
:class:`~pcapkit.protocols.misc.pcapng.PCAPNG`: the packet data is
``min(original_len, snaplen)`` octets, with the snaplen of interface 0.

Args:
buf: Whole block, as read from the file.

Returns:
Timestamp and packet data, as :meth:`_read_packet` returns them. A
Simple Packet Block carries no timestamp, so the packet is dated at
the UNIX epoch, as the default engine dates it; the original length
is the block's own.
the UNIX epoch; the original length is the block's own.

Raises:
FormatError: If the section describes no interface. A Simple Packet
Expand Down Expand Up @@ -293,8 +291,8 @@ def __iter__(self) -> 'Iterator[tuple[float, bytes]]':
if nanosecond:
yield DecimalTimestamp(value, resolution, hdr.len), buf
else:
# at the default engine's precision, not the caller's, which
# may be too low to hold it (:issue:`1668`)
# at a precision of 64 digits, not the caller's, which may
# be too low to hold it (:issue:`1668`)
with localcontext(prec=64):
exact = decimal.Decimal(hdr.tv_sec) + decimal.Decimal(hdr.tv_usec) / resolution
yield Timestamp(value, exact, resolution, hdr.len), buf
Expand All @@ -305,17 +303,16 @@ def __iter__(self) -> 'Iterator[tuple[float, bytes]]':
@functools.lru_cache(maxsize=None)
def _loopback_protocol(base: 'Type[DPKTPacket]', ip: 'Type[DPKTPacket]', ip6: 'Type[DPKTPacket]',
linktype: 'int') -> 'Type[DPKTPacket]':
"""A :class:`dpkt.loopback.Loopback` that reads as the default engine's.
"""A :class:`dpkt.loopback.Loopback` that reads the address family by link type.

It reads the address family in the byte order
:func:`~pcapkit.protocols.link.loopback.family_byteorder` gives for
``linktype`` -- network byte order for ``LINKTYPE_LOOP``, inferred for
``LINKTYPE_NULL`` -- and dispatches on the same families as
:class:`~pcapkit.protocols.link.loopback.Loopback`. Unlike `DPKT`_'s own, it
names an IPv4 or IPv6 payload ``ip`` or ``ip6``, as
:class:`dpkt.ethernet.Ethernet` does and as :mod:`pcapkit.toolkit.dpkt`
looks for it, and keeps a payload of any other family as raw octets rather
than parsing it as Ethernet (:issue:`1574`).
``LINKTYPE_NULL`` -- and dispatches on families 2 (IPv4) and 24, 28 and 30
(IPv6). Unlike `DPKT`_'s own, it names an IPv4 or IPv6 payload ``ip`` or
``ip6``, as :class:`dpkt.ethernet.Ethernet` does and as
:mod:`pcapkit.toolkit.dpkt` looks for it, and keeps a payload of any other
family as raw octets rather than parsing it as Ethernet (:issue:`1574`).

Args:
base: :class:`dpkt.loopback.Loopback` itself.
Expand Down Expand Up @@ -361,9 +358,9 @@ class DPKT(EngineBase['DPKTPacket']):
:class:`dpkt.pcapng.Reader`, so each packet takes the link type and the
timestamp resolution and offset of the interface it was captured on.

It reads every packet block the default engine reads -- the Enhanced, the
Simple and the (obsolete) Packet Block -- and skips every other block
without a warning. `DPKT`_'s own reader skips the Simple Packet Block too.
It reads the Enhanced, the Simple and the (obsolete) Packet Block, and
skips every other block without a warning. `DPKT`_'s own reader skips the
Simple Packet Block too.

.. _DPKT: https://dpkt.readthedocs.io

Expand Down Expand Up @@ -500,9 +497,8 @@ def read_frame(self) -> 'DPKTPacket':
# ``frag_off`` from whatever header follows an IPv6 Fragment header,
# so a Fragment header followed by ESP or Destination Options raises
# :exc:`AttributeError` (:issue:`1351`). The frame is kept as a raw
# packet, so one undecodable frame does not end the extraction and
# the frame numbering still matches the other engines, which fall
# back to a raw payload in the same way.
# packet, so one undecodable frame neither ends the extraction nor
# shifts the numbering of the frames after it.
warn(f'Frame {ext._frnum + 1}: dpkt cannot parse the frame '
f'({type(exc).__name__}: {exc}); kept as raw data',
DPKTWarning, stacklevel=stacklevel())
Expand Down
2 changes: 1 addition & 1 deletion pcapkit/foundation/engines/pypcapfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -400,7 +400,7 @@ def _decode(self, packet: 'PCAPFilePacket', frnum: 'int') -> 'PCAPFilePacket':

"""
# A zero-length record holds nothing to decode, which is not a decoding
# failure: the ``default`` engine reads one without a warning too.
# failure.
if self._declf is None or not packet.packet:
return packet

Expand Down
8 changes: 4 additions & 4 deletions pcapkit/foundation/engines/pyshark.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,8 @@ def _is_packet(packet: 'PySharkPacket') -> 'bool':

:program:`tshark` numbers every record it reads, not only packets: a PCAP-NG
systemd Journal Export Block, Custom Block or Sysdig Event Block is a frame
to it, measured with :program:`tshark` 4.6.9. The default engine counts
Enhanced, Simple and obsolete Packet Blocks only.
to it, measured with :program:`tshark` 4.6.9. This engine counts packet
records only.

The marker is ``frame.encap_type`` on the record's :func:`_frame_layer`.
Wireshark adds it for ``REC_TYPE_PACKET`` records only:
Expand Down Expand Up @@ -273,8 +273,8 @@ def read_frame(self) -> 'PySharkPacket':
"""Read frames with PyShark engine.

A tshark record that is not a packet, such as a PCAP-NG systemd Journal
Export Block, is skipped. Frames are numbered by count, as the default
engine numbers them, not by tshark's frame number.
Export Block, is skipped. Frames are numbered by count, not by tshark's
frame number.

Returns:
Parsed frame instance.
Expand Down
5 changes: 2 additions & 3 deletions pcapkit/foundation/extraction.py
Original file line number Diff line number Diff line change
Expand Up @@ -964,9 +964,8 @@ def record_header(self) -> 'EngineBase[_P]':
:class:`~pcapkit.foundation.engines.dpkt.DPKT`,
:class:`~pcapkit.foundation.engines.scapy.Scapy` and
:class:`~pcapkit.foundation.engines.pyshark.PyShark` -- which call it at
setup, so their output opens with the same header record as the
built-in engines' and :attr:`format` is set even for a capture with no
frames.
setup, so their output opens with the header record dumped above and
:attr:`format` is set even for a capture with no frames.

Raises:
FormatError: If the input is neither a PCAP nor a PCAP-NG file.
Expand Down
23 changes: 11 additions & 12 deletions pcapkit/foundation/reassembly/ip.py
Original file line number Diff line number Diff line change
Expand Up @@ -182,18 +182,17 @@ def reassembly(self, info: 'Packet[_AT]') -> 'None':
#
# A Total Length of 0 declares nothing. It is what TCP segmentation
# offload leaves in a capture taken on the sending host, and the
# datagram then runs to the end of the captured frame -- which is the
# payload every adapter hands over (:issue:`1547`). Wireshark ("presumed
# TSO") substitutes that length for the field before it reassembles
# anything, and so does this, whatever **MF** and the offset say: an
# unfragmented one is then whole rather than ``-IHL`` octets long
# (:issue:`1555`). Linux's BIG TCP zeroes the field precisely because
# the datagram, header included, outgrows the 65535 octets it can
# declare -- and such a datagram never completes in a buffer of that
# length (:meth:`_is_complete`). So the buffer grows past it, to hold
# the datagram rather than clip it, ``RCVBT`` in step with it; it grows
# by an octet even when the data alone would fit, since its growing is
# what lifts the bound.
# datagram then runs to the end of the captured frame (:issue:`1547`).
# Wireshark ("presumed TSO") substitutes that length for the field
# before it reassembles anything, and so does this, whatever **MF** and
# the offset say: an unfragmented one is then whole rather than ``-IHL``
# octets long (:issue:`1555`). Linux's BIG TCP zeroes the field
# precisely because the datagram, header included, outgrows the 65535
# octets it can declare -- and such a datagram never completes in a
# buffer of that length (:meth:`_is_complete`). So the buffer grows past
# it, to hold the datagram rather than clip it, ``RCVBT`` in step with
# it; it grows by an octet even when the data alone would fit, since its
# growing is what lifts the bound.
#
# With nothing declared, a snaplen cut cannot be seen in the length. A
# non-final fragment's cut still shows, as a partial last block, which
Expand Down
33 changes: 15 additions & 18 deletions pcapkit/toolkit/dpkt.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,9 +83,9 @@ class RecordTimestamp:
:class:`DecimalTimestamp` instead, which *is* the value `DPKT`_ yields --
so a stored frame's timestamp is exactly what it was before -- and also
carries those three. Reassembly and flow tracing are handed a plain
:obj:`float` of it, as by every other engine (:issue:`1501`): of its exact
value, since `DPKT`_'s own for a microsecond PCAP rounds twice
(:issue:`1668`; see :func:`_capture_time`).
:obj:`float` of it (:issue:`1501`): of its exact value, since `DPKT`_'s
own for a microsecond PCAP rounds twice (:issue:`1668`; see
:func:`_capture_time`).

A copy, a deep copy and a pickle at any protocol all rebuild a timestamp
from what its :meth:`~object.__reduce__` returns, so all keep the three.
Expand Down Expand Up @@ -225,9 +225,9 @@ def _capture_time(timestamp: 'float | Decimal') -> 'float':
timestamp: Capture timestamp of the packet.

Returns:
The :obj:`float` nearest the exact timestamp, as the default engine hands
over. `DPKT`_ reads a microsecond PCAP's as ``tv_sec + tv_usec / 1e6``,
which rounds twice, so can be an ulp off at a small ``tv_sec``:
The :obj:`float` nearest the exact timestamp. `DPKT`_ reads a
microsecond PCAP's as ``tv_sec + tv_usec / 1e6``, which rounds twice,
so can be an ulp off at a small ``tv_sec``:
``1.0036909999999999`` for ``1.003691`` (:issue:`1668`). A
:class:`RecordTimestamp` is therefore read by its exact value. Any other
timestamp -- one passed by hand, or yielded by a ``dpkt`` whose reader
Expand Down Expand Up @@ -296,9 +296,9 @@ def packet2frame(packet: 'Packet', timestamp: 'float | Decimal', *,

Note:
The fraction in ``frame_info.ts_usec`` is in the capture's own resolution,
as the default engine's is, and both it and ``orig_len`` come from the
reader: :class:`~pcapkit.foundation.engines.dpkt.DPKT` reads PCAP and
PCAP-NG alike into a :class:`RecordTimestamp`. A timestamp that is not one
and both it and ``orig_len`` come from the reader:
:class:`~pcapkit.foundation.engines.dpkt.DPKT` reads PCAP and PCAP-NG
alike into a :class:`RecordTimestamp`. A timestamp that is not one
-- a packet built by hand, or read by a ``dpkt`` whose reader the engine
could not extend -- is read as `DPKT`_ yields it: a
:class:`~decimal.Decimal` counts nanoseconds and a :obj:`float`
Expand Down Expand Up @@ -565,7 +565,7 @@ def ipv4_reassembly(packet: 'Packet', timestamp: 'float', *,
does. A caller holding only a frame can read it back with
:func:`packet2timestamp`, which is where
:class:`~pcapkit.foundation.engines.dpkt.DPKT` leaves it. Handed on
as a :obj:`float`, as every engine's adapters do, though `DPKT`_ reads
as a :obj:`float`, though `DPKT`_ reads
a nanosecond PCAP's as a :class:`~decimal.Decimal` (:issue:`1501`),
and as the one nearest the exact value it carries, if any, where
`DPKT`_'s own may be an ulp off (:issue:`1668`; see
Expand Down Expand Up @@ -706,9 +706,7 @@ def ipv6_reassembly(packet: 'Packet', timestamp: 'float', *,
# (:rfc:`8200#section-4.5`), not the IPv6 header's Flow Label. The
# label is optional and routinely zero, so keying on it collapses
# every datagram between one address pair into a single buffer and
# interleaves their fragments; it would also disagree with
# ``bufid[2]`` in every other engine, which feeds
# :attr:`pcapkit.foundation.reassembly.data.ip.DatagramID.id`.
# interleaves their fragments.
ipv6_frag.id, # identification
Enum_TransType.get(ipv6_frag.nxt), # next header field in IPv6 Fragment Header
),
Expand Down Expand Up @@ -742,11 +740,10 @@ def _tcp_flags(tcp: 'TCP') -> 'tuple[bool, bool, bool]':

Note:
Each flag is masked out of :attr:`dpkt.tcp.TCP.flags` by its ``TH_*``
value, so it is read from the flags octet alone, as the default engine
reads it. `DPKT`_ 1.9.8 makes that field nine bits wide, its top bit
the NS bit (:rfc:`3540`): read by its position among the field's binary
digits, every flag was one bit low with NS set, a SYN reading as a FIN
(:issue:`1618`).
value, so it is read from the flags octet alone. `DPKT`_ 1.9.8 makes
that field nine bits wide, its top bit the NS bit (:rfc:`3540`): read by
its position among the field's binary digits, every flag was one bit low
with NS set, a SYN reading as a FIN (:issue:`1618`).

.. _DPKT: https://dpkt.readthedocs.io

Expand Down
23 changes: 8 additions & 15 deletions pcapkit/toolkit/pyshark.py
Original file line number Diff line number Diff line change
Expand Up @@ -353,13 +353,7 @@ def _parse_epoch(value: 'Any') -> 'float':
value: The field as PyShark reports it, decimal text.

Returns:
The timestamp, as the same :obj:`float` the default engine reports.

Note:
The default engine computes the epoch as a :class:`~decimal.Decimal`
and hands the tracer ``float()`` of it (:mod:`pcapkit.toolkit.pcap`).
This takes the same route, so both are the double nearest the same
decimal value: identical, not merely close.
The timestamp, as the :obj:`float` nearest the decimal value.

Raises:
ProtocolError: If ``value`` is not decimal text.
Expand Down Expand Up @@ -468,8 +462,7 @@ def _carrier(packet: 'Packet') -> 'tuple[Any, Any]':
Note:
The nearest is the innermost IP layer of an IP-in-IP tunnel
(:rfc:`2003`, :rfc:`2473`, :rfc:`4213`), whose addresses are the TCP
endpoints', as for the default engine
(:func:`pcapkit.toolkit.pcap.tcp_segment`). ``packet.ip`` is the first
endpoints'. ``packet.ip`` is the first
``ip`` layer instead, which is the tunnel's (:issue:`1581`).

"""
Expand All @@ -493,10 +486,10 @@ def _has_fixed_header(tcp: 'Any') -> 'bool':
Whether the layer carries ``tcp.urgent_pointer``.

Note:
The default engine traces a segment only if its 20-octet fixed header
(:rfc:`9293#section-3.1`) was captured and its Data Offset is at least
5 (:func:`pcapkit.toolkit.pcap.tcp_segment`). tshark omits fields in
both other cases, measured with tshark 4.6.9 (:issue:`1611`):
A segment is traced only if its 20-octet fixed header
(:rfc:`9293#section-3.1`) was captured and its Data Offset is at
least 5. tshark omits fields in both other cases, measured with
tshark 4.6.9 (:issue:`1611`):

* a Data Offset below 5 stops the dissection at ``tcp.hdr_len``, so
``tcp.ack`` and the ``tcp.flags`` fields are absent;
Expand Down Expand Up @@ -614,8 +607,8 @@ def tcp_traceflow(packet: 'Packet', *, count: 'int' = -1) -> 'TF_TCP_Packet | No

# A Simple Packet Block carries no timestamp, so tshark reports no
# ``frame.time_epoch`` for it; the PDML ``geninfo`` timestamp PyShark
# keeps as ``sniff_timestamp`` is then ``'0.000000000'``, which is what
# the default engine reports (measured on ``test.pcapng`` frame 4).
# keeps as ``sniff_timestamp`` is then ``'0.000000000'``, the UNIX epoch
# (measured on ``test.pcapng`` frame 4).
epoch = getattr(frame, 'time_epoch', None)
if epoch is None:
epoch = packet.sniff_timestamp
Expand Down
Loading
Loading