diff --git a/docs/source/ext.rst b/docs/source/ext.rst index 30c83a2f87..4e2c4a3480 100644 --- a/docs/source/ext.rst +++ b/docs/source/ext.rst @@ -422,7 +422,9 @@ The available engines and the file formats they read: matters is that the gap is *announced* -- each engine warns, or raises, for the capability it cannot provide, rather than silently producing an empty result, save that `PyPCAPFile`_ leaves TCP behind a VLAN tag, in a tunnel or behind an - extension header (an IPv6 one, or AH in IPv4) out without a warning, and the + extension header (an IPv6 one, or AH in IPv4) out without a warning; that + `DPKT`_ leaves TCP in an IP-in-IP tunnel or behind an AH or IPv6 extension + header in IPv4, and IPv4 in IPv6 or IPv6 in IPv4, out without one; and the `Scapy`_ engine reads IPv6 only as far as `Scapy`_ dissects it, which leaves IPv6 behind some BSD loopback headers, and TCP and IPv6 fragments behind some IPv6 extension headers, out without one. See diff --git a/docs/source/index.rst b/docs/source/index.rst index 6394480f43..f4e102765c 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -150,10 +150,13 @@ so IPv6 reassembly is unavailable. Each gap is announced with a warning or an exception rather than silently returning nothing, save that :class:`~pcapkit.foundation.engines.pypcapfile.PyPCAPFile` leaves TCP behind a VLAN tag, in a tunnel or behind an extension header (an IPv6 one, or AH in IPv4) -out without one, and :class:`~pcapkit.foundation.engines.scapy.Scapy` reads IPv6 -only as far as `Scapy`_ dissects it, which leaves IPv6 behind some BSD loopback -headers, and TCP and IPv6 fragments behind some IPv6 extension headers, out -without one; :doc:`pcapkit/foundation/engines/index` tabulates them. +out without one; that :class:`~pcapkit.foundation.engines.dpkt.DPKT` leaves +TCP in an IP-in-IP tunnel or behind an AH or IPv6 extension header in IPv4, and +IPv4 in IPv6 or IPv6 in IPv4, out without one; and +:class:`~pcapkit.foundation.engines.scapy.Scapy` reads IPv6 only as far as +`Scapy`_ dissects it, which leaves IPv6 behind some BSD loopback headers, and +TCP and IPv6 fragments behind some IPv6 extension headers, out without one. +:doc:`pcapkit/foundation/engines/index` tabulates them. Every engine also answers a preflight check -- :meth:`~pcapkit.foundation.engines.engine.EngineBase.unsupported_reason`, which diff --git a/docs/source/pcapkit/foundation/engines/index.rst b/docs/source/pcapkit/foundation/engines/index.rst index a673c3adac..2d4c4171a2 100644 --- a/docs/source/pcapkit/foundation/engines/index.rst +++ b/docs/source/pcapkit/foundation/engines/index.rst @@ -101,6 +101,14 @@ their row says otherwise: | | PCAP savefiles only, otherwise | | | :exc:`~pcapkit.utilities.exceptions.FormatError` | +-----------------------------------------------------------------+---------------------------------------------------------------+ +| :class:`~pcapkit.foundation.engines.dpkt.DPKT` | TCP in an IP-in-IP tunnel, or behind an AH or IPv6 extension | +| | header carried in IPv4, is left out of TCP reassembly and | +| | flow tracing, and IPv4 carried in IPv6, or IPv6 in IPv4, out | +| | of IP reassembly, all without a warning; a frame `DPKT`_ | +| | cannot decode is kept raw, with a | +| | :class:`~pcapkit.utilities.warnings.DPKTWarning`, and is | +| | neither reassembled nor traced | ++-----------------------------------------------------------------+---------------------------------------------------------------+ | :class:`~pcapkit.foundation.engines.scapy.Scapy` | reads IPv6 only as far as `Scapy`_ dissects it, without a | | | warning: behind a BSD loopback header only at the family | | | Scapy binds it to on the host it runs on, and no TCP or | diff --git a/docs/source/pcapkit/protocols/internet/internet.rst b/docs/source/pcapkit/protocols/internet/internet.rst index 0b5475b3fc..e1c313ba46 100644 --- a/docs/source/pcapkit/protocols/internet/internet.rst +++ b/docs/source/pcapkit/protocols/internet/internet.rst @@ -37,11 +37,12 @@ A chain of IPv6 extension headers is dissected to at most a row, whether it follows IPv6 or another layer such as IPv4. What follows is kept as :class:`~pcapkit.protocols.misc.raw.Raw`, with one :exc:`~pcapkit.utilities.warnings.ProtocolWarning` (:issue:`1604`). The -``dpkt`` toolkit stops at the same depth. The ``pyshark`` engine leaves the walk -to its own parser, which does not stop there, so it may still find the upper -layer past it (:issue:`1609`). So does the ``scapy`` engine behind IPv6, where -Scapy dissects each extension header itself; behind IPv4, its toolkit steps over -them in the octets to find TCP, and stops at the same depth. The ``pypcapfile`` -toolkit walks no extension header chain, so it finds no upper layer behind one. +``pyshark`` engine leaves the walk to its own parser, which does not stop there, +so it may still find the upper layer past it (:issue:`1609`). So may the +``dpkt`` toolkit, which reads the chain as :mod:`dpkt` decodes it. So does the +``scapy`` engine behind IPv6, where Scapy dissects each extension header itself; +behind IPv4, its toolkit steps over them in the octets to find TCP, and stops at +the same depth. The ``pypcapfile`` toolkit walks no extension header chain, so +it finds no upper layer behind one. .. autodata:: pcapkit.protocols.internet.internet.EXTENSION_HEADER_LIMIT diff --git a/docs/source/pcapkit/protocols/protocol.rst b/docs/source/pcapkit/protocols/protocol.rst index b2e58af056..ad3131dc40 100644 --- a/docs/source/pcapkit/protocols/protocol.rst +++ b/docs/source/pcapkit/protocols/protocol.rst @@ -118,8 +118,7 @@ starts at the first layer it dissects. One chain of extension headers is also bo its own, by :data:`~pcapkit.protocols.internet.internet.EXTENSION_HEADER_LIMIT`. The ``pyshark``, ``scapy`` and ``pypcapfile`` engines dissect with their own parsers, which do not apply the frame's layer bound (:issue:`1609`). The -``dpkt`` toolkit counts the layers it walks as the default engine does, and -stops at it. +``dpkt`` toolkit reads what :mod:`dpkt` decodes, which applies neither bound. .. autodata:: pcapkit.protocols.protocol.FRAME_LAYER_LIMIT diff --git a/pcapkit/foundation/engines/dpkt.py b/pcapkit/foundation/engines/dpkt.py index 3e6b48607d..557b933a12 100644 --- a/pcapkit/foundation/engines/dpkt.py +++ b/pcapkit/foundation/engines/dpkt.py @@ -479,9 +479,9 @@ def read_frame(self) -> 'DPKTPacket': for more operational information. """ - from pcapkit.toolkit.dpkt import (attach_buffer, attach_linktype, attach_timestamp, - ipv4_reassembly, ipv6_reassembly, packet2dict, - tcp_reassembly, tcp_traceflow) + from pcapkit.toolkit.dpkt import (attach_buffer, attach_timestamp, ipv4_reassembly, + ipv6_reassembly, packet2dict, tcp_reassembly, + tcp_traceflow) ext = self._extractor reader = self._extmp @@ -517,9 +517,6 @@ def read_frame(self) -> 'DPKTPacket': # it again recomputes zeroed checksums and lengths into it, so keep those # too for the adapters to slice the wire octets out of. attach_buffer(packet, pkt) - # And a frame kept as raw data does not say what link layer its octets - # start with, which the adapters read them by (#1626). - attach_linktype(packet, linktype) # verbose output ext._frnum += 1 diff --git a/pcapkit/toolkit/dpkt.py b/pcapkit/toolkit/dpkt.py index ada095f8db..877fe15bbb 100644 --- a/pcapkit/toolkit/dpkt.py +++ b/pcapkit/toolkit/dpkt.py @@ -15,39 +15,31 @@ import copy import decimal import ipaddress -from typing import TYPE_CHECKING, NamedTuple, cast +from typing import TYPE_CHECKING, cast -from pcapkit.const.reg.ethertype import EtherType as Enum_EtherType -from pcapkit.const.reg.linktype import LinkType as Enum_LinkType from pcapkit.const.reg.transtype import TransType as Enum_TransType from pcapkit.foundation.reassembly.data.ip import Packet as IP_Packet from pcapkit.foundation.reassembly.data.tcp import Packet as TCP_Packet from pcapkit.foundation.traceflow.data.data import FrameRecord from pcapkit.foundation.traceflow.data.tcp import Packet as TF_TCP_Packet from pcapkit.protocols.data.misc.pcap.frame import FrameInfo as Data_FrameInfo -from pcapkit.protocols.internet.internet import EXTENSION_HEADER_LIMIT -from pcapkit.protocols.link.loopback import family_byteorder -from pcapkit.protocols.protocol import FRAME_LAYER_LIMIT -from pcapkit.protocols.schema.internet.ipv6 import jumbo_payload_length -from pcapkit.toolkit.pypcapfile import (IPV4_EXTENSION_HEADERS, IPV4_HEADER_LEN, - IPV6_EXTENSION_HEADERS, IPV6_HEADER_LEN, TCP_MIN_HEADER_LEN, - VLAN_TAG_LEN, VLAN_TAGS, _default_accepts, - _walk_extension_headers) -from pcapkit.utilities.exceptions import ProtocolError, UnsupportedCall +from pcapkit.utilities.exceptions import UnsupportedCall if TYPE_CHECKING: from decimal import Decimal from ipaddress import IPv4Address, IPv6Address - from typing import Any, Iterator, Optional + from typing import Any, Optional from dpkt.dpkt import Packet - from dpkt.ethernet import Ethernet from dpkt.ip import IP - from dpkt.ip6 import IP6 + from dpkt.ip6 import IP6, IP6FragmentHeader + from dpkt.tcp import TCP + + from pcapkit.const.reg.linktype import LinkType as Enum_LinkType __all__ = [ 'RecordTimestamp', 'Timestamp', 'DecimalTimestamp', 'ipv6_hdr_len', 'attach_timestamp', 'packet2timestamp', 'attach_buffer', 'packet2bytes', - 'attach_linktype', 'packet2frame', 'packet2chain', 'packet2dict', + 'packet2frame', 'packet2chain', 'packet2dict', 'ipv4_reassembly', 'ipv6_reassembly', 'tcp_reassembly', 'tcp_traceflow' ] @@ -80,54 +72,6 @@ #: .. _DPKT: https://dpkt.readthedocs.io BUFFER_ATTR = '__pcapkit_buffer__' -#: Attribute a frame's data link type is stashed under, as the capture's -#: ``LINKTYPE_*`` number, by :func:`attach_linktype`. -LINKTYPE_ATTR = '__pcapkit_linktype__' - -#: IP protocol numbers of the IP-in-IP tunnels the default engine dissects -#: (see :attr:`Internet.__proto__ -#: `), over IPv4 and -#: IPv6 alike: IPv4 (protocol 4, :rfc:`2003`) and IPv6 (protocol 41, -#: :rfc:`2473` and :rfc:`4213`). `DPKT`_ also decodes an IPv4 packet's -#: protocol 0 as IPv4, its ``IP_PROTO_IP``, where the default engine reads an -#: IPv6 Hop-by-Hop Options header, so :func:`_ip_layers` does not enter it. -#: -#: .. _DPKT: https://dpkt.readthedocs.io -IP_TUNNELS = frozenset((int(Enum_TransType.IPv4), int(Enum_TransType.IPv6))) - -#: Length of an Ethernet II header: two MAC addresses and the EtherType. -ETHERNET_HEADER_LEN = 14 - -#: Length of a BSD loopback header: the address family. -LOOPBACK_HEADER_LEN = 4 - -#: EtherTypes of the IP layers the default engine dissects over Ethernet, each -#: mapped to its IP version (see :attr:`Link.__proto__ -#: `). -ETHERTYPE_IP_VERSIONS = { - Enum_EtherType.Internet_Protocol_version_4: 4, - Enum_EtherType.Internet_Protocol_version_6: 6, -} # type: dict[int, int] - -#: Address families of the IP layers the default engine dissects over BSD -#: loopback, each mapped to its IP version (see :attr:`Loopback.__proto__ -#: `). -LOOPBACK_IP_VERSIONS = {2: 4, 24: 6, 28: 6, 30: 6} # type: dict[int, int] - -#: Destination address prefixes by which `DPKT`_ reads an Ethernet frame whose -#: EtherType field is an 802.3 length, at most 1500, as Cisco ISL; the default -#: engine reads one as raw octets (see :func:`_ethernet_accepted`). -#: -#: .. _DPKT: https://dpkt.readthedocs.io -ISL_ADDRESSES = (b'\x01\x00\x0c\x00\x00', b'\x03\x00\x0c\x00\x00') - -#: TPIDs of the VLAN tags `DPKT`_ steps over in an Ethernet II frame: 802.1Q, -#: 802.1ad and the legacy QinQ ``0x9100`` and ``0x9200``, i.e. -#: ``dpkt.ethernet._ETH_TYPES_QINQ`` (see :func:`_vlan_tpids`). -#: -#: .. _DPKT: https://dpkt.readthedocs.io -DPKT_VLAN_TAGS = frozenset((0x8100, 0x88A8, 0x9100, 0x9200)) - class RecordTimestamp: """A record's capture timestamp, with what a PCAP frame record needs of it. @@ -311,26 +255,6 @@ def attach_buffer(packet: 'Packet', buffer: 'bytes') -> 'None': setattr(packet, BUFFER_ATTR, bytes(buffer)) -def attach_linktype(packet: 'Packet', linktype: 'int') -> 'None': - """Stash the data link type of a frame's capture on the frame. - - Args: - packet: DPKT packet. - linktype: ``LINKTYPE_*`` number: the PCAP global header's, or the - PCAP-NG interface's the packet was captured on. - - Note: - A frame `DPKT`_ cannot decode is kept as raw octets (see - :meth:`DPKT.read_frame `), - which do not say what link layer they start with. The adapters read its - layers off those octets by this (see :func:`_link_network`). - - .. _DPKT: https://dpkt.readthedocs.io - - """ - setattr(packet, LINKTYPE_ATTR, int(linktype)) - - def packet2bytes(packet: 'Packet') -> 'bytes': """Fetch the octets of a DPKT packet without modifying it. @@ -434,139 +358,90 @@ def _ipv6_ext_hdrs(ipv6: 'IP6') -> 'list[Any]': return list(getattr(ipv6, 'extension_hdrs', {}).values()) -def _vlan_tpids(wire: 'bytes | memoryview') -> 'list[int]': - """Read the TPIDs of the VLAN tags `DPKT`_ steps over in an Ethernet II frame, off its octets. +def _wire_hdr_len(packet: 'Packet', payload: 'Packet') -> 'int': + """Calculate the octets a layer occupies before its payload begins. Args: - wire: Octets of an Ethernet frame, from its destination address on. + packet: DPKT packet, i.e. one layer of a frame. + payload: That layer's payload, i.e. :attr:`packet.data `. Returns: - The frame's EtherType, then each tag's but the last's, for as long as - each is one of the :data:`DPKT_VLAN_TAGS` and a whole tag follows it; - empty for a frame whose EtherType field is an 802.3 length. - - Note: - :class:`dpkt.ethernet.Ethernet` reads at most two tags at a time, and - reads on only while the tag it read names 802.1Q. Where it then has no - decoder for the EtherType, its constructor reads again from there, and - starts its :attr:`~dpkt.ethernet.Ethernet.vlan_tags` afresh: so it keeps - only the tags of its last read, one for 802.1Q then 802.1ad, two - 802.1ad, or three tags, and its re-serialisation is 4 octets short for - each tag it dropped. Where it decodes the payload, the tags it stepped - over are all those heading it, since none of the :data:`DPKT_VLAN_TAGS` - has a decoder, so they are read off the octets instead. + The advance :meth:`~dpkt.dpkt.Packet.unpack` itself made to reach the + payload. + + For IPv6 that is the fixed header plus each extension header's + :attr:`~dpkt.ip6.IP6ExtensionHeader.length`, which is what + :meth:`dpkt.ip6.IP6.unpack` walks the buffer by. It cannot be taken from + ``len()``: :meth:`dpkt.ip6.IP6.__len__` sums ``len()`` of the extension + headers instead, and that is 12 octets plus the *rest of the frame* for + an Authentication header -- :class:`dpkt.ip6.IP6AHHeader` trims only into + :attr:`~dpkt.ip6.IP6AHHeader.auth_data` and leaves + :attr:`~dpkt.dpkt.Packet.data` running to the end of the buffer -- and 8 + octets short of the wire for a Routing header whose Hdr Ext Len is odd + (:class:`dpkt.ip6.IP6RoutingHeader` keeps whole addresses only). + + Every other layer's ``len()`` does bound its payload, so the difference + of the two is used there. For IPv4 that difference is the Internet + Header Length, options included. An Ethernet frame's falls short of the + wire where `DPKT`_ dropped some of its VLAN tags, which + :func:`_layer2bytes` then finds (:issue:`1646`). .. _DPKT: https://dpkt.readthedocs.io """ - tpids = [] # type: list[int] - offset = 12 - while len(wire) >= offset + 2 + VLAN_TAG_LEN: - tpid = int.from_bytes(wire[offset:offset + 2], 'big') - if tpid not in DPKT_VLAN_TAGS: - break - tpids.append(tpid) - offset += VLAN_TAG_LEN - return tpids + if hasattr(packet, 'extension_hdrs'): + return packet.__hdr_len__ + sum( + ext_hdr.length for ext_hdr in _ipv6_ext_hdrs(cast('IP6', packet)) + ) + return len(packet) - len(payload) -def _network_octets(packet: 'Packet', ip: 'IP | IP6') -> 'bytes': - """Fetch the octets of a DPKT packet's network layer, from its first octet on. +def _layer2bytes(packet: 'Packet', layer: 'Packet') -> 'tuple[bytes, bool]': + """Fetch the octets of a layer of a DPKT packet, from its first octet on. Args: packet: DPKT packet, i.e. the outermost layer. - ip: Its network layer, as :func:`_network` finds it: ``packet`` itself, - or the payload of its link layer. + layer: A layer nested within ``packet``. Returns: - With a buffer attached by :func:`attach_buffer`, the record octets from - the layer's offset to the end of the frame, so any link-layer trailer is - still there; otherwise the layer alone, serialised from a copy (see - :func:`packet2bytes`). Behind an Ethernet II header, the offset counts - the VLAN tags off the octets, as :class:`dpkt.ethernet.Ethernet` keeps - only some of a stack it reads in two goes (see :func:`_vlan_tpids`); - behind any other link layer, it is what the layer's ``len()`` counts - before its payload. - - Raises: - ProtocolError: If the buffer does not carry the layer's own header where - the link layer placed it. `DPKT`_ unpacks a header field by field and - :meth:`~dpkt.dpkt.Packet.pack_hdr` packs those fields back without - recomputing any of them, so the two agree octet for octet wherever - the offset is right -- which makes this an exact check rather than a - plausibility one. It is raised rather than worked around because a - misplaced slice reaches the reassembler as a believable datagram and - so is invisible in its output. + The octets, and whether they are the captured ones. With a buffer + attached by :func:`attach_buffer` they are the record octets from the + layer's offset to the end of the frame, so any link-layer trailer is + still there for the caller to trim; otherwise they are the layer alone, + serialised from a copy (see :func:`packet2bytes`). + + They are that serialisation too if the buffer does not carry the + layer's own header where the walk above placed it. `DPKT`_ unpacks a + header field by field and :meth:`~dpkt.dpkt.Packet.pack_hdr` packs those + fields back without recomputing any of them, so the two agree octet for + octet wherever the offset is right -- which makes this an exact check + rather than a plausibility one. A misplaced slice would reach the + reassembler as a believable datagram, invisible in its output, so it is + never returned. `DPKT`_'s own lengths misplace the layers behind a stack + of VLAN tags it reads in two goes, such as 802.1Q then 802.1ad, as it + keeps only the last go's tags (:issue:`1646`), and behind a Cisco ISL + header followed by an 802.1Q tag, which it keeps in place of the ISL + one. .. _DPKT: https://dpkt.readthedocs.io """ - import dpkt # isort:skip - - buffer = getattr(packet, BUFFER_ATTR, None) # type: Optional[bytes] - if buffer is None: - return packet2bytes(ip) - if ip is packet: + buffer = getattr(packet, BUFFER_ATTR, None) + if buffer is not None: offset = 0 - elif isinstance(packet, dpkt.ethernet.Ethernet): - offset = ETHERNET_HEADER_LEN + VLAN_TAG_LEN * len(_vlan_tpids(buffer)) - else: - offset = len(packet) - len(ip) - wire = buffer[offset:] - if wire[:ip.__hdr_len__] != ip.pack_hdr(): - raise ProtocolError( - f'{type(ip).__name__} header not found at offset {offset} of the ' - f'{len(buffer)}-octet record; the frame cannot be sliced without re-serialising it' - ) - return wire - - -def _ethernet_accepted(packet: 'Ethernet') -> 'bool': - """Test if the default engine reads past a DPKT Ethernet frame's link layer as `DPKT`_ did. - - Args: - packet: DPKT Ethernet frame. - - Returns: - :data:`False` if `DPKT`_ reached its payload past an MPLS label stack, a - Cisco ISL header, or a VLAN tag whose TPID is neither of the - :data:`~pcapkit.toolkit.pypcapfile.VLAN_TAGS`; else :data:`True`. - - Note: - `DPKT`_ pops an MPLS label stack and guesses IPv4 or IPv6 from the - nibble after it, unwraps an ISL header, and steps over the legacy QinQ - tags ``0x9100`` and ``0x9200``, naming the IPv4 or IPv6 packet it finds - ``ip`` or ``ip6`` as for an untagged frame. The default engine reads - what follows any of them as raw octets (see :attr:`Link.__proto__ - `), an ISL frame's 802.3 - length field included (:issue:`1638`). A tag's TPID is the EtherType - before it: the frame's, then each tag's but the last's. - - `DPKT`_ records an ISL header as a tag, but unwraps it into the frame - itself, so a VLAN tag of the frame inside replaces it; the captured - octets attached by :func:`attach_buffer` still show it, by the - :data:`ISL_ADDRESSES` and length field `DPKT`_ recognises it by. Nor - does it keep every tag of a stack it reads in two goes, so the TPIDs - are read off those octets too (see :func:`_vlan_tpids`). - - .. _DPKT: https://dpkt.readthedocs.io - - """ - import dpkt # isort:skip - - if getattr(packet, 'mpls_labels', None): - return False - tags = list(getattr(packet, 'vlan_tags', None) or ()) - if any(isinstance(tag, dpkt.ethernet.VLANtagISL) for tag in tags): - return False - buffer = getattr(packet, BUFFER_ATTR, None) # type: Optional[bytes] - if buffer is None: - tpids = [packet.type, *(tag.type for tag in tags)][:len(tags)] - elif buffer[:5] in ISL_ADDRESSES and int.from_bytes(buffer[12:14], 'big') <= 1500: - return False - else: - tpids = _vlan_tpids(buffer) - return all(tpid in VLAN_TAGS for tpid in tpids) + current = packet + while current is not layer: + payload = getattr(current, 'data', None) + if payload is None or isinstance(payload, (bytes, bytearray)): + break + offset += _wire_hdr_len(current, payload) + current = payload + else: + wire = cast('bytes', buffer)[offset:] + hdr_len = getattr(layer, '__hdr_len__', 0) + if not hdr_len or wire[:hdr_len] == layer.pack_hdr(): + return wire, True + return packet2bytes(layer), False def _network(packet: 'Packet') -> 'Optional[IP | IP6]': @@ -579,9 +454,7 @@ def _network(packet: 'Packet') -> 'Optional[IP | IP6]': The packet itself if it is an IPv4 or IPv6 packet, as `DPKT`_ reads every frame of a raw-IP link type (:issue:`1548`); otherwise the IPv4 or IPv6 payload of its link layer, which `DPKT`_ names ``ip`` or ``ip6`` -- a - bare IP packet names neither -- or :data:`None` if it has none, or the - default engine dissects no payload of an Ethernet frame where `DPKT`_ - found one (see :func:`_ethernet_accepted`). + bare IP packet names neither -- or :data:`None` if it has none. .. _DPKT: https://dpkt.readthedocs.io @@ -590,494 +463,11 @@ def _network(packet: 'Packet') -> 'Optional[IP | IP6]': if isinstance(packet, (dpkt.ip.IP, dpkt.ip6.IP6)): return packet - if isinstance(packet, dpkt.ethernet.Ethernet) and not _ethernet_accepted(packet): - return None if hasattr(packet, 'ip'): return cast('IP', packet.ip) return getattr(packet, 'ip6', None) -def _network_depth(packet: 'Packet') -> 'int': - """Count the layers of a DPKT packet the default engine dissects up to its network layer. - - Args: - packet: DPKT packet, i.e. the outermost layer, whose network layer - :func:`_network` found. - - Returns: - The place of that layer in the frame, its first layer being ``1``, as - the default engine counts towards - :data:`~pcapkit.protocols.protocol.FRAME_LAYER_LIMIT`: ``1`` for the - packet of a raw-IP link type, which is the network layer itself; else - ``2``, and one more for each VLAN tag of an Ethernet frame, which the - default engine dissects as a layer of its own. - - Note: - The tags are read off the captured octets, as `DPKT`_ keeps only some - of a stack it reads in two goes (see :func:`_vlan_tpids`). Each is - 802.1Q or 802.1ad, or :func:`_ethernet_accepted` refuses the frame. - - .. _DPKT: https://dpkt.readthedocs.io - - """ - import dpkt # isort:skip - - if isinstance(packet, (dpkt.ip.IP, dpkt.ip6.IP6)): - return 1 - tags = 0 - if isinstance(packet, dpkt.ethernet.Ethernet): - buffer = getattr(packet, BUFFER_ATTR, None) # type: Optional[bytes] - if buffer is None: - tags = len(getattr(packet, 'vlan_tags', None) or ()) - else: - tags = len(_vlan_tpids(buffer)) - return 2 + tags - - -def _ipv4_header_accepted(header: 'bytes') -> 'bool': - """Test if the default engine reads an IPv4 header as an IPv4 layer. - - Args: - header: The header's captured octets, from its first 20 on to as many as - its IHL counts, or to the end of the capture if that comes first. - - Returns: - :data:`False` if the default engine's IPv4 parser rejects it -- its - version is not 4, its options are ones the parser rejects, or its IHL - runs past the captured octets (:issue:`1607`) -- as the default engine - then dissects nothing past it; else :data:`True`. - - Note: - A header of version 4 with no options is not parsed, as the parser - checks nothing else of one; any other is asked of the parser itself - (see :func:`pcapkit.toolkit.pypcapfile._default_accepts`). - - """ - return header[0] == 0x45 or _default_accepts('IPv4', bytes(header)) - - -def _ipv4_accepted(ipv4: 'IP') -> 'bool': - """Test if the default engine reads a DPKT IPv4 packet as an IPv4 layer. - - Args: - ipv4: DPKT IPv4 packet. - - Returns: - What :func:`_ipv4_header_accepted` says of its header. `DPKT`_ decodes - what an IPv4 EtherType or tunnel carries as IPv4 whatever its version, - and never checks its options. Its octets are the fields as `DPKT`_ read - them and the options as captured, which is exact. - - .. _DPKT: https://dpkt.readthedocs.io - - """ - return _ipv4_header_accepted(ipv4.pack_hdr() + bytes(ipv4.opts)) - - -class _Segment(NamedTuple): - """A TCP segment the default engine reads, keyed by the IP layer that carries it (see :func:`_tcp_carrier`).""" - - #: Source address of the IP layer. - src: 'IPv4Address | IPv6Address' - #: Destination address of the IP layer. - dst: 'IPv4Address | IPv6Address' - #: Source port. - srcport: 'int' - #: Destination port. - dstport: 'int' - #: Sequence number. - seq: 'int' - #: Acknowledgement number. - ack: 'int' - #: Reset (RST) flag. - rst: 'bool' - #: Synchronise (SYN) flag. - syn: 'bool' - #: Finish (FIN) flag. - fin: 'bool' - #: Captured header, as far as the Data Offset goes. - header: 'bytes' - #: Captured payload, as far as the IP packet carrying it goes. - payload: 'bytes' - - -class _IPLayer(NamedTuple): - """An IPv4 or IPv6 header the default engine dissects as a layer of a frame (see :func:`_ip_layer`).""" - - #: IP version of the header, ``4`` or ``6``. - version: 'int' - #: Captured octets from the header on, to the end of the frame or of the - #: packets carrying it, whichever comes first. - octets: 'bytes' - #: Place of the layer in the frame, its first layer being ``1``, as the - #: default engine counts towards - #: :data:`~pcapkit.protocols.protocol.FRAME_LAYER_LIMIT`. - depth: 'int' - - -class _Upper(NamedTuple): - """The layer the default engine dissects behind an IP header and the headers it reads past.""" - - #: Protocol number of the layer. - protocol: 'int' - #: Captured octets from its header on, to the end of the packet carrying it. - octets: 'bytes' - #: Place of the layer in the frame, as for :attr:`_IPLayer.depth`. - depth: 'int' - #: Whether an AH or IPv6 extension header in IPv4 comes before it, which the - #: default engine dissects as a layer of its own (see :func:`_tcp_carrier`). - behind: 'bool' - - -def _ip_layer(version: 'int', octets: 'bytes', depth: 'int') -> 'Optional[_IPLayer]': - """Read an IP header as the default engine dissects it. - - Args: - version: IP version to read the header as, ``4`` or ``6``. - octets: Captured octets from the header on, to the end of the frame or - of the packets carrying it. - depth: Place of the header in the frame (see :attr:`_IPLayer.depth`). - - Returns: - The layer, or :data:`None` if the default engine dissects none there: - past :data:`~pcapkit.protocols.protocol.FRAME_LAYER_LIMIT` layers - (:issue:`1610`), short of the fixed header, or an IPv4 header whose IHL - is below 5 or which its parser rejects (see - :func:`_ipv4_header_accepted`). `DPKT`_ decodes no header that short or - with that IHL either, and neither it nor the default engine checks an - IPv6 header's version. - - .. _DPKT: https://dpkt.readthedocs.io - - """ - if depth > FRAME_LAYER_LIMIT: - return None - if version == 4: - ihl = (octets[0] & 0x0F) * 4 if len(octets) >= IPV4_HEADER_LEN else 0 - if ihl < IPV4_HEADER_LEN or not _ipv4_header_accepted(octets[:ihl]): - return None - elif len(octets) < IPV6_HEADER_LEN: - return None - return _IPLayer(version, bytes(octets), depth) - - -def _link_network(linktype: 'Optional[int]', octets: 'bytes') -> 'Optional[_IPLayer]': - """Find the IP layer the default engine dissects right after a frame's link layer, from its octets. - - Args: - linktype: The frame's ``LINKTYPE_*`` number (see :func:`attach_linktype`), - if known. - octets: The frame's record octets. - - Returns: - The layer (see :func:`_ip_layer`), or :data:`None` if there is none. - Over Ethernet II, that is an IPv4 or IPv6 EtherType after the 14-octet - header and every 802.1Q and 802.1ad tag behind it (the - :data:`~pcapkit.toolkit.pypcapfile.VLAN_TAGS`), each tag a layer of its - own; over BSD loopback (``NULL`` and ``LOOP``), an IPv4 or IPv6 address - family (:data:`LOOPBACK_IP_VERSIONS`) in the byte order - :func:`~pcapkit.protocols.link.loopback.family_byteorder` gives. An - ``IPV4`` or ``IPV6`` frame is IP of that version, and a ``RAW`` one of - the version its first octet gives. The default engine leaves every other - link type, EtherType and family raw, and a frame cut short of its link - header. - - """ - if linktype == Enum_LinkType.ETHERNET: - if len(octets) < ETHERNET_HEADER_LEN: - return None - ethertype, start = int.from_bytes(octets[12:14], 'big'), ETHERNET_HEADER_LEN - while ethertype in VLAN_TAGS: - if len(octets) < start + VLAN_TAG_LEN: - return None - ethertype, start = int.from_bytes(octets[start + 2:start + 4], 'big'), start + VLAN_TAG_LEN - version = ETHERTYPE_IP_VERSIONS.get(ethertype) - depth = 2 + (start - ETHERNET_HEADER_LEN) // VLAN_TAG_LEN - elif linktype in (Enum_LinkType.NULL, Enum_LinkType.LOOP): - if len(octets) < LOOPBACK_HEADER_LEN: - return None - family, start, depth = octets[:LOOPBACK_HEADER_LEN], LOOPBACK_HEADER_LEN, 2 - version = LOOPBACK_IP_VERSIONS.get(int.from_bytes(family, family_byteorder(family, linktype))) - elif linktype in (Enum_LinkType.IPV4, Enum_LinkType.IPV6, Enum_LinkType.RAW) and octets: - version = {Enum_LinkType.IPV4: 4, Enum_LinkType.IPV6: 6}.get(linktype, octets[0] >> 4) - start, depth = 0, 1 - else: - return None - if version not in (4, 6): - return None - return _ip_layer(version, octets[start:], depth) - - -def _network_layer(packet: 'Packet') -> 'Optional[_IPLayer]': - """Fetch the network layer of a DPKT packet, as the default engine dissects it. - - Args: - packet: DPKT packet, i.e. the outermost layer. - - Returns: - The IPv4 or IPv6 packet :func:`_network` finds, with the octets - :func:`_network_octets` slices for it and its place in the frame as - :func:`_network_depth` counts it; or, if `DPKT`_ left the frame's - payload undecoded, the layer :func:`_link_network` reads off the record - octets :func:`attach_buffer` attached. That is a frame behind a VLAN tag - stack `DPKT`_ does not finish decoding in two passes (:issue:`1654`), - an IPv6 packet whose extension headers it cannot decode, and a frame it - cannot decode at all, which the engine keeps as raw octets with its link - type attached by :func:`attach_linktype` (:issue:`1626`). Else - :data:`None`. - - Note: - A packet built by hand carries no record octets, so its network layer's - octets are `DPKT`_'s re-serialisation of it (see :func:`packet2bytes`), - and one that is no :class:`dpkt.ip.IP` or :class:`dpkt.ip6.IP6` has - none. - - .. _DPKT: https://dpkt.readthedocs.io - - """ - import dpkt # isort:skip - - ip = _network(packet) - if ip is not None: - if not isinstance(ip, (dpkt.ip.IP, dpkt.ip6.IP6)): - return None - return _ip_layer(6 if isinstance(ip, dpkt.ip6.IP6) else 4, _network_octets(packet, ip), - _network_depth(packet)) - - buffer = getattr(packet, BUFFER_ATTR, None) # type: Optional[bytes] - if buffer is None or not isinstance(getattr(packet, 'data', None), (bytes, bytearray)): - return None - if isinstance(packet, dpkt.ethernet.Ethernet): - return _link_network(Enum_LinkType.ETHERNET, buffer) - if isinstance(packet, dpkt.loopback.Loopback): # an address family it has no decoder for - return None - return _link_network(getattr(packet, LINKTYPE_ATTR, None), buffer) - - -def _ipv4_upper(layer: '_IPLayer') -> 'Optional[_Upper]': - """Step over an IPv4 header to the layer the default engine dissects behind it. - - Args: - layer: IPv4 layer. - - Returns: - The layer its protocol names, in the octets its Total Length leaves -- - a Total Length of ``0``, which TCP segmentation offload leaves, the rest - of what carries it -- past the headers the default engine reads past, - one layer each (see - :func:`~pcapkit.toolkit.pypcapfile._walk_extension_headers`); or - :data:`None` if it dissects none: in a later fragment, whose data is a - slice of the datagram's payload (:rfc:`791#section-3.2`), or behind one - of those headers it leaves undissected. - - """ - octets = layer.octets - if int.from_bytes(octets[6:8], 'big') & 0x1FFF: # a later fragment - return None - ihl, total = (octets[0] & 0x0F) * 4, int.from_bytes(octets[2:4], 'big') - payload = octets[ihl:max(total, ihl)] if total else octets[ihl:] - if octets[9] not in IPV4_EXTENSION_HEADERS: - return _Upper(octets[9], payload, layer.depth + 1, False) - walked = _walk_extension_headers(octets[9], payload) - if walked is None: - return None - protocol, upper, hops = walked - return _Upper(protocol, upper, layer.depth + hops + 1, True) - - -def _ipv6_chain(layer: '_IPLayer') -> 'tuple[bytes, Optional[int], Optional[_Upper]]': - """Walk an IPv6 packet's extension headers as the default engine does. - - Args: - layer: IPv6 layer. - - Returns: - The packet's octets as far as its Payload Length goes -- for a zero one, - the Jumbo Payload Length (:rfc:`2675`) of a Hop-by-Hop Options header - right after the IPv6 header, and lacking one, no payload at all - (:issue:`1651`) -- then where in them its first Fragment header starts, - if the walk reaches one, and the layer behind the last header walked, or - :data:`None` if the default engine dissects none. - - Each of the :data:`~pcapkit.toolkit.pypcapfile.IPV6_EXTENSION_HEADERS` - is walked, Mobility, HIP and Shim6 included (:issue:`1626`), and is a - layer of the frame: an AH is ``(Payload Length + 2) * 4`` octets long, a - Fragment header 8, and any other ``(Hdr Ext Len + 1) * 8``. The walk - ends with no layer behind past - :data:`~pcapkit.protocols.internet.internet.EXTENSION_HEADER_LIMIT` - headers (:issue:`1604`) or - :data:`~pcapkit.protocols.protocol.FRAME_LAYER_LIMIT` layers of the - frame (:issue:`1610`), at a header cut short or overrunning the packet, - and at a later fragment's Fragment header, behind which lies a slice of - the Fragmentable Part (:rfc:`8200#section-4.5`). - - Note: - Where a header's own parser rejects it, the default engine reads it - with a generic one that takes the same length, so the lengths alone - decide its walk. `DPKT`_'s walk does not stop at either bound, decodes - none of 135, 139 and 140, runs a zero Payload Length to the end of the - frame, and raises :exc:`AttributeError` at a Fragment header first in - the chain with another header behind it (:issue:`1351`), which keeps the - whole frame raw, so the walk is the default engine's own. - - .. _DPKT: https://dpkt.readthedocs.io - - """ - octets = layer.octets - extent = (int.from_bytes(octets[4:6], 'big') - or jumbo_payload_length(octets[6], octets[IPV6_HEADER_LEN:]) or 0) - data = octets[:IPV6_HEADER_LEN + extent] - nxt, offset, count, fragment = octets[6], IPV6_HEADER_LEN, 0, None # type: int, int, int, Optional[int] - while nxt in IPV6_EXTENSION_HEADERS: - if (count == EXTENSION_HEADER_LIMIT or layer.depth + count >= FRAME_LAYER_LIMIT - or len(data) < offset + 8): - return data, fragment, None - if nxt == Enum_TransType.IPv6_Frag: - if fragment is None: - fragment = offset - if int.from_bytes(data[offset + 2:offset + 4], 'big') >> 3: # a later fragment - return data, fragment, None - length = 8 - elif nxt == Enum_TransType.AH: - length = (data[offset + 1] + 2) * 4 - else: - length = (data[offset + 1] + 1) * 8 - nxt, offset, count = data[offset], offset + length, count + 1 - if len(data) < offset: - return data, fragment, None - return data, fragment, _Upper(nxt, data[offset:], layer.depth + count + 1, False) - - -def _ip_layers(packet: 'Packet') -> 'Iterator[tuple[_IPLayer, Optional[_Upper]]]': - """Walk the IP layers of a DPKT packet that the default engine dissects. - - Args: - packet: DPKT packet, i.e. the outermost layer. - - Yields: - Each IP layer, outermost first, with the layer the default engine - dissects behind it, if any (see :func:`_ipv4_upper` and - :func:`_ipv6_chain`). The first is the network layer (see - :func:`_network_layer`), and each next one the IPv4 or IPv6 packet the - last carries in an IP-in-IP tunnel (the :data:`IP_TUNNELS`), as deep as - they go. - - Note: - Every layer is read off the octets, by the default engine's rules, - rather than off what `DPKT`_ decoded of them, which differs from it in - the ways :func:`_ipv6_chain` and :func:`_network_layer` give. - - .. _DPKT: https://dpkt.readthedocs.io - - """ - layer = _network_layer(packet) - while layer is not None: - upper = _ipv4_upper(layer) if layer.version == 4 else _ipv6_chain(layer)[2] - yield layer, upper - if upper is None or upper.protocol not in IP_TUNNELS: - return - layer = _ip_layer(4 if upper.protocol == Enum_TransType.IPv4 else 6, upper.octets, upper.depth) - - -def _first_layer(packet: 'Packet', version: 'int') -> 'Optional[_IPLayer]': - """Fetch the first IP layer of a version in a DPKT packet, as the default engine dissects it. - - Args: - packet: DPKT packet, i.e. the outermost layer. - version: IP version, ``4`` or ``6``. - - Returns: - The first such layer :func:`_ip_layers` walks, or :data:`None`. - - Note: - The default engine hands IP reassembly ``frame['IPv4']`` or - ``frame['IPv6']``, the first such layer however deep - (:func:`pcapkit.toolkit.pcap.ipv4_reassembly`), so IPv4 carried in IPv6 - counts, and IPv6 in IPv4 (:issue:`1615`, :issue:`1637`). - - """ - return next((layer for layer, _ in _ip_layers(packet) if layer.version == version), None) - - -def _tcp_carrier(packet: 'Packet') -> 'Optional[_Segment]': - """Fetch the TCP segment of a DPKT packet, keyed by the IP layer that carries it. - - Args: - packet: DPKT packet, i.e. the outermost layer. - - Returns: - The segment behind the innermost IP layer :func:`_ip_layers` walks, if - the default engine reads one there, inside - :data:`~pcapkit.protocols.protocol.FRAME_LAYER_LIMIT` layers of the - frame (:issue:`1609`); else :data:`None`. It has at least - :data:`TCP_MIN_HEADER_LEN ` - octets, and a Data Offset counting no fewer; behind an AH or IPv6 - extension header in IPv4, also no more octets than were captured, and - options its TCP parser accepts (see - :func:`~pcapkit.toolkit.pypcapfile._default_accepts`). - - Note: - Directly over IP, the default engine reads a TCP header its parser - rejects out of the raw octets left behind - (:func:`pcapkit.toolkit.pcap.tcp_segment`), as `DPKT`_ decodes one, and - an IPv6 packet's extension headers do not come between, as it dissects - them itself. Behind a header that is a layer of its own, it does not, so - the parser's answer stands there. The TCP endpoints are the innermost IP - layer's addresses, as for the default engine; the outer ones are the - tunnel's (:issue:`1581`). Each flag is read from the flags octet alone, - as the default engine reads it (:issue:`1618`). - - .. _DPKT: https://dpkt.readthedocs.io - - """ - walked = None - for walked in _ip_layers(packet): - pass - if walked is None: - return None - layer, upper = walked - if upper is None or upper.protocol != Enum_TransType.TCP or upper.depth > FRAME_LAYER_LIMIT: - return None - segment = upper.octets - hdr_len = (segment[12] >> 4) * 4 if len(segment) >= TCP_MIN_HEADER_LEN else 0 - if hdr_len < TCP_MIN_HEADER_LEN: - return None - if upper.behind and (hdr_len > len(segment) or ( - hdr_len > TCP_MIN_HEADER_LEN and not _default_accepts('TCP', segment[:hdr_len]))): - return None - src, dst = _addresses(layer) - flags = segment[13] - return _Segment( - src=src, - dst=dst, - srcport=int.from_bytes(segment[0:2], 'big'), - dstport=int.from_bytes(segment[2:4], 'big'), - seq=int.from_bytes(segment[4:8], 'big'), - ack=int.from_bytes(segment[8:12], 'big'), - rst=bool(flags & 0x04), - syn=bool(flags & 0x02), - fin=bool(flags & 0x01), - header=segment[:hdr_len], - payload=segment[hdr_len:], - ) - - -def _addresses(layer: '_IPLayer') -> 'tuple[IPv4Address | IPv6Address, IPv4Address | IPv6Address]': - """Read the source and destination addresses of an IP layer. - - Args: - layer: IPv4 or IPv6 layer. - - Returns: - The source address, then the destination. - - """ - octets = layer.octets - if layer.version == 4: - return ipaddress.ip_address(octets[12:16]), ipaddress.ip_address(octets[16:20]) - return ipaddress.ip_address(octets[8:24]), ipaddress.ip_address(octets[24:40]) - - def ipv6_hdr_len(ipv6: 'IP6') -> 'int': """Calculate length of headers before IPv6 Fragment header. @@ -1186,9 +576,9 @@ def ipv4_reassembly(packet: 'Packet', timestamp: 'float', *, Data for IPv4 reassembly. * If the ``packet`` can be used for IPv4 reassembly. A packet can be reassembled - if it contains an IPv4 layer the default engine dissects -- the first of the - frame, as for the default engine, so one carried in IPv6 counts (see - :func:`_first_layer`) -- and its **DF** flag is :data:`False`. + if its network layer is IPv4 (:class:`dpkt.ip.IP`) -- one carried in IPv6 is not + read -- whose header was captured whole, and the **DF** (:attr:`dpkt.ip.IP.df`) + flag is :data:`False`. * If the ``packet`` can be reassembled, then the :obj:`dict` mapping of data for IPv4 reassembly (:term:`reasm.ipv4.packet`) will be returned; otherwise, returns :data:`None`. @@ -1198,47 +588,61 @@ def ipv4_reassembly(packet: 'Packet', timestamp: 'float', *, they are the captured ones. A packet built by hand carries no such buffer, and its octets are then :mod:`dpkt`'s re-serialisation of it, in which a zeroed checksum or length comes back recomputed -- - :func:`attach_buffer` is what makes them the wire's again. The packet - itself is left untouched either way. + :func:`attach_buffer` is what makes them the wire's again. They are that + re-serialisation too where :mod:`dpkt`'s own lengths misplace the layer + in the record, as behind some stacks of VLAN tags (:issue:`1646`). The + packet itself is left untouched either way. - The header is the IHL's octets and the payload the rest, to the end of - the frame or of the IPv6 packets carrying it, as the default engine - hands them over; the reassembler bounds the payload by the Total Length - itself. Neither is cut at the Total Length, which may be shorter than - the header (:issue:`1616`). + The header is the IHL's octets, and the payload ends at the Total + Length, or at the end of the frame for a zero one. A Total Length + shorter than the IHL leaves the header whole and the payload empty + (:issue:`1616`). See Also: :class:`pcapkit.foundation.reassembly.ipv4.IPv4` """ - ipv4 = _first_layer(packet, 4) - if ipv4 is None: - return None - octets = ipv4.octets - flags = int.from_bytes(octets[6:8], 'big') - if flags & 0x4000: # dismiss not fragmented packet - return None - src, dst = _addresses(ipv4) - # internet header length, in octets, options included - ihl = (octets[0] & 0x0F) * 4 - - data = IP_Packet( - bufid=( - cast('IPv4Address', src), # source IP address - cast('IPv4Address', dst), # destination IP address - int.from_bytes(octets[4:6], 'big'), # identification - Enum_TransType.get(octets[9]), # payload protocol type - ), - num=count, # original packet range number - fo=(flags & 0x1FFF) * 8, # fragment offset - ihl=ihl, # internet header length - mf=bool(flags & 0x2000), # more fragment flag - tl=int.from_bytes(octets[2:4], 'big'), # total length, header includes - header=octets[:ihl], # raw bytes type header - payload=bytearray(octets[ihl:]), # raw bytearray type payload - timestamp=_capture_time(timestamp), # capture timestamp - ) - return data + network = _network(packet) + # an IPv6 layer is the one with extension headers, as in ``_wire_hdr_len`` + ipv4 = None if hasattr(network, 'extension_hdrs') else network + if ipv4 is not None: + if ipv4.df: # dismiss not fragmented packet + return None + # internet header length, in octets -- ``IP.hl`` counts 32-bit words and + # covers any IP options, whereas ``IP.__hdr_len__`` is the fixed 20-octet + # struct size and would leave option octets at the head of the payload + ihl = ipv4.hl * 4 + wire, captured = _layer2bytes(packet, ipv4) + # a zero Total Length is what TCP segmentation offload leaves behind, in + # which case the datagram runs to the end of the frame, as with DPKT; nor + # does one shorter than the IHL cut into the header (#1616) + if captured and ipv4.len: + wire = wire[:max(ipv4.len, ihl)] + # DPKT decodes a header whose IHL runs past the captured octets, but + # there is no whole header to hand over + if len(wire) < ihl: + return None + + data = IP_Packet( + bufid=( + cast('IPv4Address', + ipaddress.ip_address(ipv4.src)), # source IP address + cast('IPv4Address', + ipaddress.ip_address(ipv4.dst)), # destination IP address + ipv4.id, # identification + Enum_TransType.get(ipv4.p), # payload protocol type + ), + num=count, # original packet range number + fo=ipv4.offset * 8, # fragment offset + ihl=ihl, # internet header length + mf=bool(ipv4.mf), # more fragment flag + tl=ipv4.len, # total length, header includes + header=wire[:ihl], # raw bytes type header + payload=bytearray(wire[ihl:]), # raw bytearray type payload + timestamp=_capture_time(timestamp), # capture timestamp + ) + return data + return None def ipv6_reassembly(packet: 'Packet', timestamp: 'float', *, @@ -1256,10 +660,9 @@ def ipv6_reassembly(packet: 'Packet', timestamp: 'float', *, Data for IPv6 reassembly. * If the ``packet`` can be used for IPv6 reassembly. A packet can be reassembled - if it contains an IPv6 layer the default engine dissects -- the first of the - frame, as for the default engine, so one carried in IPv4 counts (see - :func:`_first_layer`) -- and an IPv6 Fragment header (:rfc:`2460#section-4.5`) - among the extension headers the default engine walks (see :func:`_ipv6_chain`). + if its network layer is IPv6 (:class:`dpkt.ip6.IP6`) -- one carried in IPv4 is not + read -- and has IPv6 Fragment header (:rfc:`2460#section-4.5`, i.e., + :class:`dpkt.ip6.IP6FragmentHeader`). * If the ``packet`` can be reassembled, then the :obj:`dict` mapping of data for IPv6 reassembly (:term:`reasm.ipv6.packet`) will be returned; otherwise, returns :data:`None`. @@ -1269,57 +672,89 @@ def ipv6_reassembly(packet: 'Packet', timestamp: 'float', *, they are the captured ones. A packet built by hand carries no such buffer, and its octets are then :mod:`dpkt`'s re-serialisation of it, in which a zeroed checksum or length comes back recomputed -- - :func:`attach_buffer` is what makes them the wire's again. The packet - itself is left untouched either way. - - The header is the octets before the first Fragment header, and the - payload those after it, as far as the Payload Length goes, or the Jumbo - Payload Length (:rfc:`2675`) of a zero one, as the default engine cuts - them. + :func:`attach_buffer` is what makes them the wire's again. They are that + re-serialisation too where :mod:`dpkt`'s own lengths misplace the layer + in the record, as behind some stacks of VLAN tags (:issue:`1646`). The + packet itself is left untouched either way. See Also: :class:`pcapkit.foundation.reassembly.ipv6.IPv6` """ - ipv6 = _first_layer(packet, 6) - if ipv6 is None: - return None - octets, fragment, _ = _ipv6_chain(ipv6) - if fragment is None: # dismiss not fragmented packet - return None - src, dst = _addresses(ipv6) - word = int.from_bytes(octets[fragment + 2:fragment + 4], 'big') - # payload following the IPv6 Fragment header - payload = octets[fragment + 8:] - - data = IP_Packet( - bufid=( - cast('IPv6Address', src), # source IP address - cast('IPv6Address', dst), # destination IP address - # NOTE: The reassembly key is the Fragment header's Identification - # (:rfc:`8200#section-4.5`), not the IPv6 header's Flow Label. The - # label is optional and routinely zero, so keying on it collapses - # every datagram between one address pair into a single buffer and - # interleaves their fragments; it would also disagree with - # ``bufid[2]`` in every other engine, which feeds - # :attr:`pcapkit.foundation.reassembly.data.ip.DatagramID.id`. - int.from_bytes(octets[fragment + 4:fragment + 8], 'big'), # identification - Enum_TransType.get(octets[fragment]), # next header field in IPv6 Fragment Header - ), - num=count, # original packet range number - # NOTE: The Fragment Offset is the top 13 bits of its word, in 8-octet - # units (:rfc:`8200#section-4.5`). The reassembly machinery indexes the - # datagram buffer with ``fo``, so the units have to become octets here, - # exactly as :func:`pcapkit.toolkit.scapy.ipv6_reassembly` does. - fo=(word >> 3) * 8, # fragment offset - ihl=fragment, # header length, only headers before IPv6-Frag - mf=bool(word & 0x1), # more fragment flag - tl=fragment + len(payload), # total length, header includes - header=octets[:fragment], # raw bytes type header before IPv6-Frag - payload=bytearray(payload), # raw bytearray type payload after IPv6-Frag - timestamp=_capture_time(timestamp), # capture timestamp - ) - return data + network = _network(packet) + ipv6 = cast('IP6', network) if hasattr(network, 'extension_hdrs') else None + if ipv6 is not None: + ipv6_frag = ipv6.extension_hdrs.get(44) # type: Optional[IP6FragmentHeader] + if ipv6_frag is None: # dismiss not fragmented packet + return None + hdr_len = ipv6_hdr_len(ipv6) + wire, captured = _layer2bytes(packet, ipv6) + # a zero Payload Length is a jumbogram or TCP segmentation offload, in + # which case the datagram runs to the end of the frame, as with DPKT + if captured and ipv6.plen: + wire = wire[:ipv6.__hdr_len__ + ipv6.plen] + # payload following the IPv6 Fragment header + payload = wire[hdr_len + ipv6_frag.__hdr_len__:] + + data = IP_Packet( + bufid=( + cast('IPv6Address', + ipaddress.ip_address(ipv6.src)), # source IP address + cast('IPv6Address', + ipaddress.ip_address(ipv6.dst)), # destination IP address + # NOTE: The reassembly key is the Fragment header's Identification + # (:rfc:`8200#section-4.5`), not the IPv6 header's Flow Label. The + # label is optional and routinely zero, so keying on it collapses + # every datagram between one address pair into a single buffer and + # interleaves their fragments; it would also disagree with + # ``bufid[2]`` in every other engine, which feeds + # :attr:`pcapkit.foundation.reassembly.data.ip.DatagramID.id`. + ipv6_frag.id, # identification + Enum_TransType.get(ipv6_frag.nxt), # next header field in IPv6 Fragment Header + ), + num=count, # original packet range number + # NOTE: ``IP6FragmentHeader.frag_off`` is a ``__bit_fields__`` property + # over the 13-bit on-wire Fragment Offset, i.e. already shifted out of + # the flags word, so it counts 8-octet units (:rfc:`8200#section-4.5`). + # The reassembly machinery indexes the datagram buffer with ``fo``, so + # the units have to become octets here, exactly as + # :func:`pcapkit.toolkit.scapy.ipv6_reassembly` does. + fo=ipv6_frag.frag_off * 8, # fragment offset + ihl=hdr_len, # header length, only headers before IPv6-Frag + mf=bool(ipv6_frag.m_flag), # more fragment flag + tl=hdr_len + len(payload), # total length, header includes + header=wire[:hdr_len], # raw bytes type header before IPv6-Frag + payload=bytearray(payload), # raw bytearray type payload after IPv6-Frag + timestamp=_capture_time(timestamp), # capture timestamp + ) + return data + return None + + +def _tcp_flags(tcp: 'TCP') -> 'tuple[bool, bool, bool]': + """Read the RST, SYN and FIN flags of a DPKT TCP segment. + + Args: + tcp: DPKT TCP packet. + + Returns: + Whether RST, SYN and FIN are set, in that order. + + Note: + Each flag is masked out of :attr:`dpkt.tcp.TCP.flags` by its ``TH_*`` + value, so it is read from the flags octet alone, as the default engine + reads it. `DPKT`_ 1.9.8 makes that field nine bits wide, its top bit + the NS bit (:rfc:`3540`): read by its position among the field's binary + digits, every flag was one bit low with NS set, a SYN reading as a FIN + (:issue:`1618`). + + .. _DPKT: https://dpkt.readthedocs.io + + """ + from dpkt.tcp import TH_FIN, TH_RST, TH_SYN # isort:skip + + flags = tcp.flags + return bool(flags & TH_RST), bool(flags & TH_SYN), bool(flags & TH_FIN) def tcp_reassembly(packet: 'Packet', timestamp: 'float', *, @@ -1337,8 +772,9 @@ def tcp_reassembly(packet: 'Packet', timestamp: 'float', *, Data for TCP reassembly. * If the ``packet`` can be used for TCP reassembly. A packet can be reassembled - if it contains a TCP segment the default engine reads, keyed by the IP packet - that carries it -- in an IP-in-IP tunnel, the innermost (see :func:`_tcp_carrier`). + if its network layer carries TCP layer (:class:`dpkt.tcp.TCP`), behind any IPv6 + extension headers :mod:`dpkt` decodes. TCP in an IP-in-IP tunnel, or behind an AH or + IPv6 extension header carried in IPv4, is not read. * If the ``packet`` can be reassembled, then the :obj:`dict` mapping of data for TCP reassembly (:term:`reasm.tcp.packet`) will be returned; otherwise, returns :data:`None`. @@ -1348,34 +784,44 @@ def tcp_reassembly(packet: 'Packet', timestamp: 'float', *, they are the captured ones. A packet built by hand carries no such buffer, and its octets are then :mod:`dpkt`'s re-serialisation of it, in which a zeroed checksum or length comes back recomputed -- - :func:`attach_buffer` is what makes them the wire's again. The packet - itself is left untouched either way. + :func:`attach_buffer` is what makes them the wire's again. They are that + re-serialisation too where :mod:`dpkt`'s own lengths misplace the layer + in the record, as behind some stacks of VLAN tags (:issue:`1646`). The + packet itself is left untouched either way. See Also: :class:`pcapkit.foundation.reassembly.tcp.TCP` """ - segment = _tcp_carrier(packet) - if segment is not None: - raw_len = len(segment.payload) # payload length, header excludes + ip = _network(packet) + if ip is None: + return None + + tcp = getattr(ip, 'tcp', None) # type: Optional[TCP] + if tcp is None and type(getattr(ip, 'data', None)).__name__ == 'TCP': + tcp = cast('TCP', ip.data) + if tcp is not None: + rst, syn, fin = _tcp_flags(tcp) + wire, _ = _layer2bytes(packet, tcp) + raw_len = len(tcp.data) # payload length, header excludes data = TCP_Packet( bufid=( - segment.src, # source IP address - segment.srcport, # source port - segment.dst, # destination IP address - segment.dstport, # destination port + ipaddress.ip_address(ip.src), # source IP address + tcp.sport, # source port + ipaddress.ip_address(ip.dst), # destination IP address + tcp.dport, # destination port ), num=count, # original packet range number - ack=segment.ack, # acknowledgement - dsn=segment.seq, # data sequence number - rst=segment.rst, # reset connection flag - syn=segment.syn, # synchronise flag - fin=segment.fin, # finish flag - header=segment.header, # raw bytes type header - payload=bytearray(segment.payload), # raw bytearray type payload - first=segment.seq, # first sequence number of payload - last=segment.seq + raw_len - 1, # last sequence number of payload + ack=tcp.ack, # acknowledgement + dsn=tcp.seq, # data sequence number + rst=rst, # reset connection flag + syn=syn, # synchronise flag + fin=fin, # finish flag + header=wire[:tcp.off * 4], # raw bytes type header + payload=bytearray(bytes(tcp.data)), # raw bytearray type payload + first=tcp.seq, # first sequence number of payload + last=tcp.seq + raw_len - 1, # last sequence number of payload len=raw_len, # payload length, header excludes timestamp=_capture_time(timestamp), # capture timestamp ) @@ -1400,8 +846,9 @@ def tcp_traceflow(packet: 'Packet', timestamp: 'float', *, Data for TCP reassembly. * If the ``packet`` can be used for TCP flow tracing. A packet can be reassembled - if it contains a TCP segment the default engine reads, keyed by the IP packet - that carries it -- in an IP-in-IP tunnel, the innermost (see :func:`_tcp_carrier`). + if its network layer carries TCP layer (:class:`dpkt.tcp.TCP`), behind any IPv6 + extension headers :mod:`dpkt` decodes. TCP in an IP-in-IP tunnel, or behind an AH or + IPv6 extension header carried in IPv4, is not read. * If the ``packet`` can be reassembled, then the :obj:`dict` mapping of data for TCP flow tracing (:term:`trace.tcp.packet`) will be returned; otherwise, returns :data:`None`. @@ -1411,31 +858,42 @@ def tcp_traceflow(packet: 'Packet', timestamp: 'float', *, they are the captured ones. A packet built by hand carries no such buffer, and its octets are then :mod:`dpkt`'s re-serialisation of it, in which a zeroed checksum or length comes back recomputed -- - :func:`attach_buffer` is what makes them the wire's again. The packet - itself is left untouched either way. + :func:`attach_buffer` is what makes them the wire's again. They are that + re-serialisation too where :mod:`dpkt`'s own lengths misplace the layer + in the record, as behind some stacks of VLAN tags (:issue:`1646`). The + packet itself is left untouched either way. See Also: :class:`pcapkit.foundation.traceflow.tcp.TCP` """ - segment = _tcp_carrier(packet) - if segment is not None: + ip = _network(packet) + if ip is None: + return None + + tcp = getattr(ip, 'tcp', None) # type: Optional[TCP] + if tcp is None and type(getattr(ip, 'data', None)).__name__ == 'TCP': + tcp = cast('TCP', ip.data) + if tcp is not None: + rst, syn, fin = _tcp_flags(tcp) + wire, _ = _layer2bytes(packet, tcp) + data = TF_TCP_Packet( # type: ignore[type-var] protocol=data_link, # data link type from global header index=count, # frame number frame=packet2frame(packet, timestamp, data_link=data_link), # extracted packet - syn=segment.syn, # TCP synchronise (SYN) flag - fin=segment.fin, # TCP finish (FIN) flag - rst=segment.rst, # TCP reset (RST) flag - src=segment.src, # source IP - dst=segment.dst, # destination IP - srcport=segment.srcport, # TCP source port - dstport=segment.dstport, # TCP destination port + syn=syn, # TCP synchronise (SYN) flag + fin=fin, # TCP finish (FIN) flag + rst=rst, # TCP reset (RST) flag + src=ipaddress.ip_address(ip.src), # source IP + dst=ipaddress.ip_address(ip.dst), # destination IP + srcport=tcp.sport, # TCP source port + dstport=tcp.dport, # TCP destination port timestamp=_capture_time(timestamp), # timestamp - seq=segment.seq, # TCP sequence number - ack=segment.ack, # TCP acknowledgement number - header=segment.header, # raw bytes type header - payload=bytearray(segment.payload), # raw bytearray type payload + seq=tcp.seq, # TCP sequence number + ack=tcp.ack, # TCP acknowledgement number + header=wire[:tcp.off * 4], # raw bytes type header + payload=bytearray(bytes(tcp.data)), # raw bytearray type payload ) return data return None diff --git a/pcapkit/toolkit/pypcapfile.py b/pcapkit/toolkit/pypcapfile.py index a972f95075..b3766305f2 100644 --- a/pcapkit/toolkit/pypcapfile.py +++ b/pcapkit/toolkit/pypcapfile.py @@ -50,7 +50,6 @@ import textwrap from typing import TYPE_CHECKING -from pcapkit.const.ipv6.extension_header import ExtensionHeader as Enum_ExtensionHeader from pcapkit.const.reg.ethertype import EtherType as Enum_EtherType from pcapkit.const.reg.transtype import TransType as Enum_TransType from pcapkit.foundation.reassembly.data.ip import Packet as IP_Packet @@ -109,10 +108,9 @@ # NOTE: the definitions from here to ``packet2timestamp``, and # ``_default_accepts``, ``_default_extension_header`` and -# ``_walk_extension_headers`` below, are kept for :mod:`pcapkit.toolkit.dpkt` -# and :mod:`pcapkit.toolkit.scapy`: they import the rest, and -# ``_walk_extension_headers`` uses ``AH_HEADER_LEN`` and -# ``_default_extension_header``. This module's adapters use none of them +# ``_walk_extension_headers`` below, are kept for :mod:`pcapkit.toolkit.scapy`: +# it imports the rest, and ``_walk_extension_headers`` uses ``AH_HEADER_LEN`` +# and ``_default_extension_header``. This module's adapters use none of them # (#1708). #: Length of the fixed part of an IP Authentication Header (AH), before its @@ -128,30 +126,11 @@ Enum_TransType.HIP, )) -#: EtherTypes of the 802.1Q customer tag and the 802.1ad service tag. -VLAN_TAGS = frozenset(( - Enum_EtherType.Customer_VLAN_Tag_Type, - Enum_EtherType.IEEE_Std_802_1Q_Service_VLAN_tag_identifier, -)) - -#: Length of a VLAN tag (IEEE 802.1Q): two octets of tag control information, -#: then the EtherType of what follows the tag. -VLAN_TAG_LEN = 4 - #: IP protocol numbers of the IP-in-IP tunnels, each mapped to the IP version #: it carries: IPv4 (protocol 4, :rfc:`2003`) and IPv6 (protocol 41, #: :rfc:`2473` and :rfc:`4213`). IP_TUNNELS = {Enum_TransType.IPv4: 4, Enum_TransType.IPv6: 6} -#: IPv6 extension headers that precede the upper-layer protocol: the ones -#: :rfc:`8200#section-4.1` orders before it, less ESP, whose payload is -#: encrypted, plus the Mobility, HIP and Shim6 headers. -IPV6_EXTENSION_HEADERS = frozenset(( - Enum_ExtensionHeader.HOPOPT, Enum_ExtensionHeader.IPv6_Opts, Enum_ExtensionHeader.IPv6_Route, - Enum_ExtensionHeader.IPv6_Frag, Enum_ExtensionHeader.AH, Enum_ExtensionHeader.Mobility_Header, - Enum_ExtensionHeader.HIP, Enum_ExtensionHeader.Shim6, -)) - def packet2timestamp(packet: 'Packet') -> 'float': """Calculate the timestamp of a PyPCAPFile packet. diff --git a/tests/foundation/engines/test_dpkt_parse_error_unit.py b/tests/foundation/engines/test_dpkt_parse_error_unit.py index 1fc49f332f..fd49f4e539 100644 --- a/tests/foundation/engines/test_dpkt_parse_error_unit.py +++ b/tests/foundation/engines/test_dpkt_parse_error_unit.py @@ -6,7 +6,7 @@ Options raises :exc:`AttributeError` from ``dpkt/ip6.py``. The engine did not catch it, so the third-party exception ended the whole extraction. -Every case writes its own one-frame capture to a temporary directory. Everything +Every case writes its own capture to a temporary directory. Everything from :mod:`pcapkit` is imported inside each test, after :func:`~tests._support.reimport_once_per_class`, not at module load. @@ -19,7 +19,7 @@ import unittest import warnings -from tests._support import reimport_once_per_class +from tests._support import close_extractor, reimport_once_per_class HAS_DPKT = importlib.util.find_spec('dpkt') is not None @@ -67,6 +67,32 @@ def test_unparsable_frame_is_kept_as_raw_data(self) -> None: self.assertTrue(any(issubclass(item.category, DPKTWarning) and 'cannot parse' in str(item.message) for item in caught)) + def test_unparsable_frame_is_written_out(self) -> None: + # The engine writes every frame through packet2dict, which read the + # header fields a frame kept raw does not have, so AttributeError ended + # the extraction there (#1662). It maps to no fields, then the next frame + # is read. + import pcapkit + from pcapkit.const.reg.linktype import LinkType + from pcapkit.toolkit.dpkt import packet2dict + + frame = bytes(12) + b'\x86\xdd' + IPV6 + PAYLOADS[60] + with tempfile.TemporaryDirectory() as tmp: + path = os.path.join(tmp, 'frag-60.pcap') + _write_pcap(path, frame) + with open(path, 'ab') as file: + file.write(struct.pack(' None: [(lt, ts) for lt, _, ts in EXPECTED]) self.assertEqual([data for _, _, data in seen], [ETHERNET, IPV4, ETHERNET, IPV4]) - def _assert_engines_agree(self, count: int) -> None: + def _assert_frames(self, count: int) -> None: from pcapkit.toolkit.dpkt import TIMESTAMP_ATTR - default = self._extract('default') dpkt = self._extract('dpkt') self.assertEqual(dpkt.length, count) self.assertEqual([getattr(frame, TIMESTAMP_ATTR) for frame in dpkt.frame], - [float(frame.info.timestamp_epoch) for frame in default.frame]) + [ts for _, _, ts in EXPECTED[:count]]) self.assertEqual([type(frame).__name__ for frame in dpkt.frame], [name for _, name, _ in EXPECTED[:count]]) - def test_dpkt_engine_matches_default_engine_across_interfaces(self) -> None: + def test_dpkt_engine_reads_each_interface(self) -> None: # section 1 only: one byte order, two interfaces with open(self.path, 'r+b') as file: file.truncate(len(_capture()) - len(_shb('>') + _idb('>', 228, tsresol=0x8A, tsoffset=100) + _epb('>', 0, IPV4))) - self._assert_engines_agree(3) + self._assert_frames(3) - def test_dpkt_engine_matches_default_engine_across_sections(self) -> None: - self._assert_engines_agree(len(EXPECTED)) + def test_dpkt_engine_reads_each_section(self) -> None: + self._assert_frames(len(EXPECTED)) def test_unknown_interface_id_is_a_format_error(self) -> None: from pcapkit.foundation.engines.dpkt import PCAPNGReader @@ -167,7 +166,7 @@ def test_unknown_interface_id_is_a_format_error(self) -> None: @unittest.skipUnless(HAS_DPKT, 'dpkt is not installed') class TestDPKTSimplePacketBlock(unittest.TestCase): - """The DPKT engine reads a Simple Packet Block, as the default engine does.""" + """The DPKT engine reads a Simple Packet Block.""" #: Ethernet, then raw IPv4 on nanoseconds; an SPB names no interface, so it #: is on the first, whose snaplen cuts it short. Big-endian, so the block's @@ -210,21 +209,19 @@ def test_reader_yields_the_block_on_the_first_interface(self) -> None: self.assertEqual((float(stamp), stamp.exact, stamp.orig_len), (0.0, decimal.Decimal(0), len(ETHERNET) + 40)) - def test_dpkt_engine_matches_default_engine(self) -> None: + def test_dpkt_engine_reads_every_block(self) -> None: import pcapkit from pcapkit.toolkit.dpkt import packet2bytes, packet2timestamp self._write(self.CAPTURE) with warnings.catch_warnings(): warnings.simplefilter('ignore') - default = pcapkit.extract(fin=self.path, engine='default', nofile=True, store=True) dpkt = pcapkit.extract(fin=self.path, engine='dpkt', nofile=True, store=True) - self.assertEqual(dpkt.length, default.length) + # the SPB, between the two EPBs, is dated at the epoch self.assertEqual([(float(packet2timestamp(frame)), packet2bytes(frame)) for frame in dpkt.frame], - [(float(frame.info.timestamp_epoch), bytes(frame.info.packet)) - for frame in default.frame]) + [(TICKS / 10**9, IPV4), (0.0, ETHERNET), (TICKS / 10**6, ETHERNET)]) def test_packet_data_stops_at_the_original_length(self) -> None: from pcapkit.foundation.engines.dpkt import PCAPNGReader @@ -236,27 +233,28 @@ def test_packet_data_stops_at_the_original_length(self) -> None: ((stamp, data),) = list(PCAPNGReader(file)) self.assertEqual((data, stamp.orig_len), (ETHERNET, len(ETHERNET))) - def test_a_raw_ipv4_interface_is_traced_as_by_the_default_engine(self) -> None: + def test_a_raw_ipv4_interface_is_traced(self) -> None: # #1520 and #1548 at once: on a LINKTYPE_IPV4 interface, an SPB is read # as a bare IP packet, which the flow tracer has to find too import pcapkit self._write(_shb('<') + _idb('<', 228) + _spb('<', IPV4_TCP) + _epb('<', 0, IPV4_TCP)) - flows = {} - for engine in ('default', 'dpkt'): - with warnings.catch_warnings(): - warnings.simplefilter('ignore') - extractor = pcapkit.extract(fin=self.path, engine=engine, nofile=True, tcp=True, - trace=True, trace_format='pcap', - trace_fout=os.path.join(self._tmp.name, engine)) - self.addCleanup(close_extractor, extractor) - flows[engine] = [] - for flow in extractor.trace.tcp: - with open(flow.fpout, 'rb') as file: - flows[engine].append((flow.label, tuple(flow.index), file.read())) - - self.assertEqual([index for _, index, _ in flows['dpkt']], [(1, 2)]) - self.assertEqual(flows['dpkt'], flows['default']) + with warnings.catch_warnings(): + warnings.simplefilter('ignore') + extractor = pcapkit.extract(fin=self.path, engine='dpkt', nofile=True, tcp=True, + trace=True, trace_format='pcap', trace_byteorder='little', + trace_fout=os.path.join(self._tmp.name, 'dpkt')) + self.addCleanup(close_extractor, extractor) + flows = [] + for flow in extractor.trace.tcp: + with open(flow.fpout, 'rb') as file: + flows.append((flow.label, tuple(flow.index), file.read())) + + # the SPB dated at the epoch, then the EPB, each record the whole packet + traced = (struct.pack(' None: from pcapkit.foundation.engines.dpkt import PCAPNGReader diff --git a/tests/foundation/engines/test_dpkt_raw_null_linktypes_unit.py b/tests/foundation/engines/test_dpkt_raw_null_linktypes_unit.py index 3f927e4337..0965f59333 100644 --- a/tests/foundation/engines/test_dpkt_raw_null_linktypes_unit.py +++ b/tests/foundation/engines/test_dpkt_raw_null_linktypes_unit.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -"""The DPKT engine reads raw-IP and BSD loopback frames as the default engine does. +"""The DPKT engine reads raw-IP and BSD loopback frames by their version and address family. GitHub issues #1502 and #1574. :meth:`DPKT._get_protocol ` knew Ethernet, IPv4 and @@ -39,7 +39,7 @@ @unittest.skipUnless(HAS_DPKT, 'dpkt is not installed') class TestDPKTRawAndLoopback(unittest.TestCase): - """Pin the DPKT engine's raw-IP and loopback link types against the default engine's.""" + """Pin the DPKT engine's raw-IP and loopback link types.""" def setUp(self) -> None: reimport_once_per_class(self) @@ -78,14 +78,14 @@ def test_frames_parse_into_their_network_layer(self) -> None: self.assertEqual(network if loopback else frame.data, record[4:] if loopback else record) - def test_reassembly_inputs_match_the_default_engine(self) -> None: - for linktype, records, _ in CAPTURES: + def test_reassembly_inputs_are_the_captured_datagrams(self) -> None: + # each frame read as IPv4 is one whole datagram: IPV4's header, then its UDP header + for (linktype, records, _), frames in zip(CAPTURES, ((1,), (1, 2), (1,))): with self.subTest(linktype=linktype): - default = self._extract(linktype, records, 'default') dpkt = self._extract(linktype, records, 'dpkt') - self.assertEqual(len(dpkt.reassembly.ipv4), len(default.reassembly.ipv4)) - self.assertEqual([datagram.packet for datagram in dpkt.reassembly.ipv4], - [datagram.packet for datagram in default.reassembly.ipv4]) + self.assertEqual([(datagram.index, bytes(datagram.header), bytes(datagram.payload)) + for datagram in dpkt.reassembly.ipv4], + [((frame,), IPV4[:20], IPV4[20:]) for frame in frames]) def test_get_protocol_by_link_type(self) -> None: import dpkt diff --git a/tests/interface/test_misc.py b/tests/interface/test_misc.py index fdc9682ed8..3f38e2e6a7 100644 --- a/tests/interface/test_misc.py +++ b/tests/interface/test_misc.py @@ -54,8 +54,8 @@ class FollowTCPStreamTests(unittest.TestCase): (``AttributeError: 'dict' object has no attribute 'packet'``) and returned an empty, misleading result on Scapy frames. - Both third-party engines that dissect this capture -- DPKT and Scapy -- must - therefore agree with the default engine on every stream. Engines that do + Both third-party engines that dissect this capture must therefore find every + stream: DPKT the two it holds, and Scapy the default engine's. Engines that do no dissection at all (PyPCAP) or have no reassembly adapter (PyShark) are redirected to the default engine instead, and are asserted separately below. @@ -114,28 +114,25 @@ def test_per_direction_tracing_still_splits_the_conversation(self) -> None: self.assertEqual(sorted(len(stream.packets) for stream in streams), [1, 1, 1]) @unittest.skipUnless(HAS_DPKT, 'dpkt not installed') - def test_dpkt_engine_matches_the_default_engine(self) -> None: - # DPKT dissects in.pcap's Ethernet/IP/TCP in full, so it must agree with the - # native engine on both the stream count and the reassembled bytes. This is - # the regression guard for #399: before the fix the DPKT frame reached the + def test_dpkt_engine_follows_every_stream(self) -> None: + # DPKT dissects in.pcap's Ethernet/IP/TCP in full, so it finds both streams, + # of two frames and one, with no payload to reassemble. This is the + # regression guard for #399: before the fix the DPKT frame reached the # pcapkit adapter and raised AttributeError, and the counted-vs-uncounted # call convention is what makes the reassembled numbering line up. with warnings.catch_warnings(): warnings.simplefilter('ignore') - native = self._follow() foreign = self._follow(engine='dpkt') self.assertEqual(len(foreign), IN_PCAP_TCP_STREAMS) - self.assertEqual([len(stream.packets) for stream in foreign], - [len(stream.packets) for stream in native]) - self.assertEqual([stream.conversations for stream in foreign], - [stream.conversations for stream in native]) + self.assertEqual([(len(stream.packets), stream.conversations) for stream in foreign], + [(2, ()), (1, ())]) @unittest.skipUnless(HAS_SCAPY, 'scapy not installed') def test_scapy_engine_matches_the_default_engine(self) -> None: # Scapy dissects in.pcap in full -- its two ICMPv6-over-IPv6 frames, its - # three IPv4/TCP frames and its one UDP frame -- so, exactly like DPKT above, - # it must agree with the native engine rather than differ from it. + # three IPv4/TCP frames and its one UDP frame -- so it must agree with the + # native engine rather than differ from it. # This is the regression guard for #406: the engine imported only # ``scapy.sendrecv``, so Scapy's ``conf.l2types`` registry was empty, # ``PcapReader`` could not map link type 1, and all six frames arrived as diff --git a/tests/toolkit/test_dpkt_6in4_reassembly_and_mpls_isl_1637_1638_unit.py b/tests/toolkit/test_dpkt_6in4_reassembly_and_mpls_isl_1637_1638_unit.py deleted file mode 100644 index d5897d24d3..0000000000 --- a/tests/toolkit/test_dpkt_6in4_reassembly_and_mpls_isl_1637_1638_unit.py +++ /dev/null @@ -1,497 +0,0 @@ -# -*- coding: utf-8 -*- -"""dpkt reads IPv6 in IPv4, and nothing behind MPLS or Cisco ISL, as the default engine does. C.f. #1637, #1638. - -The default engine hands IPv6 reassembly the first IPv6 layer of a frame however -deep, so an IPv6 fragment carried in IPv4 (6in4, RFC 4213) is an input, with the -octets inside the Total Length of each IPv4 packet above it. dpkt took the -network layer only, so gave no input for 6in4, 6in4 inside 4in4 or 6in4 behind -an IPv4 AH (#1637). It now walks the IPv4 tunnels and the headers the default -engine reads past, and stops where that engine does: at an IPv4 header it -rejects, a later IPv4 fragment, or any other protocol, an ICMP error quoting the -packet included. The IPv6 packet ends at its Payload Length or, for a zero one, -at its Jumbo Payload Length, as the default engine reads it; a zero Payload -Length with no Jumbo Payload option leaves that engine no Fragment header to -read, carried in IPv4 or not. Nor does that engine walk more than 32 extension -headers (#1604), so a Fragment header past them is no input either, carried in -IPv4 or not (#1609), nor dissect a frame past its 64th layer (#1610), so neither -is one that deep in nested IPv4. - -dpkt's Ethernet decoder pops an MPLS label stack, unwraps a Cisco ISL header and -steps over the legacy QinQ tags ``0x9100`` and ``0x9200``, naming the IP packet -behind them ``ip`` or ``ip6``. The default engine reads all of them as raw octets, -so dpkt handed TCP reassembly, flow tracing and IP reassembly packets it never -reads, and an ISL frame carrying an 802.1Q tag ended the extraction with a -``ProtocolError`` (#1638). dpkt now takes an Ethernet frame's payload only past -the 802.1Q and 802.1ad tags the default engine dissects, and nothing else. - -That decoder also keeps only some tags of a stack it reads in two goes, e.g. -802.1Q then 802.1ad, or three tags, so the IP packet behind them was looked for -too early in the frame, and ``ProtocolError`` ended the extraction. The tags are -now read off the captured octets. - -Every capture is built here, so the module is in the unit tier. - -""" -from __future__ import annotations - -import importlib.util -import os -import tempfile -import unittest -import warnings -from typing import TYPE_CHECKING -from unittest import mock - -from tests._support import close_extractor, reimport_once_per_class -from tests.foundation import _roundtrip as wire -from tests.toolkit.test_dpkt_ipv4_reassembly_input_1615_1616_unit import ipv4, ipv6, jumbo -from tests.toolkit.test_tcp_carrier_walk_1606_1617_unit import ah, options, tcp, v4 - -if TYPE_CHECKING: - from typing import Any - -RUNTIME_DEPS = ('tbtrim', 'aenum', 'chardet', 'dictdumper') -HAS_RUNTIME = all(importlib.util.find_spec(name) is not None for name in RUNTIME_DEPS) -HAS_DPKT = importlib.util.find_spec('dpkt') is not None - -#: An Ethernet trailer, e.g. an FCS, which no IP packet reaches. -TRAILER = b'\xee' * 6 - - -def fragment(ident: int, *, offset: int = 0, mf: bool = True, nxt: int = 253) -> bytes: - """An IPv6 Fragment header; ``offset`` is in octets.""" - return bytes([nxt, 0]) + (offset | mf).to_bytes(2, 'big') + ident.to_bytes(4, 'big') - - -def first(ident: int, *, ext: bytes = b'', nxt: int = 44, plen: 'int | None' = None) -> bytes: - """The first fragment of a 24-octet datagram, behind ``ext``, its Payload Length as given.""" - return ipv6(ext + fragment(ident) + b'A' * 16, nxt, plen=plen) - - -def last(ident: int) -> bytes: - """The last fragment of that datagram.""" - return ipv6(fragment(ident, offset=16, mf=False) + b'B' * 8, 44) - - -def eth(packet: bytes, *, trailer: bytes = b'') -> bytes: - """An Ethernet frame carrying the IPv4 or IPv6 ``packet``, then ``trailer``.""" - return wire.ethernet(packet, 0x86DD if packet[0] >> 4 == 6 else 0x0800) + trailer - - -def tagged(tpids: 'list[int]', packet: bytes) -> bytes: - """An Ethernet frame whose VLAN tags have the TPIDs ``tpids``, then the IP ``packet``.""" - types = list(tpids) + [0x86DD if packet[0] >> 4 == 6 else 0x0800] - tags = b''.join((100 + i).to_bytes(2, 'big') + types[i + 1].to_bytes(2, 'big') for i in range(len(tpids))) - return wire.MAC_B + wire.MAC_A + types[0].to_bytes(2, 'big') + tags + packet - - -def mpls(packet: bytes, *, labels: int = 1, ethertype: int = 0x8847, tpid: 'int | None' = None) -> bytes: - """An Ethernet frame carrying ``packet`` behind ``labels`` MPLS labels, and an 802.1Q tag if ``tpid``.""" - stack = b''.join(((16 + i) << 12 | (i == labels - 1) << 8 | 64).to_bytes(4, 'big') for i in range(labels)) - tag = b'' if tpid is None else (100).to_bytes(2, 'big') + ethertype.to_bytes(2, 'big') - return wire.MAC_B + wire.MAC_A + (ethertype if tpid is None else tpid).to_bytes(2, 'big') + tag + stack + packet - - -def isl(frame: bytes) -> bytes: - """A Cisco ISL frame: its 26-octet header, the encapsulated ``frame`` and a 4-octet CRC.""" - body = b'\xaa\xaa\x03\x00\x00\x0c' + (1 << 1).to_bytes(2, 'big') + bytes(4) + frame - return b'\x01\x00\x0c\x00\x00\x00' + wire.MAC_A + len(body).to_bytes(2, 'big') + body + bytes(4) - - -def icmp_unreach(quote: bytes) -> bytes: - """An ICMP Destination Unreachable message quoting ``quote``.""" - return bytes([3, 1, 0, 0, 0, 0, 0, 0]) + quote - - -def gre(packet: bytes) -> bytes: - """A GRE header carrying the IPv6 ``packet``.""" - return b'\x00\x00\x86\xdd' + packet - - -def ipv6_in_ipv4() -> 'dict[str, tuple[list[int], list[bytes]]]': - """IPv6 fragments carried in IPv4, each case with the payload lengths the default engine hands over. - - Every case's datagram has an Identification of its own. - - """ - cases = {} # type: dict[str, tuple[list[int], list[bytes]]] - ident = iter(range(1, 1000)) - - def pair(label: str, carry: 'Any') -> None: - number = next(ident) - cases[label] = ([16, 8], [carry(first(number)), carry(last(number))]) - - def one(label: str, payloads: 'list[int]', carry: 'Any') -> None: - cases[label] = (payloads, [carry(next(ident))]) - - # #1637: the first IPv6 layer, in IPv4, in 4in4, and behind the headers the default engine reads past - one('IPv6 (control)', [16], lambda n: eth(first(n))) - pair('6in4', lambda p: eth(v4(p, 41))) - pair('6in4 inside 4in4', lambda p: eth(v4(v4(p, 41), 4))) - pair('6in4 behind an IPv4 AH', lambda p: eth(v4(ah(41) + p, 51))) - one('6in4 behind a Hop-by-Hop header in IPv4', [16], lambda n: eth(v4(options(41) + first(n), 0))) - one('6in4 behind an AH inside 4in4', [16], lambda n: eth(v4(v4(ah(41) + first(n), 51), 4))) - one('4in4 behind an AH, carrying 6in4', [16], lambda n: eth(v4(ah(4) + v4(first(n), 41), 51))) - one('6in4 in a first IPv4 fragment', [16], lambda n: eth(v4(first(n), 41, mf=True, ident=n))) - one('6in4 with IPv4 options', [16], - lambda n: eth(ipv4(first(n), ident=n, options=b'\x94\x04\x00\x00', proto=41))) - one('6in4, Hop-by-Hop then Fragment header', [16], lambda n: eth(v4(first(n, ext=options(44), nxt=0), 41))) - one('6in4 behind an 802.1Q tag', [16], lambda n: tagged([0x8100], v4(first(n), 41))) - # whose octets end at each Total Length above, or at the IPv6 packet's own length - one('6in4 with a trailer', [16], lambda n: eth(v4(first(n), 41), trailer=TRAILER)) - one('6in4, Total Length 0, with a trailer', [16], lambda n: eth(v4(first(n), 41, length=0), trailer=TRAILER)) - one('6in4 cut by the Total Length', [6], - lambda n: eth(v4(first(n), 41, length=20 + 48 + 6), trailer=TRAILER)) - one('6in4 cut by the outer Total Length of 4in4', [6], - lambda n: eth(v4(v4(first(n), 41), 4, length=40 + 48 + 6), trailer=TRAILER)) - one('6in4 behind an AH, cut by the Total Length', [6], - lambda n: eth(v4(ah(41) + first(n), 51, length=20 + 24 + 48 + 6), trailer=TRAILER)) - # where the default engine dissects no Fragment header of the first IPv6 layer - one('6in4, not fragmented', [], lambda n: eth(v4(ipv6(b'A' * 16, 253), 41))) - one('ICMP quoting a 6in4 fragment', [], lambda n: eth(v4(icmp_unreach(v4(first(n), 41)), 1))) - one('6in4 in a later IPv4 fragment', [], lambda n: eth(v4(first(n), 41, offset=8, ident=n))) - # dpkt decodes no later fragment's payload, but the IPv6 behind an AH or - # Hop-by-Hop header is decoded from the octets, so the fragment check is ours - one('6in4 behind an AH in a later IPv4 fragment', [], - lambda n: eth(v4(ah(41) + first(n), 51, offset=8, ident=n))) - one('6in4 behind a Hop-by-Hop header in a later IPv4 fragment', [], - lambda n: eth(v4(options(41) + first(n), 0, offset=8, ident=n))) - one('6in4 behind rejected IPv4 options', [], - lambda n: eth(ipv4(first(n), ident=n, options=b'\x02\x03\x05\xb4', proto=41))) - one('6in4, Total Length below the IHL', [], lambda n: eth(v4(first(n), 41, length=12))) - one('6in4, Total Length inside the IPv6 header', [], lambda n: eth(v4(first(n), 41, length=20 + 30))) - one('6in4 behind an AH too short', [], lambda n: eth(v4(ah(41, 0) + first(n), 51))) - one('6in4 with a zero Payload Length', [], lambda n: eth(v4(first(n, plen=0), 41), trailer=TRAILER)) - one('6in4 jumbogram ending inside the Fragment header', [], - lambda n: eth(v4(first(n, ext=jumbo(44, 8 + 4), nxt=0, plen=0), 41))) - one('4in6 carrying a 6in4 fragment', [], lambda n: eth(ipv6(v4(first(n), 41), 4))) - one('6in4 in GRE', [], lambda n: eth(v4(gre(first(n)), 47))) - one('6in4 behind MPLS', [], lambda n: mpls(v4(first(n), 41))) - return cases - - -def payload_length() -> 'dict[str, tuple[list[int], list[bytes]]]': - """IPv6 fragments in Ethernet with a zero Payload Length, each with the payload lengths handed over.""" - return { - 'a zero Payload Length': ([], [eth(first(901, plen=0))]), - 'a zero Payload Length, with a trailer': ([], [eth(first(902, plen=0), trailer=TRAILER)]), - 'a jumbogram ending inside the Fragment header': ([], [eth(first(905, ext=jumbo(44, 8 + 4), nxt=0, - plen=0))]), - 'a Payload Length short of the packet (control)': ([6], [eth(first(906, plen=8 + 6), trailer=TRAILER)]), - } - - -def chain_bound() -> 'dict[str, tuple[list[int], list[bytes]]]': - """IPv6 first fragments behind Destination Options headers, each with the payload lengths handed over. - - The default engine walks 32 extension headers at most, so a Fragment header - after 32 or more others is not among them. - - """ - cases = {} # type: dict[str, tuple[list[int], list[bytes]]] - ident = iter(range(501, 1000)) - for count, payloads in ((31, [16]), (32, []), (33, []), (40, [])): - ext = options(60) * (count - 1) + options(44) - label = f'{count} Destination Options headers' + (' (control)' if payloads else '') - cases[f'IPv6, {label}'] = (payloads, [eth(first(next(ident), ext=ext, nxt=60))]) - cases[f'6in4, {label}'] = (payloads, [eth(v4(first(next(ident), ext=ext, nxt=60), 41))]) - cases[f'6in4 inside 4in4, {label}'] = (payloads, [eth(v4(v4(first(next(ident), ext=ext, nxt=60), 41), 4))]) - return cases - - -def nested(packet: bytes, count: int, *, proto: int = 41) -> bytes: - """``packet`` carried as ``proto`` in the innermost of ``count`` nested IPv4 headers.""" - packet = v4(packet, proto) - for _ in range(count - 1): - packet = v4(packet, 4) - return packet - - -def layer_limit() -> 'dict[str, tuple[list[int], list[bytes]]]': - """IPv6 first fragments deep in nested IPv4 over Ethernet, each with the payload lengths handed over. - - The default engine dissects no layer past the 64th of a frame (#1610). The - first is Ethernet, then each VLAN tag, IPv4 header, header in IPv4 it reads - past, and the IPv6 header, then each extension header before the Fragment - header, which must be within the 64 too. - - """ - cases = {} # type: dict[str, tuple[list[int], list[bytes]]] - ident = iter(range(1001, 2000)) - - def one(label: str, payloads: 'list[int]', build: 'Any', tpids: 'tuple[int, ...]' = ()) -> None: - packet = build(next(ident)) - cases[label + (' (control)' if payloads else '')] = (payloads, [tagged(list(tpids), packet) if tpids - else eth(packet)]) - - for count, payloads in ((61, [16]), (62, [])): - one(f'6in4 in {count} IPv4', payloads, lambda n, c=count: nested(first(n), c)) - one(f'6in4 behind an AH in {count - 1} IPv4', payloads, - lambda n, c=count: nested(ah(41) + first(n), c - 1, proto=51)) - for count, payloads in ((60, [16]), (61, [])): - one(f'802.1Q, 6in4 in {count} IPv4', payloads, lambda n, c=count: nested(first(n), c), (0x8100,)) - for count, payloads in ((59, [16]), (60, [])): - # a stack dpkt reads in two goes, so keeps one tag of - one(f'802.1Q, 802.1ad, 6in4 in {count} IPv4', payloads, lambda n, c=count: nested(first(n), c), - (0x8100, 0x88A8)) - # where the extension headers before the Fragment header reach the 64th layer - # first, and where each header the default engine reads past in IPv4 counts - for count, chain, payloads in ((40, 21, [16]), (40, 22, []), (31, 30, [16]), (31, 31, [])): - ext = options(60) * (chain - 1) + options(44) - one(f'6in4 in {count} IPv4, {chain} Destination Options headers', payloads, - lambda n, c=count, e=ext: nested(first(n, ext=e, nxt=60), c)) - for hops, chain, payloads in ((10, 1, [16]), (10, 2, []), (11, 0, [16]), (12, 0, [])): - ext = options(60) * (chain - 1) + options(44) if chain else b'' - one(f'6in4 behind {hops} Hop-by-Hop headers in 50 IPv4, {chain} Destination Options headers', payloads, - lambda n, h=hops, c=chain, e=ext: nested(options(0) * (h - 1) + options(41) - + first(n, ext=e, nxt=60 if c else 44), 50, proto=0)) - return cases - - -def raw_layer_limit() -> 'dict[str, tuple[list[int], list[bytes]]]': - """IPv6 first fragments deep in nested IPv4 of a raw-IP link type, whose first layer is the IPv4 header.""" - return { - '6in4 in 62 IPv4 (control)': ([16], [nested(first(2001), 62)]), - '6in4 in 63 IPv4': ([], [nested(first(2002), 63)]), - '6in4 in 40 IPv4, 22 Destination Options headers (control)': - ([16], [nested(first(2003, ext=options(60) * 21 + options(44), nxt=60), 40)]), - '6in4 in 40 IPv4, 23 Destination Options headers': - ([], [nested(first(2004, ext=options(60) * 22 + options(44), nxt=60), 40)]), - } - - -#: VLAN tag stacks by name, each with whether the default engine reads past them. -#: dpkt reads each of them in two goes and keeps only the tags of the second. -STACKS = { - '802.1Q, 802.1ad': (True, [0x8100, 0x88A8]), - 'two 802.1ad': (True, [0x88A8, 0x88A8]), - 'three 802.1Q': (True, [0x8100] * 3), - '802.1ad, 802.1Q, 802.1Q': (True, [0x88A8, 0x8100, 0x8100]), - 'four 802.1Q': (True, [0x8100] * 4), - # where the tag dpkt drops is the one the default engine leaves raw - '802.1Q, legacy QinQ 0x9100': (False, [0x8100, 0x9100]), - '802.1ad, 802.1Q, legacy QinQ 0x9100': (False, [0x88A8, 0x8100, 0x9100]), -} - - -def stacked(tpids: 'list[int]') -> 'list[bytes]': - """Frames tagged ``tpids``: IPv4 and IPv6 TCP, then IPv4, IPv6 and 6in4 fragment pairs; then an untagged control.""" - return [ - tagged(tpids, v4(tcp(1), 6)), tagged(tpids, wire.ipv6(tcp(2), nxt=6)), - tagged(tpids, v4(b'A' * 16, 253, mf=True, ident=31)), tagged(tpids, v4(b'B' * 8, 253, offset=16, ident=31)), - tagged(tpids, first(32)), tagged(tpids, last(32)), - tagged(tpids, v4(first(33), 41)), tagged(tpids, v4(last(33), 41)), - eth(v4(tcp(3), 6)), - ] - - -def links() -> 'dict[str, tuple[tuple[int, int, int], list[bytes]]]': - """Frames by link-layer encapsulation, each with the TCP, IPv4 and IPv6 inputs the default engine hands over. - - Every TCP segment has a port of its own, and every fragmented datagram an - Identification of its own. - - """ - def each(build: 'Any', ident: int) -> 'list[bytes]': - return [build(v4(b'A' * 16, 253, mf=True, ident=ident)), build(v4(b'B' * 8, 253, offset=16, ident=ident))] - - def each6(build: 'Any', ident: int) -> 'list[bytes]': - return [build(first(ident)), build(last(ident))] - - return { - # an IPv4 packet that may be fragmented is an IPv4 reassembly input too - 'IPv4, TCP (control)': ((1, 1, 0), [eth(v4(tcp(1), 6))]), - '802.1Q, IPv4, TCP (control)': ((1, 1, 0), [tagged([0x8100], v4(tcp(2), 6))]), - '802.1ad, 802.1Q, IPv6, TCP (control)': ((1, 0, 0), [tagged([0x88A8, 0x8100], wire.ipv6(tcp(3), nxt=6))]), - 'MPLS, IPv4, TCP': ((0, 0, 0), [mpls(v4(tcp(4), 6))]), - 'MPLS, IPv6, TCP': ((0, 0, 0), [mpls(wire.ipv6(tcp(5), nxt=6))]), - 'MPLS multicast, IPv4, TCP': ((0, 0, 0), [mpls(v4(tcp(6), 6), ethertype=0x8848)]), - 'two MPLS labels, IPv4, TCP': ((0, 0, 0), [mpls(v4(tcp(7), 6), labels=2)]), - '802.1Q, MPLS, IPv4, TCP': ((0, 0, 0), [mpls(v4(tcp(8), 6), tpid=0x8100)]), - 'Cisco ISL, Ethernet, IPv4, TCP': ((0, 0, 0), [isl(eth(v4(tcp(9), 6)))]), - 'Cisco ISL, Ethernet, IPv6, TCP': ((0, 0, 0), [isl(eth(wire.ipv6(tcp(10), nxt=6)))]), - 'legacy QinQ 0x9100, IPv4, TCP': ((0, 0, 0), [tagged([0x9100], v4(tcp(11), 6))]), - 'legacy QinQ 0x9200, 802.1Q, IPv4, TCP': ((0, 0, 0), [tagged([0x9200, 0x8100], v4(tcp(12), 6))]), - 'LLC/SNAP, IPv4, TCP (control)': ((0, 0, 0), [ - wire.MAC_B + wire.MAC_A + (8 + 44).to_bytes(2, 'big') + b'\xaa\xaa\x03\x00\x00\x00\x08\x00' - + v4(tcp(13), 6)]), - 'MPLS, IPv4 fragments': ((0, 0, 0), each(mpls, 21)), - 'MPLS, IPv6 fragments': ((0, 0, 0), each6(mpls, 22)), - 'Cisco ISL, Ethernet, IPv4 fragments': ((0, 0, 0), each(lambda p: isl(eth(p)), 23)), - 'Cisco ISL, Ethernet, IPv6 fragments': ((0, 0, 0), each6(lambda p: isl(eth(p)), 24)), - 'legacy QinQ 0x9100, IPv4 fragments': ((0, 0, 0), each(lambda p: tagged([0x9100], p), 25)), - '802.1Q, IPv4 fragments (control)': ((0, 2, 0), each(lambda p: tagged([0x8100], p), 26)), - '802.1Q, IPv6 fragments (control)': ((0, 0, 2), each6(lambda p: tagged([0x8100], p), 27)), - } - - -def read(test: unittest.TestCase, engine: str, frames: 'list[bytes]', linktype: int = 1) -> 'dict[str, Any]': - """Read ``frames`` of ``linktype`` with ``engine``, with IP and TCP reassembly and TCP flow tracing. - - Returns: - Every input the engine hands IPv4, IPv6 and TCP reassembly and TCP flow - tracing, keyed by frame number -- every field, but a trace input's - ``frame`` -- the reassembled datagrams, and the traced flows. - - """ - from pcapkit import extract - from pcapkit.foundation.reassembly.ipv4 import IPv4 - from pcapkit.foundation.reassembly.ipv6 import IPv6 - from pcapkit.foundation.reassembly.tcp import TCP - from pcapkit.foundation.traceflow.traceflow import TraceFlowBase - - seen = {'IPv4': {}, 'IPv6': {}, 'TCP': {}, 'trace': {}} # type: dict[str, Any] - calls = {klass: klass.__call__ for klass in (IPv4, IPv6, TCP)} - trace = TraceFlowBase.__call__ - - def fields(packet: 'Any', skip: 'tuple[str, ...]' = ()) -> 'dict[str, Any]': - return {key: bytes(value) if isinstance(value, bytearray) else value - for key, value in packet.items() if key not in skip} - - def spy(klass: 'Any') -> 'Any': - def call(this: 'Any', packet: 'Any') -> 'Any': - seen[klass.__name__][packet.num] = fields(packet) - return calls[klass](this, packet) - return call - - def spy_trace(this: 'Any', packet: 'Any') -> 'Any': - seen['trace'][packet.index] = fields(packet, ('frame',)) - return trace(this, packet) - - with tempfile.TemporaryDirectory() as tmp, warnings.catch_warnings(), \ - mock.patch.object(IPv4, '__call__', spy(IPv4)), mock.patch.object(IPv6, '__call__', spy(IPv6)), \ - mock.patch.object(TCP, '__call__', spy(TCP)), mock.patch.object(TraceFlowBase, '__call__', spy_trace): - path = os.path.join(tmp, 'encap.pcap') - with open(path, 'wb') as file: - file.write(wire.pcap([(1, number, octets) for number, octets in enumerate(frames)], linktype=linktype)) - warnings.simplefilter('ignore') - extractor = extract(fin=path, nofile=True, engine=engine, reassembly=True, ip=True, tcp=True, - trace=True, trace_fout=os.path.join(tmp, 'trace'), trace_format='json') - try: - if extractor._exnam != engine: # pylint: disable=protected-access - test.skipTest(f'{engine} did not run') - seen['datagrams'] = { - 'IPv4': [(datagram.completed, datagram.id, datagram.index, datagram.header, datagram.payload) - for datagram in extractor.reassembly.ipv4], - 'IPv6': [(datagram.completed, datagram.id, datagram.index, datagram.header, datagram.payload) - for datagram in extractor.reassembly.ipv6], - 'TCP': [datagram.id for datagram in extractor.reassembly.tcp], - } - seen['flows'] = [(flow.label, tuple(flow.index)) for flow in extractor.trace.tcp] - finally: - close_extractor(extractor) - return seen - - -def numbered(cases: 'dict[str, tuple[Any, list[bytes]]]') -> 'tuple[list[bytes], dict[str, list[int]]]': - """The frames of ``cases`` in order, and each case's one-based frame numbers.""" - frames, numbers = [], {} # type: list[bytes], dict[str, list[int]] - for label, (_, built) in cases.items(): - numbers[label] = list(range(len(frames) + 1, len(frames) + 1 + len(built))) - frames.extend(built) - return frames, numbers - - -@unittest.skipUnless(HAS_RUNTIME, 'runtime dependencies not installed') -@unittest.skipUnless(HAS_DPKT, 'dpkt not installed') -class DPKTEncapsulationTests(unittest.TestCase): - """dpkt's reassembly and trace inputs are the default engine's, field for field.""" - - def setUp(self) -> None: - reimport_once_per_class(self) - - def agree_on_ipv6(self, cases: 'dict[str, tuple[list[int], list[bytes]]]', linktype: int = 1) -> None: - frames, numbers = numbered(cases) - base, dpkt = read(self, 'default', frames, linktype), read(self, 'dpkt', frames, linktype) - for label, (payloads, _) in cases.items(): - with self.subTest(case=label): - got = [base['IPv6'][number] for number in numbers[label] if number in base['IPv6']] - self.assertEqual([len(packet['payload']) for packet in got], payloads, 'the default engine') - self.assertEqual([dpkt['IPv6'].get(number) for number in numbers[label]], - [base['IPv6'].get(number) for number in numbers[label]]) - for aspect in ('IPv4', 'TCP', 'trace', 'datagrams', 'flows'): - self.assertEqual(dpkt[aspect], base[aspect], aspect) - - def test_ipv6_carried_in_ipv4_is_an_input(self) -> None: - # #1637 - self.agree_on_ipv6(ipv6_in_ipv4()) - - def test_a_zero_payload_length_gives_way_to_a_jumbo_payload_option(self) -> None: - # #1637: the rule the 6in4 rows need, which holds for IPv6 in Ethernet too - self.agree_on_ipv6(payload_length()) - - def test_a_fragment_header_past_the_chain_bound_is_no_input(self) -> None: - # #1604, #1609: dpkt walks the whole chain, the default engine 32 headers - self.agree_on_ipv6(chain_bound()) - - def test_a_fragment_header_past_the_frame_layer_limit_is_no_input(self) -> None: - # #1610: the default engine dissects 64 layers of a frame, dpkt any number - self.agree_on_ipv6(layer_limit()) - - def test_a_raw_ip_frame_counts_no_link_layer(self) -> None: - # #1610: LINKTYPE_RAW, so the outermost IPv4 header is the first layer - self.agree_on_ipv6(raw_layer_limit(), linktype=101) - - def test_a_stack_dpkt_reads_in_two_goes_ends_no_extraction(self) -> None: - # dpkt kept the second go's tags only, so the IP header was looked for 4 - # octets early per tag dropped, and ProtocolError ended the read - for label, (reads, tpids) in STACKS.items(): - with self.subTest(stack=label): - frames = stacked(tpids) - base, dpkt = read(self, 'default', frames), read(self, 'dpkt', frames) - self.assertEqual((sorted(base['TCP']), sorted(base['IPv6'])), - ([1, 2, 9], [5, 6, 7, 8]) if reads else ([9], []), 'the default engine') - for aspect in ('TCP', 'trace', 'IPv4', 'IPv6', 'datagrams', 'flows'): - self.assertEqual(dpkt[aspect], base[aspect], aspect) - - def test_nothing_is_read_behind_mpls_isl_or_legacy_qinq(self) -> None: - # #1638 - cases = links() - frames, numbers = numbered(cases) - base, dpkt = read(self, 'default', frames), read(self, 'dpkt', frames) - for label, (counts, _) in cases.items(): - with self.subTest(case=label): - self.assertEqual(tuple(sum(number in base[aspect] for number in numbers[label]) - for aspect in ('TCP', 'IPv4', 'IPv6')), counts, 'the default engine') - for aspect in ('TCP', 'trace', 'IPv4', 'IPv6'): - self.assertEqual([dpkt[aspect].get(number) for number in numbers[label]], - [base[aspect].get(number) for number in numbers[label]], aspect) - for aspect in ('datagrams', 'flows'): - self.assertEqual(dpkt[aspect], base[aspect], aspect) - - def test_an_isl_frame_carrying_an_8021q_tag_ends_no_extraction(self) -> None: - # #1638: dpkt overwrote the ISL tag with the 802.1Q one, so the IPv4 - # header was looked for 26 octets early and ProtocolError ended the read - frames = [isl(tagged([0x8100], v4(tcp(1), 6))), eth(v4(tcp(2), 6))] - base, dpkt = read(self, 'default', frames), read(self, 'dpkt', frames) - self.assertEqual((sorted(base['TCP']), sorted(base['trace'])), ([2], [2])) - for aspect in ('TCP', 'trace', 'IPv4', 'IPv6', 'datagrams', 'flows'): - self.assertEqual(dpkt[aspect], base[aspect], aspect) - - def test_a_packet_built_by_hand_is_read_too(self) -> None: - # with no record octets attached, DPKT's own fields tell what it read - # past, and the octets are its serialisation of the packet, which here - # is the octets it was parsed from - import dpkt - - from pcapkit.const.reg.linktype import LinkType - from pcapkit.toolkit import dpkt as toolkit - - for label, octets in (('MPLS', mpls(v4(tcp(1), 6))), ('Cisco ISL', isl(eth(v4(tcp(1), 6)))), - ('legacy QinQ', tagged([0x9100], v4(tcp(1), 6)))): - with self.subTest(case=label): - packet = dpkt.ethernet.Ethernet(octets) - self.assertEqual(type(packet.ip).__name__, 'IP') # what dpkt decodes - self.assertIsNone(toolkit.tcp_reassembly(packet, 1.0, count=1)) - self.assertIsNone(toolkit.tcp_traceflow(packet, 1.0, data_link=LinkType.ETHERNET, count=1)) - - tunnelled = eth(v4(first(1), 41), trailer=TRAILER) - data = toolkit.ipv6_reassembly(dpkt.ethernet.Ethernet(tunnelled), 1.0, count=1) - self.assertIsNotNone(data) - assert data is not None - self.assertEqual((data.ihl, bytes(data.header), bytes(data.payload)), - (40, tunnelled[34:74], b'A' * 16)) - - -if __name__ == '__main__': - unittest.main() diff --git a/tests/toolkit/test_dpkt_ip_walk_1609_1626_1651_1654_unit.py b/tests/toolkit/test_dpkt_ip_walk_1609_1626_1651_1654_unit.py deleted file mode 100644 index 804f3f08e7..0000000000 --- a/tests/toolkit/test_dpkt_ip_walk_1609_1626_1651_1654_unit.py +++ /dev/null @@ -1,361 +0,0 @@ -# -*- coding: utf-8 -*- -"""dpkt reads a frame's IP layers off its octets, as the default engine does. C.f. #1609, #1626, #1651, #1654. - -The dpkt adapters walked what dpkt decoded, which is not what the default engine -dissects: - -* #1651: dpkt runs an IPv6 payload to the end of the record when the Payload - Length is zero, and past a jumbogram's Jumbo Payload Length, so TCP and IPv6 - fragments were read where the default engine reads none, or longer. -* #1654: dpkt reads at most two VLAN tags a pass and two passes, so behind a - deeper stack it decoded nothing, and the adapters handed over nothing. -* #1609: the default engine walks 32 extension headers in a row and dissects 64 - layers of a frame; the walk for TCP and IPv4 reassembly bounded neither. -* #1626: dpkt decodes none of Mobility, HIP and Shim6, cannot read a later - fragment whose Next Header is an extension header, and raises at a Fragment - header first in the chain with another header behind it, which keeps the whole - frame raw. - -The adapters now read every IP layer, extension header and TCP segment off the -captured octets by the default engine's rules, and a frame the engine keeps raw -by the link type it attaches. Every input to TCP reassembly, flow tracing and IP -reassembly is compared with the default engine's, field for field. A frame kept -raw is written out, and traced, rather than ending the extraction. - -Every capture is built here, so the module is in the unit tier. - -""" -from __future__ import annotations - -import importlib.util -import itertools -import os -import tempfile -import unittest -import warnings -from typing import TYPE_CHECKING - -from tests._support import close_extractor, reimport_once_per_class -from tests.foundation import _roundtrip as wire -from tests.toolkit.test_dpkt_6in4_reassembly_and_mpls_isl_1637_1638_unit import (TRAILER, nested, - numbered, read, - tagged) -from tests.toolkit.test_dpkt_ipv4_reassembly_input_1615_1616_unit import ipv6, jumbo -from tests.toolkit.test_tcp_carrier_walk_1606_1617_unit import ah, eth, options, tcp, v4, v6 - -if TYPE_CHECKING: - from typing import Any, Callable - -RUNTIME_DEPS = ('tbtrim', 'aenum', 'chardet', 'dictdumper') -HAS_RUNTIME = all(importlib.util.find_spec(name) is not None for name in RUNTIME_DEPS) -HAS_DPKT = importlib.util.find_spec('dpkt') is not None - -#: What each case's counts count, in order. -ASPECTS = ('TCP', 'trace', 'IPv4', 'IPv6') - -#: TCP ports and fragment Identifications, one per use. -_NUMBERS = itertools.count(1) - - -def number() -> int: - """A port offset or Identification no other frame uses.""" - return next(_NUMBERS) - - -def segment() -> bytes: - """A TCP segment carrying four octets, on a port pair of its own.""" - return tcp(number()) - - -def frag6(nxt: int, ident: int, *, offset: int = 0, mf: bool = True) -> bytes: - """An IPv6 Fragment header; ``offset`` is in octets.""" - return bytes([nxt, 0]) + (offset | mf).to_bytes(2, 'big') + ident.to_bytes(4, 'big') - - -def v4_fragments(carry: 'Callable[[bytes], bytes]') -> 'list[bytes]': - """The two fragments of a 24-octet IPv4 datagram, each carried by ``carry``.""" - ident = number() - return [carry(v4(b'A' * 16, 253, mf=True, ident=ident)), carry(v4(b'B' * 8, 253, offset=16, ident=ident))] - - -def v6_fragments(carry: 'Callable[[bytes], bytes]') -> 'list[bytes]': - """The two fragments of a 24-octet IPv6 datagram, each carried by ``carry``.""" - ident = number() - return [carry(v6(frag6(253, ident) + b'A' * 16, 44)), - carry(v6(frag6(253, ident, offset=16, mf=False) + b'B' * 8, 44))] - - -def nested6(packet: bytes, count: int, nxt: int) -> bytes: - """``packet`` carried as ``nxt`` in the innermost of ``count`` nested IPv6 headers.""" - packet = v6(packet, nxt) - for _ in range(count - 1): - packet = v6(packet, 41) - return packet - - -def pairs(segment: bytes, count: int, *, version: int) -> bytes: - """``segment`` behind ``count`` (IPv4, AH) or (IPv6, Hop-by-Hop) pairs, as ``version`` says.""" - if version == 4: - packet = v4(ah(6) + segment, 51) - for _ in range(count - 1): - packet = v4(ah(4) + packet, 51) - return packet - packet = v6(options(6) + segment, 0) - for _ in range(count - 1): - packet = v6(options(41) + packet, 0) - return packet - - -def mobility(nxt: int) -> bytes: - """A Mobility header (135) of 8 octets, a Binding Refresh Request.""" - return bytes([nxt, 0, 0, 0, 0, 0, 0, 0]) - - -def hip(nxt: int) -> bytes: - """A HIP header (139) of 40 octets, an I1 packet.""" - return bytes([nxt, 4, 1, 0x21, 0, 0, 0, 0]) + b'\x11' * 32 - - -def shim6(nxt: int) -> bytes: - """A Shim6 header (140) of 8 octets.""" - return bytes([nxt, 0, 0, 0, 0, 0, 0, 0]) - - -def payload_length() -> 'dict[str, tuple[tuple[int, ...], list[bytes]]]': - """#1651: IPv6 packets whose Payload Length is zero, each with the default engine's input counts.""" - def jumbogram(short: int = 0) -> bytes: - return ipv6(jumbo(6, 8 + 24 - short) + segment(), 0, plen=0) - - return { - 'IPv6, TCP (control)': ((1, 1, 0, 0), [eth(v6(segment(), 6))]), - 'Payload Length 0, TCP': ((0, 0, 0, 0), [eth(ipv6(segment(), 6, plen=0))]), - 'Payload Length 0, TCP, trailer': ((0, 0, 0, 0), [eth(ipv6(segment(), 6, plen=0)) + TRAILER]), - '6in4, Payload Length 0, TCP': ((0, 0, 1, 0), [eth(v4(ipv6(segment(), 6, plen=0), 41))]), - 'Payload Length 0, Fragment header': ((0, 0, 0, 0), [eth(ipv6(frag6(253, number()) + b'A' * 16, 44, - plen=0))]), - 'Payload Length 0, Fragment header, trailer': ((0, 0, 0, 0), [eth(ipv6(frag6(253, number()) + b'A' * 16, 44, - plen=0)) + TRAILER]), - 'Payload Length 0, Hop-by-Hop with no Jumbo Payload option, TCP': ( - (0, 0, 0, 0), [eth(ipv6(options(6) + segment(), 0, plen=0))]), - 'Payload Length 0, 4in6, TCP': ((0, 0, 0, 0), [eth(ipv6(v4(segment(), 6), 4, plen=0))]), - '6in6, inner Payload Length 0, TCP': ((0, 0, 0, 0), [eth(v6(ipv6(segment(), 6, plen=0), 41))]), - 'jumbogram, Jumbo Payload Length inside the TCP header': ((0, 0, 0, 0), [eth(jumbogram(12))]), - } - - -#: VLAN tag stacks dpkt does not finish decoding in two passes, by name. -STACKS = { - 'three 802.1ad': [0x88A8] * 3, - 'five 802.1Q': [0x8100] * 5, - '802.1ad, four 802.1Q': [0x88A8] + [0x8100] * 4, - '802.1Q, two 802.1ad': [0x8100, 0x88A8, 0x88A8], - 'eight 802.1Q': [0x8100] * 8, -} - - -def vlan_stacks() -> 'dict[str, tuple[tuple[int, ...], list[bytes]]]': - """#1654: IP behind deep VLAN tag stacks, each with the default engine's input counts.""" - cases = {} # type: dict[str, tuple[tuple[int, ...], list[bytes]]] - for name, tpids in STACKS.items(): - def carry(packet: bytes, tpids: 'list[int]' = tpids) -> bytes: - return tagged(tpids, packet) - cases[f'{name}, IPv4, TCP'] = ((1, 1, 1, 0), [carry(v4(segment(), 6))]) - cases[f'{name}, IPv6, TCP'] = ((1, 1, 0, 0), [carry(v6(segment(), 6))]) - cases[f'{name}, IPv4 fragments'] = ((0, 0, 2, 0), v4_fragments(carry)) - cases[f'{name}, IPv6 fragments'] = ((0, 0, 0, 2), v6_fragments(carry)) - cases[f'{name}, 6in4 fragments'] = ((0, 0, 2, 2), v6_fragments(lambda p, c=carry: c(v4(p, 41)))) - cases['three 802.1ad, IPv4, TCP, trailer'] = ((1, 1, 1, 0), [tagged(STACKS['three 802.1ad'], - v4(segment(), 6)) + TRAILER]) - # where dpkt finishes in two passes, and where a legacy tag deep in the stack is left raw - cases['two 802.1ad, IPv4, TCP (control)'] = ((1, 1, 1, 0), [tagged([0x88A8] * 2, v4(segment(), 6))]) - cases['three 802.1ad, legacy QinQ 0x9100, IPv4, TCP'] = ((0, 0, 0, 0), [tagged([0x88A8] * 3 + [0x9100], - v4(segment(), 6))]) - return cases - - -def layer_limit() -> 'dict[str, tuple[tuple[int, ...], list[bytes]]]': - """#1609: TCP and IPv4 fragments deep in a frame, each with the default engine's input counts. - - The default engine walks 32 extension headers in a row (#1604) and dissects - 64 layers of a frame (#1610): Ethernet, then each VLAN tag, IP header, - header in IPv4 it reads past and IPv6 extension header. - - """ - cases = { - 'IPv6, 32 Hop-by-Hop headers, TCP (control)': ((1, 1, 0, 0), [eth(v6(options(0) * 31 + options(6) - + segment(), 0))]), - 'IPv6, 33 Hop-by-Hop headers, TCP': ((0, 0, 0, 0), [eth(v6(options(0) * 32 + options(6) + segment(), 0))]), - 'IPv6, 1000 Hop-by-Hop headers, TCP': ((0, 0, 0, 0), [eth(v6(options(0) * 999 + options(6) - + segment(), 0))]), - 'IPv6, 32 Destination Options headers, 4in6, TCP (control)': ( - (1, 1, 1, 0), [eth(v6(options(60) * 31 + options(4) + v4(segment(), 6), 60))]), - 'IPv6, 33 Destination Options headers, 4in6, TCP': ( - (0, 0, 0, 0), [eth(v6(options(60) * 32 + options(4) + v4(segment(), 6), 60))]), - 'TCP in 62 IPv4 (control)': ((1, 1, 1, 0), [eth(nested(segment(), 62, proto=6))]), - 'TCP in 63 IPv4': ((0, 0, 1, 0), [eth(nested(segment(), 63, proto=6))]), - 'TCP in 62 IPv6 (control)': ((1, 1, 0, 0), [eth(nested6(segment(), 62, 6))]), - 'TCP in 63 IPv6': ((0, 0, 0, 0), [eth(nested6(segment(), 63, 6))]), - 'TCP behind 31 (IPv4, AH) pairs (control)': ((1, 1, 1, 0), [eth(pairs(segment(), 31, version=4))]), - 'TCP behind 32 (IPv4, AH) pairs': ((0, 0, 1, 0), [eth(pairs(segment(), 32, version=4))]), - 'TCP behind 31 (IPv6, Hop-by-Hop) pairs (control)': ((1, 1, 0, 0), [eth(pairs(segment(), 31, version=6))]), - 'TCP behind 32 (IPv6, Hop-by-Hop) pairs': ((0, 0, 0, 0), [eth(pairs(segment(), 32, version=6))]), - 'IPv4 fragments in 62 IPv6 (control)': ((0, 0, 2, 0), v4_fragments(lambda p: eth(nested6(p, 62, 4)))), - 'IPv4 fragments in 63 IPv6': ((0, 0, 0, 0), v4_fragments(lambda p: eth(nested6(p, 63, 4)))), - 'TCP behind 61 802.1Q tags (control)': ((1, 1, 1, 0), [tagged([0x8100] * 61, v4(segment(), 6))]), - 'TCP behind 62 802.1Q tags': ((0, 0, 1, 0), [tagged([0x8100] * 62, v4(segment(), 6))]), - 'IPv4 fragments behind 62 802.1Q tags (control)': ((0, 0, 2, 0), - v4_fragments(lambda p: tagged([0x8100] * 62, p))), - 'IPv4 fragments behind 63 802.1Q tags': ((0, 0, 0, 0), v4_fragments(lambda p: tagged([0x8100] * 63, p))), - 'IPv6 fragments behind 61 802.1Q tags (control)': ((0, 0, 0, 2), - v6_fragments(lambda p: tagged([0x8100] * 61, p))), - 'IPv6 fragments behind 62 802.1Q tags': ((0, 0, 0, 0), v6_fragments(lambda p: tagged([0x8100] * 62, p))), - } - return cases - - -def raw_layer_limit() -> 'dict[str, tuple[tuple[int, ...], list[bytes]]]': - """#1609 on a raw-IP link type, whose first layer is the outermost IP header.""" - return { - 'TCP in 63 IPv4 (control)': ((1, 1, 1, 0), [nested(segment(), 63, proto=6)]), - 'TCP in 64 IPv4': ((0, 0, 1, 0), [nested(segment(), 64, proto=6)]), - } - - -def chains() -> 'dict[str, tuple[tuple[int, ...], list[bytes]]]': - """#1626: TCP and IP fragments behind IPv6 extension header chains, each with the default engine's input counts.""" - chain = { - 'Fragment (offset 0, M=0), then AH': (lambda nxt: frag6(51, number(), mf=False) + ah(nxt), 44, 1), - 'Fragment (offset 0, M=0), then Destination Options': ( - lambda nxt: frag6(60, number(), mf=False) + options(nxt), 44, 1), - 'AH': (ah, 51, 0), - 'Mobility (135)': (mobility, 135, 0), - 'HIP (139)': (hip, 139, 0), - 'Shim6 (140)': (shim6, 140, 0), - 'Destination Options (control)': (options, 60, 0), - } - cases = {} # type: dict[str, tuple[tuple[int, ...], list[bytes]]] - for name, (header, first, fragmented) in chain.items(): - cases[f'{name}, TCP'] = ((1, 1, 0, fragmented), [eth(v6(header(6) + segment(), first))]) - cases[f'{name}, TCP, trailer'] = ((1, 1, 0, fragmented), [eth(v6(header(6) + segment(), first)) + TRAILER]) - cases[f'6in4, {name}, TCP'] = ((1, 1, 1, fragmented), [eth(v4(v6(header(6) + segment(), first), 41))]) - cases[f'{name}, IPv4 fragments'] = ((0, 0, 2, 2 * fragmented), - v4_fragments(lambda p, h=header, f=first: eth(v6(h(4) + p, f)))) - # an IPv6 reassembly input whose Fragment header names another extension header - ident = number() - cases.update({ - 'Fragment (offset 0, M=1), then 253 (control)': ((0, 0, 0, 1), [eth(v6(frag6(253, number()) - + b'A' * 16, 44))]), - 'Fragment (offset 0, M=1), then Destination Options': ( - (0, 0, 0, 1), [eth(v6(frag6(60, number()) + options(253) + b'A' * 8, 44))]), - 'Fragment (offset 0, M=0), then Destination Options': ( - (0, 0, 0, 1), [eth(v6(frag6(60, number(), mf=False) + options(253) + b'A' * 8, 44))]), - 'Fragment (offset 0, M=1), then AH': ((0, 0, 0, 1), [eth(v6(frag6(51, number()) + ah(253) + b'A' * 8, 44))]), - 'Fragment (offset 8, M=0), next header 60': ( - (0, 0, 0, 1), [eth(v6(frag6(60, number(), offset=8, mf=False) + b'B' * 8, 44))]), - 'a datagram whose Fragmentable Part starts with Destination Options': ( - (0, 0, 0, 2), [eth(v6(frag6(60, ident) + options(253) + b'A' * 8, 44)), - eth(v6(frag6(60, ident, offset=16, mf=False) + b'B' * 8, 44))]), - 'two Fragment headers': ((0, 0, 0, 1), [eth(v6(frag6(44, number()) + frag6(253, number()) + b'A' * 16, 44))]), - 'a Hop-by-Hop header overrunning the packet, TCP': ( - (0, 0, 0, 0), [eth(v6(bytes([6, 5, 1, 4, 0, 0, 0, 0]) + segment(), 0))]), - '802.1Q, Fragment (offset 0, M=0), then AH, TCP': ( - (1, 1, 0, 1), [tagged([0x8100], v6(frag6(51, number(), mf=False) + ah(6) + segment(), 44))]), - }) - return cases - - -def raw_link_chains() -> 'dict[int, dict[str, tuple[tuple[int, ...], list[bytes]]]]': - """#1626 on the other link types, by link type, where dpkt's decoding raises and the frame is kept raw.""" - def first(nxt: int = 6) -> bytes: - return v6(frag6(51, number(), mf=False) + ah(nxt) + segment(), 44) - - def later() -> bytes: - return v6(frag6(60, number()) + options(253) + b'A' * 8, 44) - - return { - # and where the default engine dissects nothing past the link layer - 1: {'Ethernet, cut short of its header': ((0, 0, 0, 0), [eth(first())[:10]]), - 'Ethernet, 802.1Q tag cut short': ((0, 0, 0, 0), [wire.MAC_B + wire.MAC_A + b'\x81\x00\x00\x64'])}, - 101: {'raw IP, Fragment then AH, TCP': ((1, 1, 0, 1), [first()]), - 'raw IP, Fragment then Destination Options': ((0, 0, 0, 1), [later()]), - 'raw IP, version 5': ((0, 0, 0, 0), [bytes([0x55]) + v4(segment(), 6)[1:]]), - 'raw IP, no octets': ((0, 0, 0, 0), [b''])}, - 0: {'NULL, Fragment then AH, TCP': ((1, 1, 0, 1), [b'\x18\x00\x00\x00' + first()]), - 'NULL, Fragment then Destination Options': ((0, 0, 0, 1), [b'\x18\x00\x00\x00' + later()]), - 'NULL, cut short of its family': ((0, 0, 0, 0), [b'\x18\x00'])}, - 108: {'LOOP, Fragment then AH, TCP': ((1, 1, 0, 1), [b'\x00\x00\x00\x18' + first()])}, - 147: {'USER0, IPv4, TCP': ((0, 0, 0, 0), [v4(segment(), 6)])}, - 228: {'IPV4, 6in4, Fragment then AH, TCP': ((1, 1, 1, 1), [v4(first(), 41)])}, - 229: {'IPV6, Fragment then Destination Options': ((0, 0, 0, 1), [later()])}, - } - - -@unittest.skipUnless(HAS_RUNTIME, 'runtime dependencies not installed') -@unittest.skipUnless(HAS_DPKT, 'dpkt not installed') -class DPKTIPWalkTests(unittest.TestCase): - """dpkt's reassembly and trace inputs are the default engine's, field for field.""" - - def setUp(self) -> None: - reimport_once_per_class(self) - - def agree(self, cases: 'dict[str, tuple[tuple[int, ...], list[bytes]]]', linktype: int = 1) -> None: - frames, numbers = numbered(cases) - base, dpkt = read(self, 'default', frames, linktype), read(self, 'dpkt', frames, linktype) - for label, (counts, _) in cases.items(): - with self.subTest(case=label, linktype=linktype): - self.assertEqual(tuple(sum(frame in base[aspect] for frame in numbers[label]) for aspect in ASPECTS), - counts, 'the default engine') - for aspect in ASPECTS: - self.assertEqual([dpkt[aspect].get(frame) for frame in numbers[label]], - [base[aspect].get(frame) for frame in numbers[label]], aspect) - for aspect in ('datagrams', 'flows'): - self.assertEqual(dpkt[aspect], base[aspect], aspect) - - def test_a_zero_payload_length_carries_no_payload_unless_a_jumbogram(self) -> None: - # #1651 - self.agree(payload_length()) - - def test_ip_behind_a_deep_vlan_tag_stack_is_read(self) -> None: - # #1654 - self.agree(vlan_stacks()) - - def test_tcp_and_ipv4_past_the_chain_and_frame_bounds_are_no_input(self) -> None: - # #1609 - self.agree(layer_limit()) - self.agree(raw_layer_limit(), linktype=101) - - def test_tcp_and_fragments_behind_every_extension_header_chain_are_read(self) -> None: - # #1626 - self.agree(chains()) - - def test_a_frame_dpkt_cannot_decode_is_read_by_its_link_type(self) -> None: - # #1626: dpkt raises AttributeError for a Fragment header first in the - # chain with another header behind it, so the engine keeps the frame raw - for linktype, cases in raw_link_chains().items(): - self.agree(cases, linktype=linktype) - - def test_a_frame_kept_raw_is_written_out(self) -> None: - # The engine writes every frame through packet2dict, which read the - # header fields a frame kept raw does not have, so AttributeError ended - # the extraction at the first frame dpkt could not decode - from pcapkit import extract - - frames = [eth(v6(frag6(51, number(), mf=False) + ah(6) + segment(), 44)), eth(v4(segment(), 6))] - with tempfile.TemporaryDirectory() as tmp, warnings.catch_warnings(): - warnings.simplefilter('ignore') - path = os.path.join(tmp, 'raw.pcap') - with open(path, 'wb') as file: - file.write(wire.pcap([(1, index, octets) for index, octets in enumerate(frames)])) - extractor = extract(fin=path, fout=os.path.join(tmp, 'out'), format='json', engine='dpkt', - store=True) - try: - if extractor._exnam != 'dpkt': # pylint: disable=protected-access - self.skipTest('dpkt did not run') - self.assertEqual(len(extractor.frame), 2) - self.assertEqual(type(extractor.frame[0]).__name__, 'RawPacket') - finally: - close_extractor(extractor) - - -if __name__ == '__main__': - unittest.main() diff --git a/tests/toolkit/test_dpkt_ipv4_reassembly_input_1615_1616_unit.py b/tests/toolkit/test_dpkt_ipv4_reassembly_input_1615_1616_unit.py index c125bc5d0e..65248cd27f 100644 --- a/tests/toolkit/test_dpkt_ipv4_reassembly_input_1615_1616_unit.py +++ b/tests/toolkit/test_dpkt_ipv4_reassembly_input_1615_1616_unit.py @@ -1,13 +1,9 @@ # -*- coding: utf-8 -*- -"""dpkt hands IPv4 reassembly what the default engine hands it. C.f. #1615, #1616. +"""dpkt hands IPv4 reassembly a whole header. C.f. #1616. -The default engine hands IPv4 reassembly the first IPv4 layer of a frame however -deep, so IPv4 carried in IPv6 (RFC 2473) is an input, with the octets inside the -Payload Length of each IPv6 packet above it. dpkt took the network layer only, -so gave no input for 4in6 (#1615). It also cut the input at the Total Length, -which the reassembler bounds the payload by itself: a Total Length shorter than -the header cut the header, and a padded last fragment lost its padding (#1616). -The default engine's header is the IHL's octets and its payload the rest. +dpkt cut the input at the Total Length, so one shorter than the IHL cut the +header itself (#1616). It now cuts no shorter than the IHL. A header whose IHL +runs past the captured octets, which dpkt still decodes, is no input at all. Every capture is built here, so the module is in the unit tier. @@ -46,81 +42,9 @@ def ipv4(payload: bytes, *, ident: int, options: bytes = b'', length: 'int | Non return bytes(header) + payload -def ipv6(payload: bytes, nxt: int, *, plen: 'int | None' = None) -> bytes: - """An IPv6 packet carrying ``payload``, its Payload Length as given.""" - packet = wire.ipv6(payload, nxt=nxt) - if plen is None: - return packet - return packet[:4] + plen.to_bytes(2, 'big') + packet[6:] - - -def options(nxt: int) -> bytes: - """An 8-octet Hop-by-Hop or Destination Options header, one PadN option in it.""" - return bytes([nxt, 0, 1, 4, 0, 0, 0, 0]) - - -def fragment(nxt: int, offset: int, mf: bool) -> bytes: - """An IPv6 Fragment header; ``offset`` is in octets.""" - return bytes([nxt, 0]) + (offset | mf).to_bytes(2, 'big') + (7).to_bytes(4, 'big') - - -def jumbo(nxt: int, length: int) -> bytes: - """A Hop-by-Hop Options header carrying a Jumbo Payload option (RFC 2675).""" - return bytes([nxt, 0, 0xC2, 4]) + length.to_bytes(4, 'big') - - -def ethernet(packet: bytes, *, pad_to: int = 0, trailer: bytes = b'') -> bytes: - """An Ethernet frame carrying ``packet``, padded to ``pad_to`` octets, then ``trailer``.""" - frame = wire.ethernet(packet, 0x86DD if packet[0] >> 4 == 6 else 0x0800) - return frame + bytes(max(pad_to - len(frame), 0)) + trailer - - -def cases() -> 'dict[str, list[bytes]]': - """Frames by case, each case's IPv4 datagram with an Identification of its own. - - Each is the first fragment of a 24-octet datagram, then for a pair its last. - - """ - built = {} # type: dict[str, list[bytes]] - - def pair(label: str, carry: 'Any', ident: int) -> None: - first = ipv4(b'A' * 16, ident=ident, mf=True) - last = ipv4(b'B' * 8, ident=ident, offset=16) - built[label] = [carry(first), carry(last)] - - def one(label: str, carry: 'Any', ident: int) -> None: - built[label] = [carry(ipv4(b'A' * 16, ident=ident, mf=True))] - - # #1615: the first IPv4 layer, inside IPv6, behind extension headers, or in 6in6 - pair('4in6', lambda p: ethernet(ipv6(p, 4)), 1) - pair('4in6 behind Destination Options', lambda p: ethernet(ipv6(options(4) + p, 60)), 2) - pair('4in6 in 6in6', lambda p: ethernet(ipv6(ipv6(p, 4), 41)), 3) - one('4in6 in a first IPv6 fragment', lambda p: ethernet(ipv6(fragment(4, 0, True) + p, 44)), 4) - one('4in6 behind 32 extension headers', lambda p: ethernet(ipv6(options(60) * 31 + options(4) + p, 60)), 5) - # whose octets end at each Payload Length above, short of a trailer - pair('4in6 with a trailer', lambda p: ethernet(ipv6(p, 4), trailer=b'\xee' * 6), 6) - one('4in6 cut by the Payload Length', lambda p: ethernet(ipv6(p, 4, plen=30)), 7) - one('4in6 cut by the outer Payload Length of 6in6', - lambda p: ethernet(ipv6(ipv6(p, 4, plen=200), 41, plen=70)), 8) - # where the default engine dissects no IPv4 inside the IPv6 packet - one('4in6 in a later IPv6 fragment', lambda p: ethernet(ipv6(options(44) + fragment(4, 8, False) + p, 0)), 10) - one('4in6 behind 33 extension headers', lambda p: ethernet(ipv6(options(60) * 32 + options(4) + p, 60)), 11) - one('4in6 with a zero Payload Length', lambda p: ethernet(ipv6(p, 4, plen=0)), 12) - one('4in6 jumbogram shorter than the IPv4 header', lambda p: ethernet(ipv6(jumbo(4, 8 + 10) + p, 0, plen=0)), 13) - # #1616: the header is the IHL's octets and the payload the rest of the frame - built['Total Length below the IHL'] = [ethernet(ipv4(b'C' * 16, ident=14, options=b'\x94\x04\x00\x00', - length=10, mf=True))] - pair('padded to the Ethernet minimum', lambda p: ethernet(p, pad_to=60), 15) - one('Total Length 0 in 4in6, with a trailer', - lambda p: ethernet(ipv6(p[:2] + b'\0\0' + p[4:], 4), trailer=b'\xee' * 4), 16) - # controls: plain IPv4, the outer IPv4 of 4in4 and of 4in6in4, and no IPv4 at all - pair('plain', ethernet, 17) - one('4in4', lambda p: ethernet(ipv4(p, ident=18, proto=4)), 19) - one('4in6in4', lambda p: ethernet(ipv4(ipv6(p, 4), ident=20, proto=41)), 21) - built['IPv6 carrying UDP'] = [ethernet(ipv6(wire.udp(b'data'), 17))] - built['ARP'] = [wire.ethernet(bytes.fromhex('0001080006040001') + wire.MAC_A + wire.IPV4_A - + bytes(6) + wire.IPV4_B, 0x0806)] - return built +def ethernet(packet: bytes) -> bytes: + """An Ethernet frame carrying the IPv4 ``packet``.""" + return wire.ethernet(packet, 0x0800) def read(test: unittest.TestCase, engine: str, frames: 'list[bytes]') -> 'dict[str, Any]': @@ -162,93 +86,28 @@ def spy(this: 'Any', packet: 'Any') -> 'Any': @unittest.skipUnless(HAS_RUNTIME, 'runtime dependencies not installed') @unittest.skipUnless(HAS_DPKT, 'dpkt not installed') class DPKTIPv4ReassemblyInputTests(unittest.TestCase): - """dpkt's IPv4 reassembly inputs are the default engine's, octet for octet.""" - - @classmethod - def setUpClass(cls) -> None: - cls.cases = cases() - cls.frames = [frame for frames in cls.cases.values() for frame in frames] - # one-based frame numbers of each case - cls.numbers = {} # type: dict[str, list[int]] - number = 0 - for label, frames in cls.cases.items(): - cls.numbers[label] = list(range(number + 1, number + 1 + len(frames))) - number += len(frames) + """dpkt's IPv4 reassembly input carries its header whole, or is not handed over.""" def setUp(self) -> None: reimport_once_per_class(self) - def read(self) -> 'tuple[dict[str, Any], dict[str, Any]]': - """The default engine's reading of every case, and dpkt's.""" - if not hasattr(type(self), 'seen'): - type(self).seen = (read(self, 'default', self.frames), read(self, 'dpkt', self.frames)) - return type(self).seen - - def inputs(self, seen: 'dict[str, Any]', label: str) -> 'list[dict[str, Any]]': - return [seen['inputs'][number] for number in self.numbers[label] if number in seen['inputs']] - - def test_every_input_agrees_with_the_default_engine(self) -> None: - base, dpkt = self.read() - for label in self.cases: - with self.subTest(case=label): - self.assertEqual(self.inputs(dpkt, label), self.inputs(base, label)) - self.assertEqual(dpkt['datagrams'], base['datagrams']) - - def test_ipv4_in_ipv6_is_an_input(self) -> None: - # #1615: two inputs of the datagram tunnelled in IPv6, which completes - base, dpkt = self.read() - for seen in (base, dpkt): - got = self.inputs(seen, '4in6') - self.assertEqual([(len(packet['header']), len(packet['payload'])) for packet in got], - [(20, 16), (20, 8)]) - self.assertIn((self.numbers['4in6'][0], self.numbers['4in6'][1]), - [datagram[2] for datagram in seen['datagrams'] if datagram[0]]) - - def test_each_payload_length_above_bounds_the_input(self) -> None: - base, dpkt = self.read() - for label, payloads in ( - ('4in6 with a trailer', [16, 8]), - ('4in6 cut by the Payload Length', [10]), - ('4in6 cut by the outer Payload Length of 6in6', [10]), - ('Total Length 0 in 4in6, with a trailer', [16]), - ): - for seen in (base, dpkt): - with self.subTest(case=label): - self.assertEqual([len(packet['payload']) for packet in self.inputs(seen, label)], payloads) - - def test_no_input_where_the_default_engine_dissects_no_ipv4(self) -> None: - base, dpkt = self.read() - self.assertEqual(len(self.inputs(base, '4in6 behind 32 extension headers')), 1) - for label in ('4in6 in a later IPv6 fragment', '4in6 behind 33 extension headers', - '4in6 with a zero Payload Length', '4in6 jumbogram shorter than the IPv4 header', - 'IPv6 carrying UDP', 'ARP'): - for seen in (base, dpkt): - with self.subTest(case=label): - self.assertEqual(self.inputs(seen, label), []) - - def test_input_is_not_cut_at_the_total_length(self) -> None: - # #1616: the IHL's octets, then the rest of the frame, padding included - base, dpkt = self.read() - for seen in (base, dpkt): - short, = self.inputs(seen, 'Total Length below the IHL') - self.assertEqual((short['ihl'], short['tl']), (24, 10)) - self.assertEqual((len(short['header']), bytes(short['payload'])), (24, b'C' * 16)) - padded = self.inputs(seen, 'padded to the Ethernet minimum') - self.assertEqual([len(packet['payload']) for packet in padded], [26, 26]) - self.assertEqual(bytes(padded[1]['payload']), b'B' * 8 + bytes(18)) - - def test_a_packet_built_by_hand_is_an_input_too(self) -> None: - # with no record buffer attached, the octets are dpkt's own serialisation - import dpkt - - from pcapkit.toolkit import dpkt as toolkit - - for label, payload in (('4in6', b'A' * 16), ('4in6 cut by the Payload Length', b'A' * 10)): - with self.subTest(case=label): - data = toolkit.ipv4_reassembly(dpkt.ethernet.Ethernet(self.cases[label][0]), 1.0, count=1) - self.assertIsNotNone(data) - self.assertEqual((bytes(data.header), bytes(data.payload)), - (self.cases[label][0][54:74], payload)) + def test_a_total_length_below_the_ihl_leaves_the_header_whole(self) -> None: + # #1616: the header was cut to the Total Length's 10 octets + frame = ethernet(ipv4(b'C' * 16, ident=14, options=b'\x94\x04\x00\x00', length=10, mf=True)) + seen = read(self, 'dpkt', [frame]) + packet, = seen['inputs'].values() + self.assertEqual((packet['ihl'], packet['tl'], packet['mf']), (24, 10, True)) + self.assertEqual((packet['header'], packet['payload']), (frame[14:38], b'')) + + def test_a_header_cut_short_by_the_capture_is_no_input(self) -> None: + # an IHL of 15 with 28 of its 60 octets captured, which dpkt decodes; + # the next frame is still an input + cut = ethernet(ipv4(b'', ident=30, options=b'\x01' * 40, mf=True)[:28]) + plain = ethernet(ipv4(b'A' * 16, ident=31, mf=True)) + seen = read(self, 'dpkt', [cut, plain]) + self.assertEqual(sorted(seen['inputs']), [2]) + self.assertEqual((seen['inputs'][2]['header'], seen['inputs'][2]['payload']), + (plain[14:34], b'A' * 16)) if __name__ == '__main__': diff --git a/tests/toolkit/test_dpkt_pypcapfile_timestamp_rounding_1668_unit.py b/tests/toolkit/test_dpkt_pypcapfile_timestamp_rounding_1668_unit.py index 327365cd10..54eac417af 100644 --- a/tests/toolkit/test_dpkt_pypcapfile_timestamp_rounding_1668_unit.py +++ b/tests/toolkit/test_dpkt_pypcapfile_timestamp_rounding_1668_unit.py @@ -34,7 +34,7 @@ from tests.toolkit import test_pypcapfile_unit as pypcapfile_unit if TYPE_CHECKING: - from typing import Any, ClassVar, Iterator, Optional + from typing import Any, Iterator, Optional HAS_RUNTIME = all(importlib.util.find_spec(name) is not None for name in ('tbtrim', 'aenum', 'chardet', 'dictdumper')) @@ -110,7 +110,7 @@ def records(fractions: 'tuple[int, ...]', seconds: int = 1) -> 'list[tuple[int, 'sweep-pcapng-ns': ('pcapng', True, SWEEP_NS), } -#: Readings by engine and what :func:`read` builds and reads, kept across the classes. +#: Readings by engine and what :func:`read` builds and reads, kept across the tests. _READINGS = {} # type: dict[tuple[Any, ...], dict[str, Any]] @@ -174,77 +174,53 @@ def spy_trace(this: 'Any', packet: 'Any') -> 'Any': return seen -class _Agreement(unittest.TestCase): - """An engine's capture timestamps are the default engine's, at ``ts_sec=1``.""" - - #: Engine under test. - ENGINE = None # type: ClassVar[Optional[str]] - #: Inputs the engine hands over, of ``ipv4``, ``ipv6``, ``tcp`` and ``trace``. - KINDS = ('ipv4', 'ipv6', 'tcp', 'trace') - #: Containers the engine reads. - CONTAINERS = ('pcap', 'pcapng') +@unittest.skipUnless(HAS_RUNTIME, 'runtime dependencies not installed') +@unittest.skipUnless(HAS_DPKT, 'dpkt not installed') +class DPKTTimestampTests(unittest.TestCase): + """dpkt hands over the double nearest each exact timestamp, on PCAP and PCAP-NG alike.""" def setUp(self) -> None: - if self.ENGINE is None: - self.skipTest('base class') reimport_once_per_class(self) - def captures(self, sweep: bool) -> 'list[str]': - return [name for name, (container, _, _) in CAPTURES.items() - if container in self.CONTAINERS and name.startswith('sweep-') == sweep] + def check(self, container: str, nanosecond: bool, fractions: 'tuple[int, ...]', seconds: int = 1, + digits: 'Optional[int]' = None) -> 'list[float]': + """Assert every input, and every trace file name, carries the exact timestamp of its frame. - def check(self, *capture: 'Any') -> 'dict[str, Any]': - """Assert the engine's inputs and trace file names over ``capture`` are the default engine's. - - ``capture`` is what :func:`read` takes after the engine. + The arguments are what :func:`read` takes after the engine. """ - base, mine = read(self, 'default', *capture), read(self, self.ENGINE, *capture) - for kind in self.KINDS: - self.assertTrue(base[kind], f'default hands over no {kind} input') - self.assertEqual(mine[kind], base[kind], kind) - self.assertEqual(mine['names'], base['names']) - return base - - def test_ts_sec_1_hands_over_the_default_engines_timestamp(self) -> None: - for name in self.captures(sweep=False): - with self.subTest(capture=name): - base = self.check(*CAPTURES[name]) - # the default engine hands over the double nearest 1.003691 - self.assertEqual(set(base['trace'].values()), {1.003691}) - self.assertTrue(all(label.endswith('-1.003691.json') for label in base['names']), base['names']) - - def test_sweep_hands_over_the_default_engines_timestamp(self) -> None: + want = sorted(exact(seconds, fraction, nanosecond) for fraction in fractions) + seen = read(self, 'dpkt', container, nanosecond, fractions, seconds, digits) + for kind in ('ipv4', 'ipv6', 'tcp', 'trace'): + self.assertEqual(sorted(seen[kind].values()), want, kind) + self.assertEqual(sorted(name.rsplit('-', 1)[1] for name in seen['names']), + sorted(f'{stamp}.json' for stamp in want)) + return want + + def test_ts_sec_1_hands_over_the_exact_timestamp(self) -> None: + for name, capture in CAPTURES.items(): + if not name.startswith('sweep-'): + with self.subTest(capture=name): + self.assertEqual(self.check(*capture), [1.003691]) + + def test_sweep_hands_over_the_exact_timestamp(self) -> None: # the sweeps hold many a fraction ``ts_sec + frac / divisor`` misses self.assertGreater(double_rounded(SWEEP_US, False), 50) self.assertGreater(double_rounded(SWEEP_NS, True), 50) - for name in self.captures(sweep=True): - with self.subTest(capture=name): - base = self.check(*CAPTURES[name]) - _, nanosecond, fractions = CAPTURES[name] - self.assertEqual(sorted(base['trace'].values()), - sorted(exact(1, fraction, nanosecond) for fraction in fractions)) - - def test_low_decimal_precision_hands_over_the_default_engines_timestamp(self) -> None: + for name, capture in CAPTURES.items(): + if name.startswith('sweep-'): + with self.subTest(capture=name): + self.check(*capture) + + def test_low_decimal_precision_hands_over_the_exact_timestamp(self) -> None: before = decimal.getcontext().prec for seconds, fraction in LOW_PRECISION_STAMPS: for digits in LOW_PRECISIONS: with self.subTest(ts_sec=seconds, prec=digits): - base = self.check('pcap', False, (fraction,), seconds, digits) - self.assertEqual(set(base['trace'].values()), {exact(seconds, fraction, False)}) - label = f'-{exact(seconds, fraction, False)}.json' - self.assertTrue(all(name.endswith(label) for name in base['names']), base['names']) + self.check('pcap', False, (fraction,), seconds, digits) self.assertEqual(decimal.getcontext().prec, before) -@unittest.skipUnless(HAS_RUNTIME, 'runtime dependencies not installed') -@unittest.skipUnless(HAS_DPKT, 'dpkt not installed') -class DPKTTimestampAgreementTests(_Agreement): - """dpkt hands over the default engine's timestamp, on PCAP and PCAP-NG alike.""" - - ENGINE = 'dpkt' - - @unittest.skipUnless(HAS_RUNTIME, 'runtime dependencies not installed') @unittest.skipUnless(HAS_DPKT, 'dpkt not installed') class DPKTAdapterTimestampTests(unittest.TestCase): diff --git a/tests/toolkit/test_dpkt_reassembly_rawbytes_unit.py b/tests/toolkit/test_dpkt_reassembly_rawbytes_unit.py index 3e1706de9b..49603d1a33 100644 --- a/tests/toolkit/test_dpkt_reassembly_rawbytes_unit.py +++ b/tests/toolkit/test_dpkt_reassembly_rawbytes_unit.py @@ -12,9 +12,12 @@ ``len()``: :meth:`dpkt.ip6.IP6.__len__` sums ``len()`` of the extension headers rather than their wire lengths, which is 12 octets plus the rest of the frame for an Authentication header and 8 octets short for a Routing header whose Hdr Ext -Len is odd. So the extension headers are walked off the octets, by the lengths -their own fields declare (#1626), and the network layer's slice is checked -against its own header before anything is read from it. +Len is odd. :class:`~dpkt.ip6.IP6ExtensionHeader` records the wire length in +``length``, which is what :meth:`dpkt.ip6.IP6.unpack` walks the buffer by, so +that is what the offset is built from -- and the slice is then checked against +the layer's own header before it is handed back. Where the check fails, as +behind a stack of VLAN tags DPKT reads in two goes (#1646), the layer is +re-serialised instead, so the extraction goes on. Every frame below is written to a PCAP file and read back through the engine, so the tests exercise the path a capture actually takes rather than a packet with a @@ -278,40 +281,45 @@ def test_no_adapter_changes_what_the_next_one_sees(self) -> None: info = toolkit.packet2dict(frame, 0.0, data_link=LinkType.ETHERNET) self.assertEqual(info['packet'], wire) - def test_options_transport_matches_the_default_engine(self) -> None: + def test_options_transport_inputs_are_the_captured_octets(self) -> None: """Before the fix, 33 of this capture's IPv4 datagrams came back rewritten. - The number of datagrams is counted here rather than pinned: the capture is - generated by ``make samples``, and that is not currently reproducible -- - the same commit produces a 43-frame and a 44-frame file in different - worktrees -- so any absolute count is flaky whichever value it picks. + Each record is read by hand, and each input is the IHL's octets, then + the rest up to the Total Length. The number of datagrams is counted here + rather than pinned: the capture is generated by ``make samples``, and that + is not currently reproducible -- the same commit produces a 43-frame and a + 44-frame file in different worktrees -- so any absolute count is flaky + whichever value it picks. """ from pcapkit.toolkit import dpkt as toolkit - from pcapkit.toolkit import pcap as toolkit_pcap try: path = sample_path('options-transport.pcap') except FileNotFoundError as exc: self.skipTest(str(exc)) + with open(path, 'rb') as file: + octets = file.read() + records, offset = [], 24 + while offset + 16 <= len(octets): + incl_len = struct.unpack_from(' None: self.assertEqual(bytes(data.header) + bytes(data.payload), IPV4_TCP_REPACKED) self.assertNotEqual(IPV4_TCP_REPACKED, IPV4_TCP) - # the TCP segment is read out of the same serialisation of the network - # layer, so it carries the checksum dpkt.ip.IP.__bytes__ fills in with - # the IPv4 header's (#1626) + # the TCP layer alone round-trips: dpkt.tcp.TCP.__bytes__ recomputes + # nothing, the pseudo-header checksum being dpkt.ip.IP.__bytes__'s doing tcp = toolkit.tcp_reassembly(dpkt.ethernet.Ethernet(ETH_IPV4 + IPV4_TCP), 0.0) - self.assertEqual(bytes(tcp.header), IPV4_TCP_REPACKED[20:40]) + self.assertEqual(bytes(tcp.header), IPV4_TCP[20:40]) def test_attaching_the_buffer_makes_the_octets_the_wires_again(self) -> None: import dpkt @@ -374,19 +381,79 @@ def test_attaching_the_buffer_makes_the_octets_the_wires_again(self) -> None: data = toolkit.ipv4_reassembly(packet, 0.0) self.assertEqual(bytes(data.header) + bytes(data.payload), IPV4_TCP) - def test_a_misplaced_slice_is_refused_rather_than_returned(self) -> None: - """A buffer that is not the packet's own cannot be sliced into.""" + def test_a_misplaced_slice_is_never_returned(self) -> None: + """A buffer that is not the packet's own is not sliced into: the layer is re-serialised.""" import dpkt from pcapkit.toolkit import dpkt as toolkit - from pcapkit.utilities.exceptions import ProtocolError frame = ETH_IPV4 + IPV4_TCP packet = dpkt.ethernet.Ethernet(frame) # one octet short at the front, so every layer sits one octet early toolkit.attach_buffer(packet, frame[1:]) - with self.assertRaises(ProtocolError): - toolkit.ipv4_reassembly(packet, 0.0) + data = toolkit.ipv4_reassembly(packet, 0.0) + self.assertEqual(bytes(data.header) + bytes(data.payload), IPV4_TCP_REPACKED) + self.assertEqual((packet.ip.sum, packet.ip.tcp.sum), (0, 0)) + + +#: VLAN tag stacks DPKT reads in two goes, keeping only the last go's tags, each +#: as the octets between the source address and the IPv4 EtherType (#1646). +VLAN_STACKS = { + '802.1Q, 802.1ad': bytes.fromhex('8100 0064 88a8 0065'), + '802.1ad, 802.1ad': bytes.fromhex('88a8 0064 88a8 0065'), + 'three 802.1Q': bytes.fromhex('8100 0064 8100 0065 8100 0066'), +} +#: A Cisco ISL header, which DPKT records as a tag and then replaces with the +#: 802.1Q tag of the frame it carries; its length counts that frame and a CRC. +ISL_HDR = (bytes.fromhex('01000c0000 00') + b'\xaa' * 6 + struct.pack('>H', 26 + 18 + 44 + 4) + + bytes.fromhex('aaaa03 00000c 0064') + bytes(4)) + + +@unittest.skipUnless(HAS_RUNTIME and HAS_DPKT, 'runtime dependencies not installed') +class DPKTVLANStackTests(unittest.TestCase): + """#1646: a frame whose layers DPKT's own lengths misplace ends no extraction. + + The adapters looked for the IP header 4 octets early for each tag DPKT + dropped, found none and raised ``ProtocolError``, which ended the + extraction. The datagram behind the tags has its checksums filled in, so + DPKT's re-serialisation of it is the captured octets. + + """ + + def setUp(self) -> None: + reimport_once_per_class(self) + + @staticmethod + def frames() -> 'dict[str, bytes]': + frames = {label: b'\xbb' * 6 + b'\xaa' * 6 + tags + b'\x08\x00' + IPV4_TCP_REPACKED + for label, tags in VLAN_STACKS.items()} + frames['Cisco ISL, 802.1Q'] = (ISL_HDR + b'\xbb' * 6 + b'\xaa' * 6 + bytes.fromhex('8100 0005 0800') + + IPV4_TCP_REPACKED + bytes(4)) + return frames + + def test_the_inputs_are_the_captured_datagram(self) -> None: + import pcapkit + from pcapkit.const.reg.linktype import LinkType + from pcapkit.toolkit import dpkt as toolkit + + for label, frame in self.frames().items(): + with self.subTest(stack=label): + path = write_pcap((frame, ETH_IPV4 + IPV4_TCP_REPACKED)) + self.addCleanup(os.remove, path) + with warnings.catch_warnings(): + warnings.simplefilter('ignore') + extractor = pcapkit.extract(fin=path, nofile=True, store=True, engine='dpkt', + reassembly=True, ipv4=True, tcp=True) + self.addCleanup(close_extractor, extractor) + self.assertEqual(len(extractor.frame), 2) + tagged = extractor.frame[0] + data = toolkit.ipv4_reassembly(tagged, 0.0) + self.assertEqual((bytes(data.header), bytes(data.payload)), + (IPV4_TCP_REPACKED[:20], IPV4_TCP_REPACKED[20:])) + for record in (toolkit.tcp_reassembly(tagged, 0.0), + toolkit.tcp_traceflow(tagged, 0.0, data_link=LinkType.ETHERNET)): + self.assertEqual((bytes(record.header), bytes(record.payload)), + (IPV4_TCP_REPACKED[20:40], b'data')) if __name__ == '__main__': diff --git a/tests/toolkit/test_dpkt_tcp_ns_flags_1618_unit.py b/tests/toolkit/test_dpkt_tcp_ns_flags_1618_unit.py index 2f61981240..f8e9cbcde3 100644 --- a/tests/toolkit/test_dpkt_tcp_ns_flags_1618_unit.py +++ b/tests/toolkit/test_dpkt_tcp_ns_flags_1618_unit.py @@ -1,12 +1,12 @@ # -*- coding: utf-8 -*- -"""The dpkt engine reads RST, SYN and FIN as the default engine does, whatever the NS bit. C.f. #1618. +"""The dpkt engine reads RST, SYN and FIN from the flags octet, whatever the NS bit. C.f. #1618. `dpkt`_ 1.9.8 makes :attr:`dpkt.tcp.TCP.flags` nine bits wide, its top bit the NS bit (:rfc:`3540`). :func:`~pcapkit.toolkit.dpkt.tcp_reassembly` and :func:`~pcapkit.toolkit.dpkt.tcp_traceflow` read each flag by its position among that field's binary digits, padded to eight, so with NS set every flag was read one bit low: a SYN as a FIN, an RST as a SYN, a PSH as an RST. They now mask -each flag out of the flags octet, as the default engine reads it. +each flag out of the flags octet. Every capture is built here, so the module is in the unit tier. @@ -84,22 +84,22 @@ def test_a_syn_with_ns_set_is_read_as_a_syn(self) -> None: self.assertEqual({flag: seen[aspect][1][flag] for flag in ('rst', 'syn', 'fin')}, {'rst': False, 'syn': True, 'fin': False}) - def test_extraction_agrees_with_the_default_engine(self) -> None: - # Every input to TCP reassembly and flow tracing, every field of it, and - # what comes out, for every flags octet with NS and the reserved bits - # clear and set. + def test_extraction_reads_each_flag_from_the_flags_octet(self) -> None: + # Every input to TCP reassembly and flow tracing, for every flags octet + # with NS and the reserved bits clear and set; the header handed on is + # the captured one. cases = [(nibble, octet) for nibble in (0x0, NS, 0xE, 0xF) for octet in range(256)] - frames = [wire.ethernet(wire.ipv4(segment(number, nibble, octet), proto=6, df=True), 0x0800) - for number, (nibble, octet) in enumerate(cases)] - base = read(self, 'default', frames) - self.assertEqual(sorted(base['tcp']), list(range(1, len(cases) + 1))) - self.assertEqual([{flag: base['tcp'][number][flag] for flag in ('rst', 'syn', 'fin')} - for number in range(1, len(cases) + 1)], - [expected(octet) for _, octet in cases]) + headers = [segment(number, nibble, octet) for number, (nibble, octet) in enumerate(cases)] + frames = [wire.ethernet(wire.ipv4(header, proto=6, df=True), 0x0800) for header in headers] + numbers = list(range(1, len(cases) + 1)) seen = read(self, 'dpkt', frames) - for aspect in ('tcp', 'trace', 'ids', 'flows'): + for aspect in ('tcp', 'trace'): with self.subTest(aspect=aspect): - self.assertEqual(seen[aspect], base[aspect]) + self.assertEqual(sorted(seen[aspect]), numbers) + self.assertEqual([{flag: seen[aspect][number][flag] for flag in ('rst', 'syn', 'fin')} + for number in numbers], + [expected(octet) for _, octet in cases]) + self.assertEqual([seen[aspect][number]['header'] for number in numbers], headers) if __name__ == '__main__': diff --git a/tests/toolkit/test_dpkt_unit.py b/tests/toolkit/test_dpkt_unit.py index 6aa19334c4..a6c1d335c1 100644 --- a/tests/toolkit/test_dpkt_unit.py +++ b/tests/toolkit/test_dpkt_unit.py @@ -76,27 +76,42 @@ def pack(self) -> bytes: return b'I' * 40 + b'H' * 8 + b'F' * 8 + b'PAYLOAD' -def _ipv6_frame(payload: bytes, nxt: int, *, src: str, dst: str, flow: int = 0): - """Build an Ethernet frame carrying IPv6, as the DPKT engine hands it over: its record octets attached.""" - import dpkt +class TCP: + __hdr_fields__ = ('sport', 'dport', 'seq', 'ack', 'flags') + __hdr_len__ = 4 + #: Data Offset, in 32-bit words. ``off * 4`` has to agree with the header + #: portion of :meth:`pack`, the way it does on a real :class:`dpkt.tcp.TCP`. + off = 1 + sport = 2345 + dport = 443 + seq = 20 + ack = 15 + flags = 0b00010011 + data = b'ip6' + + def pack(self) -> bytes: + return b'HEADip6' + + +class FakeTCPIPv6: + __hdr_fields__ = ('src', 'dst') + src = ip_address('2001:db8::10').packed + dst = ip_address('2001:db8::20').packed + + def __init__(self) -> None: + self.data = TCP() - from pcapkit.toolkit import dpkt as toolkit - octets = (b'\xbb' * 6 + b'\xaa' * 6 + b'\x86\xdd' - + struct.pack('>IHBB16s16s', 0x60000000 | flow, len(payload), nxt, 64, - ip_address(src).packed, ip_address(dst).packed) + payload) - packet = dpkt.ethernet.Ethernet(octets) - toolkit.attach_buffer(packet, octets) - return packet +class FakeDPKTPacket: + __hdr_fields__ = ('link',) + link = 1 + def __init__(self) -> None: + self.ip6 = FakeTCPIPv6() + self.data = self.ip6 -def _fake_fragment_frame(): - """A later IPv6 fragment, behind a Hop-by-Hop header, with the fields :class:`FakeFragment` pins.""" - frag = FakeFragment() - payload = (bytes([44, 0, 1, 4, 0, 0, 0, 0]) # Hop-by-Hop, one PadN - + struct.pack('>BBHI', frag.nxt, 0, frag.frag_off << 3 | frag.m_flag, frag.id) - + b'PAYLOAD') - return _ipv6_frame(payload, 0, src='2001:db8::1', dst='2001:db8::2', flow=FakeIPv6.flow) + def pack(self) -> bytes: + return b'packet' @unittest.skipUnless(HAS_RUNTIME and HAS_DPKT, 'runtime dependencies not installed') @@ -218,7 +233,8 @@ def test_every_adapter_hands_on_a_float_timestamp(self) -> None: results = { 'ipv4': toolkit.ipv4_reassembly(self._make_ipv4_tcp_packet(fragmented=True), stamp, count=1), - 'ipv6': toolkit.ipv6_reassembly(_fake_fragment_frame(), stamp, count=1), + 'ipv6': toolkit.ipv6_reassembly(types.SimpleNamespace(ip6=FakeIPv6(FakeFragment())), + stamp, count=1), 'tcp': toolkit.tcp_reassembly(packet, stamp, count=1), 'trace': toolkit.tcp_traceflow(packet, stamp, data_link=LinkType.ETHERNET, count=1), } @@ -278,13 +294,11 @@ def fields(data): # type: ignore[no-untyped-def] (99, True, bytes(20))) self.assertIsNone(toolkit.ipv4_reassembly(ipv6, 1.0, count=1)) - def test_tcp_helpers_cover_tcp_over_ipv6(self) -> None: + def test_tcp_helpers_cover_ipv6_and_data_payload_fallback(self) -> None: from pcapkit.const.reg.linktype import LinkType from pcapkit.toolkit import dpkt as toolkit - # SYN and FIN set, with ACK - segment = struct.pack('>HHIIBBHHH', 2345, 443, 20, 15, 5 << 4, 0b00010011, 1024, 0, 0) + b'ip6' - packet = _ipv6_frame(segment, 6, src='2001:db8::10', dst='2001:db8::20') + packet = FakeDPKTPacket() tcp = toolkit.tcp_reassembly(packet, 60.5, count=12) self.assertIsNotNone(tcp) assert tcp is not None @@ -302,9 +316,9 @@ def test_tcp_helpers_cover_tcp_over_ipv6(self) -> None: self.assertEqual(flow.dst, ip_address('2001:db8::20')) self.assertTrue(flow.syn) self.assertTrue(flow.fin) - self.assertEqual(flow.frame['ETHERNET']['IP6']['TCP']['sport'], 2345) + self.assertEqual(flow.frame['ETHERNET']['FakeTCPIPv6']['TCP']['sport'], 2345) - def test_ipv6_header_length_and_reassembly_of_a_fragment(self) -> None: + def test_ipv6_header_length_and_reassembly_with_fragment_fake(self) -> None: from pcapkit.const.reg.transtype import TransType from pcapkit.toolkit import dpkt as toolkit @@ -312,8 +326,7 @@ def test_ipv6_header_length_and_reassembly_of_a_fragment(self) -> None: ipv6 = FakeIPv6(frag) self.assertEqual(toolkit.ipv6_hdr_len(ipv6), 48) - # the same fields on the wire, as the engine hands them over - packet = _fake_fragment_frame() + packet = types.SimpleNamespace(ip6=ipv6) reassembled = toolkit.ipv6_reassembly(packet, 0.0, count=5) self.assertIsNotNone(reassembled) assert reassembled is not None @@ -337,8 +350,6 @@ def test_ipv6_header_length_and_reassembly_of_a_fragment(self) -> None: self.assertIsNone(toolkit.ipv6_reassembly(types.SimpleNamespace(), 0.0, count=1)) self.assertIsNone(toolkit.ipv6_reassembly(types.SimpleNamespace(ip6=FakeIPv6()), 0.0, count=1)) - self.assertIsNone(toolkit.ipv6_reassembly(_ipv6_frame(b'PAYLOAD', 253, src='2001:db8::1', - dst='2001:db8::2'), 0.0, count=1)) # --------------------------------------------------------------------------- @@ -845,52 +856,79 @@ def test_engine_reassembles_ipv6_fragments_end_to_end(self) -> None: self.assertEqual(bytes(datagrams[0].payload), body) +def _tcp_payloads(path: str) -> 'dict[int, tuple[int, bytes]]': + """Each TCP segment straight over Ethernet and IP in a little-endian PCAP, read by hand. + + Returns: + The segment's sequence number and payload, by frame number. + + """ + with open(path, 'rb') as file: + data = file.read() + segments = {} # type: dict[int, tuple[int, bytes]] + offset, number = 24, 0 + while offset + 16 <= len(data): + incl_len = struct.unpack_from('> 4) * 4:]) + return segments + + +def _assemble(segments: 'list[tuple[int, bytes]]') -> bytes: + """The octets ``(seq, data)`` segments carry from the lowest sequence number on, the first copy winning.""" + start = min(seq for seq, _ in segments) + octets = {} # type: dict[int, int] + for seq, data in segments: + for index, octet in enumerate(data): + octets.setdefault(seq - start + index, octet) + return bytes(octets[index] for index in range(len(octets))) + + @unittest.skipUnless(HAS_RUNTIME and HAS_DPKT, 'runtime dependencies not installed') -class DPKTEngineParityTests(unittest.TestCase): - """The ``dpkt`` engine must agree with the default engine octet for octet. +class DPKTEngineReassemblyTests(unittest.TestCase): + """The ``dpkt`` engine reassembles the captured octets, octet for octet. This is the check that would have caught the Data Offset defect: the split was wrong in a way that left the datagram *lengths* untouched, because the reassembly buffer is indexed by sequence number, so only a byte-level - comparison against a known-good engine shows it. + comparison against the capture's own octets shows it. """ def setUp(self) -> None: reimport_once_per_class(self) - def _reassemble(self, path: str, engine: str): + def test_tcp_reassembly_is_the_captured_payload(self) -> None: import pcapkit - extractor = pcapkit.extract(fin=path, nofile=True, store=False, engine=engine, - tcp=True, reassembly=True, reasm_strict=True) - try: - return [ - ( - datagram.completed, - datagram.payload if isinstance(datagram.payload, bytes) - else b''.join(datagram.payload), - ) - for datagram in extractor.reassembly.tcp - ] - finally: - close_extractor(extractor) - - def test_tcp_reassembly_matches_the_default_engine(self) -> None: try: path = sample_path('test.pcap') except FileNotFoundError as exc: self.skipTest(str(exc)) - default = self._reassemble(path, 'default') - dpkt_out = self._reassemble(path, 'dpkt') + extractor = pcapkit.extract(fin=path, nofile=True, store=False, engine='dpkt', + tcp=True, reassembly=True, reasm_strict=True) + try: + datagrams = list(extractor.reassembly.tcp) + finally: + close_extractor(extractor) - self.assertGreater(len(default), 0, 'sample capture reassembled nothing') - self.assertEqual(len(dpkt_out), len(default)) - for index, (expected, actual) in enumerate(zip(default, dpkt_out)): - with self.subTest(datagram=index): - self.assertEqual(actual[0], expected[0]) - self.assertEqual(actual[1], expected[1]) + segments = _tcp_payloads(path) + self.assertGreater(len(datagrams), 0, 'sample capture reassembled nothing') + for datagram in datagrams: + with self.subTest(frames=datagram.index): + payload = (datagram.payload if isinstance(datagram.payload, bytes) + else b''.join(datagram.payload)) + self.assertTrue(datagram.completed) + self.assertEqual(payload, _assemble([segments[number] for number in datagram.index])) @unittest.skipUnless(HAS_RUNTIME and HAS_DPKT, 'runtime dependencies not installed') @@ -941,24 +979,24 @@ def test_a_frame_from_elsewhere_has_no_timestamp_and_says_so(self) -> None: with self.assertRaises(UnsupportedCall): packet2timestamp(bare) - def test_following_a_stream_agrees_with_the_default_engine(self) -> None: + def test_following_a_stream_reads_every_stream(self) -> None: """And the real timestamp changes nothing, which is worth pinning. TCP reassembly has no timeout by default, so the value never reaches a decision -- the fix removes a workaround rather than altering a result. + ``in.pcap`` holds two TCP conversations, of two frames and one, each + segment alone, so none has a payload to reassemble. """ import tempfile from pcapkit.interface.misc import follow_tcp_stream - def follow(engine: str): - with tempfile.TemporaryDirectory() as tempdir: - streams = follow_tcp_stream(fin=sample_path('in.pcap'), engine=engine, - fout=tempdir, format='json') - return [(len(stream.packets), stream.conversations) for stream in streams] - - self.assertEqual(follow('dpkt'), follow('default')) + with tempfile.TemporaryDirectory() as tempdir: + streams = follow_tcp_stream(fin=sample_path('in.pcap'), engine='dpkt', + fout=tempdir, format='json') + self.assertEqual([(len(stream.packets), stream.conversations) for stream in streams], + [(2, ()), (1, ())]) if __name__ == '__main__': diff --git a/tests/toolkit/test_pcap_tcp_segment_ipv6_version_1666_unit.py b/tests/toolkit/test_pcap_tcp_segment_ipv6_version_1666_unit.py index fd142121e5..051ec97f6d 100644 --- a/tests/toolkit/test_pcap_tcp_segment_ipv6_version_1666_unit.py +++ b/tests/toolkit/test_pcap_tcp_segment_ipv6_version_1666_unit.py @@ -10,8 +10,8 @@ IPv6 now rejects a Version other than 6 (#1680), so such a header is kept raw: its frame hands over no input, and every frame after it is still read. -dpkt and scapy still read those headers as IPv6 and hand their segments over, -which the best-effort ruling on third-party engines allows (#1687). +scapy still reads those headers as IPv6 and hands their segments over, which +the best-effort ruling on third-party engines allows (#1687). Every capture is built here, so the module is in the unit tier. @@ -38,7 +38,7 @@ HAS_RUNTIME = all(importlib.util.find_spec(name) is not None for name in RUNTIME_DEPS) #: The third-party engines read here besides the default engine. -ENGINES = ('dpkt', 'scapy') +ENGINES = ('scapy',) #: The endpoints of every IPv6 header built here, then of every IPv4 one. IPV6 = (ipaddress.IPv6Address(wire.IPV6_A), ipaddress.IPv6Address(wire.IPV6_B)) @@ -195,7 +195,7 @@ def test_the_default_engine_reads_only_the_controls(self) -> None: self.assertEqual(seen['IPv6'], {}) self.assertEqual(seen['datagrams']['IPv6'], []) - def test_dpkt_and_scapy_read_every_segment(self) -> None: + def test_scapy_reads_every_segment(self) -> None: frames = build() for container in ('pcap', 'pcapng'): for engine in ENGINES: