refactor(toolkit): revert pypcapfile's agreement-only code (#1708)

docs/source/contributing/pep.rst

@@ -563,13 +563,15 @@ both of the following are worth knowing before reaching for them:
   sends the loader to ``find_library("pcap")``. No compiler and no
   :file:`pcap.h` are needed; a ``libpcap.so.1`` is.
 
-Both engines support less than the ``default`` engine does, deliberately and
-noisily: `pypcap`_ performs no protocol dissection, so it disables reassembly
-*and* flow tracing; `pypcapfile`_ has no IPv6 decoder, so it disables IPv6
-reassembly while keeping IPv4 and TCP. Each gap is announced through an
+Both engines support less than the ``default`` engine does, deliberately:
+`pypcap`_ performs no protocol dissection, so it disables reassembly *and* flow
+tracing; `pypcapfile`_ has no IPv6 decoder, so it disables IPv6 reassembly while
+keeping IPv4 and TCP. Each gap is announced through an
 :class:`~pcapkit.utilities.warnings.AttributeWarning` or an outright exception
-rather than by silently returning nothing --
-:doc:`/pcapkit/foundation/engines/index` tabulates them.
+rather than by silently returning nothing, save that `pypcapfile`_ leaves TCP
+behind a VLAN tag, in a tunnel or behind an extension header (an IPv6 one, or AH
+in IPv4) out without one -- :doc:`/pcapkit/foundation/engines/index` tabulates
+them.
 
 Adding a further engine does not mean adding handler methods to
 :class:`~pcapkit.foundation.extraction.Extractor`, as the thread describes. A new engine subclasses

docs/source/ext.rst

@@ -420,7 +420,9 @@ The available engines and the file formats they read:
    whatsoever, so they support neither reassembly nor flow tracing; `PyPCAPFile`_
    has no IPv6 decoder, so it supports IPv4 and TCP reassembly but not IPv6. What
    matters is that the gap is *announced* -- each engine warns, or raises, for the
-   capability it cannot provide, rather than silently producing an empty result. See
+   capability it cannot provide, rather than silently producing an empty result,
+   save that `PyPCAPFile`_ leaves TCP behind a VLAN tag, in a tunnel or behind an
+   extension header (an IPv6 one, or AH in IPv4) out without a warning. See
    :doc:`pcapkit/foundation/engines/index` for the full table.
 
 .. note::

docs/source/index.rst

@@ -147,8 +147,10 @@ Speed is not free. Every third-party engine supports **less** than the
 dissection at all, so they offer neither reassembly nor flow tracing, and
 :class:`~pcapkit.foundation.engines.pypcapfile.PyPCAPFile` has no IPv6 decoder,
 so IPv6 reassembly is unavailable. Each gap is announced with a warning or an
-exception rather than silently returning nothing;
-:doc:`pcapkit/foundation/engines/index` tabulates them.
+exception rather than silently returning nothing, save that
+:class:`~pcapkit.foundation.engines.pypcapfile.PyPCAPFile` leaves TCP behind a
+VLAN tag, in a tunnel or behind an extension header (an IPv6 one, or AH in IPv4)
+out without one; :doc:`pcapkit/foundation/engines/index` tabulates them.
 
 Every engine also answers a preflight check --
 :meth:`~pcapkit.foundation.engines.engine.EngineBase.unsupported_reason`, which

docs/source/pcapkit/foundation/engines/3rdparty.rst

@@ -399,9 +399,7 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
    TCP and UDP and nothing else. IPv6 reassembly is therefore unavailable and is
    disabled -- with an :class:`~pcapkit.utilities.warnings.AttributeWarning` --
    when requested; IPv4 and TCP reassembly and TCP flow tracing remain
-   available. The IP packet of a BSD loopback or raw-IP frame, which it does
-   not decode, is read by :mod:`pcapkit.toolkit.pypcapfile` instead
-   (:issue:`1667`). PCAP-NG is rejected with a
+   available. PCAP-NG is rejected with a
    :exc:`~pcapkit.utilities.exceptions.FormatError`. `PyPCAPFile`_ 0.12.0 drops
    every record of a big-endian savefile, every record from a zero-length one
    on, and a truncated final record, so the engine reads the records itself,

docs/source/pcapkit/foundation/engines/index.rst

@@ -70,7 +70,8 @@ class hierarchy of :mod:`pcapkit.foundation.engines`:
        click PyPCAPFile "/pcapkit/foundation/engines/3rdparty.html#pcapkit.foundation.engines.pypcapfile.PyPCAPFile"
 
 Not every engine can do everything :class:`~pcapkit.foundation.extraction.Extractor`
-offers, and the ones that cannot say so rather than quietly doing less:
+offers, and the ones that cannot say so rather than quietly doing less, unless
+their row says otherwise:
 
 +-----------------------------------------------------------------+---------------------------------------------------------------+
 | Engine                                                          | Gap, and how it is surfaced                                   |
@@ -92,11 +93,12 @@ offers, and the ones that cannot say so rather than quietly doing less:
 | :class:`~pcapkit.foundation.engines.pypcapfile.PyPCAPFile`      | no IPv6 decoder, hence no IPv6 reassembly -- disabled with an |
 |                                                                 | :class:`~pcapkit.utilities.warnings.AttributeWarning`, and    |
 |                                                                 | :func:`~pcapkit.toolkit.pypcapfile.ipv6_reassembly` raises;   |
-|                                                                 | TCP over IPv6, tunnelled in IP (e.g. 6in4) or not, is left    |
-|                                                                 | out of TCP reassembly and flow tracing, with an               |
+|                                                                 | TCP over IPv6 straight over Ethernet is left out of TCP       |
+|                                                                 | reassembly and flow tracing, with an                          |
 |                                                                 | :class:`~pcapkit.utilities.warnings.AttributeWarning` once    |
-|                                                                 | per capture; IPv4 behind 802.1Q/802.1ad VLAN tags is read as  |
-|                                                                 | it is untagged; PCAP savefiles only, otherwise                |
+|                                                                 | per capture, and TCP behind a VLAN tag, in a tunnel or behind |
+|                                                                 | an extension header (an IPv6 one, or AH in IPv4) without one; |
+|                                                                 | PCAP savefiles only, otherwise                                |
 |                                                                 | :exc:`~pcapkit.utilities.exceptions.FormatError`              |
 +-----------------------------------------------------------------+---------------------------------------------------------------+
 | :class:`~pcapkit.foundation.engines.dpkt.DPKT`,                 | no PCAP output, since each frame reaches the writer as a      |

docs/source/pcapkit/protocols/internet/internet.rst

@@ -37,8 +37,9 @@ A chain of IPv6 extension headers is dissected to at most
 a row, whether it follows IPv6 or another layer such as IPv4. What follows is
 kept as :class:`~pcapkit.protocols.misc.raw.Raw`, with one
 :exc:`~pcapkit.utilities.warnings.ProtocolWarning` (:issue:`1604`). The
-``pypcapfile``, ``dpkt`` and ``scapy`` toolkits stop at the same depth. The
-``pyshark`` engine leaves the walk to its own parser, which does not stop
-there, so it may still find the upper layer past it (:issue:`1609`).
+``dpkt`` and ``scapy`` toolkits stop at the same depth. The ``pyshark`` engine
+leaves the walk to its own parser, which does not stop there, so it may still
+find the upper layer past it (:issue:`1609`). The ``pypcapfile`` toolkit walks
+no extension header chain, so it finds no upper layer behind one.
 
 .. autodata:: pcapkit.protocols.internet.internet.EXTENSION_HEADER_LIMIT

docs/source/pcapkit/protocols/protocol.rst

@@ -116,9 +116,9 @@ there back out, as they do one the TCP parser rejected (:issue:`1518`). A
 protocol constructed from its own octets is not counted itself, so the count
 starts at the first layer it dissects. One chain of extension headers is also bounded on
 its own, by :data:`~pcapkit.protocols.internet.internet.EXTENSION_HEADER_LIMIT`.
-The ``pyshark`` engine dissects with its own parser, which does not apply this
-bound (:issue:`1609`). The ``dpkt``, ``scapy`` and ``pypcapfile`` toolkits
-count the layers they walk as the default engine does, and stop at it.
+The ``pyshark`` and ``pypcapfile`` engines dissect with their own parsers,
+which do not apply this bound (:issue:`1609`). The ``dpkt`` and ``scapy``
+toolkits count the layers they walk as the default engine does, and stop at it.
 
 .. autodata:: pcapkit.protocols.protocol.FRAME_LAYER_LIMIT
 

docs/source/pcapkit/toolkit/3rdparty.rst

@@ -184,25 +184,15 @@ cannot be used for it.
    :data:`None` -- which would be indistinguishable from "this frame carries no
    IPv6 fragment".
 
-   Nor does it decode a VLAN tag or a tunnel. The adapters read an IPv4 packet
-   behind 802.1Q and 802.1ad tags as `PyPCAPFile`_ reads an untagged one, and
-   TCP tunnelled in IP (e.g. 4in4, 4in6) from its innermost IPv4 packet, keyed by
-   that packet's addresses as the default engine keys it, while TCP over IPv6,
-   tunnelled (e.g. 6in4, 6in6) or not, is left out of
+   TCP over IPv6 is therefore left out of
    :func:`~pcapkit.toolkit.pypcapfile.tcp_reassembly` and
    :func:`~pcapkit.toolkit.pypcapfile.tcp_traceflow`, with an
-   :class:`~pcapkit.utilities.warnings.AttributeWarning` once per capture.
-
-   Nor does it decode a link layer but Ethernet. The adapters read the IP packet
-   of a BSD loopback (``NULL``, ``LOOP``) or raw-IP (``RAW``, ``IPV4``,
-   ``IPV6``) frame themselves, as the default engine dissects it (see
-   :data:`~pcapkit.toolkit.pypcapfile.READ_LINKTYPES`, :issue:`1667`). Every
-   other link type is left unread, as the default engine leaves it, and the
-   engine warns of it with an :class:`~pcapkit.utilities.warnings.AttributeWarning`.
-   Nor do the adapters read deeper than the default engine dissects: each VLAN
-   tag, IP header, AH and IPv6 extension header is one of the
-   :data:`~pcapkit.protocols.protocol.FRAME_LAYER_LIMIT` layers of a frame, and
-   nothing past the last of them is read (:issue:`1609`).
+   :class:`~pcapkit.utilities.warnings.AttributeWarning` once per capture when
+   the IPv6 packet is straight over Ethernet. `PyPCAPFile`_ decodes no link
+   layer but Ethernet, and no VLAN tag or tunnel, so a frame of any other link
+   type is left undecoded, with the engine's own warning, and TCP behind a VLAN
+   tag, in a tunnel or behind an extension header (an IPv6 one, or AH in IPv4)
+   is left out without one.
 
 .. autofunction:: pcapkit.toolkit.pypcapfile.ipv4_reassembly
 
@@ -226,9 +216,6 @@ Auxiliary Functions
 Internal Definitions
 --------------------
 
-.. autodata:: pcapkit.toolkit.pypcapfile.READ_LINKTYPES
-   :no-value:
-
 .. autodata:: pcapkit.toolkit.pypcapfile.TCP_MIN_HEADER_LEN
 
 .. autodata:: pcapkit.toolkit.pypcapfile.IPV4_FLAG_DF

pcapkit/foundation/engines/pypcapfile.py

@@ -44,18 +44,15 @@ class PyPCAPFile(EngineBase['PCAPFilePacket']):
     * reads PCAP savefiles only, raising
       :exc:`~pcapkit.utilities.exceptions.FormatError` on PCAP-NG,
     * reads the savefile's records itself, into `PyPCAPFile`_'s own packet
-      objects, as `PyPCAPFile`_ 0.12.0 drops records the ``default`` engine
-      reads -- see :meth:`_read_records`,
+      objects, as `PyPCAPFile`_ 0.12.0 drops some of them -- see
+      :meth:`_read_records`,
     * disables IPv6 reassembly, warning as it does so, while leaving IPv4 and
-      TCP reassembly and TCP flow tracing in place,
+      TCP reassembly and TCP flow tracing in place, and
     * leaves TCP over IPv6 out of TCP reassembly and flow tracing, with one
-      :exc:`~pcapkit.utilities.warnings.AttributeWarning` per capture -- see
-      :mod:`pcapkit.toolkit.pypcapfile` -- and
-    * leaves the frames of a link type `PyPCAPFile`_ has no decoder for
-      undecoded, from which :mod:`pcapkit.toolkit.pypcapfile` reads the IP
-      packet of the BSD loopback and raw-IP ones itself, as the ``default``
-      engine dissects it (see
-      :data:`~pcapkit.toolkit.pypcapfile.READ_LINKTYPES`).
+      :exc:`~pcapkit.utilities.warnings.AttributeWarning` per capture when the
+      IPv6 packet is straight over Ethernet, and TCP behind a VLAN tag, in a
+      tunnel or behind an extension header (an IPv6 one, or AH in IPv4) out
+      without one -- see :mod:`pcapkit.toolkit.pypcapfile`.
 
     The engine stops decoding at the network layer rather than descending into
     the transport layer. `PyPCAPFile`_ decoders *replace* the payload bytes of
@@ -189,10 +186,8 @@ def run(self) -> 'None':
                   support such operations.
                 * if IPv6 reassembly is enabled, as :mod:`pcapfile` has no IPv6
                   decoder.
-                * if the savefile's link layer type is left unread: neither
-                  :mod:`pcapfile` has a decoder for it nor
-                  :mod:`pcapkit.toolkit.pypcapfile` reads it (see
-                  :meth:`_get_decoder`).
+                * if :mod:`pcapfile` has no decoder for the savefile's link layer
+                  type, in which case frames are left undecoded.
 
         Raises:
             FormatError: If the file format is not supported, i.e., not a PCAP
@@ -309,36 +304,29 @@ def _get_decoder(self, linktype: 'int') -> 'Optional[Callable[..., Any]]':
             The decoder class, or :data:`None` when :mod:`pcapfile` has none.
 
         Warns:
-            AttributeWarning: If no decoder is available and the link layer type
-                is none of the :data:`~pcapkit.toolkit.pypcapfile.READ_LINKTYPES`,
-                as frames will then be left hexlified -- see :meth:`run` -- and
-                no reassembly or flow tracing is possible. The frames of those
-                link types are left undecoded too, but reassembly and flow
-                tracing read the IP packet in them, as the ``default`` engine
-                does, so nothing is lost there to warn of.
+            AttributeWarning: If no decoder is available, as frames will then be
+                left hexlified -- see :meth:`run` -- and no reassembly or flow
+                tracing is possible.
 
         """
-        from pcapkit.toolkit.pypcapfile import READ_LINKTYPES  # isort:skip
-
         try:
             decoder = self._expkg.linklayer.clookup(linktype)
         except IndexError:  # malformed entry in ``pcapfile.linklayer.__LL_TYPES__``
             decoder = None
 
         if not callable(decoder):
-            if linktype not in READ_LINKTYPES:
-                warn(f'unrecognised link layer protocol: {self._dlink!r}; frames will be left '
-                     'undecoded and all analysis functions ignored', AttributeWarning,
-                     stacklevel=stacklevel())
+            warn(f'unrecognised link layer protocol: {self._dlink!r}; frames will be left '
+                 'undecoded and all analysis functions ignored', AttributeWarning,
+                 stacklevel=stacklevel())
             return None
         return decoder
 
     def _read_records(self, file: 'NamedStream',
                       header: 'PCAPFileHeader') -> 'Iterator[PCAPFilePacket]':
-        """Read the savefile's records, as the ``default`` engine reads them.
+        """Read the savefile's records, every one of them.
 
         :func:`pcapfile.savefile._read_a_packet` (pypcapfile 0.12.0) drops
-        records the ``default`` engine reads, in two ways:
+        records in two ways:
 
         * It picks the byte order of each record header by testing
           ``hdrp[0].byteorder == 'big'`` -- :class:`bytes` against :class:`str`,
@@ -354,9 +342,8 @@ def _read_records(self, file: 'NamedStream',
         builds with ``layers=0`` -- the frame hexlified, pointing at the
         savefile's own header -- so :meth:`_decode` and
         :mod:`pcapkit.toolkit.pypcapfile` see what :mod:`pcapfile` hands them.
-        It stops where :class:`~pcapkit.protocols.misc.pcap.frame.Frame` stops,
-        at a tail too short for a record header, and keeps a record whose capture
-        length runs past the data as that does: the capture length as declared,
+        It stops at a tail too short for a record header, and keeps a record
+        whose capture length runs past the data: the capture length as declared,
         the octets present as the frame.
 
         Args:

pcapkit/toolkit/dpkt.py

@@ -649,9 +649,8 @@ def _ipv4_header_accepted(header: 'bytes') -> 'bool':
 
     Note:
         A header of version 4 with no options is not parsed, as the parser
-        checks nothing else of one; any other is asked of the parser itself, as
-        the pypcapfile engine asks it (see
-        :func:`pcapkit.toolkit.pypcapfile._default_accepts`).
+        checks nothing else of one; any other is asked of the parser itself
+        (see :func:`pcapkit.toolkit.pypcapfile._default_accepts`).
 
     """
     return header[0] == 0x45 or _default_accepts('IPv4', bytes(header))
@@ -1023,11 +1022,10 @@ def _tcp_carrier(packet: 'Packet') -> 'Optional[_Segment]':
         (:func:`pcapkit.toolkit.pcap.tcp_segment`), as `DPKT`_ decodes one, and
         an IPv6 packet's extension headers do not come between, as it dissects
         them itself. Behind a header that is a layer of its own, it does not, so
-        the parser's answer stands there, as for the pypcapfile engine. The TCP
-        endpoints are the innermost IP layer's addresses, as for the default
-        engine; the outer ones are the tunnel's (:issue:`1581`). Each flag is
-        read from the flags octet alone, as the default engine reads it
-        (:issue:`1618`).
+        the parser's answer stands there. The TCP endpoints are the innermost IP
+        layer's addresses, as for the default engine; the outer ones are the
+        tunnel's (:issue:`1581`). Each flag is read from the flags octet alone,
+        as the default engine reads it (:issue:`1618`).
 
     .. _DPKT: https://dpkt.readthedocs.io