fix(toolkit): pypcapfile reads IP over loopback and raw-IP link types, up to the frame layer bound (#1609, #1667)

docs/source/pcapkit/foundation/engines/3rdparty.rst

@@ -399,7 +399,9 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
    TCP and UDP and nothing else. IPv6 reassembly is therefore unavailable and is
    disabled -- with an :class:`~pcapkit.utilities.warnings.AttributeWarning` --
    when requested; IPv4 and TCP reassembly and TCP flow tracing remain
-   available. PCAP-NG is rejected with a
+   available. The IP packet of a BSD loopback or raw-IP frame, which it does
+   not decode, is read by :mod:`pcapkit.toolkit.pypcapfile` instead
+   (:issue:`1667`). PCAP-NG is rejected with a
    :exc:`~pcapkit.utilities.exceptions.FormatError`. `PyPCAPFile`_ 0.12.0 drops
    every record of a big-endian savefile, every record from a zero-length one
    on, and a truncated final record, so the engine reads the records itself,

docs/source/pcapkit/protocols/protocol.rst

@@ -116,9 +116,9 @@ there back out, as they do one the TCP parser rejected (:issue:`1518`). A
 protocol constructed from its own octets is not counted itself, so the count
 starts at the first layer it dissects. One chain of extension headers is also bounded on
 its own, by :data:`~pcapkit.protocols.internet.internet.EXTENSION_HEADER_LIMIT`.
-The ``pyshark`` and ``pypcapfile`` engines dissect with their own parsers,
-which do not apply this bound (:issue:`1609`). The ``dpkt`` and ``scapy``
-toolkits count the layers they walk as the default engine does, and stop at it.
+The ``pyshark`` engine dissects with its own parser, which does not apply this
+bound (:issue:`1609`). The ``dpkt``, ``scapy`` and ``pypcapfile`` toolkits
+count the layers they walk as the default engine does, and stop at it.
 
 .. autodata:: pcapkit.protocols.protocol.FRAME_LAYER_LIMIT
 

docs/source/pcapkit/toolkit/3rdparty.rst

@@ -193,6 +193,17 @@ cannot be used for it.
    :func:`~pcapkit.toolkit.pypcapfile.tcp_traceflow`, with an
    :class:`~pcapkit.utilities.warnings.AttributeWarning` once per capture.
 
+   Nor does it decode a link layer but Ethernet. The adapters read the IP packet
+   of a BSD loopback (``NULL``, ``LOOP``) or raw-IP (``RAW``, ``IPV4``,
+   ``IPV6``) frame themselves, as the default engine dissects it (see
+   :data:`~pcapkit.toolkit.pypcapfile.READ_LINKTYPES`, :issue:`1667`). Every
+   other link type is left unread, as the default engine leaves it, and the
+   engine warns of it with an :class:`~pcapkit.utilities.warnings.AttributeWarning`.
+   Nor do the adapters read deeper than the default engine dissects: each VLAN
+   tag, IP header, AH and IPv6 extension header is one of the
+   :data:`~pcapkit.protocols.protocol.FRAME_LAYER_LIMIT` layers of a frame, and
+   nothing past the last of them is read (:issue:`1609`).
+
 .. autofunction:: pcapkit.toolkit.pypcapfile.ipv4_reassembly
 
 .. autofunction:: pcapkit.toolkit.pypcapfile.ipv6_reassembly
@@ -215,6 +226,9 @@ Auxiliary Functions
 Internal Definitions
 --------------------
 
+.. autodata:: pcapkit.toolkit.pypcapfile.READ_LINKTYPES
+   :no-value:
+
 .. autodata:: pcapkit.toolkit.pypcapfile.TCP_MIN_HEADER_LEN
 
 .. autodata:: pcapkit.toolkit.pypcapfile.IPV4_FLAG_DF

pcapkit/foundation/engines/pypcapfile.py

@@ -47,10 +47,15 @@ class PyPCAPFile(EngineBase['PCAPFilePacket']):
       objects, as `PyPCAPFile`_ 0.12.0 drops records the ``default`` engine
       reads -- see :meth:`_read_records`,
     * disables IPv6 reassembly, warning as it does so, while leaving IPv4 and
-      TCP reassembly and TCP flow tracing in place, and
+      TCP reassembly and TCP flow tracing in place,
     * leaves TCP over IPv6 out of TCP reassembly and flow tracing, with one
       :exc:`~pcapkit.utilities.warnings.AttributeWarning` per capture -- see
-      :mod:`pcapkit.toolkit.pypcapfile`.
+      :mod:`pcapkit.toolkit.pypcapfile` -- and
+    * leaves the frames of a link type `PyPCAPFile`_ has no decoder for
+      undecoded, from which :mod:`pcapkit.toolkit.pypcapfile` reads the IP
+      packet of the BSD loopback and raw-IP ones itself, as the ``default``
+      engine dissects it (see
+      :data:`~pcapkit.toolkit.pypcapfile.READ_LINKTYPES`).
 
     The engine stops decoding at the network layer rather than descending into
     the transport layer. `PyPCAPFile`_ decoders *replace* the payload bytes of
@@ -184,8 +189,10 @@ def run(self) -> 'None':
                   support such operations.
                 * if IPv6 reassembly is enabled, as :mod:`pcapfile` has no IPv6
                   decoder.
-                * if :mod:`pcapfile` has no decoder for the savefile's link layer
-                  type, in which case frames are left undecoded.
+                * if the savefile's link layer type is left unread: neither
+                  :mod:`pcapfile` has a decoder for it nor
+                  :mod:`pcapkit.toolkit.pypcapfile` reads it (see
+                  :meth:`_get_decoder`).
 
         Raises:
             FormatError: If the file format is not supported, i.e., not a PCAP
@@ -302,20 +309,27 @@ def _get_decoder(self, linktype: 'int') -> 'Optional[Callable[..., Any]]':
             The decoder class, or :data:`None` when :mod:`pcapfile` has none.
 
         Warns:
-            AttributeWarning: If no decoder is available, as frames will then be
-                left hexlified -- see :meth:`run` -- and no reassembly or flow
-                tracing is possible.
+            AttributeWarning: If no decoder is available and the link layer type
+                is none of the :data:`~pcapkit.toolkit.pypcapfile.READ_LINKTYPES`,
+                as frames will then be left hexlified -- see :meth:`run` -- and
+                no reassembly or flow tracing is possible. The frames of those
+                link types are left undecoded too, but reassembly and flow
+                tracing read the IP packet in them, as the ``default`` engine
+                does, so nothing is lost there to warn of.
 
         """
+        from pcapkit.toolkit.pypcapfile import READ_LINKTYPES  # isort:skip
+
         try:
             decoder = self._expkg.linklayer.clookup(linktype)
         except IndexError:  # malformed entry in ``pcapfile.linklayer.__LL_TYPES__``
             decoder = None
 
         if not callable(decoder):
-            warn(f'unrecognised link layer protocol: {self._dlink!r}; frames will be left '
-                 'undecoded and all analysis functions ignored', AttributeWarning,
-                 stacklevel=stacklevel())
+            if linktype not in READ_LINKTYPES:
+                warn(f'unrecognised link layer protocol: {self._dlink!r}; frames will be left '
+                     'undecoded and all analysis functions ignored', AttributeWarning,
+                     stacklevel=stacklevel())
             return None
         return decoder