docs(foundation): fifth whole-repo pass over foundation, toolkit, dumpkit, interface and utilities (#1103)

docs/source/pcapkit/foundation/engines/3rdparty.rst

@@ -115,7 +115,8 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
    3.10 and 3.11 return a loop silently, 3.12 returns one with a
    :exc:`DeprecationWarning`, and **3.14 raises**
    ``RuntimeError: There is no current event loop in thread 'MainThread'``. Python
-   3.13 works, being on the deprecated-but-functional side of that change.
+   3.13 was not measured; it is expected to work, being on the
+   deprecated-but-functional side of that change.
 
    **Wireshark's** :program:`tshark` **binary.** |pyshark|_ shells out to it and
    parses nothing itself. It need not be on :envvar:`PATH`: |pyshark|_ consults
@@ -464,8 +465,9 @@ and ``import pcap`` resolves to whichever the import system finds first.
 they actually got rather than assume, and this module is the one place that
 answers it -- deliberately shared, since the engines must agree and a copy of
 the detection in each would be a chance to disagree. It is *only* detection,
-and it imports nothing from :mod:`pcapkit`, so it cannot introduce an import
-cycle.
+and it imports nothing from :mod:`pcapkit` but :mod:`pcapkit.corekit.infoclass`,
+which imports nothing from :mod:`pcapkit.foundation`, so it cannot introduce an
+import cycle.
 
 .. autodata:: pcapkit.foundation.engines._pcap_backend.PYPCAP
 

docs/source/pcapkit/foundation/engines/index.rst

@@ -171,9 +171,10 @@ A constraint that an import cannot reveal is enforced by overriding
 one warning naming the actual cause and a fall back to the built-in parser.
 :class:`~pcapkit.foundation.engines.pyshark.PyShark` checks its Python ceiling and for
 ``tshark``, :class:`~pcapkit.foundation.engines.pypcapfile.PyPCAPFile` its Python ceiling,
-:class:`~pcapkit.foundation.engines.pcap_ct.PCAP_CT` for a system ``libpcap``, and
-:class:`~pcapkit.foundation.engines.pypcap.PyPCAP` and
-:class:`~pcapkit.foundation.engines.pcap_ct.PCAP_CT` which distribution owns |pcap|_.
+and :class:`~pcapkit.foundation.engines.pypcap.PyPCAP` and
+:class:`~pcapkit.foundation.engines.pcap_ct.PCAP_CT` which distribution owns |pcap|_ and
+whether it loads at all -- for :class:`~pcapkit.foundation.engines.pcap_ct.PCAP_CT`, whether
+a system ``libpcap`` is present (:issue:`2132`).
 :class:`~pcapkit.foundation.engines.pypcap.PyPCAP` deliberately has no Python ceiling
 check, even though upstream cannot be installed on 3.12 or newer: what matters is which
 distribution is present, and a version check would refuse a working build.

docs/source/pcapkit/foundation/reassembly/ip/index.rst

@@ -96,6 +96,10 @@ reported as a separate incomplete datagram; this is intended (:issue:`1507`).
 
 Departures from the procedure above:
 
+* The timer is not set from ``TLB`` and ``TTL``. A buffer is abandoned a fixed
+  timeout after its first-arriving fragment, on the capture's clock -- 60
+  seconds by default, per :rfc:`1122#section-3.3.2` for IPv4 and
+  :rfc:`8200#section-4.5` for IPv6.
 * Only the final fragment's partial last ``RCVBT`` block is set. A non-final
   fragment carries a multiple of 8 octets, so one ending mid-block was cut or
   is malformed, and setting its last block would complete the datagram with

docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst

@@ -96,8 +96,8 @@ Terminology
        .. note::
 
           ``packet`` is analysed on the first read, not when the datagram is
-          submitted. A datagram is submitted for *every* frame -- an unfragmented
-          one included, since nothing upstream filters it out -- and the analysis
+          submitted. A datagram is submitted for *every* frame whose **DF** flag
+          is clear -- an unfragmented one included -- and the analysis
           is a second full parse of the payload, so running it eagerly would
           charge every caller for a result most never read. Reading the attribute, or any
           mapping view of it (``datagram['packet']``, ``to_dict()``, ``items()``,

docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst

@@ -140,11 +140,9 @@ Terminology
        .. note::
 
           ``packet`` is analysed on the first read, not when the datagram is
-          submitted. A datagram is submitted for *every* frame -- an unfragmented
-          one included, since nothing upstream filters it out -- and the analysis
-          is a second full parse of the payload, so running it eagerly would
-          charge every caller for a result most never read. Reading the attribute, or any
-          mapping view of it (``datagram['packet']``, ``to_dict()``, ``items()``,
+          submitted. The analysis is a second full parse of the payload, so
+          running it eagerly would charge every caller for a result most never
+          read. Reading the attribute, or any mapping view of it (``datagram['packet']``, ``to_dict()``, ``items()``,
           ``repr()``), runs it and keeps the result; see
           :class:`~pcapkit.foundation.reassembly.data.data.Deferred`.
 

docs/source/pcapkit/foundation/reassembly/tcp.rst

@@ -83,6 +83,11 @@ Algorithm
             [from octet fragment.first to octet fragment.last];
          HDL <- [one hole from PSN + fragment.len to infinity,
                  i.e. HOLE_OPEN_BOUND];
+         IF (SYN is false) {
+            /* the start of the stream is unknown: this hole is the one
+               before the capture */
+            insert one hole from -HOLE_OPEN_BOUND to PSN - 1 into HDL;
+         }
       } ELSE {
          put data from fragment into data buffer with BUFID
             [from octet fragment.first to octet fragment.last];
@@ -94,7 +99,13 @@ Algorithm
          }
       }
 
-      IF (FIN is true or RST is true) {
+      /* a FIN fixes the end of the stream; data sent before it may still
+         arrive after it */
+      IF (FIN is true) {
+         cut every hole in HDL back to end below the FIN;
+      }
+
+      IF (RST is true or HDL holds no hole but the one before the capture) {
          submit datagram to next step;
          free all reassembly resources for this BUFID;
          BREAK.
@@ -116,13 +127,14 @@ Algorithm
             new_hole.last <- fragment.first - 1;
          }
 
-         IF (fragment.last < hole.last AND FIN is false AND RST is false) {
+         IF (fragment.last < hole.last) {
             create new entry "new_hole" in HDL;
             new_hole.first <- fragment.last + 1;
             new_hole.last <- hole.last;
          }
 
-         BREAK.
+         /* one segment may span several holes */
+         CONTINUE.
       } give up until (no entry from HDL)
    }
 
@@ -141,16 +153,18 @@ here is the process:
    new hole descriptor ``new_hole`` with ``new_hole.first`` equal to
    ``hole.first``, and ``new_hole.last`` equal to ``fragment.first``
    minus one (``-1``).
-6. If ``fragment.last`` is less than ``hole.last`` and neither ``FIN``
-   nor ``RST`` is set -- TCP has no *more fragments* flag, so the
-   termination flags take its place -- then create a new hole
+6. If ``fragment.last`` is less than ``hole.last``, then create a new hole
    descriptor ``new_hole``, with ``new_hole.first`` equal to
    ``fragment.last`` plus one (``+1``) and ``new_hole.last`` equal to
    ``hole.last``.
 7. Go to step one.
-8. If the hole descriptor list is now empty, the datagram is now
-   complete. Pass it on to the higher level protocol processor
-   for further handling. Otherwise, return.
+8. If the hole descriptor list holds no hole but the one below a capture
+   that missed the handshake, the stream is complete. Pass it on to the
+   higher level protocol processor for further handling. Otherwise, return.
+
+TCP has no *more fragments* flag, so the hole above the last octet received
+stays open until a FIN cuts it back to end at the sequence number the FIN
+occupies. A RST submits the stream as it stands, holes and all.
 
 Terminology
 ===========

pcapkit/dumpkit/common.py

@@ -421,7 +421,7 @@ def excluded(info: 'Info | MultiDict[Any, Any]') -> 'Collection[Any]':
             The :attr:`~pcapkit.corekit.infoclass.Info.__excluded__` of a field
             model, or of the field model an export was made from -- metadata
             such as ``__next_type__``, which only a rebuild reads
-            (:issue:`2003`). None for an option list, whose keys are its
+            (:issue:`2003`). Empty for an option list, whose keys are its
             options.
 
         """

pcapkit/dumpkit/null.py

@@ -49,7 +49,7 @@ def __call__(self, value: 'dict[str, Any]', name: 'Optional[str]' = None) -> 'No
             name: name of current content block
 
         Returns:
-            The dumper class itself (to support chain calling).
+            The dumper itself (to support chain calling).
 
         """
         return self

pcapkit/dumpkit/pcap.py

@@ -148,7 +148,7 @@ def __call__(self, value: 'Data_Frame', name: 'Optional[str]' = None) -> 'PCAPIO
             name: name of current content block
 
         Returns:
-            The dumper class itself (to support chain calling).
+            The dumper itself (to support chain calling).
 
         """
         with open(self._file, 'ab') as file:

pcapkit/foundation/engines/_pcap_backend.py

@@ -24,7 +24,8 @@
 the one place that answers that, deliberately shared: the engines must agree on
 the answer, and a copy of the detection in each would be a chance to disagree.
 It is *only* detection -- nothing here is engine behaviour -- and it imports
-nothing from :mod:`pcapkit`, so it cannot introduce an import cycle.
+nothing from :mod:`pcapkit` but :mod:`pcapkit.corekit.infoclass`, which imports
+nothing from :mod:`pcapkit.foundation`, so it cannot introduce an import cycle.
 
 """
 import contextvars

pcapkit/foundation/engines/engine.py

@@ -290,15 +290,15 @@ def __init_subclass__(cls, /, engine: 'Optional[str]' = None, *args: 'Any', **kw
 
         Note:
             The keyword is ``engine`` rather than ``name`` because ``name``
-            cannot be passed as a class keyword at all on Python 3.10:
+            cannot be passed as a class keyword at all below Python 3.11:
             ``abc.ABCMeta.__new__`` takes ``mcls``, ``name``, ``bases`` and
             ``namespace`` as positional-*or-keyword* parameters
-            before 3.11, so a class keyword by any of those names collides
+            there, so a class keyword by any of those names collides
             with one of them and the class statement raises :exc:`TypeError` from
             the metaclass before this method is reached. ``engine`` is outside
             that set, so the documented registration path works on every
-            supported version. Measured on 3.10.21, 3.11.15 and 3.14.7; those
-            names are the whole of the ``abc.ABCMeta.__new__`` collision
+            supported version. Measured on 3.8.20, 3.9.25, 3.10.21, 3.11.15 and
+            3.14.7; those names are the whole of the ``abc.ABCMeta.__new__`` collision
             surface. Separately, and for an unrelated reason that holds on every
             version, ``metaclass`` is not usable as a class keyword either: a
             ``class`` statement consumes it to choose the metaclass, so it never
@@ -316,7 +316,7 @@ def __init_subclass__(cls, /, engine: 'Optional[str]' = None, *args: 'Any', **kw
         # ``class MyEngine(Engine, engnie='x')`` skip registration altogether --
         # no exception, no warning. So reject it.
         #
-        # One typo this cannot catch is ``name=``, and only on Python 3.10: it is
+        # One typo this cannot catch is ``name=``, and only below Python 3.11: it is
         # one of the names that collide with ``ABCMeta.__new__``, so it fails
         # in the metaclass before reaching here. It is still loud, just as a
         # ``TypeError`` rather than an ``UnsupportedCall``.

pcapkit/foundation/engines/pcap_ct.py

@@ -251,10 +251,11 @@ def __init__(self, extractor: 'Extractor') -> 'None':
         Raises:
             UnsupportedCall: If the environment cannot support this engine, for
                 either of the reasons :meth:`unsupported_reason` describes. That
-                hook is only consulted by :meth:`Extractor.run
-                <pcapkit.foundation.extraction.Extractor.run>`, which degrades to
-                the default engine with a warning; constructing the engine
-                directly bypasses it, so this is the backstop for that path.
+                hook is only consulted by
+                :class:`~pcapkit.foundation.extraction.Extractor`'s engine
+                selection, which degrades to the default engine with a warning;
+                constructing the engine directly bypasses it, so this is the
+                backstop for that path.
 
         """
         # NOTE: the probe comes *before* ``import pcap``, and the order is
@@ -464,8 +465,8 @@ def close(self) -> 'None':
             a second call, where upstream ``pypcap`` 1.3.0 segfaults on one. The
             :attr:`_closed <PCAP_CT._closed>` flag is kept regardless: it is not
             this engine's business to depend on a pre-release's undocumented
-            forgiveness, and the flag also covers the case of an engine that was
-            never opened.
+            forgiveness. An engine that was never opened has no
+            :attr:`_handle <PCAP_CT._handle>` to close, and is skipped too.
 
         """
         if self._closed or self._handle is None:

pcapkit/foundation/engines/pcapng.py

@@ -344,7 +344,7 @@ def _block_name(self, kind: 'str') -> 'str':
         with every Section Header Block, so numbering a block by their length gives the first IDB of
         every section the same name. The output dumpers key by name, so a repeated name overwrites
         the earlier block (#1256). The number is therefore counted across the whole file instead;
-        for a single-section file it is the same number as before.
+        for a single-section file the two counts agree.
 
         Args:
             kind: The kind of the block, e.g. ``'Interface Description'``.

pcapkit/foundation/engines/pypcap.py

@@ -208,7 +208,7 @@ def __init__(self, extractor: 'Extractor') -> 'None':
                 than upstream `PyPCAP`_, or is installed but cannot load. These
                 are the conditions :meth:`unsupported_reason` reports, kept here as
                 well because that hook is only consulted by
-                :meth:`Extractor.run <pcapkit.foundation.extraction.Extractor.run>`;
+                :class:`~pcapkit.foundation.extraction.Extractor`'s engine selection;
                 constructing the engine directly bypasses it, and running
                 `pcap-ct`_ under the :class:`~pcapkit.foundation.engines.pypcap.PyPCAP` name would
                 misreport what did the work. Via :class:`~pcapkit.foundation.extraction.Extractor`

pcapkit/foundation/engines/pyshark.py

@@ -64,10 +64,10 @@ def _is_packet(packet: 'PySharkPacket') -> 'bool':
     records only.
 
     The marker is ``frame.encap_type`` on the record's :func:`_frame_layer`.
-    Wireshark adds it for ``REC_TYPE_PACKET`` records only:
-    :file:`epan/dissectors/packet-frame.c` lines 1001-1002 in 4.2.2, and 828-890
-    in 4.6.9. The summary line is not used, because it differs between those
-    versions. A record with no ``frame`` layer is not a packet either.
+    Wireshark adds it for ``REC_TYPE_PACKET`` records only, in ``dissect_frame``
+    of :file:`epan/dissectors/packet-frame.c`, in 4.2.2 and 4.6.9 alike. The
+    summary line is not used, because it differs between those versions. A
+    record with no ``frame`` layer is not a packet either.
 
     """
     frame = _frame_layer(packet)
@@ -119,8 +119,8 @@ def unsupported_reason(cls) -> 'Optional[str]':
         error rather than degrading to the default engine with a warning.
 
         **The interpreter.** |pyshark|_ 0.6 does
-        ``asyncio.get_event_loop_policy().get_event_loop()`` at
-        :file:`pyshark/capture/capture.py:183`, in a fresh interpreter with no
+        ``asyncio.get_event_loop_policy().get_event_loop()`` in
+        ``Capture._setup_eventloop``, in a fresh interpreter with no
         running loop. Measured: 3.10 and 3.11 return a loop
         silently, 3.12 returns one with a :exc:`DeprecationWarning`, and 3.14
         raises ``RuntimeError: There is no current event loop in thread

pcapkit/foundation/extraction.py

@@ -515,7 +515,9 @@ class it names, restores the shipped entry rather than storing
             ext: file extension
 
         Raises:
-            RegistryError: If ``dumper`` is not a class, or not a :class:`~pcapkit.dumpkit.common.Dumper` subclass.
+            RegistryError: If ``dumper`` is not a class, or not a
+                :class:`dictdumper.dumper.Dumper` subclass, such as
+                :class:`~pcapkit.dumpkit.common.Dumper`.
 
         Warns:
             RegistryWarning: If a different dumper is already registered for
@@ -1670,10 +1672,11 @@ def __del__(self) -> 'None':
             :attr:`~object.__dict__` rather than attribute access. That matters
             for more than tidiness: ``_flag_s`` is assigned early in
             :meth:`__init__` and ``_ifile`` only much later, so a constructor
-            that fails in between -- an unreadable path, an unknown format --
-            leaves an instance whose flag says "mine" and which has no stream at
-            all. The close itself is guarded too, that being what a close during
-            interpreter shutdown can fail at.
+            that fails in between -- an unreadable path, or a negative
+            ``reasm_timeout`` with ``reassembly=True`` and ``ip``, ``ipv4``,
+            ``ipv6`` or ``tcp`` set -- leaves an instance whose flag says "mine"
+            and which has no stream at all. The close itself is guarded too, that
+            being what a close during interpreter shutdown can fail at.
 
         """
         if not self._owns_input():

pcapkit/foundation/reassembly/data/data.py

@@ -71,8 +71,9 @@ class Completion(EnumLookup, StrEnum):
     COMPLETE = auto()
 
     #: Fragments were still outstanding when the buffer was flushed -- at the end
-    #: of the capture, or when the session was torn down (a TCP FIN/RST, or an
-    #: IPv4 datagram whose identifier was reused by an unfragmented packet).
+    #: of the capture, or when the session was torn down (a TCP RST, a TCP SYN on
+    #: a 4-tuple already buffered, or an IPv4 datagram whose identifier was
+    #: reused by an unfragmented packet).
     #: The missing octets may simply not have been captured.
     PARTIAL = auto()
 
@@ -111,9 +112,9 @@ class Deferred:
     :class:`~pcapkit.foundation.reassembly.data.data.ReassemblyData` rather than in
     either protocol's data module.
 
-    IP reassembly is the case that made it necessary. It submits a datagram for
-    *every* frame -- not only the fragmented ones, since a frame that is not
-    fragmented in any sense still reaches
+    IP reassembly is the case that made it necessary. IPv4 reassembly submits a
+    datagram for *every* frame whose **DF** flag is clear -- not only the
+    fragmented ones, since an unfragmented frame still reaches
     :meth:`IP.reassembly <pcapkit.foundation.reassembly.ip.IP.reassembly>` and is
     submitted there as a trivially complete datagram -- so an eager parse would
     re-parse captures holding no fragments at all: :file:`http.pcap` has 1117

pcapkit/foundation/reassembly/ip.py

@@ -511,11 +511,12 @@ def submit(self, buf: 'Buffer[_AT]', *, bufid: 'tuple[_AT, _AT, int, TransType]'
                 index=tuple(index),
                 header=header,
                 payload=payload,
-                # NOTE: ``analyze`` is a second full parse of the payload, and a
-                # datagram is submitted for every frame rather than only for the
-                # fragmented ones, so running it here charges every caller for a
-                # result most of them never read. ``Deferred`` postpones it to the
-                # first read of ``Datagram.packet``.
+                # NOTE: ``analyze`` is a second full parse of the payload, and an
+                # IPv4 datagram is submitted for every frame with **DF** clear
+                # rather than only for the fragmented ones, so running it here
+                # charges every caller for a result most of them never read.
+                # ``Deferred`` postpones it to the first read of
+                # ``Datagram.packet``.
                 packet=Deferred(self.protocol.analyze, bufid[3], payload),
                 conflict=conflict,
             )

pcapkit/foundation/reassembly/ipv4.py

@@ -34,7 +34,7 @@ class IPv4(IP):
             :data:`None` selects :attr:`__timeout__`
 
     Example:
-        >>> from pcapkit.foundation.reassembly import IPv4
+        >>> from pcapkit.foundation.reassembly.ipv4 import IPv4
         # Initialise instance:
         >>> ipv4_reassembly = IPv4()
         # Call reassembly: