diff --git a/.github/workflows/deploy-pages.yml b/.github/workflows/deploy-pages.yml index 5e4c601aa..5918b85d7 100644 --- a/.github/workflows/deploy-pages.yml +++ b/.github/workflows/deploy-pages.yml @@ -57,7 +57,14 @@ jobs: with: python-version: '3.14' - - name: Install and Build 🔧 # This example project is built using npm and outputs the result to the 'build' folder. Replace with the commands required to build your project, or remove this step entirely if your site is pre-built. + # Builds the Sphinx documentation into `docs/build/html`, which the deploy + # step below publishes. `PCAPKIT_SPHINX` is what makes the + # `if SPHINX_TYPE_CHECKING:` TypedDicts visible to autodoc; `PCAPKIT_DEVMODE` + # puts the library logger at DEBUG on stderr and leaves tracebacks untrimmed, + # so an autodoc import failure is legible in the run log. Both are read at + # import time, so they are exported before anything imports `pcapkit` -- + # `docs/source/conf.py` sets `PCAPKIT_SPHINX` itself for the same reason. + - name: Install and Build 🔧 run: | set -x diff --git a/CHANGELOG.md b/CHANGELOG.md index f03570a46..42868f009 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ The largest release since 1.0, and the first recorded here as it happened rather - **Added** -- ESP parsing and construction [[RFC 4303](https://datatracker.ietf.org/doc/html/rfc4303)], with optional payload decryption and ICV verification through `cryptography` (`pip install pypcapkit[crypto]`) (#378). Keys reach the dissector through a new caller-state channel, `pcapkit.corekit.context`, surfaced as the `context=` keyword on `extract()` and `Extractor`, carrying an `esp.SecurityAssociation`. Nothing here raises: with no association the SPI and sequence number are still reported and the ciphertext is left opaque, with `status=NO_SA`, and a failed decryption or ICV check is recorded on the parsed result the same way. Extended Sequence Numbers, TFC padding and anti-replay are not implemented, and an unsupported cipher or MAC is refused with a clear error rather than half-processed. - **Added** -- SCTP as a transport protocol [[RFC 9260](https://datatracker.ietf.org/doc/html/rfc9260)]: all 13 chunk types, 8 chunk parameters and 13 error causes, with the CRC32c both recorded and verifiable -- it covers the SCTP packet alone, with no IP pseudo-header, so it can be checked from the SCTP bytes. Upper layers register on the DATA chunk's Payload Protocol Identifier through `register_sctp`, not on a port (#379). - **Added** -- NGAP over SCTP (3GPP TS 38.413), decoding aligned PER through `pycrate` (`pip install pypcapkit[NGAP]`) (#251, #417). Decoding is generic by ASN.1 shape rather than per-procedure, so all 81 elementary procedures and 438 protocol IEs work and a new 3GPP release needs no code change. Registered as a default on PPID 60 and 66, but only 60 decodes: PPID 66 is an NGAP PDU inside a DTLS record, and there is no DTLS dissector. `pycrate` is deliberately excluded from the `all` extra -- it is LGPL-2.1+ where this package is BSD-3-Clause, and lands some 238 MB to obtain one module. -- **Added** -- the Mobility Header registry, completed (#383, #437). The [RFC 5568](https://datatracker.ietf.org/doc/html/rfc5568) fast-handover messages and options first, then all 24 registered message data types, 70 of the 71 registered options -- with nested sub-option registries for the flow identification, access network identifier, quality-of-service and LMA-controlled MAG parameter families -- and all 4 CGA extensions. Only the CGA Parameters option remains on the generic handler. +- **Added** -- the Mobility Header registry, completed (#383, #437). The [RFC 5568](https://datatracker.ietf.org/doc/html/rfc5568) fast-handover messages and options first, then all 24 registered message data types, all 71 registered options -- with nested sub-option registries for the flow identification, access network identifier, quality-of-service and LMA-controlled MAG parameter families -- and all 4 CGA extensions. The CGA Parameters option was the last to come off the generic handler, which is what let the four CGA extensions round-trip end to end, since it is the only option that can carry one on the wire. - **Added** -- dispatch entries for dissectors that existed but were reachable from no registry (#436): FTP-DATA on TCP 20, HTTP/1 on TCP 8080, HTTP on UDP 8080, `L2TPv2` on UDP 1701 and OSPF at `TransType` 89. `VLAN` became an abstract base with `C_Tag` (802.1Q) and `S_Tag` (802.1ad) as concrete subclasses, so a Q-in-Q frame no longer collapses into one opaque `Raw`; `L2TP` likewise became a base, with `L2TPv2` carrying the [RFC 2661](https://datatracker.ietf.org/doc/html/rfc2661) implementation. - **Added** -- `pcapkit.utilities.logging` as a real interface: `get_logger()` for per-module children, `configure()` to set level, handler, stream, format or propagation at runtime, `reset()` to return to library-neutral, and `ensure_output()`. Seventeen modules now log under their own `__name__`, so a consumer can silence `pcapkit.foundation.registry` while keeping `pcapkit.foundation.extraction` (#384). - **Added** -- `conflict` on the reassembly data models: absolute, inclusive ranges where two fragments claimed the same span with different bytes, which was previously lost silently on both the IP (#482) and TCP (#443, #478) paths. diff --git a/SECURITY.md b/SECURITY.md index 8cfd5b925..cb2c735e1 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -2,18 +2,81 @@ ## Supported Versions -Use this section to tell people about which versions of your project are -currently being supported with security updates. +Security fixes land on the current stable line and on the development line. Older +lines are not patched -- upgrading is the fix. -| Version | Supported | -| ------- | ------------------ | -| 1.0.x | :white_check_mark: | -| 0.x.x | :x: | +| Version | Supported | Notes | +| --------------- | ------------------ | --------------------------------------- | +| 1.5.x | :white_check_mark: | development line, currently pre-release | +| 1.4.x | :white_check_mark: | current stable | +| 1.3.x and older | :x: | no longer patched | +| 0.x | :x: | no longer patched | + +Supported interpreters are those the test matrix actually covers, currently +CPython 3.10 through 3.14. `pyproject.toml` declares `requires-python = ">=3.6, <4"` +through the `bpc-*` source conversion, but 3.8 and 3.9 are best-effort and below +3.8 is intent rather than something that is exercised -- see the note above +`requires-python` in `pyproject.toml`. A report that only reproduces on an +interpreter outside the tested range is still welcome; it may be fixed by raising +the floor rather than by patching. ## Reporting a Vulnerability -Use this section to tell people how to report a vulnerability. +**Please do not open a public issue for a security problem.** Public issues are +the right place for ordinary bugs, and the wrong place for anything exploitable. + +Use GitHub's private vulnerability reporting, which is enabled on this repository: + +- + +That opens a private advisory visible only to the maintainer, lets patches be +prepared before anything is public, and issues a CVE if one is warranted. If you +cannot use it, email the project contact listed in +[`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md) instead and say that the report is a +security issue. + +Please include whatever you have: + +- the version of `pypcapkit` and the Python interpreter, +- what an attacker gains, and what they need in order to get it, +- a capture file or a short script that reproduces it -- **strip anything + sensitive from a capture first**, since a pcap taken from a real network tends + to carry more than the bug. + +## What to expect + +This is a single-maintainer project, so response is best-effort rather than +contractual: + +- an acknowledgement that the report arrived, normally within a week, +- an assessment of whether it is accepted, and if so a rough severity, once it has + been reproduced, +- a fix on the supported lines above, released as a patch version, with credit in + the changelog unless you would rather not be named. + +If a report is declined, you will be told why -- most often that it describes +expected behaviour for a packet parser. Note the threat model below before +deciding whether you have a finding. + +## Threat model + +`pcapkit` parses attacker-controlled bytes by design: a capture file, or a live +stream off the wire, is untrusted input. Vulnerabilities in that parsing are in +scope. In particular: + +- memory exhaustion, unbounded allocation, or a hang on a malformed packet, +- an uncaught exception escaping the public API where a `pcapkit` exception was + promised -- parse errors are supposed to arrive as + `pcapkit.utilities.exceptions` types, not as an arbitrary traceback, +- anything that executes code, writes outside the requested output path, or reads + an unrelated file as a result of the *contents* of a capture. + +Out of scope: -Tell them where to go, how often they can expect to get an update on a -reported vulnerability, what to expect if the vulnerability is accepted or -declined, etc. +- the optional third-party extraction engines' own defects (`dpkt`, `scapy`, + `pyshark`, `pypcap`, `pcap-ct`, `pypcapfile`) -- report those upstream, though + do say so here if `pcapkit` passes them something it should not, +- a capture that parses to the wrong values without any security consequence; + that is an ordinary bug and belongs in a public issue, +- needing elevated privileges to capture live traffic, which is the operating + system's requirement rather than this project's. diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index f019d022b..350a46fa4 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -52,10 +52,12 @@ pull requests between #326 and #509. package is BSD-3-Clause, and lands some 238 MB to obtain one module. * **Added** -- the Mobility Header registry, completed (#383, #437). The :rfc:`5568` fast-handover messages and options first, then all 24 registered - message data types, 70 of the 71 registered options -- with nested sub-option + message data types, all 71 registered options -- with nested sub-option registries for the flow identification, access network identifier, quality-of-service and LMA-controlled MAG parameter families -- and all 4 CGA - extensions. Only the CGA Parameters option remains on the generic handler. + extensions. The CGA Parameters option was the last to come off the generic + handler, which is what let the four CGA extensions round-trip end to end, + since it is the only option that can carry one on the wire. * **Added** -- dispatch entries for dissectors that existed but were reachable from no registry (#436): FTP-DATA on TCP 20, HTTP/1 on TCP 8080, HTTP on UDP 8080, ``L2TPv2`` on UDP 1701 and OSPF at ``TransType`` 89. ``VLAN`` became an diff --git a/docs/source/conf.py b/docs/source/conf.py index ac018b71c..aed5ba3ae 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -9,8 +9,6 @@ import logging import os import pkgutil -import sys -import typing from typing import TYPE_CHECKING # NB: a private module of ``sphinx-autodoc-typehints``, used deliberately -- see @@ -30,7 +28,7 @@ import pcapkit if TYPE_CHECKING: - from typing import Any, Dict, List, Optional + from typing import Any, Dict, Optional from sphinx.application import Sphinx logger = logging.getLogger('pcapkit-sphinx') @@ -335,56 +333,32 @@ def claim_attribute_signature(app: 'Sphinx', what: str, name: str, # pylint: di return None -def remove_module_docstring(app: 'Sphinx', what: str, name: str, # pylint: disable=unused-argument - obj: 'Any', options: 'Dict[str, Any]', lines: 'List[str]') -> None: # pylint: disable=unused-argument - if what == "module" and "pcapkit" in name: - module = sys.modules.get(name) - if module is not None: - logger.info('reloading module: %s', name) - typing.TYPE_CHECKING = True - importlib.reload(module) - logger.info('reloaded module: %s', name) - #lines.clear() - - -def process_docstring(app: 'Sphinx', what: str, name: str, # pylint: disable=unused-argument - obj: 'Any', options: 'Dict[str, Any]', lines: 'List[str]') -> None: # pylint: disable=unused-argument - if what == "module" and "pcapkit" in name: - module = importlib.import_module(name) - typing.TYPE_CHECKING = True - importlib.reload(module) - - -def process_fields(app: 'Sphinx', what: str, name: str, obj: 'Any', options: 'Dict[str, Any]', lines: 'List[str]') -> 'None': - if what == 'attribute' \ - and name.startswith('pcapkit.protocols.schema') \ - and type(obj).__module__.startswith('pcapkit.corekit.fields'): - - print(name, obj) - #lines.append(':param packet: Packet data.',) - - -def source_read(app: 'Sphinx', docname: str, source_text: str) -> 'None': # pylint: disable=unused-argument - print(docname, source_text) +# NB: four handlers used to sit here -- ``remove_module_docstring``, +# ``process_docstring``, ``process_fields`` and ``source_read`` -- each with its +# ``app.connect`` call commented out in ``setup`` below, so none of them had run +# for as long as they had been in the file. They are deleted rather than restored, +# because restoring them would either duplicate work now done properly or make the +# build unusable: +# +# ``remove_module_docstring`` and ``process_docstring`` were two attempts at the +# same thing -- flipping :data:`typing.TYPE_CHECKING` and reloading each module +# so autodoc could see the ``if TYPE_CHECKING:`` names. ``bind_type_checking_names`` +# above now solves that, per module and without mutating a global mid-build. +# +# ``process_fields`` only ever printed; the ``lines.append`` that was its point +# was itself commented out. +# +# ``source_read`` printed every document's entire source text, which on this +# project is several thousand pages of build log. +# +# If the ``TYPE_CHECKING`` behaviour is ever wanted again, ``bind_type_checking_names`` +# is where it belongs, not a reload hook. def setup(app: 'Sphinx') -> None: - #app.connect('autodoc-process-docstring', process_docstring, 0) - #app.connect("autodoc-process-docstring", remove_module_docstring) app.connect('builder-inited', bind_type_checking_names) app.connect('autodoc-skip-member', maybe_skip_member) # NB: below ``sphinx_autodoc_typehints``, which connects at the default 500 and # would otherwise win the tie on registration order -- conf.py's ``setup`` runs # after the extensions in ``extensions`` have been set up. app.connect('autodoc-process-signature', claim_attribute_signature, priority=400) - #app.connect('source-read', source_read) - #app.connect('autodoc-process-docstring', process_fields) - - # typing.TYPE_CHECKING = True - # for name, module in sys.modules.copy().items(): - # if 'pcapkit' not in name: - # continue - - # logger.info('reloading module: %s', name) - # importlib.reload(module) - # logger.info('reloaded module: %s', name) diff --git a/docs/source/ext.rst b/docs/source/ext.rst index c8afbe06b..5ac47fce2 100644 --- a/docs/source/ext.rst +++ b/docs/source/ext.rst @@ -83,6 +83,8 @@ The following table shows all available protocol classes in :mod:`pcapkit`: | | HTTP Family | :class:`pcapkit.protocols.application.httpv1.HTTP` | + + +-----------------------+-------------------------------------------------------------+ | | | :class:`pcapkit.protocols.application.httpv2.HTTP` | ++ +----------------+-----------------------+-------------------------------------------------------------+ +| | :class:`pcapkit.protocols.application.ngap.NGAP` | +------------------------------------------------------------------+----------------+-----------------------+-------------------------------------------------------------+ | | | :class:`pcapkit.protocols.misc.pcap.header.Header` | + + PCAP Format +-----------------------+-------------------------------------------------------------+ diff --git a/docs/source/pcapkit/corekit/context.rst b/docs/source/pcapkit/corekit/context.rst index 6b7fad0b6..89c0fe9f1 100644 --- a/docs/source/pcapkit/corekit/context.rst +++ b/docs/source/pcapkit/corekit/context.rst @@ -20,13 +20,13 @@ Rather than adding protocol specific keyword arguments to :class:`~pcapkit.foundation.extraction.Extractor`, such information is passed as a :class:`~pcapkit.corekit.context.ContextRegistry` -- a mapping of protocol index ID (c.f. :meth:`Protocol.id -`) to a +`) to a :class:`~pcapkit.corekit.context.ProtocolContext` instance. The registry is handed to :class:`~pcapkit.foundation.extraction.Extractor` once, and is then propagated down the protocol stack by -:meth:`Protocol._import_next_layer `, +:meth:`Protocol._import_next_layer `, so that a protocol nested arbitrarily deep can reach it through -:meth:`Protocol._get_context `. +:meth:`Protocol._get_context `. Decoding an ESP tunnel end to end: diff --git a/docs/source/pcapkit/foundation/extraction.rst b/docs/source/pcapkit/foundation/extraction.rst index 07056dece..6944259e0 100644 --- a/docs/source/pcapkit/foundation/extraction.rst +++ b/docs/source/pcapkit/foundation/extraction.rst @@ -62,6 +62,7 @@ extracts parametres from a PCAP file. .. autoattribute:: _flag_d .. autoattribute:: _flag_e .. autoattribute:: _flag_q + .. autoattribute:: _flag_r .. autoattribute:: _flag_t .. autoattribute:: _flag_v .. autoattribute:: _flag_n @@ -79,6 +80,7 @@ extracts parametres from a PCAP file. .. autoattribute:: _exlyr .. autoattribute:: _exptl + .. autoattribute:: _exctx .. automethod:: __iter__ .. automethod:: __next__ diff --git a/docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst b/docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst index 4bbe6266f..22407f409 100644 --- a/docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst +++ b/docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst @@ -24,7 +24,7 @@ Terminology Data structure for **IPv4 datagram reassembly** (:meth:`IPv4.reassembly `) is as following, with ``ipv4`` the protocol instance - (``frame['IPv4']``) and ``ipv4_info`` its :attr:`~pcapkit.protocols.protocol.ProtocolBase.info` + (``frame['IPv4']``) and ``ipv4_info`` its :attr:`~pcapkit.protocols.protocol.Protocol.info` -- the header fields come off the latter, the raw octets off the former: .. code-block:: python diff --git a/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst b/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst index 1b9bb37b5..5146ff1b2 100644 --- a/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst +++ b/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst @@ -24,7 +24,7 @@ Terminology Data structure for **IPv6 datagram reassembly** (:meth:`IPv6.reassembly `) is as following, with ``ipv6_info`` the IPv6 - :attr:`~pcapkit.protocols.protocol.ProtocolBase.info` and + :attr:`~pcapkit.protocols.protocol.Protocol.info` and ``ipv6_frag_info`` the Fragment header's: .. code-block:: python diff --git a/docs/source/pcapkit/protocols/application/application.rst b/docs/source/pcapkit/protocols/application/application.rst index 363f3023d..9cfb822a6 100644 --- a/docs/source/pcapkit/protocols/application/application.rst +++ b/docs/source/pcapkit/protocols/application/application.rst @@ -6,8 +6,8 @@ Base Protocol :mod:`pcapkit.protocols.application.application` contains only :class:`~pcapkit.protocols.application.application.Application`, which is a base class for application layer protocols, eg. -:class:`HTTP/1.* `, -:class:`HTTP/2 ` +:class:`HTTP/1.* `, +:class:`HTTP/2 ` and etc. .. autoclass:: pcapkit.protocols.application.application.Application diff --git a/docs/source/pcapkit/protocols/application/http.rst b/docs/source/pcapkit/protocols/application/http.rst index 3af596e7e..05d6a25d6 100644 --- a/docs/source/pcapkit/protocols/application/http.rst +++ b/docs/source/pcapkit/protocols/application/http.rst @@ -7,8 +7,8 @@ HTTP - Hypertext Transfer Protocol :class:`~pcapkit.protocols.application.http.HTTP` only, which is a base class for Hypertext Transfer Protocol (HTTP) [*]_ family, eg. -:class:`HTTP/1.* ` -and :class:`HTTP/2 `. +:class:`HTTP/1.* ` +and :class:`HTTP/2 `. .. autoclass:: pcapkit.protocols.application.http.HTTP :no-members: diff --git a/docs/source/pcapkit/protocols/link/l2tp.rst b/docs/source/pcapkit/protocols/link/l2tp.rst index 72fa0922a..554074ec8 100644 --- a/docs/source/pcapkit/protocols/link/l2tp.rst +++ b/docs/source/pcapkit/protocols/link/l2tp.rst @@ -36,7 +36,7 @@ and directly over IP as **protocol number 115**. That second route is why :attr:`Internet.__proto__ ` leaves 115 unbound today: the binding waits on an ``L2TPv3`` class, not on a different framing decision. It also means v3 is the first member of this family -to have a real :meth:`~pcapkit.protocols.protocol.ProtocolBase.__index__`, and so +to have a real :meth:`~pcapkit.protocols.protocol.Protocol.__index__`, and so the first that must have a module of its own under the project's one-module-per-index rule. @@ -45,7 +45,7 @@ an earlier version of L2TP: :rfc:`2661` §3.1 requires ``Ver`` to be 2 and reser the value 1 "to permit detection of L2F packets should they arrive intermixed with L2TP packets". L2F is a separate protocol with its own header. It is therefore to be implemented as ``L2F``, the canonical name, carrying ``L2TPv1`` only as an -alias in its :meth:`~pcapkit.protocols.protocol.ProtocolBase.id` -- the same +alias in its :meth:`~pcapkit.protocols.protocol.Protocol.id` -- the same relationship HTTP/3 has to QUIC. c.f. :meth:`HTTPv1.id ` for how a version-flavoured alias is spelled: canonical name first, alias second, since diff --git a/docs/source/pcapkit/protocols/link/vlan.rst b/docs/source/pcapkit/protocols/link/vlan.rst index 73660f61b..c7b52559d 100644 --- a/docs/source/pcapkit/protocols/link/vlan.rst +++ b/docs/source/pcapkit/protocols/link/vlan.rst @@ -50,14 +50,14 @@ tags appear in one frame: ethernet.s_tag.c_tag.tci.vid = 200 <- customer tag, 802.1Q ethernet.s_tag.c_tag.type = 0x0800 -:attr:`~pcapkit.protocols.protocol.ProtocolBase.info_name` -- ``s_tag`` against +:attr:`~pcapkit.protocols.protocol.Protocol.info_name` -- ``s_tag`` against ``c_tag`` -- is what keeps the two apart in the parsed :class:`~pcapkit.corekit.infoclass.Info`. A single class bound at both EtherTypes would nest one ``c_tag`` inside another, leaving nothing in the output to say which of the two was the service tag. Two distinct EtherTypes also means two distinct -:meth:`~pcapkit.protocols.protocol.ProtocolBase.__index__` values, which is the +:meth:`~pcapkit.protocols.protocol.Protocol.__index__` values, which is the project's rule for when protocols get separate modules: siblings that *share* an index may share a module, as :class:`~pcapkit.protocols.link.arp.InARP` shares :mod:`~pcapkit.protocols.link.arp` and diff --git a/docs/source/pcapkit/protocols/protocol.rst b/docs/source/pcapkit/protocols/protocol.rst index 555394cc1..0b8790c67 100644 --- a/docs/source/pcapkit/protocols/protocol.rst +++ b/docs/source/pcapkit/protocols/protocol.rst @@ -65,7 +65,9 @@ utility arguments and methods of specified protocols. .. automethod:: _lookup_next_layer .. automethod:: _decode_next_layer .. automethod:: _import_next_layer + .. automethod:: _get_context + .. autoattribute:: _data .. autoattribute:: _file .. autoattribute:: _info .. autoattribute:: __data__ @@ -83,6 +85,7 @@ utility arguments and methods of specified protocols. .. autoattribute:: _exlayer .. autoattribute:: _exproto + .. autoattribute:: _exctx Data Models ----------- diff --git a/docs/source/pcapkit/utilities/chardet.rst b/docs/source/pcapkit/utilities/chardet.rst index 29515d5cb..86e253b16 100644 --- a/docs/source/pcapkit/utilities/chardet.rst +++ b/docs/source/pcapkit/utilities/chardet.rst @@ -7,7 +7,7 @@ Character Set Detection turning the bytes of a text field into a :obj:`str`. It is shared by :meth:`StringField.post_process ` and -:meth:`ProtocolBase.decode `, +:meth:`ProtocolBase.decode `, which is why it lives here rather than beside either of them. .. _chardet: https://chardet.readthedocs.io diff --git a/docs/source/pep.rst b/docs/source/pep.rst index 7f8a99b3a..b6d3455f4 100644 --- a/docs/source/pep.rst +++ b/docs/source/pep.rst @@ -346,7 +346,7 @@ Three follow-ups the above deliberately left alone: :class:`~pcapkit.protocols.link.l2tpv2.L2TPv2`, the :rfc:`2661` v2 framing, reached over UDP 1701. So the binding waits on a v3 dissector, which is also the first member of the family to carry an - :meth:`~pcapkit.protocols.protocol.ProtocolBase.__index__` of its own -- 115 + :meth:`~pcapkit.protocols.protocol.Protocol.__index__` of its own -- 115 being that index. :mod:`pcapkit.protocols.link.l2tp` records what v3 needs, and what ``L2F`` needs alongside it: the version nibble reading ``1`` selects L2F [:rfc:`2341`], a separate protocol, not an earlier L2TP. @@ -420,7 +420,7 @@ landed in full and the other two in part: ``pcapkit/toolkit/pcap.py:53`` filters only on the *DF* flag -- so a capture with no fragments at all still produces one "datagram" per frame. Parsing its payload is no longer part of that cost: - :meth:`~pcapkit.protocols.protocol.ProtocolBase.analyze` used to run eagerly on + :meth:`~pcapkit.protocols.protocol.Protocol.analyze` used to run eagerly on each one, 86% of the IP-reassembly cost plus a 133 ms garbage-collection bill, and :attr:`Datagram.packet ` is now a diff --git a/pcapkit/corekit/context.py b/pcapkit/corekit/context.py index 8866cb93d..fee9fcf17 100644 --- a/pcapkit/corekit/context.py +++ b/pcapkit/corekit/context.py @@ -20,13 +20,13 @@ Rather than adding protocol specific keyword arguments to :class:`~pcapkit.foundation.extraction.Extractor`, such information is passed as a :class:`ContextRegistry` -- a mapping of protocol index ID -(c.f. :meth:`Protocol.id `) +(c.f. :meth:`Protocol.id `) to a :class:`ProtocolContext` instance. The registry is handed to :class:`~pcapkit.foundation.extraction.Extractor` once, and is then propagated down the protocol stack by -:meth:`Protocol._import_next_layer `, +:meth:`Protocol._import_next_layer `, so that a protocol nested arbitrarily deep can reach it through -:meth:`Protocol._get_context `. +:meth:`Protocol._get_context `. Example: Decoding an ESP tunnel end to end:: @@ -90,7 +90,7 @@ def protocol(cls) -> 'tuple[str, ...]': """Index ID of the protocol(s) this context applies to. The returned names are matched against - :meth:`Protocol.id `, + :meth:`Protocol.id `, and are case insensitive. """ diff --git a/pcapkit/dumpkit/common.py b/pcapkit/dumpkit/common.py index af8183b31..7451bb681 100644 --- a/pcapkit/dumpkit/common.py +++ b/pcapkit/dumpkit/common.py @@ -230,7 +230,7 @@ def default(self, o: 'Any') -> 'Literal["fallback"]': # pylint: disable=unused- This function is a fallback for :meth:`dictdumper.dumper.Dumper.default`. It will be called when :meth:`dictdumper.dumper.Dumper.default` fails to find a suitable function for dumping and it should pair with - :func:`pcapkit.dumpkit.common._append_fallback` for use. + ``_append_fallback`` for use. """ return 'fallback' @@ -247,7 +247,7 @@ def _append_fallback(self, value: 'Any', file: 'TextIO') -> 'None': This function is a fallback for :meth:`dictdumper.dumper.Dumper.default`. It will be called when :meth:`dictdumper.dumper.Dumper.default` fails to find a suitable function for dumping and it should pair with - :func:`pcapkit.dumpkit.common.default` for use. + ``default`` for use. """ if hasattr(value, '__slots__'): diff --git a/pcapkit/foundation/extraction.py b/pcapkit/foundation/extraction.py index a62422989..d78753537 100644 --- a/pcapkit/foundation/extraction.py +++ b/pcapkit/foundation/extraction.py @@ -126,6 +126,11 @@ class Extractor(Generic[_P]): _flag_f: 'bool' #: No output file, i.e., no output file is to be generated. _flag_q: 'bool' + #: Reassembly flag. It indicates if datagram reassembly is enabled. + #: Every engine reads it to decide whether to feed + #: :attr:`~pcapkit.foundation.extraction.Extractor._reasm`, and the + #: engines that cannot reassemble clear it on startup. + _flag_r: 'bool' #: Trace flag. It indicates if the flow tracing is enabled. _flag_t: 'bool' #: Verbose flag. This is used to determine if the verbose callback @@ -648,7 +653,7 @@ def record_header(self) -> 'Engine': The method will parse the PCAP global header and save the parsed result to its extraction context. Information such as PCAP version, data link layer protocol type, nanosecond flag and byteorder will also be save - the current :class:`~pcapkit.foundation.engins.engine.Engine` instance + the current :class:`~pcapkit.foundation.engines.engine.Engine` instance as well. If TCP flow tracing is enabled, the nanosecond flag and byteorder will @@ -784,7 +789,7 @@ def __init__(self, buffer_save: if save buffer to file (for :class:`~pcapkit.corekit.io.SeekableReader` only) buffer_path: path name for buffer file if necessary (for :class:`~pcapkit.corekit.io.SeekableReader` only) - no_eof: if raise :exc:`EOFError` when EOF + no_eof: if not raise :exc:`EOFError` when reach EOF context: caller supplied parsing context for protocols that need information not carried on the wire, keyed by protocol index diff --git a/pcapkit/foundation/reassembly/data/data.py b/pcapkit/foundation/reassembly/data/data.py index fb670094f..9f79e9919 100644 --- a/pcapkit/foundation/reassembly/data/data.py +++ b/pcapkit/foundation/reassembly/data/data.py @@ -121,7 +121,7 @@ class Deferred: Args: analyze: The analyser to call, i.e. - :meth:`Protocol.analyze ` + :meth:`Protocol.analyze ` bound to the reassembly object's protocol. proto: Payload protocol type. payload: Reassembled payload to parse. diff --git a/pcapkit/foundation/reassembly/data/tcp.py b/pcapkit/foundation/reassembly/data/tcp.py index df58ee113..be666b217 100644 --- a/pcapkit/foundation/reassembly/data/tcp.py +++ b/pcapkit/foundation/reassembly/data/tcp.py @@ -103,9 +103,9 @@ class Datagram(DeferredPacket, Info, Generic[_AT]): header: 'bytes' #: Reassembled payload (application layer data). payload: 'bytes | tuple[bytes, ...]' - #: Parsed reassembled payload. #: Parsed TCP payload. Analysed on first read rather than at construction; - #: a :class:`Deferred` may be passed in its place. + #: a :class:`Deferred` may be passed in its place, and reading this + #: attribute then runs it and keeps the result. packet: 'Optional[Protocol]' #: Sequence ranges on which two segments disagreed, i.e. where an arriving #: segment overlapped bytes already buffered but did not repeat them. diff --git a/pcapkit/foundation/registry/foundation.py b/pcapkit/foundation/registry/foundation.py index 2de4fd9db..eb42eda1a 100644 --- a/pcapkit/foundation/registry/foundation.py +++ b/pcapkit/foundation/registry/foundation.py @@ -71,7 +71,7 @@ def register_extractor_engine(name: 'str', module: 'ModuleDescriptor[Engine] | T :data:`pcapkit.foundation.extraction.Extractor.__engine__` registry. Arguments: - engine: engine name + name: engine name module: module name or module descriptor or an :class:`~pcapkit.foundation.engines.engine.Engine` subclass class\_: class name diff --git a/pcapkit/foundation/registry/protocols.py b/pcapkit/foundation/registry/protocols.py index 421f75974..53f4b1659 100644 --- a/pcapkit/foundation/registry/protocols.py +++ b/pcapkit/foundation/registry/protocols.py @@ -492,7 +492,7 @@ def register_mh_message(code: 'MH_Packet', meth: 'str | tuple[MH_PacketParser, M :data:`pcapkit.protocols.internet.mh.MH.__message__` registry. Args: - code: :class:`~pcapkit.protocols.internet.mh.MH>` + code: :class:`~pcapkit.protocols.internet.mh.MH` data type code as in :class:`~pcapkit.const.mh.packet.Packet`. meth: Method name or callable to parse and/or construct the data. schema: :class:`~pcapkit.protocols.schema.schema.Schema` class for the message type. @@ -517,7 +517,7 @@ def register_mh_option(code: 'MH_Option', meth: 'str | tuple[MH_OptionParser, MH :data:`pcapkit.protocols.internet.mh.MH.__option__` registry. Args: - code: :class:`~pcapkit.protocols.internet.mh.MH>` + code: :class:`~pcapkit.protocols.internet.mh.MH` data type code as in :class:`~pcapkit.const.mh.option.Option`. meth: Method name or callable to parse and/or construct the data. schema: :class:`~pcapkit.protocols.schema.schema.Schema` class for the message type. @@ -542,7 +542,7 @@ def register_mh_extension(code: 'MH_CGAExtension', meth: 'str | tuple[MH_Extensi :data:`pcapkit.protocols.internet.mh.MH.__extension__` registry. Args: - code: :class:`~pcapkit.protocols.internet.mh.MH>` + code: :class:`~pcapkit.protocols.internet.mh.MH` data type code as in :class:`~pcapkit.const.mh.cga_extension.CGAExtension`. meth: Method name or callable to parse and/or construct the data. schema: :class:`~pcapkit.protocols.schema.schema.Schema` class for the message type. diff --git a/pcapkit/protocols/application/application.py b/pcapkit/protocols/application/application.py index 3fcf4c831..b00fe99ac 100644 --- a/pcapkit/protocols/application/application.py +++ b/pcapkit/protocols/application/application.py @@ -7,8 +7,8 @@ :mod:`pcapkit.protocols.application.application` contains only :class:`~pcapkit.protocols.application.application.Application`, which is a base class for application layer protocols, eg. -:class:`HTTP/1.* `, -:class:`HTTP/2 ` +:class:`HTTP/1.* `, +:class:`HTTP/2 ` and etc. """ diff --git a/pcapkit/protocols/application/http.py b/pcapkit/protocols/application/http.py index 592315b9b..4cbdb2aba 100644 --- a/pcapkit/protocols/application/http.py +++ b/pcapkit/protocols/application/http.py @@ -8,8 +8,8 @@ :class:`~pcapkit.protocols.application.http.HTTP` only, which is a base class for Hypertext Transfer Protocol (HTTP) [*]_ family, eg. -:class:`HTTP/1.* ` -and :class:`HTTP/2 `. +:class:`HTTP/1.* ` +and :class:`HTTP/2 `. .. [*] https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol diff --git a/pcapkit/protocols/internet/esp.py b/pcapkit/protocols/internet/esp.py index 39615b680..73753d28e 100644 --- a/pcapkit/protocols/internet/esp.py +++ b/pcapkit/protocols/internet/esp.py @@ -1395,7 +1395,7 @@ def _make_opaque(self, spi: 'int', seq: 'int', total: 'int', payload_data: 'byte The payload is surfaced as :class:`~pcapkit.protocols.misc.raw.Raw` -- which is what a next header of :data:`None` resolves to -- so that - :attr:`self.payload ` + :attr:`self.payload ` and the protocol chain behave as they do for any other protocol, and the trailer fields are left :data:`None` rather than guessed at. diff --git a/pcapkit/protocols/internet/hopopt.py b/pcapkit/protocols/internet/hopopt.py index 4748a96f6..19802bd46 100644 --- a/pcapkit/protocols/internet/hopopt.py +++ b/pcapkit/protocols/internet/hopopt.py @@ -597,7 +597,7 @@ def _read_opt_pad(self, schema: 'Schema_PadOption', *, options: 'Option') -> 'Da Note: A ``Pad1`` option occupies a single octet and carries no ``Opt Data Len`` field, so its - :attr:`~pcapkit.protocols.data.internet.hopopt.PadOption.length` is + :attr:`~pcapkit.protocols.data.internet.hopopt.Option.length` is ``1`` rather than ``len + 2``. That one-octet wire shape is enforced by :class:`~pcapkit.protocols.schema.internet.hopopt.PadOption` itself, which sizes both the length octet and the padding data from @@ -1421,7 +1421,7 @@ def _make_opt_pad(self, code: 'Enum_Option', opt: 'Optional[Data_PadOption]' = N Note: :attr:`Data_PadOption.length - ` counts the + ` counts the *whole* option, whereas :attr:`Schema_PadOption.len ` is the ``Opt Data Len`` field -- two octets fewer, and absent altogether for diff --git a/pcapkit/protocols/internet/ipv6_opts.py b/pcapkit/protocols/internet/ipv6_opts.py index 80e174131..73b20c077 100644 --- a/pcapkit/protocols/internet/ipv6_opts.py +++ b/pcapkit/protocols/internet/ipv6_opts.py @@ -599,7 +599,7 @@ def _read_opt_pad(self, schema: 'Schema_PadOption', *, options: 'Option') -> 'Da Note: A ``Pad1`` option occupies a single octet and carries no ``Opt Data Len`` field, so its - :attr:`~pcapkit.protocols.data.internet.ipv6_opts.PadOption.length` + :attr:`~pcapkit.protocols.data.internet.ipv6_opts.Option.length` is ``1`` rather than ``len + 2``. That one-octet wire shape is enforced by :class:`~pcapkit.protocols.schema.internet.ipv6_opts.PadOption` @@ -1424,7 +1424,7 @@ def _make_opt_pad(self, code: 'Enum_Option', opt: 'Optional[Data_PadOption]' = N Note: :attr:`Data_PadOption.length - ` counts + ` counts the *whole* option, whereas :attr:`Schema_PadOption.len ` is the ``Opt Data Len`` field -- two octets fewer, and absent altogether for diff --git a/pcapkit/protocols/link/l2tp.py b/pcapkit/protocols/link/l2tp.py index ffc1601a2..43eebbc3c 100644 --- a/pcapkit/protocols/link/l2tp.py +++ b/pcapkit/protocols/link/l2tp.py @@ -38,14 +38,14 @@ :attr:`Internet.__proto__ ` leaves 115 unbound today: the binding waits on an ``L2TPv3`` class, not on a different framing decision. It also means v3 is the first member of this family -to have a real :meth:`~pcapkit.protocols.protocol.ProtocolBase.__index__`. +to have a real :meth:`~pcapkit.protocols.protocol.Protocol.__index__`. **L2F** [:rfc:`2341`] is reached when the version nibble reads ``1``. It is *not* an earlier version of L2TP: :rfc:`2661` §3.1 requires ``Ver`` to be 2 and reserves the value 1 "to permit detection of L2F packets should they arrive intermixed with L2TP packets". L2F is a separate protocol with its own header. It is therefore to be implemented as ``L2F``, the canonical name, carrying ``L2TPv1`` -only as an alias in its :meth:`~pcapkit.protocols.protocol.ProtocolBase.id` -- +only as an alias in its :meth:`~pcapkit.protocols.protocol.Protocol.id` -- the same relationship HTTP/3 has to QUIC. c.f. :meth:`HTTPv1.id ` for how a version-flavoured alias is spelled: canonical name first, alias second, since @@ -90,8 +90,8 @@ class L2TP(Link[_PT, _ST], Generic[_PT, _ST]): # pylint: disable=abstract-metho It is abstract for the same mechanical reason :class:`~pcapkit.protocols.internet.ip.IP` is: - :attr:`~pcapkit.protocols.protocol.ProtocolBase.name` and - :meth:`~pcapkit.protocols.protocol.ProtocolBase.read` are both declared + :attr:`~pcapkit.protocols.protocol.Protocol.name` and + :meth:`~pcapkit.protocols.protocol.Protocol.read` are both declared abstract by :class:`~pcapkit.protocols.protocol.ProtocolBase` and neither is defined here, so the class cannot be instantiated. Bind a version, never this class. @@ -105,7 +105,7 @@ class L2TP(Link[_PT, _ST], Generic[_PT, _ST]): # pylint: disable=abstract-metho #: NOTE: Declared on the base, and so shared by every version, deliberately. #: This is the key the parsed datagram appears under, and a consumer wants #: ``udp.l2tp`` whichever version was on the wire -- the version is reported - #: by :attr:`~pcapkit.protocols.protocol.ProtocolBase.alias` instead. Left to + #: by :attr:`~pcapkit.protocols.protocol.Protocol.alias` instead. Left to #: the class-name default it would read ``l2tpv2``, ``l2tpv3`` and so on, and #: every consumer would have to know the version to find the data. @property diff --git a/pcapkit/protocols/link/ospf.py b/pcapkit/protocols/link/ospf.py index 309efac56..e5f8ee422 100644 --- a/pcapkit/protocols/link/ospf.py +++ b/pcapkit/protocols/link/ospf.py @@ -90,13 +90,14 @@ class OSPF(Link[Data_OSPF, Schema_OSPF], """ #: Version number of corresponding protocol, as read off the header. Held on #: the instance rather than read back out of :attr:`self._info - #: ` because :attr:`name` and + #: ` because :attr:`name` and #: :attr:`alias` are both needed *during* :meth:`read` -- it is #: :meth:`self._decode_next_layer - #: ` that builds + #: ` that builds #: the protocol chain out of :attr:`alias` -- and ``_info`` is not assigned - #: until :meth:`read` has returned. c.f. :attr:`ARP._acnm - #: `, which carries the same constraint. + #: until :meth:`read` has returned. c.f. ``ARP._acnm`` on + #: :class:`~pcapkit.protocols.link.arp.ARP`, which carries the same + #: constraint. _version: 'int' ########################################################################## diff --git a/pcapkit/protocols/link/vlan.py b/pcapkit/protocols/link/vlan.py index 477dcee8f..812f5b290 100644 --- a/pcapkit/protocols/link/vlan.py +++ b/pcapkit/protocols/link/vlan.py @@ -32,14 +32,14 @@ They are nonetheless distinct classes rather than one class bound at two EtherTypes, because 802.1ad *stacks* them: a Q-in-Q frame carries a service tag whose next EtherType is ``0x8100``, selecting a customer tag in turn. Both tags -therefore appear in one frame, and :attr:`~pcapkit.protocols.protocol.ProtocolBase.info_name` +therefore appear in one frame, and :attr:`~pcapkit.protocols.protocol.Protocol.info_name` -- ``s_tag`` against ``c_tag`` -- is what keeps them apart in the parsed :class:`~pcapkit.corekit.infoclass.Info`. A single class bound at both EtherTypes would nest one ``c_tag`` inside another, leaving nothing in the output to say which of the two was the service tag. Two distinct EtherTypes also means two distinct -:meth:`~pcapkit.protocols.protocol.ProtocolBase.__index__` values, which is the +:meth:`~pcapkit.protocols.protocol.Protocol.__index__` values, which is the project's rule for when protocols get separate modules: siblings that *share* an index may share a module, as :class:`~pcapkit.protocols.link.arp.InARP` shares :mod:`~pcapkit.protocols.link.arp` and @@ -83,10 +83,10 @@ class VLAN(Link[Data_VLAN, Schema_VLAN], # pylint: disable=abstract-method since the customer and service tags are byte-for-byte identical. What it deliberately leaves to its subclasses is only how the tag *names* itself: :attr:`name`, :attr:`alias` and - :attr:`~pcapkit.protocols.protocol.ProtocolBase.info_name`. + :attr:`~pcapkit.protocols.protocol.Protocol.info_name`. It is abstract for the same reason :class:`~pcapkit.protocols.internet.ip.IP` - is: :attr:`~pcapkit.protocols.protocol.ProtocolBase.name` is declared + is: :attr:`~pcapkit.protocols.protocol.Protocol.name` is declared abstract by :class:`~pcapkit.protocols.protocol.ProtocolBase` and is not defined here, so the class cannot be instantiated. Bind :class:`C_Tag` or :class:`S_Tag`, never this class. diff --git a/pcapkit/protocols/misc/pcap/frame.py b/pcapkit/protocols/misc/pcap/frame.py index 7e98692df..2b67d3c4e 100644 --- a/pcapkit/protocols/misc/pcap/frame.py +++ b/pcapkit/protocols/misc/pcap/frame.py @@ -403,8 +403,8 @@ def __index__(self: 'Optional[Frame]' = None) -> 'int': # type: ignore[override self: :class:`Frame` object or :obj:`None`. Returns: - If the object is initiated, i.e. :attr:`self._fnum ` - exists, returns the frame index number of itself; else raises :exc:`UnsupportedCall`. + If the object is initiated, i.e. ``self._fnum`` exists, returns the + frame index number of itself; else raises :exc:`UnsupportedCall`. Raises: UnsupportedCall: This protocol has no registry entry. diff --git a/pcapkit/protocols/misc/pcapng.py b/pcapkit/protocols/misc/pcapng.py index 6994249c5..bfbe1b95d 100644 --- a/pcapkit/protocols/misc/pcapng.py +++ b/pcapkit/protocols/misc/pcapng.py @@ -1067,7 +1067,7 @@ def __post_init__(self, file: 'Optional[IO[bytes] | bytes]' = None, length: 'Opt self._sect = sct #: int: Block index number. self._fnum = num - #: pcapkit.foundation.engins.pcapng.Context: Context of the PCAP-NG file. + #: pcapkit.foundation.engines.pcapng.Context: Context of the PCAP-NG file. self._ctx = ctx #: collections.Counter: Counter for option types. self._opt = collections.Counter() # type: Counter[Enum_OptionType] @@ -1100,9 +1100,9 @@ def __index__(self: 'Optional[PCAPNG]' = None) -> 'int': # type: ignore[overrid self: :class:`PCAPNG` object or :obj:`None`. Returns: - If the object is initiated, i.e. :attr:`self._fnum ` - exists, and is of a packet block (EPB, ISB or Packet), returns the - block index number of itself; else raises :exc:`UnsupportedCall`. + If the object is initiated, i.e. ``self._fnum`` exists, and is of a + packet block (EPB, ISB or Packet), returns the block index number of + itself; else raises :exc:`UnsupportedCall`. Raises: UnsupportedCall: This protocol has no registry entry. diff --git a/pcapkit/protocols/protocol.py b/pcapkit/protocols/protocol.py index e296a879b..34ec9c08c 100644 --- a/pcapkit/protocols/protocol.py +++ b/pcapkit/protocols/protocol.py @@ -696,8 +696,8 @@ def __init_subclass__(cls, /, schema: 'Optional[Type[_ST]]' = None, try to find the corresponding class in the :mod:`~pcapkit.protocols.schema` and :mod:`~pcapkit.protocols.data` modules respectively. If the class is not found, the default - :class:`~pcapkit.protocols.schema.schema.Schema_Raw` and - :class:`~pcapkit.protocols.data.data.Data_Raw` classes will be used. + :class:`~pcapkit.protocols.schema.misc.raw.Raw` and + :class:`~pcapkit.protocols.data.misc.raw.Raw` classes will be used. """ super().__init_subclass__() @@ -1428,8 +1428,8 @@ def __init_subclass__(cls, /, schema: 'Optional[Type[_ST]]' = None, try to find the corresponding class in the :mod:`~pcapkit.protocols.schema` and :mod:`~pcapkit.protocols.data` modules respectively. If the class is not found, the default - :class:`~pcapkit.protocols.schema.schema.Schema_Raw` and - :class:`~pcapkit.protocols.data.data.Data_Raw` classes will be used. + :class:`~pcapkit.protocols.schema.misc.raw.Raw` and + :class:`~pcapkit.protocols.data.misc.raw.Raw` classes will be used. This method also registers the subclass to the protocol registry, i.e., :attr:`pcapkit.protocols.__proto__`. diff --git a/pcapkit/protocols/schema/internet/esp.py b/pcapkit/protocols/schema/internet/esp.py index c513bd105..00603a2fe 100644 --- a/pcapkit/protocols/schema/internet/esp.py +++ b/pcapkit/protocols/schema/internet/esp.py @@ -29,7 +29,7 @@ class ESP(Schema): This also means :attr:`payload` is **not** the next layer's data: the next layer lives inside the ciphertext, and is handed to - :meth:`Protocol._decode_next_layer ` + :meth:`Protocol._decode_next_layer ` explicitly once decrypted. """ diff --git a/pcapkit/protocols/schema/internet/mh.py b/pcapkit/protocols/schema/internet/mh.py index c08fff894..ea66b5ce1 100644 --- a/pcapkit/protocols/schema/internet/mh.py +++ b/pcapkit/protocols/schema/internet/mh.py @@ -1798,7 +1798,7 @@ class ANIGeoLocationSuboption(ANISuboption, code=Enum_ANISuboption.Geo_Location) with 9 integer bits [:rfc:`6757#section-3.1.2`]. A :class:`~pcapkit.corekit.fields.strings.BitField` reads them unsigned, so the sign is applied when - :meth:`~pcapkit.protocols.internet.mh.MH._read_ani_geo` builds the data + :meth:`~pcapkit.protocols.internet.mh.MH._read_ani_suboptions` builds the data model. """ diff --git a/pcapkit/protocols/schema/link/ethernet.py b/pcapkit/protocols/schema/link/ethernet.py index 9448b56c3..f36e92fbe 100644 --- a/pcapkit/protocols/schema/link/ethernet.py +++ b/pcapkit/protocols/schema/link/ethernet.py @@ -30,7 +30,7 @@ def callback_payload(self: 'PayloadField', packet: 'dict[str, Any]') -> 'None': Important: The lookup goes through :meth:`ProtocolBase._lookup_next_layer - ` rather than + ` rather than subscripting the registry. :attr:`Ethernet.__proto__ ` *is* :attr:`Link.__proto__ `, a diff --git a/pcapkit/protocols/schema/misc/pcapng.py b/pcapkit/protocols/schema/misc/pcapng.py index 0a77a9023..b0e6418d4 100644 --- a/pcapkit/protocols/schema/misc/pcapng.py +++ b/pcapkit/protocols/schema/misc/pcapng.py @@ -237,15 +237,15 @@ def dsb_secrets_selector(packet: 'dict[str, Any]') -> 'Field': Returns: * If ``secrets_type`` is unknown, returns a :class:`~pcapkit.corekit.fields.strings.BytesField` instance. - * If ``secret_type`` is :attr:`~pcapkit.const.pcapng.secrets_type.Secrets_Type.TLS_Key_Log` - and/or :attr:`~pcapkit.const.pcapng.secrets_type.Secrets_Type.WireGuard_Key_Log`, + * If ``secret_type`` is :attr:`~pcapkit.const.pcapng.secrets_type.SecretsType.TLS_Key_Log` + and/or :attr:`~pcapkit.const.pcapng.secrets_type.SecretsType.WireGuard_Key_Log`, returns a :class:`~pcapkit.corekit.fields.strings.StringField` instance. * Otherwise, returns a :class:`~pcapkit.corekit.fields.misc.SchemaField` wrapped :class:`~pcapkit.protocols.schema.misc.pcapng.DSBSecrets` subclass instance. See Also: - * :class:`pcapkit.const.pcapng.secrets_type.Secrets_Type` + * :class:`pcapkit.const.pcapng.secrets_type.SecretsType` * :class:`pcapkit.protocols.schema.misc.pcapng.DSBSecrets` """ diff --git a/pcapkit/protocols/transport/sctp.py b/pcapkit/protocols/transport/sctp.py index 3090794cc..b70e9b181 100644 --- a/pcapkit/protocols/transport/sctp.py +++ b/pcapkit/protocols/transport/sctp.py @@ -813,10 +813,10 @@ def _decode_next_layer(self, dict_: 'Data_SCTP', proto: 'Optional[int]' = None, does for an unregistered transport type. Resolving it to :class:`~pcapkit.protocols.misc.raw.Raw` is :meth:`ProtocolBase._import_next_layer - `'s job, + `'s job, which looks the PPID up through :meth:`ProtocolBase._lookup_next_layer - ` and so + ` and so leaves :attr:`self.__proto__ ` untouched. """ diff --git a/pcapkit/protocols/transport/transport.py b/pcapkit/protocols/transport/transport.py index fc55b51db..6f347bf5f 100644 --- a/pcapkit/protocols/transport/transport.py +++ b/pcapkit/protocols/transport/transport.py @@ -184,7 +184,7 @@ def _decode_next_layer(self, dict_: '_PT', ports: 'tuple[int, int]', length: 'Op Important: The port is forwarded **whether or not it is registered**, since :meth:`ProtocolBase._import_next_layer - ` passes + ` passes it on as ``alias`` and :class:`~pcapkit.protocols.misc.raw.Raw` records it as :attr:`Data_Raw.protocol `. Dropping it -- as diff --git a/pcapkit/utilities/chardet.py b/pcapkit/utilities/chardet.py index 3a8018c42..a8cd7c67c 100644 --- a/pcapkit/utilities/chardet.py +++ b/pcapkit/utilities/chardet.py @@ -41,7 +41,7 @@ def detect(value: 'bytes') -> 'str': retains 19.1 MB. :data:`DETECT_CACHE_SIZE` therefore caps the entries rather than the footprint, which matters because :meth:`ProtocolBase.decode - ` is public and a caller + ` is public and a caller may hand it a whole payload. Use :meth:`detect.cache_clear ` to release it in a long-running process. diff --git a/pcapkit/utilities/exceptions.py b/pcapkit/utilities/exceptions.py index 7e032130c..72d6c9a48 100644 --- a/pcapkit/utilities/exceptions.py +++ b/pcapkit/utilities/exceptions.py @@ -4,7 +4,7 @@ .. module:: pcapkit.utilities.exceptions -:mod:`pcapkit.exceptions` refined built-in exceptions. +:mod:`pcapkit.utilities.exceptions` refined built-in exceptions. Make it possible to show only user error stack infomation [*]_, when exception raised on user's operation. diff --git a/pcapkit/vendor/mh/status_code.py b/pcapkit/vendor/mh/status_code.py index b4fff214f..357560cf9 100644 --- a/pcapkit/vendor/mh/status_code.py +++ b/pcapkit/vendor/mh/status_code.py @@ -5,7 +5,7 @@ .. module:: pcapkit.vendor.mh.status_code This module contains the vendor crawler for **Status Codes**, -which is automatically generating :class:`pcapkit.const.mh.dns_status_code.StatusCode`. +which is automatically generating :class:`pcapkit.const.mh.dns_status_code.DNSStatusCode`. """ diff --git a/pyproject.toml b/pyproject.toml index 39d4db134..ffc415408 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -183,10 +183,13 @@ PyPCAP = [ "pypcap; python_version < '3.12'" ] # The ``>= '3.10'`` marker is the same courtesy as the ceilings above, pointing # the other way. Taken from the wheel metadata rather than guessed: ``pcap-ct`` # 1.3.0b3 declares ``Requires-Python: <4.0.0,>=3.9.0`` and ``libpcap`` -# 1.11.0b29 declares ``<4.0.0,>=3.10.0``, so 3.10 is the binding floor. This -# project's own floor is now ``>=3.10`` too, so the marker is belt-and-braces -# rather than load-bearing -- it keeps the requirement honest on its own terms, and -# would matter again if the project floor ever moved back down. +# 1.11.0b29 declares ``<4.0.0,>=3.10.0``, so 3.10 is the binding floor. The +# marker is load-bearing rather than decorative, because this project's own +# ``requires-python`` is ``>=3.6, <4`` (see the note above it): without the +# marker, ``pip install pypcapkit[PCAP_CT]`` on a 3.6-3.9 interpreter would try +# to resolve two distributions that exclude it and fail the whole install, where +# with it the extra simply resolves to nothing there. CI covers 3.10 upwards, so +# the marker never excludes a tested interpreter. PCAP_CT = [ "pcap-ct>=1.3.0b3; python_version >= '3.10'", "libpcap>=1.11.0b29; python_version >= '3.10'", diff --git a/tests/project/test_documentation_claims.py b/tests/project/test_documentation_claims.py new file mode 100644 index 000000000..2b7783e46 --- /dev/null +++ b/tests/project/test_documentation_claims.py @@ -0,0 +1,164 @@ +# -*- coding: utf-8 -*- +"""Tests for documentation that states something checkable about the code. + +Most of what #546 collected is prose, and prose cannot fail a test. These are the +parts that can, so that the specific defects it found cannot come back silently: + +* ``no_eof`` was documented **backwards** on :class:`~pcapkit.foundation.extraction.Extractor` + while :func:`~pcapkit.interface.core.extract` documented it correctly. Two + docstrings for one parameter disagreed, and the flag is on the primary public + entry point, so a caller reading the wrong one gets a non-terminating + extraction. :class:`TestNoEOFDocumentedOnce` pins the two together *and* to the + branch in the code that decides the behaviour, so a future edit to either + docstring alone fails here rather than in somebody's capture loop. + +* ``Extractor._flag_r`` was referenced by :mod:`pcapkit`'s own documentation but + was the one flag missing from the ``if TYPE_CHECKING:`` block that documents + its siblings, so the reference pointed at nothing. + :class:`TestExtractorFlagsDocumented` asserts every flag the constructor + assigns is declared, which is the general form of that defect. + +#546's item 2 -- ``register_extractor_engine`` documenting a parameter named +``engine`` where the signature reads ``name`` -- is **not** tested here, because +:file:`tests/test_docstring_contract.py` already checks that property across the +whole of :mod:`pcapkit` and carried the defect as a :data:`KNOWN_DEFECTS` entry +reading "owned by the registry docstring change". Fixing it is therefore a +*deletion* from that tuple rather than a new test, and a second checker here would +only be a narrower copy of a better one. + +Deliberately **not** tested here either: whether a Sphinx cross-reference +resolves. That is a property of the built inventory, not of the source, and the +honest check for it is a nitpicky ``sphinx-build`` -- some 20 minutes, and it needs +the docs toolchain installed. A unit test that re-implemented Sphinx's resolution +rules would pass while the real build failed, which is worse than not testing it. +The measured counts are recorded in the pull request instead. + +""" + +from __future__ import annotations + +import ast +import inspect +import pathlib +import textwrap +import unittest + +ROOT = pathlib.Path(__file__).resolve().parents[2] + + +class TestNoEOFDocumentedOnce(unittest.TestCase): + """``no_eof`` means "keep going", and both docstrings have to say so.""" + + @staticmethod + def _no_eof_line(doc: 'str | None') -> 'str': + for line in (doc or '').splitlines(): + stripped = line.strip() + if stripped.startswith('no_eof:'): + return ' '.join(stripped.split()) + raise AssertionError('no no_eof entry found in docstring') + + def test_extractor_and_extract_agree(self) -> None: + """The two public docstrings for ``no_eof`` say the same thing.""" + from pcapkit.foundation.extraction import Extractor + from pcapkit.interface.core import extract + + self.assertEqual( + self._no_eof_line(inspect.getdoc(Extractor.__init__)), + self._no_eof_line(inspect.getdoc(extract)), + 'Extractor.__init__ and extract() document no_eof differently; they ' + 'are the same flag and disagreeing is how #546 arose', + ) + + def test_documented_sense_matches_the_code(self) -> None: + """The docstring says "not raise", and the code continues on EOF. + + The behaviour is decided by ``if self._flag_n: continue`` inside the + ``except (EOFError, StopIteration)`` handler of ``record_frames``: the + flag makes extraction *carry on*. So the documentation has to be phrased + as suppressing the error, not as raising it. + + """ + from pcapkit.foundation.extraction import Extractor + + line = self._no_eof_line(inspect.getdoc(Extractor.__init__)) + self.assertIn('not raise', line, + f'no_eof is documented as {line!r}, but the flag suppresses ' + 'the EOF stop rather than raising') + + tree = ast.parse(textwrap.dedent(inspect.getsource(Extractor.record_frames))) + + # Find the handler for EOFError and confirm it can `continue` on the flag. + handlers = [node for node in ast.walk(tree) if isinstance(node, ast.ExceptHandler)] + eof_handlers = [ + h for h in handlers + if h.type is not None and 'EOFError' in ast.unparse(h.type) + ] + self.assertTrue(eof_handlers, 'record_frames no longer handles EOFError') + + continues_on_flag = False + for handler in eof_handlers: + for node in ast.walk(handler): + if (isinstance(node, ast.If) + and '_flag_n' in ast.unparse(node.test) + and any(isinstance(inner, ast.Continue) for inner in node.body)): + continues_on_flag = True + self.assertTrue( + continues_on_flag, + 'record_frames no longer continues on _flag_n when EOF is reached, so ' + 'the documented sense of no_eof may have changed', + ) + + +class TestExtractorFlagsDocumented(unittest.TestCase): + """Every ``_flag_*`` the constructor sets is a documented attribute.""" + + @staticmethod + def _parse_extraction() -> 'ast.ClassDef': + path = ROOT / 'pcapkit' / 'foundation' / 'extraction.py' + tree = ast.parse(path.read_text(encoding='utf-8')) + for node in tree.body: + if isinstance(node, ast.ClassDef) and node.name == 'Extractor': + return node + raise AssertionError('Extractor class not found in extraction.py') + + def test_assigned_flags_are_declared(self) -> None: + """``self._flag_x = ...`` in ``__init__`` implies a ``_flag_x`` declaration. + + The declarations live in the class's ``if TYPE_CHECKING:`` block, each + with a ``#:`` comment, and that is what Sphinx documents them from. A + flag that is assigned but never declared is undocumented, and any + ``:attr:`` reference to it is dead -- which is what happened to + ``_flag_r``. + + """ + cls = self._parse_extraction() + + assigned = set() + for node in ast.walk(cls): + if isinstance(node, ast.Assign): + for target in node.targets: + if (isinstance(target, ast.Attribute) + and isinstance(target.value, ast.Name) + and target.value.id == 'self' + and target.attr.startswith('_flag_')): + assigned.add(target.attr) + + declared = { + node.target.id + for node in ast.walk(cls) + if isinstance(node, ast.AnnAssign) + and isinstance(node.target, ast.Name) + and node.target.id.startswith('_flag_') + } + + self.assertTrue(assigned, 'no _flag_* assignments found; the test has ' + 'stopped checking anything') + self.assertEqual( + assigned - declared, set(), + 'these Extractor flags are assigned but not declared, so they are ' + f'undocumented: {sorted(assigned - declared)}', + ) + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/test_docstring_contract.py b/tests/test_docstring_contract.py index 68d1417c1..2baefca93 100644 --- a/tests/test_docstring_contract.py +++ b/tests/test_docstring_contract.py @@ -221,10 +221,11 @@ class Known(NamedTuple): #: that work. The corrections are recorded in the issue thread; nothing here #: is a claim that the docstring is right. KNOWN_DEFECTS = ( - Known(Finding('pcapkit/foundation/registry/foundation.py', - 'register_extractor_engine', 'engine'), - "documents 'engine' where the parameter is 'name'; owned by the " - 'registry docstring change'), + # NOTE: the ``register_extractor_engine`` entry that used to head this tuple + # is gone because the defect is fixed -- it documented 'engine' where the + # parameter is 'name', which is #546's item 2. The remaining 34 register_* + # functions in that file were checked at the same time and none of them + # carries the same defect, so there is nothing left to record here for it. Known(Finding('pcapkit/protocols/internet/ipv4.py', '_make_ipv4_options', 'option'), "documents 'option' where the parameter is 'options'; ipv4.py is "