diff --git a/docs/source/changelog.rst b/docs/source/changelog.rst index a9f78ec76..5092122bb 100644 --- a/docs/source/changelog.rst +++ b/docs/source/changelog.rst @@ -36,9 +36,8 @@ All notable changes to PyPCAPKit are recorded here. collapsed into a single line per release below. Where a post-release did carry something real, it is called out. -Each release below has a short summary and a link to its own page, newest -first. The repository root's :file:`CHANGELOG.md` carries only the version -currently being released; this is the whole history. +Releases are listed newest first. The repository root's :file:`CHANGELOG.md` +carries only the version being released; this page is the whole history. * **1.5.0** (unreleased) -- The largest release since 1.0, and the first recorded here as it happened rather than reconstructed: three more diff --git a/docs/source/demo.rst b/docs/source/demo.rst index 395a4f8ff..234b8f7ee 100644 --- a/docs/source/demo.rst +++ b/docs/source/demo.rst @@ -39,7 +39,8 @@ Basic Samples .. code-block:: python from pcapkit import HTTP, extract - # set reasm_strict to make sure full reassembly + # reasm_strict is the default: it submits incompletely reassembled + # datagrams too, which is why the packet check below is needed extraction = extract(fin='in.pcap', store=False, nofile=True, reassembly=True, tcp=True, reasm_strict=True) # print extracted packet if HTTP in reassembled payloads for datagram in extraction.reassembly.tcp: diff --git a/docs/source/ext.rst b/docs/source/ext.rst index 2d74fb352..8595c8d73 100644 --- a/docs/source/ext.rst +++ b/docs/source/ext.rst @@ -1,10 +1,8 @@ Customisation & Extensions ========================== -:mod:`pcapkit` is designed with extensive ability in customisation and -extension. It is easy to add new protocols, layers, fields, and even new file -formats. This section will introduce how to customise and extend :mod:`pcapkit` -to meet your own needs. +:mod:`pcapkit` is built for customisation: new protocols, layers, fields and +even new file formats can all be added from outside the library. ------------------------ What's in for Protocols? @@ -15,7 +13,7 @@ and class in :mod:`pcapkit`. Every protocol is represented by a :class:`~pcapkit.protocols.protocol.Protocol` subclass, which is responsible for parsing and/or constructing the protocol packets, as in the network stack. -The following table shows all available protocol classes in :mod:`pcapkit`: +The protocol classes :mod:`pcapkit` ships: +------------------------------------------------------------------+----------------+-----------------------+-------------------------------------------------------------+ | Protocol Type | Protocol Class | @@ -132,16 +130,12 @@ attributes of the protocol. attribute should be defined as a class attribute with no value besides the type annotations of the corresponding field. -Once the protocol class is implemented, you need to register it to the protocol -registry, which is managed by the APIs provided by -:mod:`pcapkit.foundation.registry.protocols` module. Depending on the protocol -type, you need to register the protocol class to the corresponding registry, e.g., -for a link layer protocol, you need to register it to the link layer protocol -registry thru :func:`~pcapkit.foundation.registry.protocols.register_linktype` -function. +Once the class exists, register it through +:mod:`pcapkit.foundation.registry.protocols`, picking the registry that matches +the protocol's layer -- a link layer protocol goes to +:func:`~pcapkit.foundation.registry.protocols.register_linktype`, and so on. -The following table shows the type of protocols and the corresponding registry -functions: +The registry function per protocol type: +-------------------+-------------------------------------------------------------------+----------------------------------------------------------------+--------------------+ | Protocol Type | Registry Function | Notes | @@ -275,13 +269,13 @@ The following code snippet shows how to create a new protocol class: The check lives in :meth:`ProtocolBase.__init__ `, where every producer's keywords converge, so it covers ``SomeProtocol(...)`` and the ``pack`` it - leads to -- but **not a direct ``SomeProtocol.make(...)`` call**, which still + leads to -- but **not** a direct ``SomeProtocol.make(...)`` call, which still discards an undeclared keyword in silence. ``object.__new__(cls).make(...)`` is the idiom that reaches it, used by this package's own tests and by :meth:`HTTP.make ` to reach its versioned implementation. Covering that would mean interposing on every - ``make`` in the tree -- a larger change that was deliberately not made. - Construct through the constructor to get the check. + ``make`` in the tree -- a larger change this package deliberately does not + make. Construct through the constructor to get the check. .. note:: @@ -315,27 +309,21 @@ The following code snippet shows how to create a new protocol class: Extending Existing Protocol --------------------------- -In many cases, existing protocols have customisable attributes, e.g., the -option headers in the IPv4 protocol. To extend an existing protocol with -additional methods for handling option headers, and etc., you need to first -redirect to the corresponding protocol class, and check if the protocol class -supports the expected extension. If so, you can add the extension methods to -the protocol class, as it may be discussed in the protocol documentation. +Several protocols carry customisable attributes -- the option headers of IPv4, +for one. To add handling for one of them, check the protocol class for the +extension point you need, then register your methods against it. .. warning:: - It is not recommended to directly inherit from the protocol class, as it - may cause unexpected errors. Instead, you should use the provided helper - functions to extend the protocol class. + Do not inherit from the protocol class directly -- it may cause unexpected + errors. Use the helper registry functions in + :mod:`pcapkit.foundation.registry.protocols` instead. - See :mod:`pcapkit.foundation.registry.protocols` module for available - helper registry functions and the corresponding protocol classes. - -The following table shows the protocol classes with their corresponding -available extensible items and the helper registry functions: +The extension points of each protocol class, with the registry function that +writes them: +-------------------+------------------------------------------------------------+-----------------------------------------------------------------------------------------------+----------------------------------------------------------------------------+ -| Protocol Type | Protocol Class | Extensable Registry | Registry Function | +| Protocol Type | Protocol Class | Extensible Registry | Registry Function | +===================+============================================================+===============================================================================================+============================================================================+ | | :class:`~pcapkit.protocols.internet.ipv4.IPv4` | :attr:`IPv4.__option__ ` | :func:`~pcapkit.foundation.registry.protocols.register_ipv4_option` | + +------------------------------------------------------------+-----------------------------------------------------------------------------------------------+----------------------------------------------------------------------------+ @@ -376,30 +364,27 @@ available extensible items and the helper registry functions: .. important:: - All extensable items are to be implemented as class methods, whose first argument - is the protocol class itself, and the rest arguments are the same as the other - built-in methods' signatures. + Every extension point takes a class method: its first argument is the protocol + class, and the rest match the built-in methods it sits beside. -------------------------------------- And Speaking of Library Foundations... -------------------------------------- -The :mod:`pcapkit` library is built on top of the :mod:`pcapkit.foundation` -module, which provides the basic functionalities for the library. The -:mod:`pcapkit.foundation` module is designed to be extensible, and it is -easy to add new features to the library. +:mod:`pcapkit.foundation` carries the library's own machinery -- extraction, +reassembly, flow tracing and the registries behind them -- and each part of it +is extensible in the same way the protocols are. Extractor Engines ----------------- -The :mod:`pcapkit.foundation.engines` module provides several built-in engines -for extracting network packets from the intput PCAP and/or PCAP-NG, etc., files. -All engines are implemented as :class:`~pcapkit.foundation.engines.engine.Engine` -subclasses, which are responsible for parsing the input files and extracting -the network packets for further processing. +:mod:`pcapkit.foundation.engines` holds the built-in engines that read packets +out of an input PCAP, PCAP-NG or other capture file. The shipped engines derive +from :class:`~pcapkit.foundation.engines.engine.EngineBase`; +:class:`~pcapkit.foundation.engines.engine.Engine` is the subclass to build your +own on, since it is the one carrying the ``engine=`` registration hook. -The following table shows the available engines and the corresponding supported -file formats: +The available engines and the file formats they read: +---------------------+-----------------------------------------------------------+----------------------------------------+ | Engine Type | Engine Class | Supported File Formats | @@ -489,7 +474,7 @@ The following code snippet shows how to create a new engine class: from scapy import sendrecv # import API entry point self._expkg = sendrecv # store API entry point - self._extmp = None # intermediate storage for the interator + self._extmp = None # intermediate storage for the iterator # generated by the API entry point super().__init__(extractor) # initialise base class @@ -514,10 +499,10 @@ The following code snippet shows how to create a new engine class: # third-party libraries, e.g., the `packet2chain` function is # to convert a given Packet instance of the Scapy library to a # human-readable string, which is used for verbose output. - from pcapkit.toolkit.scapy import pack2chain # import utility function + from pcapkit.toolkit.scapy import packet2chain # import utility function ext._vfunc = lambda extractor, packet: print( - f'Frame {extractor._frnum:>3d}: {pack2chain(packet)}' + f'Frame {extractor._frnum:>3d}: {packet2chain(packet)}' ) # print verbose message as `Frame XXX: Ethernet:IP:TCP:...` # NOTE: Here we use the API entry point, i.e., scapy.sendrecv.sniff, @@ -606,20 +591,17 @@ The following code snippet shows how to create a new engine class: Output Dumpers -------------- -The :mod:`pcapkit.dumpkit` module wraps the :mod:`DictDumper ` -library, which provides the basic functionalities for dumping the extracted -network packets to the output file. The :mod:`pcapkit.dumpkit` module is -designed to be extensible, and it is easy to add new output formats to the -library, based on the extensibility of the :mod:`DictDumper ` -library. +:mod:`pcapkit.dumpkit` wraps :mod:`DictDumper `, which writes the +extracted packets to the output file. New output formats are added by +subclassing a dumper, on the same extensibility :mod:`DictDumper ` +itself offers. .. seealso:: - Please refer to the documentation of :mod:`DictDumper ` library - for more information about the output dumpers. + :mod:`DictDumper `'s own documentation covers the dumpers in + detail. -The following table shows the available output dumpers and the corresponding -formats: +The available output dumpers and their formats: +----------------------------------+------------------------------------------------+------------------+ | Dumper Format | Dumper Class | Output Extension | @@ -691,7 +673,7 @@ The following code snippet shows how to create a new dumper class: def __call__(self, value: 'Frame', name: 'Optional[str]' = None) -> 'MyDumper': with open(self._file, 'ab') as file: self._append_value(value, file, name or f'Frame {self._fnum}') # append value to the file - return self + return self # NOTE: The following method will be called at the initialisation of the # dumper class, which is expected to write the top-level header of the @@ -719,18 +701,16 @@ The following code snippet shows how to create a new dumper class: Reassembly and Flow Tracing --------------------------- -The :mod:`pcapkit.foundation.reassembly` module provides several built-in -reassembly classes for reassembling the fragmented network packets; and the -:mod:`pcapkit.foundation.traceflow` module provides several built-in flow -tracing classes for tracing the network packets as they will compose as a -flow and/or stream. All reassembly and flow tracing classes are implemented -as :class:`~pcapkit.foundation.reassembly.reassembly.Reassembly` and -:class:`~pcapkit.foundation.traceflow.traceflow.TraceFlow` subclasses, -respectively, which are responsible for reassembling and tracing the network -packets for further processing. +:mod:`pcapkit.foundation.reassembly` reassembles fragmented packets; +:mod:`pcapkit.foundation.traceflow` follows packets that compose a flow or +stream. The shipped classes derive from +:class:`~pcapkit.foundation.reassembly.reassembly.ReassemblyBase` and +:class:`~pcapkit.foundation.traceflow.traceflow.TraceFlowBase`; +:class:`~pcapkit.foundation.reassembly.reassembly.Reassembly` and +:class:`~pcapkit.foundation.traceflow.traceflow.TraceFlow` are the ones to build +your own on, since they carry the ``protocol=`` registration hook. -The following table shows the available reassembly and flow tracing classes -per the supported protocols: +The reassembly and flow tracing classes, per protocol: +------------------------------------------------+--------------------------------------------------+-----------------------------------------------+ | Protocol | Reassembly Class | Flow Tracing Class | @@ -751,13 +731,9 @@ per the supported protocols: New Implementation ~~~~~~~~~~~~~~~~~~ -To add a new reassembly or flow tracing class, you need to create a new class -inherited from :class:`~pcapkit.foundation.reassembly.reassembly.Reassembly` -or :class:`~pcapkit.foundation.traceflow.traceflow.TraceFlow` class, which -is responsible for reassembling or tracing the network packets for further -processing. - -The following code snippet shows how to create a new reassembly class: +A new reassembly or flow tracing class subclasses +:class:`~pcapkit.foundation.reassembly.reassembly.Reassembly` or +:class:`~pcapkit.foundation.traceflow.traceflow.TraceFlow`: .. note:: @@ -785,8 +761,9 @@ The following code snippet shows how to create a new reassembly class: # the reassembled packet and the reassembly table, respectively. BufferID: 'TypeAlias' = ... # a tuple-like object to identify the reassembly - # table entry's ID, e.g., (src, dst, id, proto) - # for IPv4, (src, dst, spi) for IPv6, etc. + # table entry's ID, e.g. (src, dst, id, proto) + # for both IPv4 and IPv6, and + # (src, srcport, dst, dstport) for TCP @info_final @@ -878,30 +855,23 @@ The following code snippet shows how to create a new reassembly class: Callback Functions ~~~~~~~~~~~~~~~~~~ -It is possible to register callback functions to the reassembly and flow -tracing classes, which will be called at the end of the reassembly and -flow tracing process, respectively. This feature is designed to create -the ability to process the reassembled datagrams and/or flows, e.g., to -check if the datagram and/or flows are to be discarded, etc. +Callback functions can be registered on the reassembly and flow tracing +classes, and run at the end of the respective process -- to inspect the +reassembled datagrams or flows, or to discard some of them. .. seealso:: - For more information, you may refer to the documentation of - :mod:`pcapkit.foundation.registry.foundation` for the callback - registry functions: + :mod:`pcapkit.foundation.registry.foundation` holds the callback registry + functions: - :func:`~pcapkit.foundation.registry.foundation.register_reassembly_ipv4_callback` - :func:`~pcapkit.foundation.registry.foundation.register_reassembly_ipv6_callback` - :func:`~pcapkit.foundation.registry.foundation.register_reassembly_tcp_callback` - :func:`~pcapkit.foundation.registry.foundation.register_traceflow_tcp_callback` -All callback functions are expected to be a callback function, which -accepts a single argument, i.e., the list of reassembled datagrams -and/or flows, and returns :obj:`None`. Any return value will be ignored. +A callback takes one argument -- the list of reassembled datagrams or flows -- +and returns :obj:`None`; any return value is ignored. -It is possible to modify the reassembled datagrams and/or flows in the -callback functions, e.e., to discard certain reassembled datagrams and/or -flows, etc. However, it is not recommended to modify the reassembled -datagrams and/or flows directly. Should that is the intended behaviour, -you should create a new reassembly and/or flow tracing class, and modify -the corresponding reassembly and/or flow tracing algorithm. +That list can be modified in place, but prefer not to: if the reassembly or +tracing result itself is wrong for your purpose, subclass the reassembly or +flow tracing class and change its algorithm instead. diff --git a/docs/source/index.rst b/docs/source/index.rst index d0d0fc51f..10ed8ebe3 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,15 +1,12 @@ -.. PyPCAPKit documentation master file, created by - sphinx-quickstart on Sat Mar 28 21:29:54 2020. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. +.. The documentation's root document. It must carry the root ``toctree`` + directives, which is what populates the global sidebar on every page. ========================================================= PyPCAPKit - Comprehensive Network Packet Analysis Library ========================================================= -The PyPCAPKit project is an open source Python program focus on network packet -parsing and analysis, which works as a comprehensive `PCAP`_ file extraction, -construction and analysis library. +PyPCAPKit is an open source Python library for `PCAP`_ file extraction, +construction and analysis. .. important:: @@ -74,7 +71,7 @@ interface. Module Structure ---------------- -In :mod:`pcapkit`, all files can be described as following eight parts. +:mod:`pcapkit` is made of nine subpackages. - Interface (:mod:`pcapkit.interface`) @@ -115,6 +112,11 @@ In :mod:`pcapkit`, all files can be described as following eight parts. Constant enumerations used in :mod:`pcapkit` for protocol family extraction and representation. +- Vendor (:mod:`pcapkit.vendor`) + + Crawlers that regenerate the :mod:`pcapkit.const` enumerations + from the upstream registries. + ----------------- Engine Comparison ----------------- @@ -225,30 +227,29 @@ compared with a measured row. .. [3] `PyPCAP`_ 1.3.0 could not be built here: its build does not search Homebrew's libpcap library prefix. It is also unsupported on Python 3.12+. -.. [4] The prerelease `pcap-ct`_ / `libpcap`_ wheels currently load Linux - ``libc.so.6`` on this macOS host, so their engine could not be preflighted. +.. [4] The prerelease `pcap-ct`_ / `libpcap`_ wheels load Linux ``libc.so.6`` on + this macOS host, so their engine could not be preflighted. Installation ============ .. note:: - :mod:`pcapkit` declares support for **Python 3.6 and later**, and CI verifies - **3.10 through 3.14**, plus 3.15 as an allowed-to-fail leg. - The sources themselves use 3.8 syntax; the ``bpc-walrus``/``bpc-poseur`` backport tools in :file:`setup.py` convert it at install time, which is what - makes the lower bound possible. Measured: 3.9 and 3.8 import and extract - straight from source with no conversion needed, and 3.7 needs the conversion. + makes the **3.6** lower bound possible. Measured: 3.9 and 3.8 import and + extract straight from source with no conversion needed, and 3.7 needs the + conversion. .. warning:: - **The conversion is currently blocked by an upstream bug**, so below 3.8 the - declaration is intent rather than something that works today. ``bpc-poseur`` - 0.4.3.post1 crashes on positional-only parameters declared on a *method* - rather than a plain function, and exits 0 so the build does not notice; the 12 - such parameters in :mod:`pcapkit.corekit.io` then survive into the installed - package and ``import pcapkit`` fails with :exc:`SyntaxError`. + **The conversion is blocked by an upstream bug**, so below 3.8 the declaration + is intent rather than something that works. ``bpc-poseur`` 0.4.3.post1 crashes + on positional-only parameters declared on a *method* rather than a plain + function, and exits 0 so the build does not notice; such methods -- including + the 13 on :class:`~pcapkit.corekit.io.SeekableReader`, plus a dozen more across + the protocols, foundation, dumpkit and corekit modules -- then survive into the + installed package and ``import pcapkit`` fails with :exc:`SyntaxError`. It is a one-line fix upstream -- ``poseur.py:744`` passes ``cls_ctx=name.name`` where ``name`` is already a parso ``Name`` and wants ``.value`` -- and with it @@ -339,7 +340,7 @@ plug-in functions, you may want to install the optional ones: .. warning:: - **Install only one of ``pypcap`` and ``pcap-ct``.** Both own the top-level + **Install only one of** ``pypcap`` **and** ``pcap-ct``. Both own the top-level :mod:`pcap` module, and ``pip`` will install both without complaint. With both present the ``pcap-ct`` package wins the import and ``pypcap``'s extension module is shadowed and unreachable, so ``engine='pypcap'`` stops working -- @@ -351,12 +352,10 @@ plug-in functions, you may want to install the optional ones: Engine prerequisites -------------------- -Four of the engines need something beyond a ``pip install``. Each constraint is -also enforced in code, through the engine's -:meth:`~pcapkit.foundation.engines.engine.EngineBase.unsupported_reason`, so -hitting one produces a warning naming the cause and a fall back to -:mod:`pcapkit`'s own parser rather than an error from inside the third-party -package. +Four of the engines need something beyond a ``pip install``. Each constraint +below is also enforced in code, by the +:meth:`~pcapkit.foundation.engines.engine.EngineBase.unsupported_reason` +preflight described under `Engine Comparison`_. :class:`~pcapkit.foundation.engines.pyshark.PyShark` Two requirements, and neither is visible to an import: the package imports @@ -372,9 +371,9 @@ package. **3.14** :func:`asyncio.get_event_loop` raises :exc:`RuntimeError` when there is no current event loop instead of quietly creating one. Measured: a loop is returned silently on 3.10 and 3.11, returned with a - :exc:`DeprecationWarning` on 3.12, and refused on 3.14. (3.13 was not - available to test and is expected to work, being on the - deprecated-but-functional side of that change.) + :exc:`DeprecationWarning` on 3.12, and refused on 3.14. 3.13 sits on the + deprecated-but-functional side of that change, and ``pyshark`` extracted a + capture there. :class:`~pcapkit.foundation.engines.pypcap.PyPCAP` `PyPCAP`_ ships **no wheels** -- only an sdist -- so :program:`pip` compiles @@ -436,13 +435,10 @@ package. .. note:: - :mod:`pcapkit` itself, and its ``default``, ``dpkt`` and ``scapy`` engines, - work fine on current Python versions -- ``dpkt`` 1.9.8 and ``scapy`` 2.7.0 were - both measured reading a capture on Python 3.14. Only the four engines above - carry extra constraints, and asking for an engine that cannot run in the - current environment emits an - :class:`~pcapkit.utilities.warnings.EngineWarning` naming the reason and falls - back to :mod:`pcapkit`'s own parser rather than failing outright. + Only the four engines above carry extra constraints. :mod:`pcapkit` itself and + its ``default``, ``dpkt`` and ``scapy`` engines need nothing further -- + ``dpkt`` 1.9.8 and ``scapy`` 2.7.0 were both measured reading a capture on + Python 3.14. For CLI usage, you will need to install the optional packages: @@ -468,4 +464,3 @@ Indices and tables .. _pcap-ct: https://pypi.org/project/pcap-ct/ .. _PyPCAPFile: https://github.com/kisom/pypcapfile .. _libpcap: https://www.tcpdump.org -.. _DictDumper: https://github.com/JarryShaw/DictDumper