Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/source/pcapkit/const/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,8 @@ Protocol Numbers

reg

Miscellanous
------------
Miscellaneous
-------------

.. toctree::
:maxdepth: 2
Expand Down
81 changes: 36 additions & 45 deletions docs/source/pcapkit/foundation/engines/3rdparty.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,19 +14,18 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.

.. note::

Constructing this engine imports :mod:`scapy.all`, which is what populates
`Scapy`_'s layer registries -- ``conf.l2types`` and the ``bind_layers``
payload table, both of which exist only as import side effects of the layer
modules. Importing a narrower submodule leaves them empty, and
Constructing this engine imports :mod:`scapy.all`, which populates `Scapy`_'s
layer registries -- ``conf.l2types`` and the ``bind_layers`` payload table,
both of which exist only as import side effects of the layer modules.
Importing a narrower submodule leaves them empty, and
:class:`~scapy.utils.PcapReader` then returns every frame as one opaque
:class:`~scapy.packet.Raw` layer without raising, so the engine dissected
nothing at all and said so only on :data:`sys.stderr`. See
:meth:`Scapy.__init__` for why naming the layer modules individually is not a
cheaper route to the same place.

One side effect is worth knowing about in advance: :mod:`scapy.all` loads
:mod:`scapy.layers.dcerpc`, which reaches `Scapy`_'s TLS layer and there
triggers a ``CryptographyDeprecationWarning`` from :mod:`cryptography` about
:class:`~scapy.packet.Raw` layer without raising, so the engine would dissect
nothing and say so only on :data:`sys.stderr`. See :meth:`Scapy.__init__` for
why naming the layer modules individually is not a cheaper route.

One side effect: :mod:`scapy.all` loads :mod:`scapy.layers.dcerpc`, which
reaches `Scapy`_'s TLS layer and there triggers a
``CryptographyDeprecationWarning`` from :mod:`cryptography` about
finite-field Diffie-Hellman. It concerns a key-exchange code path
:mod:`pcapkit` never executes, but it subclasses :exc:`UserWarning` rather
than :exc:`DeprecationWarning`, so Python's default filters show it.
Expand Down Expand Up @@ -97,12 +96,12 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
both up front.

**Python 3.13 or older.** ``pyshark`` 0.6 builds its event loop with
``asyncio.get_event_loop_policy().get_event_loop()``. Measured in a fresh
interpreter with no running loop: 3.10 and 3.11 return a loop silently, 3.12
returns one with a :exc:`DeprecationWarning`, and **3.14 raises**
``asyncio.get_event_loop_policy().get_event_loop()``. With no running loop,
3.10 and 3.11 return a loop silently, 3.12 returns one with a
:exc:`DeprecationWarning`, and **3.14 raises**
``RuntimeError: There is no current event loop in thread 'MainThread'``. Python
3.13 was not available to measure and is expected to work, being on the
deprecated-but-functional side of that change.
3.13 is unchecked and expected to work, being on the deprecated-but-functional
side of that change.

**Wireshark's** :program:`tshark` **binary.** ``pyshark`` shells out to it and
parses nothing itself. It need not be on :envvar:`PATH`: ``pyshark`` consults
Expand Down Expand Up @@ -153,8 +152,8 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
**On Python 3.12 and newer it cannot be installed at all**, which is why the
extra carries a ``python_version < '3.12'`` marker. `PyPCAP`_ 1.3.0 ships a
``pcap.c`` pre-generated by Cython 0.29.32 and never runs Cython at build
time, and that generated C does not compile against the 3.12+ C API. Measured
on four interpreters with libpcap present and found:
time, and that generated C does not compile against the 3.12+ C API. With
libpcap present:

=============== ===============================================================
Python ``pip install pypcap``
Expand All @@ -166,9 +165,9 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
``_PyLong_AsByteArray`` arity
=============== ===============================================================

Upstream is unmaintained, and its Python 3.12 issue (`pynetwork/pypcap#116
<https://github.com/pynetwork/pypcap/issues/116>`_) remains open and
uncommented, so the cap is not expected to lift. Use
Upstream is unmaintained (see `pynetwork/pypcap#116
<https://github.com/pynetwork/pypcap/issues/116>`_), so the cap is not
expected to lift. Use
:class:`~pcapkit.foundation.engines.pcap_ct.PCAP_CT` on 3.12 and newer.

.. important::
Expand Down Expand Up @@ -247,11 +246,8 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.

pip install pypcapkit[PCAP_CT]

Verified end to end on Python **3.10.20** and **3.14.7**: both read
:file:`examples/captures/in.pcap` through ``engine='pcap_ct'`` and return the
same six frames with identical timestamps. So the two engines together cover
every interpreter the project supports, and ``PCAP_CT`` covers all of it on
its own:
The two engines together cover every interpreter the project supports, and
``PCAP_CT`` covers all of it on its own:

========================================================================= ==================
Engine Python
Expand All @@ -269,11 +265,10 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
published, which sends the loader to :func:`ctypes.util.find_library`, so what
gets mapped is the host's ``libpcap.so.1``.

Measured, not assumed: the same ``libpcap`` 1.11.0b29 wheel loaded
:file:`/usr/lib64/libpcap.so.1.5.3` under one interpreter on this host and
linuxbrew's 1.11.0 under another, purely because their loader search paths
differ. Setting ``LIBPCAP = tcpdump`` in :file:`libpcap.cfg` selects the
vendored copy instead, which reports libpcap 1.10.6.
Which system library gets mapped therefore depends on the interpreter's
loader search path, so the same wheel can load different ``libpcap`` versions
under different interpreters. Setting ``LIBPCAP = tcpdump`` in
:file:`libpcap.cfg` selects the vendored copy instead.

With no system :manpage:`libpcap(3)` at all, ``import pcap`` raises
:exc:`OSError` rather than :exc:`ImportError`, which would escape
Expand All @@ -286,18 +281,17 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.

.. warning::

**Both distributions are pre-releases.** ``pcap-ct`` 1.3.0b3 and ``libpcap``
1.11.0b29 are the newest published versions, and neither project has ever
published a stable release -- which is also why ``pip install`` resolves them
without ``--pre``. ``pcap-ct`` further documents itself as tracking the
`PyPCAP`_ **1.2.3** interface rather than 1.3.0.
**Both distributions are pre-releases only** (``pcap-ct`` 1.3.0b3, ``libpcap``
1.11.0b29), which is also why ``pip install`` resolves them without
``--pre``. ``pcap-ct`` documents itself as tracking the `PyPCAP`_ **1.2.3**
interface rather than 1.3.0.

Every attribute this engine touches -- the ``pcap.pcap(name=..., promisc=...)``
constructor, :meth:`~pcap.pcap.datalink`, ``snaplen``, iteration yielding
``(timestamp, bytes)``, and :meth:`~pcap.pcap.close` -- is present on both and
was measured to behave identically, byte for byte and timestamp for timestamp,
on ``pcap-ct`` 1.3.0b3 against ``pypcap`` 1.3.0. That is a statement about the
versions tested, not a guarantee from upstream. ``PCAP_CT`` is therefore not
behaves identically, byte for byte and timestamp for timestamp, on
``pcap-ct`` 1.3.0b3 against ``pypcap`` 1.3.0 -- a statement about those
versions, not a guarantee from upstream. ``PCAP_CT`` is therefore not
included in the ``all`` extra: a beta should be asked for by name.

.. important::
Expand All @@ -316,9 +310,7 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
The timestamp is seconds since the epoch as a :class:`float`. `pcap-ct`_ opens
savefiles asking for nanosecond precision via
:c:func:`pcap_open_offline_with_tstamp_precision`, so a microsecond-resolution
savefile is scaled up by :manpage:`libpcap(3)` rather than truncated; for
:file:`examples/captures/in.pcap` the result matches each record header's
``ts_sec + ts_usec * 1e-6`` exactly.
savefile is scaled up by :manpage:`libpcap(3)` rather than truncated.

.. note::

Expand All @@ -327,8 +319,7 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
reason is that accepting it would be *unpredictable* rather than merely
limited. :manpage:`libpcap(3)` can read a PCAP-NG savefile, but how well
depends on the version the host provides -- which, per the warning above, the
`libpcap`_ wheel does not pin. Both measured on
:file:`examples/captures/dhcp.pcapng`:
`libpcap`_ wheel does not pin. Reading :file:`examples/captures/dhcp.pcapng`:

================ ===========================================================
System libpcap Result
Expand Down
31 changes: 17 additions & 14 deletions docs/source/pcapkit/foundation/engines/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -99,22 +99,20 @@ offers, and the ones that cannot say so rather than quietly doing less:
Availability
------------

A third-party engine is only usable where its backing package is, and three of
them are harder to obtain than a plain :program:`pip install` suggests. Asking
for an engine whose package is missing is not fatal -- :mod:`pcapkit` emits an
A third-party engine is usable only where its backing package is, and three are
harder to obtain than a plain :program:`pip install` suggests. A missing package
is not fatal -- :mod:`pcapkit` emits an
:class:`~pcapkit.utilities.warnings.EngineWarning` and falls back to its own
parser -- but the extraction then has nothing to do with the engine requested,
so it is worth knowing in advance.
parser -- but the extraction then does not use the engine requested.

+-----------------------------------------------------------------+---------------------------------------------------------------+
| Engine | What it needs beyond ``pip install`` |
+=================================================================+===============================================================+
| :class:`~pcapkit.foundation.engines.pcap.PCAP`, | nothing -- built in |
| :class:`~pcapkit.foundation.engines.pcapng.PCAPNG` | |
+-----------------------------------------------------------------+---------------------------------------------------------------+
| :class:`~pcapkit.foundation.engines.dpkt.DPKT`, | nothing -- both ship pure-Python wheels, and both were |
| :class:`~pcapkit.foundation.engines.scapy.Scapy` | measured reading a capture on Python 3.14, so neither |
| | overrides ``unsupported_reason()`` |
| :class:`~pcapkit.foundation.engines.dpkt.DPKT`, | nothing -- both are pure Python, and neither |
| :class:`~pcapkit.foundation.engines.scapy.Scapy` | overrides ``unsupported_reason()`` |
+-----------------------------------------------------------------+---------------------------------------------------------------+
| :class:`~pcapkit.foundation.engines.pyshark.PyShark` | Wireshark's :program:`tshark` binary -- on :envvar:`PATH`, or |
| | wherever ``pyshark``'s :file:`config.ini` points -- **and** |
Expand All @@ -136,21 +134,26 @@ so it is worth knowing in advance.
| | ``imp`` module, removed in Python 3.12 |
+-----------------------------------------------------------------+---------------------------------------------------------------+

Every one of these constraints is also enforced in code rather than only
documented: each engine overrides
A constraint that an import cannot reveal is enforced by overriding
:meth:`~pcapkit.foundation.engines.engine.EngineBase.unsupported_reason`, which
:meth:`Extractor.run <pcapkit.foundation.extraction.Extractor.run>` consults
*before* the import test, so asking for an engine that cannot run here produces
one warning naming the actual cause and a clean fall back to the built-in parser.
one warning naming the actual cause and a fall back to the built-in parser.
``PyShark`` checks its Python ceiling and for ``tshark``, ``PyPCAPFile`` its
Python ceiling, ``PCAP_CT`` for a system ``libpcap``, and ``PyPCAP`` and
``PCAP_CT`` which distribution owns ``pcap``. ``PyPCAP`` deliberately has no
Python ceiling check, even though upstream cannot be installed on 3.12 or newer:
what matters is which distribution is present, and a version check would refuse
a working build.

.. seealso::

:doc:`../../../index` covers the installation prerequisites in full,
including how ``pypcap``'s :file:`setup.py` looks for :file:`pcap.h` and why
a Homebrew ``libpcap`` is not always found.

Two engines, one interface: choosing between PyPCAP and PCAP_CT
---------------------------------------------------------------
Choosing Between PyPCAP and PCAP_CT
-----------------------------------

:class:`~pcapkit.foundation.engines.pypcap.PyPCAP` and
:class:`~pcapkit.foundation.engines.pcap_ct.PCAP_CT` read the same
Expand Down Expand Up @@ -189,7 +192,7 @@ installed.** Pick on availability, not on features.
hard error it would otherwise be.

**The two distributions collide, so install exactly one.** Both own the top-level
:mod:`pcap` module, and pip will happily install both -- measured on Python 3.10,
:mod:`pcap` module, and pip will happily install both -- on Python 3.10,
the ``pcap-ct`` package then wins the import and upstream's extension module is
shadowed and unreachable. Each engine therefore detects which distribution it
actually got, via
Expand Down
8 changes: 4 additions & 4 deletions docs/source/pcapkit/foundation/extraction.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,17 @@ File Extractor
:class:`~pcapkit.foundation.extraction.Extractor` only,
which synthesises file I/O and protocol analysis,
coordinates information exchange in all network layers,
extracts parametres from a PCAP file.
extracts parameters from a PCAP file.

.. seealso::

Engine support for |pypcap|_, |pcap-ct|_ and |pypcapfile|_ has since landed,
as :class:`pcapkit.foundation.engines.pypcap.PyPCAP` (``engine='pypcap'``),
|pypcap|_, |pcap-ct|_ and |pypcapfile|_ are supported by
:class:`pcapkit.foundation.engines.pypcap.PyPCAP` (``engine='pypcap'``),
:class:`pcapkit.foundation.engines.pcap_ct.PCAP_CT` (``engine='pcap_ct'``)
and :class:`pcapkit.foundation.engines.pypcapfile.PyPCAPFile`
(``engine='pypcapfile'``). All three support less than the ``default``
engine does; :doc:`engines/index` tabulates the gaps, and :doc:`../../index`
documents the installation prerequisites each of the three carries.
documents their installation prerequisites.

.. autoclass:: pcapkit.foundation.extraction.Extractor
:no-members:
Expand Down
8 changes: 3 additions & 5 deletions docs/source/pcapkit/foundation/reassembly/ip/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,9 @@
IP Datagram Reassembly
======================

The following algorithm implement is based on IP
reassembly procedure introduced in :rfc:`791`, using
``RCVBT`` (fragment receivedbit table). Though another
algorithm is explained in :rfc:`815`, replacing ``RCVBT``,
however, this implement still used the original one.
The algorithm below follows the IP reassembly procedure of :rfc:`791`, using
``RCVBT`` (fragment received bit table). :rfc:`815` explains an alternative
that replaces ``RCVBT``; it is not used here.

.. toctree::
:maxdepth: 2
Expand Down
6 changes: 1 addition & 5 deletions docs/source/pcapkit/foundation/reassembly/ip/ip.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,7 @@ Base Class
:mod:`pcapkit.foundation.reassembly.ip` contains
:class:`~pcapkit.foundation.reassembly.ip.IP`
only, which reconstructs fragmented IP packets back to
origin. The following algorithm implement is based on IP
reassembly procedure introduced in :rfc:`791`, using
``RCVBT`` (fragment receivedbit table). Though another
algorithm is explained in :rfc:`815`, replacing ``RCVBT``,
however, this implement still used the elder one.
origin, using the :rfc:`791` procedure described in :doc:`index`.

.. autoclass:: pcapkit.foundation.reassembly.ip.IP
:no-members:
Expand Down
4 changes: 2 additions & 2 deletions docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst
Original file line number Diff line number Diff line change
Expand Up @@ -97,8 +97,8 @@ Terminology
``packet`` is analysed on the first read, not when the datagram is
submitted. A datagram is submitted for *every* frame -- an unfragmented
one included, since nothing upstream filters it out -- and the analysis
is a second full parse of the payload, so running it eagerly charged
every caller for a result most never read. Reading the attribute, or any
is a second full parse of the payload, so running it eagerly would
charge every caller for a result most never read. Reading the attribute, or any
mapping view of it (``datagram['packet']``, ``to_dict()``, ``items()``,
``repr()``), runs it and keeps the result; see
:class:`~pcapkit.foundation.reassembly.data.data.Deferred`.
Expand Down
4 changes: 2 additions & 2 deletions docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst
Original file line number Diff line number Diff line change
Expand Up @@ -129,8 +129,8 @@ Terminology
``packet`` is analysed on the first read, not when the datagram is
submitted. A datagram is submitted for *every* frame -- an unfragmented
one included, since nothing upstream filters it out -- and the analysis
is a second full parse of the payload, so running it eagerly charged
every caller for a result most never read. Reading the attribute, or any
is a second full parse of the payload, so running it eagerly would
charge every caller for a result most never read. Reading the attribute, or any
mapping view of it (``datagram['packet']``, ``to_dict()``, ``items()``,
``repr()``), runs it and keeps the result; see
:class:`~pcapkit.foundation.reassembly.data.data.Deferred`.
Expand Down
4 changes: 2 additions & 2 deletions docs/source/pcapkit/foundation/reassembly/tcp.rst
Original file line number Diff line number Diff line change
Expand Up @@ -242,8 +242,8 @@ Terminology
conflicting overlap is first-write-wins (:rfc:`9293#section-3.10`), so
it never leaves a hole, and a contested range that was later filled in
around does not stop the datagram from completing. ``conflict`` is
what lets a caller tell a clean stream from a contested one, now that
``completed`` alone no longer can.
what lets a caller tell a clean stream from a contested one, which
``completed`` alone cannot.

reasm.tcp.buffer
Data structure for internal buffering when performing reassembly algorithms
Expand Down
4 changes: 1 addition & 3 deletions docs/source/pcapkit/foundation/traceflow/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,7 @@ Flow Tracing

.. note::

This was implemented at the demand of my mate
`@gousaiyang <https://github.com/gousaiyang>`__. It is
a approximate functionality of *Follow TCP Streams* in
This approximates the *Follow TCP Stream* feature of
`Wireshark <https://www.wireshark.org/>`__.

:mod:`pcapkit.foundation.traceflow` implements flow tracing functions
Expand Down
2 changes: 1 addition & 1 deletion docs/source/pcapkit/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ Command Line Tool

This module requires ``emoji`` package to be installed.

:mod:`pcapkit.__main__` provides the CLI, merged in from the now-deprecated
:mod:`pcapkit.__main__` provides the CLI, merged in from the deprecated
|jspcapy|_ project.

.. |jspcapy| replace:: ``jspcapy``
Expand Down
Loading
Loading