Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe
- **Added** -- `examples/generators/endian.py`, and the byte-order tests that read what it writes. There was no big-endian `.pcap` in the repository at all, which is why #605 survived its own code review: the one-character fix leaves the corrected path exactly as untested as the broken one. The generator writes three captures -- `big_endian.pcap` (magic `a1 b2 c3 d4`), `big_endian_nanosecond.pcap` (`a1 b2 3c 4d`, the first fixture to take that branch of the magic-number table) and `little_endian.pcap` (`d4 c3 b2 a1`) -- carrying the *same three records* in each container, so the tests can assert that the byte order makes no difference to what is read out rather than only that the big-endian file matches numbers written down in a test. Frame 3 is captured short, 1200 octets on the wire cut to a 96-octet `snaplen`, so `incl_len` and `orig_len` differ and cannot both be satisfied by one byte-swapped value. `test_frame_endian_runtime.py` drives all three through `extract()` and walks each file's record chain with `struct` to derive its own expectations; a unit-tier case in `test_header_frame_unit.py` builds a two-record big-endian capture in memory instead, so the regression is also caught by the fixture-free selection CI runs on every push. All four fail on the unfixed tree -- the three fixture-backed ones by that `ValueError`, the in-memory one by `AssertionError: 3106905 != 1500000000` -- while the little-endian twin passes on both trees, which is what shows the records themselves are not the variable (#605).
- **Changed** -- `tests/protocols/transport/test_tcp_udp_unit.py` now reaches the MP_JOIN dispatchers through `TCP()` itself, instead of assigning a Python `set` to `_flags` on a bare `TCP.__new__(TCP)`. A `set` answers the membership tests `_make_mptcp_join` and `_read_mptcp_join` use, so every flag branch ran and both TCP modules read 100% statement and branch coverage -- while the attribute had neither the `aenum.IntFlag` type production assigns nor the ordering that governs when it exists at all, which is how #587 stayed invisible behind that number and how the `cast('Enum_Flags', 0)` no-op behind it went unnoticed too. Measured on the rewrite, against the 17 tests of that file: revert #587's hoist and two of them fail with `AttributeError: 'TCP' object has no attribute '_flags'` where all 17 passed before; restore the `cast` and two fail with `TypeError: argument of type 'int' is not a container or iterable`, again where all 17 passed. The library is unchanged and the file's tests still pass, so the coverage numbers do not move -- the point is what the same numbers are now worth (#603).
- **Fixed** -- documentation. `mptcp_dss_ack_selector`'s note said a corrected field-width lambda "would not have worked" and that fixing it belonged to `pcapkit.corekit.fields.numbers`, which is exactly where #598 then fixed it; the same paragraph sat in `test_tcp_mptcp_length_arithmetic_unit.py`'s module docstring, whose other stale claim was that MP_JOIN "cannot be built through the public `TCP()` constructor at all", true only until #587. A callable-length `NumberField` packs and unpacks both DSS widths now, and wire *absence* was never the obstacle either: `MPTCPDSS.ssn`, `dl_len` and `checksum` have always been `ConditionalField` on the sibling `M` flag, so the class already relied on that wrapper to keep a field off the wire. The `SwitchField` form is kept for the narrower reason the note now gives -- `ConditionalField`'s `length` forwards to the wrapped field without consulting the condition, so it is safe here only because `Schema.pack` and `Schema.unpack` special-case that wrapper by name, whereas a `SwitchField` always resolves to a concrete field. Replacing it would be a behaviour change and is not made (#603).
- **Changed** -- the README is a landing page now, and Markdown rather than reStructuredText. `README.rst` (424 lines) became `README.md` (103), keeping what a reader arriving from PyPI or a search result actually needs -- what the library is, why it exists rather than Scapy or DPKT, how to install it, a worked example, and where the documentation lives -- and dropping the technical detail the documentation already carried. **Module Structure**, **Engine Comparison**, **Engine support by Python version**, **Test Environment**, **Test Results** and **Installation Notes** were each already duplicated in `docs/source/index.rst`, in a fuller form, so they are linked rather than restated. Two blocks existed nowhere else and moved rather than going: **Testing** is now `docs/source/testing.rst`, registered in the index toctree, and the `pipenv` and `make setup` local development block joined the Installation section of `docs/source/index.rst`. Requested by the project owner, and a deliberate exception to the convention that documentation here is reStructuredText -- for the README only, since it is the one documentation file whose renderers are GitHub and PyPI rather than Sphinx. Accordingly `setup.py` reads `README.md` and declares its content type as `text/markdown`, and the `include README.md` line in `MANIFEST.in` is now the only thing that puts the README in a source distribution, because `global-include *.rst` no longer matches it -- which matters, since `setup.py` reads the file unguarded and an sdist without it cannot be installed. Verified with `twine check --strict` against a built sdist and wheel, both of which pass. The rename's own references moved with it, since a change that renames a file owns the references to it: `examples/benchmark/Dockerfile` copies `README.md` -- a literal `COPY` of the old name would have failed the layer outright and taken `make bench`, `make bench-quick` and `run.sh` with it -- and the benchmark harness prose that named the root README as the destination of its generated tables now names `docs/source/index.rst`, which is where those tables went. That covers the `Makefile` comment, `report.py`, `test_harness.py`, `run.sh` and the suite's own README, including the one place whose stated reason had inverted: the emitted markup is kept parseable by plain docutils, which is now a conservative choice rather than a hard requirement, because the page it lands on is rendered by Sphinx. `examples/benchmark/benchmark.py` still says `README.rst` and is left alone, because it means the benchmark suite's own README in the same directory, not the project's.
- **Changed** -- `CODE_OF_CONDUCT.md` moves from Contributor Covenant 1.4 to Contributor Covenant 3.0, at the maintainer's request. The text is the canonical 3.0 Markdown fetched from https://www.contributor-covenant.org/version/3/0/code_of_conduct/code_of_conduct.md rather than a transcription, so the pledge, the encouraged and restricted behaviours and the scope are unaltered. Three things needed deciding rather than copying. 3.0 ships two `[NOTE` placeholders an adopter must fill: the reporting channel, which now names `jarryshaw@icloud.com` -- the same contact 1.4 carried and the one `SECURITY.md` already points at as its email fallback -- plus GitHub's report-abuse form for the case a single-maintainer project cannot otherwise cover, a report about the maintainer; and the enforcement section, whose placeholder is an instruction to the adopter and is removed. 3.0 then assigns enforcement throughout to plural "Community Moderators" (and once, inconsistently, to "Community Managers"), which this repository does not have, so all eight occurrences become the singular maintainer. The four-rung ladder -- Warning, Temporarily Limited Activities, Temporary Suspension, Permanent Ban -- is offered as a suggestion and is **kept**, because each rung maps onto a lever one person actually holds on GitHub: a private message, a locked thread, an interaction limit or block, a permanent block. Finally, 3.0 is licensed CC BY-SA 4.0 where 1.4's attribution paragraph carried no licence notice at all, so the attribution now names version 3.0, links the permanent `version/3/0/` URL, carries the CC BY-SA 4.0 notice and link, indicates that changes were made as BY requires, and says explicitly that the share-alike term covers this document only -- the code remains BSD-3-Clause and `LICENSE` is untouched. Rendering was checked against GitHub's own Markdown API rather than assumed: the ladder comes back as four list items each nesting three, which is what #613 had to repair in the 1.4 file when a stray list marker collapsed the whole document into one nested item (#624).

Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498).
Expand Down
8 changes: 6 additions & 2 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,11 @@ prune examples/captures
prune temp
prune test

include README.rst
# Load-bearing, not belt-and-braces. ``global-include *.rst`` above matches only
# ``*.rst``, so since the README became Markdown this line is the only thing that
# puts it in an sdist -- and ``setup.py`` reads it unguarded for the
# ``long_description``, so an sdist without it cannot be installed at all.
include README.md
include LICENSE

# The changelog needs two lines of its own, and neither is redundant.
Expand All @@ -24,7 +28,7 @@ include CHANGELOG.md
# ``CHANGELOG.md`` carries only the version being released, so on its own the
# distribution would describe one release and no history. ``prune docs`` above
# drops the rest, which is where the history lives, so re-add just that one
# directory afterwards -- the same shape as ``include README.rst``, and for the
# directory afterwards -- the same shape as ``include README.md``, and for the
# same reason. Each of those files is a self-contained reStructuredText entry
# that renders standalone; the Sphinx index beside them is deliberately *not*
# shipped, because it is a ``toctree`` that only Sphinx can read.
Expand Down
10 changes: 5 additions & 5 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -87,11 +87,11 @@ coverage: samples
pipenv run coverage run -m pytest -q
pipenv run coverage report

# The engine speed table in README.rst -- every supported Python version, one
# image each, measured in containers so the host does not affect the result. Needs
# docker, and nothing else: deliberately not run through pipenv, since the whole
# point is that the measuring environments are the pinned ones inside the images
# rather than whatever is installed here.
# The engine speed table in docs/source/index.rst -- every supported Python
# version, one image each, measured in containers so the host does not affect the
# result. Needs docker, and nothing else: deliberately not run through pipenv,
# since the whole point is that the measuring environments are the pinned ones
# inside the images rather than whatever is installed here.
#
# Expect around two hours at the defaults: five interpreters, seven environments,
# and almost all of the measuring time is pyshark, which spawns a tshark process
Expand Down
102 changes: 102 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# PyPCAPKit -- Comprehensive Network Packet Analysis Library

> For any technical and/or maintenance information, please kindly refer to the
> **[Official Documentation](https://jarryshaw.github.io/PyPCAPKit/)**.

The PyPCAPKit project is an open source Python program focused on network packet
parsing and analysis, which works as a comprehensive
[PCAP](https://en.wikipedia.org/wiki/Pcap) file extraction, construction and
analysis library, with [DictDumper](https://github.com/JarryShaw/DictDumper) as
its formatted output dumper.

Unlike popular PCAP file extractors such as [Scapy](https://scapy.net),
[DPKT](https://dpkt.readthedocs.io) and [PyShark](https://kiminewt.github.io/pyshark),
`pcapkit` is designed to be much more comprehensive: it reports more detailed
information about each packet, and offers a more *Pythonic* interface to work
with it. When that depth is not what you need, the same interface will also drive
six third-party extraction engines instead.

The whole project supports **Python 3.6** or later.

## Installation

```shell
pip install pypcapkit
```

Or from a clone, for the latest version and for development:

```shell
git clone https://github.com/JarryShaw/PyPCAPKit.git
cd PyPCAPKit
pip install -e .
```

The extraction engines and plug-ins are optional extras:

```shell
pip install pypcapkit[DPKT] # or Scapy, PyShark, PyPCAPFile, PyPCAP, PCAP_CT
pip install pypcapkit[crypto] # ESP payload decryption
pip install pypcapkit[cli] # command line interface
pip install pypcapkit[all] # every pure-Python extra
```

Four of the engines need something beyond a `pip install` -- a `tshark` binary, a
C compiler, `libpcap` headers, or an older interpreter -- and `all` deliberately
excludes both `pypcap` and `pcap-ct`, which must never be installed together.
The [installation guide](https://jarryshaw.github.io/PyPCAPKit/#installation)
covers every constraint and the reason for it, and `pcapkit` enforces each one in
code: asking for an engine that cannot run in the current environment warns with
the actual cause and falls back to `pcapkit`'s own parser.

## Usage

```python
>>> import pcapkit
>>> extraction = pcapkit.extract('in.pcap', nofile=True)
>>> len(extraction.frame)
6
>>> frame = extraction.frame[0]
>>> str(frame.protochain)
'Ethernet:IPv6:IPv6_ICMP'
>>> frame.info.time
datetime.datetime(2017, 11, 19, 15, 49, 5, 471719, tzinfo=datetime.timezone.utc)
>>> frame.payload.payload.src
IPv6Address('fe80::a6:87f9:2793:16ee')
```

The output above is from `examples/captures/in.pcap`, which is committed, so it
is reproducible from a clone.

Reassembly, TCP flow tracing and a different engine are all keyword arguments on
the same call:

```python
>>> scapy = pcapkit.extract('in.pcap', nofile=True, engine='scapy')
>>> reasm = pcapkit.extract('in.pcap', nofile=True, reassembly=True, ipv6=True)
>>> flows = pcapkit.extract('in.pcap', nofile=True, trace=True, tcp=True)
>>> len(flows.trace)
3
```

More worked examples, including the command line interface, are in
[How to ...](https://jarryshaw.github.io/PyPCAPKit/demo.html).

## Documentation

The [official documentation](https://jarryshaw.github.io/PyPCAPKit/) is the
reference for everything below. The pages worth knowing by name:

| Page | What is in it |
|---|---|
| [API reference](https://jarryshaw.github.io/PyPCAPKit/pcapkit/index.html) | Every module, protocol and constant |
| [Module structure](https://jarryshaw.github.io/PyPCAPKit/#module-structure) | What each of the eight subpackages is for |
| [Engine comparison](https://jarryshaw.github.io/PyPCAPKit/#engine-comparison) | Which engines exist, which Python versions they run on, and measured speed per packet |
| [Engine support](https://jarryshaw.github.io/PyPCAPKit/pcapkit/foundation/engines/index.html) | What each engine does *not* support, and how the gap is surfaced |
| [Installation](https://jarryshaw.github.io/PyPCAPKit/#installation) | Extras, engine prerequisites and the local development setup |
| [Testing](https://jarryshaw.github.io/PyPCAPKit/testing.html) | Running the suite, and the sample captures it needs |
| [How to ...](https://jarryshaw.github.io/PyPCAPKit/demo.html) | Worked examples, library and CLI |
| [Extensions](https://jarryshaw.github.io/PyPCAPKit/ext.html) | Registering your own protocols, engines and dumpers |

Release history is in [CHANGELOG.md](CHANGELOG.md), and contribution guidelines
are in [CONTRIBUTING.md](CONTRIBUTING.md).
Loading
Loading