diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000000..946afc9a7a --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,40 @@ +## 1.5.0 -- unreleased + +The largest release since 1.0, and the first recorded here as it happened rather than reconstructed. Three more extraction engines, ESP with payload decryption, SCTP and NGAP over SCTP, a library logger that no longer hijacks the consumer's, and a defect programme run through the issue tracker across some 140 issues and pull requests between #326 and #509. + +- **Added** -- three extraction engines: `engine='pypcap'` and `engine='pcap_ct'`, two independent distributions of the same `libpcap` interface, and `engine='pypcapfile'` (#386, #405). They buy speed by doing less -- neither `pypcap` nor `pcap_ct` dissects at all, so they offer neither reassembly nor flow tracing, and `pypcapfile` has no IPv6 decoder. Install only **one** of `pypcap` and `pcap-ct`: both own the top-level `pcap` module, and with both present `pcap-ct` wins the import and the other becomes unselectable. The matching interface constants `PyPCAP`, `PCAP_CT` and `PyPCAPFile` were missing and are now exported alongside `DPKT`, `Scapy`, `PyShark` and `PCAPKit` (#412). That brings the built-in set to seven engines; 3.11 is the last interpreter on which every one of them can run, and even there two of them cannot coexist. +- **Added** -- `EngineBase.unsupported_reason`, a preflight every engine answers and `Extractor.run` consults before anything is imported. Asking for an engine that cannot run in the current environment now gives one warning naming the real cause -- a Python version, a missing `tshark`, a missing `libpcap`, the wrong `pcap` distribution -- and a clean fall back to `pcapkit`'s own parser, rather than an error from inside the third-party package (#396, #405). +- **Added** -- ESP parsing and construction [[RFC 4303](https://datatracker.ietf.org/doc/html/rfc4303)], with optional payload decryption and ICV verification through `cryptography` (`pip install pypcapkit[crypto]`) (#378). Keys reach the dissector through a new caller-state channel, `pcapkit.corekit.context`, surfaced as the `context=` keyword on `extract()` and `Extractor`, carrying an `esp.SecurityAssociation`. Nothing here raises: with no association the SPI and sequence number are still reported and the ciphertext is left opaque, with `status=NO_SA`, and a failed decryption or ICV check is recorded on the parsed result the same way. Extended Sequence Numbers, TFC padding and anti-replay are not implemented, and an unsupported cipher or MAC is refused with a clear error rather than half-processed. +- **Added** -- SCTP as a transport protocol [[RFC 9260](https://datatracker.ietf.org/doc/html/rfc9260)]: all 13 chunk types, 8 chunk parameters and 13 error causes, with the CRC32c both recorded and verifiable -- it covers the SCTP packet alone, with no IP pseudo-header, so it can be checked from the SCTP bytes. Upper layers register on the DATA chunk's Payload Protocol Identifier through `register_sctp`, not on a port (#379). +- **Added** -- NGAP over SCTP (3GPP TS 38.413), decoding aligned PER through `pycrate` (`pip install pypcapkit[NGAP]`) (#251, #417). Decoding is generic by ASN.1 shape rather than per-procedure, so all 81 elementary procedures and 438 protocol IEs work and a new 3GPP release needs no code change. Registered as a default on PPID 60 and 66, but only 60 decodes: PPID 66 is an NGAP PDU inside a DTLS record, and there is no DTLS dissector. `pycrate` is deliberately excluded from the `all` extra -- it is LGPL-2.1+ where this package is BSD-3-Clause, and lands some 238 MB to obtain one module. +- **Added** -- the Mobility Header registry, completed (#383, #437). The [RFC 5568](https://datatracker.ietf.org/doc/html/rfc5568) fast-handover messages and options first, then all 24 registered message data types, 70 of the 71 registered options -- with nested sub-option registries for the flow identification, access network identifier, quality-of-service and LMA-controlled MAG parameter families -- and all 4 CGA extensions. Only the CGA Parameters option remains on the generic handler. +- **Added** -- dispatch entries for dissectors that existed but were reachable from no registry (#436): FTP-DATA on TCP 20, HTTP/1 on TCP 8080, HTTP on UDP 8080, `L2TPv2` on UDP 1701 and OSPF at `TransType` 89. `VLAN` became an abstract base with `C_Tag` (802.1Q) and `S_Tag` (802.1ad) as concrete subclasses, so a Q-in-Q frame no longer collapses into one opaque `Raw`; `L2TP` likewise became a base, with `L2TPv2` carrying the [RFC 2661](https://datatracker.ietf.org/doc/html/rfc2661) implementation. +- **Added** -- `pcapkit.utilities.logging` as a real interface: `get_logger()` for per-module children, `configure()` to set level, handler, stream, format or propagation at runtime, `reset()` to return to library-neutral, and `ensure_output()`. Seventeen modules now log under their own `__name__`, so a consumer can silence `pcapkit.foundation.registry` while keeping `pcapkit.foundation.extraction` (#384). +- **Added** -- `conflict` on the reassembly data models: absolute, inclusive ranges where two fragments claimed the same span with different bytes, which was previously lost silently on both the IP (#482) and TCP (#443, #478) paths. +- **Added** -- an end-to-end test tier (#376), sample-capture generators so a fresh clone can rebuild every fixture (#340), a Dockerised engine benchmark covering every supported Python version (#410), and registry round-trip coverage that records the entries which cannot close the cycle rather than skipping them (#440, #504). +- **Changed** -- `pcapkit` no longer configures logging at import. It installs a `NullHandler` and sets no level, so verbosity is inherited from the application instead of being seized by whichever library was imported second; the old stderr handler stays as the `PCAPKIT_DEVMODE` opt-in. Three consequences worth knowing: the previous behaviour is `configure(logging.INFO, stream=sys.stderr)`; 38 registry and extractor `info` calls became `debug`, so those messages are invisible even at `INFO`; and the handler is no longer `logger.handlers[0]`. `verbose=` output stays on stdout and is not logging (#384). +- **Changed** -- each warning is reported once per channel, and `pcapkit` no longer inserts a `simplefilter('ignore', ...)` at the front of the process-global `warnings.filters` (#362--#364, #390). The application's own filter therefore wins now, which is the point of the change and also the sharp edge in it: under `-W error`, or pytest's `filterwarnings = error`, a pcapkit warning that used to be suppressed will raise. Suppress them deliberately with `warnings.filterwarnings('ignore', category=BaseWarning)`. `quiet=True` now means no record at any level and no longer sets `sys.tracebacklimit`, and the `pcapkit.utilities.warnings.DEVMODE` re-export is gone -- its canonical home is `pcapkit.utilities.logging`. +- **Changed** -- `layer=` and `protocol=` are honoured rather than inert. Both were read under the wrong names, so every value a caller passed was dropped into `**kwargs` and discarded; the CLI's `-L` also now validates its argument instead of accepting anything. The packet context reaches the schema layer for the first time as well, so a field the wire elides can be resolved from its enclosing packet (#404). `follow_tcp_stream` dispatches on the engine type, where both branches of the old test were dead and the native adapter ran against every engine's frames (#402). +- **Changed** -- two reassembly and flow-tracing defaults moved (#435), and both are visible to a caller. `Datagram.completed` widened from `bool` to a `Completion` enumeration (`COMPLETE`, `PARTIAL`, `TIMEOUT`); only `COMPLETE` is truthy, so `if datagram.completed:` is unaffected but `datagram.completed == True` no longer holds. TCP flow tracing is **bidirectional by default**, which merges each flow's two halves and closes one only once both have FINed -- 331 flows become 111 on the sample HTTP capture, the difference being single-frame stray tails; pass `trace_bidirectional=False` for the old behaviour. IP reassembly also gained the 60-second timeout [[RFC 1122](https://datatracker.ietf.org/doc/html/rfc1122), [RFC 8200](https://datatracker.ietf.org/doc/html/rfc8200)], clocked off the capture's own timestamps rather than the wall clock, tunable with `reasm_timeout=`; TCP reassembly gets no timeout by default. `trace_analyse=` is new, and reassembles each traced flow's application layer. +- **Changed** -- conflicting TCP overlaps resolve first-write-wins, per [RFC 9293](https://datatracker.ietf.org/doc/html/rfc9293) section 3.10, where they had silently resolved last-write-wins (#443, #478). A deliberate behaviour break, and a narrow one: a conforming retransmission carries identical bytes, so nothing changes for it. IP fragment reassembly keeps last-write-wins, because [RFC 791](https://datatracker.ietf.org/doc/html/rfc791) specifies the opposite resolution, and records the disagreement instead (#482). +- **Changed** -- `Probe`, `CipherSuite` and `IntegritySuite` are `Info` subclasses rather than `typing.NamedTuple`, and no `NamedTuple` remains in the package. They are Mappings now, so `len()` and iteration yield field names rather than values. +- **Changed** -- renames with no compatibility alias left behind: `HoleDiscriptor` is spelled `HoleDescriptor` and its package alias `TCP_HoleDiscriptor` is `TCP_HoleDescriptor` (#350); PCAP-NG `Option` subclasses spell the namespace class keyword `ns=` instead of `namespace=` (#439); and `examples/sample` and `examples/samples` -- one letter apart, holding different things -- are now `examples/captures` and `examples/generators`. +- **Changed** -- extraction is around 46% faster on a 1,117-frame HTTP capture, with byte-identical output (#420). A reassembled datagram's payload is now analysed on first read rather than eagerly, which cuts IP reassembly's own cost by 90.7% and TCP's by 23.7% -- IP reassembly submits a datagram for every frame, fragmented or not (#424). Flow tracing over the same capture went from 1416.6 ms to 744.0 ms, because the flow dumper had been handing each record to a `Frame` constructor that re-dissected the whole protocol stack to return bytes it had just been given; options are no longer parsed twice either (#427). All output compared byte-for-byte across the sample captures in each case. +- **Fixed** -- next-layer, option, chunk, block and parameter dispatch all read `defaultdict` registries, so a lookup miss inserted the key into class-level state shared by every later instance, after which a legitimate `register_*` call warned that the code was already registered. Every read now goes through a lookup that does not grow the table, and `IPv4.__option__` and `HIP.__parameter__` became inspectable class attributes rather than names assembled at call time (#426, #428, #429, #434). One break comes with it: a tuple-registered handler pair written to the documented `OptionParser`/`OptionConstructor` signature now works where it could previously never be called at all, and a pair written with an explicit leading `self` -- the only shape that used to work -- now does not. +- **Fixed** -- on Python 3.10 and older, no `Schema` subclass got its own `_abc_impl`: all of them fell through to `collections.abc.Mapping`'s, so a single `isinstance` or `issubclass` answer poisoned every later question about that class for the rest of the process. A terminating PCAP-NG `EndRecord` tested `True` as an `IPv4Record` (#439). +- **Fixed** -- construction, which was broken in several places at once: the generated typed `__init__` was never installed, so `__post_init__` did not run and a schema built from a subset of its fields could not be packed at all -- `UDP(srcport=53, dstport=5353)` now packs (#430); IPv6 and Mobility Header option padding was wrong, leaving construction wholly broken (#398); `HTTP.make` called the versioned `make` unbound, so every real call raised `TypeError` (#452, #462); and `IPv4._make_data` returned the fragment offset in octets where the wire wants 8-octet units, and read `data.options` on a packet that has none (#494, #499). +- **Fixed** -- a truncated or under-declared area no longer parses "successfully", and no longer wedges the process. The option and list loops could spin forever with no exception on a truncated area, reachable from untrusted input through HOPOPT, IPv6-Opts, MH, HIP and SCTP; each iteration must now advance the stream by at least one octet, and the error names the option, the offset and the octets remaining (#431, #432). Separately, wire-derived lengths in `ipv6_opts`, CALIPSO, MPL, REG_INFO and four HIP list callbacks underflowed below zero, which `ListField`'s own `while length > 0` then turned into a silent empty list; they are floored and raise instead (#449, #456, #460, #463). +- **Fixed** -- field widths and units, each measured against the specification rather than inferred: HIP's `TRANSPORT_FORMAT_LIST`, `NAT_TRAVERSAL_MODE` and `ESP_TRANSFORM` list entries are two octets, not one [[RFC 7401](https://datatracker.ietf.org/doc/html/rfc7401), [RFC 5770](https://datatracker.ietf.org/doc/html/rfc5770), [RFC 7402](https://datatracker.ietf.org/doc/html/rfc7402)] (#463, #472); the MN-ID option sizes from its subtype, not from `identifier`'s Python type (#448, #464, #467); IPv6-Route's `Hdr Ext Len` is computed in 8-octet units on both sides [[RFC 8200](https://datatracker.ietf.org/doc/html/rfc8200)] (#487, #489); and the Fast Binding Update and Acknowledgment Lifetimes are plain seconds [[RFC 5568](https://datatracker.ietf.org/doc/html/rfc5568)], not [RFC 6275](https://datatracker.ietf.org/doc/html/rfc6275)'s four-second units (#502). +- **Fixed** -- stdlib exceptions leaking out where the library's own were promised: a malformed IP field value raised a bare `ValueError` instead of `FieldValueError` (#465); a `bool` address was silently packed as `0.0.0.1` or `0.0.0.0`, `bool` being an `int` subclass (#491, #500); and `@prepare` discarded extra arguments silently and treated a *declared* zero length as end of stream, which is now distinguished from a genuinely exhausted one and raises `StreamEOFError` (#454, #458). +- **Fixed** -- the engine adapters, which were quietly wrong rather than loud. The `dpkt` toolkit split TCP and IPv4 headers at their fixed struct size instead of their real length, so option octets overwrote payload in the sequence-indexed reassembly buffer; it also read an `ipv6_frag.nh` that `dpkt` does not have, and passed fragment offsets unscaled (#351, #370, #385, #395). `scapy` never loaded its layer registry, so that engine did not dissect at all (#409), and its IPv4 fragment offset reached reassembly unscaled (#483, #484). The four IPv6 adapters disagreed about whether the 8-octet Fragment header belongs to `ihl`, `header` and `tl`; per [RFC 8200](https://datatracker.ietf.org/doc/html/rfc8200) section 4.5 it belongs to none of them, and all four now agree (#415, #424). +- **Fixed** -- PCAP and PCAP-NG output and parsing: `bytes(frame)` returned the *next* frame's octets, `files=True` wrote names like `Frame 1..json`, and a PCAP-NG `timestamp_epoch` was shifted by the reading host's timezone (#403); seven further parser defects (#341--#347, #371) and, on the write path, four more block-parsing ones (#388); `BitField` packed every named bit as set (#359, #374); the extension-header walk failed to advance past the last IPv6 extension header (#348, #373); and IPv6 fragment offsets went unscaled, with reassembly keyed on the flow label -- optional, and routinely zero, so distinct datagrams collapsed together -- rather than on the fragment identification (#389). +- **Fixed** -- TCP reassembly mixed absolute sequence numbers with buffer-relative slicing, so on any capture carrying a SYN with a realistic initial sequence number incomplete datagrams were dropped silently, and the `completed=False` branch of the public API was unreachable (#349, #376). +- **Fixed** -- constant lookups that rejected a value the registry defines. `RouterAlert(0)` is the only value [RFC 2113](https://datatracker.ietf.org/doc/html/rfc2113) defines and the one IGMP, RSVP and MLD actually send, and it was discarded because the vendor crawler skipped a header row IANA's CSV does not have; IPX `Socket(0)` is that protocol's own default, so `bytes(IPX(...))` crashed on its own defaults; and two FTP `_missing_` overrides were plain methods rather than classmethods, so every unregistered value raised `TypeError` instead of extending the enumeration (#492, #503). +- **Fixed** -- `format='text'` raised `AttributeError` before writing anything, naming a `dictdumper.Text` that has never existed. It now points at `Tree`, as the `'txt'` alias beside it already did. +- **Fixed** -- 45 places where a documentation page contradicted the code (#413), ambiguous cross-references and five autodoc signature failures (#416), and `Extractor`'s documented exception plus 40 phantom or stale `Args:` labels (#501). + +Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). + +--- + +Full changelog: diff --git a/MANIFEST.in b/MANIFEST.in index 55cbf739c0..89194f4fcb 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -14,5 +14,21 @@ prune test include README.rst include LICENSE +# The changelog needs two lines of its own, and neither is redundant. +# +# ``global-include *.rst`` above matches only ``*.rst``, so the root +# ``CHANGELOG.md`` is picked up by nothing and would be absent from every source +# distribution without this line -- measured, not assumed. +include CHANGELOG.md + +# ``CHANGELOG.md`` carries only the version being released, so on its own the +# distribution would describe one release and no history. ``prune docs`` above +# drops the rest, which is where the history lives, so re-add just that one +# directory afterwards -- the same shape as ``include README.rst``, and for the +# same reason. Each of those files is a self-contained reStructuredText entry +# that renders standalone; the Sphinx index beside them is deliberately *not* +# shipped, because it is a ``toctree`` that only Sphinx can read. +recursive-include docs/source/changelog *.rst + include pcapkit/_extern exclude pcapkit/protocols/*/NotImplemented diff --git a/docs/source/changelog.rst b/docs/source/changelog.rst new file mode 100644 index 0000000000..a702867fe0 --- /dev/null +++ b/docs/source/changelog.rst @@ -0,0 +1,74 @@ +.. The changelog's index. Every entry is its own document under + ``changelog/``, one per released version, and this page is the title, the + preamble and the table of contents over them -- there is exactly one copy of + each entry and nothing to keep in step by hand. + + The repository root carries ``CHANGELOG.md`` rather than a second copy of + this history. It holds only the version being released, in Markdown, because + its consumers are the ``Create Release`` workflow's release body and the + source distribution, and both of those read Markdown. It is generated from + ``changelog/.rst``, so this tree stays the single source. + + The preamble below is verbatim from the single-file changelog it replaces. + +========= +Changelog +========= + +All notable changes to PyPCAPKit are recorded here. + + **Note** -- Versions before 1.5.0 are reconstructed from the git history, so + they summarise each release rather than enumerate every change. The record + starts at 0.13.0 (2018-12-08); the earlier 0.x releases are not covered. + + **Note** -- Post-releases (``X.Y.Z.postN``) are, from 1.0.1 onwards, automated + publications of the weekly registry refresh: a bot regenerates the vendor + constant enumerations under ``pcapkit.const`` from the upstream IANA + registries and bumps the version. They carry no library changes, and are + collapsed into a single line per release below. Where a post-release did + carry something real, it is called out. + +Each release has its own page below, newest first. The repository root's +:file:`CHANGELOG.md` carries only the version currently being released; this is +the whole history. + +.. toctree:: + :maxdepth: 1 + + changelog/1.5.0 + changelog/1.4.1 + changelog/1.4.0 + changelog/1.3.5 + changelog/1.3.4 + changelog/1.3.3 + changelog/1.3.1 + changelog/1.3.0 + changelog/1.2.2 + changelog/1.2.1 + changelog/1.2.0 + changelog/1.1.1 + changelog/1.1.0 + changelog/1.0.3 + changelog/1.0.2 + changelog/1.0.1 + changelog/1.0.0 + changelog/0.16.3 + changelog/0.16.2 + changelog/0.16.1 + changelog/0.16.0 + changelog/0.15.5 + changelog/0.15.4 + changelog/0.15.3 + changelog/0.15.2 + changelog/0.15.1 + changelog/0.15.0 + changelog/0.14.5 + changelog/0.14.4 + changelog/0.14.3 + changelog/0.14.2 + changelog/0.14.1 + changelog/0.14.0 + changelog/0.13.3 + changelog/0.13.2 + changelog/0.13.1 + changelog/0.13.0 diff --git a/docs/source/changelog/0.13.0.rst b/docs/source/changelog/0.13.0.rst new file mode 100644 index 0000000000..58bddc06e2 --- /dev/null +++ b/docs/source/changelog/0.13.0.rst @@ -0,0 +1,7 @@ +0.13.0 -- 2018-12-08 +==================== + +No library changes -- a licence and packaging metadata refresh. + +*Also released:* ``0.13.0.post1``, ``0.13.0.post2`` (through 2018-12-12); +``post2`` fixed encoding errors in ``setup.py``. diff --git a/docs/source/changelog/0.13.1.rst b/docs/source/changelog/0.13.1.rst new file mode 100644 index 0000000000..264dbf0528 --- /dev/null +++ b/docs/source/changelog/0.13.1.rst @@ -0,0 +1,8 @@ +0.13.1 -- 2019-01-23 +==================== + +* **Added** -- ``tbtrim`` is used to trim PyPCAPKit's own frames out of + tracebacks through a custom ``excepthook``, so a traceback points at the + caller's code. +* **Fixed** -- compatibility issues, with a new ``pcapkit.utilities.compat`` + module. diff --git a/docs/source/changelog/0.13.2.rst b/docs/source/changelog/0.13.2.rst new file mode 100644 index 0000000000..4362af65fa --- /dev/null +++ b/docs/source/changelog/0.13.2.rst @@ -0,0 +1,4 @@ +0.13.2 -- 2019-01-24 +==================== + +Maintenance release: the ``tbtrim`` dependency was updated. diff --git a/docs/source/changelog/0.13.3.rst b/docs/source/changelog/0.13.3.rst new file mode 100644 index 0000000000..8070d4b369 --- /dev/null +++ b/docs/source/changelog/0.13.3.rst @@ -0,0 +1,9 @@ +0.13.3 -- 2019-02-26 +==================== + +**Fixed** -- numeric literals in the constant modules that were incompatible +with Python 3.5. + +*Also released:* ``0.13.3.post1``, ``0.13.3.post2`` (through 2019-03-01); +``post1`` declared official support for Python 3.5 and below, and ``post2`` +refreshed the dependencies. diff --git a/docs/source/changelog/0.14.0.rst b/docs/source/changelog/0.14.0.rst new file mode 100644 index 0000000000..011c7a6dd9 --- /dev/null +++ b/docs/source/changelog/0.14.0.rst @@ -0,0 +1,5 @@ +0.14.0 -- 2019-03-02 +==================== + +**Changed** -- the TCP reassembly pipeline handles the RST flag. Docker build +files were added. diff --git a/docs/source/changelog/0.14.1.rst b/docs/source/changelog/0.14.1.rst new file mode 100644 index 0000000000..7811324951 --- /dev/null +++ b/docs/source/changelog/0.14.1.rst @@ -0,0 +1,5 @@ +0.14.1 -- 2019-03-07 +==================== + +**Added** -- the ``PCAPKIT_DEVMODE`` environment variable, which turns on +development-mode behaviour without a code change. diff --git a/docs/source/changelog/0.14.2.rst b/docs/source/changelog/0.14.2.rst new file mode 100644 index 0000000000..3ea6bc3cf6 --- /dev/null +++ b/docs/source/changelog/0.14.2.rst @@ -0,0 +1,4 @@ +0.14.2 -- 2019-03-28 +==================== + +**Fixed** -- the exception classes, and the TCP reassembly algorithm. diff --git a/docs/source/changelog/0.14.3.rst b/docs/source/changelog/0.14.3.rst new file mode 100644 index 0000000000..13ca9964ef --- /dev/null +++ b/docs/source/changelog/0.14.3.rst @@ -0,0 +1,5 @@ +0.14.3 -- 2019-08-08 +==================== + +**Fixed** -- reported bugs (#29, #30), and a Wikipedia block that broke a vendor +crawler. Constant enumerations and the sample output were regenerated. diff --git a/docs/source/changelog/0.14.4.rst b/docs/source/changelog/0.14.4.rst new file mode 100644 index 0000000000..7da8893ada --- /dev/null +++ b/docs/source/changelog/0.14.4.rst @@ -0,0 +1,6 @@ +0.14.4 -- 2019-09-01 +==================== + +* **Added** -- the vendor crawler scripts ship in the distribution. +* **Changed** -- the CLI and vendor dependencies are separate extras in + ``setup.py``; constant enumerations regenerated. diff --git a/docs/source/changelog/0.14.5.rst b/docs/source/changelog/0.14.5.rst new file mode 100644 index 0000000000..ca7e4586ff --- /dev/null +++ b/docs/source/changelog/0.14.5.rst @@ -0,0 +1,6 @@ +0.14.5 -- 2019-10-24 +==================== + +* **Added** -- a CLI for the vendor crawlers. +* **Changed** -- vendor imports are deferred, so the crawler dependencies are + not needed to import ``pcapkit``; dependencies revised. diff --git a/docs/source/changelog/0.15.0.rst b/docs/source/changelog/0.15.0.rst new file mode 100644 index 0000000000..cdfd3bc26a --- /dev/null +++ b/docs/source/changelog/0.15.0.rst @@ -0,0 +1,11 @@ +0.15.0 -- 2020-06-07 +==================== + +* **Changed** -- the parsing and construction logic was merged, so a protocol is + no longer implemented twice; TCP flow tracing and several interfaces were + revised. +* **Added** -- full API documentation, published at + https://pypcapkit.jarryshaw.me. +* **Fixed** -- the DictDumper dependency and the failures it caused (#37, #40). + +Preceded by ``0.15.0rc1`` (2020-06-06). diff --git a/docs/source/changelog/0.15.1.rst b/docs/source/changelog/0.15.1.rst new file mode 100644 index 0000000000..ec3ae7e30d --- /dev/null +++ b/docs/source/changelog/0.15.1.rst @@ -0,0 +1,8 @@ +0.15.1 -- 2020-06-18 +==================== + +Maintenance release: the constant enumerations and the vendor crawlers that +generate them were revised. + +*Also released:* ``0.15.1.post1`` (2020-06-19), a refresh of the constant +enumerations. diff --git a/docs/source/changelog/0.15.2.rst b/docs/source/changelog/0.15.2.rst new file mode 100644 index 0000000000..95592198bc --- /dev/null +++ b/docs/source/changelog/0.15.2.rst @@ -0,0 +1,7 @@ +0.15.2 -- 2020-06-27 +==================== + +* **Added** -- a register interface on the protocol classes, and the same on + ``pcapkit.foundation.analysis``, so both can be extended without patching the + library. +* **Fixed** -- flow tracing output format and content consistency. diff --git a/docs/source/changelog/0.15.3.rst b/docs/source/changelog/0.15.3.rst new file mode 100644 index 0000000000..5550c45d7b --- /dev/null +++ b/docs/source/changelog/0.15.3.rst @@ -0,0 +1,5 @@ +0.15.3 -- 2020-08-17 +==================== + +* **Fixed** -- ARP parsing (#55). +* **Added** -- logging, through a ``pcapkit`` logger. diff --git a/docs/source/changelog/0.15.4.rst b/docs/source/changelog/0.15.4.rst new file mode 100644 index 0000000000..0808e969d6 --- /dev/null +++ b/docs/source/changelog/0.15.4.rst @@ -0,0 +1,5 @@ +0.15.4 -- 2020-08-28 +==================== + +**Fixed** -- the missing ``stacklevel`` attribute on ``pcapkit.utilities`` (#58). +Travis CI was set up. diff --git a/docs/source/changelog/0.15.5.rst b/docs/source/changelog/0.15.5.rst new file mode 100644 index 0000000000..a9a2a6850f --- /dev/null +++ b/docs/source/changelog/0.15.5.rst @@ -0,0 +1,5 @@ +0.15.5 -- 2020-10-11 +==================== + +**Fixed** -- a fallback encoder was added so that data DictDumper cannot encode +no longer raises out of the dumper (#65). diff --git a/docs/source/changelog/0.16.0.rst b/docs/source/changelog/0.16.0.rst new file mode 100644 index 0000000000..11925d4afe --- /dev/null +++ b/docs/source/changelog/0.16.0.rst @@ -0,0 +1,23 @@ +0.16.0 -- 2022-05-31 +==================== + +A project-wide revision, and the last release of the pre-1.0 design. + +* **Changed** -- type annotations throughout, and linter compliance + (pylint, mypy, bandit, vermin) across every module. +* **Changed** -- protocol classes were separated from their data models, which + moved to ``pcapkit.protocols.data``; a single ``Protocol`` class handles both + parsing and construction, and subclassing it to add a protocol is far less + work. +* **Added** -- ``pcapkit.foundation.registry``, so protocols can be subscribed + by name; ``Protocol.analyze`` for analysing a payload directly, used by the + reassembly classes; ``pcapkit.corekit.multidict``, adapted from Werkzeug. +* **Changed** -- reassembly became a package (IP, IPv4, IPv6, TCP) with revised + data models; PCAP and the auxiliary protocols moved under ``pcapkit.misc``. +* **Removed** -- the ``validators`` module, several decorators from + ``pcapkit.utilities``, and multiprocessing support in ``Extractor``. +* **Fixed** -- the DPKT toolkit (#101), IP reassembly, and missing ``id`` + methods on subclassed protocols. + +The PyPCAPKit Enhancement Proposals discussion channel was opened with this +release (#106). diff --git a/docs/source/changelog/0.16.1.rst b/docs/source/changelog/0.16.1.rst new file mode 100644 index 0000000000..5ac4e60b1c --- /dev/null +++ b/docs/source/changelog/0.16.1.rst @@ -0,0 +1,6 @@ +0.16.1 -- 2022-06-07 +==================== + +**Changed** -- warnings are raised through ``pcapkit.utilities.warnings`` +instead of ``warnings.warn`` directly, and a missing optional dependency now +warns for the CLI and vendor extras too. diff --git a/docs/source/changelog/0.16.2.rst b/docs/source/changelog/0.16.2.rst new file mode 100644 index 0000000000..c78d53adc1 --- /dev/null +++ b/docs/source/changelog/0.16.2.rst @@ -0,0 +1,5 @@ +0.16.2 -- 2022-08-03 +==================== + +**Fixed** -- ``Info`` ``__init__`` generation for classes without annotations +(#113); the IP reassembly algorithm (#82). diff --git a/docs/source/changelog/0.16.3.rst b/docs/source/changelog/0.16.3.rst new file mode 100644 index 0000000000..1cbb91cf39 --- /dev/null +++ b/docs/source/changelog/0.16.3.rst @@ -0,0 +1,5 @@ +0.16.3 -- 2022-10-31 +==================== + +**Fixed** -- import crashes on some interpreters (#114, #116); the reassembly +property caches; README rendering for PyPI. The build chain was revised. diff --git a/docs/source/changelog/1.0.0.rst b/docs/source/changelog/1.0.0.rst new file mode 100644 index 0000000000..e4562011d7 --- /dev/null +++ b/docs/source/changelog/1.0.0.rst @@ -0,0 +1,31 @@ +1.0.0 -- 2023-05-09 +=================== + +The 1.0 rewrite: parsing and construction are one declarative definition per +protocol, and the extraction backend became pluggable. Roughly 55,000 lines +changed across 617 files, released after 21 beta builds and a release candidate. + +* **Added** -- the schema and field layer, ``pcapkit.protocols.schema`` and + ``pcapkit.corekit.fields``. A protocol declares its wire format once, as + fields (numbers, strings, IP addresses, payloads, options, lists, switches), + and both parsing and construction follow from it. +* **Added** -- PCAP-NG support [1]_: the protocol implementation covering every + block type, the generic and per-block options, name-resolution records and + decryption secrets, plus a matching extraction engine, toolkit functions and + ``pcapkit.protocols.schema.misc.pcapng``. +* **Added** -- ``pcapkit.foundation.engines``, making the extraction backend a + registered, swappable component: the built-in PCAP and PCAP-NG engines + alongside DPKT, Scapy and PyShark, with engine registry APIs. +* **Added** -- ``SchemaWarning``, ``DeprecatedFormatWarning`` and + ``RegistryWarning``; ``Protocol._get_payload`` for customised payload + retrieval; ``packet`` passed down the protocol chain so a layer can see its + parent's context, which is how IPv6 hands source and destination addresses to + the transport layer. +* **Changed** -- flow tracing became the ``pcapkit.foundation.traceflow`` + package, and reassembly gained data-model modules; + ``pcapkit.toolkit.default`` was renamed ``pcapkit.toolkit.pcap``; + ``pcapkit.foundation.engine`` became ``engines``; ``IPField`` was split into + ``IPAddressField`` and ``IPInterfaceField``; the option and parameter + registries for HIP and IPv4 warn on overwrite. + +.. [1] PCAP-NG is specified by ``draft-tuexen-opsawg-pcapng``, not by an RFC. diff --git a/docs/source/changelog/1.0.1.rst b/docs/source/changelog/1.0.1.rst new file mode 100644 index 0000000000..16c1e5909a --- /dev/null +++ b/docs/source/changelog/1.0.1.rst @@ -0,0 +1,8 @@ +1.0.1 -- 2023-05-14 +=================== + +**Changed** -- ``Info`` and ``Schema`` subclasses are finalised when the class is +created rather than on each instantiation, which is a straight runtime saving on +parsing. + +*Also released:* ``1.0.1.post1`` through ``1.0.1.post3`` (through 2023-05-27). diff --git a/docs/source/changelog/1.0.2.rst b/docs/source/changelog/1.0.2.rst new file mode 100644 index 0000000000..f81efa9b89 --- /dev/null +++ b/docs/source/changelog/1.0.2.rst @@ -0,0 +1,12 @@ +1.0.2 -- 2023-06-05 +=================== + +* **Added** -- Mobility Header (MH) parsing and construction [:rfc:`6275`], + including the message types, mobility options, CGA parameters and link-layer + address option codes, and the binding error status codes. +* **Fixed** -- ``pcapkit.extract`` failing on interpreters where + ``decimal.localcontext`` takes no keyword arguments (#139); wheel filenames + now carry the correct Python tags. + +*Also released:* ``1.0.2.post1`` through ``1.0.2.post8`` (through 2023-06-27); +``post6`` also reset the conda build number. diff --git a/docs/source/changelog/1.0.3.rst b/docs/source/changelog/1.0.3.rst new file mode 100644 index 0000000000..53be2ba55d --- /dev/null +++ b/docs/source/changelog/1.0.3.rst @@ -0,0 +1,11 @@ +1.0.3 -- 2023-06-29 +=================== + +* **Added** -- the IANA service name and transport protocol port number registry + as the ``AppType`` enumeration [:rfc:`6335`], wired into TCP and UDP parsing + so ports resolve to service names; a generated enumeration for the TCP header + flags; an undefined-command member for FTP commands. +* **Changed** -- registry management was redesigned; the MPTCP and PCAP-NG + option enumerations were revised. + +*Also released:* ``1.0.3.post1`` through ``1.0.3.post3`` (through 2023-07-04). diff --git a/docs/source/changelog/1.1.0.rst b/docs/source/changelog/1.1.0.rst new file mode 100644 index 0000000000..68a7fd81f9 --- /dev/null +++ b/docs/source/changelog/1.1.0.rst @@ -0,0 +1,12 @@ +1.1.0 -- 2023-07-09 +=================== + +* **Changed** -- the schema and field classes were rebuilt on metaclasses, with + ``EnumSchema`` for enum-dispatched schemas, and every protocol schema was + revised onto them: IPv4, HOPOPT, IPv6-Opts, IPv6-Route, HIP, MH, TCP, HTTP/2 + and PCAP-NG. +* **Added** -- schema registration through the protocol registry APIs; + persistent additional and excluded fields on ``Info``; numeric and comparison + support on the PCAP-NG option code and application-layer port enumerations. + +*Also released:* ``1.1.0.post1`` through ``1.1.0.post3`` (through 2023-08-12). diff --git a/docs/source/changelog/1.1.1.rst b/docs/source/changelog/1.1.1.rst new file mode 100644 index 0000000000..a405ccfacb --- /dev/null +++ b/docs/source/changelog/1.1.1.rst @@ -0,0 +1,8 @@ +1.1.1 -- 2023-08-15 +=================== + +**Fixed** -- ``AppType`` enumeration lookups; ``typing.TypeAlias`` compatibility +on older interpreters; the output file name chosen by flow tracing; dumping of +``Schema`` objects through a custom dumper. + +*Also released:* ``1.1.1.post1``, ``1.1.1.post2`` (through 2023-08-23). diff --git a/docs/source/changelog/1.2.0.rst b/docs/source/changelog/1.2.0.rst new file mode 100644 index 0000000000..f4eaa15143 --- /dev/null +++ b/docs/source/changelog/1.2.0.rst @@ -0,0 +1,9 @@ +1.2.0 -- 2023-08-26 +=================== + +* **Added** -- streaming input. ``pcapkit.corekit.io.SeekableReader`` buffers a + non-seekable stream so a capture can be read from ``stdin`` or a pipe, which + is what live ``tcpdump`` output needs (#156). +* **Fixed** -- reassembly, broken since 0.16.3 (#155). ``Extractor`` now raises + an explicit unsupported-call error, and logs a notice, when reassembly or flow + tracing is requested where it cannot be provided. diff --git a/docs/source/changelog/1.2.1.rst b/docs/source/changelog/1.2.1.rst new file mode 100644 index 0000000000..78e70abfd6 --- /dev/null +++ b/docs/source/changelog/1.2.1.rst @@ -0,0 +1,6 @@ +1.2.1 -- 2023-08-27 +=================== + +**Fixed** -- bugs in the ``SeekableReader`` introduced one day earlier in 1.2.0. + +*Also released:* ``1.2.1.post1`` through ``1.2.1.post3`` (through 2023-09-09). diff --git a/docs/source/changelog/1.2.2.rst b/docs/source/changelog/1.2.2.rst new file mode 100644 index 0000000000..39c4eb3f23 --- /dev/null +++ b/docs/source/changelog/1.2.2.rst @@ -0,0 +1,6 @@ +1.2.2 -- 2023-09-14 +=================== + +**Fixed** -- extraction failing outright (#166). + +*Also released:* ``1.2.2.post1``, ``1.2.2.post2`` (through 2023-10-04). diff --git a/docs/source/changelog/1.3.0.rst b/docs/source/changelog/1.3.0.rst new file mode 100644 index 0000000000..d6e2b12611 --- /dev/null +++ b/docs/source/changelog/1.3.0.rst @@ -0,0 +1,14 @@ +1.3.0 -- 2023-10-04 +=================== + +A redesign of the extension points, on metaclasses. + +* **Changed** -- the protocol, reassembly, flow-tracing, engine, vendor and + field base classes are all built on metaclasses, which is what registers a + subclass and fixes its name and module. +* **Added** -- ``ModuleDescriptor``, so a registry entry can name a module and + class that are imported lazily on first use rather than at registration; the + registry functions accept it throughout. ``VersionInfo`` gained a ``version`` + property. +* **Changed** -- ``_Engine`` renamed ``EngineBase``; the ``beholder`` decorator + logs the error it swallows under devmode and verbose mode. diff --git a/docs/source/changelog/1.3.1.rst b/docs/source/changelog/1.3.1.rst new file mode 100644 index 0000000000..364d1d8225 --- /dev/null +++ b/docs/source/changelog/1.3.1.rst @@ -0,0 +1,11 @@ +1.3.1 -- 2023-12-21 +=================== + +* **Changed** -- ``register_port()`` was renamed ``register_apptype()``, with no + alias left behind; callers registering application-layer protocols by port + must be updated. +* **Changed** -- the documentation was revised across the whole project -- + ``pcapkit.protocols``, ``pcapkit.corekit`` and ``pcapkit.dumpkit`` in + particular. + +*Also released:* ``1.3.1.post1`` through ``1.3.1.post28`` (through 2024-10-26). diff --git a/docs/source/changelog/1.3.3.rst b/docs/source/changelog/1.3.3.rst new file mode 100644 index 0000000000..9adfca4d20 --- /dev/null +++ b/docs/source/changelog/1.3.3.rst @@ -0,0 +1,16 @@ +1.3.3 -- 2024-11-03 +=================== + +* **Fixed** -- errors reading PCAP files from both the CLI and the library + (#240): missing optional CLI dependencies are handled gracefully and console + output is far less verbose; installation from source, which ``setup.py`` had + broken (#221); PCAP output during flow tracing (#180). +* **Changed** -- the base protocol data class carries ``packet``; ``Info`` + iteration skips excluded fields. + +1.3.2 was never released. The version string existed in-tree for a few hours on +2024-11-03 and was superseded by 1.3.3 the same day, so there is no ``v1.3.2`` +tag and no distribution. + +*Also released:* ``1.3.3.post1``, ``1.3.3.post2`` (through 2024-11-14); +``post1`` fixed a PyPI metadata problem. diff --git a/docs/source/changelog/1.3.4.rst b/docs/source/changelog/1.3.4.rst new file mode 100644 index 0000000000..fc8d6c9813 --- /dev/null +++ b/docs/source/changelog/1.3.4.rst @@ -0,0 +1,9 @@ +1.3.4 -- 2024-11-14 +=================== + +* **Fixed** -- IPv6 Routing header length handling, and extraction of the IPv6 + extension header chain (#218). +* **Changed** -- the build matrix moved to Python 3.13 and stopped testing 3.8. + +This release also introduced the transport-layer parsing regression fixed in +1.3.5; upgrade past it rather than to it. diff --git a/docs/source/changelog/1.3.5.rst b/docs/source/changelog/1.3.5.rst new file mode 100644 index 0000000000..cef3273c1d --- /dev/null +++ b/docs/source/changelog/1.3.5.rst @@ -0,0 +1,12 @@ +1.3.5 -- 2024-11-16 +=================== + +**Fixed** -- transport-layer payloads were not decoded at all. An inverted +``payload is None`` check in ``pcapkit.protocols.internet.internet`` left every +TCP and UDP frame under IPv4/IPv6 reported as ``raw``, typically with +``'int' object has no attribute 'port'``. The regression was introduced by the +IPv6 extension-header fix in 1.3.4, so 1.3.4 should be skipped entirely. + +*Also released:* ``1.3.5.post1`` through ``1.3.5.post43`` (through 2026-08-21), +all registry refreshes except ``post41``, which switched the Anaconda upload to +the official action. diff --git a/docs/source/changelog/1.4.0.rst b/docs/source/changelog/1.4.0.rst new file mode 100644 index 0000000000..bf4b763342 --- /dev/null +++ b/docs/source/changelog/1.4.0.rst @@ -0,0 +1,14 @@ +1.4.0 -- 2026-08-21 +=================== + +The first release in nearly two years to carry code changes, and the first with +a test suite. + +* **Added** -- a unit-test suite under ``tests/``, covering ``pcapkit.corekit``, + the extraction engines, reassembly and flow tracing, the protocol schemas, the + CLI and the utility modules. +* **Fixed** -- PCAP-NG option dispatch regressions; the fallback name generated + by the ``linktype`` vendor crawler. +* **Changed** -- CI tests current Python versions again; every GitHub Actions + job carries an explicit permissions block; the conda and cron release tooling + was repaired, and Anaconda uploads use the official upload action. diff --git a/docs/source/changelog/1.4.1.rst b/docs/source/changelog/1.4.1.rst new file mode 100644 index 0000000000..1aa554f2ec --- /dev/null +++ b/docs/source/changelog/1.4.1.rst @@ -0,0 +1,13 @@ +1.4.1 -- 2026-08-22 +=================== + +Release engineering only -- no library behaviour changed. + +* **Added** -- a unit-test CI workflow, and the packaging, vendor-cron and + documentation workflows are now gated on it, so a release can no longer be cut + from a tree whose tests fail. +* **Changed** -- the vendor cron refuses to publish when the registry update + itself failed, rather than shipping a half-refreshed release. +* **Fixed** -- unit-test compatibility on Python 3.10. + +*Also released:* ``1.4.1.post1``, ``1.4.1.post2`` (through 2026-09-12). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst new file mode 100644 index 0000000000..9851e1d86e --- /dev/null +++ b/docs/source/changelog/1.5.0.rst @@ -0,0 +1,248 @@ +1.5.0 -- unreleased +=================== + +The largest release since 1.0, and the first recorded here as it happened rather +than reconstructed. Three more extraction engines, ESP with payload decryption, +SCTP and NGAP over SCTP, a library logger that no longer hijacks the consumer's, +and a defect programme run through the issue tracker across some 140 issues and +pull requests between #326 and #509. + +* **Added** -- three extraction engines: ``engine='pypcap'`` and + ``engine='pcap_ct'``, two independent distributions of the same ``libpcap`` + interface, and ``engine='pypcapfile'`` (#386, #405). They buy speed by doing + less -- neither ``pypcap`` nor ``pcap_ct`` dissects at all, so they offer + neither reassembly nor flow tracing, and ``pypcapfile`` has no IPv6 decoder. + Install only **one** of ``pypcap`` and ``pcap-ct``: both own the top-level + ``pcap`` module, and with both present ``pcap-ct`` wins the import and the + other becomes unselectable. The matching interface constants ``PyPCAP``, + ``PCAP_CT`` and ``PyPCAPFile`` were missing and are now exported alongside + ``DPKT``, ``Scapy``, ``PyShark`` and ``PCAPKit`` (#412). That brings the + built-in set to seven engines; 3.11 is the last interpreter on which every one + of them can run, and even there two of them cannot coexist. +* **Added** -- ``EngineBase.unsupported_reason``, a preflight every engine + answers and ``Extractor.run`` consults before anything is imported. Asking for + an engine that cannot run in the current environment now gives one warning + naming the real cause -- a Python version, a missing ``tshark``, a missing + ``libpcap``, the wrong ``pcap`` distribution -- and a clean fall back to + ``pcapkit``'s own parser, rather than an error from inside the third-party + package (#396, #405). +* **Added** -- ESP parsing and construction [:rfc:`4303`], with optional payload + decryption and ICV verification through ``cryptography`` + (``pip install pypcapkit[crypto]``) (#378). Keys reach the dissector through a + new caller-state channel, ``pcapkit.corekit.context``, surfaced as the + ``context=`` keyword on ``extract()`` and ``Extractor``, carrying an + ``esp.SecurityAssociation``. Nothing here raises: with no association the SPI + and sequence number are still reported and the ciphertext is left opaque, with + ``status=NO_SA``, and a failed decryption or ICV check is recorded on the + parsed result the same way. Extended Sequence Numbers, TFC padding and + anti-replay are not implemented, and an unsupported cipher or MAC is refused + with a clear error rather than half-processed. +* **Added** -- SCTP as a transport protocol [:rfc:`9260`]: all 13 chunk types, 8 + chunk parameters and 13 error causes, with the CRC32c both recorded and + verifiable -- it covers the SCTP packet alone, with no IP pseudo-header, so it + can be checked from the SCTP bytes. Upper layers register on the DATA chunk's + Payload Protocol Identifier through ``register_sctp``, not on a port (#379). +* **Added** -- NGAP over SCTP (3GPP TS 38.413), decoding aligned PER through + ``pycrate`` (``pip install pypcapkit[NGAP]``) (#251, #417). Decoding is generic + by ASN.1 shape rather than per-procedure, so all 81 elementary procedures and + 438 protocol IEs work and a new 3GPP release needs no code change. Registered + as a default on PPID 60 and 66, but only 60 decodes: PPID 66 is an NGAP PDU + inside a DTLS record, and there is no DTLS dissector. ``pycrate`` is + deliberately excluded from the ``all`` extra -- it is LGPL-2.1+ where this + package is BSD-3-Clause, and lands some 238 MB to obtain one module. +* **Added** -- the Mobility Header registry, completed (#383, #437). The + :rfc:`5568` fast-handover messages and options first, then all 24 registered + message data types, 70 of the 71 registered options -- with nested sub-option + registries for the flow identification, access network identifier, + quality-of-service and LMA-controlled MAG parameter families -- and all 4 CGA + extensions. Only the CGA Parameters option remains on the generic handler. +* **Added** -- dispatch entries for dissectors that existed but were reachable + from no registry (#436): FTP-DATA on TCP 20, HTTP/1 on TCP 8080, HTTP on UDP + 8080, ``L2TPv2`` on UDP 1701 and OSPF at ``TransType`` 89. ``VLAN`` became an + abstract base with ``C_Tag`` (802.1Q) and ``S_Tag`` (802.1ad) as concrete + subclasses, so a Q-in-Q frame no longer collapses into one opaque ``Raw``; + ``L2TP`` likewise became a base, with ``L2TPv2`` carrying the :rfc:`2661` + implementation. +* **Added** -- ``pcapkit.utilities.logging`` as a real interface: + ``get_logger()`` for per-module children, ``configure()`` to set level, + handler, stream, format or propagation at runtime, ``reset()`` to return to + library-neutral, and ``ensure_output()``. Seventeen modules now log under their + own ``__name__``, so a consumer can silence ``pcapkit.foundation.registry`` + while keeping ``pcapkit.foundation.extraction`` (#384). +* **Added** -- ``conflict`` on the reassembly data models: absolute, inclusive + ranges where two fragments claimed the same span with different bytes, which + was previously lost silently on both the IP (#482) and TCP (#443, #478) paths. +* **Added** -- an end-to-end test tier (#376), sample-capture generators so a + fresh clone can rebuild every fixture (#340), a Dockerised engine benchmark + covering every supported Python version (#410), and registry round-trip + coverage that records the entries which cannot close the cycle rather than + skipping them (#440, #504). +* **Changed** -- ``pcapkit`` no longer configures logging at import. It + installs a ``NullHandler`` and sets no level, so verbosity is inherited from + the application instead of being seized by whichever library was imported + second; the old stderr handler stays as the ``PCAPKIT_DEVMODE`` opt-in. Three + consequences worth knowing: the previous behaviour is + ``configure(logging.INFO, stream=sys.stderr)``; 38 registry and extractor + ``info`` calls became ``debug``, so those messages are invisible even at + ``INFO``; and the handler is no longer ``logger.handlers[0]``. ``verbose=`` + output stays on stdout and is not logging (#384). +* **Changed** -- each warning is reported once per channel, and ``pcapkit`` no + longer inserts a ``simplefilter('ignore', ...)`` at the front of the + process-global ``warnings.filters`` (#362--#364, #390). The application's own + filter therefore wins now, which is the point of the change and also the sharp + edge in it: under ``-W error``, or pytest's ``filterwarnings = error``, a + pcapkit warning that used to be suppressed will raise. Suppress them + deliberately with + ``warnings.filterwarnings('ignore', category=BaseWarning)``. ``quiet=True`` + now means no record at any level and no longer sets ``sys.tracebacklimit``, + and the ``pcapkit.utilities.warnings.DEVMODE`` re-export is gone -- its + canonical home is ``pcapkit.utilities.logging``. +* **Changed** -- ``layer=`` and ``protocol=`` are honoured rather than inert. + Both were read under the wrong names, so every value a caller passed was + dropped into ``**kwargs`` and discarded; the CLI's ``-L`` also now validates + its argument instead of accepting anything. The packet context reaches the + schema layer for the first time as well, so a field the wire elides can be + resolved from its enclosing packet (#404). ``follow_tcp_stream`` dispatches on + the engine type, where both branches of the old test were dead and the native + adapter ran against every engine's frames (#402). +* **Changed** -- two reassembly and flow-tracing defaults moved (#435), and both + are visible to a caller. ``Datagram.completed`` widened from ``bool`` to a + ``Completion`` enumeration (``COMPLETE``, ``PARTIAL``, ``TIMEOUT``); only + ``COMPLETE`` is truthy, so ``if datagram.completed:`` is unaffected but + ``datagram.completed == True`` no longer holds. TCP flow tracing is + **bidirectional by default**, which merges each flow's two halves and closes + one only once both have FINed -- 331 flows become 111 on the sample HTTP + capture, the difference being single-frame stray tails; pass + ``trace_bidirectional=False`` for the old behaviour. IP reassembly also gained + the 60-second timeout [:rfc:`1122`, :rfc:`8200`], clocked off the capture's own + timestamps rather than the wall clock, tunable with ``reasm_timeout=``; TCP + reassembly gets no timeout by default. ``trace_analyse=`` is new, and + reassembles each traced flow's application layer. +* **Changed** -- conflicting TCP overlaps resolve first-write-wins, per + :rfc:`9293` section 3.10, where they had silently resolved last-write-wins + (#443, #478). A deliberate behaviour break, and a narrow one: a conforming + retransmission carries identical bytes, so nothing changes for it. IP fragment + reassembly keeps last-write-wins, because :rfc:`791` specifies the opposite + resolution, and records the disagreement instead (#482). +* **Changed** -- ``Probe``, ``CipherSuite`` and ``IntegritySuite`` are ``Info`` + subclasses rather than ``typing.NamedTuple``, and no ``NamedTuple`` remains in + the package. They are Mappings now, so ``len()`` and iteration yield field + names rather than values. +* **Changed** -- renames with no compatibility alias left behind: + ``HoleDiscriptor`` is spelled ``HoleDescriptor`` and its package alias + ``TCP_HoleDiscriptor`` is ``TCP_HoleDescriptor`` (#350); PCAP-NG ``Option`` + subclasses spell the namespace class keyword ``ns=`` instead of ``namespace=`` + (#439); and ``examples/sample`` and ``examples/samples`` -- one letter apart, + holding different things -- are now ``examples/captures`` and + ``examples/generators``. +* **Changed** -- extraction is around 46% faster on a 1,117-frame HTTP capture, + with byte-identical output (#420). A reassembled datagram's payload is now + analysed on first read rather than eagerly, which cuts IP reassembly's own + cost by 90.7% and TCP's by 23.7% -- IP reassembly submits a datagram for every + frame, fragmented or not (#424). Flow tracing over the same capture went from + 1416.6 ms to 744.0 ms, because the flow dumper had been handing each record to + a ``Frame`` constructor that re-dissected the whole protocol stack to return + bytes it had just been given; options are no longer parsed twice either + (#427). All output compared byte-for-byte across the sample captures in each + case. +* **Fixed** -- next-layer, option, chunk, block and parameter dispatch all read + ``defaultdict`` registries, so a lookup miss inserted the key into class-level + state shared by every later instance, after which a legitimate ``register_*`` + call warned that the code was already registered. Every read now goes through + a lookup that does not grow the table, and ``IPv4.__option__`` and + ``HIP.__parameter__`` became inspectable class attributes rather than names + assembled at call time (#426, #428, #429, #434). One break comes with it: a + tuple-registered handler pair written to the documented + ``OptionParser``/``OptionConstructor`` signature now works where it could + previously never be called at all, and a pair written with an explicit leading + ``self`` -- the only shape that used to work -- now does not. +* **Fixed** -- on Python 3.10 and older, no ``Schema`` subclass got its own + ``_abc_impl``: all of them fell through to ``collections.abc.Mapping``'s, so a + single ``isinstance`` or ``issubclass`` answer poisoned every later question + about that class for the rest of the process. A terminating PCAP-NG + ``EndRecord`` tested ``True`` as an ``IPv4Record`` (#439). +* **Fixed** -- construction, which was broken in several places at once: the + generated typed ``__init__`` was never installed, so ``__post_init__`` did not + run and a schema built from a subset of its fields could not be packed at all + -- ``UDP(srcport=53, dstport=5353)`` now packs (#430); IPv6 and Mobility Header + option padding was wrong, leaving construction wholly broken (#398); + ``HTTP.make`` called the versioned ``make`` unbound, so every real call raised + ``TypeError`` (#452, #462); and ``IPv4._make_data`` returned the fragment + offset in octets where the wire wants 8-octet units, and read ``data.options`` + on a packet that has none (#494, #499). +* **Fixed** -- a truncated or under-declared area no longer parses + "successfully", and no longer wedges the process. The option and list loops + could spin forever with no exception on a truncated area, reachable from + untrusted input through HOPOPT, IPv6-Opts, MH, HIP and SCTP; each iteration + must now advance the stream by at least one octet, and the error names the + option, the offset and the octets remaining (#431, #432). Separately, + wire-derived lengths in ``ipv6_opts``, CALIPSO, MPL, REG_INFO and four HIP list + callbacks underflowed below zero, which ``ListField``'s own + ``while length > 0`` then turned into a silent empty list; they are floored and + raise instead (#449, #456, #460, #463). +* **Fixed** -- field widths and units, each measured against the specification + rather than inferred: HIP's ``TRANSPORT_FORMAT_LIST``, ``NAT_TRAVERSAL_MODE`` + and ``ESP_TRANSFORM`` list entries are two octets, not one [:rfc:`7401`, + :rfc:`5770`, :rfc:`7402`] (#463, #472); the MN-ID option sizes from its + subtype, not from ``identifier``'s Python type (#448, #464, #467); + IPv6-Route's ``Hdr Ext Len`` is computed in 8-octet units on both sides + [:rfc:`8200`] (#487, #489); and the Fast Binding Update and Acknowledgment + Lifetimes are plain seconds [:rfc:`5568`], not :rfc:`6275`'s four-second units + (#502). +* **Fixed** -- stdlib exceptions leaking out where the library's own were + promised: a malformed IP field value raised a bare ``ValueError`` instead of + ``FieldValueError`` (#465); a ``bool`` address was silently packed as + ``0.0.0.1`` or ``0.0.0.0``, ``bool`` being an ``int`` subclass (#491, #500); + and ``@prepare`` discarded extra arguments silently and treated a *declared* + zero length as end of stream, which is now distinguished from a genuinely + exhausted one and raises ``StreamEOFError`` (#454, #458). +* **Fixed** -- the engine adapters, which were quietly wrong rather than loud. + The ``dpkt`` toolkit split TCP and IPv4 headers at their fixed struct size + instead of their real length, so option octets overwrote payload in the + sequence-indexed reassembly buffer; it also read an ``ipv6_frag.nh`` that + ``dpkt`` does not have, and passed fragment offsets unscaled (#351, #370, + #385, #395). ``scapy`` never loaded its layer registry, so that engine did not + dissect at all (#409), and its IPv4 fragment offset reached reassembly + unscaled (#483, #484). The four IPv6 adapters disagreed about whether the + 8-octet Fragment header belongs to ``ihl``, ``header`` and ``tl``; per + :rfc:`8200` section 4.5 it belongs to none of them, and all four now agree + (#415, #424). +* **Fixed** -- PCAP and PCAP-NG output and parsing: ``bytes(frame)`` returned the + *next* frame's octets, ``files=True`` wrote names like ``Frame 1..json``, and a + PCAP-NG ``timestamp_epoch`` was shifted by the reading host's timezone (#403); + seven further parser defects (#341--#347, #371) and, on the write path, four + more block-parsing ones (#388); ``BitField`` packed every named bit as set + (#359, #374); the extension-header walk failed to advance past the last IPv6 + extension header (#348, #373); and IPv6 fragment offsets went unscaled, with + reassembly keyed on the flow label -- optional, and routinely zero, so distinct + datagrams collapsed together -- rather than on the fragment identification + (#389). +* **Fixed** -- TCP reassembly mixed absolute sequence numbers with + buffer-relative slicing, so on any capture carrying a SYN with a realistic + initial sequence number incomplete datagrams were dropped silently, and the + ``completed=False`` branch of the public API was unreachable (#349, #376). +* **Fixed** -- constant lookups that rejected a value the registry defines. + ``RouterAlert(0)`` is the only value :rfc:`2113` defines and the one IGMP, + RSVP and MLD actually send, and it was discarded because the vendor crawler + skipped a header row IANA's CSV does not have; IPX ``Socket(0)`` is that + protocol's own default, so ``bytes(IPX(...))`` crashed on its own defaults; + and two FTP ``_missing_`` overrides were plain methods rather than + classmethods, so every unregistered value raised ``TypeError`` instead of + extending the enumeration (#492, #503). +* **Fixed** -- ``format='text'`` raised ``AttributeError`` before writing + anything, naming a ``dictdumper.Text`` that has never existed. It now points + at ``Tree``, as the ``'txt'`` alias beside it already did. +* **Fixed** -- 45 places where a documentation page contradicted the code + (#413), ambiguous cross-references and five autodoc signature failures (#416), + and ``Extractor``'s documented exception plus 40 phantom or stale ``Args:`` + labels (#501). + +Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both +2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so +resolved only by ``pip install --pre``. ``1.5.0b1`` half-shipped: the tag, the +GitHub release and the Conda deployments landed, but PyPI rejected the wheel +because ``twine check`` found a Sphinx-only ``:mod:`` role in ``README.rst``, +which ``pyproject.toml`` declares as the dynamic long description. ``1.5.0b2`` +is what reshipped it -- the release workflow is version-driven, so an existing +version cannot republish -- and ``1.5.0b3`` followed the CI change that stops a +TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/index.rst b/docs/source/index.rst index c0a6948a9c..950f4f8442 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -30,6 +30,7 @@ construction and analysis library. ext demo pep + changelog About ===== diff --git a/pyproject.toml b/pyproject.toml index 2d69ea4ee8..39d4db1349 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -100,7 +100,7 @@ dependencies = [ homepage = "https://jarryshaw.github.io/PyPCAPKit/" documentation = "https://jarryshaw.github.io/PyPCAPKit/" repository = "https://github.com/JarryShaw/PyPCAPKit" -changelog = "https://github.com/JarryShaw/PyPCAPKit/releases" +changelog = "https://jarryshaw.github.io/PyPCAPKit/changelog.html" [project.scripts] pcapkit-cli = "pcapkit.__main__:main" diff --git a/tests/project/test_changelog_md.py b/tests/project/test_changelog_md.py new file mode 100644 index 0000000000..66e793935f --- /dev/null +++ b/tests/project/test_changelog_md.py @@ -0,0 +1,448 @@ +# -*- coding: utf-8 -*- +"""Tests for :file:`util/changelog_md.py`, the ``CHANGELOG.md`` generator. + +``CHANGELOG.md`` is a derived artefact: it is the newest +:file:`docs/source/changelog/.rst` entry converted to Markdown, so that +the ``Create Release`` workflow's release body and the source distribution get +Markdown without a second copy of the history to keep in step by hand. + +The conversion is six mechanical rules over a small reStructuredText subset, and +there is one test per rule. Each asserts both halves -- that the Markdown form +arrived *and* that the reStructuredText form is gone -- because a rule that fires +in the wrong place and a rule that does not fire at all both leave markup in a +published release body, and only checking for the new spelling would miss the +second. + +The rest cover the two things that are easy to get wrong and invisible when they +are: which entry counts as "newest" (the toctree's first line, not a sort of the +version strings), and ``--check``, which is what a CI gate calls and is worthless +if it cannot fail. + +Almost everything here builds its own two-file changelog tree in a temporary +directory rather than reading the repository's. That keeps the rule tests honest +-- each entry is written to exercise one rule, instead of hoping the real +changelog happens to contain the construct -- and it keeps them passing on a +source tarball, where :file:`docs/` has been pruned away. + +""" + +from __future__ import annotations + +import contextlib +import importlib.util +import io +import pathlib +import re +import shutil +import tempfile +import textwrap +import unittest + +ROOT = pathlib.Path(__file__).resolve().parents[2] + + +def _load_generator(): + """Load :file:`util/changelog_md.py` as a module. + + ``util/`` is a directory of scripts rather than a package, so there is no + import path to it. The script has no import-time side effects, which is what + makes loading it by location safe. + + """ + path = ROOT / 'util' / 'changelog_md.py' + spec = importlib.util.spec_from_file_location('changelog_md', path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +changelog_md = _load_generator() + + +#: One entry exercising all six rules at once: a setext heading, a wrapped +#: paragraph, ``*`` bullets whose text wraps, an ``:rfc:`` role, a double-backtick +#: literal, and a footnote reference with its definition. +ENTRY = """\ +9.9.9 -- 2026-01-02 +=================== + +An entry whose prose is wrapped across +two source lines, so that rule 6 has +something to collapse. + +* **Added** -- ESP parsing [:rfc:`4303`], with a ``literal`` in it + and a continuation line. +* **Fixed** -- PCAP-NG support [1]_. + +.. [1] PCAP-NG is specified by a draft, not by an RFC. +""" + +INDEX = """\ +========= +Changelog +========= + +Preamble. + +.. toctree:: + :maxdepth: 1 + + changelog/9.9.9 +""" + + +class ChangelogTreeMixin: + """Builds a throwaway changelog tree for one test.""" + + def make_tree(self, entry: str = ENTRY, index: str = INDEX, + version: str = '9.9.9') -> pathlib.Path: + """Write *index* and *entry* into a temporary tree; return the index path.""" + root = pathlib.Path(tempfile.mkdtemp()) + self.addCleanup(shutil.rmtree, root, True) + + (root / 'changelog').mkdir() + (root / 'changelog' / f'{version}.rst').write_text(entry, encoding='utf-8') + index_path = root / 'changelog.rst' + index_path.write_text(index, encoding='utf-8') + return index_path + + def convert(self, entry: str = ENTRY) -> str: + """Convert *entry* through the generator, as ``render`` would.""" + return changelog_md.convert(entry) + + +class ConversionRuleTests(ChangelogTreeMixin, unittest.TestCase): + """One test per conversion rule.""" + + def test_rule_1_setext_heading_becomes_atx(self) -> None: + markdown = self.convert() + + self.assertTrue( + markdown.startswith('## 9.9.9 -- 2026-01-02\n'), + f'expected an ATX heading first, got {markdown[:60]!r}', + ) + # The underline must be consumed, not merely followed by a heading. + self.assertNotRegex(markdown, r'(?m)^=+$') + + def test_rule_2_rfc_role_becomes_a_datatracker_link(self) -> None: + markdown = self.convert() + + self.assertIn('[RFC 4303](https://datatracker.ietf.org/doc/html/rfc4303)', markdown) + self.assertNotIn(':rfc:', markdown) + + def test_rule_3_double_backtick_literal_becomes_a_code_span(self) -> None: + markdown = self.convert() + + self.assertIn('`literal`', markdown) + self.assertNotIn('``', markdown) + + def test_rule_4_star_bullets_become_dash_bullets(self) -> None: + markdown = self.convert() + + bullets = [line for line in markdown.split('\n') if line.startswith('- ')] + self.assertEqual(len(bullets), 2, f'expected two dash bullets, got {bullets!r}') + self.assertNotRegex(markdown, r'(?m)^\*[ \t]') + + def test_rule_5_footnotes_become_github_footnotes(self) -> None: + markdown = self.convert() + + self.assertIn('[^1]', markdown) + self.assertRegex(markdown, r'(?m)^\[\^1\]: PCAP-NG is specified by a draft') + self.assertNotIn('[1]_', markdown) + self.assertNotRegex(markdown, r'(?m)^\.\. \[1\]') + + def test_rule_6_paragraphs_and_bullets_are_unwrapped(self) -> None: + markdown = self.convert() + + # A GitHub release body renders a single newline as a hard break, so every + # block has to be one line: the source's own wrapping must be gone. + self.assertIn( + 'An entry whose prose is wrapped across two source lines, so that ' + 'rule 6 has something to collapse.', + markdown, + ) + self.assertIn('with a `literal` in it and a continuation line.', markdown) + + # Nothing but blank lines separates the blocks, so no line may be a bare + # continuation of the one above it. + for line in markdown.split('\n'): + if line and not line.startswith(('## ', '- ', '[^')): + self.assertTrue( + line[0].isupper() or line.startswith('---'), + f'{line[:50]!r} looks like a wrapped continuation line', + ) + + +class NewestEntryTests(ChangelogTreeMixin, unittest.TestCase): + """How the generator decides which entry is the current release.""" + + def test_newest_is_the_toctree_head_not_a_version_sort(self) -> None: + index = self.make_tree( + index=textwrap.dedent("""\ + ========= + Changelog + ========= + + .. toctree:: + :maxdepth: 1 + + changelog/1.10.0 + changelog/1.9.0 + """), + version='1.10.0', + ) + (index.parent / 'changelog' / '1.9.0.rst').write_text(ENTRY, encoding='utf-8') + + version, entry = changelog_md.newest(index) + + # ``1.9.0`` sorts above ``1.10.0`` lexically and below it under PEP 440; + # the toctree settles it without the generator having to know either rule. + self.assertEqual(version, '1.10.0') + self.assertEqual(entry.name, '1.10.0.rst') + + def test_toctree_options_and_blank_lines_are_not_entries(self) -> None: + index = self.make_tree() + + self.assertEqual(changelog_md.read_toctree(index), ['changelog/9.9.9']) + + def test_explicit_title_form_is_reduced_to_the_docname(self) -> None: + index = self.make_tree(index=INDEX.replace( + ' changelog/9.9.9', ' The 9.9.9 release ')) + + self.assertEqual(changelog_md.read_toctree(index), ['changelog/9.9.9']) + + def test_dotted_version_is_not_mistaken_for_a_file_extension(self) -> None: + # ``pathlib.Path.with_suffix`` would turn ``changelog/9.9.9`` into + # ``changelog/9.9.rst``, because a version string ends in what looks like + # an extension. + version, entry = changelog_md.newest(self.make_tree()) + + self.assertEqual(version, '9.9.9') + self.assertEqual(entry.name, '9.9.9.rst') + + def test_missing_toctree_is_an_error(self) -> None: + index = self.make_tree(index='=========\nChangelog\n=========\n') + + with self.assertRaises(ValueError): + changelog_md.read_toctree(index) + + def test_empty_toctree_is_an_error(self) -> None: + index = self.make_tree(index='.. toctree::\n :maxdepth: 1\n') + + with self.assertRaises(ValueError): + changelog_md.read_toctree(index) + + def test_toctree_head_naming_a_missing_file_is_an_error(self) -> None: + index = self.make_tree(index=INDEX.replace('9.9.9', '9.9.8')) + + with self.assertRaises(FileNotFoundError): + changelog_md.newest(index) + + +class ResidualMarkupTests(ChangelogTreeMixin, unittest.TestCase): + """The guard against six regexes quietly copying through what they cannot convert.""" + + def test_an_unconvertible_directive_is_fatal(self) -> None: + index = self.make_tree(entry=ENTRY + '\n.. note::\n\n Not in the subset.\n') + + with self.assertRaises(changelog_md.ResidualMarkupError) as error: + changelog_md.render(index) + + self.assertIn('directive', str(error.exception)) + + def test_an_unconverted_role_is_fatal(self) -> None: + index = self.make_tree(entry=ENTRY + '\nSee :mod:`pcapkit.const` for more.\n') + + with self.assertRaises(changelog_md.ResidualMarkupError) as error: + changelog_md.render(index) + + self.assertIn('role', str(error.exception)) + + def _reject(self, tail: str, expected: str) -> str: + """Assert an entry ending in *tail* is refused, naming *expected*.""" + index = self.make_tree(entry=ENTRY + tail) + + with self.assertRaises(changelog_md.ResidualMarkupError) as error: + changelog_md.render(index) + + message = str(error.exception) + self.assertIn(expected, message) + return message + + def test_an_inline_hyperlink_reference_is_fatal(self) -> None: + # The worst of the five: GFM renders ```text `_`` as broken inline + # code followed by a stray underscore, so the link target vanishes + # silently rather than merely looking wrong. + self._reject('\nSee `the RFC index `_ for more.\n', + 'hyperlink reference') + + def test_an_anonymous_hyperlink_reference_is_fatal(self) -> None: + self._reject('\nSee `the RFC index `__ for more.\n', + 'hyperlink reference') + + def test_a_substitution_reference_is_fatal(self) -> None: + self._reject('\nShipped in |version| of the library.\n', 'substitution') + + def test_a_field_list_is_fatal(self) -> None: + self._reject('\n:Author: Jarry Shaw\n:Version: 9.9.9\n', 'field list') + + def test_a_line_block_is_fatal(self) -> None: + # Rule 6 joins these into ``| One line | Another line``, which GFM can + # read as a table row. + self._reject('\n| One line\n| Another line\n', 'line block') + + def test_a_grid_table_is_fatal(self) -> None: + self._reject( + '\n' + '+----------+----------+\n' + '| Column A | Column B |\n' + '+==========+==========+\n' + '| a | b |\n' + '+----------+----------+\n', + 'grid table', + ) + + def test_a_simple_table_is_fatal(self) -> None: + self._reject( + '\n' + '======== ========\n' + 'Column A Column B\n' + '======== ========\n' + 'a b\n' + '======== ========\n', + 'simple table', + ) + + def test_a_sub_heading_underline_joined_into_the_prose_is_fatal(self) -> None: + # Rule 1 only consumes ``=`` underlines, so a ``-`` underlined sub-heading + # is joined onto the heading text by rule 6 and would otherwise disappear + # into a paragraph. + self._reject('\nA sub heading\n-------------\n\nSome prose.\n', + 'setext underline joined') + + def test_an_over_long_equals_underline_is_fatal(self) -> None: + # Rule 1 fires only when the underline is exactly as long as the title, + # but reStructuredText merely requires it to be no shorter -- so an + # over-long one falls past rule 1 and is joined like any other underline. + self._reject('\nA sub heading\n==================\n\nSome prose.\n', + 'setext underline joined') + + def test_the_generated_files_own_trailer_is_not_a_leftover(self) -> None: + # The trailer's ``---`` sits alone on its line, and the joined-underline + # pattern requires text before the run, so the guard cannot fire on the + # file it is protecting. Checked on the whole rendered output, trailer + # included, rather than on the body render() already checks. + self.assertEqual(changelog_md.residual(changelog_md.render(self.make_tree())), []) + + def test_markup_quoted_inside_a_code_span_is_not_a_leftover(self) -> None: + # The real 1.5.0 entry says ``a Sphinx-only ``:mod:`` role``, which + # converts to the code span ```:mod:```. That is prose about a role, not a + # role that escaped rule 2, and it must not trip the guard. + markdown = self.convert( + ENTRY + '\nPyPI rejected a Sphinx-only ``:mod:`` role in ``README.rst``.\n') + + self.assertEqual(changelog_md.residual(markdown), []) + + def test_the_real_entries_are_all_within_the_subset(self) -> None: + directory = changelog_md.INDEX.parent / 'changelog' + if not directory.is_dir(): + self.skipTest(f'{directory} is absent (docs/ is pruned from a source tarball)') + + for entry in sorted(directory.glob('*.rst')): + with self.subTest(entry=entry.name): + self.assertEqual( + changelog_md.residual(changelog_md.convert( + entry.read_text(encoding='utf-8'))), + [], + ) + + +class CheckModeTests(ChangelogTreeMixin, unittest.TestCase): + """``--check`` is what a CI gate calls, so it has to fail when it should.""" + + def _run(self, *argv: str) -> tuple[int, str, str]: + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + status = changelog_md.main(list(argv)) + return status, out.getvalue(), err.getvalue() + + def test_check_passes_when_the_file_is_in_step(self) -> None: + index = self.make_tree() + output = index.parent / 'CHANGELOG.md' + output.write_text(changelog_md.render(index), encoding='utf-8') + + status, stdout, _ = self._run('--check', '--index', str(index), + '--output', str(output)) + + self.assertEqual(status, 0) + self.assertIn('in step', stdout) + + def test_check_fails_on_a_one_character_edit(self) -> None: + index = self.make_tree() + output = index.parent / 'CHANGELOG.md' + generated = changelog_md.render(index) + output.write_text(generated.replace('ESP parsing', 'ESP parsinG', 1), encoding='utf-8') + + status, _, stderr = self._run('--check', '--index', str(index), + '--output', str(output)) + + self.assertEqual(status, 1) + self.assertIn('drifted', stderr) + self.assertIn('---', stderr) # a unified diff, so the reviewer sees what moved + + def test_check_fails_when_the_file_is_missing(self) -> None: + index = self.make_tree() + + status, _, stderr = self._run('--check', '--index', str(index), + '--output', str(index.parent / 'CHANGELOG.md')) + + self.assertEqual(status, 1) + self.assertIn('drifted', stderr) + + def test_writing_then_checking_round_trips(self) -> None: + index = self.make_tree() + output = index.parent / 'CHANGELOG.md' + + write_status, _, _ = self._run('--index', str(index), '--output', str(output)) + check_status, _, _ = self._run('--check', '--index', str(index), + '--output', str(output)) + + self.assertEqual((write_status, check_status), (0, 0)) + + def test_generated_file_carries_the_trailer(self) -> None: + markdown = changelog_md.render(self.make_tree()) + + self.assertIn(changelog_md.DOCS_URL, markdown) + self.assertTrue(markdown.endswith('\n')) + + +class RepositoryStateTests(unittest.TestCase): + """Checks against the repository rather than a fixture.""" + + def test_docs_url_matches_the_pyproject_changelog_url(self) -> None: + # ``MANIFEST.in`` prunes ``docs/``, so the trailer's link is the only route + # from the shipped changelog to the rest of the history. If the two drift, + # the sdist points at a page that may not exist. + pyproject = (ROOT / 'pyproject.toml').read_text(encoding='utf-8') + declared = re.search(r'(?m)^changelog\s*=\s*"([^"]+)"', pyproject) + + self.assertIsNotNone(declared, 'pyproject.toml declares no [project.urls].changelog') + assert declared is not None + self.assertEqual(changelog_md.DOCS_URL, declared.group(1)) + + def test_committed_changelog_is_in_step_with_the_newest_entry(self) -> None: + if not changelog_md.INDEX.is_file(): + self.skipTest(f'{changelog_md.INDEX} is absent') + if not changelog_md.OUTPUT.is_file(): + self.skipTest(f'{changelog_md.OUTPUT} is absent') + + self.assertEqual( + changelog_md.OUTPUT.read_text(encoding='utf-8'), + changelog_md.render(), + 'CHANGELOG.md is stale; regenerate it with util/changelog_md.py', + ) + + +if __name__ == '__main__': + unittest.main() diff --git a/util/changelog_md.py b/util/changelog_md.py new file mode 100644 index 0000000000..31068f542a --- /dev/null +++ b/util/changelog_md.py @@ -0,0 +1,483 @@ +# -*- coding: utf-8 -*- +"""Generate the root ``CHANGELOG.md`` from the newest per-version changelog entry. + +``docs/source/changelog/.rst`` is the single source of every changelog +entry, and Sphinx renders the lot as the project's history. ``CHANGELOG.md`` is a +*derivative* of exactly one of them -- the version being released -- because its +two consumers, the ``Create Release`` workflow's release body and the source +distribution, both read Markdown rather than reStructuredText. + +Nothing edits ``CHANGELOG.md`` by hand. That is the point: there is no "promote +the latest entry into the history" step to forget, because the entry was written +in the history in the first place, and the two surfaces cannot drift while one is +generated from the other. It is the same arrangement as ``pcapkit/const/**``, +which is generated from ``pcapkit/vendor/**``. + +Markdown at the repository root is a deliberate exception to the "documentation is +reStructuredText" rule, justified by those two consumers. + +Which entry is "newest" +----------------------- + +**The first entry of the toctree in** ``docs/source/changelog.rst``. + +Not a sort of the version strings. ``1.5.0b3``, ``1.5.0`` and ``1.10.0`` order +correctly under :pep:`440` and incorrectly under every lexical comparison, and +reimplementing :pep:`440` here to answer a question the toctree already answers +would be a second source of truth rather than a convenience. Not the file +modification time either, which says when a file was touched rather than which +release is current. + +The toctree has to exist for Sphinx, and it has to be newest-first for the +rendered history to read correctly, so it is *already* the ordering of record. +Reading it means a release is added in exactly one place -- a new entry file, and +its name at the top of the toctree -- and this script follows. + +The conversion +-------------- + +The entries are written in a deliberately small reStructuredText subset, so the +conversion is six mechanical rules rather than a document converter: + + 1. the setext version heading becomes an ATX ``##`` heading; + 2. ``:rfc:`NNNN``` becomes a Markdown link to the RFC on the IETF datatracker; + 3. ``double backtick`` literals become single-backtick code spans; + 4. ``*`` bullets become ``-`` bullets; + 5. ``[n]_`` / ``.. [n]`` footnotes become GitHub's ``[^n]`` / ``[^n]:``; + 6. each paragraph and each bullet is unwrapped onto one line. + +Everything else -- ``**strong**``, ``*emphasis*``, and the ``#nnn`` issue +references that GitHub autolinks in a release body -- is already valid Markdown +and is copied verbatim. + +Rule 6 is the only one about the *consumer* rather than the syntax. A GitHub +release body is rendered like a comment, where a single newline is a hard line +break rather than a space, so a paragraph wrapped at 79 columns renders as a +column of short ragged lines. A paragraph with no newline inside it renders the +same either way, which is why unwrapping is the safe direction. + +Six regexes cannot recognise everything reStructuredText can express, and a rule +that does not fire copies its construct straight through as literal text. That is +how a release body ends up with ``.. note::`` printed in it -- published to GitHub +and PyPI, where it is expensive and irreversible. So :func:`residual` re-reads the +converted entry and :func:`render` refuses to emit while anything is left. + +What it guards, exactly -- the claim is kept level with the code, because a guard +believed to be broader than it is, is worse than no guard: + + * a ``double backtick`` literal rule 3 did not reach; + * an interpreted-text role, ``:mod:`x```, rule 2 does not know; + * a directive or comment, any line opening ``..`` -- which also catches the + ``.. _name: url`` target that a bare ``name_`` reference needs, so that form is + covered through its definition rather than through the reference; + * a ``*`` bullet rule 4 missed; + * a ``[n]_`` footnote reference rule 5 missed; + * an inline or phrase hyperlink reference, ```text `_``, which GFM renders + as broken inline code plus a stray underscore, losing the link outright; + * a substitution reference, ``|version|``, which has no Markdown equivalent; + * a field list, ``:Author: ...``, which rule 6 would join onto one line; + * a line block, ``| line``, which rule 6 joins into something GFM may read as a + table row; + * a grid or simple table, which rule 6 mangles onto one line; + * a setext underline, whether left on its own line or joined into the prose + above it by rule 6 -- the second is how a sub-heading would silently + disappear into a paragraph. + +Two things it deliberately does **not** catch. A bare ``name_`` reference on its +own, because a pattern loose enough to spot a trailing underscore in prose fires on +ordinary identifiers; its target line is caught instead, as above. And a construct +that rule 6 joins into the *middle* of a line rather than the start of one, since +every line-anchored pattern here then has nothing to anchor to -- a field list +buried inside a bullet, say. Both would need the guard to run before unwrapping, +which is a larger change than the insurance is worth. + +Every pattern was measured against all 37 committed entries and matches none of +them, so the guard costs nothing until an entry actually leaves the subset. + +Usage +----- + +.. code-block:: shell + + python util/changelog_md.py # regenerate CHANGELOG.md + python util/changelog_md.py --check # exit non-zero if it has drifted + +``--check`` regenerates into memory, compares, and prints a unified diff of what +moved. It is what a CI gate calls, and it is the same shape as the byte- +reproduction gate over the generated ``pcapkit.const`` modules. + +""" + +from __future__ import annotations + +import argparse +import difflib +import pathlib +import re +import sys +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from typing import Optional, Sequence + +#: Repository root, taken from this file's location rather than the working +#: directory, so the script gives the same answer run from anywhere. +ROOT = pathlib.Path(__file__).resolve().parent.parent + +#: The page carrying the changelog title, preamble and toctree. The toctree in it +#: is the ordering of record; see the module docstring. +INDEX = ROOT / 'docs' / 'source' / 'changelog.rst' + +#: The generated artefact. +OUTPUT = ROOT / 'CHANGELOG.md' + +#: Base of the RFC links rule 2 emits. This is the form Sphinx's own ``:rfc:`` +#: role resolves to, so the Markdown and the rendered documentation point at the +#: same page; docutils on its own would resolve the role to a ``tools.ietf.org`` +#: URL instead. +RFC_URL = 'https://datatracker.ietf.org/doc/html/rfc' + +#: Where the full rendered history lives. Must match ``[project.urls].changelog`` +#: in ``pyproject.toml`` -- ``MANIFEST.in`` prunes ``docs/``, so in a source +#: distribution this is the only route from the shipped entry to the rest of the +#: history. ``tests/project/test_changelog_md.py`` asserts the two agree. +DOCS_URL = 'https://jarryshaw.github.io/PyPCAPKit/changelog.html' + +#: Closing note on the generated file, pointing at the rest of the history. +#: +#: Deliberately short. This file is read on a GitHub release page and on PyPI, +#: where a reader has no repository in front of them -- so an explanation of +#: which ``docs/source/changelog/.rst`` the entry came from is noise to +#: them, while the link is the one thing that is useful everywhere. +TRAILER = ( + '---\n' + '\n' + f'Full changelog: <{DOCS_URL}>\n' +) + +#: A single-backtick code span. Replaced by :data:`_SPAN` before :func:`residual` +#: looks for unconverted markup, because an entry may legitimately *discuss* +#: markup: the 1.5.0 entry says ``a Sphinx-only ``:mod:`` role``, which converts +#: to the code span ```:mod:``` and must not be read as a role that escaped rule +#: 2. Deliberately cannot match across a newline, so masking leaves every line +#: break -- and therefore every reported line number -- where it was. +_CODE_SPAN = re.compile(r'`[^`\n]*`') + +#: Stands in for a masked code span. A *marker* rather than nothing, because the +#: difference between a leftover role and prose about a role is precisely whether +#: a code span follows the ``:name:`` or encloses it: ``:mod:`pcapkit.const``` is +#: a role that escaped rule 2, and ```:mod:``` is a sentence mentioning one. +_SPAN = '\x00' + +#: reStructuredText that should be gone by the time conversion finishes. Checked +#: against the masked text, and each one means a rule did not fire. Every pattern +#: matches none of the 37 committed entries; see the module docstring for what is +#: covered and what is deliberately not. +_RESIDUAL = ( + (re.compile(r':[a-zA-Z][a-zA-Z0-9_+:.-]*:' + _SPAN), + 'an unconverted interpreted-text role'), + (re.compile(r'(?m)^[ \t]*\.\.[ \t]'), 'an unconverted directive or comment'), + (re.compile(r'(?m)^[ \t]*\*[ \t]'), 'an unconverted bullet'), + (re.compile(r'\[\d+\]_'), 'an unconverted footnote reference'), + # ``\x00_`` is a masked code span followed by an underscore, which is what + # ```text `_`` and its anonymous ```text `__ form both reduce to. + (re.compile(_SPAN + '_'), + 'an unconverted hyperlink reference (the link target would be lost)'), + (re.compile(r'\|\S[^|\n]*\|'), 'an unconverted substitution reference'), + (re.compile(r'(?m)^[ \t]*:[^:\n]+:[ \t]'), 'an unconverted field list'), + (re.compile(r'(?m)^[ \t]*\|[ \t]'), 'an unconverted line block'), + (re.compile(r'(?m)^[ \t]*\+[-=]+\+'), 'an unconverted grid table'), + (re.compile(r'(?m)^[ \t]*=+[ \t]+=+'), 'an unconverted simple table'), + (re.compile(r'(?m)^=+$'), 'a leftover setext underline'), + (re.compile(r'(?m)^~+$'), 'a leftover setext underline'), + # A sub-heading's underline does not survive as its own line: rule 6 joins it + # onto the end of the heading text, so it has to be caught mid-line too. The + # leading ``\S[ \t]+`` is what makes this specific to the joined case, so it + # cannot fire on a ``---`` that is alone on its line -- which is what the + # generated file's own trailer is made of. + # ``(?m)`` matters: the run is followed by a newline rather than end of input, + # and a bare ``$`` only matches at the very end of the string. + # + # ``=`` belongs in the alternation even though rule 1 handles ``=`` underlines, + # because rule 1 requires the underline to be exactly as long as the title + # while reStructuredText merely requires it to be no shorter. An over-long + # underline therefore falls straight past rule 1 and gets joined like any other. + (re.compile(r'(?m)\S[ \t]+(?:={3,}|-{3,}|~{3,}|\^{3,}|"{3,})(?:[ \t]|$)'), + 'a setext underline joined into the prose above it'), +) + + +class ResidualMarkupError(RuntimeError): + """Conversion finished with reStructuredText still in the output. + + Deliberately fatal. The six rules cover the subset the entries are written + in; anything outside it would otherwise be copied through as literal text and + render as itself in a release body, which is a silent defect in a published + artefact. Failing here instead names the construct and the line. + + """ + + +def read_toctree(index: pathlib.Path) -> list[str]: + """Return the document names in *index*'s first ``toctree``, in order. + + Args: + index: The page carrying the toctree. + + Returns: + The toctree entries, newest first, each as written -- so relative to + *index*'s own directory -- with any ``Title `` wrapper reduced + to the document name. + + Raises: + ValueError: If *index* carries no ``toctree``, or it lists no entries. + + """ + lines = index.read_text(encoding='utf-8').split('\n') + + for start, line in enumerate(lines): + if line.strip() == '.. toctree::': + indent = len(line) - len(line.lstrip()) + break + else: + raise ValueError(f'no toctree directive in {index}') + + entries = [] # type: list[str] + for line in lines[start + 1:]: + if not line.strip(): + continue # blank lines are allowed inside a directive body + if len(line) - len(line.lstrip()) <= indent: + break # a dedent ends the directive + body = line.strip() + if body.startswith(':'): + continue # a directive option, such as :maxdepth: + explicit = re.search(r'<([^<>]+)>\Z', body) + entries.append(explicit.group(1) if explicit else body) + + if not entries: + raise ValueError(f'the toctree in {index} lists no entries') + return entries + + +def newest(index: pathlib.Path = INDEX) -> tuple[str, pathlib.Path]: + """Resolve the newest changelog entry from *index*'s toctree. + + Args: + index: The page carrying the toctree. + + Returns: + The newest entry's version string and the path to its file. + + Raises: + FileNotFoundError: If the toctree's first entry names a document that + does not exist, which is what half of a release addition looks like. + + """ + first = read_toctree(index)[0] + # ``+ '.rst'`` rather than :meth:`~pathlib.Path.with_suffix`: a version string + # ends in what looks like a file extension, so ``with_suffix`` on + # ``changelog/1.5.0`` yields ``changelog/1.5.rst``. + entry = index.parent / (first + '.rst') + if not entry.is_file(): + raise FileNotFoundError( + f'{index} lists {first!r} as the newest entry, but {entry} does not exist' + ) + return entry.stem, entry + + +def convert(rst: str) -> str: + """Apply the six rules to one per-version entry. + + Args: + rst: The entry's reStructuredText. + + Returns: + The entry as Markdown, ending in a single newline. + + """ + lines = rst.rstrip('\n').split('\n') + out = [] # type: list[str] + + index = 0 + while index < len(lines): + line = lines[index] + + # 1. setext heading -> ATX. Only ``=`` is used in these files, and only + # for the version heading, so the underline can be consumed outright. + if (index + 1 < len(lines) and line and set(lines[index + 1]) == {'='} + and len(lines[index + 1]) == len(line)): + out.append(f'## {line}') + index += 2 + continue + + # 2. the one role these entries use. + line = re.sub(r':rfc:`(\d+)`', lambda match: f'[RFC {match[1]}]({RFC_URL}{match[1]})', line) + # 3. literals. + line = re.sub(r'``([^`]+)``', r'`\1`', line) + # 4. bullets, at any indent. + line = re.sub(r'^(\s*)\* ', r'\1- ', line) + # 5. footnotes. + line = re.sub(r'^\.\. \[(\d+)\] ', r'[^\1]: ', line) + line = re.sub(r'\[(\d+)\]_', r'[^\1]', line) + + out.append(line) + index += 1 + + return unwrap(out).rstrip('\n') + '\n' + + +def unwrap(lines: Sequence[str]) -> str: + """Rule 6: collapse each paragraph and each bullet onto a single line. + + A block ends at a blank line, at the next bullet, at a heading, or at a + footnote definition. Nothing in these entries is indentation-sensitive -- + no literal blocks, no tables, no definition lists -- so joining a block's + lines with a single space is lossless. + + Args: + lines: The entry's lines, with rules 1 to 5 already applied. + + Returns: + The lines with each block joined onto one line. + + """ + blocks = [] # type: list[str] + current = [] # type: list[str] + + def flush() -> None: + if current: + blocks.append(' '.join(item.strip() for item in current)) + current.clear() + + for line in lines: + if not line.strip(): + flush() + blocks.append('') + continue + if re.match(r'^\s*- ', line) or line.startswith('## ') \ + or re.match(r'^\[\^\d+\]: ', line): + flush() + current.append(line) + continue + current.append(line) + flush() + + # Collapse the runs of blank lines the flush cycle can leave behind. + return re.sub(r'\n{3,}', '\n\n', '\n'.join(blocks)) + + +def residual(markdown: str) -> list[str]: + """Report reStructuredText left in *markdown* that the six rules did not convert. + + See the module docstring for the guarded set, and for the two constructs left + deliberately unguarded. + + Args: + markdown: One converted entry, as :func:`convert` returned it. The + generated file's trailer is not part of this and does not need to be: + :func:`render` checks the body before appending it. + + Returns: + One human-readable complaint per leftover construct, empty when clean. + + """ + problems = [] # type: list[str] + + # Checked before code spans are masked: masking would eat ``````` as an empty + # code span and hide exactly the case this looks for. + for number, line in enumerate(markdown.split('\n'), 1): + if '``' in line: + problems.append(f'line {number}: an unconverted `` literal: {line.strip()[:70]!r}') + + masked = _CODE_SPAN.sub(_SPAN, markdown) + for pattern, label in _RESIDUAL: + for match in pattern.finditer(masked): + number = masked.count('\n', 0, match.start()) + 1 + found = match.group(0).replace(_SPAN, '`...`') + problems.append(f'line {number}: {label}: {found!r}') + return problems + + +def render(index: pathlib.Path = INDEX) -> str: + """Render the complete ``CHANGELOG.md``. + + Args: + index: The page carrying the toctree that decides which entry is newest. + + Returns: + The file's content, ready to be written. + + Raises: + ResidualMarkupError: If the entry uses reStructuredText the rules do not + cover, so the output would carry markup through as literal text. + + """ + _, entry = newest(index) + body = convert(entry.read_text(encoding='utf-8')) + + problems = residual(body) + if problems: + raise ResidualMarkupError( + f'{entry} uses reStructuredText the six conversion rules do not cover, ' + f'so CHANGELOG.md would carry it through as literal text:\n ' + + '\n '.join(problems) + + '\nEither rewrite the entry in the supported subset, or teach ' + 'util/changelog_md.py the construct.' + ) + + return body + '\n' + TRAILER + + +def main(argv: Optional[Sequence[str]] = None) -> int: + """Command line entry point. + + Args: + argv: Argument list, defaulting to :data:`sys.argv`. + + Returns: + ``0`` on success; ``1`` if ``--check`` found the output stale or missing. + + """ + parser = argparse.ArgumentParser( + prog='changelog_md.py', + description='Generate CHANGELOG.md from the newest per-version changelog entry.', + ) + parser.add_argument( + '--check', action='store_true', + help='write nothing; exit non-zero if the committed file has drifted', + ) + parser.add_argument( + '--index', type=pathlib.Path, default=INDEX, + help='page carrying the toctree that orders the entries (default: %(default)s)', + ) + parser.add_argument( + '--output', type=pathlib.Path, default=OUTPUT, + help='file to generate (default: %(default)s)', + ) + args = parser.parse_args(argv) + + version, entry = newest(args.index) + want = render(args.index) + + if not args.check: + args.output.write_text(want, encoding='utf-8') + print(f'wrote {args.output} from {entry} ({len(want.splitlines())} lines)') + return 0 + + have = args.output.read_text(encoding='utf-8') if args.output.is_file() else '' + if have == want: + print(f'{args.output} is in step with {entry}') + return 0 + + print(f'{args.output} has drifted from {entry}', file=sys.stderr) + sys.stderr.writelines(difflib.unified_diff( + have.splitlines(keepends=True), + want.splitlines(keepends=True), + fromfile=f'{args.output.name} (committed)', + tofile=f'{args.output.name} (regenerated from {version})', + )) + return 1 + + +if __name__ == '__main__': + sys.exit(main())