From b117b976bea44a6b466a55e7faf7de544260ea08 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Sat, 19 Sep 2026 13:04:11 -0400 Subject: [PATCH 1/4] docs: add CHANGELOG.rst and render it in the Sphinx docs The repository had no changelog at all -- release history lived only in git tags and GitHub Releases. - `CHANGELOG.rst` at the repository root, 632 lines, covering 0.13.0 through a new `1.5.0 -- unreleased` section - `docs/source/changelog.rst`, a stub that only does `.. include:: ../../CHANGELOG.rst`, so no content is duplicated - `changelog` added to the second toctree in `docs/source/index.rst` - `pyproject.toml` `[project.urls].changelog` repointed from the Releases page to the rendered page Root rather than `docs/` because `MANIFEST.in` carries both `global-include *.rst` and `prune docs`: a changelog kept only under `docs/` would be missing from every sdist, which is where packagers read it. A copy in both places would drift. The stub deliberately has no title of its own -- measured, one carrying its own title renders two `

` elements. `:start-after:` was rejected for silently dropping the file's intro line. Nitpicky build warning count is unchanged at 1166 with an identical warning set, and the file parses clean under plain docutils, so the sdist copy survives a PyPI-style render. --- CHANGELOG.rst | 632 ++++++++++++++++++++++++++++++++++++++ docs/source/changelog.rst | 14 + docs/source/index.rst | 1 + pyproject.toml | 2 +- 4 files changed, 648 insertions(+), 1 deletion(-) create mode 100644 CHANGELOG.rst create mode 100644 docs/source/changelog.rst diff --git a/CHANGELOG.rst b/CHANGELOG.rst new file mode 100644 index 000000000..932cb9e71 --- /dev/null +++ b/CHANGELOG.rst @@ -0,0 +1,632 @@ +========= +Changelog +========= + +All notable changes to PyPCAPKit are recorded here. + + **Note** -- Versions before 1.5.0 are reconstructed from the git history, so + they summarise each release rather than enumerate every change. The record + starts at 0.13.0 (2018-12-08); the earlier 0.x releases are not covered. + + **Note** -- Post-releases (``X.Y.Z.postN``) are, from 1.0.1 onwards, automated + publications of the weekly registry refresh: a bot regenerates the vendor + constant enumerations under ``pcapkit.const`` from the upstream IANA + registries and bumps the version. They carry no library changes, and are + collapsed into a single line per release below. Where a post-release did + carry something real, it is called out. + +1.5.0 -- unreleased +=================== + +The largest release since 1.0, and the first recorded here as it happened rather +than reconstructed. Three more extraction engines, ESP with payload decryption, +SCTP and NGAP over SCTP, a library logger that no longer hijacks the consumer's, +and a defect programme run through the issue tracker across some 140 issues and +pull requests between #326 and #509. + +* **Added** -- three extraction engines: ``engine='pypcap'`` and + ``engine='pcap_ct'``, two independent distributions of the same ``libpcap`` + interface, and ``engine='pypcapfile'`` (#386, #405). They buy speed by doing + less -- neither ``pypcap`` nor ``pcap_ct`` dissects at all, so they offer + neither reassembly nor flow tracing, and ``pypcapfile`` has no IPv6 decoder. + Install only **one** of ``pypcap`` and ``pcap-ct``: both own the top-level + ``pcap`` module, and with both present ``pcap-ct`` wins the import and the + other becomes unselectable. The matching interface constants ``PyPCAP``, + ``PCAP_CT`` and ``PyPCAPFile`` were missing and are now exported alongside + ``DPKT``, ``Scapy``, ``PyShark`` and ``PCAPKit`` (#412). That brings the + built-in set to seven engines; 3.11 is the last interpreter on which every one + of them can run, and even there two of them cannot coexist. +* **Added** -- ``EngineBase.unsupported_reason``, a preflight every engine + answers and ``Extractor.run`` consults before anything is imported. Asking for + an engine that cannot run in the current environment now gives one warning + naming the real cause -- a Python version, a missing ``tshark``, a missing + ``libpcap``, the wrong ``pcap`` distribution -- and a clean fall back to + ``pcapkit``'s own parser, rather than an error from inside the third-party + package (#396, #405). +* **Added** -- ESP parsing and construction [:rfc:`4303`], with optional payload + decryption and ICV verification through ``cryptography`` + (``pip install pypcapkit[crypto]``) (#378). Keys reach the dissector through a + new caller-state channel, ``pcapkit.corekit.context``, surfaced as the + ``context=`` keyword on ``extract()`` and ``Extractor``, carrying an + ``esp.SecurityAssociation``. Nothing here raises: with no association the SPI + and sequence number are still reported and the ciphertext is left opaque, with + ``status=NO_SA``, and a failed decryption or ICV check is recorded on the + parsed result the same way. Extended Sequence Numbers, TFC padding and + anti-replay are not implemented, and an unsupported cipher or MAC is refused + with a clear error rather than half-processed. +* **Added** -- SCTP as a transport protocol [:rfc:`9260`]: all 13 chunk types, 8 + chunk parameters and 13 error causes, with the CRC32c both recorded and + verifiable -- it covers the SCTP packet alone, with no IP pseudo-header, so it + can be checked from the SCTP bytes. Upper layers register on the DATA chunk's + Payload Protocol Identifier through ``register_sctp``, not on a port (#379). +* **Added** -- NGAP over SCTP (3GPP TS 38.413), decoding aligned PER through + ``pycrate`` (``pip install pypcapkit[NGAP]``) (#251, #417). Decoding is generic + by ASN.1 shape rather than per-procedure, so all 81 elementary procedures and + 438 protocol IEs work and a new 3GPP release needs no code change. Registered + as a default on PPID 60 and 66, but only 60 decodes: PPID 66 is an NGAP PDU + inside a DTLS record, and there is no DTLS dissector. ``pycrate`` is + deliberately excluded from the ``all`` extra -- it is LGPL-2.1+ where this + package is BSD-3-Clause, and lands some 238 MB to obtain one module. +* **Added** -- the Mobility Header registry, completed (#383, #437). The + :rfc:`5568` fast-handover messages and options first, then all 24 registered + message data types, 70 of the 71 registered options -- with nested sub-option + registries for the flow identification, access network identifier, + quality-of-service and LMA-controlled MAG parameter families -- and all 4 CGA + extensions. Only the CGA Parameters option remains on the generic handler. +* **Added** -- dispatch entries for dissectors that existed but were reachable + from no registry (#436): FTP-DATA on TCP 20, HTTP/1 on TCP 8080, HTTP on UDP + 8080, ``L2TPv2`` on UDP 1701 and OSPF at ``TransType`` 89. ``VLAN`` became an + abstract base with ``C_Tag`` (802.1Q) and ``S_Tag`` (802.1ad) as concrete + subclasses, so a Q-in-Q frame no longer collapses into one opaque ``Raw``; + ``L2TP`` likewise became a base, with ``L2TPv2`` carrying the :rfc:`2661` + implementation. +* **Added** -- ``pcapkit.utilities.logging`` as a real interface: + ``get_logger()`` for per-module children, ``configure()`` to set level, + handler, stream, format or propagation at runtime, ``reset()`` to return to + library-neutral, and ``ensure_output()``. Seventeen modules now log under their + own ``__name__``, so a consumer can silence ``pcapkit.foundation.registry`` + while keeping ``pcapkit.foundation.extraction`` (#384). +* **Added** -- ``conflict`` on the reassembly data models: absolute, inclusive + ranges where two fragments claimed the same span with different bytes, which + was previously lost silently on both the IP (#482) and TCP (#443, #478) paths. +* **Added** -- an end-to-end test tier (#376), sample-capture generators so a + fresh clone can rebuild every fixture (#340), a Dockerised engine benchmark + covering every supported Python version (#410), and registry round-trip + coverage that records the entries which cannot close the cycle rather than + skipping them (#440, #504). +* **Changed** -- ``pcapkit`` no longer configures logging at import. It + installs a ``NullHandler`` and sets no level, so verbosity is inherited from + the application instead of being seized by whichever library was imported + second; the old stderr handler stays as the ``PCAPKIT_DEVMODE`` opt-in. Three + consequences worth knowing: the previous behaviour is + ``configure(logging.INFO, stream=sys.stderr)``; 38 registry and extractor + ``info`` calls became ``debug``, so those messages are invisible even at + ``INFO``; and the handler is no longer ``logger.handlers[0]``. ``verbose=`` + output stays on stdout and is not logging (#384). +* **Changed** -- each warning is reported once per channel, and ``pcapkit`` no + longer inserts a ``simplefilter('ignore', ...)`` at the front of the + process-global ``warnings.filters`` (#362--#364, #390). The application's own + filter therefore wins now, which is the point of the change and also the sharp + edge in it: under ``-W error``, or pytest's ``filterwarnings = error``, a + pcapkit warning that used to be suppressed will raise. Suppress them + deliberately with + ``warnings.filterwarnings('ignore', category=BaseWarning)``. ``quiet=True`` + now means no record at any level and no longer sets ``sys.tracebacklimit``, + and the ``pcapkit.utilities.warnings.DEVMODE`` re-export is gone -- its + canonical home is ``pcapkit.utilities.logging``. +* **Changed** -- ``layer=`` and ``protocol=`` are honoured rather than inert. + Both were read under the wrong names, so every value a caller passed was + dropped into ``**kwargs`` and discarded; the CLI's ``-L`` also now validates + its argument instead of accepting anything. The packet context reaches the + schema layer for the first time as well, so a field the wire elides can be + resolved from its enclosing packet (#404). ``follow_tcp_stream`` dispatches on + the engine type, where both branches of the old test were dead and the native + adapter ran against every engine's frames (#402). +* **Changed** -- two reassembly and flow-tracing defaults moved (#435), and both + are visible to a caller. ``Datagram.completed`` widened from ``bool`` to a + ``Completion`` enumeration (``COMPLETE``, ``PARTIAL``, ``TIMEOUT``); only + ``COMPLETE`` is truthy, so ``if datagram.completed:`` is unaffected but + ``datagram.completed == True`` no longer holds. TCP flow tracing is + **bidirectional by default**, which merges each flow's two halves and closes + one only once both have FINed -- 331 flows become 111 on the sample HTTP + capture, the difference being single-frame stray tails; pass + ``trace_bidirectional=False`` for the old behaviour. IP reassembly also gained + the 60-second timeout [:rfc:`1122`, :rfc:`8200`], clocked off the capture's own + timestamps rather than the wall clock, tunable with ``reasm_timeout=``; TCP + reassembly gets no timeout by default. ``trace_analyse=`` is new, and + reassembles each traced flow's application layer. +* **Changed** -- conflicting TCP overlaps resolve first-write-wins, per + :rfc:`9293` section 3.10, where they had silently resolved last-write-wins + (#443, #478). A deliberate behaviour break, and a narrow one: a conforming + retransmission carries identical bytes, so nothing changes for it. IP fragment + reassembly keeps last-write-wins, because :rfc:`791` specifies the opposite + resolution, and records the disagreement instead (#482). +* **Changed** -- ``Probe``, ``CipherSuite`` and ``IntegritySuite`` are ``Info`` + subclasses rather than ``typing.NamedTuple``, and no ``NamedTuple`` remains in + the package. They are Mappings now, so ``len()`` and iteration yield field + names rather than values. +* **Changed** -- renames with no compatibility alias left behind: + ``HoleDiscriptor`` is spelled ``HoleDescriptor`` and its package alias + ``TCP_HoleDiscriptor`` is ``TCP_HoleDescriptor`` (#350); PCAP-NG ``Option`` + subclasses spell the namespace class keyword ``ns=`` instead of ``namespace=`` + (#439); and ``examples/sample`` and ``examples/samples`` -- one letter apart, + holding different things -- are now ``examples/captures`` and + ``examples/generators``. +* **Changed** -- extraction is around 46% faster on a 1,117-frame HTTP capture, + with byte-identical output (#420). A reassembled datagram's payload is now + analysed on first read rather than eagerly, which cuts IP reassembly's own + cost by 90.7% and TCP's by 23.7% -- IP reassembly submits a datagram for every + frame, fragmented or not (#424). Flow tracing over the same capture went from + 1416.6 ms to 744.0 ms, because the flow dumper had been handing each record to + a ``Frame`` constructor that re-dissected the whole protocol stack to return + bytes it had just been given; options are no longer parsed twice either + (#427). All output compared byte-for-byte across the sample captures in each + case. +* **Fixed** -- next-layer, option, chunk, block and parameter dispatch all read + ``defaultdict`` registries, so a lookup miss inserted the key into class-level + state shared by every later instance, after which a legitimate ``register_*`` + call warned that the code was already registered. Every read now goes through + a lookup that does not grow the table, and ``IPv4.__option__`` and + ``HIP.__parameter__`` became inspectable class attributes rather than names + assembled at call time (#426, #428, #429, #434). One break comes with it: a + tuple-registered handler pair written to the documented + ``OptionParser``/``OptionConstructor`` signature now works where it could + previously never be called at all, and a pair written with an explicit leading + ``self`` -- the only shape that used to work -- now does not. +* **Fixed** -- on Python 3.10 and older, no ``Schema`` subclass got its own + ``_abc_impl``: all of them fell through to ``collections.abc.Mapping``'s, so a + single ``isinstance`` or ``issubclass`` answer poisoned every later question + about that class for the rest of the process. A terminating PCAP-NG + ``EndRecord`` tested ``True`` as an ``IPv4Record`` (#439). +* **Fixed** -- construction, which was broken in several places at once: the + generated typed ``__init__`` was never installed, so ``__post_init__`` did not + run and a schema built from a subset of its fields could not be packed at all + -- ``UDP(srcport=53, dstport=5353)`` now packs (#430); IPv6 and Mobility Header + option padding was wrong, leaving construction wholly broken (#398); + ``HTTP.make`` called the versioned ``make`` unbound, so every real call raised + ``TypeError`` (#452, #462); and ``IPv4._make_data`` returned the fragment + offset in octets where the wire wants 8-octet units, and read ``data.options`` + on a packet that has none (#494, #499). +* **Fixed** -- a truncated or under-declared area no longer parses + "successfully", and no longer wedges the process. The option and list loops + could spin forever with no exception on a truncated area, reachable from + untrusted input through HOPOPT, IPv6-Opts, MH, HIP and SCTP; each iteration + must now advance the stream by at least one octet, and the error names the + option, the offset and the octets remaining (#431, #432). Separately, + wire-derived lengths in ``ipv6_opts``, CALIPSO, MPL, REG_INFO and four HIP list + callbacks underflowed below zero, which ``ListField``'s own + ``while length > 0`` then turned into a silent empty list; they are floored and + raise instead (#449, #456, #460, #463). +* **Fixed** -- field widths and units, each measured against the specification + rather than inferred: HIP's ``TRANSPORT_FORMAT_LIST``, ``NAT_TRAVERSAL_MODE`` + and ``ESP_TRANSFORM`` list entries are two octets, not one [:rfc:`7401`, + :rfc:`5770`, :rfc:`7402`] (#463, #472); the MN-ID option sizes from its + subtype, not from ``identifier``'s Python type (#448, #464, #467); + IPv6-Route's ``Hdr Ext Len`` is computed in 8-octet units on both sides + [:rfc:`8200`] (#487, #489); and the Fast Binding Update and Acknowledgment + Lifetimes are plain seconds [:rfc:`5568`], not :rfc:`6275`'s four-second units + (#502). +* **Fixed** -- stdlib exceptions leaking out where the library's own were + promised: a malformed IP field value raised a bare ``ValueError`` instead of + ``FieldValueError`` (#465); a ``bool`` address was silently packed as + ``0.0.0.1`` or ``0.0.0.0``, ``bool`` being an ``int`` subclass (#491, #500); + and ``@prepare`` discarded extra arguments silently and treated a *declared* + zero length as end of stream, which is now distinguished from a genuinely + exhausted one and raises ``StreamEOFError`` (#454, #458). +* **Fixed** -- the engine adapters, which were quietly wrong rather than loud. + The ``dpkt`` toolkit split TCP and IPv4 headers at their fixed struct size + instead of their real length, so option octets overwrote payload in the + sequence-indexed reassembly buffer; it also read an ``ipv6_frag.nh`` that + ``dpkt`` does not have, and passed fragment offsets unscaled (#351, #370, + #385, #395). ``scapy`` never loaded its layer registry, so that engine did not + dissect at all (#409), and its IPv4 fragment offset reached reassembly + unscaled (#483, #484). The four IPv6 adapters disagreed about whether the + 8-octet Fragment header belongs to ``ihl``, ``header`` and ``tl``; per + :rfc:`8200` section 4.5 it belongs to none of them, and all four now agree + (#415, #424). +* **Fixed** -- PCAP and PCAP-NG output and parsing: ``bytes(frame)`` returned the + *next* frame's octets, ``files=True`` wrote names like ``Frame 1..json``, and a + PCAP-NG ``timestamp_epoch`` was shifted by the reading host's timezone (#403); + seven further parser defects (#341--#347, #371) and, on the write path, four + more block-parsing ones (#388); ``BitField`` packed every named bit as set + (#359, #374); the extension-header walk failed to advance past the last IPv6 + extension header (#348, #373); and IPv6 fragment offsets went unscaled, with + reassembly keyed on the flow label -- optional, and routinely zero, so distinct + datagrams collapsed together -- rather than on the fragment identification + (#389). +* **Fixed** -- TCP reassembly mixed absolute sequence numbers with + buffer-relative slicing, so on any capture carrying a SYN with a realistic + initial sequence number incomplete datagrams were dropped silently, and the + ``completed=False`` branch of the public API was unreachable (#349, #376). +* **Fixed** -- constant lookups that rejected a value the registry defines. + ``RouterAlert(0)`` is the only value :rfc:`2113` defines and the one IGMP, + RSVP and MLD actually send, and it was discarded because the vendor crawler + skipped a header row IANA's CSV does not have; IPX ``Socket(0)`` is that + protocol's own default, so ``bytes(IPX(...))`` crashed on its own defaults; + and two FTP ``_missing_`` overrides were plain methods rather than + classmethods, so every unregistered value raised ``TypeError`` instead of + extending the enumeration (#492, #503). +* **Fixed** -- ``format='text'`` raised ``AttributeError`` before writing + anything, naming a ``dictdumper.Text`` that has never existed. It now points + at ``Tree``, as the ``'txt'`` alias beside it already did. +* **Fixed** -- 45 places where a documentation page contradicted the code + (#413), ambiguous cross-references and five autodoc signature failures (#416), + and ``Extractor``'s documented exception plus 40 phantom or stale ``Args:`` + labels (#501). + +Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both +2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so +resolved only by ``pip install --pre``. ``1.5.0b1`` half-shipped: the tag, the +GitHub release and the Conda deployments landed, but PyPI rejected the wheel +because ``twine check`` found a Sphinx-only ``:mod:`` role in ``README.rst``, +which ``pyproject.toml`` declares as the dynamic long description. ``1.5.0b2`` +is what reshipped it -- the release workflow is version-driven, so an existing +version cannot republish -- and ``1.5.0b3`` followed the CI change that stops a +TestPyPI outage from costing a release its wheels (#497, #498). + +1.4.1 -- 2026-08-22 +=================== + +Release engineering only -- no library behaviour changed. + +* **Added** -- a unit-test CI workflow, and the packaging, vendor-cron and + documentation workflows are now gated on it, so a release can no longer be cut + from a tree whose tests fail. +* **Changed** -- the vendor cron refuses to publish when the registry update + itself failed, rather than shipping a half-refreshed release. +* **Fixed** -- unit-test compatibility on Python 3.10. + +*Also released:* ``1.4.1.post1``, ``1.4.1.post2`` (through 2026-09-12). + +1.4.0 -- 2026-08-21 +=================== + +The first release in nearly two years to carry code changes, and the first with +a test suite. + +* **Added** -- a unit-test suite under ``tests/``, covering ``pcapkit.corekit``, + the extraction engines, reassembly and flow tracing, the protocol schemas, the + CLI and the utility modules. +* **Fixed** -- PCAP-NG option dispatch regressions; the fallback name generated + by the ``linktype`` vendor crawler. +* **Changed** -- CI tests current Python versions again; every GitHub Actions + job carries an explicit permissions block; the conda and cron release tooling + was repaired, and Anaconda uploads use the official upload action. + +1.3.5 -- 2024-11-16 +=================== + +**Fixed** -- transport-layer payloads were not decoded at all. An inverted +``payload is None`` check in ``pcapkit.protocols.internet.internet`` left every +TCP and UDP frame under IPv4/IPv6 reported as ``raw``, typically with +``'int' object has no attribute 'port'``. The regression was introduced by the +IPv6 extension-header fix in 1.3.4, so 1.3.4 should be skipped entirely. + +*Also released:* ``1.3.5.post1`` through ``1.3.5.post43`` (through 2026-08-21), +all registry refreshes except ``post41``, which switched the Anaconda upload to +the official action. + +1.3.4 -- 2024-11-14 +=================== + +* **Fixed** -- IPv6 Routing header length handling, and extraction of the IPv6 + extension header chain (#218). +* **Changed** -- the build matrix moved to Python 3.13 and stopped testing 3.8. + +This release also introduced the transport-layer parsing regression fixed in +1.3.5; upgrade past it rather than to it. + +1.3.3 -- 2024-11-03 +=================== + +* **Fixed** -- errors reading PCAP files from both the CLI and the library + (#240): missing optional CLI dependencies are handled gracefully and console + output is far less verbose; installation from source, which ``setup.py`` had + broken (#221); PCAP output during flow tracing (#180). +* **Changed** -- the base protocol data class carries ``packet``; ``Info`` + iteration skips excluded fields. + +1.3.2 was never released. The version string existed in-tree for a few hours on +2024-11-03 and was superseded by 1.3.3 the same day, so there is no ``v1.3.2`` +tag and no distribution. + +*Also released:* ``1.3.3.post1``, ``1.3.3.post2`` (through 2024-11-14); +``post1`` fixed a PyPI metadata problem. + +1.3.1 -- 2023-12-21 +=================== + +* **Changed** -- ``register_port()`` was renamed ``register_apptype()``, with no + alias left behind; callers registering application-layer protocols by port + must be updated. +* **Changed** -- the documentation was revised across the whole project -- + ``pcapkit.protocols``, ``pcapkit.corekit`` and ``pcapkit.dumpkit`` in + particular. + +*Also released:* ``1.3.1.post1`` through ``1.3.1.post28`` (through 2024-10-26). + +1.3.0 -- 2023-10-04 +=================== + +A redesign of the extension points, on metaclasses. + +* **Changed** -- the protocol, reassembly, flow-tracing, engine, vendor and + field base classes are all built on metaclasses, which is what registers a + subclass and fixes its name and module. +* **Added** -- ``ModuleDescriptor``, so a registry entry can name a module and + class that are imported lazily on first use rather than at registration; the + registry functions accept it throughout. ``VersionInfo`` gained a ``version`` + property. +* **Changed** -- ``_Engine`` renamed ``EngineBase``; the ``beholder`` decorator + logs the error it swallows under devmode and verbose mode. + +1.2.2 -- 2023-09-14 +=================== + +**Fixed** -- extraction failing outright (#166). + +*Also released:* ``1.2.2.post1``, ``1.2.2.post2`` (through 2023-10-04). + +1.2.1 -- 2023-08-27 +=================== + +**Fixed** -- bugs in the ``SeekableReader`` introduced one day earlier in 1.2.0. + +*Also released:* ``1.2.1.post1`` through ``1.2.1.post3`` (through 2023-09-09). + +1.2.0 -- 2023-08-26 +=================== + +* **Added** -- streaming input. ``pcapkit.corekit.io.SeekableReader`` buffers a + non-seekable stream so a capture can be read from ``stdin`` or a pipe, which + is what live ``tcpdump`` output needs (#156). +* **Fixed** -- reassembly, broken since 0.16.3 (#155). ``Extractor`` now raises + an explicit unsupported-call error, and logs a notice, when reassembly or flow + tracing is requested where it cannot be provided. + +1.1.1 -- 2023-08-15 +=================== + +**Fixed** -- ``AppType`` enumeration lookups; ``typing.TypeAlias`` compatibility +on older interpreters; the output file name chosen by flow tracing; dumping of +``Schema`` objects through a custom dumper. + +*Also released:* ``1.1.1.post1``, ``1.1.1.post2`` (through 2023-08-23). + +1.1.0 -- 2023-07-09 +=================== + +* **Changed** -- the schema and field classes were rebuilt on metaclasses, with + ``EnumSchema`` for enum-dispatched schemas, and every protocol schema was + revised onto them: IPv4, HOPOPT, IPv6-Opts, IPv6-Route, HIP, MH, TCP, HTTP/2 + and PCAP-NG. +* **Added** -- schema registration through the protocol registry APIs; + persistent additional and excluded fields on ``Info``; numeric and comparison + support on the PCAP-NG option code and application-layer port enumerations. + +*Also released:* ``1.1.0.post1`` through ``1.1.0.post3`` (through 2023-08-12). + +1.0.3 -- 2023-06-29 +=================== + +* **Added** -- the IANA service name and transport protocol port number registry + as the ``AppType`` enumeration [:rfc:`6335`], wired into TCP and UDP parsing + so ports resolve to service names; a generated enumeration for the TCP header + flags; an undefined-command member for FTP commands. +* **Changed** -- registry management was redesigned; the MPTCP and PCAP-NG + option enumerations were revised. + +*Also released:* ``1.0.3.post1`` through ``1.0.3.post3`` (through 2023-07-04). + +1.0.2 -- 2023-06-05 +=================== + +* **Added** -- Mobility Header (MH) parsing and construction [:rfc:`6275`], + including the message types, mobility options, CGA parameters and link-layer + address option codes, and the binding error status codes. +* **Fixed** -- ``pcapkit.extract`` failing on interpreters where + ``decimal.localcontext`` takes no keyword arguments (#139); wheel filenames + now carry the correct Python tags. + +*Also released:* ``1.0.2.post1`` through ``1.0.2.post8`` (through 2023-06-27); +``post6`` also reset the conda build number. + +1.0.1 -- 2023-05-14 +=================== + +**Changed** -- ``Info`` and ``Schema`` subclasses are finalised when the class is +created rather than on each instantiation, which is a straight runtime saving on +parsing. + +*Also released:* ``1.0.1.post1`` through ``1.0.1.post3`` (through 2023-05-27). + +1.0.0 -- 2023-05-09 +=================== + +The 1.0 rewrite: parsing and construction are one declarative definition per +protocol, and the extraction backend became pluggable. Roughly 55,000 lines +changed across 617 files, released after 21 beta builds and a release candidate. + +* **Added** -- the schema and field layer, ``pcapkit.protocols.schema`` and + ``pcapkit.corekit.fields``. A protocol declares its wire format once, as + fields (numbers, strings, IP addresses, payloads, options, lists, switches), + and both parsing and construction follow from it. +* **Added** -- PCAP-NG support [1]_: the protocol implementation covering every + block type, the generic and per-block options, name-resolution records and + decryption secrets, plus a matching extraction engine, toolkit functions and + ``pcapkit.protocols.schema.misc.pcapng``. +* **Added** -- ``pcapkit.foundation.engines``, making the extraction backend a + registered, swappable component: the built-in PCAP and PCAP-NG engines + alongside DPKT, Scapy and PyShark, with engine registry APIs. +* **Added** -- ``SchemaWarning``, ``DeprecatedFormatWarning`` and + ``RegistryWarning``; ``Protocol._get_payload`` for customised payload + retrieval; ``packet`` passed down the protocol chain so a layer can see its + parent's context, which is how IPv6 hands source and destination addresses to + the transport layer. +* **Changed** -- flow tracing became the ``pcapkit.foundation.traceflow`` + package, and reassembly gained data-model modules; + ``pcapkit.toolkit.default`` was renamed ``pcapkit.toolkit.pcap``; + ``pcapkit.foundation.engine`` became ``engines``; ``IPField`` was split into + ``IPAddressField`` and ``IPInterfaceField``; the option and parameter + registries for HIP and IPv4 warn on overwrite. + +.. [1] PCAP-NG is specified by ``draft-tuexen-opsawg-pcapng``, not by an RFC. + +0.16.3 -- 2022-10-31 +==================== + +**Fixed** -- import crashes on some interpreters (#114, #116); the reassembly +property caches; README rendering for PyPI. The build chain was revised. + +0.16.2 -- 2022-08-03 +==================== + +**Fixed** -- ``Info`` ``__init__`` generation for classes without annotations +(#113); the IP reassembly algorithm (#82). + +0.16.1 -- 2022-06-07 +==================== + +**Changed** -- warnings are raised through ``pcapkit.utilities.warnings`` +instead of ``warnings.warn`` directly, and a missing optional dependency now +warns for the CLI and vendor extras too. + +0.16.0 -- 2022-05-31 +==================== + +A project-wide revision, and the last release of the pre-1.0 design. + +* **Changed** -- type annotations throughout, and linter compliance + (pylint, mypy, bandit, vermin) across every module. +* **Changed** -- protocol classes were separated from their data models, which + moved to ``pcapkit.protocols.data``; a single ``Protocol`` class handles both + parsing and construction, and subclassing it to add a protocol is far less + work. +* **Added** -- ``pcapkit.foundation.registry``, so protocols can be subscribed + by name; ``Protocol.analyze`` for analysing a payload directly, used by the + reassembly classes; ``pcapkit.corekit.multidict``, adapted from Werkzeug. +* **Changed** -- reassembly became a package (IP, IPv4, IPv6, TCP) with revised + data models; PCAP and the auxiliary protocols moved under ``pcapkit.misc``. +* **Removed** -- the ``validators`` module, several decorators from + ``pcapkit.utilities``, and multiprocessing support in ``Extractor``. +* **Fixed** -- the DPKT toolkit (#101), IP reassembly, and missing ``id`` + methods on subclassed protocols. + +The PyPCAPKit Enhancement Proposals discussion channel was opened with this +release (#106). + +0.15.5 -- 2020-10-11 +==================== + +**Fixed** -- a fallback encoder was added so that data DictDumper cannot encode +no longer raises out of the dumper (#65). + +0.15.4 -- 2020-08-28 +==================== + +**Fixed** -- the missing ``stacklevel`` attribute on ``pcapkit.utilities`` (#58). +Travis CI was set up. + +0.15.3 -- 2020-08-17 +==================== + +* **Fixed** -- ARP parsing (#55). +* **Added** -- logging, through a ``pcapkit`` logger. + +0.15.2 -- 2020-06-27 +==================== + +* **Added** -- a register interface on the protocol classes, and the same on + ``pcapkit.foundation.analysis``, so both can be extended without patching the + library. +* **Fixed** -- flow tracing output format and content consistency. + +0.15.1 -- 2020-06-18 +==================== + +Maintenance release: the constant enumerations and the vendor crawlers that +generate them were revised. + +*Also released:* ``0.15.1.post1`` (2020-06-19), a refresh of the constant +enumerations. + +0.15.0 -- 2020-06-07 +==================== + +* **Changed** -- the parsing and construction logic was merged, so a protocol is + no longer implemented twice; TCP flow tracing and several interfaces were + revised. +* **Added** -- full API documentation, published at + https://pypcapkit.jarryshaw.me. +* **Fixed** -- the DictDumper dependency and the failures it caused (#37, #40). + +Preceded by ``0.15.0rc1`` (2020-06-06). + +0.14.5 -- 2019-10-24 +==================== + +* **Added** -- a CLI for the vendor crawlers. +* **Changed** -- vendor imports are deferred, so the crawler dependencies are + not needed to import ``pcapkit``; dependencies revised. + +0.14.4 -- 2019-09-01 +==================== + +* **Added** -- the vendor crawler scripts ship in the distribution. +* **Changed** -- the CLI and vendor dependencies are separate extras in + ``setup.py``; constant enumerations regenerated. + +0.14.3 -- 2019-08-08 +==================== + +**Fixed** -- reported bugs (#29, #30), and a Wikipedia block that broke a vendor +crawler. Constant enumerations and the sample output were regenerated. + +0.14.2 -- 2019-03-28 +==================== + +**Fixed** -- the exception classes, and the TCP reassembly algorithm. + +0.14.1 -- 2019-03-07 +==================== + +**Added** -- the ``PCAPKIT_DEVMODE`` environment variable, which turns on +development-mode behaviour without a code change. + +0.14.0 -- 2019-03-02 +==================== + +**Changed** -- the TCP reassembly pipeline handles the RST flag. Docker build +files were added. + +0.13.3 -- 2019-02-26 +==================== + +**Fixed** -- numeric literals in the constant modules that were incompatible +with Python 3.5. + +*Also released:* ``0.13.3.post1``, ``0.13.3.post2`` (through 2019-03-01); +``post1`` declared official support for Python 3.5 and below, and ``post2`` +refreshed the dependencies. + +0.13.2 -- 2019-01-24 +==================== + +Maintenance release: the ``tbtrim`` dependency was updated. + +0.13.1 -- 2019-01-23 +==================== + +* **Added** -- ``tbtrim`` is used to trim PyPCAPKit's own frames out of + tracebacks through a custom ``excepthook``, so a traceback points at the + caller's code. +* **Fixed** -- compatibility issues, with a new ``pcapkit.utilities.compat`` + module. + +0.13.0 -- 2018-12-08 +==================== + +No library changes -- a licence and packaging metadata refresh. + +*Also released:* ``0.13.0.post1``, ``0.13.0.post2`` (through 2018-12-12); +``post2`` fixed encoding errors in ``setup.py``. diff --git a/docs/source/changelog.rst b/docs/source/changelog.rst new file mode 100644 index 000000000..086e8b564 --- /dev/null +++ b/docs/source/changelog.rst @@ -0,0 +1,14 @@ +.. This page is a thin wrapper around the repository-root ``CHANGELOG.rst``, + which is the single source of truth. It lives at the root rather than here + because ``MANIFEST.in`` carries both ``global-include *.rst`` and + ``prune docs``: a changelog kept only under ``docs/`` would be absent from + every source distribution, which is where packagers read it. A copy in both + places would drift, so this file duplicates no content. + + Deliberately no title of its own. The included file opens with its own + ``Changelog`` document title, which becomes this page's title and its + toctree label; adding a second one here renders two ``

`` elements on the + same page. Measured, not assumed -- a stub carrying its own title produced + ``h1 Release Notes`` followed by ``h1 Changelog``. + +.. include:: ../../CHANGELOG.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index c0a6948a9..950f4f844 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -30,6 +30,7 @@ construction and analysis library. ext demo pep + changelog About ===== diff --git a/pyproject.toml b/pyproject.toml index 3f160dd70..2df23d892 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -100,7 +100,7 @@ dependencies = [ homepage = "https://jarryshaw.github.io/PyPCAPKit/" documentation = "https://jarryshaw.github.io/PyPCAPKit/" repository = "https://github.com/JarryShaw/PyPCAPKit" -changelog = "https://github.com/JarryShaw/PyPCAPKit/releases" +changelog = "https://jarryshaw.github.io/PyPCAPKit/changelog.html" [project.scripts] pcapkit-cli = "pcapkit.__main__:main" From 3e5fc53cf89e2ef8de44dbb392477b4a7fec86a3 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Sat, 19 Sep 2026 18:42:35 -0400 Subject: [PATCH 2/4] docs: split the changelog per version and derive CHANGELOG.md from it Replaces the single root CHANGELOG.rst with the layout the release chain actually needs: a Markdown CHANGELOG.md at the root holding only the newest version's entry, and the full history under docs/ as .rst wired into Sphinx. - docs/source/changelog/.rst: 37 per-version entries, split out of the 632-line root file. Concatenating the preamble and all 37 in toctree order reproduces CHANGELOG.rst byte-for-byte (32,671 bytes, 632 lines). - docs/source/changelog.rst: title, preamble and a 37-entry toctree, already reachable from docs/source/index.rst:33. - CHANGELOG.md: generated, not maintained. 40 lines of GFM holding the 1.5.0 entry, for the Create Release body and the sdist surface, both of which read Markdown rather than reStructuredText. - util/changelog_md.py: the generator, plus a --check mode that exits non-zero when the committed CHANGELOG.md has drifted from its source entry. "Newest" comes from the toctree's first line rather than a version sort, since PEP 440 ordering and lexical ordering disagree on 1.5.0b3 against 1.5.0 against 1.10.0. A residual-markup guard of 13 patterns refuses to emit when any unconverted RST survives, so nothing malformed can reach a published release. - MANIFEST.in: include CHANGELOG.md explicitly, since global-include *.rst does not match it, and ship the per-version entries. CHANGELOG.md is Markdown by deliberate exception to the .rst convention: its consumers are GitHub's release renderer and PyPI, neither of which reads RST. 34 tests, 37 subtests. Sphinx warning set unchanged against main. sdist ships CHANGELOG.md and all 37 entries, and twine check passes. --- CHANGELOG.md | 40 ++ MANIFEST.in | 16 + docs/source/changelog.rst | 88 +++- docs/source/changelog/0.13.0.rst | 7 + docs/source/changelog/0.13.1.rst | 8 + docs/source/changelog/0.13.2.rst | 4 + docs/source/changelog/0.13.3.rst | 9 + docs/source/changelog/0.14.0.rst | 5 + docs/source/changelog/0.14.1.rst | 5 + docs/source/changelog/0.14.2.rst | 4 + docs/source/changelog/0.14.3.rst | 5 + docs/source/changelog/0.14.4.rst | 6 + docs/source/changelog/0.14.5.rst | 6 + docs/source/changelog/0.15.0.rst | 11 + docs/source/changelog/0.15.1.rst | 8 + docs/source/changelog/0.15.2.rst | 7 + docs/source/changelog/0.15.3.rst | 5 + docs/source/changelog/0.15.4.rst | 5 + docs/source/changelog/0.15.5.rst | 5 + docs/source/changelog/0.16.0.rst | 23 + docs/source/changelog/0.16.1.rst | 6 + docs/source/changelog/0.16.2.rst | 5 + docs/source/changelog/0.16.3.rst | 5 + docs/source/changelog/1.0.0.rst | 31 ++ docs/source/changelog/1.0.1.rst | 8 + docs/source/changelog/1.0.2.rst | 12 + docs/source/changelog/1.0.3.rst | 11 + docs/source/changelog/1.1.0.rst | 12 + docs/source/changelog/1.1.1.rst | 8 + docs/source/changelog/1.2.0.rst | 9 + docs/source/changelog/1.2.1.rst | 6 + docs/source/changelog/1.2.2.rst | 6 + docs/source/changelog/1.3.0.rst | 14 + docs/source/changelog/1.3.1.rst | 11 + docs/source/changelog/1.3.3.rst | 16 + docs/source/changelog/1.3.4.rst | 9 + docs/source/changelog/1.3.5.rst | 12 + docs/source/changelog/1.4.0.rst | 14 + docs/source/changelog/1.4.1.rst | 13 + .../source/changelog/1.5.0.rst | 384 -------------- tests/project/test_changelog_md.py | 448 ++++++++++++++++ util/changelog_md.py | 480 ++++++++++++++++++ 42 files changed, 1389 insertions(+), 398 deletions(-) create mode 100644 CHANGELOG.md create mode 100644 docs/source/changelog/0.13.0.rst create mode 100644 docs/source/changelog/0.13.1.rst create mode 100644 docs/source/changelog/0.13.2.rst create mode 100644 docs/source/changelog/0.13.3.rst create mode 100644 docs/source/changelog/0.14.0.rst create mode 100644 docs/source/changelog/0.14.1.rst create mode 100644 docs/source/changelog/0.14.2.rst create mode 100644 docs/source/changelog/0.14.3.rst create mode 100644 docs/source/changelog/0.14.4.rst create mode 100644 docs/source/changelog/0.14.5.rst create mode 100644 docs/source/changelog/0.15.0.rst create mode 100644 docs/source/changelog/0.15.1.rst create mode 100644 docs/source/changelog/0.15.2.rst create mode 100644 docs/source/changelog/0.15.3.rst create mode 100644 docs/source/changelog/0.15.4.rst create mode 100644 docs/source/changelog/0.15.5.rst create mode 100644 docs/source/changelog/0.16.0.rst create mode 100644 docs/source/changelog/0.16.1.rst create mode 100644 docs/source/changelog/0.16.2.rst create mode 100644 docs/source/changelog/0.16.3.rst create mode 100644 docs/source/changelog/1.0.0.rst create mode 100644 docs/source/changelog/1.0.1.rst create mode 100644 docs/source/changelog/1.0.2.rst create mode 100644 docs/source/changelog/1.0.3.rst create mode 100644 docs/source/changelog/1.1.0.rst create mode 100644 docs/source/changelog/1.1.1.rst create mode 100644 docs/source/changelog/1.2.0.rst create mode 100644 docs/source/changelog/1.2.1.rst create mode 100644 docs/source/changelog/1.2.2.rst create mode 100644 docs/source/changelog/1.3.0.rst create mode 100644 docs/source/changelog/1.3.1.rst create mode 100644 docs/source/changelog/1.3.3.rst create mode 100644 docs/source/changelog/1.3.4.rst create mode 100644 docs/source/changelog/1.3.5.rst create mode 100644 docs/source/changelog/1.4.0.rst create mode 100644 docs/source/changelog/1.4.1.rst rename CHANGELOG.rst => docs/source/changelog/1.5.0.rst (54%) create mode 100644 tests/project/test_changelog_md.py create mode 100644 util/changelog_md.py diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 000000000..e1a9d3210 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,40 @@ +## 1.5.0 -- unreleased + +The largest release since 1.0, and the first recorded here as it happened rather than reconstructed. Three more extraction engines, ESP with payload decryption, SCTP and NGAP over SCTP, a library logger that no longer hijacks the consumer's, and a defect programme run through the issue tracker across some 140 issues and pull requests between #326 and #509. + +- **Added** -- three extraction engines: `engine='pypcap'` and `engine='pcap_ct'`, two independent distributions of the same `libpcap` interface, and `engine='pypcapfile'` (#386, #405). They buy speed by doing less -- neither `pypcap` nor `pcap_ct` dissects at all, so they offer neither reassembly nor flow tracing, and `pypcapfile` has no IPv6 decoder. Install only **one** of `pypcap` and `pcap-ct`: both own the top-level `pcap` module, and with both present `pcap-ct` wins the import and the other becomes unselectable. The matching interface constants `PyPCAP`, `PCAP_CT` and `PyPCAPFile` were missing and are now exported alongside `DPKT`, `Scapy`, `PyShark` and `PCAPKit` (#412). That brings the built-in set to seven engines; 3.11 is the last interpreter on which every one of them can run, and even there two of them cannot coexist. +- **Added** -- `EngineBase.unsupported_reason`, a preflight every engine answers and `Extractor.run` consults before anything is imported. Asking for an engine that cannot run in the current environment now gives one warning naming the real cause -- a Python version, a missing `tshark`, a missing `libpcap`, the wrong `pcap` distribution -- and a clean fall back to `pcapkit`'s own parser, rather than an error from inside the third-party package (#396, #405). +- **Added** -- ESP parsing and construction [[RFC 4303](https://datatracker.ietf.org/doc/html/rfc4303)], with optional payload decryption and ICV verification through `cryptography` (`pip install pypcapkit[crypto]`) (#378). Keys reach the dissector through a new caller-state channel, `pcapkit.corekit.context`, surfaced as the `context=` keyword on `extract()` and `Extractor`, carrying an `esp.SecurityAssociation`. Nothing here raises: with no association the SPI and sequence number are still reported and the ciphertext is left opaque, with `status=NO_SA`, and a failed decryption or ICV check is recorded on the parsed result the same way. Extended Sequence Numbers, TFC padding and anti-replay are not implemented, and an unsupported cipher or MAC is refused with a clear error rather than half-processed. +- **Added** -- SCTP as a transport protocol [[RFC 9260](https://datatracker.ietf.org/doc/html/rfc9260)]: all 13 chunk types, 8 chunk parameters and 13 error causes, with the CRC32c both recorded and verifiable -- it covers the SCTP packet alone, with no IP pseudo-header, so it can be checked from the SCTP bytes. Upper layers register on the DATA chunk's Payload Protocol Identifier through `register_sctp`, not on a port (#379). +- **Added** -- NGAP over SCTP (3GPP TS 38.413), decoding aligned PER through `pycrate` (`pip install pypcapkit[NGAP]`) (#251, #417). Decoding is generic by ASN.1 shape rather than per-procedure, so all 81 elementary procedures and 438 protocol IEs work and a new 3GPP release needs no code change. Registered as a default on PPID 60 and 66, but only 60 decodes: PPID 66 is an NGAP PDU inside a DTLS record, and there is no DTLS dissector. `pycrate` is deliberately excluded from the `all` extra -- it is LGPL-2.1+ where this package is BSD-3-Clause, and lands some 238 MB to obtain one module. +- **Added** -- the Mobility Header registry, completed (#383, #437). The [RFC 5568](https://datatracker.ietf.org/doc/html/rfc5568) fast-handover messages and options first, then all 24 registered message data types, 70 of the 71 registered options -- with nested sub-option registries for the flow identification, access network identifier, quality-of-service and LMA-controlled MAG parameter families -- and all 4 CGA extensions. Only the CGA Parameters option remains on the generic handler. +- **Added** -- dispatch entries for dissectors that existed but were reachable from no registry (#436): FTP-DATA on TCP 20, HTTP/1 on TCP 8080, HTTP on UDP 8080, `L2TPv2` on UDP 1701 and OSPF at `TransType` 89. `VLAN` became an abstract base with `C_Tag` (802.1Q) and `S_Tag` (802.1ad) as concrete subclasses, so a Q-in-Q frame no longer collapses into one opaque `Raw`; `L2TP` likewise became a base, with `L2TPv2` carrying the [RFC 2661](https://datatracker.ietf.org/doc/html/rfc2661) implementation. +- **Added** -- `pcapkit.utilities.logging` as a real interface: `get_logger()` for per-module children, `configure()` to set level, handler, stream, format or propagation at runtime, `reset()` to return to library-neutral, and `ensure_output()`. Seventeen modules now log under their own `__name__`, so a consumer can silence `pcapkit.foundation.registry` while keeping `pcapkit.foundation.extraction` (#384). +- **Added** -- `conflict` on the reassembly data models: absolute, inclusive ranges where two fragments claimed the same span with different bytes, which was previously lost silently on both the IP (#482) and TCP (#443, #478) paths. +- **Added** -- an end-to-end test tier (#376), sample-capture generators so a fresh clone can rebuild every fixture (#340), a Dockerised engine benchmark covering every supported Python version (#410), and registry round-trip coverage that records the entries which cannot close the cycle rather than skipping them (#440, #504). +- **Changed** -- `pcapkit` no longer configures logging at import. It installs a `NullHandler` and sets no level, so verbosity is inherited from the application instead of being seized by whichever library was imported second; the old stderr handler stays as the `PCAPKIT_DEVMODE` opt-in. Three consequences worth knowing: the previous behaviour is `configure(logging.INFO, stream=sys.stderr)`; 38 registry and extractor `info` calls became `debug`, so those messages are invisible even at `INFO`; and the handler is no longer `logger.handlers[0]`. `verbose=` output stays on stdout and is not logging (#384). +- **Changed** -- each warning is reported once per channel, and `pcapkit` no longer inserts a `simplefilter('ignore', ...)` at the front of the process-global `warnings.filters` (#362--#364, #390). The application's own filter therefore wins now, which is the point of the change and also the sharp edge in it: under `-W error`, or pytest's `filterwarnings = error`, a pcapkit warning that used to be suppressed will raise. Suppress them deliberately with `warnings.filterwarnings('ignore', category=BaseWarning)`. `quiet=True` now means no record at any level and no longer sets `sys.tracebacklimit`, and the `pcapkit.utilities.warnings.DEVMODE` re-export is gone -- its canonical home is `pcapkit.utilities.logging`. +- **Changed** -- `layer=` and `protocol=` are honoured rather than inert. Both were read under the wrong names, so every value a caller passed was dropped into `**kwargs` and discarded; the CLI's `-L` also now validates its argument instead of accepting anything. The packet context reaches the schema layer for the first time as well, so a field the wire elides can be resolved from its enclosing packet (#404). `follow_tcp_stream` dispatches on the engine type, where both branches of the old test were dead and the native adapter ran against every engine's frames (#402). +- **Changed** -- two reassembly and flow-tracing defaults moved (#435), and both are visible to a caller. `Datagram.completed` widened from `bool` to a `Completion` enumeration (`COMPLETE`, `PARTIAL`, `TIMEOUT`); only `COMPLETE` is truthy, so `if datagram.completed:` is unaffected but `datagram.completed == True` no longer holds. TCP flow tracing is **bidirectional by default**, which merges each flow's two halves and closes one only once both have FINed -- 331 flows become 111 on the sample HTTP capture, the difference being single-frame stray tails; pass `trace_bidirectional=False` for the old behaviour. IP reassembly also gained the 60-second timeout [[RFC 1122](https://datatracker.ietf.org/doc/html/rfc1122), [RFC 8200](https://datatracker.ietf.org/doc/html/rfc8200)], clocked off the capture's own timestamps rather than the wall clock, tunable with `reasm_timeout=`; TCP reassembly gets no timeout by default. `trace_analyse=` is new, and reassembles each traced flow's application layer. +- **Changed** -- conflicting TCP overlaps resolve first-write-wins, per [RFC 9293](https://datatracker.ietf.org/doc/html/rfc9293) section 3.10, where they had silently resolved last-write-wins (#443, #478). A deliberate behaviour break, and a narrow one: a conforming retransmission carries identical bytes, so nothing changes for it. IP fragment reassembly keeps last-write-wins, because [RFC 791](https://datatracker.ietf.org/doc/html/rfc791) specifies the opposite resolution, and records the disagreement instead (#482). +- **Changed** -- `Probe`, `CipherSuite` and `IntegritySuite` are `Info` subclasses rather than `typing.NamedTuple`, and no `NamedTuple` remains in the package. They are Mappings now, so `len()` and iteration yield field names rather than values. +- **Changed** -- renames with no compatibility alias left behind: `HoleDiscriptor` is spelled `HoleDescriptor` and its package alias `TCP_HoleDiscriptor` is `TCP_HoleDescriptor` (#350); PCAP-NG `Option` subclasses spell the namespace class keyword `ns=` instead of `namespace=` (#439); and `examples/sample` and `examples/samples` -- one letter apart, holding different things -- are now `examples/captures` and `examples/generators`. +- **Changed** -- extraction is around 46% faster on a 1,117-frame HTTP capture, with byte-identical output (#420). A reassembled datagram's payload is now analysed on first read rather than eagerly, which cuts IP reassembly's own cost by 90.7% and TCP's by 23.7% -- IP reassembly submits a datagram for every frame, fragmented or not (#424). Flow tracing over the same capture went from 1416.6 ms to 744.0 ms, because the flow dumper had been handing each record to a `Frame` constructor that re-dissected the whole protocol stack to return bytes it had just been given; options are no longer parsed twice either (#427). All output compared byte-for-byte across the sample captures in each case. +- **Fixed** -- next-layer, option, chunk, block and parameter dispatch all read `defaultdict` registries, so a lookup miss inserted the key into class-level state shared by every later instance, after which a legitimate `register_*` call warned that the code was already registered. Every read now goes through a lookup that does not grow the table, and `IPv4.__option__` and `HIP.__parameter__` became inspectable class attributes rather than names assembled at call time (#426, #428, #429, #434). One break comes with it: a tuple-registered handler pair written to the documented `OptionParser`/`OptionConstructor` signature now works where it could previously never be called at all, and a pair written with an explicit leading `self` -- the only shape that used to work -- now does not. +- **Fixed** -- on Python 3.10 and older, no `Schema` subclass got its own `_abc_impl`: all of them fell through to `collections.abc.Mapping`'s, so a single `isinstance` or `issubclass` answer poisoned every later question about that class for the rest of the process. A terminating PCAP-NG `EndRecord` tested `True` as an `IPv4Record` (#439). +- **Fixed** -- construction, which was broken in several places at once: the generated typed `__init__` was never installed, so `__post_init__` did not run and a schema built from a subset of its fields could not be packed at all -- `UDP(srcport=53, dstport=5353)` now packs (#430); IPv6 and Mobility Header option padding was wrong, leaving construction wholly broken (#398); `HTTP.make` called the versioned `make` unbound, so every real call raised `TypeError` (#452, #462); and `IPv4._make_data` returned the fragment offset in octets where the wire wants 8-octet units, and read `data.options` on a packet that has none (#494, #499). +- **Fixed** -- a truncated or under-declared area no longer parses "successfully", and no longer wedges the process. The option and list loops could spin forever with no exception on a truncated area, reachable from untrusted input through HOPOPT, IPv6-Opts, MH, HIP and SCTP; each iteration must now advance the stream by at least one octet, and the error names the option, the offset and the octets remaining (#431, #432). Separately, wire-derived lengths in `ipv6_opts`, CALIPSO, MPL, REG_INFO and four HIP list callbacks underflowed below zero, which `ListField`'s own `while length > 0` then turned into a silent empty list; they are floored and raise instead (#449, #456, #460, #463). +- **Fixed** -- field widths and units, each measured against the specification rather than inferred: HIP's `TRANSPORT_FORMAT_LIST`, `NAT_TRAVERSAL_MODE` and `ESP_TRANSFORM` list entries are two octets, not one [[RFC 7401](https://datatracker.ietf.org/doc/html/rfc7401), [RFC 5770](https://datatracker.ietf.org/doc/html/rfc5770), [RFC 7402](https://datatracker.ietf.org/doc/html/rfc7402)] (#463, #472); the MN-ID option sizes from its subtype, not from `identifier`'s Python type (#448, #464, #467); IPv6-Route's `Hdr Ext Len` is computed in 8-octet units on both sides [[RFC 8200](https://datatracker.ietf.org/doc/html/rfc8200)] (#487, #489); and the Fast Binding Update and Acknowledgment Lifetimes are plain seconds [[RFC 5568](https://datatracker.ietf.org/doc/html/rfc5568)], not [RFC 6275](https://datatracker.ietf.org/doc/html/rfc6275)'s four-second units (#502). +- **Fixed** -- stdlib exceptions leaking out where the library's own were promised: a malformed IP field value raised a bare `ValueError` instead of `FieldValueError` (#465); a `bool` address was silently packed as `0.0.0.1` or `0.0.0.0`, `bool` being an `int` subclass (#491, #500); and `@prepare` discarded extra arguments silently and treated a *declared* zero length as end of stream, which is now distinguished from a genuinely exhausted one and raises `StreamEOFError` (#454, #458). +- **Fixed** -- the engine adapters, which were quietly wrong rather than loud. The `dpkt` toolkit split TCP and IPv4 headers at their fixed struct size instead of their real length, so option octets overwrote payload in the sequence-indexed reassembly buffer; it also read an `ipv6_frag.nh` that `dpkt` does not have, and passed fragment offsets unscaled (#351, #370, #385, #395). `scapy` never loaded its layer registry, so that engine did not dissect at all (#409), and its IPv4 fragment offset reached reassembly unscaled (#483, #484). The four IPv6 adapters disagreed about whether the 8-octet Fragment header belongs to `ihl`, `header` and `tl`; per [RFC 8200](https://datatracker.ietf.org/doc/html/rfc8200) section 4.5 it belongs to none of them, and all four now agree (#415, #424). +- **Fixed** -- PCAP and PCAP-NG output and parsing: `bytes(frame)` returned the *next* frame's octets, `files=True` wrote names like `Frame 1..json`, and a PCAP-NG `timestamp_epoch` was shifted by the reading host's timezone (#403); seven further parser defects (#341--#347, #371) and, on the write path, four more block-parsing ones (#388); `BitField` packed every named bit as set (#359, #374); the extension-header walk failed to advance past the last IPv6 extension header (#348, #373); and IPv6 fragment offsets went unscaled, with reassembly keyed on the flow label -- optional, and routinely zero, so distinct datagrams collapsed together -- rather than on the fragment identification (#389). +- **Fixed** -- TCP reassembly mixed absolute sequence numbers with buffer-relative slicing, so on any capture carrying a SYN with a realistic initial sequence number incomplete datagrams were dropped silently, and the `completed=False` branch of the public API was unreachable (#349, #376). +- **Fixed** -- constant lookups that rejected a value the registry defines. `RouterAlert(0)` is the only value [RFC 2113](https://datatracker.ietf.org/doc/html/rfc2113) defines and the one IGMP, RSVP and MLD actually send, and it was discarded because the vendor crawler skipped a header row IANA's CSV does not have; IPX `Socket(0)` is that protocol's own default, so `bytes(IPX(...))` crashed on its own defaults; and two FTP `_missing_` overrides were plain methods rather than classmethods, so every unregistered value raised `TypeError` instead of extending the enumeration (#492, #503). +- **Fixed** -- `format='text'` raised `AttributeError` before writing anything, naming a `dictdumper.Text` that has never existed. It now points at `Tree`, as the `'txt'` alias beside it already did. +- **Fixed** -- 45 places where a documentation page contradicted the code (#413), ambiguous cross-references and five autodoc signature failures (#416), and `Extractor`'s documented exception plus 40 phantom or stale `Args:` labels (#501). + +Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). + +--- + +Only the version being released is kept here. Every entry, including this one, lives in the repository as `docs/source/changelog/.rst`, and the whole history is rendered at . diff --git a/MANIFEST.in b/MANIFEST.in index 55cbf739c..89194f4fc 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -14,5 +14,21 @@ prune test include README.rst include LICENSE +# The changelog needs two lines of its own, and neither is redundant. +# +# ``global-include *.rst`` above matches only ``*.rst``, so the root +# ``CHANGELOG.md`` is picked up by nothing and would be absent from every source +# distribution without this line -- measured, not assumed. +include CHANGELOG.md + +# ``CHANGELOG.md`` carries only the version being released, so on its own the +# distribution would describe one release and no history. ``prune docs`` above +# drops the rest, which is where the history lives, so re-add just that one +# directory afterwards -- the same shape as ``include README.rst``, and for the +# same reason. Each of those files is a self-contained reStructuredText entry +# that renders standalone; the Sphinx index beside them is deliberately *not* +# shipped, because it is a ``toctree`` that only Sphinx can read. +recursive-include docs/source/changelog *.rst + include pcapkit/_extern exclude pcapkit/protocols/*/NotImplemented diff --git a/docs/source/changelog.rst b/docs/source/changelog.rst index 086e8b564..a702867fe 100644 --- a/docs/source/changelog.rst +++ b/docs/source/changelog.rst @@ -1,14 +1,74 @@ -.. This page is a thin wrapper around the repository-root ``CHANGELOG.rst``, - which is the single source of truth. It lives at the root rather than here - because ``MANIFEST.in`` carries both ``global-include *.rst`` and - ``prune docs``: a changelog kept only under ``docs/`` would be absent from - every source distribution, which is where packagers read it. A copy in both - places would drift, so this file duplicates no content. - - Deliberately no title of its own. The included file opens with its own - ``Changelog`` document title, which becomes this page's title and its - toctree label; adding a second one here renders two ``

`` elements on the - same page. Measured, not assumed -- a stub carrying its own title produced - ``h1 Release Notes`` followed by ``h1 Changelog``. - -.. include:: ../../CHANGELOG.rst +.. The changelog's index. Every entry is its own document under + ``changelog/``, one per released version, and this page is the title, the + preamble and the table of contents over them -- there is exactly one copy of + each entry and nothing to keep in step by hand. + + The repository root carries ``CHANGELOG.md`` rather than a second copy of + this history. It holds only the version being released, in Markdown, because + its consumers are the ``Create Release`` workflow's release body and the + source distribution, and both of those read Markdown. It is generated from + ``changelog/.rst``, so this tree stays the single source. + + The preamble below is verbatim from the single-file changelog it replaces. + +========= +Changelog +========= + +All notable changes to PyPCAPKit are recorded here. + + **Note** -- Versions before 1.5.0 are reconstructed from the git history, so + they summarise each release rather than enumerate every change. The record + starts at 0.13.0 (2018-12-08); the earlier 0.x releases are not covered. + + **Note** -- Post-releases (``X.Y.Z.postN``) are, from 1.0.1 onwards, automated + publications of the weekly registry refresh: a bot regenerates the vendor + constant enumerations under ``pcapkit.const`` from the upstream IANA + registries and bumps the version. They carry no library changes, and are + collapsed into a single line per release below. Where a post-release did + carry something real, it is called out. + +Each release has its own page below, newest first. The repository root's +:file:`CHANGELOG.md` carries only the version currently being released; this is +the whole history. + +.. toctree:: + :maxdepth: 1 + + changelog/1.5.0 + changelog/1.4.1 + changelog/1.4.0 + changelog/1.3.5 + changelog/1.3.4 + changelog/1.3.3 + changelog/1.3.1 + changelog/1.3.0 + changelog/1.2.2 + changelog/1.2.1 + changelog/1.2.0 + changelog/1.1.1 + changelog/1.1.0 + changelog/1.0.3 + changelog/1.0.2 + changelog/1.0.1 + changelog/1.0.0 + changelog/0.16.3 + changelog/0.16.2 + changelog/0.16.1 + changelog/0.16.0 + changelog/0.15.5 + changelog/0.15.4 + changelog/0.15.3 + changelog/0.15.2 + changelog/0.15.1 + changelog/0.15.0 + changelog/0.14.5 + changelog/0.14.4 + changelog/0.14.3 + changelog/0.14.2 + changelog/0.14.1 + changelog/0.14.0 + changelog/0.13.3 + changelog/0.13.2 + changelog/0.13.1 + changelog/0.13.0 diff --git a/docs/source/changelog/0.13.0.rst b/docs/source/changelog/0.13.0.rst new file mode 100644 index 000000000..58bddc06e --- /dev/null +++ b/docs/source/changelog/0.13.0.rst @@ -0,0 +1,7 @@ +0.13.0 -- 2018-12-08 +==================== + +No library changes -- a licence and packaging metadata refresh. + +*Also released:* ``0.13.0.post1``, ``0.13.0.post2`` (through 2018-12-12); +``post2`` fixed encoding errors in ``setup.py``. diff --git a/docs/source/changelog/0.13.1.rst b/docs/source/changelog/0.13.1.rst new file mode 100644 index 000000000..264dbf052 --- /dev/null +++ b/docs/source/changelog/0.13.1.rst @@ -0,0 +1,8 @@ +0.13.1 -- 2019-01-23 +==================== + +* **Added** -- ``tbtrim`` is used to trim PyPCAPKit's own frames out of + tracebacks through a custom ``excepthook``, so a traceback points at the + caller's code. +* **Fixed** -- compatibility issues, with a new ``pcapkit.utilities.compat`` + module. diff --git a/docs/source/changelog/0.13.2.rst b/docs/source/changelog/0.13.2.rst new file mode 100644 index 000000000..4362af65f --- /dev/null +++ b/docs/source/changelog/0.13.2.rst @@ -0,0 +1,4 @@ +0.13.2 -- 2019-01-24 +==================== + +Maintenance release: the ``tbtrim`` dependency was updated. diff --git a/docs/source/changelog/0.13.3.rst b/docs/source/changelog/0.13.3.rst new file mode 100644 index 000000000..8070d4b36 --- /dev/null +++ b/docs/source/changelog/0.13.3.rst @@ -0,0 +1,9 @@ +0.13.3 -- 2019-02-26 +==================== + +**Fixed** -- numeric literals in the constant modules that were incompatible +with Python 3.5. + +*Also released:* ``0.13.3.post1``, ``0.13.3.post2`` (through 2019-03-01); +``post1`` declared official support for Python 3.5 and below, and ``post2`` +refreshed the dependencies. diff --git a/docs/source/changelog/0.14.0.rst b/docs/source/changelog/0.14.0.rst new file mode 100644 index 000000000..011c7a6dd --- /dev/null +++ b/docs/source/changelog/0.14.0.rst @@ -0,0 +1,5 @@ +0.14.0 -- 2019-03-02 +==================== + +**Changed** -- the TCP reassembly pipeline handles the RST flag. Docker build +files were added. diff --git a/docs/source/changelog/0.14.1.rst b/docs/source/changelog/0.14.1.rst new file mode 100644 index 000000000..781132495 --- /dev/null +++ b/docs/source/changelog/0.14.1.rst @@ -0,0 +1,5 @@ +0.14.1 -- 2019-03-07 +==================== + +**Added** -- the ``PCAPKIT_DEVMODE`` environment variable, which turns on +development-mode behaviour without a code change. diff --git a/docs/source/changelog/0.14.2.rst b/docs/source/changelog/0.14.2.rst new file mode 100644 index 000000000..3ea6bc3cf --- /dev/null +++ b/docs/source/changelog/0.14.2.rst @@ -0,0 +1,4 @@ +0.14.2 -- 2019-03-28 +==================== + +**Fixed** -- the exception classes, and the TCP reassembly algorithm. diff --git a/docs/source/changelog/0.14.3.rst b/docs/source/changelog/0.14.3.rst new file mode 100644 index 000000000..13ca9964e --- /dev/null +++ b/docs/source/changelog/0.14.3.rst @@ -0,0 +1,5 @@ +0.14.3 -- 2019-08-08 +==================== + +**Fixed** -- reported bugs (#29, #30), and a Wikipedia block that broke a vendor +crawler. Constant enumerations and the sample output were regenerated. diff --git a/docs/source/changelog/0.14.4.rst b/docs/source/changelog/0.14.4.rst new file mode 100644 index 000000000..7da8893ad --- /dev/null +++ b/docs/source/changelog/0.14.4.rst @@ -0,0 +1,6 @@ +0.14.4 -- 2019-09-01 +==================== + +* **Added** -- the vendor crawler scripts ship in the distribution. +* **Changed** -- the CLI and vendor dependencies are separate extras in + ``setup.py``; constant enumerations regenerated. diff --git a/docs/source/changelog/0.14.5.rst b/docs/source/changelog/0.14.5.rst new file mode 100644 index 000000000..ca7e4586f --- /dev/null +++ b/docs/source/changelog/0.14.5.rst @@ -0,0 +1,6 @@ +0.14.5 -- 2019-10-24 +==================== + +* **Added** -- a CLI for the vendor crawlers. +* **Changed** -- vendor imports are deferred, so the crawler dependencies are + not needed to import ``pcapkit``; dependencies revised. diff --git a/docs/source/changelog/0.15.0.rst b/docs/source/changelog/0.15.0.rst new file mode 100644 index 000000000..cdfd3bc26 --- /dev/null +++ b/docs/source/changelog/0.15.0.rst @@ -0,0 +1,11 @@ +0.15.0 -- 2020-06-07 +==================== + +* **Changed** -- the parsing and construction logic was merged, so a protocol is + no longer implemented twice; TCP flow tracing and several interfaces were + revised. +* **Added** -- full API documentation, published at + https://pypcapkit.jarryshaw.me. +* **Fixed** -- the DictDumper dependency and the failures it caused (#37, #40). + +Preceded by ``0.15.0rc1`` (2020-06-06). diff --git a/docs/source/changelog/0.15.1.rst b/docs/source/changelog/0.15.1.rst new file mode 100644 index 000000000..ec3ae7e30 --- /dev/null +++ b/docs/source/changelog/0.15.1.rst @@ -0,0 +1,8 @@ +0.15.1 -- 2020-06-18 +==================== + +Maintenance release: the constant enumerations and the vendor crawlers that +generate them were revised. + +*Also released:* ``0.15.1.post1`` (2020-06-19), a refresh of the constant +enumerations. diff --git a/docs/source/changelog/0.15.2.rst b/docs/source/changelog/0.15.2.rst new file mode 100644 index 000000000..95592198b --- /dev/null +++ b/docs/source/changelog/0.15.2.rst @@ -0,0 +1,7 @@ +0.15.2 -- 2020-06-27 +==================== + +* **Added** -- a register interface on the protocol classes, and the same on + ``pcapkit.foundation.analysis``, so both can be extended without patching the + library. +* **Fixed** -- flow tracing output format and content consistency. diff --git a/docs/source/changelog/0.15.3.rst b/docs/source/changelog/0.15.3.rst new file mode 100644 index 000000000..5550c45d7 --- /dev/null +++ b/docs/source/changelog/0.15.3.rst @@ -0,0 +1,5 @@ +0.15.3 -- 2020-08-17 +==================== + +* **Fixed** -- ARP parsing (#55). +* **Added** -- logging, through a ``pcapkit`` logger. diff --git a/docs/source/changelog/0.15.4.rst b/docs/source/changelog/0.15.4.rst new file mode 100644 index 000000000..0808e969d --- /dev/null +++ b/docs/source/changelog/0.15.4.rst @@ -0,0 +1,5 @@ +0.15.4 -- 2020-08-28 +==================== + +**Fixed** -- the missing ``stacklevel`` attribute on ``pcapkit.utilities`` (#58). +Travis CI was set up. diff --git a/docs/source/changelog/0.15.5.rst b/docs/source/changelog/0.15.5.rst new file mode 100644 index 000000000..a9a2a6850 --- /dev/null +++ b/docs/source/changelog/0.15.5.rst @@ -0,0 +1,5 @@ +0.15.5 -- 2020-10-11 +==================== + +**Fixed** -- a fallback encoder was added so that data DictDumper cannot encode +no longer raises out of the dumper (#65). diff --git a/docs/source/changelog/0.16.0.rst b/docs/source/changelog/0.16.0.rst new file mode 100644 index 000000000..11925d4af --- /dev/null +++ b/docs/source/changelog/0.16.0.rst @@ -0,0 +1,23 @@ +0.16.0 -- 2022-05-31 +==================== + +A project-wide revision, and the last release of the pre-1.0 design. + +* **Changed** -- type annotations throughout, and linter compliance + (pylint, mypy, bandit, vermin) across every module. +* **Changed** -- protocol classes were separated from their data models, which + moved to ``pcapkit.protocols.data``; a single ``Protocol`` class handles both + parsing and construction, and subclassing it to add a protocol is far less + work. +* **Added** -- ``pcapkit.foundation.registry``, so protocols can be subscribed + by name; ``Protocol.analyze`` for analysing a payload directly, used by the + reassembly classes; ``pcapkit.corekit.multidict``, adapted from Werkzeug. +* **Changed** -- reassembly became a package (IP, IPv4, IPv6, TCP) with revised + data models; PCAP and the auxiliary protocols moved under ``pcapkit.misc``. +* **Removed** -- the ``validators`` module, several decorators from + ``pcapkit.utilities``, and multiprocessing support in ``Extractor``. +* **Fixed** -- the DPKT toolkit (#101), IP reassembly, and missing ``id`` + methods on subclassed protocols. + +The PyPCAPKit Enhancement Proposals discussion channel was opened with this +release (#106). diff --git a/docs/source/changelog/0.16.1.rst b/docs/source/changelog/0.16.1.rst new file mode 100644 index 000000000..5ac4e60b1 --- /dev/null +++ b/docs/source/changelog/0.16.1.rst @@ -0,0 +1,6 @@ +0.16.1 -- 2022-06-07 +==================== + +**Changed** -- warnings are raised through ``pcapkit.utilities.warnings`` +instead of ``warnings.warn`` directly, and a missing optional dependency now +warns for the CLI and vendor extras too. diff --git a/docs/source/changelog/0.16.2.rst b/docs/source/changelog/0.16.2.rst new file mode 100644 index 000000000..c78d53adc --- /dev/null +++ b/docs/source/changelog/0.16.2.rst @@ -0,0 +1,5 @@ +0.16.2 -- 2022-08-03 +==================== + +**Fixed** -- ``Info`` ``__init__`` generation for classes without annotations +(#113); the IP reassembly algorithm (#82). diff --git a/docs/source/changelog/0.16.3.rst b/docs/source/changelog/0.16.3.rst new file mode 100644 index 000000000..1cbb91cf3 --- /dev/null +++ b/docs/source/changelog/0.16.3.rst @@ -0,0 +1,5 @@ +0.16.3 -- 2022-10-31 +==================== + +**Fixed** -- import crashes on some interpreters (#114, #116); the reassembly +property caches; README rendering for PyPI. The build chain was revised. diff --git a/docs/source/changelog/1.0.0.rst b/docs/source/changelog/1.0.0.rst new file mode 100644 index 000000000..e4562011d --- /dev/null +++ b/docs/source/changelog/1.0.0.rst @@ -0,0 +1,31 @@ +1.0.0 -- 2023-05-09 +=================== + +The 1.0 rewrite: parsing and construction are one declarative definition per +protocol, and the extraction backend became pluggable. Roughly 55,000 lines +changed across 617 files, released after 21 beta builds and a release candidate. + +* **Added** -- the schema and field layer, ``pcapkit.protocols.schema`` and + ``pcapkit.corekit.fields``. A protocol declares its wire format once, as + fields (numbers, strings, IP addresses, payloads, options, lists, switches), + and both parsing and construction follow from it. +* **Added** -- PCAP-NG support [1]_: the protocol implementation covering every + block type, the generic and per-block options, name-resolution records and + decryption secrets, plus a matching extraction engine, toolkit functions and + ``pcapkit.protocols.schema.misc.pcapng``. +* **Added** -- ``pcapkit.foundation.engines``, making the extraction backend a + registered, swappable component: the built-in PCAP and PCAP-NG engines + alongside DPKT, Scapy and PyShark, with engine registry APIs. +* **Added** -- ``SchemaWarning``, ``DeprecatedFormatWarning`` and + ``RegistryWarning``; ``Protocol._get_payload`` for customised payload + retrieval; ``packet`` passed down the protocol chain so a layer can see its + parent's context, which is how IPv6 hands source and destination addresses to + the transport layer. +* **Changed** -- flow tracing became the ``pcapkit.foundation.traceflow`` + package, and reassembly gained data-model modules; + ``pcapkit.toolkit.default`` was renamed ``pcapkit.toolkit.pcap``; + ``pcapkit.foundation.engine`` became ``engines``; ``IPField`` was split into + ``IPAddressField`` and ``IPInterfaceField``; the option and parameter + registries for HIP and IPv4 warn on overwrite. + +.. [1] PCAP-NG is specified by ``draft-tuexen-opsawg-pcapng``, not by an RFC. diff --git a/docs/source/changelog/1.0.1.rst b/docs/source/changelog/1.0.1.rst new file mode 100644 index 000000000..16c1e5909 --- /dev/null +++ b/docs/source/changelog/1.0.1.rst @@ -0,0 +1,8 @@ +1.0.1 -- 2023-05-14 +=================== + +**Changed** -- ``Info`` and ``Schema`` subclasses are finalised when the class is +created rather than on each instantiation, which is a straight runtime saving on +parsing. + +*Also released:* ``1.0.1.post1`` through ``1.0.1.post3`` (through 2023-05-27). diff --git a/docs/source/changelog/1.0.2.rst b/docs/source/changelog/1.0.2.rst new file mode 100644 index 000000000..f81efa9b8 --- /dev/null +++ b/docs/source/changelog/1.0.2.rst @@ -0,0 +1,12 @@ +1.0.2 -- 2023-06-05 +=================== + +* **Added** -- Mobility Header (MH) parsing and construction [:rfc:`6275`], + including the message types, mobility options, CGA parameters and link-layer + address option codes, and the binding error status codes. +* **Fixed** -- ``pcapkit.extract`` failing on interpreters where + ``decimal.localcontext`` takes no keyword arguments (#139); wheel filenames + now carry the correct Python tags. + +*Also released:* ``1.0.2.post1`` through ``1.0.2.post8`` (through 2023-06-27); +``post6`` also reset the conda build number. diff --git a/docs/source/changelog/1.0.3.rst b/docs/source/changelog/1.0.3.rst new file mode 100644 index 000000000..53be2ba55 --- /dev/null +++ b/docs/source/changelog/1.0.3.rst @@ -0,0 +1,11 @@ +1.0.3 -- 2023-06-29 +=================== + +* **Added** -- the IANA service name and transport protocol port number registry + as the ``AppType`` enumeration [:rfc:`6335`], wired into TCP and UDP parsing + so ports resolve to service names; a generated enumeration for the TCP header + flags; an undefined-command member for FTP commands. +* **Changed** -- registry management was redesigned; the MPTCP and PCAP-NG + option enumerations were revised. + +*Also released:* ``1.0.3.post1`` through ``1.0.3.post3`` (through 2023-07-04). diff --git a/docs/source/changelog/1.1.0.rst b/docs/source/changelog/1.1.0.rst new file mode 100644 index 000000000..68a7fd81f --- /dev/null +++ b/docs/source/changelog/1.1.0.rst @@ -0,0 +1,12 @@ +1.1.0 -- 2023-07-09 +=================== + +* **Changed** -- the schema and field classes were rebuilt on metaclasses, with + ``EnumSchema`` for enum-dispatched schemas, and every protocol schema was + revised onto them: IPv4, HOPOPT, IPv6-Opts, IPv6-Route, HIP, MH, TCP, HTTP/2 + and PCAP-NG. +* **Added** -- schema registration through the protocol registry APIs; + persistent additional and excluded fields on ``Info``; numeric and comparison + support on the PCAP-NG option code and application-layer port enumerations. + +*Also released:* ``1.1.0.post1`` through ``1.1.0.post3`` (through 2023-08-12). diff --git a/docs/source/changelog/1.1.1.rst b/docs/source/changelog/1.1.1.rst new file mode 100644 index 000000000..a405ccfac --- /dev/null +++ b/docs/source/changelog/1.1.1.rst @@ -0,0 +1,8 @@ +1.1.1 -- 2023-08-15 +=================== + +**Fixed** -- ``AppType`` enumeration lookups; ``typing.TypeAlias`` compatibility +on older interpreters; the output file name chosen by flow tracing; dumping of +``Schema`` objects through a custom dumper. + +*Also released:* ``1.1.1.post1``, ``1.1.1.post2`` (through 2023-08-23). diff --git a/docs/source/changelog/1.2.0.rst b/docs/source/changelog/1.2.0.rst new file mode 100644 index 000000000..f4eaa1514 --- /dev/null +++ b/docs/source/changelog/1.2.0.rst @@ -0,0 +1,9 @@ +1.2.0 -- 2023-08-26 +=================== + +* **Added** -- streaming input. ``pcapkit.corekit.io.SeekableReader`` buffers a + non-seekable stream so a capture can be read from ``stdin`` or a pipe, which + is what live ``tcpdump`` output needs (#156). +* **Fixed** -- reassembly, broken since 0.16.3 (#155). ``Extractor`` now raises + an explicit unsupported-call error, and logs a notice, when reassembly or flow + tracing is requested where it cannot be provided. diff --git a/docs/source/changelog/1.2.1.rst b/docs/source/changelog/1.2.1.rst new file mode 100644 index 000000000..78e70abfd --- /dev/null +++ b/docs/source/changelog/1.2.1.rst @@ -0,0 +1,6 @@ +1.2.1 -- 2023-08-27 +=================== + +**Fixed** -- bugs in the ``SeekableReader`` introduced one day earlier in 1.2.0. + +*Also released:* ``1.2.1.post1`` through ``1.2.1.post3`` (through 2023-09-09). diff --git a/docs/source/changelog/1.2.2.rst b/docs/source/changelog/1.2.2.rst new file mode 100644 index 000000000..39c4eb3f2 --- /dev/null +++ b/docs/source/changelog/1.2.2.rst @@ -0,0 +1,6 @@ +1.2.2 -- 2023-09-14 +=================== + +**Fixed** -- extraction failing outright (#166). + +*Also released:* ``1.2.2.post1``, ``1.2.2.post2`` (through 2023-10-04). diff --git a/docs/source/changelog/1.3.0.rst b/docs/source/changelog/1.3.0.rst new file mode 100644 index 000000000..d6e2b1261 --- /dev/null +++ b/docs/source/changelog/1.3.0.rst @@ -0,0 +1,14 @@ +1.3.0 -- 2023-10-04 +=================== + +A redesign of the extension points, on metaclasses. + +* **Changed** -- the protocol, reassembly, flow-tracing, engine, vendor and + field base classes are all built on metaclasses, which is what registers a + subclass and fixes its name and module. +* **Added** -- ``ModuleDescriptor``, so a registry entry can name a module and + class that are imported lazily on first use rather than at registration; the + registry functions accept it throughout. ``VersionInfo`` gained a ``version`` + property. +* **Changed** -- ``_Engine`` renamed ``EngineBase``; the ``beholder`` decorator + logs the error it swallows under devmode and verbose mode. diff --git a/docs/source/changelog/1.3.1.rst b/docs/source/changelog/1.3.1.rst new file mode 100644 index 000000000..364d1d822 --- /dev/null +++ b/docs/source/changelog/1.3.1.rst @@ -0,0 +1,11 @@ +1.3.1 -- 2023-12-21 +=================== + +* **Changed** -- ``register_port()`` was renamed ``register_apptype()``, with no + alias left behind; callers registering application-layer protocols by port + must be updated. +* **Changed** -- the documentation was revised across the whole project -- + ``pcapkit.protocols``, ``pcapkit.corekit`` and ``pcapkit.dumpkit`` in + particular. + +*Also released:* ``1.3.1.post1`` through ``1.3.1.post28`` (through 2024-10-26). diff --git a/docs/source/changelog/1.3.3.rst b/docs/source/changelog/1.3.3.rst new file mode 100644 index 000000000..9adfca4d2 --- /dev/null +++ b/docs/source/changelog/1.3.3.rst @@ -0,0 +1,16 @@ +1.3.3 -- 2024-11-03 +=================== + +* **Fixed** -- errors reading PCAP files from both the CLI and the library + (#240): missing optional CLI dependencies are handled gracefully and console + output is far less verbose; installation from source, which ``setup.py`` had + broken (#221); PCAP output during flow tracing (#180). +* **Changed** -- the base protocol data class carries ``packet``; ``Info`` + iteration skips excluded fields. + +1.3.2 was never released. The version string existed in-tree for a few hours on +2024-11-03 and was superseded by 1.3.3 the same day, so there is no ``v1.3.2`` +tag and no distribution. + +*Also released:* ``1.3.3.post1``, ``1.3.3.post2`` (through 2024-11-14); +``post1`` fixed a PyPI metadata problem. diff --git a/docs/source/changelog/1.3.4.rst b/docs/source/changelog/1.3.4.rst new file mode 100644 index 000000000..fc8d6c981 --- /dev/null +++ b/docs/source/changelog/1.3.4.rst @@ -0,0 +1,9 @@ +1.3.4 -- 2024-11-14 +=================== + +* **Fixed** -- IPv6 Routing header length handling, and extraction of the IPv6 + extension header chain (#218). +* **Changed** -- the build matrix moved to Python 3.13 and stopped testing 3.8. + +This release also introduced the transport-layer parsing regression fixed in +1.3.5; upgrade past it rather than to it. diff --git a/docs/source/changelog/1.3.5.rst b/docs/source/changelog/1.3.5.rst new file mode 100644 index 000000000..cef3273c1 --- /dev/null +++ b/docs/source/changelog/1.3.5.rst @@ -0,0 +1,12 @@ +1.3.5 -- 2024-11-16 +=================== + +**Fixed** -- transport-layer payloads were not decoded at all. An inverted +``payload is None`` check in ``pcapkit.protocols.internet.internet`` left every +TCP and UDP frame under IPv4/IPv6 reported as ``raw``, typically with +``'int' object has no attribute 'port'``. The regression was introduced by the +IPv6 extension-header fix in 1.3.4, so 1.3.4 should be skipped entirely. + +*Also released:* ``1.3.5.post1`` through ``1.3.5.post43`` (through 2026-08-21), +all registry refreshes except ``post41``, which switched the Anaconda upload to +the official action. diff --git a/docs/source/changelog/1.4.0.rst b/docs/source/changelog/1.4.0.rst new file mode 100644 index 000000000..bf4b76334 --- /dev/null +++ b/docs/source/changelog/1.4.0.rst @@ -0,0 +1,14 @@ +1.4.0 -- 2026-08-21 +=================== + +The first release in nearly two years to carry code changes, and the first with +a test suite. + +* **Added** -- a unit-test suite under ``tests/``, covering ``pcapkit.corekit``, + the extraction engines, reassembly and flow tracing, the protocol schemas, the + CLI and the utility modules. +* **Fixed** -- PCAP-NG option dispatch regressions; the fallback name generated + by the ``linktype`` vendor crawler. +* **Changed** -- CI tests current Python versions again; every GitHub Actions + job carries an explicit permissions block; the conda and cron release tooling + was repaired, and Anaconda uploads use the official upload action. diff --git a/docs/source/changelog/1.4.1.rst b/docs/source/changelog/1.4.1.rst new file mode 100644 index 000000000..1aa554f2e --- /dev/null +++ b/docs/source/changelog/1.4.1.rst @@ -0,0 +1,13 @@ +1.4.1 -- 2026-08-22 +=================== + +Release engineering only -- no library behaviour changed. + +* **Added** -- a unit-test CI workflow, and the packaging, vendor-cron and + documentation workflows are now gated on it, so a release can no longer be cut + from a tree whose tests fail. +* **Changed** -- the vendor cron refuses to publish when the registry update + itself failed, rather than shipping a half-refreshed release. +* **Fixed** -- unit-test compatibility on Python 3.10. + +*Also released:* ``1.4.1.post1``, ``1.4.1.post2`` (through 2026-09-12). diff --git a/CHANGELOG.rst b/docs/source/changelog/1.5.0.rst similarity index 54% rename from CHANGELOG.rst rename to docs/source/changelog/1.5.0.rst index 932cb9e71..9851e1d86 100644 --- a/CHANGELOG.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1,20 +1,3 @@ -========= -Changelog -========= - -All notable changes to PyPCAPKit are recorded here. - - **Note** -- Versions before 1.5.0 are reconstructed from the git history, so - they summarise each release rather than enumerate every change. The record - starts at 0.13.0 (2018-12-08); the earlier 0.x releases are not covered. - - **Note** -- Post-releases (``X.Y.Z.postN``) are, from 1.0.1 onwards, automated - publications of the weekly registry refresh: a bot regenerates the vendor - constant enumerations under ``pcapkit.const`` from the upstream IANA - registries and bumps the version. They carry no library changes, and are - collapsed into a single line per release below. Where a post-release did - carry something real, it is called out. - 1.5.0 -- unreleased =================== @@ -263,370 +246,3 @@ which ``pyproject.toml`` declares as the dynamic long description. ``1.5.0b2`` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and ``1.5.0b3`` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). - -1.4.1 -- 2026-08-22 -=================== - -Release engineering only -- no library behaviour changed. - -* **Added** -- a unit-test CI workflow, and the packaging, vendor-cron and - documentation workflows are now gated on it, so a release can no longer be cut - from a tree whose tests fail. -* **Changed** -- the vendor cron refuses to publish when the registry update - itself failed, rather than shipping a half-refreshed release. -* **Fixed** -- unit-test compatibility on Python 3.10. - -*Also released:* ``1.4.1.post1``, ``1.4.1.post2`` (through 2026-09-12). - -1.4.0 -- 2026-08-21 -=================== - -The first release in nearly two years to carry code changes, and the first with -a test suite. - -* **Added** -- a unit-test suite under ``tests/``, covering ``pcapkit.corekit``, - the extraction engines, reassembly and flow tracing, the protocol schemas, the - CLI and the utility modules. -* **Fixed** -- PCAP-NG option dispatch regressions; the fallback name generated - by the ``linktype`` vendor crawler. -* **Changed** -- CI tests current Python versions again; every GitHub Actions - job carries an explicit permissions block; the conda and cron release tooling - was repaired, and Anaconda uploads use the official upload action. - -1.3.5 -- 2024-11-16 -=================== - -**Fixed** -- transport-layer payloads were not decoded at all. An inverted -``payload is None`` check in ``pcapkit.protocols.internet.internet`` left every -TCP and UDP frame under IPv4/IPv6 reported as ``raw``, typically with -``'int' object has no attribute 'port'``. The regression was introduced by the -IPv6 extension-header fix in 1.3.4, so 1.3.4 should be skipped entirely. - -*Also released:* ``1.3.5.post1`` through ``1.3.5.post43`` (through 2026-08-21), -all registry refreshes except ``post41``, which switched the Anaconda upload to -the official action. - -1.3.4 -- 2024-11-14 -=================== - -* **Fixed** -- IPv6 Routing header length handling, and extraction of the IPv6 - extension header chain (#218). -* **Changed** -- the build matrix moved to Python 3.13 and stopped testing 3.8. - -This release also introduced the transport-layer parsing regression fixed in -1.3.5; upgrade past it rather than to it. - -1.3.3 -- 2024-11-03 -=================== - -* **Fixed** -- errors reading PCAP files from both the CLI and the library - (#240): missing optional CLI dependencies are handled gracefully and console - output is far less verbose; installation from source, which ``setup.py`` had - broken (#221); PCAP output during flow tracing (#180). -* **Changed** -- the base protocol data class carries ``packet``; ``Info`` - iteration skips excluded fields. - -1.3.2 was never released. The version string existed in-tree for a few hours on -2024-11-03 and was superseded by 1.3.3 the same day, so there is no ``v1.3.2`` -tag and no distribution. - -*Also released:* ``1.3.3.post1``, ``1.3.3.post2`` (through 2024-11-14); -``post1`` fixed a PyPI metadata problem. - -1.3.1 -- 2023-12-21 -=================== - -* **Changed** -- ``register_port()`` was renamed ``register_apptype()``, with no - alias left behind; callers registering application-layer protocols by port - must be updated. -* **Changed** -- the documentation was revised across the whole project -- - ``pcapkit.protocols``, ``pcapkit.corekit`` and ``pcapkit.dumpkit`` in - particular. - -*Also released:* ``1.3.1.post1`` through ``1.3.1.post28`` (through 2024-10-26). - -1.3.0 -- 2023-10-04 -=================== - -A redesign of the extension points, on metaclasses. - -* **Changed** -- the protocol, reassembly, flow-tracing, engine, vendor and - field base classes are all built on metaclasses, which is what registers a - subclass and fixes its name and module. -* **Added** -- ``ModuleDescriptor``, so a registry entry can name a module and - class that are imported lazily on first use rather than at registration; the - registry functions accept it throughout. ``VersionInfo`` gained a ``version`` - property. -* **Changed** -- ``_Engine`` renamed ``EngineBase``; the ``beholder`` decorator - logs the error it swallows under devmode and verbose mode. - -1.2.2 -- 2023-09-14 -=================== - -**Fixed** -- extraction failing outright (#166). - -*Also released:* ``1.2.2.post1``, ``1.2.2.post2`` (through 2023-10-04). - -1.2.1 -- 2023-08-27 -=================== - -**Fixed** -- bugs in the ``SeekableReader`` introduced one day earlier in 1.2.0. - -*Also released:* ``1.2.1.post1`` through ``1.2.1.post3`` (through 2023-09-09). - -1.2.0 -- 2023-08-26 -=================== - -* **Added** -- streaming input. ``pcapkit.corekit.io.SeekableReader`` buffers a - non-seekable stream so a capture can be read from ``stdin`` or a pipe, which - is what live ``tcpdump`` output needs (#156). -* **Fixed** -- reassembly, broken since 0.16.3 (#155). ``Extractor`` now raises - an explicit unsupported-call error, and logs a notice, when reassembly or flow - tracing is requested where it cannot be provided. - -1.1.1 -- 2023-08-15 -=================== - -**Fixed** -- ``AppType`` enumeration lookups; ``typing.TypeAlias`` compatibility -on older interpreters; the output file name chosen by flow tracing; dumping of -``Schema`` objects through a custom dumper. - -*Also released:* ``1.1.1.post1``, ``1.1.1.post2`` (through 2023-08-23). - -1.1.0 -- 2023-07-09 -=================== - -* **Changed** -- the schema and field classes were rebuilt on metaclasses, with - ``EnumSchema`` for enum-dispatched schemas, and every protocol schema was - revised onto them: IPv4, HOPOPT, IPv6-Opts, IPv6-Route, HIP, MH, TCP, HTTP/2 - and PCAP-NG. -* **Added** -- schema registration through the protocol registry APIs; - persistent additional and excluded fields on ``Info``; numeric and comparison - support on the PCAP-NG option code and application-layer port enumerations. - -*Also released:* ``1.1.0.post1`` through ``1.1.0.post3`` (through 2023-08-12). - -1.0.3 -- 2023-06-29 -=================== - -* **Added** -- the IANA service name and transport protocol port number registry - as the ``AppType`` enumeration [:rfc:`6335`], wired into TCP and UDP parsing - so ports resolve to service names; a generated enumeration for the TCP header - flags; an undefined-command member for FTP commands. -* **Changed** -- registry management was redesigned; the MPTCP and PCAP-NG - option enumerations were revised. - -*Also released:* ``1.0.3.post1`` through ``1.0.3.post3`` (through 2023-07-04). - -1.0.2 -- 2023-06-05 -=================== - -* **Added** -- Mobility Header (MH) parsing and construction [:rfc:`6275`], - including the message types, mobility options, CGA parameters and link-layer - address option codes, and the binding error status codes. -* **Fixed** -- ``pcapkit.extract`` failing on interpreters where - ``decimal.localcontext`` takes no keyword arguments (#139); wheel filenames - now carry the correct Python tags. - -*Also released:* ``1.0.2.post1`` through ``1.0.2.post8`` (through 2023-06-27); -``post6`` also reset the conda build number. - -1.0.1 -- 2023-05-14 -=================== - -**Changed** -- ``Info`` and ``Schema`` subclasses are finalised when the class is -created rather than on each instantiation, which is a straight runtime saving on -parsing. - -*Also released:* ``1.0.1.post1`` through ``1.0.1.post3`` (through 2023-05-27). - -1.0.0 -- 2023-05-09 -=================== - -The 1.0 rewrite: parsing and construction are one declarative definition per -protocol, and the extraction backend became pluggable. Roughly 55,000 lines -changed across 617 files, released after 21 beta builds and a release candidate. - -* **Added** -- the schema and field layer, ``pcapkit.protocols.schema`` and - ``pcapkit.corekit.fields``. A protocol declares its wire format once, as - fields (numbers, strings, IP addresses, payloads, options, lists, switches), - and both parsing and construction follow from it. -* **Added** -- PCAP-NG support [1]_: the protocol implementation covering every - block type, the generic and per-block options, name-resolution records and - decryption secrets, plus a matching extraction engine, toolkit functions and - ``pcapkit.protocols.schema.misc.pcapng``. -* **Added** -- ``pcapkit.foundation.engines``, making the extraction backend a - registered, swappable component: the built-in PCAP and PCAP-NG engines - alongside DPKT, Scapy and PyShark, with engine registry APIs. -* **Added** -- ``SchemaWarning``, ``DeprecatedFormatWarning`` and - ``RegistryWarning``; ``Protocol._get_payload`` for customised payload - retrieval; ``packet`` passed down the protocol chain so a layer can see its - parent's context, which is how IPv6 hands source and destination addresses to - the transport layer. -* **Changed** -- flow tracing became the ``pcapkit.foundation.traceflow`` - package, and reassembly gained data-model modules; - ``pcapkit.toolkit.default`` was renamed ``pcapkit.toolkit.pcap``; - ``pcapkit.foundation.engine`` became ``engines``; ``IPField`` was split into - ``IPAddressField`` and ``IPInterfaceField``; the option and parameter - registries for HIP and IPv4 warn on overwrite. - -.. [1] PCAP-NG is specified by ``draft-tuexen-opsawg-pcapng``, not by an RFC. - -0.16.3 -- 2022-10-31 -==================== - -**Fixed** -- import crashes on some interpreters (#114, #116); the reassembly -property caches; README rendering for PyPI. The build chain was revised. - -0.16.2 -- 2022-08-03 -==================== - -**Fixed** -- ``Info`` ``__init__`` generation for classes without annotations -(#113); the IP reassembly algorithm (#82). - -0.16.1 -- 2022-06-07 -==================== - -**Changed** -- warnings are raised through ``pcapkit.utilities.warnings`` -instead of ``warnings.warn`` directly, and a missing optional dependency now -warns for the CLI and vendor extras too. - -0.16.0 -- 2022-05-31 -==================== - -A project-wide revision, and the last release of the pre-1.0 design. - -* **Changed** -- type annotations throughout, and linter compliance - (pylint, mypy, bandit, vermin) across every module. -* **Changed** -- protocol classes were separated from their data models, which - moved to ``pcapkit.protocols.data``; a single ``Protocol`` class handles both - parsing and construction, and subclassing it to add a protocol is far less - work. -* **Added** -- ``pcapkit.foundation.registry``, so protocols can be subscribed - by name; ``Protocol.analyze`` for analysing a payload directly, used by the - reassembly classes; ``pcapkit.corekit.multidict``, adapted from Werkzeug. -* **Changed** -- reassembly became a package (IP, IPv4, IPv6, TCP) with revised - data models; PCAP and the auxiliary protocols moved under ``pcapkit.misc``. -* **Removed** -- the ``validators`` module, several decorators from - ``pcapkit.utilities``, and multiprocessing support in ``Extractor``. -* **Fixed** -- the DPKT toolkit (#101), IP reassembly, and missing ``id`` - methods on subclassed protocols. - -The PyPCAPKit Enhancement Proposals discussion channel was opened with this -release (#106). - -0.15.5 -- 2020-10-11 -==================== - -**Fixed** -- a fallback encoder was added so that data DictDumper cannot encode -no longer raises out of the dumper (#65). - -0.15.4 -- 2020-08-28 -==================== - -**Fixed** -- the missing ``stacklevel`` attribute on ``pcapkit.utilities`` (#58). -Travis CI was set up. - -0.15.3 -- 2020-08-17 -==================== - -* **Fixed** -- ARP parsing (#55). -* **Added** -- logging, through a ``pcapkit`` logger. - -0.15.2 -- 2020-06-27 -==================== - -* **Added** -- a register interface on the protocol classes, and the same on - ``pcapkit.foundation.analysis``, so both can be extended without patching the - library. -* **Fixed** -- flow tracing output format and content consistency. - -0.15.1 -- 2020-06-18 -==================== - -Maintenance release: the constant enumerations and the vendor crawlers that -generate them were revised. - -*Also released:* ``0.15.1.post1`` (2020-06-19), a refresh of the constant -enumerations. - -0.15.0 -- 2020-06-07 -==================== - -* **Changed** -- the parsing and construction logic was merged, so a protocol is - no longer implemented twice; TCP flow tracing and several interfaces were - revised. -* **Added** -- full API documentation, published at - https://pypcapkit.jarryshaw.me. -* **Fixed** -- the DictDumper dependency and the failures it caused (#37, #40). - -Preceded by ``0.15.0rc1`` (2020-06-06). - -0.14.5 -- 2019-10-24 -==================== - -* **Added** -- a CLI for the vendor crawlers. -* **Changed** -- vendor imports are deferred, so the crawler dependencies are - not needed to import ``pcapkit``; dependencies revised. - -0.14.4 -- 2019-09-01 -==================== - -* **Added** -- the vendor crawler scripts ship in the distribution. -* **Changed** -- the CLI and vendor dependencies are separate extras in - ``setup.py``; constant enumerations regenerated. - -0.14.3 -- 2019-08-08 -==================== - -**Fixed** -- reported bugs (#29, #30), and a Wikipedia block that broke a vendor -crawler. Constant enumerations and the sample output were regenerated. - -0.14.2 -- 2019-03-28 -==================== - -**Fixed** -- the exception classes, and the TCP reassembly algorithm. - -0.14.1 -- 2019-03-07 -==================== - -**Added** -- the ``PCAPKIT_DEVMODE`` environment variable, which turns on -development-mode behaviour without a code change. - -0.14.0 -- 2019-03-02 -==================== - -**Changed** -- the TCP reassembly pipeline handles the RST flag. Docker build -files were added. - -0.13.3 -- 2019-02-26 -==================== - -**Fixed** -- numeric literals in the constant modules that were incompatible -with Python 3.5. - -*Also released:* ``0.13.3.post1``, ``0.13.3.post2`` (through 2019-03-01); -``post1`` declared official support for Python 3.5 and below, and ``post2`` -refreshed the dependencies. - -0.13.2 -- 2019-01-24 -==================== - -Maintenance release: the ``tbtrim`` dependency was updated. - -0.13.1 -- 2019-01-23 -==================== - -* **Added** -- ``tbtrim`` is used to trim PyPCAPKit's own frames out of - tracebacks through a custom ``excepthook``, so a traceback points at the - caller's code. -* **Fixed** -- compatibility issues, with a new ``pcapkit.utilities.compat`` - module. - -0.13.0 -- 2018-12-08 -==================== - -No library changes -- a licence and packaging metadata refresh. - -*Also released:* ``0.13.0.post1``, ``0.13.0.post2`` (through 2018-12-12); -``post2`` fixed encoding errors in ``setup.py``. diff --git a/tests/project/test_changelog_md.py b/tests/project/test_changelog_md.py new file mode 100644 index 000000000..66e793935 --- /dev/null +++ b/tests/project/test_changelog_md.py @@ -0,0 +1,448 @@ +# -*- coding: utf-8 -*- +"""Tests for :file:`util/changelog_md.py`, the ``CHANGELOG.md`` generator. + +``CHANGELOG.md`` is a derived artefact: it is the newest +:file:`docs/source/changelog/.rst` entry converted to Markdown, so that +the ``Create Release`` workflow's release body and the source distribution get +Markdown without a second copy of the history to keep in step by hand. + +The conversion is six mechanical rules over a small reStructuredText subset, and +there is one test per rule. Each asserts both halves -- that the Markdown form +arrived *and* that the reStructuredText form is gone -- because a rule that fires +in the wrong place and a rule that does not fire at all both leave markup in a +published release body, and only checking for the new spelling would miss the +second. + +The rest cover the two things that are easy to get wrong and invisible when they +are: which entry counts as "newest" (the toctree's first line, not a sort of the +version strings), and ``--check``, which is what a CI gate calls and is worthless +if it cannot fail. + +Almost everything here builds its own two-file changelog tree in a temporary +directory rather than reading the repository's. That keeps the rule tests honest +-- each entry is written to exercise one rule, instead of hoping the real +changelog happens to contain the construct -- and it keeps them passing on a +source tarball, where :file:`docs/` has been pruned away. + +""" + +from __future__ import annotations + +import contextlib +import importlib.util +import io +import pathlib +import re +import shutil +import tempfile +import textwrap +import unittest + +ROOT = pathlib.Path(__file__).resolve().parents[2] + + +def _load_generator(): + """Load :file:`util/changelog_md.py` as a module. + + ``util/`` is a directory of scripts rather than a package, so there is no + import path to it. The script has no import-time side effects, which is what + makes loading it by location safe. + + """ + path = ROOT / 'util' / 'changelog_md.py' + spec = importlib.util.spec_from_file_location('changelog_md', path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +changelog_md = _load_generator() + + +#: One entry exercising all six rules at once: a setext heading, a wrapped +#: paragraph, ``*`` bullets whose text wraps, an ``:rfc:`` role, a double-backtick +#: literal, and a footnote reference with its definition. +ENTRY = """\ +9.9.9 -- 2026-01-02 +=================== + +An entry whose prose is wrapped across +two source lines, so that rule 6 has +something to collapse. + +* **Added** -- ESP parsing [:rfc:`4303`], with a ``literal`` in it + and a continuation line. +* **Fixed** -- PCAP-NG support [1]_. + +.. [1] PCAP-NG is specified by a draft, not by an RFC. +""" + +INDEX = """\ +========= +Changelog +========= + +Preamble. + +.. toctree:: + :maxdepth: 1 + + changelog/9.9.9 +""" + + +class ChangelogTreeMixin: + """Builds a throwaway changelog tree for one test.""" + + def make_tree(self, entry: str = ENTRY, index: str = INDEX, + version: str = '9.9.9') -> pathlib.Path: + """Write *index* and *entry* into a temporary tree; return the index path.""" + root = pathlib.Path(tempfile.mkdtemp()) + self.addCleanup(shutil.rmtree, root, True) + + (root / 'changelog').mkdir() + (root / 'changelog' / f'{version}.rst').write_text(entry, encoding='utf-8') + index_path = root / 'changelog.rst' + index_path.write_text(index, encoding='utf-8') + return index_path + + def convert(self, entry: str = ENTRY) -> str: + """Convert *entry* through the generator, as ``render`` would.""" + return changelog_md.convert(entry) + + +class ConversionRuleTests(ChangelogTreeMixin, unittest.TestCase): + """One test per conversion rule.""" + + def test_rule_1_setext_heading_becomes_atx(self) -> None: + markdown = self.convert() + + self.assertTrue( + markdown.startswith('## 9.9.9 -- 2026-01-02\n'), + f'expected an ATX heading first, got {markdown[:60]!r}', + ) + # The underline must be consumed, not merely followed by a heading. + self.assertNotRegex(markdown, r'(?m)^=+$') + + def test_rule_2_rfc_role_becomes_a_datatracker_link(self) -> None: + markdown = self.convert() + + self.assertIn('[RFC 4303](https://datatracker.ietf.org/doc/html/rfc4303)', markdown) + self.assertNotIn(':rfc:', markdown) + + def test_rule_3_double_backtick_literal_becomes_a_code_span(self) -> None: + markdown = self.convert() + + self.assertIn('`literal`', markdown) + self.assertNotIn('``', markdown) + + def test_rule_4_star_bullets_become_dash_bullets(self) -> None: + markdown = self.convert() + + bullets = [line for line in markdown.split('\n') if line.startswith('- ')] + self.assertEqual(len(bullets), 2, f'expected two dash bullets, got {bullets!r}') + self.assertNotRegex(markdown, r'(?m)^\*[ \t]') + + def test_rule_5_footnotes_become_github_footnotes(self) -> None: + markdown = self.convert() + + self.assertIn('[^1]', markdown) + self.assertRegex(markdown, r'(?m)^\[\^1\]: PCAP-NG is specified by a draft') + self.assertNotIn('[1]_', markdown) + self.assertNotRegex(markdown, r'(?m)^\.\. \[1\]') + + def test_rule_6_paragraphs_and_bullets_are_unwrapped(self) -> None: + markdown = self.convert() + + # A GitHub release body renders a single newline as a hard break, so every + # block has to be one line: the source's own wrapping must be gone. + self.assertIn( + 'An entry whose prose is wrapped across two source lines, so that ' + 'rule 6 has something to collapse.', + markdown, + ) + self.assertIn('with a `literal` in it and a continuation line.', markdown) + + # Nothing but blank lines separates the blocks, so no line may be a bare + # continuation of the one above it. + for line in markdown.split('\n'): + if line and not line.startswith(('## ', '- ', '[^')): + self.assertTrue( + line[0].isupper() or line.startswith('---'), + f'{line[:50]!r} looks like a wrapped continuation line', + ) + + +class NewestEntryTests(ChangelogTreeMixin, unittest.TestCase): + """How the generator decides which entry is the current release.""" + + def test_newest_is_the_toctree_head_not_a_version_sort(self) -> None: + index = self.make_tree( + index=textwrap.dedent("""\ + ========= + Changelog + ========= + + .. toctree:: + :maxdepth: 1 + + changelog/1.10.0 + changelog/1.9.0 + """), + version='1.10.0', + ) + (index.parent / 'changelog' / '1.9.0.rst').write_text(ENTRY, encoding='utf-8') + + version, entry = changelog_md.newest(index) + + # ``1.9.0`` sorts above ``1.10.0`` lexically and below it under PEP 440; + # the toctree settles it without the generator having to know either rule. + self.assertEqual(version, '1.10.0') + self.assertEqual(entry.name, '1.10.0.rst') + + def test_toctree_options_and_blank_lines_are_not_entries(self) -> None: + index = self.make_tree() + + self.assertEqual(changelog_md.read_toctree(index), ['changelog/9.9.9']) + + def test_explicit_title_form_is_reduced_to_the_docname(self) -> None: + index = self.make_tree(index=INDEX.replace( + ' changelog/9.9.9', ' The 9.9.9 release ')) + + self.assertEqual(changelog_md.read_toctree(index), ['changelog/9.9.9']) + + def test_dotted_version_is_not_mistaken_for_a_file_extension(self) -> None: + # ``pathlib.Path.with_suffix`` would turn ``changelog/9.9.9`` into + # ``changelog/9.9.rst``, because a version string ends in what looks like + # an extension. + version, entry = changelog_md.newest(self.make_tree()) + + self.assertEqual(version, '9.9.9') + self.assertEqual(entry.name, '9.9.9.rst') + + def test_missing_toctree_is_an_error(self) -> None: + index = self.make_tree(index='=========\nChangelog\n=========\n') + + with self.assertRaises(ValueError): + changelog_md.read_toctree(index) + + def test_empty_toctree_is_an_error(self) -> None: + index = self.make_tree(index='.. toctree::\n :maxdepth: 1\n') + + with self.assertRaises(ValueError): + changelog_md.read_toctree(index) + + def test_toctree_head_naming_a_missing_file_is_an_error(self) -> None: + index = self.make_tree(index=INDEX.replace('9.9.9', '9.9.8')) + + with self.assertRaises(FileNotFoundError): + changelog_md.newest(index) + + +class ResidualMarkupTests(ChangelogTreeMixin, unittest.TestCase): + """The guard against six regexes quietly copying through what they cannot convert.""" + + def test_an_unconvertible_directive_is_fatal(self) -> None: + index = self.make_tree(entry=ENTRY + '\n.. note::\n\n Not in the subset.\n') + + with self.assertRaises(changelog_md.ResidualMarkupError) as error: + changelog_md.render(index) + + self.assertIn('directive', str(error.exception)) + + def test_an_unconverted_role_is_fatal(self) -> None: + index = self.make_tree(entry=ENTRY + '\nSee :mod:`pcapkit.const` for more.\n') + + with self.assertRaises(changelog_md.ResidualMarkupError) as error: + changelog_md.render(index) + + self.assertIn('role', str(error.exception)) + + def _reject(self, tail: str, expected: str) -> str: + """Assert an entry ending in *tail* is refused, naming *expected*.""" + index = self.make_tree(entry=ENTRY + tail) + + with self.assertRaises(changelog_md.ResidualMarkupError) as error: + changelog_md.render(index) + + message = str(error.exception) + self.assertIn(expected, message) + return message + + def test_an_inline_hyperlink_reference_is_fatal(self) -> None: + # The worst of the five: GFM renders ```text `_`` as broken inline + # code followed by a stray underscore, so the link target vanishes + # silently rather than merely looking wrong. + self._reject('\nSee `the RFC index `_ for more.\n', + 'hyperlink reference') + + def test_an_anonymous_hyperlink_reference_is_fatal(self) -> None: + self._reject('\nSee `the RFC index `__ for more.\n', + 'hyperlink reference') + + def test_a_substitution_reference_is_fatal(self) -> None: + self._reject('\nShipped in |version| of the library.\n', 'substitution') + + def test_a_field_list_is_fatal(self) -> None: + self._reject('\n:Author: Jarry Shaw\n:Version: 9.9.9\n', 'field list') + + def test_a_line_block_is_fatal(self) -> None: + # Rule 6 joins these into ``| One line | Another line``, which GFM can + # read as a table row. + self._reject('\n| One line\n| Another line\n', 'line block') + + def test_a_grid_table_is_fatal(self) -> None: + self._reject( + '\n' + '+----------+----------+\n' + '| Column A | Column B |\n' + '+==========+==========+\n' + '| a | b |\n' + '+----------+----------+\n', + 'grid table', + ) + + def test_a_simple_table_is_fatal(self) -> None: + self._reject( + '\n' + '======== ========\n' + 'Column A Column B\n' + '======== ========\n' + 'a b\n' + '======== ========\n', + 'simple table', + ) + + def test_a_sub_heading_underline_joined_into_the_prose_is_fatal(self) -> None: + # Rule 1 only consumes ``=`` underlines, so a ``-`` underlined sub-heading + # is joined onto the heading text by rule 6 and would otherwise disappear + # into a paragraph. + self._reject('\nA sub heading\n-------------\n\nSome prose.\n', + 'setext underline joined') + + def test_an_over_long_equals_underline_is_fatal(self) -> None: + # Rule 1 fires only when the underline is exactly as long as the title, + # but reStructuredText merely requires it to be no shorter -- so an + # over-long one falls past rule 1 and is joined like any other underline. + self._reject('\nA sub heading\n==================\n\nSome prose.\n', + 'setext underline joined') + + def test_the_generated_files_own_trailer_is_not_a_leftover(self) -> None: + # The trailer's ``---`` sits alone on its line, and the joined-underline + # pattern requires text before the run, so the guard cannot fire on the + # file it is protecting. Checked on the whole rendered output, trailer + # included, rather than on the body render() already checks. + self.assertEqual(changelog_md.residual(changelog_md.render(self.make_tree())), []) + + def test_markup_quoted_inside_a_code_span_is_not_a_leftover(self) -> None: + # The real 1.5.0 entry says ``a Sphinx-only ``:mod:`` role``, which + # converts to the code span ```:mod:```. That is prose about a role, not a + # role that escaped rule 2, and it must not trip the guard. + markdown = self.convert( + ENTRY + '\nPyPI rejected a Sphinx-only ``:mod:`` role in ``README.rst``.\n') + + self.assertEqual(changelog_md.residual(markdown), []) + + def test_the_real_entries_are_all_within_the_subset(self) -> None: + directory = changelog_md.INDEX.parent / 'changelog' + if not directory.is_dir(): + self.skipTest(f'{directory} is absent (docs/ is pruned from a source tarball)') + + for entry in sorted(directory.glob('*.rst')): + with self.subTest(entry=entry.name): + self.assertEqual( + changelog_md.residual(changelog_md.convert( + entry.read_text(encoding='utf-8'))), + [], + ) + + +class CheckModeTests(ChangelogTreeMixin, unittest.TestCase): + """``--check`` is what a CI gate calls, so it has to fail when it should.""" + + def _run(self, *argv: str) -> tuple[int, str, str]: + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + status = changelog_md.main(list(argv)) + return status, out.getvalue(), err.getvalue() + + def test_check_passes_when_the_file_is_in_step(self) -> None: + index = self.make_tree() + output = index.parent / 'CHANGELOG.md' + output.write_text(changelog_md.render(index), encoding='utf-8') + + status, stdout, _ = self._run('--check', '--index', str(index), + '--output', str(output)) + + self.assertEqual(status, 0) + self.assertIn('in step', stdout) + + def test_check_fails_on_a_one_character_edit(self) -> None: + index = self.make_tree() + output = index.parent / 'CHANGELOG.md' + generated = changelog_md.render(index) + output.write_text(generated.replace('ESP parsing', 'ESP parsinG', 1), encoding='utf-8') + + status, _, stderr = self._run('--check', '--index', str(index), + '--output', str(output)) + + self.assertEqual(status, 1) + self.assertIn('drifted', stderr) + self.assertIn('---', stderr) # a unified diff, so the reviewer sees what moved + + def test_check_fails_when_the_file_is_missing(self) -> None: + index = self.make_tree() + + status, _, stderr = self._run('--check', '--index', str(index), + '--output', str(index.parent / 'CHANGELOG.md')) + + self.assertEqual(status, 1) + self.assertIn('drifted', stderr) + + def test_writing_then_checking_round_trips(self) -> None: + index = self.make_tree() + output = index.parent / 'CHANGELOG.md' + + write_status, _, _ = self._run('--index', str(index), '--output', str(output)) + check_status, _, _ = self._run('--check', '--index', str(index), + '--output', str(output)) + + self.assertEqual((write_status, check_status), (0, 0)) + + def test_generated_file_carries_the_trailer(self) -> None: + markdown = changelog_md.render(self.make_tree()) + + self.assertIn(changelog_md.DOCS_URL, markdown) + self.assertTrue(markdown.endswith('\n')) + + +class RepositoryStateTests(unittest.TestCase): + """Checks against the repository rather than a fixture.""" + + def test_docs_url_matches_the_pyproject_changelog_url(self) -> None: + # ``MANIFEST.in`` prunes ``docs/``, so the trailer's link is the only route + # from the shipped changelog to the rest of the history. If the two drift, + # the sdist points at a page that may not exist. + pyproject = (ROOT / 'pyproject.toml').read_text(encoding='utf-8') + declared = re.search(r'(?m)^changelog\s*=\s*"([^"]+)"', pyproject) + + self.assertIsNotNone(declared, 'pyproject.toml declares no [project.urls].changelog') + assert declared is not None + self.assertEqual(changelog_md.DOCS_URL, declared.group(1)) + + def test_committed_changelog_is_in_step_with_the_newest_entry(self) -> None: + if not changelog_md.INDEX.is_file(): + self.skipTest(f'{changelog_md.INDEX} is absent') + if not changelog_md.OUTPUT.is_file(): + self.skipTest(f'{changelog_md.OUTPUT} is absent') + + self.assertEqual( + changelog_md.OUTPUT.read_text(encoding='utf-8'), + changelog_md.render(), + 'CHANGELOG.md is stale; regenerate it with util/changelog_md.py', + ) + + +if __name__ == '__main__': + unittest.main() diff --git a/util/changelog_md.py b/util/changelog_md.py new file mode 100644 index 000000000..dcabc60ed --- /dev/null +++ b/util/changelog_md.py @@ -0,0 +1,480 @@ +# -*- coding: utf-8 -*- +"""Generate the root ``CHANGELOG.md`` from the newest per-version changelog entry. + +``docs/source/changelog/.rst`` is the single source of every changelog +entry, and Sphinx renders the lot as the project's history. ``CHANGELOG.md`` is a +*derivative* of exactly one of them -- the version being released -- because its +two consumers, the ``Create Release`` workflow's release body and the source +distribution, both read Markdown rather than reStructuredText. + +Nothing edits ``CHANGELOG.md`` by hand. That is the point: there is no "promote +the latest entry into the history" step to forget, because the entry was written +in the history in the first place, and the two surfaces cannot drift while one is +generated from the other. It is the same arrangement as ``pcapkit/const/**``, +which is generated from ``pcapkit/vendor/**``. + +Markdown at the repository root is a deliberate exception to the "documentation is +reStructuredText" rule, justified by those two consumers. + +Which entry is "newest" +----------------------- + +**The first entry of the toctree in** ``docs/source/changelog.rst``. + +Not a sort of the version strings. ``1.5.0b3``, ``1.5.0`` and ``1.10.0`` order +correctly under :pep:`440` and incorrectly under every lexical comparison, and +reimplementing :pep:`440` here to answer a question the toctree already answers +would be a second source of truth rather than a convenience. Not the file +modification time either, which says when a file was touched rather than which +release is current. + +The toctree has to exist for Sphinx, and it has to be newest-first for the +rendered history to read correctly, so it is *already* the ordering of record. +Reading it means a release is added in exactly one place -- a new entry file, and +its name at the top of the toctree -- and this script follows. + +The conversion +-------------- + +The entries are written in a deliberately small reStructuredText subset, so the +conversion is six mechanical rules rather than a document converter: + + 1. the setext version heading becomes an ATX ``##`` heading; + 2. ``:rfc:`NNNN``` becomes a Markdown link to the RFC on the IETF datatracker; + 3. ``double backtick`` literals become single-backtick code spans; + 4. ``*`` bullets become ``-`` bullets; + 5. ``[n]_`` / ``.. [n]`` footnotes become GitHub's ``[^n]`` / ``[^n]:``; + 6. each paragraph and each bullet is unwrapped onto one line. + +Everything else -- ``**strong**``, ``*emphasis*``, and the ``#nnn`` issue +references that GitHub autolinks in a release body -- is already valid Markdown +and is copied verbatim. + +Rule 6 is the only one about the *consumer* rather than the syntax. A GitHub +release body is rendered like a comment, where a single newline is a hard line +break rather than a space, so a paragraph wrapped at 79 columns renders as a +column of short ragged lines. A paragraph with no newline inside it renders the +same either way, which is why unwrapping is the safe direction. + +Six regexes cannot recognise everything reStructuredText can express, and a rule +that does not fire copies its construct straight through as literal text. That is +how a release body ends up with ``.. note::`` printed in it -- published to GitHub +and PyPI, where it is expensive and irreversible. So :func:`residual` re-reads the +converted entry and :func:`render` refuses to emit while anything is left. + +What it guards, exactly -- the claim is kept level with the code, because a guard +believed to be broader than it is, is worse than no guard: + + * a ``double backtick`` literal rule 3 did not reach; + * an interpreted-text role, ``:mod:`x```, rule 2 does not know; + * a directive or comment, any line opening ``..`` -- which also catches the + ``.. _name: url`` target that a bare ``name_`` reference needs, so that form is + covered through its definition rather than through the reference; + * a ``*`` bullet rule 4 missed; + * a ``[n]_`` footnote reference rule 5 missed; + * an inline or phrase hyperlink reference, ```text `_``, which GFM renders + as broken inline code plus a stray underscore, losing the link outright; + * a substitution reference, ``|version|``, which has no Markdown equivalent; + * a field list, ``:Author: ...``, which rule 6 would join onto one line; + * a line block, ``| line``, which rule 6 joins into something GFM may read as a + table row; + * a grid or simple table, which rule 6 mangles onto one line; + * a setext underline, whether left on its own line or joined into the prose + above it by rule 6 -- the second is how a sub-heading would silently + disappear into a paragraph. + +Two things it deliberately does **not** catch. A bare ``name_`` reference on its +own, because a pattern loose enough to spot a trailing underscore in prose fires on +ordinary identifiers; its target line is caught instead, as above. And a construct +that rule 6 joins into the *middle* of a line rather than the start of one, since +every line-anchored pattern here then has nothing to anchor to -- a field list +buried inside a bullet, say. Both would need the guard to run before unwrapping, +which is a larger change than the insurance is worth. + +Every pattern was measured against all 37 committed entries and matches none of +them, so the guard costs nothing until an entry actually leaves the subset. + +Usage +----- + +.. code-block:: shell + + python util/changelog_md.py # regenerate CHANGELOG.md + python util/changelog_md.py --check # exit non-zero if it has drifted + +``--check`` regenerates into memory, compares, and prints a unified diff of what +moved. It is what a CI gate calls, and it is the same shape as the byte- +reproduction gate over the generated ``pcapkit.const`` modules. + +""" + +from __future__ import annotations + +import argparse +import difflib +import pathlib +import re +import sys +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from typing import Optional, Sequence + +#: Repository root, taken from this file's location rather than the working +#: directory, so the script gives the same answer run from anywhere. +ROOT = pathlib.Path(__file__).resolve().parent.parent + +#: The page carrying the changelog title, preamble and toctree. The toctree in it +#: is the ordering of record; see the module docstring. +INDEX = ROOT / 'docs' / 'source' / 'changelog.rst' + +#: The generated artefact. +OUTPUT = ROOT / 'CHANGELOG.md' + +#: Base of the RFC links rule 2 emits. This is the form Sphinx's own ``:rfc:`` +#: role resolves to, so the Markdown and the rendered documentation point at the +#: same page; docutils on its own would resolve the role to a ``tools.ietf.org`` +#: URL instead. +RFC_URL = 'https://datatracker.ietf.org/doc/html/rfc' + +#: Where the full rendered history lives. Must match ``[project.urls].changelog`` +#: in ``pyproject.toml`` -- ``MANIFEST.in`` prunes ``docs/``, so in a source +#: distribution this is the only route from the shipped entry to the rest of the +#: history. ``tests/project/test_changelog_md.py`` asserts the two agree. +DOCS_URL = 'https://jarryshaw.github.io/PyPCAPKit/changelog.html' + +#: Closing note on the generated file, explaining why it holds one release. +TRAILER = ( + '---\n' + '\n' + 'Only the version being released is kept here. Every entry, including this' + ' one, lives in the repository as `docs/source/changelog/.rst`,' + f' and the whole history is rendered at <{DOCS_URL}>.\n' +) + +#: A single-backtick code span. Replaced by :data:`_SPAN` before :func:`residual` +#: looks for unconverted markup, because an entry may legitimately *discuss* +#: markup: the 1.5.0 entry says ``a Sphinx-only ``:mod:`` role``, which converts +#: to the code span ```:mod:``` and must not be read as a role that escaped rule +#: 2. Deliberately cannot match across a newline, so masking leaves every line +#: break -- and therefore every reported line number -- where it was. +_CODE_SPAN = re.compile(r'`[^`\n]*`') + +#: Stands in for a masked code span. A *marker* rather than nothing, because the +#: difference between a leftover role and prose about a role is precisely whether +#: a code span follows the ``:name:`` or encloses it: ``:mod:`pcapkit.const``` is +#: a role that escaped rule 2, and ```:mod:``` is a sentence mentioning one. +_SPAN = '\x00' + +#: reStructuredText that should be gone by the time conversion finishes. Checked +#: against the masked text, and each one means a rule did not fire. Every pattern +#: matches none of the 37 committed entries; see the module docstring for what is +#: covered and what is deliberately not. +_RESIDUAL = ( + (re.compile(r':[a-zA-Z][a-zA-Z0-9_+:.-]*:' + _SPAN), + 'an unconverted interpreted-text role'), + (re.compile(r'(?m)^[ \t]*\.\.[ \t]'), 'an unconverted directive or comment'), + (re.compile(r'(?m)^[ \t]*\*[ \t]'), 'an unconverted bullet'), + (re.compile(r'\[\d+\]_'), 'an unconverted footnote reference'), + # ``\x00_`` is a masked code span followed by an underscore, which is what + # ```text `_`` and its anonymous ```text `__ form both reduce to. + (re.compile(_SPAN + '_'), + 'an unconverted hyperlink reference (the link target would be lost)'), + (re.compile(r'\|\S[^|\n]*\|'), 'an unconverted substitution reference'), + (re.compile(r'(?m)^[ \t]*:[^:\n]+:[ \t]'), 'an unconverted field list'), + (re.compile(r'(?m)^[ \t]*\|[ \t]'), 'an unconverted line block'), + (re.compile(r'(?m)^[ \t]*\+[-=]+\+'), 'an unconverted grid table'), + (re.compile(r'(?m)^[ \t]*=+[ \t]+=+'), 'an unconverted simple table'), + (re.compile(r'(?m)^=+$'), 'a leftover setext underline'), + (re.compile(r'(?m)^~+$'), 'a leftover setext underline'), + # A sub-heading's underline does not survive as its own line: rule 6 joins it + # onto the end of the heading text, so it has to be caught mid-line too. The + # leading ``\S[ \t]+`` is what makes this specific to the joined case, so it + # cannot fire on a ``---`` that is alone on its line -- which is what the + # generated file's own trailer is made of. + # ``(?m)`` matters: the run is followed by a newline rather than end of input, + # and a bare ``$`` only matches at the very end of the string. + # + # ``=`` belongs in the alternation even though rule 1 handles ``=`` underlines, + # because rule 1 requires the underline to be exactly as long as the title + # while reStructuredText merely requires it to be no shorter. An over-long + # underline therefore falls straight past rule 1 and gets joined like any other. + (re.compile(r'(?m)\S[ \t]+(?:={3,}|-{3,}|~{3,}|\^{3,}|"{3,})(?:[ \t]|$)'), + 'a setext underline joined into the prose above it'), +) + + +class ResidualMarkupError(RuntimeError): + """Conversion finished with reStructuredText still in the output. + + Deliberately fatal. The six rules cover the subset the entries are written + in; anything outside it would otherwise be copied through as literal text and + render as itself in a release body, which is a silent defect in a published + artefact. Failing here instead names the construct and the line. + + """ + + +def read_toctree(index: pathlib.Path) -> list[str]: + """Return the document names in *index*'s first ``toctree``, in order. + + Args: + index: The page carrying the toctree. + + Returns: + The toctree entries, newest first, each as written -- so relative to + *index*'s own directory -- with any ``Title `` wrapper reduced + to the document name. + + Raises: + ValueError: If *index* carries no ``toctree``, or it lists no entries. + + """ + lines = index.read_text(encoding='utf-8').split('\n') + + for start, line in enumerate(lines): + if line.strip() == '.. toctree::': + indent = len(line) - len(line.lstrip()) + break + else: + raise ValueError(f'no toctree directive in {index}') + + entries = [] # type: list[str] + for line in lines[start + 1:]: + if not line.strip(): + continue # blank lines are allowed inside a directive body + if len(line) - len(line.lstrip()) <= indent: + break # a dedent ends the directive + body = line.strip() + if body.startswith(':'): + continue # a directive option, such as :maxdepth: + explicit = re.search(r'<([^<>]+)>\Z', body) + entries.append(explicit.group(1) if explicit else body) + + if not entries: + raise ValueError(f'the toctree in {index} lists no entries') + return entries + + +def newest(index: pathlib.Path = INDEX) -> tuple[str, pathlib.Path]: + """Resolve the newest changelog entry from *index*'s toctree. + + Args: + index: The page carrying the toctree. + + Returns: + The newest entry's version string and the path to its file. + + Raises: + FileNotFoundError: If the toctree's first entry names a document that + does not exist, which is what half of a release addition looks like. + + """ + first = read_toctree(index)[0] + # ``+ '.rst'`` rather than :meth:`~pathlib.Path.with_suffix`: a version string + # ends in what looks like a file extension, so ``with_suffix`` on + # ``changelog/1.5.0`` yields ``changelog/1.5.rst``. + entry = index.parent / (first + '.rst') + if not entry.is_file(): + raise FileNotFoundError( + f'{index} lists {first!r} as the newest entry, but {entry} does not exist' + ) + return entry.stem, entry + + +def convert(rst: str) -> str: + """Apply the six rules to one per-version entry. + + Args: + rst: The entry's reStructuredText. + + Returns: + The entry as Markdown, ending in a single newline. + + """ + lines = rst.rstrip('\n').split('\n') + out = [] # type: list[str] + + index = 0 + while index < len(lines): + line = lines[index] + + # 1. setext heading -> ATX. Only ``=`` is used in these files, and only + # for the version heading, so the underline can be consumed outright. + if (index + 1 < len(lines) and line and set(lines[index + 1]) == {'='} + and len(lines[index + 1]) == len(line)): + out.append(f'## {line}') + index += 2 + continue + + # 2. the one role these entries use. + line = re.sub(r':rfc:`(\d+)`', lambda match: f'[RFC {match[1]}]({RFC_URL}{match[1]})', line) + # 3. literals. + line = re.sub(r'``([^`]+)``', r'`\1`', line) + # 4. bullets, at any indent. + line = re.sub(r'^(\s*)\* ', r'\1- ', line) + # 5. footnotes. + line = re.sub(r'^\.\. \[(\d+)\] ', r'[^\1]: ', line) + line = re.sub(r'\[(\d+)\]_', r'[^\1]', line) + + out.append(line) + index += 1 + + return unwrap(out).rstrip('\n') + '\n' + + +def unwrap(lines: Sequence[str]) -> str: + """Rule 6: collapse each paragraph and each bullet onto a single line. + + A block ends at a blank line, at the next bullet, at a heading, or at a + footnote definition. Nothing in these entries is indentation-sensitive -- + no literal blocks, no tables, no definition lists -- so joining a block's + lines with a single space is lossless. + + Args: + lines: The entry's lines, with rules 1 to 5 already applied. + + Returns: + The lines with each block joined onto one line. + + """ + blocks = [] # type: list[str] + current = [] # type: list[str] + + def flush() -> None: + if current: + blocks.append(' '.join(item.strip() for item in current)) + current.clear() + + for line in lines: + if not line.strip(): + flush() + blocks.append('') + continue + if re.match(r'^\s*- ', line) or line.startswith('## ') \ + or re.match(r'^\[\^\d+\]: ', line): + flush() + current.append(line) + continue + current.append(line) + flush() + + # Collapse the runs of blank lines the flush cycle can leave behind. + return re.sub(r'\n{3,}', '\n\n', '\n'.join(blocks)) + + +def residual(markdown: str) -> list[str]: + """Report reStructuredText left in *markdown* that the six rules did not convert. + + See the module docstring for the guarded set, and for the two constructs left + deliberately unguarded. + + Args: + markdown: One converted entry, as :func:`convert` returned it. The + generated file's trailer is not part of this and does not need to be: + :func:`render` checks the body before appending it. + + Returns: + One human-readable complaint per leftover construct, empty when clean. + + """ + problems = [] # type: list[str] + + # Checked before code spans are masked: masking would eat ``````` as an empty + # code span and hide exactly the case this looks for. + for number, line in enumerate(markdown.split('\n'), 1): + if '``' in line: + problems.append(f'line {number}: an unconverted `` literal: {line.strip()[:70]!r}') + + masked = _CODE_SPAN.sub(_SPAN, markdown) + for pattern, label in _RESIDUAL: + for match in pattern.finditer(masked): + number = masked.count('\n', 0, match.start()) + 1 + found = match.group(0).replace(_SPAN, '`...`') + problems.append(f'line {number}: {label}: {found!r}') + return problems + + +def render(index: pathlib.Path = INDEX) -> str: + """Render the complete ``CHANGELOG.md``. + + Args: + index: The page carrying the toctree that decides which entry is newest. + + Returns: + The file's content, ready to be written. + + Raises: + ResidualMarkupError: If the entry uses reStructuredText the rules do not + cover, so the output would carry markup through as literal text. + + """ + _, entry = newest(index) + body = convert(entry.read_text(encoding='utf-8')) + + problems = residual(body) + if problems: + raise ResidualMarkupError( + f'{entry} uses reStructuredText the six conversion rules do not cover, ' + f'so CHANGELOG.md would carry it through as literal text:\n ' + + '\n '.join(problems) + + '\nEither rewrite the entry in the supported subset, or teach ' + 'util/changelog_md.py the construct.' + ) + + return body + '\n' + TRAILER + + +def main(argv: Optional[Sequence[str]] = None) -> int: + """Command line entry point. + + Args: + argv: Argument list, defaulting to :data:`sys.argv`. + + Returns: + ``0`` on success; ``1`` if ``--check`` found the output stale or missing. + + """ + parser = argparse.ArgumentParser( + prog='changelog_md.py', + description='Generate CHANGELOG.md from the newest per-version changelog entry.', + ) + parser.add_argument( + '--check', action='store_true', + help='write nothing; exit non-zero if the committed file has drifted', + ) + parser.add_argument( + '--index', type=pathlib.Path, default=INDEX, + help='page carrying the toctree that orders the entries (default: %(default)s)', + ) + parser.add_argument( + '--output', type=pathlib.Path, default=OUTPUT, + help='file to generate (default: %(default)s)', + ) + args = parser.parse_args(argv) + + version, entry = newest(args.index) + want = render(args.index) + + if not args.check: + args.output.write_text(want, encoding='utf-8') + print(f'wrote {args.output} from {entry} ({len(want.splitlines())} lines)') + return 0 + + have = args.output.read_text(encoding='utf-8') if args.output.is_file() else '' + if have == want: + print(f'{args.output} is in step with {entry}') + return 0 + + print(f'{args.output} has drifted from {entry}', file=sys.stderr) + sys.stderr.writelines(difflib.unified_diff( + have.splitlines(keepends=True), + want.splitlines(keepends=True), + fromfile=f'{args.output.name} (committed)', + tofile=f'{args.output.name} (regenerated from {version})', + )) + return 1 + + +if __name__ == '__main__': + sys.exit(main()) From 807e376077f90a0cc9224c69290b5ca20c17c56b Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Sat, 19 Sep 2026 19:55:59 -0400 Subject: [PATCH 3/4] docs: shorten the generated changelog trailer to just the history link The trailer explained which `docs/source/changelog/.rst` the entry came from. CHANGELOG.md is read on a GitHub release page and on PyPI, where the reader has no repository in front of them, so that half was noise to its only two audiences while the link is useful to both. Regenerated accordingly; `--check` confirms the committed file is in step. --- CHANGELOG.md | 2 +- util/changelog_md.py | 11 +++++++---- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e1a9d3210..946afc9a7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,4 +37,4 @@ Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) an --- -Only the version being released is kept here. Every entry, including this one, lives in the repository as `docs/source/changelog/.rst`, and the whole history is rendered at . +Full changelog: diff --git a/util/changelog_md.py b/util/changelog_md.py index dcabc60ed..31068f542 100644 --- a/util/changelog_md.py +++ b/util/changelog_md.py @@ -143,13 +143,16 @@ #: history. ``tests/project/test_changelog_md.py`` asserts the two agree. DOCS_URL = 'https://jarryshaw.github.io/PyPCAPKit/changelog.html' -#: Closing note on the generated file, explaining why it holds one release. +#: Closing note on the generated file, pointing at the rest of the history. +#: +#: Deliberately short. This file is read on a GitHub release page and on PyPI, +#: where a reader has no repository in front of them -- so an explanation of +#: which ``docs/source/changelog/.rst`` the entry came from is noise to +#: them, while the link is the one thing that is useful everywhere. TRAILER = ( '---\n' '\n' - 'Only the version being released is kept here. Every entry, including this' - ' one, lives in the repository as `docs/source/changelog/.rst`,' - f' and the whole history is rendered at <{DOCS_URL}>.\n' + f'Full changelog: <{DOCS_URL}>\n' ) #: A single-backtick code span. Replaced by :data:`_SPAN` before :func:`residual` From 99f02ebfe19ea61ab1e603eeee13203e3908e377 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Sat, 19 Sep 2026 20:14:57 -0400 Subject: [PATCH 4/4] ci: feed the generated CHANGELOG.md to the release body, and gate its drift - The release step set `generate_release_notes: true` with no body, so the hand-written changelog reached no reader of a GitHub Release. Add `body_path`; the action joins body-then-generated, so the entry leads and the automatic pull-request list follows it. - `CHANGELOG.md` carries the newest toctree entry, which is the version being prepared rather than released: `bump_version.py` produces `1.5.0b4` and `.postN` without adding an entry, so an unconditional body would republish a base version's notes under a vendor refresh that changed no library code. The body is used only when the file's heading names the version being released, and both outcomes are announced in the log. - `util/changelog_md.py --check` was wired into nothing, so drift in a published file was caught only by eye. Add a `changelog` job to `unit-tests.yml`, deliberately ungated by `gate-only` so it also guards the release path that reads the file. Verified: actionlint and shellcheck report the same 15 findings as the base; `--check` exits 0 in step and 1 both on a one-character edit and on a missing file; the selector was exercised against final, pre-release, post-release, dated and absent inputs. Behaviour on a real release is unverified until one runs. --- .github/workflows/create-release.yml | 55 ++++++++++++++++++++++++++++ .github/workflows/unit-tests.yml | 41 +++++++++++++++++++++ 2 files changed, 96 insertions(+) diff --git a/.github/workflows/create-release.yml b/.github/workflows/create-release.yml index 0414f60a1..39e4e97b1 100644 --- a/.github/workflows/create-release.yml +++ b/.github/workflows/create-release.yml @@ -97,6 +97,55 @@ jobs: # ref: v${{ needs.version_check.outputs.PCAPKIT_VERSION }} fetch-depth: 0 + # ``CHANGELOG.md`` is generated by ``util/changelog_md.py`` from the newest + # entry in the ``docs/source/changelog.rst`` toctree, so it describes the + # version being *prepared* -- which is not always the version being + # *released*. The two agree on a normal release and disagree on every + # automated one, because ``util/bump_version.py`` bumps without ever adding + # an entry: ``1.5.0b3`` becomes ``1.5.0b4``, and a final release becomes + # ``.post1``. A ``.postN`` is the weekly vendor registry refresh, carries no + # library change, and is deliberately recorded as an "Also released" line + # inside its base version's entry rather than as an entry of its own. + # + # Handing the file over unconditionally would therefore republish the base + # version's notes as the body of a release that changed no library code. So + # the body is used only when the generated file's own heading names the + # version being released, and otherwise the release keeps the generated notes + # alone -- which for a vendor refresh is the honest summary, since the commit + # list is the only thing that did change. Both outcomes are announced in the + # log rather than left to be inferred from the published result. + # + # Only the version token is compared, not the rest of the heading, so an + # entry still reading "-- unreleased" is matched and dated-or-not does not + # change the decision. That case gets its own warning below instead. + - name: Select release body + id: release_body + shell: bash + env: + PCAPKIT_VERSION: ${{ needs.version_check.outputs.PCAPKIT_VERSION }} + run: | + set -euo pipefail + + if [ ! -f CHANGELOG.md ]; then + echo "::warning title=Release body generated::CHANGELOG.md is absent, so this release carries the automatically generated notes alone. The file is generated by util/changelog_md.py and the 'Changelog drift' job gates it on the release path, so reaching this branch means that gate did not run." + exit 0 + fi + + # The generated file opens with ``## -- ``. + heading="$(sed -n '1s/^## \([^[:space:]]*\).*/\1/p' CHANGELOG.md)" + + if [ "$heading" != "$PCAPKIT_VERSION" ]; then + echo "::notice title=Release body generated::CHANGELOG.md holds the entry for '${heading:-}', not ${PCAPKIT_VERSION}, so this release carries the automatically generated notes alone. Expected for a pre-release and for a .postN vendor refresh, neither of which has a changelog entry of its own." + exit 0 + fi + + echo 'path=CHANGELOG.md' >> "$GITHUB_OUTPUT" + echo "::notice title=Release body from CHANGELOG.md::The changelog entry for ${PCAPKIT_VERSION} leads the release body; the generated pull-request list follows it." + + if sed -n '1p' CHANGELOG.md | grep -qi 'unreleased'; then + echo "::warning title=Changelog entry is undated::The entry for ${PCAPKIT_VERSION} still reads 'unreleased', so the release body says so too. Date the heading in docs/source/changelog/${PCAPKIT_VERSION}.rst, regenerate CHANGELOG.md and the body can be corrected in place." + fi + - name: Create Release id: create_release uses: softprops/action-gh-release@v3.0.2 @@ -106,6 +155,12 @@ jobs: # allowUpdates: true # artifacts: | # dist/* + # An empty ``body_path`` is skipped by the action's own truthy guard, so + # an unset output here means "no body" rather than "read the file ''". + # ``generate_release_notes`` stays on either way: the action fetches the + # generated notes itself and joins them as ``\n\n``, so + # the hand-written entry leads and the pull-request list follows. + body_path: ${{ steps.release_body.outputs.path }} generate_release_notes: true # makeLatest: true name: PyPCAPKit v${{ needs.version_check.outputs.PCAPKIT_VERSION }} diff --git a/.github/workflows/unit-tests.yml b/.github/workflows/unit-tests.yml index 124278bd0..fac6e5f99 100644 --- a/.github/workflows/unit-tests.yml +++ b/.github/workflows/unit-tests.yml @@ -155,6 +155,47 @@ jobs: - name: Run full test suite run: python -m pytest -q + # ``CHANGELOG.md`` is generated from the newest entry under + # ``docs/source/changelog/`` by ``util/changelog_md.py``, so it falls out of step + # the moment an entry is edited without regenerating it. That is worth its own + # job rather than leaving it to review, because ``Create Release`` feeds the file + # to the GitHub Release body: a drifted copy is not merely wrong in the tree, it + # is published. + # + # Deliberately *not* gated on ``gate-only``, unlike the three jobs above, and + # that is the point of putting it here at all. ``create-release.yml`` calls this + # workflow as its release gate, so an ungated job runs on the release path and + # the release body cannot be built from a file that has drifted. The matrix is + # skipped per caller because it is expensive and already ran for the commit; + # this is neither expensive nor covered elsewhere on that path. + # + # ``tests/project/test_changelog_md.py`` asserts the same identity, so the + # matrix covers it too -- but that test ``skipTest``s when either file is + # absent, and reports a mismatch as two 17 KB strings. ``--check`` cannot skip, + # exits 1 when the file is missing, and prints a unified diff. + changelog: + name: Changelog drift + runs-on: ubuntu-latest + timeout-minutes: 5 + + steps: + - uses: actions/checkout@v7 + + - uses: actions/setup-python@v7 + with: + python-version: "3.14" + + # No install step: the generator imports argparse, difflib, pathlib, re, sys + # and typing and nothing else, so it runs against a bare interpreter. + # Verified with ``python -S``. + - name: Check CHANGELOG.md is in step with its source entry + shell: bash + run: | + if ! python util/changelog_md.py --check; then + echo "::error title=CHANGELOG.md has drifted::CHANGELOG.md is generated from the newest entry under docs/source/changelog/ and no longer matches it. Regenerate it with 'python util/changelog_md.py' and commit the result. The unified diff above shows what moved." + exit 1 + fi + # The verdict a shipping workflow asks for: the full suite, fixtures and # all, on one interpreter. It is deliberately not the matrix -- the matrix # has already run against this very commit, from this workflow's own push or