From da697fa930e71a2379ff9877eaddb30a32785a29 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 12:10:47 -0400 Subject: [PATCH 01/46] docs(changelog): the 1.5.0 entry for #616, whose code ships in #634 The bullet #634 originally carried, moved here verbatim so that #634 touches only `pcapkit/protocols/transport/tcp.py` and its two test files. Covers: `TCP.read` seeding its connection-flag accumulator with a `typing.cast` no-op rather than `Flags(0)`, so a flagless segment left `self._flags` a plain `int`. 35 lines added to the entry file; `CHANGELOG.md` regenerated, not edited. --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 35 +++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 838c2e164..bcee8c360 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -86,6 +86,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- `CITATION.cff` shipped in no source distribution. `MANIFEST.in` carries an `include` line each for `README.md`, `LICENSE` and `CHANGELOG.md` but had none for the citation file, and its two `global-include` patterns are `*.rst` and `*.py`, neither of which can match a `.cff` -- so the file sat in the repository and in nothing published. Nor was there a default to fall back on. Removing all three of those `include` lines and rebuilding shows `README.md` and `LICENSE` shipping anyway, setuptools adding the latter from `license_files` and recording it as `License-File: LICENSE` in `PKG-INFO`, while `CHANGELOG.md` vanishes -- so of the three only `CHANGELOG.md` is load-bearing, and a citation file, which no packaging default covers at all, is in the same position. The gap is invisible from the web UI, because GitHub renders the "Cite this repository" button from the repository itself; citation managers, Zenodo and dependency inventories read the published artifact, which is exactly the surface that was missing it, so the machine-readable half of the attribution stopped travelling with the code at the one point nobody can see the repository. #615 flagged the omission when it added the file and #619 did not address it. It matters now because #625 has just taught `util/bump_version.py` to keep the file's `version` and `date-released` in step with the bump, and a release exercising that path would otherwise publish an sdist omitting the very artefact under test. It also lets `RepositoryCitationTests` in `tests/project/test_bump_version.py` -- the gate #625 added for the hand-authored bumps that never run the script -- execute against an unpacked sdist, where today it skips itself with "CITATION.cff is not shipped in the source distribution". Measured both ways with `python -m build --sdist`: `tar tzf` found no `CITATION.cff` before and `pypcapkit-1.5.0b4/CITATION.cff` after, the two archive listings differ by that one added entry and nothing else -- 860 against 861 -- the shipped copy is byte-identical to the repository's, and `twine check --strict` reports `PASSED` on both archives (#631). - **Fixed** -- two inaccurate claims in the #630 entry above, caught by review after #630 had already merged. Both were about the history of `LICENSE` rather than about the change itself, and `LICENSE` is untouched here: 2017 was and remains the right answer, so this is a changelog-prose correction only. The entry asserted that the file "until then was stock MPL text carrying no author notice at all, so 2018 is the first start year the project ever asserted and it was already a year late when it was written". Every clause of it is wrong as a description of the era, though each was true of the single blob `bc836cfa2` happened to replace, which is where the mistake came from. The pre-BSD file was MIT, then GPL v3, then Apache 2.0, and only then MPL 2.0 -- four licences before BSD, not the one implied. The MIT and GPL texts both carried an author notice naming **2017**, the MIT one on line 3 of the initial commit. And `2018` first appeared in the Apache notice of 2018-12-08 as a single year, current for the year it was written rather than late. The claim also contradicted the entry's own opening sentence, which dates the first commit to 2017-11-07. What the history actually supports is a stronger argument for the change than the one that was written, which is why this is corrected rather than deleted: 2017 is the year the project asserted from its first commit, and #630 restores it rather than asserting it anew. The `BSD 3-Clause License` title line is reattributed too, from the opensource.org template -- whose licence text begins at the copyright line -- to choosealicense.com's, whose first line is literally that title, and which also carries the `Copyright (c) [year], [fullname]` comma this file uses. The error came from generalising the pre-BSD era off two blobs that turned out to be the same file, `bc836cfa2^:LICENSE` and `1c69341dc:LICENSE`; enumerating all 14 commits that ever touched `LICENSE` is what caught it (#638). +- **Fixed** -- `TCP.read` seeded its connection-flag accumulator with `cast('Enum_Flags', 0)`. `typing.cast` is a runtime no-op -- it returns its second argument unchanged -- so the accumulator began life as the plain `int` `0`, and the `|=` that promotes it to a `pcapkit.const.tcp.flags.Flags` member is the only thing that ever did. A segment whose flags octet is all zero, an nmap NULL scan among them, promoted nothing and left `self._flags` an `int`, so a membership test against it raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering, and `TCP.connection` returned an `int` where both that property and `Data_TCP.connection` annotate `Flags`. Any flag at all masked it, which is how it survived a module at 100% statement and branch coverage. The seed is `Flags(0)` now, which is what #597 had already done to the sibling accumulator in `make` for the same reason; `Flags` declares no `_missing_` of its own, so `Flags(0)` is an ordinary `aenum.IntFlag` pseudo-member and does not meet the `RecursionError` that the same construction hits on the flag enumerations under `pcapkit.const.mh` (#623). Nothing a caller can reach changed: the three MP_JOIN layouts of [RFC 8684](https://datatracker.ietf.org/doc/html/rfc8684) section 3.2 are chosen by `_read_mptcp_join` from these very membership tests, but `mptcp_data_selector` rejects a flagless MP_JOIN with a `FieldError` before the dispatcher runs -- measured identical either side of the fix -- so the `TypeError` was latent rather than live, and latent only by virtue of a guard in another file. Called directly on a flagless parsed segment the dispatcher now raises the library's own `ProtocolError` naming an invalid flags combination instead. One difference *is* observable, and it is a dump-format change rather than a value change: a flagless segment's `connection` renders as the string `Flags::None [0]` where it rendered as the number `0`, in all three of the JSON, tree and PLIST outputs. The value is numerically the same and the wire bytes are untouched; what changed is that `connection` no longer switches JSON type with the flags, having been a number for a flagless segment and a string -- `Flags::ACK [2048]` -- for every other one. The literal `None` inside it is a separate rendering defect: the hook `pcapkit.dumpkit.common.make_dumper` installs interpolates `o.name` without accounting for a nameless composite member, so it would spell any zero-valued flag enumeration in the library the same way. That is left to its own change and pinned by a test here so it cannot drift unnoticed. The committed example dumps are unaffected, `examples/captures/in.pcap` carrying no flagless segment. Coverage cannot see the fix itself, because the changed line already executed and `tcp.py` reads 100% statement and branch either side; the added subtests are the evidence (#616). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 7486ebde1..a69df155c 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1263,6 +1263,41 @@ pull requests between #326 and #509. file uses. The error came from generalising the pre-BSD era off two blobs that turned out to be the same file, ``bc836cfa2^:LICENSE`` and ``1c69341dc:LICENSE``; enumerating all 14 commits that ever touched ``LICENSE`` is what caught it (#638). +* **Fixed** -- ``TCP.read`` seeded its connection-flag accumulator with + ``cast('Enum_Flags', 0)``. ``typing.cast`` is a runtime no-op -- it returns its second + argument unchanged -- so the accumulator began life as the plain ``int`` ``0``, and the + ``|=`` that promotes it to a ``pcapkit.const.tcp.flags.Flags`` member is the only thing + that ever did. A segment whose flags octet is all zero, an nmap NULL scan among them, + promoted nothing and left ``self._flags`` an ``int``, so a membership test against it + raised ``TypeError: argument of type 'int' is not a container or iterable`` rather than + answering, and ``TCP.connection`` returned an ``int`` where both that property and + ``Data_TCP.connection`` annotate ``Flags``. Any flag at all masked it, which is how it + survived a module at 100% statement and branch coverage. The seed is ``Flags(0)`` now, + which is what #597 had already done to the sibling accumulator in ``make`` for the same + reason; ``Flags`` declares no ``_missing_`` of its own, so ``Flags(0)`` is an ordinary + ``aenum.IntFlag`` pseudo-member and does not meet the ``RecursionError`` that the same + construction hits on the flag enumerations under ``pcapkit.const.mh`` (#623). Nothing a + caller can reach changed: the three MP_JOIN layouts of :rfc:`8684` section 3.2 are + chosen by ``_read_mptcp_join`` from these very membership tests, but + ``mptcp_data_selector`` rejects a flagless MP_JOIN with a ``FieldError`` before the + dispatcher runs -- measured identical either side of the fix -- so the ``TypeError`` was + latent rather than live, and latent only by virtue of a guard in another file. Called + directly on a flagless parsed segment the dispatcher now raises the library's own + ``ProtocolError`` naming an invalid flags combination instead. One difference *is* + observable, and it is a dump-format change rather than a value change: a flagless + segment's ``connection`` renders as the string ``Flags::None [0]`` where it rendered as + the number ``0``, in all three of the JSON, tree and PLIST outputs. The value is + numerically the same and the wire bytes are untouched; what changed is that + ``connection`` no longer switches JSON type with the flags, having been a number for a + flagless segment and a string -- ``Flags::ACK [2048]`` -- for every other one. The + literal ``None`` inside it is a separate rendering defect: the hook + ``pcapkit.dumpkit.common.make_dumper`` installs interpolates ``o.name`` without + accounting for a nameless composite member, so it would spell any zero-valued flag + enumeration in the library the same way. That is left to its own change and pinned by a + test here so it cannot drift unnoticed. The committed example dumps are unaffected, + ``examples/captures/in.pcap`` carrying no flagless segment. Coverage cannot see the fix + itself, because the changed line already executed and ``tcp.py`` reads 100% statement + and branch either side; the added subtests are the evidence (#616). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 784228b5c012d4b29d6f749c0252bc87696c2ee1 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 12:11:01 -0400 Subject: [PATCH 02/46] docs(changelog): the 1.5.0 entry for #618, whose code ships in #635 The bullet #635 originally carried, moved here verbatim so that #635 touches only the two `Frame` modules, `pcapkit/toolkit/pcapng.py` and its two test files. Covers: the breaking change to a public attribute -- `Frame.len` is the on-wire length and `cap_len` the captured one, which the PCAP and PCAP-NG readers had filled from opposite wire fields. 41 lines added to the entry file; `CHANGELOG.md` regenerated, not edited. --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 41 +++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index bcee8c360..57e0064c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -87,6 +87,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- two inaccurate claims in the #630 entry above, caught by review after #630 had already merged. Both were about the history of `LICENSE` rather than about the change itself, and `LICENSE` is untouched here: 2017 was and remains the right answer, so this is a changelog-prose correction only. The entry asserted that the file "until then was stock MPL text carrying no author notice at all, so 2018 is the first start year the project ever asserted and it was already a year late when it was written". Every clause of it is wrong as a description of the era, though each was true of the single blob `bc836cfa2` happened to replace, which is where the mistake came from. The pre-BSD file was MIT, then GPL v3, then Apache 2.0, and only then MPL 2.0 -- four licences before BSD, not the one implied. The MIT and GPL texts both carried an author notice naming **2017**, the MIT one on line 3 of the initial commit. And `2018` first appeared in the Apache notice of 2018-12-08 as a single year, current for the year it was written rather than late. The claim also contradicted the entry's own opening sentence, which dates the first commit to 2017-11-07. What the history actually supports is a stronger argument for the change than the one that was written, which is why this is corrected rather than deleted: 2017 is the year the project asserted from its first commit, and #630 restores it rather than asserting it anew. The `BSD 3-Clause License` title line is reattributed too, from the opensource.org template -- whose licence text begins at the copyright line -- to choosealicense.com's, whose first line is literally that title, and which also carries the `Copyright (c) [year], [fullname]` comma this file uses. The error came from generalising the pre-BSD era off two blobs that turned out to be the same file, `bc836cfa2^:LICENSE` and `1c69341dc:LICENSE`; enumerating all 14 commits that ever touched `LICENSE` is what caught it (#638). - **Fixed** -- `TCP.read` seeded its connection-flag accumulator with `cast('Enum_Flags', 0)`. `typing.cast` is a runtime no-op -- it returns its second argument unchanged -- so the accumulator began life as the plain `int` `0`, and the `|=` that promotes it to a `pcapkit.const.tcp.flags.Flags` member is the only thing that ever did. A segment whose flags octet is all zero, an nmap NULL scan among them, promoted nothing and left `self._flags` an `int`, so a membership test against it raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering, and `TCP.connection` returned an `int` where both that property and `Data_TCP.connection` annotate `Flags`. Any flag at all masked it, which is how it survived a module at 100% statement and branch coverage. The seed is `Flags(0)` now, which is what #597 had already done to the sibling accumulator in `make` for the same reason; `Flags` declares no `_missing_` of its own, so `Flags(0)` is an ordinary `aenum.IntFlag` pseudo-member and does not meet the `RecursionError` that the same construction hits on the flag enumerations under `pcapkit.const.mh` (#623). Nothing a caller can reach changed: the three MP_JOIN layouts of [RFC 8684](https://datatracker.ietf.org/doc/html/rfc8684) section 3.2 are chosen by `_read_mptcp_join` from these very membership tests, but `mptcp_data_selector` rejects a flagless MP_JOIN with a `FieldError` before the dispatcher runs -- measured identical either side of the fix -- so the `TypeError` was latent rather than live, and latent only by virtue of a guard in another file. Called directly on a flagless parsed segment the dispatcher now raises the library's own `ProtocolError` naming an invalid flags combination instead. One difference *is* observable, and it is a dump-format change rather than a value change: a flagless segment's `connection` renders as the string `Flags::None [0]` where it rendered as the number `0`, in all three of the JSON, tree and PLIST outputs. The value is numerically the same and the wire bytes are untouched; what changed is that `connection` no longer switches JSON type with the flags, having been a number for a flagless segment and a string -- `Flags::ACK [2048]` -- for every other one. The literal `None` inside it is a separate rendering defect: the hook `pcapkit.dumpkit.common.make_dumper` installs interpolates `o.name` without accounting for a nameless composite member, so it would spell any zero-valued flag enumeration in the library the same way. That is left to its own change and pinned by a test here so it cannot drift unnoticed. The committed example dumps are unaffected, `examples/captures/in.pcap` carrying no flagless segment. Coverage cannot see the fix itself, because the changed line already executed and `tcp.py` reads 100% statement and branch either side; the added subtests are the evidence (#616). +- **Fixed** -- **a breaking change to a public attribute.** `Frame.len` and `Frame.cap_len` were filled from *opposite* wire fields depending on which container format was read, so `frame.len` meant the captured length out of a `.pcap` and the on-wire length out of a `.pcapng`. The PCAP reader is the one that moved, and now matches the PCAP-NG reader: `len` is the on-wire length (the record header's `orig_len`) and `cap_len` the octets actually stored (`incl_len`). **Code reading `frame.len` or `frame.cap_len` from a `.pcap` gets the other field's value than it did before**, and for a truncated frame that is a different number rather than a relabelling. Which reader to move was a real decision rather than the correction of a typo, and not settled by seniority: the PCAP spelling is the *older* of the two, dating to `c43892af` (2022-01-11) with the data model's docstrings agreeing with it a day later, while the opposite spelling in `pcapkit.toolkit.pcapng` arrived 15 months afterwards in `25f216f4` (2023-04-27). Both were internally consistent. What settles it is that the *names* are Wireshark's, and its `epan/dissectors/packet-frame.c` registers `frame.len` as "Frame length on the wire" and `frame.cap_len` as "Frame length stored into the capture file", and raises the `frame.len_lt_caplen` expert info, `PI_MALFORMED`/`PI_ERROR`, on `frame_len < cap_len` -- which could not be malformed if `len` were the smaller, captured one. So the later spelling is the one that matches the borrowed names. Both formats define the underlying fields the same way: `pcap-savefile(5)` gives `incl_len` as "the number of bytes of captured data that follow the per-packet header" and `orig_len` as "the number of bytes that would have been present had the packet not been truncated by the snapshot length", and `draft-ietf-opsawg-pcapng` gives Captured Packet Length as "the number of octets captured from the packet" against Original Packet Length's "number of octets of packet data that would have been provided had the packet not been truncated", which "SHOULD NOT be less than the Captured Packet Length". The internal consumer moved with it: `Frame.read` hands `_decode_next_layer` the octets that are *present*, which is now spelled `frame.cap_len` and was `frame.len` when that was the captured length -- handing over the on-wire length instead would be the declared-length-exceeds-available-octets fault of #554, #573 and #594 on every snapped frame. The dumpers are untouched, since `PCAPIO` packs its record header from `frame_info.incl_len` and `frame_info.orig_len` rather than from these two, and `frame_info` was already right in both readers. This went unnoticed because the two lengths differ only for a frame the snapshot length cut short, and no such fixture existed until #614 added one -- `big_endian.pcap`'s third frame, 1200 octets on the wire against 96 captured -- so every earlier assertion compared a value against itself (#618). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index a69df155c..de2315c3a 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1298,6 +1298,47 @@ pull requests between #326 and #509. ``examples/captures/in.pcap`` carrying no flagless segment. Coverage cannot see the fix itself, because the changed line already executed and ``tcp.py`` reads 100% statement and branch either side; the added subtests are the evidence (#616). +* **Fixed** -- **a breaking change to a public attribute.** ``Frame.len`` and + ``Frame.cap_len`` were filled from *opposite* wire fields depending on which + container format was read, so ``frame.len`` meant the captured length out of a + ``.pcap`` and the on-wire length out of a ``.pcapng``. The PCAP reader is the + one that moved, and now matches the PCAP-NG reader: ``len`` is the on-wire + length (the record header's ``orig_len``) and ``cap_len`` the octets actually + stored (``incl_len``). **Code reading ``frame.len`` or ``frame.cap_len`` from a + ``.pcap`` gets the other field's value than it did before**, and for a + truncated frame that is a different number rather than a relabelling. Which + reader to move was a real decision rather than the correction of a typo, and + not settled by seniority: the PCAP spelling is the *older* of the two, dating + to ``c43892af`` (2022-01-11) with the data model's docstrings agreeing with it + a day later, while the opposite spelling in ``pcapkit.toolkit.pcapng`` arrived + 15 months afterwards in ``25f216f4`` (2023-04-27). Both were internally + consistent. What settles it is that the *names* are Wireshark's, and its + ``epan/dissectors/packet-frame.c`` registers ``frame.len`` as "Frame length on + the wire" and ``frame.cap_len`` as "Frame length stored into the capture file", + and raises the ``frame.len_lt_caplen`` expert info, + ``PI_MALFORMED``/``PI_ERROR``, on ``frame_len < cap_len`` -- which could not be + malformed if ``len`` were the smaller, captured one. So the later spelling is + the one that matches the borrowed names. + Both formats define the underlying fields the same way: ``pcap-savefile(5)`` + gives ``incl_len`` as "the number of bytes of captured data that follow the + per-packet header" and ``orig_len`` as "the number of bytes that would have + been present had the packet not been truncated by the snapshot length", and + ``draft-ietf-opsawg-pcapng`` gives Captured Packet Length as "the number of + octets captured from the packet" against Original Packet Length's "number of + octets of packet data that would have been provided had the packet not been + truncated", which "SHOULD NOT be less than the Captured Packet Length". The + internal consumer moved with it: ``Frame.read`` hands ``_decode_next_layer`` + the octets that are *present*, which is now spelled ``frame.cap_len`` and was + ``frame.len`` when that was the captured length -- handing over the on-wire + length instead would be the declared-length-exceeds-available-octets fault of + #554, #573 and #594 on every snapped frame. The dumpers are untouched, since + ``PCAPIO`` packs its record header from ``frame_info.incl_len`` and + ``frame_info.orig_len`` rather than from these two, and ``frame_info`` was + already right in both readers. This went unnoticed because the two lengths + differ only for a frame the snapshot length cut short, and no such fixture + existed until #614 added one -- ``big_endian.pcap``'s third frame, 1200 octets + on the wire against 96 captured -- so every earlier assertion compared a value + against itself (#618). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 87a8eabb8a96adaadd9efc6149c06ff8c0ad532f Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 12:11:11 -0400 Subject: [PATCH 03/46] docs(changelog): the 1.5.0 entry for #610, whose code ships in #636 The bullet #636 originally carried, moved here verbatim so that #636 touches only `pcapkit/foundation/extraction.py` and its two test files. Covers: `Extractor` closing the caller's input stream and leaking the one it opened itself, both handlers now reading a single `_owns_input` predicate. 23 lines added to the entry file; `CHANGELOG.md` regenerated, not edited. --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 23 +++++++++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 57e0064c2..c45fd1042 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -88,6 +88,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- two inaccurate claims in the #630 entry above, caught by review after #630 had already merged. Both were about the history of `LICENSE` rather than about the change itself, and `LICENSE` is untouched here: 2017 was and remains the right answer, so this is a changelog-prose correction only. The entry asserted that the file "until then was stock MPL text carrying no author notice at all, so 2018 is the first start year the project ever asserted and it was already a year late when it was written". Every clause of it is wrong as a description of the era, though each was true of the single blob `bc836cfa2` happened to replace, which is where the mistake came from. The pre-BSD file was MIT, then GPL v3, then Apache 2.0, and only then MPL 2.0 -- four licences before BSD, not the one implied. The MIT and GPL texts both carried an author notice naming **2017**, the MIT one on line 3 of the initial commit. And `2018` first appeared in the Apache notice of 2018-12-08 as a single year, current for the year it was written rather than late. The claim also contradicted the entry's own opening sentence, which dates the first commit to 2017-11-07. What the history actually supports is a stronger argument for the change than the one that was written, which is why this is corrected rather than deleted: 2017 is the year the project asserted from its first commit, and #630 restores it rather than asserting it anew. The `BSD 3-Clause License` title line is reattributed too, from the opensource.org template -- whose licence text begins at the copyright line -- to choosealicense.com's, whose first line is literally that title, and which also carries the `Copyright (c) [year], [fullname]` comma this file uses. The error came from generalising the pre-BSD era off two blobs that turned out to be the same file, `bc836cfa2^:LICENSE` and `1c69341dc:LICENSE`; enumerating all 14 commits that ever touched `LICENSE` is what caught it (#638). - **Fixed** -- `TCP.read` seeded its connection-flag accumulator with `cast('Enum_Flags', 0)`. `typing.cast` is a runtime no-op -- it returns its second argument unchanged -- so the accumulator began life as the plain `int` `0`, and the `|=` that promotes it to a `pcapkit.const.tcp.flags.Flags` member is the only thing that ever did. A segment whose flags octet is all zero, an nmap NULL scan among them, promoted nothing and left `self._flags` an `int`, so a membership test against it raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering, and `TCP.connection` returned an `int` where both that property and `Data_TCP.connection` annotate `Flags`. Any flag at all masked it, which is how it survived a module at 100% statement and branch coverage. The seed is `Flags(0)` now, which is what #597 had already done to the sibling accumulator in `make` for the same reason; `Flags` declares no `_missing_` of its own, so `Flags(0)` is an ordinary `aenum.IntFlag` pseudo-member and does not meet the `RecursionError` that the same construction hits on the flag enumerations under `pcapkit.const.mh` (#623). Nothing a caller can reach changed: the three MP_JOIN layouts of [RFC 8684](https://datatracker.ietf.org/doc/html/rfc8684) section 3.2 are chosen by `_read_mptcp_join` from these very membership tests, but `mptcp_data_selector` rejects a flagless MP_JOIN with a `FieldError` before the dispatcher runs -- measured identical either side of the fix -- so the `TypeError` was latent rather than live, and latent only by virtue of a guard in another file. Called directly on a flagless parsed segment the dispatcher now raises the library's own `ProtocolError` naming an invalid flags combination instead. One difference *is* observable, and it is a dump-format change rather than a value change: a flagless segment's `connection` renders as the string `Flags::None [0]` where it rendered as the number `0`, in all three of the JSON, tree and PLIST outputs. The value is numerically the same and the wire bytes are untouched; what changed is that `connection` no longer switches JSON type with the flags, having been a number for a flagless segment and a string -- `Flags::ACK [2048]` -- for every other one. The literal `None` inside it is a separate rendering defect: the hook `pcapkit.dumpkit.common.make_dumper` installs interpolates `o.name` without accounting for a nameless composite member, so it would spell any zero-valued flag enumeration in the library the same way. That is left to its own change and pinned by a test here so it cannot drift unnoticed. The committed example dumps are unaffected, `examples/captures/in.pcap` carrying no flagless segment. Coverage cannot see the fix itself, because the changed line already executed and `tcp.py` reads 100% statement and branch either side; the added subtests are the evidence (#616). - **Fixed** -- **a breaking change to a public attribute.** `Frame.len` and `Frame.cap_len` were filled from *opposite* wire fields depending on which container format was read, so `frame.len` meant the captured length out of a `.pcap` and the on-wire length out of a `.pcapng`. The PCAP reader is the one that moved, and now matches the PCAP-NG reader: `len` is the on-wire length (the record header's `orig_len`) and `cap_len` the octets actually stored (`incl_len`). **Code reading `frame.len` or `frame.cap_len` from a `.pcap` gets the other field's value than it did before**, and for a truncated frame that is a different number rather than a relabelling. Which reader to move was a real decision rather than the correction of a typo, and not settled by seniority: the PCAP spelling is the *older* of the two, dating to `c43892af` (2022-01-11) with the data model's docstrings agreeing with it a day later, while the opposite spelling in `pcapkit.toolkit.pcapng` arrived 15 months afterwards in `25f216f4` (2023-04-27). Both were internally consistent. What settles it is that the *names* are Wireshark's, and its `epan/dissectors/packet-frame.c` registers `frame.len` as "Frame length on the wire" and `frame.cap_len` as "Frame length stored into the capture file", and raises the `frame.len_lt_caplen` expert info, `PI_MALFORMED`/`PI_ERROR`, on `frame_len < cap_len` -- which could not be malformed if `len` were the smaller, captured one. So the later spelling is the one that matches the borrowed names. Both formats define the underlying fields the same way: `pcap-savefile(5)` gives `incl_len` as "the number of bytes of captured data that follow the per-packet header" and `orig_len` as "the number of bytes that would have been present had the packet not been truncated by the snapshot length", and `draft-ietf-opsawg-pcapng` gives Captured Packet Length as "the number of octets captured from the packet" against Original Packet Length's "number of octets of packet data that would have been provided had the packet not been truncated", which "SHOULD NOT be less than the Captured Packet Length". The internal consumer moved with it: `Frame.read` hands `_decode_next_layer` the octets that are *present*, which is now spelled `frame.cap_len` and was `frame.len` when that was the captured length -- handing over the on-wire length instead would be the declared-length-exceeds-available-octets fault of #554, #573 and #594 on every snapped frame. The dumpers are untouched, since `PCAPIO` packs its record header from `frame_info.incl_len` and `frame_info.orig_len` rather than from these two, and `frame_info` was already right in both readers. This went unnoticed because the two lengths differ only for a frame the snapshot length cut short, and no such fixture existed until #614 added one -- `big_endian.pcap`'s third frame, 1200 octets on the wire against 96 captured -- so every earlier assertion compared a value against itself (#618). +- **Fixed** -- `Extractor` had the ownership of its input stream inverted, so it did both halves of the wrong thing at once: a handle it opened itself, from `fin` given as a path, was **never closed**, while a stream the *caller* supplied and still needed **was**. `_cleanup` closed under `not self._flag_s` where `self._flag_s` is the flag that gated the `open()`, and the `SeekableReader` wrapping a non-seekable input asked for `stream_closing=not self._flag_s` in the same wrong direction. The leak was one descriptor per extraction for the life of the process, and the production root cause of the `ResourceWarning` that reddened #577, #596 and #600 from three unrelated pull requests -- #606 made the assertion in `tests/utilities/test_stacklevel.py` immune to foreign warnings, which repaired the CI signal but could not stop the leak, because the leak was in library code. Closing the caller's stream was the more dangerous half: silent data loss for anyone passing an open file they meant to keep reading, with no warning and no error to notice it by. Both conditions now follow ownership, via a single `Extractor._owns_input`, and a finaliser releases the handle of an extraction *abandoned* before end of file -- `auto=False`, iterated part way and dropped -- which reaches `_cleanup` by no route at all and was the second of the two warnings. Measured on `6c3d1b0d9`: a path-given extraction left 1 descriptor open and a stream-given one came back closed and unreadable; the two files #610 names now emit 0 such warnings where they emitted 2. `__exit__` is deliberately left closing unconditionally, since a caller who scopes an `Extractor` with `with` has asked for exactly that (#610). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index de2315c3a..597d410ff 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1339,6 +1339,29 @@ pull requests between #326 and #509. existed until #614 added one -- ``big_endian.pcap``'s third frame, 1200 octets on the wire against 96 captured -- so every earlier assertion compared a value against itself (#618). +* **Fixed** -- ``Extractor`` had the ownership of its input stream inverted, so it + did both halves of the wrong thing at once: a handle it opened itself, from + ``fin`` given as a path, was **never closed**, while a stream the *caller* + supplied and still needed **was**. ``_cleanup`` closed under + ``not self._flag_s`` where ``self._flag_s`` is the flag that gated the + ``open()``, and the ``SeekableReader`` wrapping a non-seekable input asked for + ``stream_closing=not self._flag_s`` in the same wrong direction. The leak was + one descriptor per extraction for the life of the process, and the production + root cause of the ``ResourceWarning`` that reddened #577, #596 and #600 from + three unrelated pull requests -- #606 made the assertion in + ``tests/utilities/test_stacklevel.py`` immune to foreign warnings, which + repaired the CI signal but could not stop the leak, because the leak was in + library code. Closing the caller's stream was the more dangerous half: silent + data loss for anyone passing an open file they meant to keep reading, with no + warning and no error to notice it by. Both conditions now follow ownership, via + a single ``Extractor._owns_input``, and a finaliser releases the handle of an + extraction *abandoned* before end of file -- ``auto=False``, iterated part way + and dropped -- which reaches ``_cleanup`` by no route at all and was the second + of the two warnings. Measured on ``6c3d1b0d9``: a path-given extraction left 1 + descriptor open and a stream-given one came back closed and unreadable; the two + files #610 names now emit 0 such warnings where they emitted 2. ``__exit__`` is + deliberately left closing unconditionally, since a caller who scopes an + ``Extractor`` with ``with`` has asked for exactly that (#610). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From b41f80f1600e30a2228d0a30098a3d8d500b48ba Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 12:11:21 -0400 Subject: [PATCH 04/46] docs(changelog): the 1.5.0 entry for #620, whose code ships in #639 The bullet #639 originally carried, moved here verbatim so that #639 touches only `pcapkit/foundation/extraction.py`, `pcapkit/interface/core.py` and its three test files. Covers: `extract(..., no_eof=True)` never returning, and the progress check that now ends it -- including the deliberate narrowing for a seekable input still being appended to. 43 lines added to the entry file; `CHANGELOG.md` regenerated, not edited. --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 43 +++++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index c45fd1042..152944a38 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -89,6 +89,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- `TCP.read` seeded its connection-flag accumulator with `cast('Enum_Flags', 0)`. `typing.cast` is a runtime no-op -- it returns its second argument unchanged -- so the accumulator began life as the plain `int` `0`, and the `|=` that promotes it to a `pcapkit.const.tcp.flags.Flags` member is the only thing that ever did. A segment whose flags octet is all zero, an nmap NULL scan among them, promoted nothing and left `self._flags` an `int`, so a membership test against it raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering, and `TCP.connection` returned an `int` where both that property and `Data_TCP.connection` annotate `Flags`. Any flag at all masked it, which is how it survived a module at 100% statement and branch coverage. The seed is `Flags(0)` now, which is what #597 had already done to the sibling accumulator in `make` for the same reason; `Flags` declares no `_missing_` of its own, so `Flags(0)` is an ordinary `aenum.IntFlag` pseudo-member and does not meet the `RecursionError` that the same construction hits on the flag enumerations under `pcapkit.const.mh` (#623). Nothing a caller can reach changed: the three MP_JOIN layouts of [RFC 8684](https://datatracker.ietf.org/doc/html/rfc8684) section 3.2 are chosen by `_read_mptcp_join` from these very membership tests, but `mptcp_data_selector` rejects a flagless MP_JOIN with a `FieldError` before the dispatcher runs -- measured identical either side of the fix -- so the `TypeError` was latent rather than live, and latent only by virtue of a guard in another file. Called directly on a flagless parsed segment the dispatcher now raises the library's own `ProtocolError` naming an invalid flags combination instead. One difference *is* observable, and it is a dump-format change rather than a value change: a flagless segment's `connection` renders as the string `Flags::None [0]` where it rendered as the number `0`, in all three of the JSON, tree and PLIST outputs. The value is numerically the same and the wire bytes are untouched; what changed is that `connection` no longer switches JSON type with the flags, having been a number for a flagless segment and a string -- `Flags::ACK [2048]` -- for every other one. The literal `None` inside it is a separate rendering defect: the hook `pcapkit.dumpkit.common.make_dumper` installs interpolates `o.name` without accounting for a nameless composite member, so it would spell any zero-valued flag enumeration in the library the same way. That is left to its own change and pinned by a test here so it cannot drift unnoticed. The committed example dumps are unaffected, `examples/captures/in.pcap` carrying no flagless segment. Coverage cannot see the fix itself, because the changed line already executed and `tcp.py` reads 100% statement and branch either side; the added subtests are the evidence (#616). - **Fixed** -- **a breaking change to a public attribute.** `Frame.len` and `Frame.cap_len` were filled from *opposite* wire fields depending on which container format was read, so `frame.len` meant the captured length out of a `.pcap` and the on-wire length out of a `.pcapng`. The PCAP reader is the one that moved, and now matches the PCAP-NG reader: `len` is the on-wire length (the record header's `orig_len`) and `cap_len` the octets actually stored (`incl_len`). **Code reading `frame.len` or `frame.cap_len` from a `.pcap` gets the other field's value than it did before**, and for a truncated frame that is a different number rather than a relabelling. Which reader to move was a real decision rather than the correction of a typo, and not settled by seniority: the PCAP spelling is the *older* of the two, dating to `c43892af` (2022-01-11) with the data model's docstrings agreeing with it a day later, while the opposite spelling in `pcapkit.toolkit.pcapng` arrived 15 months afterwards in `25f216f4` (2023-04-27). Both were internally consistent. What settles it is that the *names* are Wireshark's, and its `epan/dissectors/packet-frame.c` registers `frame.len` as "Frame length on the wire" and `frame.cap_len` as "Frame length stored into the capture file", and raises the `frame.len_lt_caplen` expert info, `PI_MALFORMED`/`PI_ERROR`, on `frame_len < cap_len` -- which could not be malformed if `len` were the smaller, captured one. So the later spelling is the one that matches the borrowed names. Both formats define the underlying fields the same way: `pcap-savefile(5)` gives `incl_len` as "the number of bytes of captured data that follow the per-packet header" and `orig_len` as "the number of bytes that would have been present had the packet not been truncated by the snapshot length", and `draft-ietf-opsawg-pcapng` gives Captured Packet Length as "the number of octets captured from the packet" against Original Packet Length's "number of octets of packet data that would have been provided had the packet not been truncated", which "SHOULD NOT be less than the Captured Packet Length". The internal consumer moved with it: `Frame.read` hands `_decode_next_layer` the octets that are *present*, which is now spelled `frame.cap_len` and was `frame.len` when that was the captured length -- handing over the on-wire length instead would be the declared-length-exceeds-available-octets fault of #554, #573 and #594 on every snapped frame. The dumpers are untouched, since `PCAPIO` packs its record header from `frame_info.incl_len` and `frame_info.orig_len` rather than from these two, and `frame_info` was already right in both readers. This went unnoticed because the two lengths differ only for a frame the snapshot length cut short, and no such fixture existed until #614 added one -- `big_endian.pcap`'s third frame, 1200 octets on the wire against 96 captured -- so every earlier assertion compared a value against itself (#618). - **Fixed** -- `Extractor` had the ownership of its input stream inverted, so it did both halves of the wrong thing at once: a handle it opened itself, from `fin` given as a path, was **never closed**, while a stream the *caller* supplied and still needed **was**. `_cleanup` closed under `not self._flag_s` where `self._flag_s` is the flag that gated the `open()`, and the `SeekableReader` wrapping a non-seekable input asked for `stream_closing=not self._flag_s` in the same wrong direction. The leak was one descriptor per extraction for the life of the process, and the production root cause of the `ResourceWarning` that reddened #577, #596 and #600 from three unrelated pull requests -- #606 made the assertion in `tests/utilities/test_stacklevel.py` immune to foreign warnings, which repaired the CI signal but could not stop the leak, because the leak was in library code. Closing the caller's stream was the more dangerous half: silent data loss for anyone passing an open file they meant to keep reading, with no warning and no error to notice it by. Both conditions now follow ownership, via a single `Extractor._owns_input`, and a finaliser releases the handle of an extraction *abandoned* before end of file -- `auto=False`, iterated part way and dropped -- which reaches `_cleanup` by no route at all and was the second of the two warnings. Measured on `6c3d1b0d9`: a path-given extraction left 1 descriptor open and a stream-given one came back closed and unreadable; the two files #610 names now emit 0 such warnings where they emitted 2. `__exit__` is deliberately left closing unconditionally, since a caller who scopes an `Extractor` with `with` has asked for exactly that (#610). +- **Fixed** -- `extract(..., no_eof=True)` never returned. End of stream was detected correctly and `ExtractionWarning: EOF reached` fired, but all three loops that handle it -- `record_frames`, `__next__` and `__call__` -- read `if self._flag_n: continue` with nothing else to stop them, so the flag suppressed the error that *ended* the loop without supplying any other ending and an exhausted input was retried forever. The flag is not pointless, which is why this is a termination condition rather than a deletion: `__main__` sets `no_eof` exactly for `fin='-'`, and the end of what has arrived on a live capture is not the end of the capture. What was missing was a way to tell a capture that has paused from one that is over, and the input's own position answers it -- `prepare` raises end of stream when the bytes remaining measure zero and restores the position before raising, so two consecutive ends of stream at the same position mean nothing arrived between them. `Extractor._note_eof_progress` is that check. A pipe is unaffected, because a read there *blocks* while the writer is open but idle rather than reporting end of stream -- measured at a full 1.5s pause on `6c3d1b0d9`, after which the frames arrived -- so a paused pipe never reaches the check at all. Also fixed thereby, and not mentioned in the issue: `pcapkit -` hung on **any** finished stdin, a pipe reporting end of stream once its writer closes; verified through the real CLI, where `cat in.pcap | python -m pcapkit -` went from killed at 25s to exiting 0 with a byte-identical dump. **One deliberate narrowing**: a *seekable* input does not block, so its two probes fall microseconds apart and a file still being appended to now ends at the data present when the extraction reached it -- measured with the append landing on a record boundary 0.6s in, six frames before and five after. The previous behaviour was unbounded by construction, which is the defect itself, so some stopping rule had to be chosen; a timed grace period would make the cut-off intermittent rather than absent, so following a growing file is left to a policy of its own and the decision is pinned by a test. Worth knowing alongside it, pre-existing and untouched: an append landing *mid-record* raises `ValueError: read length must be non-negative or -1` instead, so `no_eof` over a growing file only ever worked for a writer flushing whole records. Both docstrings for `no_eof` now say all of this. The end-to-end regression tests are bounded by a **child process** rather than by `tests._support.time_limit`, because the in-process deadline proved *intermittent* on this loop: it usually expired on time and once escaped entirely, running past ten minutes and 13.2 GB RSS before being killed by hand. An intermittent guard against a hang is worse than none, since the run it misses is a wedged suite rather than a red test. Two guesses at the cause are recorded as ruled out, so they are not made again: the parse path's two `except Exception` handlers are never entered during the spin, because `prepare` raises before any next-layer decode, and the memory is the retry loop emitting tens of thousands of `EOF reached` records a second which the runner retains (#620). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 597d410ff..beeaeabc2 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1362,6 +1362,49 @@ pull requests between #326 and #509. files #610 names now emit 0 such warnings where they emitted 2. ``__exit__`` is deliberately left closing unconditionally, since a caller who scopes an ``Extractor`` with ``with`` has asked for exactly that (#610). +* **Fixed** -- ``extract(..., no_eof=True)`` never returned. End of stream was + detected correctly and ``ExtractionWarning: EOF reached`` fired, but all three + loops that handle it -- ``record_frames``, ``__next__`` and ``__call__`` -- read + ``if self._flag_n: continue`` with nothing else to stop them, so the flag + suppressed the error that *ended* the loop without supplying any other ending + and an exhausted input was retried forever. The flag is not pointless, which is + why this is a termination condition rather than a deletion: ``__main__`` sets + ``no_eof`` exactly for ``fin='-'``, and the end of what has arrived on a live + capture is not the end of the capture. What was missing was a way to tell a + capture that has paused from one that is over, and the input's own position + answers it -- ``prepare`` raises end of stream when the bytes remaining measure + zero and restores the position before raising, so two consecutive ends of stream + at the same position mean nothing arrived between them. + ``Extractor._note_eof_progress`` is that check. A pipe is unaffected, because a + read there *blocks* while the writer is open but idle rather than reporting end + of stream -- measured at a full 1.5s pause on ``6c3d1b0d9``, after which the + frames arrived -- so a paused pipe never reaches the check at all. Also fixed + thereby, and not mentioned in the issue: ``pcapkit -`` hung on **any** finished + stdin, a pipe reporting end of stream once its writer closes; verified through + the real CLI, where ``cat in.pcap | python -m pcapkit -`` went from killed at + 25s to exiting 0 with a byte-identical dump. **One deliberate narrowing**: a + *seekable* input does not block, so its two probes fall microseconds apart and a + file still being appended to now ends at the data present when the extraction + reached it -- measured with the append landing on a record boundary 0.6s in, + six frames before and five after. The previous behaviour was unbounded by + construction, which is the defect itself, so some stopping rule had to be + chosen; a timed grace period would make the cut-off intermittent rather than + absent, so following a growing file is left to a policy of its own and the + decision is pinned by a test. Worth knowing alongside it, pre-existing and + untouched: an append landing *mid-record* raises + ``ValueError: read length must be non-negative or -1`` + instead, so ``no_eof`` over a growing file only ever worked for a writer + flushing whole records. Both docstrings for ``no_eof`` now say all of this. The end-to-end regression tests are bounded by a **child process** rather + than by ``tests._support.time_limit``, because the in-process deadline proved + *intermittent* on this loop: it usually expired on time and once escaped + entirely, running past ten minutes and 13.2 GB RSS before being killed by hand. + An intermittent guard against a hang is worse than none, since the run it misses + is a wedged suite rather than a red test. Two guesses at the cause are recorded + as ruled out, so they are not made again: the parse path's two + ``except Exception`` handlers are never entered during the spin, because + ``prepare`` raises before any next-layer decode, and the memory is the retry loop + emitting tens of thousands of ``EOF reached`` records a second which the runner + retains (#620). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 69a6e13001b2aeb403d5c275e45b0d05babbc631 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 12:11:31 -0400 Subject: [PATCH 05/46] docs(changelog): the 1.5.0 entry for #617, whose code ships in #640 The bullet #640 originally carried, moved here verbatim so that #640 touches only `pcapkit/protocols/protocol.py`, `pcapkit/protocols/application/http.py`, `docs/source/ext.rst`, `examples/generators/dispatch.py` and its six test files. Covers: the behaviour change to a public API -- building a protocol through its constructor with a keyword no signature declares now raises `UnsupportedCall` instead of discarding it. 55 lines added to the entry file; `CHANGELOG.md` regenerated, not edited. --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 55 +++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 152944a38..a0ad2633c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -90,6 +90,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- **a breaking change to a public attribute.** `Frame.len` and `Frame.cap_len` were filled from *opposite* wire fields depending on which container format was read, so `frame.len` meant the captured length out of a `.pcap` and the on-wire length out of a `.pcapng`. The PCAP reader is the one that moved, and now matches the PCAP-NG reader: `len` is the on-wire length (the record header's `orig_len`) and `cap_len` the octets actually stored (`incl_len`). **Code reading `frame.len` or `frame.cap_len` from a `.pcap` gets the other field's value than it did before**, and for a truncated frame that is a different number rather than a relabelling. Which reader to move was a real decision rather than the correction of a typo, and not settled by seniority: the PCAP spelling is the *older* of the two, dating to `c43892af` (2022-01-11) with the data model's docstrings agreeing with it a day later, while the opposite spelling in `pcapkit.toolkit.pcapng` arrived 15 months afterwards in `25f216f4` (2023-04-27). Both were internally consistent. What settles it is that the *names* are Wireshark's, and its `epan/dissectors/packet-frame.c` registers `frame.len` as "Frame length on the wire" and `frame.cap_len` as "Frame length stored into the capture file", and raises the `frame.len_lt_caplen` expert info, `PI_MALFORMED`/`PI_ERROR`, on `frame_len < cap_len` -- which could not be malformed if `len` were the smaller, captured one. So the later spelling is the one that matches the borrowed names. Both formats define the underlying fields the same way: `pcap-savefile(5)` gives `incl_len` as "the number of bytes of captured data that follow the per-packet header" and `orig_len` as "the number of bytes that would have been present had the packet not been truncated by the snapshot length", and `draft-ietf-opsawg-pcapng` gives Captured Packet Length as "the number of octets captured from the packet" against Original Packet Length's "number of octets of packet data that would have been provided had the packet not been truncated", which "SHOULD NOT be less than the Captured Packet Length". The internal consumer moved with it: `Frame.read` hands `_decode_next_layer` the octets that are *present*, which is now spelled `frame.cap_len` and was `frame.len` when that was the captured length -- handing over the on-wire length instead would be the declared-length-exceeds-available-octets fault of #554, #573 and #594 on every snapped frame. The dumpers are untouched, since `PCAPIO` packs its record header from `frame_info.incl_len` and `frame_info.orig_len` rather than from these two, and `frame_info` was already right in both readers. This went unnoticed because the two lengths differ only for a frame the snapshot length cut short, and no such fixture existed until #614 added one -- `big_endian.pcap`'s third frame, 1200 octets on the wire against 96 captured -- so every earlier assertion compared a value against itself (#618). - **Fixed** -- `Extractor` had the ownership of its input stream inverted, so it did both halves of the wrong thing at once: a handle it opened itself, from `fin` given as a path, was **never closed**, while a stream the *caller* supplied and still needed **was**. `_cleanup` closed under `not self._flag_s` where `self._flag_s` is the flag that gated the `open()`, and the `SeekableReader` wrapping a non-seekable input asked for `stream_closing=not self._flag_s` in the same wrong direction. The leak was one descriptor per extraction for the life of the process, and the production root cause of the `ResourceWarning` that reddened #577, #596 and #600 from three unrelated pull requests -- #606 made the assertion in `tests/utilities/test_stacklevel.py` immune to foreign warnings, which repaired the CI signal but could not stop the leak, because the leak was in library code. Closing the caller's stream was the more dangerous half: silent data loss for anyone passing an open file they meant to keep reading, with no warning and no error to notice it by. Both conditions now follow ownership, via a single `Extractor._owns_input`, and a finaliser releases the handle of an extraction *abandoned* before end of file -- `auto=False`, iterated part way and dropped -- which reaches `_cleanup` by no route at all and was the second of the two warnings. Measured on `6c3d1b0d9`: a path-given extraction left 1 descriptor open and a stream-given one came back closed and unreadable; the two files #610 names now emit 0 such warnings where they emitted 2. `__exit__` is deliberately left closing unconditionally, since a caller who scopes an `Extractor` with `with` has asked for exactly that (#610). - **Fixed** -- `extract(..., no_eof=True)` never returned. End of stream was detected correctly and `ExtractionWarning: EOF reached` fired, but all three loops that handle it -- `record_frames`, `__next__` and `__call__` -- read `if self._flag_n: continue` with nothing else to stop them, so the flag suppressed the error that *ended* the loop without supplying any other ending and an exhausted input was retried forever. The flag is not pointless, which is why this is a termination condition rather than a deletion: `__main__` sets `no_eof` exactly for `fin='-'`, and the end of what has arrived on a live capture is not the end of the capture. What was missing was a way to tell a capture that has paused from one that is over, and the input's own position answers it -- `prepare` raises end of stream when the bytes remaining measure zero and restores the position before raising, so two consecutive ends of stream at the same position mean nothing arrived between them. `Extractor._note_eof_progress` is that check. A pipe is unaffected, because a read there *blocks* while the writer is open but idle rather than reporting end of stream -- measured at a full 1.5s pause on `6c3d1b0d9`, after which the frames arrived -- so a paused pipe never reaches the check at all. Also fixed thereby, and not mentioned in the issue: `pcapkit -` hung on **any** finished stdin, a pipe reporting end of stream once its writer closes; verified through the real CLI, where `cat in.pcap | python -m pcapkit -` went from killed at 25s to exiting 0 with a byte-identical dump. **One deliberate narrowing**: a *seekable* input does not block, so its two probes fall microseconds apart and a file still being appended to now ends at the data present when the extraction reached it -- measured with the append landing on a record boundary 0.6s in, six frames before and five after. The previous behaviour was unbounded by construction, which is the defect itself, so some stopping rule had to be chosen; a timed grace period would make the cut-off intermittent rather than absent, so following a growing file is left to a policy of its own and the decision is pinned by a test. Worth knowing alongside it, pre-existing and untouched: an append landing *mid-record* raises `ValueError: read length must be non-negative or -1` instead, so `no_eof` over a growing file only ever worked for a writer flushing whole records. Both docstrings for `no_eof` now say all of this. The end-to-end regression tests are bounded by a **child process** rather than by `tests._support.time_limit`, because the in-process deadline proved *intermittent* on this loop: it usually expired on time and once escaped entirely, running past ten minutes and 13.2 GB RSS before being killed by hand. An intermittent guard against a hang is worse than none, since the run it misses is a wedged suite rather than a red test. Two guesses at the cause are recorded as ruled out, so they are not made again: the parse path's two `except Exception` handlers are never entered during the spin, because `prepare` raises before any next-layer decode, and the memory is the retry loop emitting tens of thousands of `EOF reached` records a second which the runner retains (#620). +- **Changed** -- building a protocol through its constructor with a keyword that names nothing now raises `UnsupportedCall` instead of discarding it. **This is a behaviour change to a public API**: every `make` in the tree ends its signature with `**kwargs` and reads nothing out of it, so until now a misspelled keyword was accepted, dropped, and the field it named kept its default -- wrong octets, with nothing said. That is what #602 cost: `examples/generators/options.py` asked for `seq=1` where `TCP.make` spells the parameter `seq_no`, and 25 generated fixture frames carried sequence number `0` against an empty `warnings` list. #541 and #556 were the same silence. The schema layer has never been so permissive -- `Schema.__update__` warns `UnknownFieldWarning` for a field it does not know -- and the asymmetry between the two halves of the same construction is what this closes. Checked in `ProtocolBase.__init__` rather than in `make`, because `make` is not the only consumer of the keywords it is handed: `__post_init__` passes one `**kwargs` to the construction *and* to the parse of what it has just constructed, so a keyword declared only by `read` legitimately travels through `make` -- `HIP.read` declares `extension` where `HIP.make` does not, and the option generator depends on it. The accepted set is therefore the union of every keyword-taking parameter of `make`, `read`, `pack`, `unpack`, `__post_init__` and `__init__` across the whole MRO, computed once per class from `inspect.signature`. Parsing is deliberately untouched, since there the keywords are whatever the engines and the four `_import_next_layer` implementations forward and a protocol cannot know which of its ancestors' its parent passed on -- and nothing was ever lost that way, a dropped parse keyword changing how a packet is read rather than what its octets say. Two escapes exist for the shapes a signature cannot express, both opt-in per class through a new `__keywords__`: a set, for a keyword read out of `**kwargs` by name as `ESP.read` does with `packet`; and `None`, for a dispatcher whose real signature belongs to a class chosen at call time, which is exactly `HTTP.make` forwarding to `HTTPv1`/`HTTPv2` and the one place in the tree that uses it. The message names the near neighbour it found, so `seq` reports *did you mean 'seq_no'?*. `from_data` warns `UnknownFieldWarning` where a caller would be raised at, because the keywords there are whatever `_make_data` returned rather than anything anybody typed, so the defect is a key of that mapping disagreeing with the signature it is spread into and the person who meets it is not the person who can fix it. Expect this to surface latent bugs in code that has been quietly losing a field, which is the point. It surfaced four in this repository, all the residue of #602 and all fixed here: the `_TCP_BASE` of `examples/generators/dispatch.py` and three stale copies of it under `tests/protocols/transport/`, each still passing `seq`/`ack_flag`/`urgent_pointer` and so building segments with sequence number `0` where they read as `1`. It surfaced three more that are reported rather than fixed, being a defect per protocol rather than one in this mechanism: `Frame._make_data` returns `ts_src` where `make` declares `ts_sec`, `L2TPv2._make_data` returns `prio` where it declares `priority`, and `Header._make_data` returns a `magic_number` that `Header.make` does not take at all -- so `from_data` has been dropping a frame's timestamp, an L2TPv2 priority bit and a capture's byte order, and now says so. One limitation worth knowing rather than discovering: a *direct* `SomeProtocol.make(...)` call is not checked and still discards in silence, since the check sits where every producer's keywords converge rather than inside each of the 30 `make` implementations -- `object.__new__(cls).make(**kwargs)` is the idiom that reaches it, and `HTTP.make` uses it to reach its versioned implementation (#617). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index beeaeabc2..0e9ebf23f 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1405,6 +1405,61 @@ pull requests between #326 and #509. ``prepare`` raises before any next-layer decode, and the memory is the retry loop emitting tens of thousands of ``EOF reached`` records a second which the runner retains (#620). +* **Changed** -- building a protocol through its constructor with a keyword that + names nothing now raises ``UnsupportedCall`` instead of discarding it. **This is + a behaviour + change to a public API**: every ``make`` in the tree ends its signature with + ``**kwargs`` and reads nothing out of it, so until now a misspelled keyword was + accepted, dropped, and the field it named kept its default -- wrong octets, with + nothing said. That is what #602 cost: ``examples/generators/options.py`` asked + for ``seq=1`` where ``TCP.make`` spells the parameter ``seq_no``, and 25 + generated fixture frames carried sequence number ``0`` against an empty + ``warnings`` list. #541 and #556 were the same silence. The schema layer has + never been so permissive -- ``Schema.__update__`` warns ``UnknownFieldWarning`` + for a field it does not know -- and the asymmetry between the two halves of the + same construction is what this closes. Checked in ``ProtocolBase.__init__`` + rather than in ``make``, because ``make`` is not the only consumer of the + keywords it is handed: ``__post_init__`` passes one ``**kwargs`` to the + construction *and* to the parse of what it has just constructed, so a keyword + declared only by ``read`` legitimately travels through ``make`` -- ``HIP.read`` + declares ``extension`` where ``HIP.make`` does not, and the option generator + depends on it. The accepted set is therefore the union of every keyword-taking + parameter of ``make``, ``read``, ``pack``, ``unpack``, ``__post_init__`` and + ``__init__`` across the whole MRO, computed once per class from + ``inspect.signature``. Parsing is deliberately untouched, since there the + keywords are whatever the engines and the four ``_import_next_layer`` + implementations forward and a protocol cannot know which of its ancestors' its + parent passed on -- and nothing was ever lost that way, a dropped parse keyword + changing how a packet is read rather than what its octets say. Two escapes exist + for the shapes a signature cannot express, both opt-in per class through a new + ``__keywords__``: a set, for a keyword read out of ``**kwargs`` by name as + ``ESP.read`` does with ``packet``; and ``None``, for a dispatcher whose real + signature belongs to a class chosen at call time, which is exactly ``HTTP.make`` + forwarding to ``HTTPv1``/``HTTPv2`` and the one place in the tree that uses it. + The message names the near neighbour it found, so ``seq`` reports *did you mean + 'seq_no'?*. ``from_data`` warns ``UnknownFieldWarning`` where a caller would be + raised at, because the keywords there are whatever ``_make_data`` returned + rather than anything anybody typed, so the defect is a key of that mapping + disagreeing with the signature it is spread into and the person who meets it is + not the person who can fix it. Expect this to surface latent bugs in code that + has been quietly losing a field, which is the point. It surfaced four in this + repository, all the residue of #602 and all fixed here: the ``_TCP_BASE`` of + ``examples/generators/dispatch.py`` and three stale copies of it under + ``tests/protocols/transport/``, each still passing + ``seq``/``ack_flag``/``urgent_pointer`` and so building segments with sequence + number ``0`` where they read as ``1``. It surfaced three more that are reported + rather than fixed, being a defect per protocol rather than one in this mechanism: + ``Frame._make_data`` returns ``ts_src`` where ``make`` declares ``ts_sec``, + ``L2TPv2._make_data`` returns ``prio`` where it declares ``priority``, and + ``Header._make_data`` returns a ``magic_number`` that ``Header.make`` does not + take at all -- so ``from_data`` has been dropping a frame's timestamp, an + L2TPv2 priority bit and a capture's byte order, and now says so. One limitation + worth knowing rather than discovering: a *direct* ``SomeProtocol.make(...)`` + call is not checked and still discards in silence, since the check sits where + every producer's keywords converge rather than inside each of the 30 ``make`` + implementations -- ``object.__new__(cls).make(**kwargs)`` is the idiom that + reaches it, and ``HTTP.make`` uses it to reach its versioned implementation + (#617). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 2ce368721b0c0d86fdcfb98e555d3f394dea03b8 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 14:34:56 -0400 Subject: [PATCH 06/46] docs(changelog): the 1.5.0 entry for #653, #654 and #655, whose code ships in #665 The bullet #665 would otherwise have carried, kept here so that #665 touches only `pcapkit/protocols/internet/hip.py`, `pcapkit/protocols/schema/internet/hip.py`, `pcapkit/protocols/data/internet/hip.py` and `tests/protocols/internet/test_hip_unit.py`. Covers all three as one bullet, because they are one root cause: the HIP `PUZZLE` and `SOLUTION` builders derived the field width, the `Reserved` octet and the version-dependent length from the payload value instead of from the data model. Splitting the entry would tell the story three times and explain it none. Two public data models change, so the bullet says so in bold and carries a migration sentence: `SolutionParameter.lifetime` becomes `reserved` and an `int` rather than a `timedelta`, both parameter models gain a required `rhash_len`, and `_make_param_solution` no longer takes `lifetime=`. 46 lines added to the entry file; `CHANGELOG.md` regenerated with `util/changelog_md.py`, not edited. `--check` exits 0 and `tests/project/` is green at 96 passed, 469 subtests. Committed from a detached HEAD on 69a6e1300 and pushed to the branch ref, because `docs/changelog-1.5.0` is checked out in another agent's worktree at a stale f84652303 and could not be taken here. --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 46 +++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index a0ad2633c..3c0d3a447 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -91,6 +91,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- `Extractor` had the ownership of its input stream inverted, so it did both halves of the wrong thing at once: a handle it opened itself, from `fin` given as a path, was **never closed**, while a stream the *caller* supplied and still needed **was**. `_cleanup` closed under `not self._flag_s` where `self._flag_s` is the flag that gated the `open()`, and the `SeekableReader` wrapping a non-seekable input asked for `stream_closing=not self._flag_s` in the same wrong direction. The leak was one descriptor per extraction for the life of the process, and the production root cause of the `ResourceWarning` that reddened #577, #596 and #600 from three unrelated pull requests -- #606 made the assertion in `tests/utilities/test_stacklevel.py` immune to foreign warnings, which repaired the CI signal but could not stop the leak, because the leak was in library code. Closing the caller's stream was the more dangerous half: silent data loss for anyone passing an open file they meant to keep reading, with no warning and no error to notice it by. Both conditions now follow ownership, via a single `Extractor._owns_input`, and a finaliser releases the handle of an extraction *abandoned* before end of file -- `auto=False`, iterated part way and dropped -- which reaches `_cleanup` by no route at all and was the second of the two warnings. Measured on `6c3d1b0d9`: a path-given extraction left 1 descriptor open and a stream-given one came back closed and unreadable; the two files #610 names now emit 0 such warnings where they emitted 2. `__exit__` is deliberately left closing unconditionally, since a caller who scopes an `Extractor` with `with` has asked for exactly that (#610). - **Fixed** -- `extract(..., no_eof=True)` never returned. End of stream was detected correctly and `ExtractionWarning: EOF reached` fired, but all three loops that handle it -- `record_frames`, `__next__` and `__call__` -- read `if self._flag_n: continue` with nothing else to stop them, so the flag suppressed the error that *ended* the loop without supplying any other ending and an exhausted input was retried forever. The flag is not pointless, which is why this is a termination condition rather than a deletion: `__main__` sets `no_eof` exactly for `fin='-'`, and the end of what has arrived on a live capture is not the end of the capture. What was missing was a way to tell a capture that has paused from one that is over, and the input's own position answers it -- `prepare` raises end of stream when the bytes remaining measure zero and restores the position before raising, so two consecutive ends of stream at the same position mean nothing arrived between them. `Extractor._note_eof_progress` is that check. A pipe is unaffected, because a read there *blocks* while the writer is open but idle rather than reporting end of stream -- measured at a full 1.5s pause on `6c3d1b0d9`, after which the frames arrived -- so a paused pipe never reaches the check at all. Also fixed thereby, and not mentioned in the issue: `pcapkit -` hung on **any** finished stdin, a pipe reporting end of stream once its writer closes; verified through the real CLI, where `cat in.pcap | python -m pcapkit -` went from killed at 25s to exiting 0 with a byte-identical dump. **One deliberate narrowing**: a *seekable* input does not block, so its two probes fall microseconds apart and a file still being appended to now ends at the data present when the extraction reached it -- measured with the append landing on a record boundary 0.6s in, six frames before and five after. The previous behaviour was unbounded by construction, which is the defect itself, so some stopping rule had to be chosen; a timed grace period would make the cut-off intermittent rather than absent, so following a growing file is left to a policy of its own and the decision is pinned by a test. Worth knowing alongside it, pre-existing and untouched: an append landing *mid-record* raises `ValueError: read length must be non-negative or -1` instead, so `no_eof` over a growing file only ever worked for a writer flushing whole records. Both docstrings for `no_eof` now say all of this. The end-to-end regression tests are bounded by a **child process** rather than by `tests._support.time_limit`, because the in-process deadline proved *intermittent* on this loop: it usually expired on time and once escaped entirely, running past ten minutes and 13.2 GB RSS before being killed by hand. An intermittent guard against a hang is worse than none, since the run it misses is a wedged suite rather than a red test. Two guesses at the cause are recorded as ruled out, so they are not made again: the parse path's two `except Exception` handlers are never entered during the spin, because `prepare` raises before any next-layer decode, and the memory is the retry loop emitting tens of thousands of `EOF reached` records a second which the runner retains (#620). - **Changed** -- building a protocol through its constructor with a keyword that names nothing now raises `UnsupportedCall` instead of discarding it. **This is a behaviour change to a public API**: every `make` in the tree ends its signature with `**kwargs` and reads nothing out of it, so until now a misspelled keyword was accepted, dropped, and the field it named kept its default -- wrong octets, with nothing said. That is what #602 cost: `examples/generators/options.py` asked for `seq=1` where `TCP.make` spells the parameter `seq_no`, and 25 generated fixture frames carried sequence number `0` against an empty `warnings` list. #541 and #556 were the same silence. The schema layer has never been so permissive -- `Schema.__update__` warns `UnknownFieldWarning` for a field it does not know -- and the asymmetry between the two halves of the same construction is what this closes. Checked in `ProtocolBase.__init__` rather than in `make`, because `make` is not the only consumer of the keywords it is handed: `__post_init__` passes one `**kwargs` to the construction *and* to the parse of what it has just constructed, so a keyword declared only by `read` legitimately travels through `make` -- `HIP.read` declares `extension` where `HIP.make` does not, and the option generator depends on it. The accepted set is therefore the union of every keyword-taking parameter of `make`, `read`, `pack`, `unpack`, `__post_init__` and `__init__` across the whole MRO, computed once per class from `inspect.signature`. Parsing is deliberately untouched, since there the keywords are whatever the engines and the four `_import_next_layer` implementations forward and a protocol cannot know which of its ancestors' its parent passed on -- and nothing was ever lost that way, a dropped parse keyword changing how a packet is read rather than what its octets say. Two escapes exist for the shapes a signature cannot express, both opt-in per class through a new `__keywords__`: a set, for a keyword read out of `**kwargs` by name as `ESP.read` does with `packet`; and `None`, for a dispatcher whose real signature belongs to a class chosen at call time, which is exactly `HTTP.make` forwarding to `HTTPv1`/`HTTPv2` and the one place in the tree that uses it. The message names the near neighbour it found, so `seq` reports *did you mean 'seq_no'?*. `from_data` warns `UnknownFieldWarning` where a caller would be raised at, because the keywords there are whatever `_make_data` returned rather than anything anybody typed, so the defect is a key of that mapping disagreeing with the signature it is spread into and the person who meets it is not the person who can fix it. Expect this to surface latent bugs in code that has been quietly losing a field, which is the point. It surfaced four in this repository, all the residue of #602 and all fixed here: the `_TCP_BASE` of `examples/generators/dispatch.py` and three stale copies of it under `tests/protocols/transport/`, each still passing `seq`/`ack_flag`/`urgent_pointer` and so building segments with sequence number `0` where they read as `1`. It surfaced three more that are reported rather than fixed, being a defect per protocol rather than one in this mechanism: `Frame._make_data` returns `ts_src` where `make` declares `ts_sec`, `L2TPv2._make_data` returns `prio` where it declares `priority`, and `Header._make_data` returns a `magic_number` that `Header.make` does not take at all -- so `from_data` has been dropping a frame's timestamp, an L2TPv2 priority bit and a capture's byte order, and now says so. One limitation worth knowing rather than discovering: a *direct* `SomeProtocol.make(...)` call is not checked and still discards in silence, since the check sits where every producer's keywords converge rather than inside each of the 30 `make` implementations -- `object.__new__(cls).make(**kwargs)` is the idiom that reaches it, and `HTTP.make` uses it to reach its versioned implementation (#617). +- **Fixed** -- the HIP `PUZZLE` and `SOLUTION` builders derived three wire-format values from the payload value instead of taking them from the data model, and all three were wrong. **This changes two public data models and the octets both parameters emit.** The width of `Random #I`, and of `Puzzle solution #J`, came from `int.bit_length()` alone, and nothing else was available to derive it from, so every leading zero octet was dropped on re-serialisation: a `SOLUTION` read with `Length = 20` rebuilt as `Length = 6`, a `PUZZLE` read with `Length = 12` as `Length = 5`, and silently, because the integers survive and nothing raises. That width is `RHASH_len / 8` octets [[RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4), [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5)], a property of the Responder's HIT Suite rather than of the number that happens to sit in the field, so both data models now carry it as `rhash_len` and both builders prefer it; a new `rhash_len=` keyword states it for a build from scratch, which under HIPv2 is the only place it can come from, since `RSA,DSA/SHA-256` is the REQUIRED HIT Suite [[RFC 7401 Section 5.2.10](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.10)] and makes the field 32 octets rather than 8. On the mandatory path roughly one parameter in 256 has a zero top octet and lost it (#653). `SOLUTION`'s second contents octet is `Reserved`, "zero when sent, ignored when received" [[RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5), and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) identically] -- not the `Lifetime` that only [RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4) defines, and that pcapkit was encoding there as `2^(value - 32)` seconds. It wrote `0x20`, `0x21`, `0x25` or `0x2b` into a field the RFC requires to be zero, and could not write that zero at all: an RFC-conformant `SOLUTION` whose `Reserved` is `0x00` parsed to `timedelta(0)` and then could not be re-serialised, escaping a bare `ValueError` from `math.log2(0.0)` that was not a `BaseError` and so bypassed the library's own error handling entirely. The field is renamed `reserved`, defaults to the mandated zero, and is carried verbatim across a round trip rather than re-derived (#654). And neither builder read its own `version` keyword, so `version=1` and `version=2` computed identical lengths at every bit width, where [RFC 5201 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.4) and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) state both fields as literally 8 bytes and `Length` as literally 12 and 20. Under HIPv1 each builder therefore accepted only a `bit_length()` of 57..64 and built, for everything narrower, a parameter this library's own reader rejects -- the same shape as #608, in the `PUZZLE` builder that #629 never opened (#655). The two remaining `math.log2` sites, both in `PUZZLE` where a lifetime is real, now raise `ProtocolError` rather than letting `ValueError` escape; `ProtocolError` is `(BaseError, ValueError)`, so a caller written around the old bare exception still catches it. Migration: `SolutionParameter.lifetime` is now `reserved` and an `int` rather than a `timedelta`; both `PuzzleParameter` and `SolutionParameter` gained a required `rhash_len`; and `_make_param_solution` no longer takes `lifetime=`. All three were only reachable end to end once #608 was fixed in #629, which removed the parity guard that had been failing these rebuilds loudly first -- so the round trip stopped raising and started quietly emitting a different parameter, which is why they were worth fixing together (#653, #654, #655). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 0e9ebf23f..6c2aa3457 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1460,6 +1460,52 @@ pull requests between #326 and #509. implementations -- ``object.__new__(cls).make(**kwargs)`` is the idiom that reaches it, and ``HTTP.make`` uses it to reach its versioned implementation (#617). +* **Fixed** -- the HIP ``PUZZLE`` and ``SOLUTION`` builders derived three + wire-format values from the payload value instead of taking them from the data + model, and all three were wrong. **This changes two public data models and the + octets both parameters emit.** The width of ``Random #I``, and of + ``Puzzle solution #J``, came from ``int.bit_length()`` alone, and nothing else + was available to derive it from, so every leading zero octet was dropped on + re-serialisation: a ``SOLUTION`` read with ``Length = 20`` rebuilt as + ``Length = 6``, a ``PUZZLE`` read with ``Length = 12`` as ``Length = 5``, and + silently, because the integers survive and nothing raises. That width is + ``RHASH_len / 8`` octets [:rfc:`7401#section-5.2.4`, :rfc:`7401#section-5.2.5`], + a property of the Responder's HIT Suite rather than of the number that happens + to sit in the field, so both data models now carry it as ``rhash_len`` and both + builders prefer it; a new ``rhash_len=`` keyword states it for a build from + scratch, which under HIPv2 is the only place it can come from, since + ``RSA,DSA/SHA-256`` is the REQUIRED HIT Suite [:rfc:`7401#section-5.2.10`] and + makes the field 32 octets rather than 8. On the mandatory path roughly one + parameter in 256 has a zero top octet and lost it (#653). ``SOLUTION``'s second + contents octet is ``Reserved``, "zero when sent, ignored when received" + [:rfc:`7401#section-5.2.5`, and :rfc:`5201#section-5.2.5` identically] -- not the + ``Lifetime`` that only :rfc:`7401#section-5.2.4` defines, and that pcapkit was + encoding there as ``2^(value - 32)`` seconds. It wrote ``0x20``, ``0x21``, + ``0x25`` or ``0x2b`` into a field the RFC requires to be zero, and could not + write that zero at all: an RFC-conformant ``SOLUTION`` whose ``Reserved`` is + ``0x00`` parsed to ``timedelta(0)`` and then could not be re-serialised, escaping + a bare ``ValueError`` from ``math.log2(0.0)`` that was not a ``BaseError`` and so + bypassed the library's own error handling entirely. The field is renamed + ``reserved``, defaults to the mandated zero, and is carried verbatim across a + round trip rather than re-derived (#654). And neither builder read its own + ``version`` keyword, so ``version=1`` and ``version=2`` computed identical + lengths at every bit width, where :rfc:`5201#section-5.2.4` and + :rfc:`5201#section-5.2.5` state both fields as literally 8 bytes and ``Length`` + as literally 12 and 20. Under HIPv1 each builder therefore accepted only a + ``bit_length()`` of 57..64 and built, for everything narrower, a parameter this + library's own reader rejects -- the same shape as #608, in the ``PUZZLE`` builder + that #629 never opened (#655). The two remaining ``math.log2`` sites, both in + ``PUZZLE`` where a lifetime is real, now raise ``ProtocolError`` rather than + letting ``ValueError`` escape; ``ProtocolError`` is ``(BaseError, ValueError)``, + so a caller written around the old bare exception still catches it. Migration: + ``SolutionParameter.lifetime`` is now ``reserved`` and an ``int`` rather than a + ``timedelta``; both ``PuzzleParameter`` and ``SolutionParameter`` gained a + required ``rhash_len``; and ``_make_param_solution`` no longer takes + ``lifetime=``. All three were only reachable end to end once #608 was fixed in + #629, which removed the parity guard that had been failing these rebuilds loudly + first -- so the round trip stopped raising and started quietly emitting a + different parameter, which is why they were worth fixing together (#653, #654, + #655). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From e55ba367c744eadd156574cf9e6b4a668e24a53f Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 14:39:11 -0400 Subject: [PATCH 07/46] docs(changelog): the 1.5.0 entry for #651, whose code ships in #664 The bullet #664 would otherwise have carried, kept here so that #664 touches only `pcapkit/protocols/internet/hip.py`, `pcapkit/protocols/schema/internet/hip.py`, `tests/protocols/internet/test_hip_unit.py`, `tests/protocols/test_option_roundtrip_unit.py`, `examples/generators/options.py` and `docs/source/pcapkit/protocols/internet/hip.rst`. One bullet, because it is one root cause in 95 places: every HIP padding site aligned the parameter's *contents* to eight octets rather than the record, ignoring the four-octet type-and-length header, so every parameter pcapkit wrote was `4 (mod 8)` for every possible `Length`. The bullet says in bold that both the emitted octets and the data model's reported `length` change, and carries a migration sentence: a `SEQ` parameter's `length` is 8 where it was 12, so code comparing stored output byte for byte or asserting on `Data_*Parameter.length` sees different values. It also records what was deliberately *not* changed, since both look like part of the same defect and are not: `HIP.make`'s `len = total_length // 8 + 4`, which RFC 7401 section 5.1.3 shows is correct and merely needed 8-aligned parameters; and `HIP_COPIES`, which stays at two for `R1_COUNTER`'s four-octet `counter` against section 5.2.3's eight -- a separate, still-unfiled defect this one had been masking. The `EncryptedParameter.data` length callback is named as fixed in the same change because the two four-octet errors cancelled at four of the eight residues of `Length`, so correcting the padding alone would have regressed it. 41 lines added to the entry file; `CHANGELOG.md` regenerated with `util/changelog_md.py`, not edited. `--check` exits 0 and `tests/project/` is green at 96 passed, 469 subtests. Committed from a detached HEAD on 2ce368721 and pushed to the branch ref, because `docs/changelog-1.5.0` is checked out in another agent's worktree at a stale f84652303 and could not be taken here. Note 2ce368721, not the 69a6e1300 I was given: the branch had already moved on with #665's entry. --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 41 +++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3c0d3a447..71e5eedb6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,6 +92,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- `extract(..., no_eof=True)` never returned. End of stream was detected correctly and `ExtractionWarning: EOF reached` fired, but all three loops that handle it -- `record_frames`, `__next__` and `__call__` -- read `if self._flag_n: continue` with nothing else to stop them, so the flag suppressed the error that *ended* the loop without supplying any other ending and an exhausted input was retried forever. The flag is not pointless, which is why this is a termination condition rather than a deletion: `__main__` sets `no_eof` exactly for `fin='-'`, and the end of what has arrived on a live capture is not the end of the capture. What was missing was a way to tell a capture that has paused from one that is over, and the input's own position answers it -- `prepare` raises end of stream when the bytes remaining measure zero and restores the position before raising, so two consecutive ends of stream at the same position mean nothing arrived between them. `Extractor._note_eof_progress` is that check. A pipe is unaffected, because a read there *blocks* while the writer is open but idle rather than reporting end of stream -- measured at a full 1.5s pause on `6c3d1b0d9`, after which the frames arrived -- so a paused pipe never reaches the check at all. Also fixed thereby, and not mentioned in the issue: `pcapkit -` hung on **any** finished stdin, a pipe reporting end of stream once its writer closes; verified through the real CLI, where `cat in.pcap | python -m pcapkit -` went from killed at 25s to exiting 0 with a byte-identical dump. **One deliberate narrowing**: a *seekable* input does not block, so its two probes fall microseconds apart and a file still being appended to now ends at the data present when the extraction reached it -- measured with the append landing on a record boundary 0.6s in, six frames before and five after. The previous behaviour was unbounded by construction, which is the defect itself, so some stopping rule had to be chosen; a timed grace period would make the cut-off intermittent rather than absent, so following a growing file is left to a policy of its own and the decision is pinned by a test. Worth knowing alongside it, pre-existing and untouched: an append landing *mid-record* raises `ValueError: read length must be non-negative or -1` instead, so `no_eof` over a growing file only ever worked for a writer flushing whole records. Both docstrings for `no_eof` now say all of this. The end-to-end regression tests are bounded by a **child process** rather than by `tests._support.time_limit`, because the in-process deadline proved *intermittent* on this loop: it usually expired on time and once escaped entirely, running past ten minutes and 13.2 GB RSS before being killed by hand. An intermittent guard against a hang is worse than none, since the run it misses is a wedged suite rather than a red test. Two guesses at the cause are recorded as ruled out, so they are not made again: the parse path's two `except Exception` handlers are never entered during the spin, because `prepare` raises before any next-layer decode, and the memory is the retry loop emitting tens of thousands of `EOF reached` records a second which the runner retains (#620). - **Changed** -- building a protocol through its constructor with a keyword that names nothing now raises `UnsupportedCall` instead of discarding it. **This is a behaviour change to a public API**: every `make` in the tree ends its signature with `**kwargs` and reads nothing out of it, so until now a misspelled keyword was accepted, dropped, and the field it named kept its default -- wrong octets, with nothing said. That is what #602 cost: `examples/generators/options.py` asked for `seq=1` where `TCP.make` spells the parameter `seq_no`, and 25 generated fixture frames carried sequence number `0` against an empty `warnings` list. #541 and #556 were the same silence. The schema layer has never been so permissive -- `Schema.__update__` warns `UnknownFieldWarning` for a field it does not know -- and the asymmetry between the two halves of the same construction is what this closes. Checked in `ProtocolBase.__init__` rather than in `make`, because `make` is not the only consumer of the keywords it is handed: `__post_init__` passes one `**kwargs` to the construction *and* to the parse of what it has just constructed, so a keyword declared only by `read` legitimately travels through `make` -- `HIP.read` declares `extension` where `HIP.make` does not, and the option generator depends on it. The accepted set is therefore the union of every keyword-taking parameter of `make`, `read`, `pack`, `unpack`, `__post_init__` and `__init__` across the whole MRO, computed once per class from `inspect.signature`. Parsing is deliberately untouched, since there the keywords are whatever the engines and the four `_import_next_layer` implementations forward and a protocol cannot know which of its ancestors' its parent passed on -- and nothing was ever lost that way, a dropped parse keyword changing how a packet is read rather than what its octets say. Two escapes exist for the shapes a signature cannot express, both opt-in per class through a new `__keywords__`: a set, for a keyword read out of `**kwargs` by name as `ESP.read` does with `packet`; and `None`, for a dispatcher whose real signature belongs to a class chosen at call time, which is exactly `HTTP.make` forwarding to `HTTPv1`/`HTTPv2` and the one place in the tree that uses it. The message names the near neighbour it found, so `seq` reports *did you mean 'seq_no'?*. `from_data` warns `UnknownFieldWarning` where a caller would be raised at, because the keywords there are whatever `_make_data` returned rather than anything anybody typed, so the defect is a key of that mapping disagreeing with the signature it is spread into and the person who meets it is not the person who can fix it. Expect this to surface latent bugs in code that has been quietly losing a field, which is the point. It surfaced four in this repository, all the residue of #602 and all fixed here: the `_TCP_BASE` of `examples/generators/dispatch.py` and three stale copies of it under `tests/protocols/transport/`, each still passing `seq`/`ack_flag`/`urgent_pointer` and so building segments with sequence number `0` where they read as `1`. It surfaced three more that are reported rather than fixed, being a defect per protocol rather than one in this mechanism: `Frame._make_data` returns `ts_src` where `make` declares `ts_sec`, `L2TPv2._make_data` returns `prio` where it declares `priority`, and `Header._make_data` returns a `magic_number` that `Header.make` does not take at all -- so `from_data` has been dropping a frame's timestamp, an L2TPv2 priority bit and a capture's byte order, and now says so. One limitation worth knowing rather than discovering: a *direct* `SomeProtocol.make(...)` call is not checked and still discards in silence, since the check sits where every producer's keywords converge rather than inside each of the 30 `make` implementations -- `object.__new__(cls).make(**kwargs)` is the idiom that reaches it, and `HTTP.make` uses it to reach its versioned implementation (#617). - **Fixed** -- the HIP `PUZZLE` and `SOLUTION` builders derived three wire-format values from the payload value instead of taking them from the data model, and all three were wrong. **This changes two public data models and the octets both parameters emit.** The width of `Random #I`, and of `Puzzle solution #J`, came from `int.bit_length()` alone, and nothing else was available to derive it from, so every leading zero octet was dropped on re-serialisation: a `SOLUTION` read with `Length = 20` rebuilt as `Length = 6`, a `PUZZLE` read with `Length = 12` as `Length = 5`, and silently, because the integers survive and nothing raises. That width is `RHASH_len / 8` octets [[RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4), [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5)], a property of the Responder's HIT Suite rather than of the number that happens to sit in the field, so both data models now carry it as `rhash_len` and both builders prefer it; a new `rhash_len=` keyword states it for a build from scratch, which under HIPv2 is the only place it can come from, since `RSA,DSA/SHA-256` is the REQUIRED HIT Suite [[RFC 7401 Section 5.2.10](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.10)] and makes the field 32 octets rather than 8. On the mandatory path roughly one parameter in 256 has a zero top octet and lost it (#653). `SOLUTION`'s second contents octet is `Reserved`, "zero when sent, ignored when received" [[RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5), and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) identically] -- not the `Lifetime` that only [RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4) defines, and that pcapkit was encoding there as `2^(value - 32)` seconds. It wrote `0x20`, `0x21`, `0x25` or `0x2b` into a field the RFC requires to be zero, and could not write that zero at all: an RFC-conformant `SOLUTION` whose `Reserved` is `0x00` parsed to `timedelta(0)` and then could not be re-serialised, escaping a bare `ValueError` from `math.log2(0.0)` that was not a `BaseError` and so bypassed the library's own error handling entirely. The field is renamed `reserved`, defaults to the mandated zero, and is carried verbatim across a round trip rather than re-derived (#654). And neither builder read its own `version` keyword, so `version=1` and `version=2` computed identical lengths at every bit width, where [RFC 5201 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.4) and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) state both fields as literally 8 bytes and `Length` as literally 12 and 20. Under HIPv1 each builder therefore accepted only a `bit_length()` of 57..64 and built, for everything narrower, a parameter this library's own reader rejects -- the same shape as #608, in the `PUZZLE` builder that #629 never opened (#655). The two remaining `math.log2` sites, both in `PUZZLE` where a lifetime is real, now raise `ProtocolError` rather than letting `ValueError` escape; `ProtocolError` is `(BaseError, ValueError)`, so a caller written around the old bare exception still catches it. Migration: `SolutionParameter.lifetime` is now `reserved` and an `int` rather than a `timedelta`; both `PuzzleParameter` and `SolutionParameter` gained a required `rhash_len`; and `_make_param_solution` no longer takes `lifetime=`. All three were only reachable end to end once #608 was fixed in #629, which removed the parity guard that had been failing these rebuilds loudly first -- so the round trip stopped raising and started quietly emitting a different parameter, which is why they were worth fixing together (#653, #654, #655). +- **Fixed** -- every HIP parameter pcapkit emitted was `4 (mod 8)` octets long, because the padding was computed to align the *contents* rather than the record. **This changes the octets of every HIP parameter written, and the `length` every parameter's data model reports.** [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1) requires that "all of the encoded TLV parameters have a length (that includes the Type and Length fields), which is a multiple of 8 bytes", and states the arithmetic outright as `Total Length = 11 + Length - (Length + 3) % 8`; all 95 padding sites instead computed `(8 - (Length % 8)) % 8`, which has no `+ 4` inside the modulus and so aligns the contents alone. Across every `Length` from 0 to 63 the result was never a multiple of eight and never the value the RFC gives, and it erred in both directions: at `Length = 4` -- a whole `SEQ`, and [RFC 7401 Section 5.3.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.3.5) puts a `SEQ` or an `ACK` on every `UPDATE` -- the record is complete in eight octets and pcapkit appended four that must not be there, while at `Length = 8` the contents were already 8-aligned, nothing was appended, and the record went out four octets short. A conformant peer reading `Length` and consuming `11 + Length - (Length + 3) % 8` octets therefore lands mid-parameter and reads the rest of the parameter area at a wrong offset, in both directions; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test in the suite could see this and why the fix is asserted against the RFC's arithmetic rather than against a round trip. The 46 `PaddingField` callbacks in `pcapkit/protocols/schema/internet/hip.py` and the 49 record lengths in `pcapkit/protocols/internet/hip.py` are now one `parameter_total_len` and one `parameter_padding_len`, stating the RFC formula once instead of 95 times (#651). `EncryptedParameter`'s `data` length callback is fixed in the same change and could not have been left: it subtracted the sixteen `iv` octets but not the four `reserved` ones, and those four cancelled the padding's four at `Length % 8` in `{0, 5, 6, 7}` -- so correcting the padding alone would have taken `ENCRYPTED` from right at four of the eight residues to four octets too long at all eight. `HIP.make`'s `len = total_length // 8 + 4` is *not* part of the defect and is unchanged: [RFC 7401 Section 5.1.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.1.3) defines Header Length as the header and parameters "in 8-byte units, excluding the first 8 bytes", so the floor division is exact once each parameter is a multiple of eight, where before it was exact only for an even number of them. Migration: a `SEQ` parameter's `length` is now 8 where it was 12, and every other parameter's moves likewise, so code comparing stored pcapkit output byte for byte, or asserting on `Data_*Parameter.length`, sees different values -- the RFC's values. `examples/generators/options.py`'s `HIP_COPIES` stays at two, no longer for this reason but for `R1_COUNTER`'s four-octet `counter` where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires eight, which this defect had been masking (#651). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 6c2aa3457..8b4118801 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1506,6 +1506,47 @@ pull requests between #326 and #509. first -- so the round trip stopped raising and started quietly emitting a different parameter, which is why they were worth fixing together (#653, #654, #655). +* **Fixed** -- every HIP parameter pcapkit emitted was ``4 (mod 8)`` octets long, + because the padding was computed to align the *contents* rather than the record. + **This changes the octets of every HIP parameter written, and the ``length`` + every parameter's data model reports.** :rfc:`7401#section-5.2.1` requires that + "all of the encoded TLV parameters have a length (that includes the Type and + Length fields), which is a multiple of 8 bytes", and states the arithmetic + outright as ``Total Length = 11 + Length - (Length + 3) % 8``; all 95 padding + sites instead computed ``(8 - (Length % 8)) % 8``, which has no ``+ 4`` inside + the modulus and so aligns the contents alone. Across every ``Length`` from 0 to + 63 the result was never a multiple of eight and never the value the RFC gives, + and it erred in both directions: at ``Length = 4`` -- a whole ``SEQ``, and + :rfc:`7401#section-5.3.5` puts a ``SEQ`` or an ``ACK`` on every ``UPDATE`` -- + the record is complete in eight octets and pcapkit appended four that must not + be there, while at ``Length = 8`` the contents were already 8-aligned, nothing + was appended, and the record went out four octets short. A conformant peer + reading ``Length`` and consuming ``11 + Length - (Length + 3) % 8`` octets + therefore lands mid-parameter and reads the rest of the parameter area at a + wrong offset, in both directions; pcapkit did not, because its reader consumed + the same wrong count its writer wrote, which is why no round-trip test in the + suite could see this and why the fix is asserted against the RFC's arithmetic + rather than against a round trip. The 46 ``PaddingField`` callbacks in + ``pcapkit/protocols/schema/internet/hip.py`` and the 49 record lengths in + ``pcapkit/protocols/internet/hip.py`` are now one ``parameter_total_len`` and one + ``parameter_padding_len``, stating the RFC formula once instead of 95 times + (#651). ``EncryptedParameter``'s ``data`` length callback is fixed in the same + change and could not have been left: it subtracted the sixteen ``iv`` octets but + not the four ``reserved`` ones, and those four cancelled the padding's four at + ``Length % 8`` in ``{0, 5, 6, 7}`` -- so correcting the padding alone would have + taken ``ENCRYPTED`` from right at four of the eight residues to four octets too + long at all eight. ``HIP.make``'s ``len = total_length // 8 + 4`` is *not* part + of the defect and is unchanged: :rfc:`7401#section-5.1.3` defines Header Length + as the header and parameters "in 8-byte units, excluding the first 8 bytes", so + the floor division is exact once each parameter is a multiple of eight, where + before it was exact only for an even number of them. Migration: a ``SEQ`` + parameter's ``length`` is now 8 where it was 12, and every other parameter's + moves likewise, so code comparing stored pcapkit output byte for byte, or + asserting on ``Data_*Parameter.length``, sees different values -- the RFC's + values. ``examples/generators/options.py``'s ``HIP_COPIES`` stays at two, no + longer for this reason but for ``R1_COUNTER``'s four-octet ``counter`` where + :rfc:`7401#section-5.2.3` requires eight, which this defect had been masking + (#651). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 367b6e65bcbd734372c26ffca28007053c198503 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 14:47:27 -0400 Subject: [PATCH 08/46] docs(changelog): the 1.5.0 entry for #661, whose code ships in #667 The bullet #667 would otherwise have carried, kept here so that #667 touches only `pcapkit/corekit/multidict.py` and `tests/corekit/test_multidict.py`. One bullet, because it is one convention gap in one class: `_Missing` behind `MultiDict.pop` and `OrderedMultiDict.pop` lacked the `@final` and the falsy `__bool__` that `NoValueType` in `pcapkit.corekit.fields.field` sets as the package's convention for a marker of this kind. The bullet says plainly that no behaviour changes, and says why rather than asserting it: both `pop()` implementations decide by identity, never by truthiness, and `pop()` structurally cannot return the marker -- it returns `default` only on the branch where `default is not _missing`. It also names the one way the old truthiness was observable, which is what justifies touching it at all: `inspect.signature(MultiDict.pop).parameters['default'].default` hands the marker to any caller who asks, and `if default:` on it reported "a default was supplied" where none had been. It closes by recording the disposition of the other two sites from the #640 sweep, so the entry is the whole story: site 1 needed nothing, and `_NOT_FOUND` in `pcapkit.utilities.compat` stays a bare `object()` deliberately, being a verbatim line of CPython's `functools.cached_property` inside a `sys.version_info < (3, 8)` branch no supported interpreter reaches. The reasoning behind that one is on #661, not here. `:obj:` roles had to come out: `util/changelog_md.py` rejects them with `ResidualMarkupError`, since its six conversion rules do not cover interpreted text and `CHANGELOG.md` would carry the role through as literal text. Double backticks instead, which is what the rest of the entry file uses. 20 lines added to the entry file; `CHANGELOG.md` regenerated with `util/changelog_md.py`, not edited. `--check` exits 0 and `tests/project/` is green at 96 passed, 469 subtests. Committed from a detached HEAD on e55ba367c and pushed to the branch ref, because `docs/changelog-1.5.0` is checked out in another agent's worktree at a stale f84652303 and could not be taken here. Note e55ba367c, not the 69a6e1300 I was given: the branch had already moved on with #665's and #651's entries. Refs #661 --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 71e5eedb6..f5b18ed34 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -93,6 +93,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Changed** -- building a protocol through its constructor with a keyword that names nothing now raises `UnsupportedCall` instead of discarding it. **This is a behaviour change to a public API**: every `make` in the tree ends its signature with `**kwargs` and reads nothing out of it, so until now a misspelled keyword was accepted, dropped, and the field it named kept its default -- wrong octets, with nothing said. That is what #602 cost: `examples/generators/options.py` asked for `seq=1` where `TCP.make` spells the parameter `seq_no`, and 25 generated fixture frames carried sequence number `0` against an empty `warnings` list. #541 and #556 were the same silence. The schema layer has never been so permissive -- `Schema.__update__` warns `UnknownFieldWarning` for a field it does not know -- and the asymmetry between the two halves of the same construction is what this closes. Checked in `ProtocolBase.__init__` rather than in `make`, because `make` is not the only consumer of the keywords it is handed: `__post_init__` passes one `**kwargs` to the construction *and* to the parse of what it has just constructed, so a keyword declared only by `read` legitimately travels through `make` -- `HIP.read` declares `extension` where `HIP.make` does not, and the option generator depends on it. The accepted set is therefore the union of every keyword-taking parameter of `make`, `read`, `pack`, `unpack`, `__post_init__` and `__init__` across the whole MRO, computed once per class from `inspect.signature`. Parsing is deliberately untouched, since there the keywords are whatever the engines and the four `_import_next_layer` implementations forward and a protocol cannot know which of its ancestors' its parent passed on -- and nothing was ever lost that way, a dropped parse keyword changing how a packet is read rather than what its octets say. Two escapes exist for the shapes a signature cannot express, both opt-in per class through a new `__keywords__`: a set, for a keyword read out of `**kwargs` by name as `ESP.read` does with `packet`; and `None`, for a dispatcher whose real signature belongs to a class chosen at call time, which is exactly `HTTP.make` forwarding to `HTTPv1`/`HTTPv2` and the one place in the tree that uses it. The message names the near neighbour it found, so `seq` reports *did you mean 'seq_no'?*. `from_data` warns `UnknownFieldWarning` where a caller would be raised at, because the keywords there are whatever `_make_data` returned rather than anything anybody typed, so the defect is a key of that mapping disagreeing with the signature it is spread into and the person who meets it is not the person who can fix it. Expect this to surface latent bugs in code that has been quietly losing a field, which is the point. It surfaced four in this repository, all the residue of #602 and all fixed here: the `_TCP_BASE` of `examples/generators/dispatch.py` and three stale copies of it under `tests/protocols/transport/`, each still passing `seq`/`ack_flag`/`urgent_pointer` and so building segments with sequence number `0` where they read as `1`. It surfaced three more that are reported rather than fixed, being a defect per protocol rather than one in this mechanism: `Frame._make_data` returns `ts_src` where `make` declares `ts_sec`, `L2TPv2._make_data` returns `prio` where it declares `priority`, and `Header._make_data` returns a `magic_number` that `Header.make` does not take at all -- so `from_data` has been dropping a frame's timestamp, an L2TPv2 priority bit and a capture's byte order, and now says so. One limitation worth knowing rather than discovering: a *direct* `SomeProtocol.make(...)` call is not checked and still discards in silence, since the check sits where every producer's keywords converge rather than inside each of the 30 `make` implementations -- `object.__new__(cls).make(**kwargs)` is the idiom that reaches it, and `HTTP.make` uses it to reach its versioned implementation (#617). - **Fixed** -- the HIP `PUZZLE` and `SOLUTION` builders derived three wire-format values from the payload value instead of taking them from the data model, and all three were wrong. **This changes two public data models and the octets both parameters emit.** The width of `Random #I`, and of `Puzzle solution #J`, came from `int.bit_length()` alone, and nothing else was available to derive it from, so every leading zero octet was dropped on re-serialisation: a `SOLUTION` read with `Length = 20` rebuilt as `Length = 6`, a `PUZZLE` read with `Length = 12` as `Length = 5`, and silently, because the integers survive and nothing raises. That width is `RHASH_len / 8` octets [[RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4), [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5)], a property of the Responder's HIT Suite rather than of the number that happens to sit in the field, so both data models now carry it as `rhash_len` and both builders prefer it; a new `rhash_len=` keyword states it for a build from scratch, which under HIPv2 is the only place it can come from, since `RSA,DSA/SHA-256` is the REQUIRED HIT Suite [[RFC 7401 Section 5.2.10](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.10)] and makes the field 32 octets rather than 8. On the mandatory path roughly one parameter in 256 has a zero top octet and lost it (#653). `SOLUTION`'s second contents octet is `Reserved`, "zero when sent, ignored when received" [[RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5), and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) identically] -- not the `Lifetime` that only [RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4) defines, and that pcapkit was encoding there as `2^(value - 32)` seconds. It wrote `0x20`, `0x21`, `0x25` or `0x2b` into a field the RFC requires to be zero, and could not write that zero at all: an RFC-conformant `SOLUTION` whose `Reserved` is `0x00` parsed to `timedelta(0)` and then could not be re-serialised, escaping a bare `ValueError` from `math.log2(0.0)` that was not a `BaseError` and so bypassed the library's own error handling entirely. The field is renamed `reserved`, defaults to the mandated zero, and is carried verbatim across a round trip rather than re-derived (#654). And neither builder read its own `version` keyword, so `version=1` and `version=2` computed identical lengths at every bit width, where [RFC 5201 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.4) and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) state both fields as literally 8 bytes and `Length` as literally 12 and 20. Under HIPv1 each builder therefore accepted only a `bit_length()` of 57..64 and built, for everything narrower, a parameter this library's own reader rejects -- the same shape as #608, in the `PUZZLE` builder that #629 never opened (#655). The two remaining `math.log2` sites, both in `PUZZLE` where a lifetime is real, now raise `ProtocolError` rather than letting `ValueError` escape; `ProtocolError` is `(BaseError, ValueError)`, so a caller written around the old bare exception still catches it. Migration: `SolutionParameter.lifetime` is now `reserved` and an `int` rather than a `timedelta`; both `PuzzleParameter` and `SolutionParameter` gained a required `rhash_len`; and `_make_param_solution` no longer takes `lifetime=`. All three were only reachable end to end once #608 was fixed in #629, which removed the parity guard that had been failing these rebuilds loudly first -- so the round trip stopped raising and started quietly emitting a different parameter, which is why they were worth fixing together (#653, #654, #655). - **Fixed** -- every HIP parameter pcapkit emitted was `4 (mod 8)` octets long, because the padding was computed to align the *contents* rather than the record. **This changes the octets of every HIP parameter written, and the `length` every parameter's data model reports.** [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1) requires that "all of the encoded TLV parameters have a length (that includes the Type and Length fields), which is a multiple of 8 bytes", and states the arithmetic outright as `Total Length = 11 + Length - (Length + 3) % 8`; all 95 padding sites instead computed `(8 - (Length % 8)) % 8`, which has no `+ 4` inside the modulus and so aligns the contents alone. Across every `Length` from 0 to 63 the result was never a multiple of eight and never the value the RFC gives, and it erred in both directions: at `Length = 4` -- a whole `SEQ`, and [RFC 7401 Section 5.3.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.3.5) puts a `SEQ` or an `ACK` on every `UPDATE` -- the record is complete in eight octets and pcapkit appended four that must not be there, while at `Length = 8` the contents were already 8-aligned, nothing was appended, and the record went out four octets short. A conformant peer reading `Length` and consuming `11 + Length - (Length + 3) % 8` octets therefore lands mid-parameter and reads the rest of the parameter area at a wrong offset, in both directions; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test in the suite could see this and why the fix is asserted against the RFC's arithmetic rather than against a round trip. The 46 `PaddingField` callbacks in `pcapkit/protocols/schema/internet/hip.py` and the 49 record lengths in `pcapkit/protocols/internet/hip.py` are now one `parameter_total_len` and one `parameter_padding_len`, stating the RFC formula once instead of 95 times (#651). `EncryptedParameter`'s `data` length callback is fixed in the same change and could not have been left: it subtracted the sixteen `iv` octets but not the four `reserved` ones, and those four cancelled the padding's four at `Length % 8` in `{0, 5, 6, 7}` -- so correcting the padding alone would have taken `ENCRYPTED` from right at four of the eight residues to four octets too long at all eight. `HIP.make`'s `len = total_length // 8 + 4` is *not* part of the defect and is unchanged: [RFC 7401 Section 5.1.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.1.3) defines Header Length as the header and parameters "in 8-byte units, excluding the first 8 bytes", so the floor division is exact once each parameter is a multiple of eight, where before it was exact only for an even number of them. Migration: a `SEQ` parameter's `length` is now 8 where it was 12, and every other parameter's moves likewise, so code comparing stored pcapkit output byte for byte, or asserting on `Data_*Parameter.length`, sees different values -- the RFC's values. `examples/generators/options.py`'s `HIP_COPIES` stays at two, no longer for this reason but for `R1_COUNTER`'s four-octet `counter` where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires eight, which this defect had been masking (#651). +- **Fixed** -- the `_missing` sentinel behind `MultiDict.pop` and `OrderedMultiDict.pop` answered `True` to `bool()`, stating the opposite of what it means, and its class was subclassable. Both now follow `NoValueType` in `pcapkit.corekit.fields.field`, the package's convention for a marker of this kind: `_Missing` carries `@final` and defines `__bool__` returning `False`. The marker is not private in practice -- it is the runtime default of both `pop()` methods, so `inspect.signature(MultiDict.pop).parameters['default'].default` hands it to any caller who asks, and `if default:` on that value reported "a default was supplied" where none had been. No behaviour changes: both `pop()` implementations decide by identity (`default is not _missing`) and never by truthiness, so `repr`, `__reduce__`, the pickle bytes at every protocol, and every `pop()` result -- including the falsy defaults `None`, `False`, `0`, `''` and `[]` -- are unchanged, and `pop()` structurally cannot return the marker itself. This closes the second of the three `object()` sentinel sites the sweep for #640 left open; the first needed nothing, and `_NOT_FOUND` in `pcapkit.utilities.compat` is deliberately left alone as a verbatim line of CPython's own `functools.cached_property`, inside a `sys.version_info < (3, 8)` branch no supported interpreter reaches (#661). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 8b4118801..2c7b35e87 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1547,6 +1547,25 @@ pull requests between #326 and #509. longer for this reason but for ``R1_COUNTER``'s four-octet ``counter`` where :rfc:`7401#section-5.2.3` requires eight, which this defect had been masking (#651). +* **Fixed** -- the ``_missing`` sentinel behind ``MultiDict.pop`` and + ``OrderedMultiDict.pop`` answered ``True`` to ``bool()``, stating the + opposite of what it means, and its class was subclassable. Both now follow + ``NoValueType`` in ``pcapkit.corekit.fields.field``, the package's convention + for a marker of this kind: ``_Missing`` carries ``@final`` and defines + ``__bool__`` returning ``False``. The marker is not private in practice -- + it is the runtime default of both ``pop()`` methods, so + ``inspect.signature(MultiDict.pop).parameters['default'].default`` hands it to + any caller who asks, and ``if default:`` on that value reported "a default was + supplied" where none had been. No behaviour changes: both ``pop()`` + implementations decide by identity (``default is not _missing``) and never by + truthiness, so ``repr``, ``__reduce__``, the pickle bytes at every protocol, + and every ``pop()`` result -- including the falsy defaults ``None``, ``False``, + ``0``, ``''`` and ``[]`` -- are unchanged, and ``pop()`` structurally cannot + return the marker itself. This closes the second of the three ``object()`` + sentinel sites the sweep for #640 left open; the first needed nothing, and + ``_NOT_FOUND`` in ``pcapkit.utilities.compat`` is deliberately left alone as a + verbatim line of CPython's own ``functools.cached_property``, inside a + ``sys.version_info < (3, 8)`` branch no supported interpreter reaches (#661). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From d14577d34714e968737f7b8f976e2f75c2089ff9 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 14:49:53 -0400 Subject: [PATCH 09/46] docs(changelog): the 1.5.0 entry for #652 and #650, whose code ships in #669 One bullet, because it is one round trip with a defect on each side of it, in the same two files: `_make_http_data` never read `frame.flags` on the construct side, and `FrameType.post_process` seeded its accumulator with a bare `0` on the parse side. The bullet leads with what changes rather than with the mechanism, since both halves alter output: the reconstructed DATA frame's flags octet, and the dumped `__value__` of a flagless frame. It says why #650 was worth fixing at all, which its issue had left as an open question -- the dump rendered `__value__` as a JSON number for a flagless frame and a JSON string for every other frame in the same capture, so the fix removes a type inconsistency rather than introducing one. It also records two things a reader would otherwise be surprised by. The seed is guarded rather than unconditional, because `FrameType.Flags` has no members and a memberless `enum.Flag` subclass refuses `Flags(0)` -- the one-token fix the issue proposed would have crashed six of the twelve frame schemas. And a DATA round trip is still lossy after this, for the unrelated mis-parenthesised length callbacks filed as #668, so the entry does not let the reader infer a clean round trip that does not exist yet. The `TypeError` message had to sit on one line: `util/changelog_md.py` rejects a `` literal spanning a line break with `ResidualMarkupError`, since its six conversion rules do not cover it. 37 lines added to the entry file; `CHANGELOG.md` regenerated with `util/changelog_md.py`, not edited. `--check` exits 0 and `tests/project/` is green at 96 passed, 469 subtests. Committed from a detached HEAD on 367b6e65b and pushed to the branch ref, because `docs/changelog-1.5.0` is checked out in another agent's worktree and could not be taken here. Note 367b6e65b, not the 69a6e1300 I was given: the branch had already moved on with #665's, #651's and #661's entries. Refs #652 Refs #650 --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 35 +++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f5b18ed34..1a6d4f70a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -94,6 +94,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- the HIP `PUZZLE` and `SOLUTION` builders derived three wire-format values from the payload value instead of taking them from the data model, and all three were wrong. **This changes two public data models and the octets both parameters emit.** The width of `Random #I`, and of `Puzzle solution #J`, came from `int.bit_length()` alone, and nothing else was available to derive it from, so every leading zero octet was dropped on re-serialisation: a `SOLUTION` read with `Length = 20` rebuilt as `Length = 6`, a `PUZZLE` read with `Length = 12` as `Length = 5`, and silently, because the integers survive and nothing raises. That width is `RHASH_len / 8` octets [[RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4), [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5)], a property of the Responder's HIT Suite rather than of the number that happens to sit in the field, so both data models now carry it as `rhash_len` and both builders prefer it; a new `rhash_len=` keyword states it for a build from scratch, which under HIPv2 is the only place it can come from, since `RSA,DSA/SHA-256` is the REQUIRED HIT Suite [[RFC 7401 Section 5.2.10](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.10)] and makes the field 32 octets rather than 8. On the mandatory path roughly one parameter in 256 has a zero top octet and lost it (#653). `SOLUTION`'s second contents octet is `Reserved`, "zero when sent, ignored when received" [[RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5), and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) identically] -- not the `Lifetime` that only [RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4) defines, and that pcapkit was encoding there as `2^(value - 32)` seconds. It wrote `0x20`, `0x21`, `0x25` or `0x2b` into a field the RFC requires to be zero, and could not write that zero at all: an RFC-conformant `SOLUTION` whose `Reserved` is `0x00` parsed to `timedelta(0)` and then could not be re-serialised, escaping a bare `ValueError` from `math.log2(0.0)` that was not a `BaseError` and so bypassed the library's own error handling entirely. The field is renamed `reserved`, defaults to the mandated zero, and is carried verbatim across a round trip rather than re-derived (#654). And neither builder read its own `version` keyword, so `version=1` and `version=2` computed identical lengths at every bit width, where [RFC 5201 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.4) and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) state both fields as literally 8 bytes and `Length` as literally 12 and 20. Under HIPv1 each builder therefore accepted only a `bit_length()` of 57..64 and built, for everything narrower, a parameter this library's own reader rejects -- the same shape as #608, in the `PUZZLE` builder that #629 never opened (#655). The two remaining `math.log2` sites, both in `PUZZLE` where a lifetime is real, now raise `ProtocolError` rather than letting `ValueError` escape; `ProtocolError` is `(BaseError, ValueError)`, so a caller written around the old bare exception still catches it. Migration: `SolutionParameter.lifetime` is now `reserved` and an `int` rather than a `timedelta`; both `PuzzleParameter` and `SolutionParameter` gained a required `rhash_len`; and `_make_param_solution` no longer takes `lifetime=`. All three were only reachable end to end once #608 was fixed in #629, which removed the parity guard that had been failing these rebuilds loudly first -- so the round trip stopped raising and started quietly emitting a different parameter, which is why they were worth fixing together (#653, #654, #655). - **Fixed** -- every HIP parameter pcapkit emitted was `4 (mod 8)` octets long, because the padding was computed to align the *contents* rather than the record. **This changes the octets of every HIP parameter written, and the `length` every parameter's data model reports.** [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1) requires that "all of the encoded TLV parameters have a length (that includes the Type and Length fields), which is a multiple of 8 bytes", and states the arithmetic outright as `Total Length = 11 + Length - (Length + 3) % 8`; all 95 padding sites instead computed `(8 - (Length % 8)) % 8`, which has no `+ 4` inside the modulus and so aligns the contents alone. Across every `Length` from 0 to 63 the result was never a multiple of eight and never the value the RFC gives, and it erred in both directions: at `Length = 4` -- a whole `SEQ`, and [RFC 7401 Section 5.3.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.3.5) puts a `SEQ` or an `ACK` on every `UPDATE` -- the record is complete in eight octets and pcapkit appended four that must not be there, while at `Length = 8` the contents were already 8-aligned, nothing was appended, and the record went out four octets short. A conformant peer reading `Length` and consuming `11 + Length - (Length + 3) % 8` octets therefore lands mid-parameter and reads the rest of the parameter area at a wrong offset, in both directions; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test in the suite could see this and why the fix is asserted against the RFC's arithmetic rather than against a round trip. The 46 `PaddingField` callbacks in `pcapkit/protocols/schema/internet/hip.py` and the 49 record lengths in `pcapkit/protocols/internet/hip.py` are now one `parameter_total_len` and one `parameter_padding_len`, stating the RFC formula once instead of 95 times (#651). `EncryptedParameter`'s `data` length callback is fixed in the same change and could not have been left: it subtracted the sixteen `iv` octets but not the four `reserved` ones, and those four cancelled the padding's four at `Length % 8` in `{0, 5, 6, 7}` -- so correcting the padding alone would have taken `ENCRYPTED` from right at four of the eight residues to four octets too long at all eight. `HIP.make`'s `len = total_length // 8 + 4` is *not* part of the defect and is unchanged: [RFC 7401 Section 5.1.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.1.3) defines Header Length as the header and parameters "in 8-byte units, excluding the first 8 bytes", so the floor division is exact once each parameter is a multiple of eight, where before it was exact only for an even number of them. Migration: a `SEQ` parameter's `length` is now 8 where it was 12, and every other parameter's moves likewise, so code comparing stored pcapkit output byte for byte, or asserting on `Data_*Parameter.length`, sees different values -- the RFC's values. `examples/generators/options.py`'s `HIP_COPIES` stays at two, no longer for this reason but for `R1_COUNTER`'s four-octet `counter` where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires eight, which this defect had been masking (#651). - **Fixed** -- the `_missing` sentinel behind `MultiDict.pop` and `OrderedMultiDict.pop` answered `True` to `bool()`, stating the opposite of what it means, and its class was subclassable. Both now follow `NoValueType` in `pcapkit.corekit.fields.field`, the package's convention for a marker of this kind: `_Missing` carries `@final` and defines `__bool__` returning `False`. The marker is not private in practice -- it is the runtime default of both `pop()` methods, so `inspect.signature(MultiDict.pop).parameters['default'].default` hands it to any caller who asks, and `if default:` on that value reported "a default was supplied" where none had been. No behaviour changes: both `pop()` implementations decide by identity (`default is not _missing`) and never by truthiness, so `repr`, `__reduce__`, the pickle bytes at every protocol, and every `pop()` result -- including the falsy defaults `None`, `False`, `0`, `''` and `[]` -- are unchanged, and `pop()` structurally cannot return the marker itself. This closes the second of the three `object()` sentinel sites the sweep for #640 left open; the first needed nothing, and `_NOT_FOUND` in `pcapkit.utilities.compat` is deliberately left alone as a verbatim line of CPython's own `functools.cached_property`, inside a `sys.version_info < (3, 8)` branch no supported interpreter reaches (#661). +- **Fixed** -- two HTTP/2 flag defects, one on each side of a round trip. **This changes the octets a reconstructed DATA frame emits, and the dumped value of a flagless frame's flags.** `_make_http_data` was the only one of the six `_make_http_*` methods that never read `frame.flags`, so a DATA frame parsed with `END_STREAM` set rebuilt with the bit clear: the flags octet went `0x01` to `0x00`, silently, producing a well-formed frame saying the stream continues where the capture said it ended [[RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1)]. `PADDED` was never affected, being re-derived from `pad_len` rather than read back, and all five sibling builders already restored theirs -- so this was a missing read-back rather than a design, and it now reads `frame.flags.END_STREAM` the way they do (#652). Separately, `FrameType.post_process` seeded its flag accumulator with a bare `0`, and `|=` promotes that only as a side effect, so a frame with **no** bit set left `__flags__` a plain `int` where both the schema and the data model declare a `Flags` -- and `END_STREAM in __value__` raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering `False`. That is the common case rather than an edge one: a `0x00` flags octet is every opening `SETTINGS`, every non-final `DATA` and every non-ACK `PING`. It now seeds `self.Flags(0)`, which the construct path had been doing correctly at all six of its own sites. The visible consequence is in the dump, where `__value__` was a JSON *number* for a flagless frame and a JSON *string* for every other frame in the same capture; it is consistently a string now, `"Flags::None [0]"` where it read `0`, the same shape of change #616 made to TCP's `connection`. The seed is guarded rather than unconditional because `FrameType.Flags` declares no members and a memberless `enum.Flag` subclass refuses `Flags(0)` outright, so the five frame schemas that inherit it -- `UnassignedFrame`, `PriorityFrame`, `RSTStreamFrame`, `GoawayFrame` and `WindowUpdateFrame` -- keep the plain `int`, which nothing observes since all five pass `flags=None` into their data objects. Note that a DATA round trip is **still** lossy after this, for an unrelated reason found while measuring it and left to its own change: three payload length callbacks in the frame schemas are mis-parenthesised, so an unpadded `DATA`, `HEADERS` or `PUSH_PROMISE` frame parses with its whole payload dropped (#668). (#652, #650). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 2c7b35e87..6dafbc742 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1566,6 +1566,41 @@ pull requests between #326 and #509. ``_NOT_FOUND`` in ``pcapkit.utilities.compat`` is deliberately left alone as a verbatim line of CPython's own ``functools.cached_property``, inside a ``sys.version_info < (3, 8)`` branch no supported interpreter reaches (#661). +* **Fixed** -- two HTTP/2 flag defects, one on each side of a round trip. + **This changes the octets a reconstructed DATA frame emits, and the dumped + value of a flagless frame's flags.** ``_make_http_data`` was the only one of + the six ``_make_http_*`` methods that never read ``frame.flags``, so a DATA + frame parsed with ``END_STREAM`` set rebuilt with the bit clear: the flags + octet went ``0x01`` to ``0x00``, silently, producing a well-formed frame + saying the stream continues where the capture said it ended + [:rfc:`9113#section-6.1`]. ``PADDED`` was never affected, being re-derived + from ``pad_len`` rather than read back, and all five sibling builders already + restored theirs -- so this was a missing read-back rather than a design, and + it now reads ``frame.flags.END_STREAM`` the way they do (#652). Separately, + ``FrameType.post_process`` seeded its flag accumulator with a bare ``0``, and + ``|=`` promotes that only as a side effect, so a frame with **no** bit set + left ``__flags__`` a plain ``int`` where both the schema and the data model + declare a ``Flags`` -- and ``END_STREAM in __value__`` raised + ``TypeError: argument of type 'int' is not a container or iterable`` rather + than answering ``False``. That is the common case rather than an edge one: a + ``0x00`` flags + octet is every opening ``SETTINGS``, every non-final ``DATA`` and every + non-ACK ``PING``. It now seeds ``self.Flags(0)``, which the construct path had + been doing correctly at all six of its own sites. The visible consequence is + in the dump, where ``__value__`` was a JSON *number* for a flagless frame and + a JSON *string* for every other frame in the same capture; it is consistently + a string now, ``"Flags::None [0]"`` where it read ``0``, the same shape of + change #616 made to TCP's ``connection``. The seed is guarded rather than + unconditional because ``FrameType.Flags`` declares no members and a + memberless ``enum.Flag`` subclass refuses ``Flags(0)`` outright, so the five + frame schemas that inherit it -- ``UnassignedFrame``, ``PriorityFrame``, + ``RSTStreamFrame``, ``GoawayFrame`` and ``WindowUpdateFrame`` -- keep the + plain ``int``, which nothing observes since all five pass ``flags=None`` into + their data objects. Note that a DATA round trip is **still** lossy after + this, for an unrelated reason found while measuring it and left to its own + change: three payload length callbacks in the frame schemas are + mis-parenthesised, so an unpadded ``DATA``, ``HEADERS`` or ``PUSH_PROMISE`` + frame parses with its whole payload dropped (#668). (#652, #650). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 6a956c4781d1507b1be4799d2dea3f2c5a78fbd3 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 14:54:48 -0400 Subject: [PATCH 10/46] docs(changelog): the 1.5.0 entries for #648 and #649, whose code ships in #670 and #671 Two bullets, not one, because the two defects are unrelated: one changes what the dumpers emit, the other only the text of four exception messages. They share a file only by accident of being found in the same pass. The #648 bullet leads with the output change and says so in bold, because that is what a reader upgrading needs to see: a flag value with no declared bits dumped as `Type::None [0]` in all six textual format names, out of both `Extractor` and `TraceFlow`. It then justifies the *replacement* rather than just stating it, since "render it as its decimal value" looks arbitrary until you know the enumeration libraries already spell an undeclared residue that way -- and that a decimal cannot collide with a member name where `None` can, `NONE` being a real declared name elsewhere. Three things a reader would otherwise get wrong are recorded: three sites carried the interpolation and not one, the guard is on `name is None` rather than on zero because the defect never was about zero, and it is not an `aenum` quirk since stdlib `enum.IntFlag` behaves identically. It also corrects the issue on a point of fact. #648 said `Flags` was the only registry nameless at zero; a sweep of all seven finds five, the four Mobility Header flag registries included. And it states that the committed example dumps do not move, which was measured by regenerating all three with and without the change rather than assumed -- a reader of a bullet this emphatic will otherwise wonder whether `examples/captures/` drifted. The #649 bullet says "cosmetic" in its second sentence so nobody reads it as a behavioural change, then gives the one reason it was worth doing at all: it is the text a user sees when an option is rejected. It names all four sites, and the 28-against-4 count in the same file, because that count is what makes the correct form a fact about the module rather than a preference. Neither bullet claims a guard it does not have: #648's third site, the `addon` branch, is not reachable from any registry in the library today, and the bullet does not imply otherwise. `CHANGELOG.md` regenerated with `util/changelog_md.py`, not edited by hand. `--check` exits 0 and `tests/project/` is green at 96 passed, 469 subtests. Each `` literal is kept on one line, since the generator rejects one spanning a line break with `ResidualMarkupError`. Committed from a detached HEAD on d14577d34 and pushed to the branch ref, because `docs/changelog-1.5.0` is checked out in another agent's worktree and could not be taken here. Note d14577d34, not the 69a6e1300 I was given: the branch had already moved on with #665's, #651's, #661's and #652/#650's entries. Refs #648 Refs #649 --- CHANGELOG.md | 2 ++ docs/source/changelog/1.5.0.rst | 43 +++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1a6d4f70a..86ac219a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -95,6 +95,8 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- every HIP parameter pcapkit emitted was `4 (mod 8)` octets long, because the padding was computed to align the *contents* rather than the record. **This changes the octets of every HIP parameter written, and the `length` every parameter's data model reports.** [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1) requires that "all of the encoded TLV parameters have a length (that includes the Type and Length fields), which is a multiple of 8 bytes", and states the arithmetic outright as `Total Length = 11 + Length - (Length + 3) % 8`; all 95 padding sites instead computed `(8 - (Length % 8)) % 8`, which has no `+ 4` inside the modulus and so aligns the contents alone. Across every `Length` from 0 to 63 the result was never a multiple of eight and never the value the RFC gives, and it erred in both directions: at `Length = 4` -- a whole `SEQ`, and [RFC 7401 Section 5.3.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.3.5) puts a `SEQ` or an `ACK` on every `UPDATE` -- the record is complete in eight octets and pcapkit appended four that must not be there, while at `Length = 8` the contents were already 8-aligned, nothing was appended, and the record went out four octets short. A conformant peer reading `Length` and consuming `11 + Length - (Length + 3) % 8` octets therefore lands mid-parameter and reads the rest of the parameter area at a wrong offset, in both directions; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test in the suite could see this and why the fix is asserted against the RFC's arithmetic rather than against a round trip. The 46 `PaddingField` callbacks in `pcapkit/protocols/schema/internet/hip.py` and the 49 record lengths in `pcapkit/protocols/internet/hip.py` are now one `parameter_total_len` and one `parameter_padding_len`, stating the RFC formula once instead of 95 times (#651). `EncryptedParameter`'s `data` length callback is fixed in the same change and could not have been left: it subtracted the sixteen `iv` octets but not the four `reserved` ones, and those four cancelled the padding's four at `Length % 8` in `{0, 5, 6, 7}` -- so correcting the padding alone would have taken `ENCRYPTED` from right at four of the eight residues to four octets too long at all eight. `HIP.make`'s `len = total_length // 8 + 4` is *not* part of the defect and is unchanged: [RFC 7401 Section 5.1.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.1.3) defines Header Length as the header and parameters "in 8-byte units, excluding the first 8 bytes", so the floor division is exact once each parameter is a multiple of eight, where before it was exact only for an even number of them. Migration: a `SEQ` parameter's `length` is now 8 where it was 12, and every other parameter's moves likewise, so code comparing stored pcapkit output byte for byte, or asserting on `Data_*Parameter.length`, sees different values -- the RFC's values. `examples/generators/options.py`'s `HIP_COPIES` stays at two, no longer for this reason but for `R1_COUNTER`'s four-octet `counter` where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires eight, which this defect had been masking (#651). - **Fixed** -- the `_missing` sentinel behind `MultiDict.pop` and `OrderedMultiDict.pop` answered `True` to `bool()`, stating the opposite of what it means, and its class was subclassable. Both now follow `NoValueType` in `pcapkit.corekit.fields.field`, the package's convention for a marker of this kind: `_Missing` carries `@final` and defines `__bool__` returning `False`. The marker is not private in practice -- it is the runtime default of both `pop()` methods, so `inspect.signature(MultiDict.pop).parameters['default'].default` hands it to any caller who asks, and `if default:` on that value reported "a default was supplied" where none had been. No behaviour changes: both `pop()` implementations decide by identity (`default is not _missing`) and never by truthiness, so `repr`, `__reduce__`, the pickle bytes at every protocol, and every `pop()` result -- including the falsy defaults `None`, `False`, `0`, `''` and `[]` -- are unchanged, and `pop()` structurally cannot return the marker itself. This closes the second of the three `object()` sentinel sites the sweep for #640 left open; the first needed nothing, and `_NOT_FOUND` in `pcapkit.utilities.compat` is deliberately left alone as a verbatim line of CPython's own `functools.cached_property`, inside a `sys.version_info < (3, 8)` branch no supported interpreter reaches (#661). - **Fixed** -- two HTTP/2 flag defects, one on each side of a round trip. **This changes the octets a reconstructed DATA frame emits, and the dumped value of a flagless frame's flags.** `_make_http_data` was the only one of the six `_make_http_*` methods that never read `frame.flags`, so a DATA frame parsed with `END_STREAM` set rebuilt with the bit clear: the flags octet went `0x01` to `0x00`, silently, producing a well-formed frame saying the stream continues where the capture said it ended [[RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1)]. `PADDED` was never affected, being re-derived from `pad_len` rather than read back, and all five sibling builders already restored theirs -- so this was a missing read-back rather than a design, and it now reads `frame.flags.END_STREAM` the way they do (#652). Separately, `FrameType.post_process` seeded its flag accumulator with a bare `0`, and `|=` promotes that only as a side effect, so a frame with **no** bit set left `__flags__` a plain `int` where both the schema and the data model declare a `Flags` -- and `END_STREAM in __value__` raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering `False`. That is the common case rather than an edge one: a `0x00` flags octet is every opening `SETTINGS`, every non-final `DATA` and every non-ACK `PING`. It now seeds `self.Flags(0)`, which the construct path had been doing correctly at all six of its own sites. The visible consequence is in the dump, where `__value__` was a JSON *number* for a flagless frame and a JSON *string* for every other frame in the same capture; it is consistently a string now, `"Flags::None [0]"` where it read `0`, the same shape of change #616 made to TCP's `connection`. The seed is guarded rather than unconditional because `FrameType.Flags` declares no members and a memberless `enum.Flag` subclass refuses `Flags(0)` outright, so the five frame schemas that inherit it -- `UnassignedFrame`, `PriorityFrame`, `RSTStreamFrame`, `GoawayFrame` and `WindowUpdateFrame` -- keep the plain `int`, which nothing observes since all five pass `flags=None` into their data objects. Note that a DATA round trip is **still** lossy after this, for an unrelated reason found while measuring it and left to its own change: three payload length callbacks in the frame schemas are mis-parenthesised, so an unpadded `DATA`, `HEADERS` or `PUSH_PROMISE` frame parses with its whole payload dropped (#668). (#652, #650). +- **Fixed** -- a flag value with no declared bits no longer dumps as `Type::None [0]`. **This changes user-visible output in every textual dump format.** `make_dumper`'s `object_hook` renders each enumeration member as `Type::name [value]` and interpolated the name unguarded, but a `Flag` value composed entirely of undeclared bits has `name is None` rather than a string -- so the literal four characters `None` landed in the name half and `Flags(0)` rendered as `Flags::None [0]` in `json`, `tree`, `text`, `txt`, `plist` and `xml`, out of both `Extractor` and `TraceFlow`. A nameless member now renders with the value's own decimal spelling instead, so `Flags(0)` gives `Flags::0 [0]` and `Flags(8)` gives `Flags::8 [8]`. That is the spelling the enumeration libraries already use for an undeclared residue -- `Flags(2057).name` is `ACK|9`, naming the declared bit and giving the leftovers as one decimal number -- and it cannot be mistaken for a member name, since a Python identifier may not begin with a digit, whereas `NONE` is a real declared name elsewhere in the library. Three sites carried the identical interpolation, not one: the `OrderedMultiDict` key path, the `addon` branch and the scalar return, which now share one `render_enum` helper so the guard cannot be applied to two of three. That guard is on `name is None` and not on the value, because the defect never was about zero -- `Flags(1)`, `Flags(8)`, `Flags(9)` and `Flags(65536)` are equally nameless -- and it is not an `aenum` quirk either, since a stdlib `enum.IntFlag` answers `name is None` at the same values. Five of the library's seven flag registries are nameless at zero rather than the one reported: `Flags` plus the four Mobility Header flag registries, the other two declaring an explicit `undefined = 0`. Pre-existing since 2023-04-28 and surfaced rather than caused by #634, which made a flagless TCP segment reach this path. The committed example dumps do not move, because no member in them lacks a name (#648). +- **Fixed** -- four MPTCP error messages carried a doubled separator, rendering as `TCP: : [OptNo 30] 1: invalid flags combination` with an empty field between the protocol alias and the option number. Cosmetic: the exception type, the option number and the subtype were always right, and nothing downstream parses these strings, but it is the text a user sees when an MP_JOIN or DSS option is rejected and an empty field reads as a value that failed to interpolate. Four sites, not the one reported -- `_read_mptcp_join` and `_make_mptcp_join` for the invalid flag combination, and both guards of `_make_mptcp_dss` for the missing required fields -- against 28 messages in the same file that already spelled the prefix correctly, so the four were outliers against a house form in their own module and the counts are now 32 correct against 0 doubled. Pre-existing since 2023-04-10. The new test asserts each message in full rather than by substring, which is what the existing `assertIn` on the message body could not do, since it passes either side of the change (#649). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 6dafbc742..c56daaba5 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1601,6 +1601,49 @@ pull requests between #326 and #509. change: three payload length callbacks in the frame schemas are mis-parenthesised, so an unpadded ``DATA``, ``HEADERS`` or ``PUSH_PROMISE`` frame parses with its whole payload dropped (#668). (#652, #650). +* **Fixed** -- a flag value with no declared bits no longer dumps as + ``Type::None [0]``. **This changes user-visible output in every textual dump + format.** ``make_dumper``'s ``object_hook`` renders each enumeration member as + ``Type::name [value]`` and interpolated the name unguarded, but a ``Flag`` + value composed entirely of undeclared bits has ``name is None`` rather than a + string -- so the literal four characters ``None`` landed in the name half and + ``Flags(0)`` rendered as ``Flags::None [0]`` in ``json``, ``tree``, ``text``, + ``txt``, ``plist`` and ``xml``, out of both ``Extractor`` and ``TraceFlow``. A + nameless member now renders with the value's own decimal spelling instead, so + ``Flags(0)`` gives ``Flags::0 [0]`` and ``Flags(8)`` gives ``Flags::8 [8]``. + That is the spelling the enumeration libraries already use for an undeclared + residue -- ``Flags(2057).name`` is ``ACK|9``, naming the declared bit and + giving the leftovers as one decimal number -- and it cannot be mistaken for a + member name, since a Python identifier may not begin with a digit, whereas + ``NONE`` is a real declared name elsewhere in the library. Three sites carried + the identical interpolation, not one: the ``OrderedMultiDict`` key path, the + ``addon`` branch and the scalar return, which now share one ``render_enum`` + helper so the guard cannot be applied to two of three. That guard is on + ``name is None`` and not on the value, because the defect never was about zero + -- ``Flags(1)``, ``Flags(8)``, ``Flags(9)`` and ``Flags(65536)`` are equally + nameless -- and it is not an ``aenum`` quirk either, since a stdlib + ``enum.IntFlag`` answers ``name is None`` at the same values. Five of the + library's seven flag registries are nameless at zero rather than the one + reported: ``Flags`` plus the four Mobility Header flag registries, the other + two declaring an explicit ``undefined = 0``. Pre-existing since 2023-04-28 and + surfaced rather than caused by #634, which made a flagless TCP segment reach + this path. The committed example dumps do not move, because no member in them + lacks a name (#648). +* **Fixed** -- four MPTCP error messages carried a doubled separator, rendering + as ``TCP: : [OptNo 30] 1: invalid flags combination`` with an empty field + between the protocol alias and the option number. Cosmetic: the exception + type, the option number and the subtype were always right, and nothing + downstream parses these strings, but it is the text a user sees when an + MP_JOIN or DSS option is rejected and an empty field reads as a value that + failed to interpolate. Four sites, not the one reported -- ``_read_mptcp_join`` + and ``_make_mptcp_join`` for the invalid flag combination, and both guards of + ``_make_mptcp_dss`` for the missing required fields -- against 28 messages in + the same file that already spelled the prefix correctly, so the four were + outliers against a house form in their own module and the counts are now 32 + correct against 0 doubled. Pre-existing since 2023-04-10. The new test asserts + each message in full rather than by substring, which is what the existing + ``assertIn`` on the message body could not do, since it passes either side of + the change (#649). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From b2ec64bf1d16626501c15b852504ead3ab9b74ed Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 14:58:24 -0400 Subject: [PATCH 11/46] docs(changelog): version-qualify the empty-enum claim in the #652/#650 entry The bullet added in d14577d34 said a memberless `enum.Flag` subclass "refuses `Flags(0)` outright", flatly. That is only true from Python 3.12, where the enum rewrite made `EnumType.__call__` raise for an enum with no members; earlier interpreters take the plain value-lookup path and hand back a pseudo-member. Measured on the two available here: version 3.14.7 members: 0 Flags(0) -> TypeError: has no members version 3.7.16 members: 0 Flags(0) -> OK `requires-python` is `>=3.6`, so the unqualified form overstated it. Three words added, no other change to the bullet: the guard in #669 is correct on every supported interpreter either way, being keyed on the memberless-ness rather than on the refusal. The same imprecision was corrected in #669's own source comment and PR body, and its test now gates only the `TypeError` assertion behind `sys.version_info >= (3, 12)` -- CI runs the unit tier on 3.10 through 3.15, so an unconditional `assertRaises` would have gone red on the older two. `CHANGELOG.md` regenerated with `util/changelog_md.py`, not edited; `--check` exits 0 and `tests/project/` is green at 96 passed, 469 subtests. Committed from a detached HEAD on 6a956c478 and pushed to the branch ref, because `docs/changelog-1.5.0` is checked out in another agent's worktree. The branch had moved on with #648's and #649's entries since d14577d34. Refs #652 Refs #650 --- CHANGELOG.md | 2 +- docs/source/changelog/1.5.0.rst | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 86ac219a4..7515a346c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -94,7 +94,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- the HIP `PUZZLE` and `SOLUTION` builders derived three wire-format values from the payload value instead of taking them from the data model, and all three were wrong. **This changes two public data models and the octets both parameters emit.** The width of `Random #I`, and of `Puzzle solution #J`, came from `int.bit_length()` alone, and nothing else was available to derive it from, so every leading zero octet was dropped on re-serialisation: a `SOLUTION` read with `Length = 20` rebuilt as `Length = 6`, a `PUZZLE` read with `Length = 12` as `Length = 5`, and silently, because the integers survive and nothing raises. That width is `RHASH_len / 8` octets [[RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4), [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5)], a property of the Responder's HIT Suite rather than of the number that happens to sit in the field, so both data models now carry it as `rhash_len` and both builders prefer it; a new `rhash_len=` keyword states it for a build from scratch, which under HIPv2 is the only place it can come from, since `RSA,DSA/SHA-256` is the REQUIRED HIT Suite [[RFC 7401 Section 5.2.10](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.10)] and makes the field 32 octets rather than 8. On the mandatory path roughly one parameter in 256 has a zero top octet and lost it (#653). `SOLUTION`'s second contents octet is `Reserved`, "zero when sent, ignored when received" [[RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5), and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) identically] -- not the `Lifetime` that only [RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4) defines, and that pcapkit was encoding there as `2^(value - 32)` seconds. It wrote `0x20`, `0x21`, `0x25` or `0x2b` into a field the RFC requires to be zero, and could not write that zero at all: an RFC-conformant `SOLUTION` whose `Reserved` is `0x00` parsed to `timedelta(0)` and then could not be re-serialised, escaping a bare `ValueError` from `math.log2(0.0)` that was not a `BaseError` and so bypassed the library's own error handling entirely. The field is renamed `reserved`, defaults to the mandated zero, and is carried verbatim across a round trip rather than re-derived (#654). And neither builder read its own `version` keyword, so `version=1` and `version=2` computed identical lengths at every bit width, where [RFC 5201 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.4) and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) state both fields as literally 8 bytes and `Length` as literally 12 and 20. Under HIPv1 each builder therefore accepted only a `bit_length()` of 57..64 and built, for everything narrower, a parameter this library's own reader rejects -- the same shape as #608, in the `PUZZLE` builder that #629 never opened (#655). The two remaining `math.log2` sites, both in `PUZZLE` where a lifetime is real, now raise `ProtocolError` rather than letting `ValueError` escape; `ProtocolError` is `(BaseError, ValueError)`, so a caller written around the old bare exception still catches it. Migration: `SolutionParameter.lifetime` is now `reserved` and an `int` rather than a `timedelta`; both `PuzzleParameter` and `SolutionParameter` gained a required `rhash_len`; and `_make_param_solution` no longer takes `lifetime=`. All three were only reachable end to end once #608 was fixed in #629, which removed the parity guard that had been failing these rebuilds loudly first -- so the round trip stopped raising and started quietly emitting a different parameter, which is why they were worth fixing together (#653, #654, #655). - **Fixed** -- every HIP parameter pcapkit emitted was `4 (mod 8)` octets long, because the padding was computed to align the *contents* rather than the record. **This changes the octets of every HIP parameter written, and the `length` every parameter's data model reports.** [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1) requires that "all of the encoded TLV parameters have a length (that includes the Type and Length fields), which is a multiple of 8 bytes", and states the arithmetic outright as `Total Length = 11 + Length - (Length + 3) % 8`; all 95 padding sites instead computed `(8 - (Length % 8)) % 8`, which has no `+ 4` inside the modulus and so aligns the contents alone. Across every `Length` from 0 to 63 the result was never a multiple of eight and never the value the RFC gives, and it erred in both directions: at `Length = 4` -- a whole `SEQ`, and [RFC 7401 Section 5.3.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.3.5) puts a `SEQ` or an `ACK` on every `UPDATE` -- the record is complete in eight octets and pcapkit appended four that must not be there, while at `Length = 8` the contents were already 8-aligned, nothing was appended, and the record went out four octets short. A conformant peer reading `Length` and consuming `11 + Length - (Length + 3) % 8` octets therefore lands mid-parameter and reads the rest of the parameter area at a wrong offset, in both directions; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test in the suite could see this and why the fix is asserted against the RFC's arithmetic rather than against a round trip. The 46 `PaddingField` callbacks in `pcapkit/protocols/schema/internet/hip.py` and the 49 record lengths in `pcapkit/protocols/internet/hip.py` are now one `parameter_total_len` and one `parameter_padding_len`, stating the RFC formula once instead of 95 times (#651). `EncryptedParameter`'s `data` length callback is fixed in the same change and could not have been left: it subtracted the sixteen `iv` octets but not the four `reserved` ones, and those four cancelled the padding's four at `Length % 8` in `{0, 5, 6, 7}` -- so correcting the padding alone would have taken `ENCRYPTED` from right at four of the eight residues to four octets too long at all eight. `HIP.make`'s `len = total_length // 8 + 4` is *not* part of the defect and is unchanged: [RFC 7401 Section 5.1.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.1.3) defines Header Length as the header and parameters "in 8-byte units, excluding the first 8 bytes", so the floor division is exact once each parameter is a multiple of eight, where before it was exact only for an even number of them. Migration: a `SEQ` parameter's `length` is now 8 where it was 12, and every other parameter's moves likewise, so code comparing stored pcapkit output byte for byte, or asserting on `Data_*Parameter.length`, sees different values -- the RFC's values. `examples/generators/options.py`'s `HIP_COPIES` stays at two, no longer for this reason but for `R1_COUNTER`'s four-octet `counter` where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires eight, which this defect had been masking (#651). - **Fixed** -- the `_missing` sentinel behind `MultiDict.pop` and `OrderedMultiDict.pop` answered `True` to `bool()`, stating the opposite of what it means, and its class was subclassable. Both now follow `NoValueType` in `pcapkit.corekit.fields.field`, the package's convention for a marker of this kind: `_Missing` carries `@final` and defines `__bool__` returning `False`. The marker is not private in practice -- it is the runtime default of both `pop()` methods, so `inspect.signature(MultiDict.pop).parameters['default'].default` hands it to any caller who asks, and `if default:` on that value reported "a default was supplied" where none had been. No behaviour changes: both `pop()` implementations decide by identity (`default is not _missing`) and never by truthiness, so `repr`, `__reduce__`, the pickle bytes at every protocol, and every `pop()` result -- including the falsy defaults `None`, `False`, `0`, `''` and `[]` -- are unchanged, and `pop()` structurally cannot return the marker itself. This closes the second of the three `object()` sentinel sites the sweep for #640 left open; the first needed nothing, and `_NOT_FOUND` in `pcapkit.utilities.compat` is deliberately left alone as a verbatim line of CPython's own `functools.cached_property`, inside a `sys.version_info < (3, 8)` branch no supported interpreter reaches (#661). -- **Fixed** -- two HTTP/2 flag defects, one on each side of a round trip. **This changes the octets a reconstructed DATA frame emits, and the dumped value of a flagless frame's flags.** `_make_http_data` was the only one of the six `_make_http_*` methods that never read `frame.flags`, so a DATA frame parsed with `END_STREAM` set rebuilt with the bit clear: the flags octet went `0x01` to `0x00`, silently, producing a well-formed frame saying the stream continues where the capture said it ended [[RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1)]. `PADDED` was never affected, being re-derived from `pad_len` rather than read back, and all five sibling builders already restored theirs -- so this was a missing read-back rather than a design, and it now reads `frame.flags.END_STREAM` the way they do (#652). Separately, `FrameType.post_process` seeded its flag accumulator with a bare `0`, and `|=` promotes that only as a side effect, so a frame with **no** bit set left `__flags__` a plain `int` where both the schema and the data model declare a `Flags` -- and `END_STREAM in __value__` raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering `False`. That is the common case rather than an edge one: a `0x00` flags octet is every opening `SETTINGS`, every non-final `DATA` and every non-ACK `PING`. It now seeds `self.Flags(0)`, which the construct path had been doing correctly at all six of its own sites. The visible consequence is in the dump, where `__value__` was a JSON *number* for a flagless frame and a JSON *string* for every other frame in the same capture; it is consistently a string now, `"Flags::None [0]"` where it read `0`, the same shape of change #616 made to TCP's `connection`. The seed is guarded rather than unconditional because `FrameType.Flags` declares no members and a memberless `enum.Flag` subclass refuses `Flags(0)` outright, so the five frame schemas that inherit it -- `UnassignedFrame`, `PriorityFrame`, `RSTStreamFrame`, `GoawayFrame` and `WindowUpdateFrame` -- keep the plain `int`, which nothing observes since all five pass `flags=None` into their data objects. Note that a DATA round trip is **still** lossy after this, for an unrelated reason found while measuring it and left to its own change: three payload length callbacks in the frame schemas are mis-parenthesised, so an unpadded `DATA`, `HEADERS` or `PUSH_PROMISE` frame parses with its whole payload dropped (#668). (#652, #650). +- **Fixed** -- two HTTP/2 flag defects, one on each side of a round trip. **This changes the octets a reconstructed DATA frame emits, and the dumped value of a flagless frame's flags.** `_make_http_data` was the only one of the six `_make_http_*` methods that never read `frame.flags`, so a DATA frame parsed with `END_STREAM` set rebuilt with the bit clear: the flags octet went `0x01` to `0x00`, silently, producing a well-formed frame saying the stream continues where the capture said it ended [[RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1)]. `PADDED` was never affected, being re-derived from `pad_len` rather than read back, and all five sibling builders already restored theirs -- so this was a missing read-back rather than a design, and it now reads `frame.flags.END_STREAM` the way they do (#652). Separately, `FrameType.post_process` seeded its flag accumulator with a bare `0`, and `|=` promotes that only as a side effect, so a frame with **no** bit set left `__flags__` a plain `int` where both the schema and the data model declare a `Flags` -- and `END_STREAM in __value__` raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering `False`. That is the common case rather than an edge one: a `0x00` flags octet is every opening `SETTINGS`, every non-final `DATA` and every non-ACK `PING`. It now seeds `self.Flags(0)`, which the construct path had been doing correctly at all six of its own sites. The visible consequence is in the dump, where `__value__` was a JSON *number* for a flagless frame and a JSON *string* for every other frame in the same capture; it is consistently a string now, `"Flags::None [0]"` where it read `0`, the same shape of change #616 made to TCP's `connection`. The seed is guarded rather than unconditional because `FrameType.Flags` declares no members and, from Python 3.12, a memberless `enum.Flag` subclass refuses `Flags(0)` outright, so the five frame schemas that inherit it -- `UnassignedFrame`, `PriorityFrame`, `RSTStreamFrame`, `GoawayFrame` and `WindowUpdateFrame` -- keep the plain `int`, which nothing observes since all five pass `flags=None` into their data objects. Note that a DATA round trip is **still** lossy after this, for an unrelated reason found while measuring it and left to its own change: three payload length callbacks in the frame schemas are mis-parenthesised, so an unpadded `DATA`, `HEADERS` or `PUSH_PROMISE` frame parses with its whole payload dropped (#668). (#652, #650). - **Fixed** -- a flag value with no declared bits no longer dumps as `Type::None [0]`. **This changes user-visible output in every textual dump format.** `make_dumper`'s `object_hook` renders each enumeration member as `Type::name [value]` and interpolated the name unguarded, but a `Flag` value composed entirely of undeclared bits has `name is None` rather than a string -- so the literal four characters `None` landed in the name half and `Flags(0)` rendered as `Flags::None [0]` in `json`, `tree`, `text`, `txt`, `plist` and `xml`, out of both `Extractor` and `TraceFlow`. A nameless member now renders with the value's own decimal spelling instead, so `Flags(0)` gives `Flags::0 [0]` and `Flags(8)` gives `Flags::8 [8]`. That is the spelling the enumeration libraries already use for an undeclared residue -- `Flags(2057).name` is `ACK|9`, naming the declared bit and giving the leftovers as one decimal number -- and it cannot be mistaken for a member name, since a Python identifier may not begin with a digit, whereas `NONE` is a real declared name elsewhere in the library. Three sites carried the identical interpolation, not one: the `OrderedMultiDict` key path, the `addon` branch and the scalar return, which now share one `render_enum` helper so the guard cannot be applied to two of three. That guard is on `name is None` and not on the value, because the defect never was about zero -- `Flags(1)`, `Flags(8)`, `Flags(9)` and `Flags(65536)` are equally nameless -- and it is not an `aenum` quirk either, since a stdlib `enum.IntFlag` answers `name is None` at the same values. Five of the library's seven flag registries are nameless at zero rather than the one reported: `Flags` plus the four Mobility Header flag registries, the other two declaring an explicit `undefined = 0`. Pre-existing since 2023-04-28 and surfaced rather than caused by #634, which made a flagless TCP segment reach this path. The committed example dumps do not move, because no member in them lacks a name (#648). - **Fixed** -- four MPTCP error messages carried a doubled separator, rendering as `TCP: : [OptNo 30] 1: invalid flags combination` with an empty field between the protocol alias and the option number. Cosmetic: the exception type, the option number and the subtype were always right, and nothing downstream parses these strings, but it is the text a user sees when an MP_JOIN or DSS option is rejected and an empty field reads as a value that failed to interpolate. Four sites, not the one reported -- `_read_mptcp_join` and `_make_mptcp_join` for the invalid flag combination, and both guards of `_make_mptcp_dss` for the missing required fields -- against 28 messages in the same file that already spelled the prefix correctly, so the four were outliers against a house form in their own module and the counts are now 32 correct against 0 doubled. Pre-existing since 2023-04-10. The new test asserts each message in full rather than by substring, which is what the existing `assertIn` on the message body could not do, since it passes either side of the change (#649). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index c56daaba5..bfddc8f56 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1591,8 +1591,9 @@ pull requests between #326 and #509. a JSON *string* for every other frame in the same capture; it is consistently a string now, ``"Flags::None [0]"`` where it read ``0``, the same shape of change #616 made to TCP's ``connection``. The seed is guarded rather than - unconditional because ``FrameType.Flags`` declares no members and a - memberless ``enum.Flag`` subclass refuses ``Flags(0)`` outright, so the five + unconditional because ``FrameType.Flags`` declares no members and, from + Python 3.12, a memberless ``enum.Flag`` subclass refuses ``Flags(0)`` + outright, so the five frame schemas that inherit it -- ``UnassignedFrame``, ``PriorityFrame``, ``RSTStreamFrame``, ``GoawayFrame`` and ``WindowUpdateFrame`` -- keep the plain ``int``, which nothing observes since all five pass ``flags=None`` into From 57ef5d92e8042c402632c0b78ddbc5fffebaf24a Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 15:19:28 -0400 Subject: [PATCH 12/46] docs(changelog): record the release-pipeline approval gates (#641) The `pypi` job's `environment: release` was commented out and `conda` never had one, so a scheduled vendor bump could publish to PyPI and Anaconda unapproved. Four jobs are now gated, one environment per credential, and the entry says plainly that the gate is inert until the environments carry required reviewers. Regenerated CHANGELOG.md with `util/changelog_md.py`; `--check` exits 0. --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7515a346c..b1f5067d4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -97,6 +97,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- two HTTP/2 flag defects, one on each side of a round trip. **This changes the octets a reconstructed DATA frame emits, and the dumped value of a flagless frame's flags.** `_make_http_data` was the only one of the six `_make_http_*` methods that never read `frame.flags`, so a DATA frame parsed with `END_STREAM` set rebuilt with the bit clear: the flags octet went `0x01` to `0x00`, silently, producing a well-formed frame saying the stream continues where the capture said it ended [[RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1)]. `PADDED` was never affected, being re-derived from `pad_len` rather than read back, and all five sibling builders already restored theirs -- so this was a missing read-back rather than a design, and it now reads `frame.flags.END_STREAM` the way they do (#652). Separately, `FrameType.post_process` seeded its flag accumulator with a bare `0`, and `|=` promotes that only as a side effect, so a frame with **no** bit set left `__flags__` a plain `int` where both the schema and the data model declare a `Flags` -- and `END_STREAM in __value__` raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering `False`. That is the common case rather than an edge one: a `0x00` flags octet is every opening `SETTINGS`, every non-final `DATA` and every non-ACK `PING`. It now seeds `self.Flags(0)`, which the construct path had been doing correctly at all six of its own sites. The visible consequence is in the dump, where `__value__` was a JSON *number* for a flagless frame and a JSON *string* for every other frame in the same capture; it is consistently a string now, `"Flags::None [0]"` where it read `0`, the same shape of change #616 made to TCP's `connection`. The seed is guarded rather than unconditional because `FrameType.Flags` declares no members and, from Python 3.12, a memberless `enum.Flag` subclass refuses `Flags(0)` outright, so the five frame schemas that inherit it -- `UnassignedFrame`, `PriorityFrame`, `RSTStreamFrame`, `GoawayFrame` and `WindowUpdateFrame` -- keep the plain `int`, which nothing observes since all five pass `flags=None` into their data objects. Note that a DATA round trip is **still** lossy after this, for an unrelated reason found while measuring it and left to its own change: three payload length callbacks in the frame schemas are mis-parenthesised, so an unpadded `DATA`, `HEADERS` or `PUSH_PROMISE` frame parses with its whole payload dropped (#668). (#652, #650). - **Fixed** -- a flag value with no declared bits no longer dumps as `Type::None [0]`. **This changes user-visible output in every textual dump format.** `make_dumper`'s `object_hook` renders each enumeration member as `Type::name [value]` and interpolated the name unguarded, but a `Flag` value composed entirely of undeclared bits has `name is None` rather than a string -- so the literal four characters `None` landed in the name half and `Flags(0)` rendered as `Flags::None [0]` in `json`, `tree`, `text`, `txt`, `plist` and `xml`, out of both `Extractor` and `TraceFlow`. A nameless member now renders with the value's own decimal spelling instead, so `Flags(0)` gives `Flags::0 [0]` and `Flags(8)` gives `Flags::8 [8]`. That is the spelling the enumeration libraries already use for an undeclared residue -- `Flags(2057).name` is `ACK|9`, naming the declared bit and giving the leftovers as one decimal number -- and it cannot be mistaken for a member name, since a Python identifier may not begin with a digit, whereas `NONE` is a real declared name elsewhere in the library. Three sites carried the identical interpolation, not one: the `OrderedMultiDict` key path, the `addon` branch and the scalar return, which now share one `render_enum` helper so the guard cannot be applied to two of three. That guard is on `name is None` and not on the value, because the defect never was about zero -- `Flags(1)`, `Flags(8)`, `Flags(9)` and `Flags(65536)` are equally nameless -- and it is not an `aenum` quirk either, since a stdlib `enum.IntFlag` answers `name is None` at the same values. Five of the library's seven flag registries are nameless at zero rather than the one reported: `Flags` plus the four Mobility Header flag registries, the other two declaring an explicit `undefined = 0`. Pre-existing since 2023-04-28 and surfaced rather than caused by #634, which made a flagless TCP segment reach this path. The committed example dumps do not move, because no member in them lacks a name (#648). - **Fixed** -- four MPTCP error messages carried a doubled separator, rendering as `TCP: : [OptNo 30] 1: invalid flags combination` with an empty field between the protocol alias and the option number. Cosmetic: the exception type, the option number and the subtype were always right, and nothing downstream parses these strings, but it is the text a user sees when an MP_JOIN or DSS option is rejected and an empty field reads as a value that failed to interpolate. Four sites, not the one reported -- `_read_mptcp_join` and `_make_mptcp_join` for the invalid flag combination, and both guards of `_make_mptcp_dss` for the missing required fields -- against 28 messages in the same file that already spelled the prefix correctly, so the four were outliers against a house form in their own module and the counts are now 32 correct against 0 doubled. Pre-existing since 2023-04-10. The new test asserts each message in full rather than by substring, which is what the existing `assertIn` on the message body could not do, since it passes either side of the change (#649). +- **Fixed** -- the release workflow published to PyPI and to Anaconda with nothing a human had to approve. `create-release.yml`'s `pypi` job carried its `environment: release` commented out while the `id-token: write` from the same upstream snippet had been re-added live below it, so the block read as disabled as a unit when only its gate still was; the `conda` job had no `environment:` at all, not even a commented one. Neither needed a tag either: the workflow also fires on the completion of `Vendor Update`, so a scheduled registry crawl that bumped the version was enough to publish to two public indexes, at a median three minutes from bump to upload. Four jobs reach outside the run and all four are now gated -- `github` on `github-release` for the GitHub Release and the `v*` tag it creates, `tag` on `conda-tag` for the commit it pushes to `main`, `pypi` on `pypi`, and `conda` on `anaconda`. One environment per credential rather than one shared `release`, because approval is granted to an environment and not to a job, so a single `release` approved once would release both indexes together. PyPI is the one that has to be answerable alone: a version number it has accepted cannot be reused, and `skip-existing: true` makes the re-upload *succeed* having published nothing, so an unintended publish permanently consumes the number the intended release wanted. The workflow half is only half the fix -- an `environment:` naming an environment that does not exist is created implicitly with no protection rules and the job proceeds unapproved, so this is inert until each of the four exists in repository settings with a required reviewer on it, and each one's "Deployment branches and tags" has to stay unrestricted or the `tags: v*` trigger fails outright instead of pausing. `github-pages` is the standing example of that failure in this repository, carrying no protection rule since 2021. The new test asserts the general rule rather than the current file -- no job running a publishing action or a `git push` may omit an `environment:` -- so a publishing job added later without a gate fails rather than ships (#641). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index bfddc8f56..f90a2994c 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1645,6 +1645,35 @@ pull requests between #326 and #509. each message in full rather than by substring, which is what the existing ``assertIn`` on the message body could not do, since it passes either side of the change (#649). +* **Fixed** -- the release workflow published to PyPI and to Anaconda with + nothing a human had to approve. ``create-release.yml``'s ``pypi`` job carried + its ``environment: release`` commented out while the ``id-token: write`` from + the same upstream snippet had been re-added live below it, so the block read as + disabled as a unit when only its gate still was; the ``conda`` job had no + ``environment:`` at all, not even a commented one. Neither needed a tag either: + the workflow also fires on the completion of ``Vendor Update``, so a scheduled + registry crawl that bumped the version was enough to publish to two public + indexes, at a median three minutes from bump to upload. Four jobs reach outside + the run and all four are now gated -- ``github`` on ``github-release`` for the + GitHub Release and the ``v*`` tag it creates, ``tag`` on ``conda-tag`` for the + commit it pushes to ``main``, ``pypi`` on ``pypi``, and ``conda`` on + ``anaconda``. One environment per credential rather than one shared + ``release``, because approval is granted to an environment and not to a job, so + a single ``release`` approved once would release both indexes together. PyPI is + the one that has to be answerable alone: a version number it has accepted + cannot be reused, and ``skip-existing: true`` makes the re-upload *succeed* + having published nothing, so an unintended publish permanently consumes the + number the intended release wanted. The workflow half is only half the fix -- + an ``environment:`` naming an environment that does not exist is created + implicitly with no protection rules and the job proceeds unapproved, so this is + inert until each of the four exists in repository settings with a required + reviewer on it, and each one's "Deployment branches and tags" has to stay + unrestricted or the ``tags: v*`` trigger fails outright instead of pausing. + ``github-pages`` is the standing example of that failure in this repository, + carrying no protection rule since 2021. The new test asserts the general rule + rather than the current file -- no job running a publishing action or a + ``git push`` may omit an ``environment:`` -- so a publishing job added later + without a gate fails rather than ships (#641). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From d47ada0e7834e9a8dde75dc8d756b8b3450922fb Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 15:19:50 -0400 Subject: [PATCH 13/46] docs(changelog): the empty-enum boundary in the #652/#650 entry is 3.11, not 3.12 b2ec64bf1 qualified the claim with the wrong version. The refusal does not start with a 3.12 change -- it starts at 3.11, and 3.12 only reworded the message. The cross-review on #669 caught it; I had inferred 3.12 from the message text I happened to measure on, which is the wrong evidence for a boundary. Measured across every interpreter available here rather than inferred, with a bare `class Flags(enum.IntFlag): pass`: 3.8.20 Flags(0) -> OK 3.9.25 Flags(0) -> OK 3.10.21 Flags(0) -> OK 3.11.15 Flags(0) -> TypeError: has no members defined 3.12.13 Flags(0) -> TypeError: ... has no members; specify `names=()` ... 3.14.7 Flags(0) -> TypeError: ... has no members; specify `names=()` ... Confirmed in CPython's source, not just behaviourally. 3.11's `enum.py:1117`, inside `Enum.__new__`, raises `TypeError("%r has no members defined" % cls)` when `not cls._member_map_`, and it runs *before* the `_missing_` hook that manufactured the pseudo-member on 3.10. 3.10's `enum.py` has no such raise -- its only "has no members" occurrence is a comment at :616. 3.11 is also where the metaclass was renamed (`class EnumType(type)` at :479 with `EnumMeta = EnumType` at :1052, against 3.10's `class EnumMeta(type)` at :161), so "the enum rewrite" is the 3.11 release. One word in the bullet. #669 carries the matching correction to its source comment and to its test's `sys.version_info` gate, which had been skipping the assertion on 3.11 -- a version the unit-test matrix runs -- even though 3.11 does raise. `CHANGELOG.md` regenerated with `util/changelog_md.py`, not edited; `--check` exits 0 and `tests/project/` is green at 96 passed, 469 subtests. Committed from a detached HEAD on b2ec64bf1 and pushed to the branch ref, because `docs/changelog-1.5.0` is checked out in another agent's worktree. Refs #652 Refs #650 --- CHANGELOG.md | 2 +- docs/source/changelog/1.5.0.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b1f5067d4..4197b4878 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -94,7 +94,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- the HIP `PUZZLE` and `SOLUTION` builders derived three wire-format values from the payload value instead of taking them from the data model, and all three were wrong. **This changes two public data models and the octets both parameters emit.** The width of `Random #I`, and of `Puzzle solution #J`, came from `int.bit_length()` alone, and nothing else was available to derive it from, so every leading zero octet was dropped on re-serialisation: a `SOLUTION` read with `Length = 20` rebuilt as `Length = 6`, a `PUZZLE` read with `Length = 12` as `Length = 5`, and silently, because the integers survive and nothing raises. That width is `RHASH_len / 8` octets [[RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4), [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5)], a property of the Responder's HIT Suite rather than of the number that happens to sit in the field, so both data models now carry it as `rhash_len` and both builders prefer it; a new `rhash_len=` keyword states it for a build from scratch, which under HIPv2 is the only place it can come from, since `RSA,DSA/SHA-256` is the REQUIRED HIT Suite [[RFC 7401 Section 5.2.10](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.10)] and makes the field 32 octets rather than 8. On the mandatory path roughly one parameter in 256 has a zero top octet and lost it (#653). `SOLUTION`'s second contents octet is `Reserved`, "zero when sent, ignored when received" [[RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5), and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) identically] -- not the `Lifetime` that only [RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4) defines, and that pcapkit was encoding there as `2^(value - 32)` seconds. It wrote `0x20`, `0x21`, `0x25` or `0x2b` into a field the RFC requires to be zero, and could not write that zero at all: an RFC-conformant `SOLUTION` whose `Reserved` is `0x00` parsed to `timedelta(0)` and then could not be re-serialised, escaping a bare `ValueError` from `math.log2(0.0)` that was not a `BaseError` and so bypassed the library's own error handling entirely. The field is renamed `reserved`, defaults to the mandated zero, and is carried verbatim across a round trip rather than re-derived (#654). And neither builder read its own `version` keyword, so `version=1` and `version=2` computed identical lengths at every bit width, where [RFC 5201 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.4) and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) state both fields as literally 8 bytes and `Length` as literally 12 and 20. Under HIPv1 each builder therefore accepted only a `bit_length()` of 57..64 and built, for everything narrower, a parameter this library's own reader rejects -- the same shape as #608, in the `PUZZLE` builder that #629 never opened (#655). The two remaining `math.log2` sites, both in `PUZZLE` where a lifetime is real, now raise `ProtocolError` rather than letting `ValueError` escape; `ProtocolError` is `(BaseError, ValueError)`, so a caller written around the old bare exception still catches it. Migration: `SolutionParameter.lifetime` is now `reserved` and an `int` rather than a `timedelta`; both `PuzzleParameter` and `SolutionParameter` gained a required `rhash_len`; and `_make_param_solution` no longer takes `lifetime=`. All three were only reachable end to end once #608 was fixed in #629, which removed the parity guard that had been failing these rebuilds loudly first -- so the round trip stopped raising and started quietly emitting a different parameter, which is why they were worth fixing together (#653, #654, #655). - **Fixed** -- every HIP parameter pcapkit emitted was `4 (mod 8)` octets long, because the padding was computed to align the *contents* rather than the record. **This changes the octets of every HIP parameter written, and the `length` every parameter's data model reports.** [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1) requires that "all of the encoded TLV parameters have a length (that includes the Type and Length fields), which is a multiple of 8 bytes", and states the arithmetic outright as `Total Length = 11 + Length - (Length + 3) % 8`; all 95 padding sites instead computed `(8 - (Length % 8)) % 8`, which has no `+ 4` inside the modulus and so aligns the contents alone. Across every `Length` from 0 to 63 the result was never a multiple of eight and never the value the RFC gives, and it erred in both directions: at `Length = 4` -- a whole `SEQ`, and [RFC 7401 Section 5.3.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.3.5) puts a `SEQ` or an `ACK` on every `UPDATE` -- the record is complete in eight octets and pcapkit appended four that must not be there, while at `Length = 8` the contents were already 8-aligned, nothing was appended, and the record went out four octets short. A conformant peer reading `Length` and consuming `11 + Length - (Length + 3) % 8` octets therefore lands mid-parameter and reads the rest of the parameter area at a wrong offset, in both directions; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test in the suite could see this and why the fix is asserted against the RFC's arithmetic rather than against a round trip. The 46 `PaddingField` callbacks in `pcapkit/protocols/schema/internet/hip.py` and the 49 record lengths in `pcapkit/protocols/internet/hip.py` are now one `parameter_total_len` and one `parameter_padding_len`, stating the RFC formula once instead of 95 times (#651). `EncryptedParameter`'s `data` length callback is fixed in the same change and could not have been left: it subtracted the sixteen `iv` octets but not the four `reserved` ones, and those four cancelled the padding's four at `Length % 8` in `{0, 5, 6, 7}` -- so correcting the padding alone would have taken `ENCRYPTED` from right at four of the eight residues to four octets too long at all eight. `HIP.make`'s `len = total_length // 8 + 4` is *not* part of the defect and is unchanged: [RFC 7401 Section 5.1.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.1.3) defines Header Length as the header and parameters "in 8-byte units, excluding the first 8 bytes", so the floor division is exact once each parameter is a multiple of eight, where before it was exact only for an even number of them. Migration: a `SEQ` parameter's `length` is now 8 where it was 12, and every other parameter's moves likewise, so code comparing stored pcapkit output byte for byte, or asserting on `Data_*Parameter.length`, sees different values -- the RFC's values. `examples/generators/options.py`'s `HIP_COPIES` stays at two, no longer for this reason but for `R1_COUNTER`'s four-octet `counter` where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires eight, which this defect had been masking (#651). - **Fixed** -- the `_missing` sentinel behind `MultiDict.pop` and `OrderedMultiDict.pop` answered `True` to `bool()`, stating the opposite of what it means, and its class was subclassable. Both now follow `NoValueType` in `pcapkit.corekit.fields.field`, the package's convention for a marker of this kind: `_Missing` carries `@final` and defines `__bool__` returning `False`. The marker is not private in practice -- it is the runtime default of both `pop()` methods, so `inspect.signature(MultiDict.pop).parameters['default'].default` hands it to any caller who asks, and `if default:` on that value reported "a default was supplied" where none had been. No behaviour changes: both `pop()` implementations decide by identity (`default is not _missing`) and never by truthiness, so `repr`, `__reduce__`, the pickle bytes at every protocol, and every `pop()` result -- including the falsy defaults `None`, `False`, `0`, `''` and `[]` -- are unchanged, and `pop()` structurally cannot return the marker itself. This closes the second of the three `object()` sentinel sites the sweep for #640 left open; the first needed nothing, and `_NOT_FOUND` in `pcapkit.utilities.compat` is deliberately left alone as a verbatim line of CPython's own `functools.cached_property`, inside a `sys.version_info < (3, 8)` branch no supported interpreter reaches (#661). -- **Fixed** -- two HTTP/2 flag defects, one on each side of a round trip. **This changes the octets a reconstructed DATA frame emits, and the dumped value of a flagless frame's flags.** `_make_http_data` was the only one of the six `_make_http_*` methods that never read `frame.flags`, so a DATA frame parsed with `END_STREAM` set rebuilt with the bit clear: the flags octet went `0x01` to `0x00`, silently, producing a well-formed frame saying the stream continues where the capture said it ended [[RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1)]. `PADDED` was never affected, being re-derived from `pad_len` rather than read back, and all five sibling builders already restored theirs -- so this was a missing read-back rather than a design, and it now reads `frame.flags.END_STREAM` the way they do (#652). Separately, `FrameType.post_process` seeded its flag accumulator with a bare `0`, and `|=` promotes that only as a side effect, so a frame with **no** bit set left `__flags__` a plain `int` where both the schema and the data model declare a `Flags` -- and `END_STREAM in __value__` raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering `False`. That is the common case rather than an edge one: a `0x00` flags octet is every opening `SETTINGS`, every non-final `DATA` and every non-ACK `PING`. It now seeds `self.Flags(0)`, which the construct path had been doing correctly at all six of its own sites. The visible consequence is in the dump, where `__value__` was a JSON *number* for a flagless frame and a JSON *string* for every other frame in the same capture; it is consistently a string now, `"Flags::None [0]"` where it read `0`, the same shape of change #616 made to TCP's `connection`. The seed is guarded rather than unconditional because `FrameType.Flags` declares no members and, from Python 3.12, a memberless `enum.Flag` subclass refuses `Flags(0)` outright, so the five frame schemas that inherit it -- `UnassignedFrame`, `PriorityFrame`, `RSTStreamFrame`, `GoawayFrame` and `WindowUpdateFrame` -- keep the plain `int`, which nothing observes since all five pass `flags=None` into their data objects. Note that a DATA round trip is **still** lossy after this, for an unrelated reason found while measuring it and left to its own change: three payload length callbacks in the frame schemas are mis-parenthesised, so an unpadded `DATA`, `HEADERS` or `PUSH_PROMISE` frame parses with its whole payload dropped (#668). (#652, #650). +- **Fixed** -- two HTTP/2 flag defects, one on each side of a round trip. **This changes the octets a reconstructed DATA frame emits, and the dumped value of a flagless frame's flags.** `_make_http_data` was the only one of the six `_make_http_*` methods that never read `frame.flags`, so a DATA frame parsed with `END_STREAM` set rebuilt with the bit clear: the flags octet went `0x01` to `0x00`, silently, producing a well-formed frame saying the stream continues where the capture said it ended [[RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1)]. `PADDED` was never affected, being re-derived from `pad_len` rather than read back, and all five sibling builders already restored theirs -- so this was a missing read-back rather than a design, and it now reads `frame.flags.END_STREAM` the way they do (#652). Separately, `FrameType.post_process` seeded its flag accumulator with a bare `0`, and `|=` promotes that only as a side effect, so a frame with **no** bit set left `__flags__` a plain `int` where both the schema and the data model declare a `Flags` -- and `END_STREAM in __value__` raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering `False`. That is the common case rather than an edge one: a `0x00` flags octet is every opening `SETTINGS`, every non-final `DATA` and every non-ACK `PING`. It now seeds `self.Flags(0)`, which the construct path had been doing correctly at all six of its own sites. The visible consequence is in the dump, where `__value__` was a JSON *number* for a flagless frame and a JSON *string* for every other frame in the same capture; it is consistently a string now, `"Flags::None [0]"` where it read `0`, the same shape of change #616 made to TCP's `connection`. The seed is guarded rather than unconditional because `FrameType.Flags` declares no members and, from Python 3.11, a memberless `enum.Flag` subclass refuses `Flags(0)` outright, so the five frame schemas that inherit it -- `UnassignedFrame`, `PriorityFrame`, `RSTStreamFrame`, `GoawayFrame` and `WindowUpdateFrame` -- keep the plain `int`, which nothing observes since all five pass `flags=None` into their data objects. Note that a DATA round trip is **still** lossy after this, for an unrelated reason found while measuring it and left to its own change: three payload length callbacks in the frame schemas are mis-parenthesised, so an unpadded `DATA`, `HEADERS` or `PUSH_PROMISE` frame parses with its whole payload dropped (#668). (#652, #650). - **Fixed** -- a flag value with no declared bits no longer dumps as `Type::None [0]`. **This changes user-visible output in every textual dump format.** `make_dumper`'s `object_hook` renders each enumeration member as `Type::name [value]` and interpolated the name unguarded, but a `Flag` value composed entirely of undeclared bits has `name is None` rather than a string -- so the literal four characters `None` landed in the name half and `Flags(0)` rendered as `Flags::None [0]` in `json`, `tree`, `text`, `txt`, `plist` and `xml`, out of both `Extractor` and `TraceFlow`. A nameless member now renders with the value's own decimal spelling instead, so `Flags(0)` gives `Flags::0 [0]` and `Flags(8)` gives `Flags::8 [8]`. That is the spelling the enumeration libraries already use for an undeclared residue -- `Flags(2057).name` is `ACK|9`, naming the declared bit and giving the leftovers as one decimal number -- and it cannot be mistaken for a member name, since a Python identifier may not begin with a digit, whereas `NONE` is a real declared name elsewhere in the library. Three sites carried the identical interpolation, not one: the `OrderedMultiDict` key path, the `addon` branch and the scalar return, which now share one `render_enum` helper so the guard cannot be applied to two of three. That guard is on `name is None` and not on the value, because the defect never was about zero -- `Flags(1)`, `Flags(8)`, `Flags(9)` and `Flags(65536)` are equally nameless -- and it is not an `aenum` quirk either, since a stdlib `enum.IntFlag` answers `name is None` at the same values. Five of the library's seven flag registries are nameless at zero rather than the one reported: `Flags` plus the four Mobility Header flag registries, the other two declaring an explicit `undefined = 0`. Pre-existing since 2023-04-28 and surfaced rather than caused by #634, which made a flagless TCP segment reach this path. The committed example dumps do not move, because no member in them lacks a name (#648). - **Fixed** -- four MPTCP error messages carried a doubled separator, rendering as `TCP: : [OptNo 30] 1: invalid flags combination` with an empty field between the protocol alias and the option number. Cosmetic: the exception type, the option number and the subtype were always right, and nothing downstream parses these strings, but it is the text a user sees when an MP_JOIN or DSS option is rejected and an empty field reads as a value that failed to interpolate. Four sites, not the one reported -- `_read_mptcp_join` and `_make_mptcp_join` for the invalid flag combination, and both guards of `_make_mptcp_dss` for the missing required fields -- against 28 messages in the same file that already spelled the prefix correctly, so the four were outliers against a house form in their own module and the counts are now 32 correct against 0 doubled. Pre-existing since 2023-04-10. The new test asserts each message in full rather than by substring, which is what the existing `assertIn` on the message body could not do, since it passes either side of the change (#649). - **Fixed** -- the release workflow published to PyPI and to Anaconda with nothing a human had to approve. `create-release.yml`'s `pypi` job carried its `environment: release` commented out while the `id-token: write` from the same upstream snippet had been re-added live below it, so the block read as disabled as a unit when only its gate still was; the `conda` job had no `environment:` at all, not even a commented one. Neither needed a tag either: the workflow also fires on the completion of `Vendor Update`, so a scheduled registry crawl that bumped the version was enough to publish to two public indexes, at a median three minutes from bump to upload. Four jobs reach outside the run and all four are now gated -- `github` on `github-release` for the GitHub Release and the `v*` tag it creates, `tag` on `conda-tag` for the commit it pushes to `main`, `pypi` on `pypi`, and `conda` on `anaconda`. One environment per credential rather than one shared `release`, because approval is granted to an environment and not to a job, so a single `release` approved once would release both indexes together. PyPI is the one that has to be answerable alone: a version number it has accepted cannot be reused, and `skip-existing: true` makes the re-upload *succeed* having published nothing, so an unintended publish permanently consumes the number the intended release wanted. The workflow half is only half the fix -- an `environment:` naming an environment that does not exist is created implicitly with no protection rules and the job proceeds unapproved, so this is inert until each of the four exists in repository settings with a required reviewer on it, and each one's "Deployment branches and tags" has to stay unrestricted or the `tags: v*` trigger fails outright instead of pausing. `github-pages` is the standing example of that failure in this repository, carrying no protection rule since 2021. The new test asserts the general rule rather than the current file -- no job running a publishing action or a `git push` may omit an `environment:` -- so a publishing job added later without a gate fails rather than ships (#641). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index f90a2994c..b2c700898 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1592,7 +1592,7 @@ pull requests between #326 and #509. a string now, ``"Flags::None [0]"`` where it read ``0``, the same shape of change #616 made to TCP's ``connection``. The seed is guarded rather than unconditional because ``FrameType.Flags`` declares no members and, from - Python 3.12, a memberless ``enum.Flag`` subclass refuses ``Flags(0)`` + Python 3.11, a memberless ``enum.Flag`` subclass refuses ``Flags(0)`` outright, so the five frame schemas that inherit it -- ``UnassignedFrame``, ``PriorityFrame``, ``RSTStreamFrame``, ``GoawayFrame`` and ``WindowUpdateFrame`` -- keep the From 29aa75247dc601e9d56637b478ebc6b6c603b38f Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 15:20:24 -0400 Subject: [PATCH 14/46] docs(changelog): the 1.5.0 entry for #642, whose code ships in #666 The changelog changes #666 was carrying on its own branch, moved here so that #666 touches only `MANIFEST.in`, `pcapkit/protocols/schema/internet/ipv6_route.py`, `pcapkit/utilities/logging.py` and `tests/project/test_annotation_names.py`. #666 was the last open branch still editing `CHANGELOG.md` itself; it no longer does. Four pieces, not one, because #666 had amended existing entries as well as needing a new one. A new **Fixed** bullet for #642: three names used in string annotations that their own module never imported -- `Any` in `pcapkit/utilities/logging.py`, and `Protocol` and `Optional` in `pcapkit/protocols/schema/internet/ipv6_route.py`. The bullet names the `typing.cast` case specifically, because that is the one no running test can catch: `cast` never evaluates its first argument. It states plainly that nothing resolves at runtime that did not before, since `TYPE_CHECKING` is `False` when the interpreter runs, so that the entry is not read as a runtime fix. mypy 2.3.1's before/after is quoted as the measurement -- three `name-defined` errors to none, 115 total to 112 -- and the new `tests/project/test_annotation_names.py` is described as what pins the invariant without a type checker installed. Three missing citations recovered: `(#570)` on the L2TPv3 worked-example line, `(#577)` on the `register_extractor_engine` keyword line, and `(#619)` on the README rename entry. And the #619 entry's packaging claim corrected in place. It asserted that `include README.md` in `MANIFEST.in` was "the only thing that puts the README in a source distribution" and that an sdist without it "cannot be installed". Both halves are false: setuptools' own `sdist` command ships the README before `MANIFEST.in` is read at all, so dropping the line leaves the listing byte-identical at 861 entries, and `setup.py` reads the file from wherever it is executing, which for a `pip` install of an sdist is the unpacked sdist. #666 corrects the same claim in the `MANIFEST.in` comment, so the two stay in step. No `:pep:` role, though the new bullet discusses PEP 695: `util/changelog_md.py` converts only double-backtick literals and the `:rfc:` role, and raises `ResidualMarkupError` on anything else, exactly as the #661 entry hit with `:obj:`. Plain prose instead. 50 lines added to the two files, 11 reflowed. `CHANGELOG.md` regenerated with `util/changelog_md.py`, not edited; `--check` exits 0 and `tests/project/` is green at 96 passed, 469 subtests, exit 0 -- the same counts the previous commit on this branch reported, so nothing else moved. Committed from a detached HEAD and pushed to the branch ref, because `docs/changelog-1.5.0` is checked out in another agent's worktree at a stale f84652303 and could not be taken here. Written against b2ec64bf1 and rebased onto d47ada0e7, the branch having taken #641's entry and the #652/#650 boundary correction in the meantime. Both files conflicted, both at the append point rather than in substance -- #641's bullet and this one land in the same place at the end of **Fixed** -- so the resolution keeps both, #641's first. `CHANGELOG.md` was not hand-resolved: it is generated, so it was regenerated from the resolved entry file and `--check` re-run, which is the only resolution that cannot drift. Refs #642 --- CHANGELOG.md | 7 +++-- docs/source/changelog/1.5.0.rst | 54 ++++++++++++++++++++++++++++----- 2 files changed, 50 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4197b4878..62684c083 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,7 +21,7 @@ The largest release since 1.0, and the first recorded here as it happened rather - **Changed** -- `Probe`, `CipherSuite` and `IntegritySuite` are `Info` subclasses rather than `typing.NamedTuple`, and no `NamedTuple` remains in the package. They are Mappings now, so `len()` and iteration yield field names rather than values. - **Changed** -- renames with no compatibility alias left behind: `HoleDiscriptor` is spelled `HoleDescriptor` and its package alias `TCP_HoleDiscriptor` is `TCP_HoleDescriptor` (#350); PCAP-NG `Option` subclasses spell the namespace class keyword `ns=` instead of `namespace=` (#439); and `examples/sample` and `examples/samples` -- one letter apart, holding different things -- are now `examples/captures` and `examples/generators`. - **Changed** -- subclass registration is **opt-in** for `Engine`, `Reassembly`, `TraceFlow` and `dumpkit`'s `Dumper` (#514). Each registers if and only if its registry keyword is given -- `engine=` for `Engine`, `protocol=` for `Reassembly` and `TraceFlow`, `fmt=` for `Dumper`. Previously an absent keyword fell back to the class' own name, so *every* subclass of the public class was registered, and declining meant subclassing the parallel `*Base` class under an alias -- which is what every built-in does, and why the public classes had **0** subclasses between them against the `*Base` classes' 9, 5, 2 and 3. **This breaks out-of-tree code that subclasses one of the four and relies on the derived key**; pass the keyword, or call the matching `register_*` function. Nothing the library ships is affected, and the `*Base` classes remain importable. Two things that were silent are now loud: an unrecognised class keyword raises `UnsupportedCall` instead of being swallowed by `**kwargs` -- which used to register the class under its own name, so passing `name=` to a `Reassembly` subclass silently ignored the key it was given, `protocol=` being the real one -- and `Dumper`'s `ext=` without `fmt=` likewise. A class attribute is not an opt-in: `__engine_name__` and `__protocol_name__` still set the name a class reports, registered or not. Each metaclass also gained a class-level `registry` property mirroring `EnumSchema.registry`. As a side effect a `Dumper` subclass no longer touches the filesystem while its `class` statement runs: inferring `fmt` from the `kind` property meant instantiating the class against a `NamedTemporaryFile` mid-definition. `Engine`'s keyword is `engine=` rather than the `name=` this first shipped with, because `name` cannot be passed as a class keyword at all on Python 3.10: `mcls`, `name`, `bases` and `namespace` collide with `abc.ABCMeta.__new__`'s own parameters, which are positional-or-keyword before 3.11 and positional-only from 3.11, so a class statement naming any of the four raises `TypeError` from the metaclass before the hook is reached. Those four are the whole of the `ABCMeta.__new__` collision surface, measured on 3.10.21, 3.11.15 and 3.14.7; `engine=`, `protocol=` and `fmt=` are all outside it, so the documented registration path works on every supported version. There is no `name=` alias -- a keyword that worked on some interpreters and not others is the trap being removed, not a compatibility measure. -- **Added** -- `Protocol`/`ProtocolBase` gain a `code=` class keyword for the next-layer dispatch registries -- `Link.__proto__`, `Internet.__proto__`, `TCP.__proto__`, `UDP.__proto__`, `SCTP.__proto__`, `Frame.__proto__` and `PCAPNG.__proto__` -- the same opt-in treatment #514 gave `Engine`, `Reassembly`, `TraceFlow` and `Dumper` above, extended to the one family that needed a registration key invented rather than merely un-guarded. Omitting `code` leaves a subclass unregistered, exactly as before this keyword existed: the built-in dispatch tables are still populated by literal assignment in each layer module, not by `__init_subclass__`, so nothing the library ships moves. `code` accepts a bare enum member, whose *type* infers the destination -- `EtherType` means `Link`, `TransType` means `Internet`, `PayloadProtocolIdentifier` means `SCTP`, and `LinkType` means *both* `Frame` **and** `PCAPNG`, deterministically, mirroring what `register_linktype` already does by hand -- or a `{destination: key}` mapping, required for a raw `int` such as a TCP/UDP port number, which cannot say by itself which transport it belongs to. Either form may appear in an iterable, so one declaration can register a class into several registries at once, e.g. a `L2TP` subclass reachable both by IP protocol number and by a UDP port. The explicit mapping form is accepted even for a key whose type could be inferred -- being more explicit than required is never an error. Inference refuses rather than guesses: an enum member whose type names no known destination raises `RegistryError` instead of silently doing nothing or picking an arbitrary registry, and an unrecognised class keyword raises `UnsupportedCall`, matching the other four families. Backed by the new `pcapkit.foundation.registry.protocols.register_protocol_code`, which can also be called directly to register a class that declined at class-definition time. This was written up as the mechanism #548 (`TransType.L2TP` registered nowhere) needs, with fixing that issue described here as "now a one-declaration change". Investigating #548 found otherwise -- 115 is an [RFC 3931](https://datatracker.ietf.org/doc/html/rfc3931) L2TPv3-over-IP header with no class to dispatch to, so the declaration would have pointed the [RFC 2661](https://datatracker.ietf.org/doc/html/rfc2661) parser at it. See the corresponding **Fixed** entry below; the mechanism itself is unaffected, and its worked example now names `L2TPv3` rather than `L2TPv2`. +- **Added** -- `Protocol`/`ProtocolBase` gain a `code=` class keyword for the next-layer dispatch registries -- `Link.__proto__`, `Internet.__proto__`, `TCP.__proto__`, `UDP.__proto__`, `SCTP.__proto__`, `Frame.__proto__` and `PCAPNG.__proto__` -- the same opt-in treatment #514 gave `Engine`, `Reassembly`, `TraceFlow` and `Dumper` above, extended to the one family that needed a registration key invented rather than merely un-guarded. Omitting `code` leaves a subclass unregistered, exactly as before this keyword existed: the built-in dispatch tables are still populated by literal assignment in each layer module, not by `__init_subclass__`, so nothing the library ships moves. `code` accepts a bare enum member, whose *type* infers the destination -- `EtherType` means `Link`, `TransType` means `Internet`, `PayloadProtocolIdentifier` means `SCTP`, and `LinkType` means *both* `Frame` **and** `PCAPNG`, deterministically, mirroring what `register_linktype` already does by hand -- or a `{destination: key}` mapping, required for a raw `int` such as a TCP/UDP port number, which cannot say by itself which transport it belongs to. Either form may appear in an iterable, so one declaration can register a class into several registries at once, e.g. a `L2TP` subclass reachable both by IP protocol number and by a UDP port. The explicit mapping form is accepted even for a key whose type could be inferred -- being more explicit than required is never an error. Inference refuses rather than guesses: an enum member whose type names no known destination raises `RegistryError` instead of silently doing nothing or picking an arbitrary registry, and an unrecognised class keyword raises `UnsupportedCall`, matching the other four families. Backed by the new `pcapkit.foundation.registry.protocols.register_protocol_code`, which can also be called directly to register a class that declined at class-definition time. This was written up as the mechanism #548 (`TransType.L2TP` registered nowhere) needs, with fixing that issue described here as "now a one-declaration change". Investigating #548 found otherwise -- 115 is an [RFC 3931](https://datatracker.ietf.org/doc/html/rfc3931) L2TPv3-over-IP header with no class to dispatch to, so the declaration would have pointed the [RFC 2661](https://datatracker.ietf.org/doc/html/rfc2661) parser at it. See the corresponding **Fixed** entry below; the mechanism itself is unaffected, and its worked example now names `L2TPv3` rather than `L2TPv2` (#570). - **Changed** -- extraction is around 46% faster on a 1,117-frame HTTP capture, with byte-identical output (#420). A reassembled datagram's payload is now analysed on first read rather than eagerly, which cuts IP reassembly's own cost by 90.7% and TCP's by 23.7% -- IP reassembly submits a datagram for every frame, fragmented or not (#424). Flow tracing over the same capture went from 1416.6 ms to 744.0 ms, because the flow dumper had been handing each record to a `Frame` constructor that re-dissected the whole protocol stack to return bytes it had just been given; options are no longer parsed twice either (#427). All output compared byte-for-byte across the sample captures in each case. - **Fixed** -- next-layer, option, chunk, block and parameter dispatch all read `defaultdict` registries, so a lookup miss inserted the key into class-level state shared by every later instance, after which a legitimate `register_*` call warned that the code was already registered. Every read now goes through a lookup that does not grow the table, and `IPv4.__option__` and `HIP.__parameter__` became inspectable class attributes rather than names assembled at call time (#426, #428, #429, #434). One break comes with it: a tuple-registered handler pair written to the documented `OptionParser`/`OptionConstructor` signature now works where it could previously never be called at all, and a pair written with an explicit leading `self` -- the only shape that used to work -- now does not. - **Fixed** -- the identical defect one layer up, in the schema layer's own `EnumSchema.registry`: `Option.registry[code]` for an unregistered `code` inserted the default schema under that code, so a single lookup made an unassigned TCP option number, e.g. `156`, read back as registered for the rest of the process. `EnumSchema.__enum__` is now built (or, when a subclass seeds it manually in its own class body -- `PCAPNG.Option`'s namespaced mapping, `TCP.MPTCP`'s plain one) as a retention-safe mapping that still returns the registered default on a miss, it just stops recording it; `.registry` keeps returning the same object it always did, so nothing that held a reference to it is affected (#555). @@ -37,7 +37,7 @@ The largest release since 1.0, and the first recorded here as it happened rather - **Fixed** -- constant lookups that rejected a value the registry defines. `RouterAlert(0)` is the only value [RFC 2113](https://datatracker.ietf.org/doc/html/rfc2113) defines and the one IGMP, RSVP and MLD actually send, and it was discarded because the vendor crawler skipped a header row IANA's CSV does not have; IPX `Socket(0)` is that protocol's own default, so `bytes(IPX(...))` crashed on its own defaults; and two FTP `_missing_` overrides were plain methods rather than classmethods, so every unregistered value raised `TypeError` instead of extending the enumeration (#492, #503). - **Fixed** -- `format='text'` raised `AttributeError` before writing anything, naming a `dictdumper.Text` that has never existed. It now points at `Tree`, as the `'txt'` alias beside it already did. - **Fixed** -- 45 places where a documentation page contradicted the code (#413), ambiguous cross-references and five autodoc signature failures (#416), and `Extractor`'s documented exception plus 40 phantom or stale `Args:` labels (#501). -- **Fixed** -- two more gaps the #514 keyword audit turned up, neither previously covered by a test: `StreamEOFError`'s docstring did not say that `@prepare` always raises it with `quiet=True` -- the same end-of-stream convention `StructError` follows via its own `eof=True` -- so nothing pinned that silence against a future regression; and `register_extractor_engine`'s real keyword, `name`, was not itself under test, only its already-corrected docstring, so a future rename could put the two out of step again exactly as quietly as before. +- **Fixed** -- two more gaps the #514 keyword audit turned up, neither previously covered by a test: `StreamEOFError`'s docstring did not say that `@prepare` always raises it with `quiet=True` -- the same end-of-stream convention `StructError` follows via its own `eof=True` -- so nothing pinned that silence against a future regression; and `register_extractor_engine`'s real keyword, `name`, was not itself under test, only its already-corrected docstring, so a future rename could put the two out of step again exactly as quietly as before (#577). - **Fixed** -- `FieldBase.unpack` zero-padded straight up to a field's declared `length` with `rjust()`, regardless of how little data `buffer` actually held; a ~40-octet PCAP-NG Decryption Secrets Block with a bogus inner length was enough to force a multi-gigabyte allocation, since `length` is frequently wire-derived and so attacker-controlled. A declared length past 262144 octets -- libpcap's own `MAXIMUM_SNAPLEN`, and this package's own default `snaplen` -- that the buffer cannot back now raises `FieldValueError` instead of padding for it; the option and list loops' own tolerance for a short read past a truncated area (#431) is far under that ceiling and is untouched (#554). - **Fixed** -- that ceiling bounds one field, and a packet holds many, so the *sum* of a parse's zero padding was still unbounded: a declared length just under 262144 octets is honoured however often it is declared. 200 minimal PCAP-NG Decryption Secrets Blocks -- 4,800 wire octets, each declaring a `secrets_length` of 262,142 against two supplied octets -- retained 50.0 MiB, an amplification of 10,922x per block, with the per-field guard never firing because every individual field was within it. `FieldBase.unpack` now keeps a running per-context ledger of octets supplied against octets synthesised, and raises `FieldValueError` once the total of shortfalls *past 65,536 octets* passes `262144 + 16 * supplied`. The same 200 blocks now retain 0.2 MiB, 54.6x rather than 10,922x. A shortfall of 65,536 octets or fewer is padded unconditionally and charged to nothing, and that band is the load-bearing part rather than a concession. 65,536 is the whole span of a 16-bit wire length -- how an IP header, an IPv6 payload, a TCP or IPv4 option and a PCAP-NG option all declare their size -- so no shortfall a capture cut short by its snapshot length can produce is subject to the budget at all, on the first frame or the ten-thousandth. Without that band, a running budget alone made the *same* legitimate 54-octet frame declaring an IPv4 total length of 65,535 parse to one result on 37 of 40 identical calls and to another on calls 26, 33 and 39, since whether it fit depended on what had been parsed before it; a guard whose answer moves with history is worse than the amplification it bounds. The worst legitimate single shortfall measured anywhere was 65,495 octets, from exactly that frame, and the worst from a truncated PCAP-NG option was 64,750. **What this does not close, measured rather than assumed**: the 16-bit band is deliberately untouched, so a length declared by a 16-bit wire field can still be repeated without limit. A crafted 80,048-octet PCAP-NG file of 2,000 Enhanced Packet Blocks, each carrying one option declaring 65,535 octets against four real ones, parses end to end through `Extractor(store=True)` and retains 125.00 MiB of synthesised zeros for 216.88 MiB of RSS -- 1,637x its own size, linear in the block count -- both before this change and after it. That is not an oversight in the bound: parsing a bare 40-octet IPv4 header declaring a total length of 65,535, which is what a legitimate capture of offload-sized segments truncated to its snapshot length looks like, amplifies by **the same 1,637x**. The two are not separable by any budget at this layer. Separating them needs the frame's own `incl_len`/`orig_len` -- a crafted block claims nothing was truncated while declaring more than it holds, and a snapshot-truncated frame says so on the wire -- which is knowable at the protocol layer and not here. Nor is the budget scoped per file: nothing in the package resets the ledger, so as shipped the bound is over everything a context has parsed rather than over one `Extractor` run. That is still proportionate to the octets that context was genuinely given, and tightening it is a one-line change at whichever layer owns a run. Verified against every capture in `examples/captures/`, every one of them truncated at some 8,700 offsets, 190 snapshot-length rewrites, and the synthetic offload shapes above. The comparison is of the instrumented padding and supplied-octet tallies *and* of each parse's outcome -- frame count, or exception type and message -- so a cut that changed from parsing to crashing would show rather than be swallowed; every one is identical to before (#573). - **Fixed** -- `TCP._make_mptcp_addaddr` could not build an `ADD_ADDR` option end to end: its `kind=`/`length=` arguments were rejected with `UnknownFieldWarning` and silently dropped, and `.pack()` then raised `KeyError: 'length'` from `port`'s own condition, `pkt['length'] in (10, 22)`. The cause was one layer up -- `MPTCP`, the base class every Multipath TCP subtype schema inherits, declared `kind` and `length` only under `typing.TYPE_CHECKING` rather than as real fields, unlike `Option`, which every non-Multipath TCP option schema inherits instead. That silently dropped `kind=`/`length=` for every `_make_mptcp_*` constructor, not only `ADD_ADDR`'s, so `MPTCP` now declares both for real, the same way `Option` already did (#541). The same missing fields broke parsing too: with no `kind`/`length` fields ahead of it, a Multipath TCP subtype schema's own leading field read the `kind` octet itself rather than the octet meant for it, an off-by-two in field alignment rather than a wire-format change -- a correct sender's octets were always right, only this library's reading of them was shifted. Spec-correct `ADD_ADDR` and `MP_PRIO` options failed to parse with `FieldError: TCP: [OptNo 30] 3 invalid IP version` and `KeyError: 'length'` respectively; both parse correctly now. @@ -72,7 +72,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Added** -- `examples/generators/endian.py`, and the byte-order tests that read what it writes. There was no big-endian `.pcap` in the repository at all, which is why #605 survived its own code review: the one-character fix leaves the corrected path exactly as untested as the broken one. The generator writes three captures -- `big_endian.pcap` (magic `a1 b2 c3 d4`), `big_endian_nanosecond.pcap` (`a1 b2 3c 4d`, the first fixture to take that branch of the magic-number table) and `little_endian.pcap` (`d4 c3 b2 a1`) -- carrying the *same three records* in each container, so the tests can assert that the byte order makes no difference to what is read out rather than only that the big-endian file matches numbers written down in a test. Frame 3 is captured short, 1200 octets on the wire cut to a 96-octet `snaplen`, so `incl_len` and `orig_len` differ and cannot both be satisfied by one byte-swapped value. `test_frame_endian_runtime.py` drives all three through `extract()` and walks each file's record chain with `struct` to derive its own expectations; a unit-tier case in `test_header_frame_unit.py` builds a two-record big-endian capture in memory instead, so the regression is also caught by the fixture-free selection CI runs on every push. All four fail on the unfixed tree -- the three fixture-backed ones by that `ValueError`, the in-memory one by `AssertionError: 3106905 != 1500000000` -- while the little-endian twin passes on both trees, which is what shows the records themselves are not the variable (#605). - **Changed** -- `tests/protocols/transport/test_tcp_udp_unit.py` now reaches the MP_JOIN dispatchers through `TCP()` itself, instead of assigning a Python `set` to `_flags` on a bare `TCP.__new__(TCP)`. A `set` answers the membership tests `_make_mptcp_join` and `_read_mptcp_join` use, so every flag branch ran and both TCP modules read 100% statement and branch coverage -- while the attribute had neither the `aenum.IntFlag` type production assigns nor the ordering that governs when it exists at all, which is how #587 stayed invisible behind that number and how the `cast('Enum_Flags', 0)` no-op behind it went unnoticed too. Measured on the rewrite, against the 17 tests of that file: revert #587's hoist and two of them fail with `AttributeError: 'TCP' object has no attribute '_flags'` where all 17 passed before; restore the `cast` and two fail with `TypeError: argument of type 'int' is not a container or iterable`, again where all 17 passed. The library is unchanged and the file's tests still pass, so the coverage numbers do not move -- the point is what the same numbers are now worth (#603). - **Fixed** -- documentation. `mptcp_dss_ack_selector`'s note said a corrected field-width lambda "would not have worked" and that fixing it belonged to `pcapkit.corekit.fields.numbers`, which is exactly where #598 then fixed it; the same paragraph sat in `test_tcp_mptcp_length_arithmetic_unit.py`'s module docstring, whose other stale claim was that MP_JOIN "cannot be built through the public `TCP()` constructor at all", true only until #587. A callable-length `NumberField` packs and unpacks both DSS widths now, and wire *absence* was never the obstacle either: `MPTCPDSS.ssn`, `dl_len` and `checksum` have always been `ConditionalField` on the sibling `M` flag, so the class already relied on that wrapper to keep a field off the wire. The `SwitchField` form is kept for the narrower reason the note now gives -- `ConditionalField`'s `length` forwards to the wrapped field without consulting the condition, so it is safe here only because `Schema.pack` and `Schema.unpack` special-case that wrapper by name, whereas a `SwitchField` always resolves to a concrete field. Replacing it would be a behaviour change and is not made (#603). -- **Changed** -- the README is a landing page now, and Markdown rather than reStructuredText. `README.rst` (424 lines) became `README.md` (103), keeping what a reader arriving from PyPI or a search result actually needs -- what the library is, why it exists rather than Scapy or DPKT, how to install it, a worked example, and where the documentation lives -- and dropping the technical detail the documentation already carried. **Module Structure**, **Engine Comparison**, **Engine support by Python version**, **Test Environment**, **Test Results** and **Installation Notes** were each already duplicated in `docs/source/index.rst`, in a fuller form, so they are linked rather than restated. Two blocks existed nowhere else and moved rather than going: **Testing** is now `docs/source/testing.rst`, registered in the index toctree, and the `pipenv` and `make setup` local development block joined the Installation section of `docs/source/index.rst`. Requested by the project owner, and a deliberate exception to the convention that documentation here is reStructuredText -- for the README only, since it is the one documentation file whose renderers are GitHub and PyPI rather than Sphinx. Accordingly `setup.py` reads `README.md` and declares its content type as `text/markdown`, and the `include README.md` line in `MANIFEST.in` is now the only thing that puts the README in a source distribution, because `global-include *.rst` no longer matches it -- which matters, since `setup.py` reads the file unguarded and an sdist without it cannot be installed. Verified with `twine check --strict` against a built sdist and wheel, both of which pass. The rename's own references moved with it, since a change that renames a file owns the references to it: `examples/benchmark/Dockerfile` copies `README.md` -- a literal `COPY` of the old name would have failed the layer outright and taken `make bench`, `make bench-quick` and `run.sh` with it -- and the benchmark harness prose that named the root README as the destination of its generated tables now names `docs/source/index.rst`, which is where those tables went. That covers the `Makefile` comment, `report.py`, `test_harness.py`, `run.sh` and the suite's own README, including the one place whose stated reason had inverted: the emitted markup is kept parseable by plain docutils, which is now a conservative choice rather than a hard requirement, because the page it lands on is rendered by Sphinx. `examples/benchmark/benchmark.py` still says `README.rst` and is left alone, because it means the benchmark suite's own README in the same directory, not the project's. +- **Changed** -- the README is a landing page now, and Markdown rather than reStructuredText. `README.rst` (424 lines) became `README.md` (103), keeping what a reader arriving from PyPI or a search result actually needs -- what the library is, why it exists rather than Scapy or DPKT, how to install it, a worked example, and where the documentation lives -- and dropping the technical detail the documentation already carried. **Module Structure**, **Engine Comparison**, **Engine support by Python version**, **Test Environment**, **Test Results** and **Installation Notes** were each already duplicated in `docs/source/index.rst`, in a fuller form, so they are linked rather than restated. Two blocks existed nowhere else and moved rather than going: **Testing** is now `docs/source/testing.rst`, registered in the index toctree, and the `pipenv` and `make setup` local development block joined the Installation section of `docs/source/index.rst`. Requested by the project owner, and a deliberate exception to the convention that documentation here is reStructuredText -- for the README only, since it is the one documentation file whose renderers are GitHub and PyPI rather than Sphinx. Accordingly `setup.py` reads `README.md` and declares its content type as `text/markdown`, and `MANIFEST.in` gains an `include README.md` line because `global-include *.rst` no longer matches the file. That line is belt-and-braces rather than load-bearing, contrary to what this entry claimed when it was written: setuptools' own `sdist` command ships whichever of `README`, `README.rst`, `README.txt` and `README.md` exists, before `MANIFEST.in` is read at all, so deleting the line leaves the sdist's file listing byte-identical -- 861 entries either way, with an empty `diff` -- and the result still installs. `setup.py` does read the file unguarded, but it reads it from wherever `setup.py` is executing, which when `pip` installs an sdist is the unpacked sdist rather than a checkout, so that read cannot be made to fail by dropping the line either. Of the `include` lines in that file only `CHANGELOG.md` is load-bearing, which the #631 entry below measured independently. Verified with `twine check --strict` against a built sdist and wheel, both of which pass. The rename's own references moved with it, since a change that renames a file owns the references to it: `examples/benchmark/Dockerfile` copies `README.md` -- a literal `COPY` of the old name would have failed the layer outright and taken `make bench`, `make bench-quick` and `run.sh` with it -- and the benchmark harness prose that named the root README as the destination of its generated tables now names `docs/source/index.rst`, which is where those tables went. That covers the `Makefile` comment, `report.py`, `test_harness.py`, `run.sh` and the suite's own README, including the one place whose stated reason had inverted: the emitted markup is kept parseable by plain docutils, which is now a conservative choice rather than a hard requirement, because the page it lands on is rendered by Sphinx. `examples/benchmark/benchmark.py` still says `README.rst` and is left alone, because it means the benchmark suite's own README in the same directory, not the project's (#619). - **Changed** -- `CODE_OF_CONDUCT.md` moves from Contributor Covenant 1.4 to Contributor Covenant 3.0, at the maintainer's request. The text is the canonical 3.0 Markdown fetched from https://www.contributor-covenant.org/version/3/0/code_of_conduct/code_of_conduct.md rather than a transcription, so the pledge, the encouraged and restricted behaviours and the scope are unaltered. Three things needed deciding rather than copying. 3.0 ships two `[NOTE` placeholders an adopter must fill: the reporting channel, which now names `jarryshaw@icloud.com` -- the same contact 1.4 carried and the one `SECURITY.md` already points at as its email fallback -- plus GitHub's report-abuse form for the case a single-maintainer project cannot otherwise cover, a report about the maintainer; and the enforcement section, whose placeholder is an instruction to the adopter and is removed. 3.0 then assigns enforcement throughout to plural "Community Moderators" (and once, inconsistently, to "Community Managers"), which this repository does not have, so all eight occurrences become the singular maintainer. The four-rung ladder -- Warning, Temporarily Limited Activities, Temporary Suspension, Permanent Ban -- is offered as a suggestion and is **kept**, because each rung maps onto a lever one person actually holds on GitHub: a private message, a locked thread, an interaction limit or block, a permanent block. Finally, 3.0 is licensed CC BY-SA 4.0 where 1.4's attribution paragraph carried no licence notice at all, so the attribution now names version 3.0, links the permanent `version/3/0/` URL, carries the CC BY-SA 4.0 notice and link, indicates that changes were made as BY requires, and says explicitly that the share-alike term covers this document only -- the code remains BSD-3-Clause and `LICENSE` is untouched. Rendering was checked against GitHub's own Markdown API rather than assumed: the ladder comes back as four list items each nesting three, which is what #613 had to repair in the 1.4 file when a stray list marker collapsed the whole document into one nested item (#624). - **Fixed** -- the one assertion #604 left pinning the old padding side, which had been red on `mainline` since #621 merged. `TCPUDPUnitTests.test_a_truncated_option_still_parses_its_declared_length` expected a truncated TCP option's `data` as the synthesised zero octets *followed by* the real ones, which is what `rjust()` produced; #621 made the padding `ljust()` everywhere but could not retarget this file, since another change (#612) owned it at the time and editing it concurrently risked discarding work that has since landed. The real octets now come first for both parametrised widths, and the docstring above the assertion says tail-padding rather than left-padding. Test-only: no library code changes, and the sibling case in `tests/protocols/internet/test_ipv4_unit.py` was already retargeted in #621. Measured against `main` at `2221c2d8f`: two subtest failures before, none after (#604). - **Fixed** -- `main` went red the moment #604's `ljust()` landed, because `TCPUDPUnitTests.test_a_truncated_option_still_parses_its_declared_length` still pinned the head-padded short read that fix removed. The `Reserved_79` option declaring `length=12` over 6 real octets now reports `aabbccddeeff00000000` where the test expected `00000000aabbccddeeff`, so both subtests -- `declared_length=12` and `=32` -- failed on that one assertion while the 17 other cases in the file stayed green: the parse itself never changed, only which end the synthesised zeros sit at. The expectation is inverted, and the docstring above it -- which said the short read was *left*-padded and described the value as four zero octets followed by the six real ones -- is corrected to match, since a docstring that contradicts its own assertion is how the stale expectation survived in the first place. The inputs do discriminate: `trailing` is non-zero and the pad width is 4 and 24, so neither subtest would hold under the other order. #621 left this file alone deliberately, because #612 owned it at the time, and merged two minutes ahead of the cross-review verdict that named it (#604, #621). @@ -98,6 +98,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- a flag value with no declared bits no longer dumps as `Type::None [0]`. **This changes user-visible output in every textual dump format.** `make_dumper`'s `object_hook` renders each enumeration member as `Type::name [value]` and interpolated the name unguarded, but a `Flag` value composed entirely of undeclared bits has `name is None` rather than a string -- so the literal four characters `None` landed in the name half and `Flags(0)` rendered as `Flags::None [0]` in `json`, `tree`, `text`, `txt`, `plist` and `xml`, out of both `Extractor` and `TraceFlow`. A nameless member now renders with the value's own decimal spelling instead, so `Flags(0)` gives `Flags::0 [0]` and `Flags(8)` gives `Flags::8 [8]`. That is the spelling the enumeration libraries already use for an undeclared residue -- `Flags(2057).name` is `ACK|9`, naming the declared bit and giving the leftovers as one decimal number -- and it cannot be mistaken for a member name, since a Python identifier may not begin with a digit, whereas `NONE` is a real declared name elsewhere in the library. Three sites carried the identical interpolation, not one: the `OrderedMultiDict` key path, the `addon` branch and the scalar return, which now share one `render_enum` helper so the guard cannot be applied to two of three. That guard is on `name is None` and not on the value, because the defect never was about zero -- `Flags(1)`, `Flags(8)`, `Flags(9)` and `Flags(65536)` are equally nameless -- and it is not an `aenum` quirk either, since a stdlib `enum.IntFlag` answers `name is None` at the same values. Five of the library's seven flag registries are nameless at zero rather than the one reported: `Flags` plus the four Mobility Header flag registries, the other two declaring an explicit `undefined = 0`. Pre-existing since 2023-04-28 and surfaced rather than caused by #634, which made a flagless TCP segment reach this path. The committed example dumps do not move, because no member in them lacks a name (#648). - **Fixed** -- four MPTCP error messages carried a doubled separator, rendering as `TCP: : [OptNo 30] 1: invalid flags combination` with an empty field between the protocol alias and the option number. Cosmetic: the exception type, the option number and the subtype were always right, and nothing downstream parses these strings, but it is the text a user sees when an MP_JOIN or DSS option is rejected and an empty field reads as a value that failed to interpolate. Four sites, not the one reported -- `_read_mptcp_join` and `_make_mptcp_join` for the invalid flag combination, and both guards of `_make_mptcp_dss` for the missing required fields -- against 28 messages in the same file that already spelled the prefix correctly, so the four were outliers against a house form in their own module and the counts are now 32 correct against 0 doubled. Pre-existing since 2023-04-10. The new test asserts each message in full rather than by substring, which is what the existing `assertIn` on the message body could not do, since it passes either side of the change (#649). - **Fixed** -- the release workflow published to PyPI and to Anaconda with nothing a human had to approve. `create-release.yml`'s `pypi` job carried its `environment: release` commented out while the `id-token: write` from the same upstream snippet had been re-added live below it, so the block read as disabled as a unit when only its gate still was; the `conda` job had no `environment:` at all, not even a commented one. Neither needed a tag either: the workflow also fires on the completion of `Vendor Update`, so a scheduled registry crawl that bumped the version was enough to publish to two public indexes, at a median three minutes from bump to upload. Four jobs reach outside the run and all four are now gated -- `github` on `github-release` for the GitHub Release and the `v*` tag it creates, `tag` on `conda-tag` for the commit it pushes to `main`, `pypi` on `pypi`, and `conda` on `anaconda`. One environment per credential rather than one shared `release`, because approval is granted to an environment and not to a job, so a single `release` approved once would release both indexes together. PyPI is the one that has to be answerable alone: a version number it has accepted cannot be reused, and `skip-existing: true` makes the re-upload *succeed* having published nothing, so an unintended publish permanently consumes the number the intended release wanted. The workflow half is only half the fix -- an `environment:` naming an environment that does not exist is created implicitly with no protection rules and the job proceeds unapproved, so this is inert until each of the four exists in repository settings with a required reviewer on it, and each one's "Deployment branches and tags" has to stay unrestricted or the `tags: v*` trigger fails outright instead of pausing. `github-pages` is the standing example of that failure in this repository, carrying no protection rule since 2021. The new test asserts the general rule rather than the current file -- no job running a publishing action or a `git push` may omit an `environment:` -- so a publishing job added later without a gate fails rather than ships (#641). +- **Fixed** -- three names appeared in string annotations that their own module never imported, so a type checker or a documentation build could not resolve them while every test went on passing: `Any` in `pcapkit/utilities/logging.py`, and in `pcapkit/protocols/schema/internet/ipv6_route.py` both `Protocol` in a `payload:` stub and `Optional` inside a `typing.cast`. All three are now in their module's `TYPE_CHECKING` block, `Protocol` spelled `ProtocolBase as Protocol` the way twenty-one sibling schema modules already spell it in that same stub, which makes twenty-two. The `cast` case is the one worth naming: `typing.cast` never evaluates its first argument, so no test that runs the line can see a bad name in it. Nothing resolves at runtime that did not resolve before -- `TYPE_CHECKING` is `False` when the interpreter runs, so a name imported under it is absent from the module namespace either way, and making these annotations resolve at runtime is a separate decision about the idiom rather than a missing import. mypy 2.3.1 over `pcapkit` reported three `name-defined` errors before and none after, its total moving 115 to 112, so nothing else shifted. A new `tests/project/test_annotation_names.py` pins the invariant without needing a type checker installed: it resolves every string annotation in the package against the names its own module binds, following a nested forward reference such as `'list["Nested"]'` while treating `Literal` members and `Annotated` metadata as the values they are, and it reports exactly those three findings on the unfixed tree and none after. That module reaches the two PEP 695 node classes it needs through `getattr` rather than naming them, because `ast.TypeAlias` and `ast.TypeVar` arrived in Python 3.12 while the supported range starts at 3.10. The same change corrects the `MANIFEST.in` comment asserting that `include README.md` was the only thing putting the README in a source distribution and that an sdist without it could not be installed; both halves are false, and the #619 entry above carried the identical claim and is corrected with it (#642). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index b2c700898..e3df1f00f 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -211,7 +211,7 @@ pull requests between #326 and #509. :rfc:`3931` L2TPv3-over-IP header with no class to dispatch to, so the declaration would have pointed the :rfc:`2661` parser at it. See the corresponding **Fixed** entry below; the mechanism itself is unaffected, and - its worked example now names ``L2TPv3`` rather than ``L2TPv2``. + its worked example now names ``L2TPv3`` rather than ``L2TPv2`` (#570). * **Changed** -- extraction is around 46% faster on a 1,117-frame HTTP capture, with byte-identical output (#420). A reassembled datagram's payload is now analysed on first read rather than eagerly, which cuts IP reassembly's own @@ -347,7 +347,7 @@ pull requests between #326 and #509. so nothing pinned that silence against a future regression; and ``register_extractor_engine``'s real keyword, ``name``, was not itself under test, only its already-corrected docstring, so a future rename could - put the two out of step again exactly as quietly as before. + put the two out of step again exactly as quietly as before (#577). * **Fixed** -- ``FieldBase.unpack`` zero-padded straight up to a field's declared ``length`` with ``rjust()``, regardless of how little data ``buffer`` actually held; a ~40-octet PCAP-NG Decryption Secrets Block with a @@ -958,11 +958,19 @@ pull requests between #326 and #509. reStructuredText -- for the README only, since it is the one documentation file whose renderers are GitHub and PyPI rather than Sphinx. Accordingly ``setup.py`` reads ``README.md`` and declares its content type as - ``text/markdown``, and the ``include README.md`` line in ``MANIFEST.in`` is now - the only thing that puts the README in a source distribution, because - ``global-include *.rst`` no longer matches it -- which matters, since - ``setup.py`` reads the file unguarded and an sdist without it cannot be - installed. Verified with ``twine check --strict`` against a built sdist and + ``text/markdown``, and ``MANIFEST.in`` gains an ``include README.md`` line + because ``global-include *.rst`` no longer matches the file. That line is + belt-and-braces rather than load-bearing, contrary to what this entry claimed + when it was written: setuptools' own ``sdist`` command ships whichever of + ``README``, ``README.rst``, ``README.txt`` and ``README.md`` exists, before + ``MANIFEST.in`` is read at all, so deleting the line leaves the sdist's file + listing byte-identical -- 861 entries either way, with an empty ``diff`` -- and + the result still installs. ``setup.py`` does read the file unguarded, but it + reads it from wherever ``setup.py`` is executing, which when ``pip`` installs an + sdist is the unpacked sdist rather than a checkout, so that read cannot be made + to fail by dropping the line either. Of the ``include`` lines in that file only + ``CHANGELOG.md`` is load-bearing, which the #631 entry below measured + independently. Verified with ``twine check --strict`` against a built sdist and wheel, both of which pass. The rename's own references moved with it, since a change that renames a file owns the references to it: ``examples/benchmark/Dockerfile`` copies ``README.md`` -- a literal ``COPY`` of @@ -976,7 +984,7 @@ pull requests between #326 and #509. choice rather than a hard requirement, because the page it lands on is rendered by Sphinx. ``examples/benchmark/benchmark.py`` still says ``README.rst`` and is left alone, because it means the benchmark suite's own README in the same - directory, not the project's. + directory, not the project's (#619). * **Changed** -- ``CODE_OF_CONDUCT.md`` moves from Contributor Covenant 1.4 to Contributor Covenant 3.0, at the maintainer's request. The text is the canonical 3.0 Markdown fetched from @@ -1674,6 +1682,36 @@ pull requests between #326 and #509. rather than the current file -- no job running a publishing action or a ``git push`` may omit an ``environment:`` -- so a publishing job added later without a gate fails rather than ships (#641). +* **Fixed** -- three names appeared in string annotations that their own module + never imported, so a type checker or a documentation build could not resolve + them while every test went on passing: ``Any`` in + ``pcapkit/utilities/logging.py``, and in + ``pcapkit/protocols/schema/internet/ipv6_route.py`` both ``Protocol`` in a + ``payload:`` stub and ``Optional`` inside a ``typing.cast``. All three are now + in their module's ``TYPE_CHECKING`` block, ``Protocol`` spelled + ``ProtocolBase as Protocol`` the way twenty-one sibling schema modules already + spell it in that same stub, which makes twenty-two. The ``cast`` case is the + one worth naming: ``typing.cast`` never evaluates its first argument, so no + test that runs the line can see a bad name in it. Nothing resolves at runtime + that did not resolve before -- ``TYPE_CHECKING`` is ``False`` when the + interpreter runs, so a name imported under it is absent from the module + namespace either way, and making these annotations resolve at runtime is a + separate decision about the idiom rather than a missing import. mypy 2.3.1 over + ``pcapkit`` reported three ``name-defined`` errors before and none after, its + total moving 115 to 112, so nothing else shifted. A new + ``tests/project/test_annotation_names.py`` pins the invariant without needing a + type checker installed: it resolves every string annotation in the package + against the names its own module binds, following a nested forward reference + such as ``'list["Nested"]'`` while treating ``Literal`` members and + ``Annotated`` metadata as the values they are, and it reports exactly those + three findings on the unfixed tree and none after. That module reaches the two + PEP 695 node classes it needs through ``getattr`` rather than naming them, + because ``ast.TypeAlias`` and ``ast.TypeVar`` arrived in Python 3.12 while the + supported range starts at 3.10. The same change corrects the ``MANIFEST.in`` + comment asserting that ``include README.md`` was the only thing putting the + README in a source distribution and that an sdist without it could not be + installed; both halves are false, and the #619 entry above carried the + identical claim and is corrected with it (#642). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 634877971e671d8754c0edebe6f04428f1ec9917 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 16:24:13 -0400 Subject: [PATCH 15/46] docs(changelog): the 1.5.0 entry for #594, whose code ships in #676 Closes the residual the #573 entry above records under "What this does not close": the 16-bit band, where a shortfall of 65,536 octets or fewer is padded unconditionally and charged to nothing, so an option declaring 65,535 octets could be repeated per block without limit. `util/changelog_md.py` regenerated `CHANGELOG.md`; `--check` exits 0. --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 34 +++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 62684c083..96535e1aa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -99,6 +99,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- four MPTCP error messages carried a doubled separator, rendering as `TCP: : [OptNo 30] 1: invalid flags combination` with an empty field between the protocol alias and the option number. Cosmetic: the exception type, the option number and the subtype were always right, and nothing downstream parses these strings, but it is the text a user sees when an MP_JOIN or DSS option is rejected and an empty field reads as a value that failed to interpolate. Four sites, not the one reported -- `_read_mptcp_join` and `_make_mptcp_join` for the invalid flag combination, and both guards of `_make_mptcp_dss` for the missing required fields -- against 28 messages in the same file that already spelled the prefix correctly, so the four were outliers against a house form in their own module and the counts are now 32 correct against 0 doubled. Pre-existing since 2023-04-10. The new test asserts each message in full rather than by substring, which is what the existing `assertIn` on the message body could not do, since it passes either side of the change (#649). - **Fixed** -- the release workflow published to PyPI and to Anaconda with nothing a human had to approve. `create-release.yml`'s `pypi` job carried its `environment: release` commented out while the `id-token: write` from the same upstream snippet had been re-added live below it, so the block read as disabled as a unit when only its gate still was; the `conda` job had no `environment:` at all, not even a commented one. Neither needed a tag either: the workflow also fires on the completion of `Vendor Update`, so a scheduled registry crawl that bumped the version was enough to publish to two public indexes, at a median three minutes from bump to upload. Four jobs reach outside the run and all four are now gated -- `github` on `github-release` for the GitHub Release and the `v*` tag it creates, `tag` on `conda-tag` for the commit it pushes to `main`, `pypi` on `pypi`, and `conda` on `anaconda`. One environment per credential rather than one shared `release`, because approval is granted to an environment and not to a job, so a single `release` approved once would release both indexes together. PyPI is the one that has to be answerable alone: a version number it has accepted cannot be reused, and `skip-existing: true` makes the re-upload *succeed* having published nothing, so an unintended publish permanently consumes the number the intended release wanted. The workflow half is only half the fix -- an `environment:` naming an environment that does not exist is created implicitly with no protection rules and the job proceeds unapproved, so this is inert until each of the four exists in repository settings with a required reviewer on it, and each one's "Deployment branches and tags" has to stay unrestricted or the `tags: v*` trigger fails outright instead of pausing. `github-pages` is the standing example of that failure in this repository, carrying no protection rule since 2021. The new test asserts the general rule rather than the current file -- no job running a publishing action or a `git push` may omit an `environment:` -- so a publishing job added later without a gate fails rather than ships (#641). - **Fixed** -- three names appeared in string annotations that their own module never imported, so a type checker or a documentation build could not resolve them while every test went on passing: `Any` in `pcapkit/utilities/logging.py`, and in `pcapkit/protocols/schema/internet/ipv6_route.py` both `Protocol` in a `payload:` stub and `Optional` inside a `typing.cast`. All three are now in their module's `TYPE_CHECKING` block, `Protocol` spelled `ProtocolBase as Protocol` the way twenty-one sibling schema modules already spell it in that same stub, which makes twenty-two. The `cast` case is the one worth naming: `typing.cast` never evaluates its first argument, so no test that runs the line can see a bad name in it. Nothing resolves at runtime that did not resolve before -- `TYPE_CHECKING` is `False` when the interpreter runs, so a name imported under it is absent from the module namespace either way, and making these annotations resolve at runtime is a separate decision about the idiom rather than a missing import. mypy 2.3.1 over `pcapkit` reported three `name-defined` errors before and none after, its total moving 115 to 112, so nothing else shifted. A new `tests/project/test_annotation_names.py` pins the invariant without needing a type checker installed: it resolves every string annotation in the package against the names its own module binds, following a nested forward reference such as `'list["Nested"]'` while treating `Literal` members and `Annotated` metadata as the values they are, and it reports exactly those three findings on the unfixed tree and none after. That module reaches the two PEP 695 node classes it needs through `getattr` rather than naming them, because `ast.TypeAlias` and `ast.TypeVar` arrived in Python 3.12 while the supported range starts at 3.10. The same change corrects the `MANIFEST.in` comment asserting that `include README.md` was the only thing putting the README in a source distribution and that an sdist without it could not be installed; both halves are false, and the #619 entry above carried the identical claim and is corrected with it (#642). +- **Fixed** -- the 16-bit band the #573 entry above records as deliberately left open. `FieldBase.unpack` pads a shortfall of 65,536 octets or fewer unconditionally and charges it to nothing, which is load-bearing for a capture cut short by its snapshot length, so a length declared by a 16-bit wire field could still be repeated without limit. The crafted 80,048-octet PCAP-NG of 2,000 Enhanced Packet Blocks that entry describes, each carrying one option declaring 65,535 octets against none present, synthesised 131,070,000 octets of zeros -- 1637.393x its own size, linear in the block count and so unbounded in the input -- and now synthesises none, with all 2,000 frames and all 2,000 options still parsed. The bound is taken one layer up, where the information the field layer lacks already exists: a PCAP-NG block's Block Total Length is authoritative and cross-checked against its own trailing copy, so the option area is that length less the fixed fields, `captured_len` and `captured_len`'s padding, and an option declaring more payload than that area has left is malformed however complete the file behind it is. A snapshot-truncated capture says so through `captured_len` instead and leaves its options whole, so it never trips this -- which is exactly what the #571 `len(buffer) < length` rejection could not distinguish, and what it was declined for. A new `bounded_option` clamps the payload to the octets the area has left at that field and warns with `SchemaWarning`; it is applied to all fifteen variable-width option and record payloads in the PCAP-NG schema, and deliberately not to the block-level fields, which are the 32-bit band the running ledger already budgets. Clamping rather than refusing is what keeps the #431 accommodation, since a PCAP-NG block read has no catch point above `FieldBase.unpack` and one refusal would abort a whole extraction rather than one block; and the clamp reads only the block's own declared framing, never a running total, so byte-identical input answers identically whatever preceded it. Zero clamps fired across all six PCAP-NG sample captures, 338 options between them, and 101 truncation levels of `dhcp.pcapng` gave byte-identical results including the failures. The bound is pinned as a property rather than as one input: every combination of one to three options against nine declared lengths asserts that the octets a block's options report holding never exceed the area, and the amplification ratio is asserted flat across 1, 8, 64 and 512 blocks, which is what separates a bound from a smaller constant (#594). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index e3df1f00f..363444abb 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1712,6 +1712,40 @@ pull requests between #326 and #509. README in a source distribution and that an sdist without it could not be installed; both halves are false, and the #619 entry above carried the identical claim and is corrected with it (#642). +* **Fixed** -- the 16-bit band the #573 entry above records as deliberately left + open. ``FieldBase.unpack`` pads a shortfall of 65,536 octets or fewer + unconditionally and charges it to nothing, which is load-bearing for a capture + cut short by its snapshot length, so a length declared by a 16-bit wire field + could still be repeated without limit. The crafted 80,048-octet PCAP-NG of + 2,000 Enhanced Packet Blocks that entry describes, each carrying one option + declaring 65,535 octets against none present, synthesised 131,070,000 octets of + zeros -- 1637.393x its own size, linear in the block count and so unbounded in + the input -- and now synthesises none, with all 2,000 frames and all 2,000 + options still parsed. The bound is taken one layer up, where the information + the field layer lacks already exists: a PCAP-NG block's Block Total Length is + authoritative and cross-checked against its own trailing copy, so the option + area is that length less the fixed fields, ``captured_len`` and + ``captured_len``'s padding, and an option declaring more payload than that area + has left is malformed however complete the file behind it is. A + snapshot-truncated capture says so through ``captured_len`` instead and leaves + its options whole, so it never trips this -- which is exactly what the #571 + ``len(buffer) < length`` rejection could not distinguish, and what it was + declined for. A new ``bounded_option`` clamps the payload to the octets the + area has left at that field and warns with ``SchemaWarning``; it is applied to + all fifteen variable-width option and record payloads in the PCAP-NG schema, + and deliberately not to the block-level fields, which are the 32-bit band the + running ledger already budgets. Clamping rather than refusing is what keeps the + #431 accommodation, since a PCAP-NG block read has no catch point above + ``FieldBase.unpack`` and one refusal would abort a whole extraction rather than + one block; and the clamp reads only the block's own declared framing, never a + running total, so byte-identical input answers identically whatever preceded + it. Zero clamps fired across all six PCAP-NG sample captures, 338 options + between them, and 101 truncation levels of ``dhcp.pcapng`` gave byte-identical + results including the failures. The bound is pinned as a property rather than + as one input: every combination of one to three options against nine declared + lengths asserts that the octets a block's options report holding never exceed + the area, and the amplification ratio is asserted flat across 1, 8, 64 and 512 + blocks, which is what separates a bound from a smaller constant (#594). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 4ed5d4a543a54617aa30e40c4f90674230c43a81 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 16:27:25 -0400 Subject: [PATCH 16/46] docs(changelog): the 1.5.0 entry for #647, whose code ships in #677 Records the six constant registries that rejected an invalid value in a way the built-in `enum` does not -- three composing a pseudo-member for any integer at all -- and why the issue's own proposal to raise `EnumError` was rejected rather than adopted. Also corrects the #623 entry above, which stated that `pcapkit/const/tcp/flags.py` "defines no `_missing_`". It defined none at the time and never had #623's recursion defect, which is what that sentence was about, but it has one now; the tense is fixed and the new entry named. `CHANGELOG.md` regenerated with `python util/changelog_md.py`; `--check` exits 0. --- CHANGELOG.md | 4 ++- docs/source/changelog/1.5.0.rst | 55 +++++++++++++++++++++++++++++++-- 2 files changed, 56 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 96535e1aa..7c2625658 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -78,7 +78,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- `main` went red the moment #604's `ljust()` landed, because `TCPUDPUnitTests.test_a_truncated_option_still_parses_its_declared_length` still pinned the head-padded short read that fix removed. The `Reserved_79` option declaring `length=12` over 6 real octets now reports `aabbccddeeff00000000` where the test expected `00000000aabbccddeeff`, so both subtests -- `declared_length=12` and `=32` -- failed on that one assertion while the 17 other cases in the file stayed green: the parse itself never changed, only which end the synthesised zeros sit at. The expectation is inverted, and the docstring above it -- which said the short read was *left*-padded and described the value as four zero octets followed by the six real ones -- is corrected to match, since a docstring that contradicts its own assertion is how the stale expectation survived in the first place. The inputs do discriminate: `trailing` is non-zero and the pad width is 4 and 24, so neither subtest would hold under the other order. #621 left this file alone deliberately, because #612 owned it at the time, and merged two minutes ahead of the cross-review verdict that named it (#604, #621). - **Fixed** -- `util/bump_version.py` left `CITATION.cff` naming the previous release. Nothing else in the repository maintains that file -- no workflow, hook or packaging file mentions it -- so every bump since it landed in #615 would have stranded the `version` and `date-released` it renders as GitHub's "Cite this repository" button and that citation managers, Zenodo and dependency inventories read directly. Both fields now move with `__version__`. They have the same standing, since the file's own header says both describe the newest *published* release, and moving only one would assert that 1.5.0b5 was released on the day 1.5.0b4 was; the date is taken in UTC, because seven of the thirty most recent bumps were made late evening in US-Eastern where a local date is a day behind the publish it describes. That the two are the same day at all is measured rather than assumed: the bump is what triggers `create-release.yml`, the median gap to the PyPI upload is three minutes, and the UTC calendar dates agree 30 times out of 30. The rewrite is line-oriented, so the comment header, key ordering and each field's existing quoting survive -- `cff-version` and a `references` entry's own `version` are anchored out at column zero -- and the result is checked with `cffconvert --validate`. An absent file is reported on stderr and skipped rather than failing the vendor cron before its `git commit`, which would discard the whole registry crawl for the sake of a documentation file; a file present with no `version` field raises instead, before anything is written, because rewriting nothing while reporting success is the staleness this fixes. Two things came with it. The script gains a `main()` guard, having previously run the entire bump at import, which is why it had no testable surface; and the `import pcapkit` fallback in its version reader, which returned `"1.5.0b4'\n"` -- closing quote and newline included, which `packaging` rejects -- is fixed, a path that had never worked and went unnoticed because the only caller installs the package first. A new gate asserts the committed file still names the packaged version, covering the version changes made by hand, which never run this script at all -- 40 of the 159 commits that have moved `__version__` on `main`, a quarter over the project's life and 11 of the most recent 25 (#625). - **Fixed** -- `SeekableReader.truncate` put its padding where the reader's own bookkeeping says the content is, and `read` then returned one octet fewer than the buffer held. Same family as #604, but two defects rather than one, and the padding side is only half of it. The buffer keeps its content at `[0:_buffer_cur]` with unwritten padding behind it, so a reduction has to keep the octets it has and an extension has to append at the *tail* -- the latter is what `io.IOBase.truncate` means by "the contents of the new file area", the area past the old end. It did neither: `temp[-size:]` sliced the buffer rather than the content, so it kept the trailing padding and discarded the octets actually read, and `temp.rjust(size)` prefixed the new zeros, displacing the content past where `_buffer_set` and `_buffer_cur` address it. Separately, `read` capped a buffered read at `min(size, self._buffer_cur - 1)`, a count less one measured from the start of the buffer rather than the run remaining from the position being read from -- one octet short at the start of the buffer, and reaching past the content into the padding anywhere further in. The shortfall was then made up from the stream, *past* the octet that had been skipped, which both dropped that octet and left the return short. The reported symptom needed both: `read(4)`, `truncate(8)`, `seek(0)`, `read(8)` over `b'abcde'` returned `b'\x00\x00\x00e'`, four octets of an eight octet request with three of them padding, and returns `b'abcde'` now -- five being the whole of what a five octet stream can answer with. Three more of the method's contract were wrong and are fixed with it: the position was reset to the start of the buffer rather than left alone, so a read after a truncation resumed from the wrong octet; an omitted `size` resized to `0` rather than to the current position; and `_buffer_cur` was left addressing octets a reduced buffer no longer had, so the next read raised `ValueError: memoryview assignment: lvalue and rvalue have different structures` from `_write_buffer` rather than returning anything. `truncate(0)` raised the same `ValueError` by a second route, and `_write_buffer` is fixed with it: `buf[-self._buffer_size:]` is `buf[-0:]` for a buffer of no size at all -- the whole of the octets just read rather than none of them -- so it is now counted from the front. That state is reachable only through `truncate`, since the constructor refuses a non-positive `buffer_size`. A reduction also advances `_buffer_set` past the octets it drops, keeping `_buffer_set + _buffer_cur` equal to how far the stream has been consumed, which `seek` reads as its licence to fetch more. Clamping `_buffer_cur` alone made that sum *under*-report the stream, and the next forward `seek` then spliced in octets from the wrong absolute offset and said nothing: `read(8)`, `truncate(3)`, `seek(6)`, `read(1)` over `b'abcdefghijklmnop'` returned `b'l'` where `b'g'` is the octet at offset 6 -- worse than the `ValueError` it replaced, being silent. All three came out of fuzzing random operation sequences against the buffer's own invariants, none from reading the code: of 3000 rounds, 2372 left the bookkeeping inconsistent before and 683 raised an undocumented exception, and none of either do now. Latent in this library rather than live -- nothing here calls `truncate`, confirmed by grep, though it is public on a public class -- and invisible to the existing tests, which asserted the return value and never the content (#622). -- **Fixed** -- `_missing_` in the four Mobility Header flag enumerations ended in `return cls(value)`, the same constructor that had just failed to find the value, so every in-range value that is not already a member re-entered `_missing_` unbounded and raised `RecursionError`. `BindingACKFlag`, `BindingUpdateFlag`, `HandoverACKFlag` and `HandoverInitiateFlag` are `IntFlag` types whose members are single bits, so the hole was not some exotic bit pattern: `F(0)` -- no flags set, the most ordinary value a flag octet can carry -- recursed on all four, and so did every composite of two defined bits, such as `BindingACKFlag(0x06)`. Defining `_missing_` at all is what caused it, because it shadowed the `aenum` `Flag` machinery that resolves exactly those values; `pcapkit/const/tcp/flags.py` defines no `_missing_` and has never had the defect. All four now end in `return super()._missing_(value)`, which is what `pcapkit/vendor/default.py` emits for every other generated enumeration and what 75 of the 117 modules under `pcapkit/const/` already do, so `F(0)` is an empty flag and `BindingACKFlag(0x06)` is `S|D`. The `extend_enum` idiom that `pcapkit/const/pcapng/record_type.py` and `secrets_type.py` use to mint a member for an unassigned integer -- the only other two modules whose `_missing_` ends in `return cls(value)`, and which do not recurse precisely because that `extend_enum` runs first -- was considered and rejected for a flag type: naming `0x06` `Unassigned_0x06` would hide the composite and pollute `_member_map_` with an entry per bit pattern, up to 65536 of them for `BindingUpdateFlag`. The range guard above it is untouched, so an out-of-range or non-integer value is still the same `ValueError`. Fixed in the four `pcapkit/vendor/mh/` templates and regenerated, the `pcapkit/const/mh/` modules being generated output that the next crawl would otherwise revert (#623). +- **Fixed** -- `_missing_` in the four Mobility Header flag enumerations ended in `return cls(value)`, the same constructor that had just failed to find the value, so every in-range value that is not already a member re-entered `_missing_` unbounded and raised `RecursionError`. `BindingACKFlag`, `BindingUpdateFlag`, `HandoverACKFlag` and `HandoverInitiateFlag` are `IntFlag` types whose members are single bits, so the hole was not some exotic bit pattern: `F(0)` -- no flags set, the most ordinary value a flag octet can carry -- recursed on all four, and so did every composite of two defined bits, such as `BindingACKFlag(0x06)`. Defining `_missing_` at all is what caused it, because it shadowed the `aenum` `Flag` machinery that resolves exactly those values; `pcapkit/const/tcp/flags.py` defined no `_missing_` at the time and never had the defect -- it has one now, added by #647 below, ending in the same `super()._missing_(value)` tail for the same reason. All four now end in `return super()._missing_(value)`, which is what `pcapkit/vendor/default.py` emits for every other generated enumeration and what 75 of the 117 modules under `pcapkit/const/` already do, so `F(0)` is an empty flag and `BindingACKFlag(0x06)` is `S|D`. The `extend_enum` idiom that `pcapkit/const/pcapng/record_type.py` and `secrets_type.py` use to mint a member for an unassigned integer -- the only other two modules whose `_missing_` ends in `return cls(value)`, and which do not recurse precisely because that `extend_enum` runs first -- was considered and rejected for a flag type: naming `0x06` `Unassigned_0x06` would hide the composite and pollute `_member_map_` with an entry per bit pattern, up to 65536 of them for `BindingUpdateFlag`. The range guard above it is untouched, so an out-of-range or non-integer value is still the same `ValueError`. Fixed in the four `pcapkit/vendor/mh/` templates and regenerated, the `pcapkit/const/mh/` modules being generated output that the next crawl would otherwise revert (#623). - **Fixed** -- the HIP `SOLUTION` builder sized the parameter with `4 + math.ceil(max(random.bit_length(), solution.bit_length()) / 4)`, which is an invalid shorthand for the two fields it actually has to describe. [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5) gives the parameter's `Length` as `4 + RHASH_len / 4` over a `Random #I` and a `Puzzle solution #J` of `RHASH_len / 8` octets **each**, and `/ 4` equals twice `/ 8` only because `RHASH_len` -- the natural output length of a hash function, in bits -- is a whole number of octets. Applied to an arbitrary `bit_length()` the identity fails, and the builder emitted an *odd* contents width, which its own reader then refused: `_read_param_solution`'s `(schema.len - 4) % 2` guard exists because `SolutionParameter` splits that width into two equal `(len - 4) // 2` halves, so an odd width cannot be framed at all. `random=0x1` with `solution=0xfff` declared `len=7` and raised `ProtocolError: HIPv2: [ParamNo 321] invalid format` on input the library had itself produced. The undersized length was the worse half of it: because both fields take their width from that same `len`, `solution=0xfff` was packed into the one octet it allowed and came back as `0xff` -- silent truncation, nothing raised. Fixed to `4 + 2 * math.ceil(max(...) / 8)`, the form the sibling `PUZZLE` builder already uses, which is even by construction and so can never trip the guard. Loosening the reader was rejected as the alternative: an odd contents width has no meaning in the wire format, since the RFC makes the two fields equal-width and says nothing about which would take an extra octet, and a laxer guard would still have truncated the value (#608). - **Added** -- `SOLUTION` parameter width coverage in `tests/protocols/internet/test_hip_unit.py`: eight widths asserting the declared length, the reader's acceptance and the construct-pack-parse cycle, plus a 57-bit case pinning the 20-octet length [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) requires of HIPv1. Five of the eight widths -- 1, 9, 12, 17 and 25 bits -- are deliberately *not* multiples of eight, because at a multiple of eight the defective `ceil(bits / 4)` coincides with the correct width, which is why every fixture that reached this builder passed through it unharmed; the round-trip case in `examples/generators/options.py` supplies no `random` or `solution` at all, so it exercised the formula at zero bits. 57 bits is what discriminates on the HIPv1 path for the same reason 64 does not: both formulas give 20 at a full-width value (#608). @@ -101,6 +101,8 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- three names appeared in string annotations that their own module never imported, so a type checker or a documentation build could not resolve them while every test went on passing: `Any` in `pcapkit/utilities/logging.py`, and in `pcapkit/protocols/schema/internet/ipv6_route.py` both `Protocol` in a `payload:` stub and `Optional` inside a `typing.cast`. All three are now in their module's `TYPE_CHECKING` block, `Protocol` spelled `ProtocolBase as Protocol` the way twenty-one sibling schema modules already spell it in that same stub, which makes twenty-two. The `cast` case is the one worth naming: `typing.cast` never evaluates its first argument, so no test that runs the line can see a bad name in it. Nothing resolves at runtime that did not resolve before -- `TYPE_CHECKING` is `False` when the interpreter runs, so a name imported under it is absent from the module namespace either way, and making these annotations resolve at runtime is a separate decision about the idiom rather than a missing import. mypy 2.3.1 over `pcapkit` reported three `name-defined` errors before and none after, its total moving 115 to 112, so nothing else shifted. A new `tests/project/test_annotation_names.py` pins the invariant without needing a type checker installed: it resolves every string annotation in the package against the names its own module binds, following a nested forward reference such as `'list["Nested"]'` while treating `Literal` members and `Annotated` metadata as the values they are, and it reports exactly those three findings on the unfixed tree and none after. That module reaches the two PEP 695 node classes it needs through `getattr` rather than naming them, because `ast.TypeAlias` and `ast.TypeVar` arrived in Python 3.12 while the supported range starts at 3.10. The same change corrects the `MANIFEST.in` comment asserting that `include README.md` was the only thing putting the README in a source distribution and that an sdist without it could not be installed; both halves are false, and the #619 entry above carried the identical claim and is corrected with it (#642). - **Fixed** -- the 16-bit band the #573 entry above records as deliberately left open. `FieldBase.unpack` pads a shortfall of 65,536 octets or fewer unconditionally and charges it to nothing, which is load-bearing for a capture cut short by its snapshot length, so a length declared by a 16-bit wire field could still be repeated without limit. The crafted 80,048-octet PCAP-NG of 2,000 Enhanced Packet Blocks that entry describes, each carrying one option declaring 65,535 octets against none present, synthesised 131,070,000 octets of zeros -- 1637.393x its own size, linear in the block count and so unbounded in the input -- and now synthesises none, with all 2,000 frames and all 2,000 options still parsed. The bound is taken one layer up, where the information the field layer lacks already exists: a PCAP-NG block's Block Total Length is authoritative and cross-checked against its own trailing copy, so the option area is that length less the fixed fields, `captured_len` and `captured_len`'s padding, and an option declaring more payload than that area has left is malformed however complete the file behind it is. A snapshot-truncated capture says so through `captured_len` instead and leaves its options whole, so it never trips this -- which is exactly what the #571 `len(buffer) < length` rejection could not distinguish, and what it was declined for. A new `bounded_option` clamps the payload to the octets the area has left at that field and warns with `SchemaWarning`; it is applied to all fifteen variable-width option and record payloads in the PCAP-NG schema, and deliberately not to the block-level fields, which are the 32-bit band the running ledger already budgets. Clamping rather than refusing is what keeps the #431 accommodation, since a PCAP-NG block read has no catch point above `FieldBase.unpack` and one refusal would abort a whole extraction rather than one block; and the clamp reads only the block's own declared framing, never a running total, so byte-identical input answers identically whatever preceded it. Zero clamps fired across all six PCAP-NG sample captures, 338 options between them, and 101 truncation levels of `dhcp.pcapng` gave byte-identical results including the failures. The bound is pinned as a property rather than as one input: every combination of one to three options against nine declared lengths asserts that the octets a block's options report holding never exceed the area, and the amplification ratio is asserted flat across 1, 8, 64 and 512 blocks, which is what separates a bound from a smaller constant (#594). +- **Fixed** -- six of the 123 constant registries under `pcapkit/const/` rejected an invalid value in a way the built-in `enum` does not, and three of them did not reject it at all. `Flags`, `ftp.command.CommandType` and `reg.apptype.TransportProtocol` are `IntFlag` types that defined no `_missing_`, so the `aenum` `Flag` machinery composed a pseudo-member for any integer whatsoever: `Flags(-1)` returned `65520`, the OR of every declared TCP header flag, so a value no 16-bit wire field can hold read back as *every* flag set at once, while `Flags(-65536)` read back as none set. `ftp.command.Command`, `ftp.command.FEATCode` and `http.method.Method` are `StrEnum` types whose `_missing_` reached `value.upper()` before checking the type, so an integer raised `AttributeError: 'int' object has no attribute 'upper'`. All six now raise a bare `ValueError`, which is what `enum.IntEnum` raises for a value it does not define and what 113 of the 117 modules already did. Issue #647's own proposal -- replacing those 113 with `pcapkit.utilities.exceptions.EnumError` -- was deliberately rejected: `EnumError` is `(BaseError, TypeError)` and not a `ValueError` at all, so it would have diverged from the built-in it was meant to improve on, walked past the `except ValueError` in all 113 generated `get()` bodies and silently undone #584, and logged CRITICAL from `BaseError.__init__` once per discarded default. The one deliberate divergence from the built-in is kept and now pinned: the mutable registries look a value up, miss, and `extend_enum` it rather than raising, so `Method('FROBNICATE')`, `ProtectionAuthority(1 << 70)` and `AppType.get(65000, proto=tcp)` still register. The three flag guards end in `return super()._missing_(value)`, so in-range composites still decompose and #623 stays fixed. Two of the four unguarded modules needed no change -- `ipv6/extension_header.py` and `ftp.command.ConformanceRequirement` define no `_missing_` either, and `aenum` already raises the bare `ValueError` for them. `pcapkit/vendor/tcp/flags.py` turned out to carry the guard's value checker all along, as `FLAG = 'isinstance(value, int) and 4 <= value <= 15'`, while its template never interpolated it; those are the registry's *bit offsets* and its members are `1 << offset`, so emitting it unchanged would have rejected every composite, every member above bit 3, and `Flags(0)`. It now reads `0 <= value <= 0xFFFF`, the 16-bit field those bits live in. `TransportProtocol` reads its bound off `cls.__members__` instead of a literal, because `TransportProtocol.get` extends the registry at runtime at `max * 2` and a written-down bound would reject the member it had just grown. Fixed in the four bespoke templates under `pcapkit/vendor/` rather than in the generated tree -- `pcapkit/vendor/default.py` needed no change, since the 113 it emits were already correct -- and regenerated from the live IANA registries: 42 insertions and zero deletions across exactly 4 of the 134 files, so nothing else in the tree was stale, including the 30k-line `reg/apptype.py`. A new `tests/const/test_const_enum_builtin_parity.py` asserts the property the fix is actually about, that a constant registry and a stdlib `enum.IntEnum` raise the same exception type for the same invalid value, over all 123 registries rather than the six; it is keyed on `aenum.Enum` because an `IntFlag` is not a subclass of `IntEnum` -- its MRO runs through `Flag` instead -- which is how three of these six escaped the two existing sweeps. `tests/const/test_const_enum_get.py` drops `Flags` from `EXPECTED_TO_RESOLVE_ANYTHING` and its sweep grows 110 to 111, a registry that bounds its domain finally having a failure for `default` to fall back from (#647). + Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). --- diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 363444abb..e6919c901 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1128,8 +1128,10 @@ pull requests between #326 and #509. four, and so did every composite of two defined bits, such as ``BindingACKFlag(0x06)``. Defining ``_missing_`` at all is what caused it, because it shadowed the ``aenum`` ``Flag`` machinery that resolves exactly those values; - ``pcapkit/const/tcp/flags.py`` defines no ``_missing_`` and has never had the - defect. All four now end in ``return super()._missing_(value)``, which is what + ``pcapkit/const/tcp/flags.py`` defined no ``_missing_`` at the time and never had + the defect -- it has one now, added by #647 below, ending in the same + ``super()._missing_(value)`` tail for the same reason. All four now end in + ``return super()._missing_(value)``, which is what ``pcapkit/vendor/default.py`` emits for every other generated enumeration and what 75 of the 117 modules under ``pcapkit/const/`` already do, so ``F(0)`` is an empty flag and ``BindingACKFlag(0x06)`` is ``S|D``. The ``extend_enum`` idiom that @@ -1747,6 +1749,55 @@ pull requests between #326 and #509. the area, and the amplification ratio is asserted flat across 1, 8, 64 and 512 blocks, which is what separates a bound from a smaller constant (#594). +* **Fixed** -- six of the 123 constant registries under ``pcapkit/const/`` rejected an + invalid value in a way the built-in ``enum`` does not, and three of them did not + reject it at all. ``Flags``, ``ftp.command.CommandType`` and + ``reg.apptype.TransportProtocol`` are ``IntFlag`` types that defined no + ``_missing_``, so the ``aenum`` ``Flag`` machinery composed a pseudo-member for any + integer whatsoever: ``Flags(-1)`` returned ``65520``, the OR of every declared TCP + header flag, so a value no 16-bit wire field can hold read back as *every* flag set + at once, while ``Flags(-65536)`` read back as none set. ``ftp.command.Command``, + ``ftp.command.FEATCode`` and ``http.method.Method`` are ``StrEnum`` types whose + ``_missing_`` reached ``value.upper()`` before checking the type, so an integer + raised ``AttributeError: 'int' object has no attribute 'upper'``. All six now raise + a bare ``ValueError``, which is what ``enum.IntEnum`` raises for a value it does not + define and what 113 of the 117 modules already did. Issue #647's own proposal -- + replacing those 113 with ``pcapkit.utilities.exceptions.EnumError`` -- was + deliberately rejected: ``EnumError`` is ``(BaseError, TypeError)`` and not a + ``ValueError`` at all, so it would have diverged from the built-in it was meant to + improve on, walked past the ``except ValueError`` in all 113 generated ``get()`` + bodies and silently undone #584, and logged CRITICAL from ``BaseError.__init__`` + once per discarded default. The one deliberate divergence from the built-in is kept + and now pinned: the mutable registries look a value up, miss, and ``extend_enum`` it + rather than raising, so ``Method('FROBNICATE')``, ``ProtectionAuthority(1 << 70)`` + and ``AppType.get(65000, proto=tcp)`` still register. The three flag guards end in + ``return super()._missing_(value)``, so in-range composites still decompose and #623 + stays fixed. Two of the four unguarded modules needed no change -- + ``ipv6/extension_header.py`` and ``ftp.command.ConformanceRequirement`` define no + ``_missing_`` either, and ``aenum`` already raises the bare ``ValueError`` for them. + ``pcapkit/vendor/tcp/flags.py`` turned out to carry the guard's value checker all + along, as ``FLAG = 'isinstance(value, int) and 4 <= value <= 15'``, while its + template never interpolated it; those are the registry's *bit offsets* and its + members are ``1 << offset``, so emitting it unchanged would have rejected every + composite, every member above bit 3, and ``Flags(0)``. It now reads + ``0 <= value <= 0xFFFF``, the 16-bit field those bits live in. + ``TransportProtocol`` reads its bound off ``cls.__members__`` instead of a literal, + because ``TransportProtocol.get`` extends the registry at runtime at ``max * 2`` and + a written-down bound would reject the member it had just grown. Fixed in the four + bespoke templates under ``pcapkit/vendor/`` rather than in the generated tree -- + ``pcapkit/vendor/default.py`` needed no change, since the 113 it emits were already + correct -- and regenerated from the live IANA registries: 42 insertions and zero + deletions across exactly 4 of the 134 files, so nothing else in the tree was stale, + including the 30k-line ``reg/apptype.py``. A new + ``tests/const/test_const_enum_builtin_parity.py`` asserts the property the fix is + actually about, that a constant registry and a stdlib ``enum.IntEnum`` raise the + same exception type for the same invalid value, over all 123 registries rather than + the six; it is keyed on ``aenum.Enum`` because an ``IntFlag`` is not a subclass of + ``IntEnum`` -- its MRO runs through ``Flag`` instead -- which is how three of these + six escaped the two existing sweeps. ``tests/const/test_const_enum_get.py`` drops ``Flags`` from + ``EXPECTED_TO_RESOLVE_ANYTHING`` and its sweep grows 110 to 111, a registry that + bounds its domain finally having a failure for ``default`` to fall back from (#647). + Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so resolved only by ``pip install --pre``. ``1.5.0b1`` half-shipped: the tag, the From aef0fb9da01b8841d6ff306fd5aad2244749d007 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 17:23:00 -0400 Subject: [PATCH 17/46] docs(changelog): expand the #594 entry for what its cross-review found The entry described only the payload bound. The cross-review of #676 found that the option *area* is sized from a Block Total Length nothing checks against the file, so a block declaring 1,000,000 octets while holding 36 synthesised 65,535 anyway -- 1,820x, unwarned. `bounded_area` closes that and the entry now says so. Also corrects two claims the same review disproved: the negative-remainder skip is load-bearing on the unpacking path rather than the packing one, and the truncation sweep is not uniformly one exception type. `util/changelog_md.py` regenerated `CHANGELOG.md`; `--check` exits 0. --- CHANGELOG.md | 2 +- docs/source/changelog/1.5.0.rst | 47 ++++++++++++++++++++++++--------- 2 files changed, 35 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7c2625658..f41daed17 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -99,7 +99,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- four MPTCP error messages carried a doubled separator, rendering as `TCP: : [OptNo 30] 1: invalid flags combination` with an empty field between the protocol alias and the option number. Cosmetic: the exception type, the option number and the subtype were always right, and nothing downstream parses these strings, but it is the text a user sees when an MP_JOIN or DSS option is rejected and an empty field reads as a value that failed to interpolate. Four sites, not the one reported -- `_read_mptcp_join` and `_make_mptcp_join` for the invalid flag combination, and both guards of `_make_mptcp_dss` for the missing required fields -- against 28 messages in the same file that already spelled the prefix correctly, so the four were outliers against a house form in their own module and the counts are now 32 correct against 0 doubled. Pre-existing since 2023-04-10. The new test asserts each message in full rather than by substring, which is what the existing `assertIn` on the message body could not do, since it passes either side of the change (#649). - **Fixed** -- the release workflow published to PyPI and to Anaconda with nothing a human had to approve. `create-release.yml`'s `pypi` job carried its `environment: release` commented out while the `id-token: write` from the same upstream snippet had been re-added live below it, so the block read as disabled as a unit when only its gate still was; the `conda` job had no `environment:` at all, not even a commented one. Neither needed a tag either: the workflow also fires on the completion of `Vendor Update`, so a scheduled registry crawl that bumped the version was enough to publish to two public indexes, at a median three minutes from bump to upload. Four jobs reach outside the run and all four are now gated -- `github` on `github-release` for the GitHub Release and the `v*` tag it creates, `tag` on `conda-tag` for the commit it pushes to `main`, `pypi` on `pypi`, and `conda` on `anaconda`. One environment per credential rather than one shared `release`, because approval is granted to an environment and not to a job, so a single `release` approved once would release both indexes together. PyPI is the one that has to be answerable alone: a version number it has accepted cannot be reused, and `skip-existing: true` makes the re-upload *succeed* having published nothing, so an unintended publish permanently consumes the number the intended release wanted. The workflow half is only half the fix -- an `environment:` naming an environment that does not exist is created implicitly with no protection rules and the job proceeds unapproved, so this is inert until each of the four exists in repository settings with a required reviewer on it, and each one's "Deployment branches and tags" has to stay unrestricted or the `tags: v*` trigger fails outright instead of pausing. `github-pages` is the standing example of that failure in this repository, carrying no protection rule since 2021. The new test asserts the general rule rather than the current file -- no job running a publishing action or a `git push` may omit an `environment:` -- so a publishing job added later without a gate fails rather than ships (#641). - **Fixed** -- three names appeared in string annotations that their own module never imported, so a type checker or a documentation build could not resolve them while every test went on passing: `Any` in `pcapkit/utilities/logging.py`, and in `pcapkit/protocols/schema/internet/ipv6_route.py` both `Protocol` in a `payload:` stub and `Optional` inside a `typing.cast`. All three are now in their module's `TYPE_CHECKING` block, `Protocol` spelled `ProtocolBase as Protocol` the way twenty-one sibling schema modules already spell it in that same stub, which makes twenty-two. The `cast` case is the one worth naming: `typing.cast` never evaluates its first argument, so no test that runs the line can see a bad name in it. Nothing resolves at runtime that did not resolve before -- `TYPE_CHECKING` is `False` when the interpreter runs, so a name imported under it is absent from the module namespace either way, and making these annotations resolve at runtime is a separate decision about the idiom rather than a missing import. mypy 2.3.1 over `pcapkit` reported three `name-defined` errors before and none after, its total moving 115 to 112, so nothing else shifted. A new `tests/project/test_annotation_names.py` pins the invariant without needing a type checker installed: it resolves every string annotation in the package against the names its own module binds, following a nested forward reference such as `'list["Nested"]'` while treating `Literal` members and `Annotated` metadata as the values they are, and it reports exactly those three findings on the unfixed tree and none after. That module reaches the two PEP 695 node classes it needs through `getattr` rather than naming them, because `ast.TypeAlias` and `ast.TypeVar` arrived in Python 3.12 while the supported range starts at 3.10. The same change corrects the `MANIFEST.in` comment asserting that `include README.md` was the only thing putting the README in a source distribution and that an sdist without it could not be installed; both halves are false, and the #619 entry above carried the identical claim and is corrected with it (#642). -- **Fixed** -- the 16-bit band the #573 entry above records as deliberately left open. `FieldBase.unpack` pads a shortfall of 65,536 octets or fewer unconditionally and charges it to nothing, which is load-bearing for a capture cut short by its snapshot length, so a length declared by a 16-bit wire field could still be repeated without limit. The crafted 80,048-octet PCAP-NG of 2,000 Enhanced Packet Blocks that entry describes, each carrying one option declaring 65,535 octets against none present, synthesised 131,070,000 octets of zeros -- 1637.393x its own size, linear in the block count and so unbounded in the input -- and now synthesises none, with all 2,000 frames and all 2,000 options still parsed. The bound is taken one layer up, where the information the field layer lacks already exists: a PCAP-NG block's Block Total Length is authoritative and cross-checked against its own trailing copy, so the option area is that length less the fixed fields, `captured_len` and `captured_len`'s padding, and an option declaring more payload than that area has left is malformed however complete the file behind it is. A snapshot-truncated capture says so through `captured_len` instead and leaves its options whole, so it never trips this -- which is exactly what the #571 `len(buffer) < length` rejection could not distinguish, and what it was declined for. A new `bounded_option` clamps the payload to the octets the area has left at that field and warns with `SchemaWarning`; it is applied to all fifteen variable-width option and record payloads in the PCAP-NG schema, and deliberately not to the block-level fields, which are the 32-bit band the running ledger already budgets. Clamping rather than refusing is what keeps the #431 accommodation, since a PCAP-NG block read has no catch point above `FieldBase.unpack` and one refusal would abort a whole extraction rather than one block; and the clamp reads only the block's own declared framing, never a running total, so byte-identical input answers identically whatever preceded it. Zero clamps fired across all six PCAP-NG sample captures, 338 options between them, and 101 truncation levels of `dhcp.pcapng` gave byte-identical results including the failures. The bound is pinned as a property rather than as one input: every combination of one to three options against nine declared lengths asserts that the octets a block's options report holding never exceed the area, and the amplification ratio is asserted flat across 1, 8, 64 and 512 blocks, which is what separates a bound from a smaller constant (#594). +- **Fixed** -- the 16-bit band the #573 entry above records as deliberately left open. `FieldBase.unpack` pads a shortfall of 65,536 octets or fewer unconditionally and charges it to nothing, which is load-bearing for a capture cut short by its snapshot length, so a length declared by a 16-bit wire field could still be repeated without limit. The crafted 80,048-octet PCAP-NG of 2,000 Enhanced Packet Blocks that entry describes, each carrying one option declaring 65,535 octets against none present, synthesised 131,070,000 octets of zeros -- 1637.393x its own size, linear in the block count and so unbounded in the input -- and now synthesises none, with all 2,000 frames and all 2,000 options still parsed. The bound is taken one layer up, where the information the field layer lacks already exists: a PCAP-NG block's Block Total Length is authoritative and cross-checked against its own trailing copy, so the option area is that length less the fixed fields, `captured_len` and `captured_len`'s padding, and an option declaring more payload than that area has left is malformed however complete the file behind it is. A snapshot-truncated capture says so through `captured_len` instead and leaves its options whole, so it never trips this -- which is exactly what the #571 `len(buffer) < length` rejection could not distinguish, and what it was declined for. A new `bounded_option` clamps the payload to the octets the area has left at that field and warns with `SchemaWarning`; it is applied to all fifteen variable-width option and record payloads in the PCAP-NG schema, and deliberately not to the block-level payload fields, which are the 32-bit band the running ledger already budgets. A second, `bounded_area`, clamps a packet block's option area to the octets the block itself holds, less the trailing Block Total Length: the area is otherwise sized from a declared length that nothing checks against the file -- `BlockType.post_process` compares it only with its own trailing copy -- so a block declaring 1,000,000 octets while holding 36 sized its area at 999,964, an option inside it declaring 65,535 was under that and went unclamped, and 65,535 octets of zeros were synthesised from 36 regardless, 1,820x with no warning at all. That came out of the change's cross-review rather than from writing it, and it is a no-op on a well-formed block, where the octets left of the block are exactly the area plus the trailing length's four. The five non-packet blocks' option areas keep the framing assumption, each computing its span with a different offset, and the general fix for a declared length reaching a read at all is #678. Clamping rather than refusing is what keeps the #431 accommodation, since a PCAP-NG block read has no catch point above `FieldBase.unpack` and one refusal would abort a whole extraction rather than one block; and the clamp reads only the block's own declared framing, never a running total, so byte-identical input answers identically whatever preceded it. The skip for a remainder already past zero is load-bearing on the unpacking path, where a ten-octet area leaves `__length__` at -2 before a payload sizes itself and clamping to it would hand the field a `'-2s'` struct template; it is *not* what protects the packing path, since `BytesField` and `StringField` both repair a negative width to `len(value)` in `pre_process`. Zero clamps fired across all six PCAP-NG sample captures, 338 options between them, and 501 truncation levels of `dhcp.pcapng` swept an octet at a time gave byte-identical results including the failures -- 497 of which are the uncaught `ValueError` that #678 records, two a `struct.error`, and two of which parse. The bound is pinned as a property rather than as one input: every combination of one to three options against nine declared lengths asserts that the octets a block's options report holding never exceed the area, and the amplification ratio is asserted flat across 1, 8, 64 and 512 blocks, which is what separates a bound from a smaller constant. The worst ratio reachable over five adversarial shapes afterwards is 0.862x, against 1,637x, 960x and 224x for the same three before (#594). - **Fixed** -- six of the 123 constant registries under `pcapkit/const/` rejected an invalid value in a way the built-in `enum` does not, and three of them did not reject it at all. `Flags`, `ftp.command.CommandType` and `reg.apptype.TransportProtocol` are `IntFlag` types that defined no `_missing_`, so the `aenum` `Flag` machinery composed a pseudo-member for any integer whatsoever: `Flags(-1)` returned `65520`, the OR of every declared TCP header flag, so a value no 16-bit wire field can hold read back as *every* flag set at once, while `Flags(-65536)` read back as none set. `ftp.command.Command`, `ftp.command.FEATCode` and `http.method.Method` are `StrEnum` types whose `_missing_` reached `value.upper()` before checking the type, so an integer raised `AttributeError: 'int' object has no attribute 'upper'`. All six now raise a bare `ValueError`, which is what `enum.IntEnum` raises for a value it does not define and what 113 of the 117 modules already did. Issue #647's own proposal -- replacing those 113 with `pcapkit.utilities.exceptions.EnumError` -- was deliberately rejected: `EnumError` is `(BaseError, TypeError)` and not a `ValueError` at all, so it would have diverged from the built-in it was meant to improve on, walked past the `except ValueError` in all 113 generated `get()` bodies and silently undone #584, and logged CRITICAL from `BaseError.__init__` once per discarded default. The one deliberate divergence from the built-in is kept and now pinned: the mutable registries look a value up, miss, and `extend_enum` it rather than raising, so `Method('FROBNICATE')`, `ProtectionAuthority(1 << 70)` and `AppType.get(65000, proto=tcp)` still register. The three flag guards end in `return super()._missing_(value)`, so in-range composites still decompose and #623 stays fixed. Two of the four unguarded modules needed no change -- `ipv6/extension_header.py` and `ftp.command.ConformanceRequirement` define no `_missing_` either, and `aenum` already raises the bare `ValueError` for them. `pcapkit/vendor/tcp/flags.py` turned out to carry the guard's value checker all along, as `FLAG = 'isinstance(value, int) and 4 <= value <= 15'`, while its template never interpolated it; those are the registry's *bit offsets* and its members are `1 << offset`, so emitting it unchanged would have rejected every composite, every member above bit 3, and `Flags(0)`. It now reads `0 <= value <= 0xFFFF`, the 16-bit field those bits live in. `TransportProtocol` reads its bound off `cls.__members__` instead of a literal, because `TransportProtocol.get` extends the registry at runtime at `max * 2` and a written-down bound would reject the member it had just grown. Fixed in the four bespoke templates under `pcapkit/vendor/` rather than in the generated tree -- `pcapkit/vendor/default.py` needed no change, since the 113 it emits were already correct -- and regenerated from the live IANA registries: 42 insertions and zero deletions across exactly 4 of the 134 files, so nothing else in the tree was stale, including the 30k-line `reg/apptype.py`. A new `tests/const/test_const_enum_builtin_parity.py` asserts the property the fix is actually about, that a constant registry and a stdlib `enum.IntEnum` raise the same exception type for the same invalid value, over all 123 registries rather than the six; it is keyed on `aenum.Enum` because an `IntFlag` is not a subclass of `IntEnum` -- its MRO runs through `Flag` instead -- which is how three of these six escaped the two existing sweeps. `tests/const/test_const_enum_get.py` drops `Flags` from `EXPECTED_TO_RESOLVE_ANYTHING` and its sweep grows 110 to 111, a registry that bounds its domain finally having a failure for `default` to fall back from (#647). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index e6919c901..859196a2c 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1735,19 +1735,40 @@ pull requests between #326 and #509. declined for. A new ``bounded_option`` clamps the payload to the octets the area has left at that field and warns with ``SchemaWarning``; it is applied to all fifteen variable-width option and record payloads in the PCAP-NG schema, - and deliberately not to the block-level fields, which are the 32-bit band the - running ledger already budgets. Clamping rather than refusing is what keeps the - #431 accommodation, since a PCAP-NG block read has no catch point above - ``FieldBase.unpack`` and one refusal would abort a whole extraction rather than - one block; and the clamp reads only the block's own declared framing, never a - running total, so byte-identical input answers identically whatever preceded - it. Zero clamps fired across all six PCAP-NG sample captures, 338 options - between them, and 101 truncation levels of ``dhcp.pcapng`` gave byte-identical - results including the failures. The bound is pinned as a property rather than - as one input: every combination of one to three options against nine declared - lengths asserts that the octets a block's options report holding never exceed - the area, and the amplification ratio is asserted flat across 1, 8, 64 and 512 - blocks, which is what separates a bound from a smaller constant (#594). + and deliberately not to the block-level payload fields, which are the 32-bit + band the running ledger already budgets. A second, ``bounded_area``, clamps a + packet block's option area to the octets the block itself holds, less the + trailing Block Total Length: the area is otherwise sized from a declared length + that nothing checks against the file -- ``BlockType.post_process`` compares it + only with its own trailing copy -- so a block declaring 1,000,000 octets while + holding 36 sized its area at 999,964, an option inside it declaring 65,535 was + under that and went unclamped, and 65,535 octets of zeros were synthesised from + 36 regardless, 1,820x with no warning at all. That came out of the change's + cross-review rather than from writing it, and it is a no-op on a well-formed + block, where the octets left of the block are exactly the area plus the + trailing length's four. The five non-packet blocks' option areas keep the + framing assumption, each computing its span with a different offset, and the + general fix for a declared length reaching a read at all is #678. Clamping + rather than refusing is what keeps the #431 accommodation, since a PCAP-NG + block read has no catch point above ``FieldBase.unpack`` and one refusal would + abort a whole extraction rather than one block; and the clamp reads only the + block's own declared framing, never a running total, so byte-identical input + answers identically whatever preceded it. The skip for a remainder already past + zero is load-bearing on the unpacking path, where a ten-octet area leaves + ``__length__`` at -2 before a payload sizes itself and clamping to it would + hand the field a ``'-2s'`` struct template; it is *not* what protects the + packing path, since ``BytesField`` and ``StringField`` both repair a negative + width to ``len(value)`` in ``pre_process``. Zero clamps fired across all six + PCAP-NG sample captures, 338 options between them, and 501 truncation levels of + ``dhcp.pcapng`` swept an octet at a time gave byte-identical results including + the failures -- 497 of which are the uncaught ``ValueError`` that #678 records, + two a ``struct.error``, and two of which parse. The bound is pinned as a + property rather than as one input: every combination of one to three options + against nine declared lengths asserts that the octets a block's options report + holding never exceed the area, and the amplification ratio is asserted flat + across 1, 8, 64 and 512 blocks, which is what separates a bound from a smaller + constant. The worst ratio reachable over five adversarial shapes afterwards is + 0.862x, against 1,637x, 960x and 224x for the same three before (#594). * **Fixed** -- six of the 123 constant registries under ``pcapkit/const/`` rejected an invalid value in a way the built-in ``enum`` does not, and three of them did not From 65ee4cb1f4b0548352adbb1472cedfbfe0a47398 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 18:16:00 -0400 Subject: [PATCH 18/46] docs(changelog): narrow the #651 entry to the 45 parameters it actually fixes #664 was narrowed per the owner's decision in #679: `LOCATOR_SET` is excluded and keeps the pre-#651 padding expression, so the change corrects 45 of the 46 HIP parameters rather than all of them. This entry claimed all of them, which would have shipped a false statement about wire output in the release notes. Three claims narrowed, in place rather than as a second bullet: - the opening, from "every HIP parameter" to 45 of 46, naming the exclusion; - the site counts, from "the 46 callbacks and the 49 record lengths ... instead of 95 times" to 93 of the 95 sites, 45 of 46 and 48 of 49; - the migration sentence, which said every other parameter's `length` moves likewise -- now the other 44, with `LOCATOR_SET` called out as unchanged in both its octets and its data model. Added the reason for the exclusion, because a reader who meets it in the code otherwise cannot tell it from an oversight: two defects in that parameter cancel exactly -- the padding callback never receives the parameter's `len`, since the nested `Locator` schemas share a packet context whose own `len` shadows it, and the parameter's `len` is in 4-octet units where the RFC's `Length` is a byte count. Always-4 padding gives `4 + 24n + 4`, and because `24n` is a multiple of 8 the RFC total for a byte-count `Length` of `24n` is the same `24n + 8`. Measured at n = 1, 2, 5 as 32, 56 and 128 octets on `b34f132f6` and on #664's head alike, so correcting only the padding would have taken a conformant parameter to four octets short. #679 carries the pair. 32 lines changed in the entry file; `CHANGELOG.md` regenerated with `util/changelog_md.py`, not edited. `--check` exits 0 and `tests/project/` is green at 96 passed, 469 subtests. Committed from a detached HEAD on aef0fb9da and pushed fast-forward to the branch ref, because `docs/changelog-1.5.0` is checked out in another agent's worktree and could not be taken here. --- CHANGELOG.md | 2 +- docs/source/changelog/1.5.0.rst | 32 +++++++++++++++++++++----------- 2 files changed, 22 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f41daed17..077f9c586 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,7 +92,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- `extract(..., no_eof=True)` never returned. End of stream was detected correctly and `ExtractionWarning: EOF reached` fired, but all three loops that handle it -- `record_frames`, `__next__` and `__call__` -- read `if self._flag_n: continue` with nothing else to stop them, so the flag suppressed the error that *ended* the loop without supplying any other ending and an exhausted input was retried forever. The flag is not pointless, which is why this is a termination condition rather than a deletion: `__main__` sets `no_eof` exactly for `fin='-'`, and the end of what has arrived on a live capture is not the end of the capture. What was missing was a way to tell a capture that has paused from one that is over, and the input's own position answers it -- `prepare` raises end of stream when the bytes remaining measure zero and restores the position before raising, so two consecutive ends of stream at the same position mean nothing arrived between them. `Extractor._note_eof_progress` is that check. A pipe is unaffected, because a read there *blocks* while the writer is open but idle rather than reporting end of stream -- measured at a full 1.5s pause on `6c3d1b0d9`, after which the frames arrived -- so a paused pipe never reaches the check at all. Also fixed thereby, and not mentioned in the issue: `pcapkit -` hung on **any** finished stdin, a pipe reporting end of stream once its writer closes; verified through the real CLI, where `cat in.pcap | python -m pcapkit -` went from killed at 25s to exiting 0 with a byte-identical dump. **One deliberate narrowing**: a *seekable* input does not block, so its two probes fall microseconds apart and a file still being appended to now ends at the data present when the extraction reached it -- measured with the append landing on a record boundary 0.6s in, six frames before and five after. The previous behaviour was unbounded by construction, which is the defect itself, so some stopping rule had to be chosen; a timed grace period would make the cut-off intermittent rather than absent, so following a growing file is left to a policy of its own and the decision is pinned by a test. Worth knowing alongside it, pre-existing and untouched: an append landing *mid-record* raises `ValueError: read length must be non-negative or -1` instead, so `no_eof` over a growing file only ever worked for a writer flushing whole records. Both docstrings for `no_eof` now say all of this. The end-to-end regression tests are bounded by a **child process** rather than by `tests._support.time_limit`, because the in-process deadline proved *intermittent* on this loop: it usually expired on time and once escaped entirely, running past ten minutes and 13.2 GB RSS before being killed by hand. An intermittent guard against a hang is worse than none, since the run it misses is a wedged suite rather than a red test. Two guesses at the cause are recorded as ruled out, so they are not made again: the parse path's two `except Exception` handlers are never entered during the spin, because `prepare` raises before any next-layer decode, and the memory is the retry loop emitting tens of thousands of `EOF reached` records a second which the runner retains (#620). - **Changed** -- building a protocol through its constructor with a keyword that names nothing now raises `UnsupportedCall` instead of discarding it. **This is a behaviour change to a public API**: every `make` in the tree ends its signature with `**kwargs` and reads nothing out of it, so until now a misspelled keyword was accepted, dropped, and the field it named kept its default -- wrong octets, with nothing said. That is what #602 cost: `examples/generators/options.py` asked for `seq=1` where `TCP.make` spells the parameter `seq_no`, and 25 generated fixture frames carried sequence number `0` against an empty `warnings` list. #541 and #556 were the same silence. The schema layer has never been so permissive -- `Schema.__update__` warns `UnknownFieldWarning` for a field it does not know -- and the asymmetry between the two halves of the same construction is what this closes. Checked in `ProtocolBase.__init__` rather than in `make`, because `make` is not the only consumer of the keywords it is handed: `__post_init__` passes one `**kwargs` to the construction *and* to the parse of what it has just constructed, so a keyword declared only by `read` legitimately travels through `make` -- `HIP.read` declares `extension` where `HIP.make` does not, and the option generator depends on it. The accepted set is therefore the union of every keyword-taking parameter of `make`, `read`, `pack`, `unpack`, `__post_init__` and `__init__` across the whole MRO, computed once per class from `inspect.signature`. Parsing is deliberately untouched, since there the keywords are whatever the engines and the four `_import_next_layer` implementations forward and a protocol cannot know which of its ancestors' its parent passed on -- and nothing was ever lost that way, a dropped parse keyword changing how a packet is read rather than what its octets say. Two escapes exist for the shapes a signature cannot express, both opt-in per class through a new `__keywords__`: a set, for a keyword read out of `**kwargs` by name as `ESP.read` does with `packet`; and `None`, for a dispatcher whose real signature belongs to a class chosen at call time, which is exactly `HTTP.make` forwarding to `HTTPv1`/`HTTPv2` and the one place in the tree that uses it. The message names the near neighbour it found, so `seq` reports *did you mean 'seq_no'?*. `from_data` warns `UnknownFieldWarning` where a caller would be raised at, because the keywords there are whatever `_make_data` returned rather than anything anybody typed, so the defect is a key of that mapping disagreeing with the signature it is spread into and the person who meets it is not the person who can fix it. Expect this to surface latent bugs in code that has been quietly losing a field, which is the point. It surfaced four in this repository, all the residue of #602 and all fixed here: the `_TCP_BASE` of `examples/generators/dispatch.py` and three stale copies of it under `tests/protocols/transport/`, each still passing `seq`/`ack_flag`/`urgent_pointer` and so building segments with sequence number `0` where they read as `1`. It surfaced three more that are reported rather than fixed, being a defect per protocol rather than one in this mechanism: `Frame._make_data` returns `ts_src` where `make` declares `ts_sec`, `L2TPv2._make_data` returns `prio` where it declares `priority`, and `Header._make_data` returns a `magic_number` that `Header.make` does not take at all -- so `from_data` has been dropping a frame's timestamp, an L2TPv2 priority bit and a capture's byte order, and now says so. One limitation worth knowing rather than discovering: a *direct* `SomeProtocol.make(...)` call is not checked and still discards in silence, since the check sits where every producer's keywords converge rather than inside each of the 30 `make` implementations -- `object.__new__(cls).make(**kwargs)` is the idiom that reaches it, and `HTTP.make` uses it to reach its versioned implementation (#617). - **Fixed** -- the HIP `PUZZLE` and `SOLUTION` builders derived three wire-format values from the payload value instead of taking them from the data model, and all three were wrong. **This changes two public data models and the octets both parameters emit.** The width of `Random #I`, and of `Puzzle solution #J`, came from `int.bit_length()` alone, and nothing else was available to derive it from, so every leading zero octet was dropped on re-serialisation: a `SOLUTION` read with `Length = 20` rebuilt as `Length = 6`, a `PUZZLE` read with `Length = 12` as `Length = 5`, and silently, because the integers survive and nothing raises. That width is `RHASH_len / 8` octets [[RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4), [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5)], a property of the Responder's HIT Suite rather than of the number that happens to sit in the field, so both data models now carry it as `rhash_len` and both builders prefer it; a new `rhash_len=` keyword states it for a build from scratch, which under HIPv2 is the only place it can come from, since `RSA,DSA/SHA-256` is the REQUIRED HIT Suite [[RFC 7401 Section 5.2.10](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.10)] and makes the field 32 octets rather than 8. On the mandatory path roughly one parameter in 256 has a zero top octet and lost it (#653). `SOLUTION`'s second contents octet is `Reserved`, "zero when sent, ignored when received" [[RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5), and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) identically] -- not the `Lifetime` that only [RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4) defines, and that pcapkit was encoding there as `2^(value - 32)` seconds. It wrote `0x20`, `0x21`, `0x25` or `0x2b` into a field the RFC requires to be zero, and could not write that zero at all: an RFC-conformant `SOLUTION` whose `Reserved` is `0x00` parsed to `timedelta(0)` and then could not be re-serialised, escaping a bare `ValueError` from `math.log2(0.0)` that was not a `BaseError` and so bypassed the library's own error handling entirely. The field is renamed `reserved`, defaults to the mandated zero, and is carried verbatim across a round trip rather than re-derived (#654). And neither builder read its own `version` keyword, so `version=1` and `version=2` computed identical lengths at every bit width, where [RFC 5201 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.4) and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) state both fields as literally 8 bytes and `Length` as literally 12 and 20. Under HIPv1 each builder therefore accepted only a `bit_length()` of 57..64 and built, for everything narrower, a parameter this library's own reader rejects -- the same shape as #608, in the `PUZZLE` builder that #629 never opened (#655). The two remaining `math.log2` sites, both in `PUZZLE` where a lifetime is real, now raise `ProtocolError` rather than letting `ValueError` escape; `ProtocolError` is `(BaseError, ValueError)`, so a caller written around the old bare exception still catches it. Migration: `SolutionParameter.lifetime` is now `reserved` and an `int` rather than a `timedelta`; both `PuzzleParameter` and `SolutionParameter` gained a required `rhash_len`; and `_make_param_solution` no longer takes `lifetime=`. All three were only reachable end to end once #608 was fixed in #629, which removed the parity guard that had been failing these rebuilds loudly first -- so the round trip stopped raising and started quietly emitting a different parameter, which is why they were worth fixing together (#653, #654, #655). -- **Fixed** -- every HIP parameter pcapkit emitted was `4 (mod 8)` octets long, because the padding was computed to align the *contents* rather than the record. **This changes the octets of every HIP parameter written, and the `length` every parameter's data model reports.** [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1) requires that "all of the encoded TLV parameters have a length (that includes the Type and Length fields), which is a multiple of 8 bytes", and states the arithmetic outright as `Total Length = 11 + Length - (Length + 3) % 8`; all 95 padding sites instead computed `(8 - (Length % 8)) % 8`, which has no `+ 4` inside the modulus and so aligns the contents alone. Across every `Length` from 0 to 63 the result was never a multiple of eight and never the value the RFC gives, and it erred in both directions: at `Length = 4` -- a whole `SEQ`, and [RFC 7401 Section 5.3.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.3.5) puts a `SEQ` or an `ACK` on every `UPDATE` -- the record is complete in eight octets and pcapkit appended four that must not be there, while at `Length = 8` the contents were already 8-aligned, nothing was appended, and the record went out four octets short. A conformant peer reading `Length` and consuming `11 + Length - (Length + 3) % 8` octets therefore lands mid-parameter and reads the rest of the parameter area at a wrong offset, in both directions; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test in the suite could see this and why the fix is asserted against the RFC's arithmetic rather than against a round trip. The 46 `PaddingField` callbacks in `pcapkit/protocols/schema/internet/hip.py` and the 49 record lengths in `pcapkit/protocols/internet/hip.py` are now one `parameter_total_len` and one `parameter_padding_len`, stating the RFC formula once instead of 95 times (#651). `EncryptedParameter`'s `data` length callback is fixed in the same change and could not have been left: it subtracted the sixteen `iv` octets but not the four `reserved` ones, and those four cancelled the padding's four at `Length % 8` in `{0, 5, 6, 7}` -- so correcting the padding alone would have taken `ENCRYPTED` from right at four of the eight residues to four octets too long at all eight. `HIP.make`'s `len = total_length // 8 + 4` is *not* part of the defect and is unchanged: [RFC 7401 Section 5.1.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.1.3) defines Header Length as the header and parameters "in 8-byte units, excluding the first 8 bytes", so the floor division is exact once each parameter is a multiple of eight, where before it was exact only for an even number of them. Migration: a `SEQ` parameter's `length` is now 8 where it was 12, and every other parameter's moves likewise, so code comparing stored pcapkit output byte for byte, or asserting on `Data_*Parameter.length`, sees different values -- the RFC's values. `examples/generators/options.py`'s `HIP_COPIES` stays at two, no longer for this reason but for `R1_COUNTER`'s four-octet `counter` where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires eight, which this defect had been masking (#651). +- **Fixed** -- every HIP parameter pcapkit emitted was `4 (mod 8)` octets long, because the padding was computed to align the *contents* rather than the record. Corrected for **45 of the 46** HIP parameters; `LOCATOR_SET` is deliberately excluded, for the reason below. **This changes the octets those 45 parameters write, and the `length` their data models report.** [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1) requires that "all of the encoded TLV parameters have a length (that includes the Type and Length fields), which is a multiple of 8 bytes", and states the arithmetic outright as `Total Length = 11 + Length - (Length + 3) % 8`; all 95 padding sites instead computed `(8 - (Length % 8)) % 8`, which has no `+ 4` inside the modulus and so aligns the contents alone. Across every `Length` from 0 to 63 the result was never a multiple of eight and never the value the RFC gives, and it erred in both directions: at `Length = 4` -- a whole `SEQ`, and [RFC 7401 Section 5.3.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.3.5) puts a `SEQ` or an `ACK` on every `UPDATE` -- the record is complete in eight octets and pcapkit appended four that must not be there, while at `Length = 8` the contents were already 8-aligned, nothing was appended, and the record went out four octets short. A conformant peer reading `Length` and consuming `11 + Length - (Length + 3) % 8` octets therefore lands mid-parameter and reads the rest of the parameter area at a wrong offset, in both directions; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test in the suite could see this and why the fix is asserted against the RFC's arithmetic rather than against a round trip. 93 of the 95 sites -- 45 of the 46 `PaddingField` callbacks in `pcapkit/protocols/schema/internet/hip.py` and 48 of the 49 record lengths in `pcapkit/protocols/internet/hip.py` -- are now one `parameter_total_len` and one `parameter_padding_len`, stating the RFC formula once instead of 95 times (#651). `LOCATOR_SET` keeps the old expression at both of its sites, on purpose, because two defects there cancel each other exactly and correcting only the padding would break a parameter that is currently right: its padding callback never receives the parameter's `len` (the nested `Locator` schemas share a packet context whose own `len` shadows it, so the value seen is always 4 for an IPv6 locator), and the parameter's `len` is written in 4-octet units where the RFC's `Length` is a byte count. Always-4 padding gives `4 + 24n + 4`, and because `24n` is a multiple of 8 the RFC total for a byte-count `Length` of `24n` is the same `24n + 8` -- measured at n = 1, 2 and 5 as 32, 56 and 128 octets both before and after. #679 carries the pair. `EncryptedParameter`'s `data` length callback is fixed in the same change and could not have been left: it subtracted the sixteen `iv` octets but not the four `reserved` ones, and those four cancelled the padding's four at `Length % 8` in `{0, 5, 6, 7}` -- so correcting the padding alone would have taken `ENCRYPTED` from right at four of the eight residues to four octets too long at all eight. `HIP.make`'s `len = total_length // 8 + 4` is *not* part of the defect and is unchanged: [RFC 7401 Section 5.1.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.1.3) defines Header Length as the header and parameters "in 8-byte units, excluding the first 8 bytes", so the floor division is exact once each parameter is a multiple of eight, where before it was exact only for an even number of them. Migration: a `SEQ` parameter's `length` is now 8 where it was 12, and the other 44 corrected parameters move likewise, so code comparing stored pcapkit output byte for byte, or asserting on `Data_*Parameter.length`, sees different values -- the RFC's values. `LOCATOR_SET` is unchanged in both respects. `examples/generators/options.py`'s `HIP_COPIES` stays at two, no longer for this reason but for `R1_COUNTER`'s four-octet `counter` where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires eight, which this defect had been masking (#651). - **Fixed** -- the `_missing` sentinel behind `MultiDict.pop` and `OrderedMultiDict.pop` answered `True` to `bool()`, stating the opposite of what it means, and its class was subclassable. Both now follow `NoValueType` in `pcapkit.corekit.fields.field`, the package's convention for a marker of this kind: `_Missing` carries `@final` and defines `__bool__` returning `False`. The marker is not private in practice -- it is the runtime default of both `pop()` methods, so `inspect.signature(MultiDict.pop).parameters['default'].default` hands it to any caller who asks, and `if default:` on that value reported "a default was supplied" where none had been. No behaviour changes: both `pop()` implementations decide by identity (`default is not _missing`) and never by truthiness, so `repr`, `__reduce__`, the pickle bytes at every protocol, and every `pop()` result -- including the falsy defaults `None`, `False`, `0`, `''` and `[]` -- are unchanged, and `pop()` structurally cannot return the marker itself. This closes the second of the three `object()` sentinel sites the sweep for #640 left open; the first needed nothing, and `_NOT_FOUND` in `pcapkit.utilities.compat` is deliberately left alone as a verbatim line of CPython's own `functools.cached_property`, inside a `sys.version_info < (3, 8)` branch no supported interpreter reaches (#661). - **Fixed** -- two HTTP/2 flag defects, one on each side of a round trip. **This changes the octets a reconstructed DATA frame emits, and the dumped value of a flagless frame's flags.** `_make_http_data` was the only one of the six `_make_http_*` methods that never read `frame.flags`, so a DATA frame parsed with `END_STREAM` set rebuilt with the bit clear: the flags octet went `0x01` to `0x00`, silently, producing a well-formed frame saying the stream continues where the capture said it ended [[RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1)]. `PADDED` was never affected, being re-derived from `pad_len` rather than read back, and all five sibling builders already restored theirs -- so this was a missing read-back rather than a design, and it now reads `frame.flags.END_STREAM` the way they do (#652). Separately, `FrameType.post_process` seeded its flag accumulator with a bare `0`, and `|=` promotes that only as a side effect, so a frame with **no** bit set left `__flags__` a plain `int` where both the schema and the data model declare a `Flags` -- and `END_STREAM in __value__` raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering `False`. That is the common case rather than an edge one: a `0x00` flags octet is every opening `SETTINGS`, every non-final `DATA` and every non-ACK `PING`. It now seeds `self.Flags(0)`, which the construct path had been doing correctly at all six of its own sites. The visible consequence is in the dump, where `__value__` was a JSON *number* for a flagless frame and a JSON *string* for every other frame in the same capture; it is consistently a string now, `"Flags::None [0]"` where it read `0`, the same shape of change #616 made to TCP's `connection`. The seed is guarded rather than unconditional because `FrameType.Flags` declares no members and, from Python 3.11, a memberless `enum.Flag` subclass refuses `Flags(0)` outright, so the five frame schemas that inherit it -- `UnassignedFrame`, `PriorityFrame`, `RSTStreamFrame`, `GoawayFrame` and `WindowUpdateFrame` -- keep the plain `int`, which nothing observes since all five pass `flags=None` into their data objects. Note that a DATA round trip is **still** lossy after this, for an unrelated reason found while measuring it and left to its own change: three payload length callbacks in the frame schemas are mis-parenthesised, so an unpadded `DATA`, `HEADERS` or `PUSH_PROMISE` frame parses with its whole payload dropped (#668). (#652, #650). - **Fixed** -- a flag value with no declared bits no longer dumps as `Type::None [0]`. **This changes user-visible output in every textual dump format.** `make_dumper`'s `object_hook` renders each enumeration member as `Type::name [value]` and interpolated the name unguarded, but a `Flag` value composed entirely of undeclared bits has `name is None` rather than a string -- so the literal four characters `None` landed in the name half and `Flags(0)` rendered as `Flags::None [0]` in `json`, `tree`, `text`, `txt`, `plist` and `xml`, out of both `Extractor` and `TraceFlow`. A nameless member now renders with the value's own decimal spelling instead, so `Flags(0)` gives `Flags::0 [0]` and `Flags(8)` gives `Flags::8 [8]`. That is the spelling the enumeration libraries already use for an undeclared residue -- `Flags(2057).name` is `ACK|9`, naming the declared bit and giving the leftovers as one decimal number -- and it cannot be mistaken for a member name, since a Python identifier may not begin with a digit, whereas `NONE` is a real declared name elsewhere in the library. Three sites carried the identical interpolation, not one: the `OrderedMultiDict` key path, the `addon` branch and the scalar return, which now share one `render_enum` helper so the guard cannot be applied to two of three. That guard is on `name is None` and not on the value, because the defect never was about zero -- `Flags(1)`, `Flags(8)`, `Flags(9)` and `Flags(65536)` are equally nameless -- and it is not an `aenum` quirk either, since a stdlib `enum.IntFlag` answers `name is None` at the same values. Five of the library's seven flag registries are nameless at zero rather than the one reported: `Flags` plus the four Mobility Header flag registries, the other two declaring an explicit `undefined = 0`. Pre-existing since 2023-04-28 and surfaced rather than caused by #634, which made a flagless TCP segment reach this path. The committed example dumps do not move, because no member in them lacks a name (#648). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 859196a2c..6583aeff8 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1518,8 +1518,9 @@ pull requests between #326 and #509. #655). * **Fixed** -- every HIP parameter pcapkit emitted was ``4 (mod 8)`` octets long, because the padding was computed to align the *contents* rather than the record. - **This changes the octets of every HIP parameter written, and the ``length`` - every parameter's data model reports.** :rfc:`7401#section-5.2.1` requires that + Corrected for **45 of the 46** HIP parameters; ``LOCATOR_SET`` is deliberately + excluded, for the reason below. **This changes the octets those 45 parameters + write, and the ``length`` their data models report.** :rfc:`7401#section-5.2.1` requires that "all of the encoded TLV parameters have a length (that includes the Type and Length fields), which is a multiple of 8 bytes", and states the arithmetic outright as ``Total Length = 11 + Length - (Length + 3) % 8``; all 95 padding @@ -1536,11 +1537,20 @@ pull requests between #326 and #509. wrong offset, in both directions; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test in the suite could see this and why the fix is asserted against the RFC's arithmetic - rather than against a round trip. The 46 ``PaddingField`` callbacks in - ``pcapkit/protocols/schema/internet/hip.py`` and the 49 record lengths in - ``pcapkit/protocols/internet/hip.py`` are now one ``parameter_total_len`` and one - ``parameter_padding_len``, stating the RFC formula once instead of 95 times - (#651). ``EncryptedParameter``'s ``data`` length callback is fixed in the same + rather than against a round trip. 93 of the 95 sites -- 45 of the 46 + ``PaddingField`` callbacks in ``pcapkit/protocols/schema/internet/hip.py`` and 48 + of the 49 record lengths in ``pcapkit/protocols/internet/hip.py`` -- are now one + ``parameter_total_len`` and one ``parameter_padding_len``, stating the RFC formula + once instead of 95 times (#651). ``LOCATOR_SET`` keeps the old expression at both + of its sites, on purpose, because two defects there cancel each other exactly and + correcting only the padding would break a parameter that is currently right: its + padding callback never receives the parameter's ``len`` (the nested ``Locator`` + schemas share a packet context whose own ``len`` shadows it, so the value seen is + always 4 for an IPv6 locator), and the parameter's ``len`` is written in 4-octet + units where the RFC's ``Length`` is a byte count. Always-4 padding gives + ``4 + 24n + 4``, and because ``24n`` is a multiple of 8 the RFC total for a + byte-count ``Length`` of ``24n`` is the same ``24n + 8`` -- measured at n = 1, 2 + and 5 as 32, 56 and 128 octets both before and after. #679 carries the pair. ``EncryptedParameter``'s ``data`` length callback is fixed in the same change and could not have been left: it subtracted the sixteen ``iv`` octets but not the four ``reserved`` ones, and those four cancelled the padding's four at ``Length % 8`` in ``{0, 5, 6, 7}`` -- so correcting the padding alone would have @@ -1550,10 +1560,10 @@ pull requests between #326 and #509. as the header and parameters "in 8-byte units, excluding the first 8 bytes", so the floor division is exact once each parameter is a multiple of eight, where before it was exact only for an even number of them. Migration: a ``SEQ`` - parameter's ``length`` is now 8 where it was 12, and every other parameter's - moves likewise, so code comparing stored pcapkit output byte for byte, or - asserting on ``Data_*Parameter.length``, sees different values -- the RFC's - values. ``examples/generators/options.py``'s ``HIP_COPIES`` stays at two, no + parameter's ``length`` is now 8 where it was 12, and the other 44 corrected + parameters move likewise, so code comparing stored pcapkit output byte for byte, + or asserting on ``Data_*Parameter.length``, sees different values -- the RFC's + values. ``LOCATOR_SET`` is unchanged in both respects. ``examples/generators/options.py``'s ``HIP_COPIES`` stays at two, no longer for this reason but for ``R1_COUNTER``'s four-octet ``counter`` where :rfc:`7401#section-5.2.3` requires eight, which this defect had been masking (#651). From 709c05608dc360c1f424ed4547144698573830e2 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 18:21:46 -0400 Subject: [PATCH 19/46] docs(changelog): the 1.5.0 entry for #645, whose code ships in #680 ``SeekableReader.truncate()`` now raises instead of returning a size, and the misspelled ``writeable()`` is spelled ``writable()``. Filed as **Changed** rather than **Fixed**, following the #617 entry: both halves are breaks to a public API, even though nothing inside the package called either method. CHANGELOG.md regenerated with ``util/changelog_md.py``; ``--check`` exits 0. --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 37 +++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 077f9c586..54b13d599 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -102,6 +102,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- the 16-bit band the #573 entry above records as deliberately left open. `FieldBase.unpack` pads a shortfall of 65,536 octets or fewer unconditionally and charges it to nothing, which is load-bearing for a capture cut short by its snapshot length, so a length declared by a 16-bit wire field could still be repeated without limit. The crafted 80,048-octet PCAP-NG of 2,000 Enhanced Packet Blocks that entry describes, each carrying one option declaring 65,535 octets against none present, synthesised 131,070,000 octets of zeros -- 1637.393x its own size, linear in the block count and so unbounded in the input -- and now synthesises none, with all 2,000 frames and all 2,000 options still parsed. The bound is taken one layer up, where the information the field layer lacks already exists: a PCAP-NG block's Block Total Length is authoritative and cross-checked against its own trailing copy, so the option area is that length less the fixed fields, `captured_len` and `captured_len`'s padding, and an option declaring more payload than that area has left is malformed however complete the file behind it is. A snapshot-truncated capture says so through `captured_len` instead and leaves its options whole, so it never trips this -- which is exactly what the #571 `len(buffer) < length` rejection could not distinguish, and what it was declined for. A new `bounded_option` clamps the payload to the octets the area has left at that field and warns with `SchemaWarning`; it is applied to all fifteen variable-width option and record payloads in the PCAP-NG schema, and deliberately not to the block-level payload fields, which are the 32-bit band the running ledger already budgets. A second, `bounded_area`, clamps a packet block's option area to the octets the block itself holds, less the trailing Block Total Length: the area is otherwise sized from a declared length that nothing checks against the file -- `BlockType.post_process` compares it only with its own trailing copy -- so a block declaring 1,000,000 octets while holding 36 sized its area at 999,964, an option inside it declaring 65,535 was under that and went unclamped, and 65,535 octets of zeros were synthesised from 36 regardless, 1,820x with no warning at all. That came out of the change's cross-review rather than from writing it, and it is a no-op on a well-formed block, where the octets left of the block are exactly the area plus the trailing length's four. The five non-packet blocks' option areas keep the framing assumption, each computing its span with a different offset, and the general fix for a declared length reaching a read at all is #678. Clamping rather than refusing is what keeps the #431 accommodation, since a PCAP-NG block read has no catch point above `FieldBase.unpack` and one refusal would abort a whole extraction rather than one block; and the clamp reads only the block's own declared framing, never a running total, so byte-identical input answers identically whatever preceded it. The skip for a remainder already past zero is load-bearing on the unpacking path, where a ten-octet area leaves `__length__` at -2 before a payload sizes itself and clamping to it would hand the field a `'-2s'` struct template; it is *not* what protects the packing path, since `BytesField` and `StringField` both repair a negative width to `len(value)` in `pre_process`. Zero clamps fired across all six PCAP-NG sample captures, 338 options between them, and 501 truncation levels of `dhcp.pcapng` swept an octet at a time gave byte-identical results including the failures -- 497 of which are the uncaught `ValueError` that #678 records, two a `struct.error`, and two of which parse. The bound is pinned as a property rather than as one input: every combination of one to three options against nine declared lengths asserts that the octets a block's options report holding never exceed the area, and the amplification ratio is asserted flat across 1, 8, 64 and 512 blocks, which is what separates a bound from a smaller constant. The worst ratio reachable over five adversarial shapes afterwards is 0.862x, against 1,637x, 960x and 224x for the same three before (#594). - **Fixed** -- six of the 123 constant registries under `pcapkit/const/` rejected an invalid value in a way the built-in `enum` does not, and three of them did not reject it at all. `Flags`, `ftp.command.CommandType` and `reg.apptype.TransportProtocol` are `IntFlag` types that defined no `_missing_`, so the `aenum` `Flag` machinery composed a pseudo-member for any integer whatsoever: `Flags(-1)` returned `65520`, the OR of every declared TCP header flag, so a value no 16-bit wire field can hold read back as *every* flag set at once, while `Flags(-65536)` read back as none set. `ftp.command.Command`, `ftp.command.FEATCode` and `http.method.Method` are `StrEnum` types whose `_missing_` reached `value.upper()` before checking the type, so an integer raised `AttributeError: 'int' object has no attribute 'upper'`. All six now raise a bare `ValueError`, which is what `enum.IntEnum` raises for a value it does not define and what 113 of the 117 modules already did. Issue #647's own proposal -- replacing those 113 with `pcapkit.utilities.exceptions.EnumError` -- was deliberately rejected: `EnumError` is `(BaseError, TypeError)` and not a `ValueError` at all, so it would have diverged from the built-in it was meant to improve on, walked past the `except ValueError` in all 113 generated `get()` bodies and silently undone #584, and logged CRITICAL from `BaseError.__init__` once per discarded default. The one deliberate divergence from the built-in is kept and now pinned: the mutable registries look a value up, miss, and `extend_enum` it rather than raising, so `Method('FROBNICATE')`, `ProtectionAuthority(1 << 70)` and `AppType.get(65000, proto=tcp)` still register. The three flag guards end in `return super()._missing_(value)`, so in-range composites still decompose and #623 stays fixed. Two of the four unguarded modules needed no change -- `ipv6/extension_header.py` and `ftp.command.ConformanceRequirement` define no `_missing_` either, and `aenum` already raises the bare `ValueError` for them. `pcapkit/vendor/tcp/flags.py` turned out to carry the guard's value checker all along, as `FLAG = 'isinstance(value, int) and 4 <= value <= 15'`, while its template never interpolated it; those are the registry's *bit offsets* and its members are `1 << offset`, so emitting it unchanged would have rejected every composite, every member above bit 3, and `Flags(0)`. It now reads `0 <= value <= 0xFFFF`, the 16-bit field those bits live in. `TransportProtocol` reads its bound off `cls.__members__` instead of a literal, because `TransportProtocol.get` extends the registry at runtime at `max * 2` and a written-down bound would reject the member it had just grown. Fixed in the four bespoke templates under `pcapkit/vendor/` rather than in the generated tree -- `pcapkit/vendor/default.py` needed no change, since the 113 it emits were already correct -- and regenerated from the live IANA registries: 42 insertions and zero deletions across exactly 4 of the 134 files, so nothing else in the tree was stale, including the 30k-line `reg/apptype.py`. A new `tests/const/test_const_enum_builtin_parity.py` asserts the property the fix is actually about, that a constant registry and a stdlib `enum.IntEnum` raise the same exception type for the same invalid value, over all 123 registries rather than the six; it is keyed on `aenum.Enum` because an `IntFlag` is not a subclass of `IntEnum` -- its MRO runs through `Flag` instead -- which is how three of these six escaped the two existing sweeps. `tests/const/test_const_enum_get.py` drops `Flags` from `EXPECTED_TO_RESOLVE_ANYTHING` and its sweep grows 110 to 111, a registry that bounds its domain finally having a failure for `default` to fall back from (#647). +- **Changed** -- `SeekableReader.truncate()` raises instead of returning a size, and the misspelled `writeable()` is now spelled `writable()`. Both are breaks for an external caller and neither breaks anything inside the package. `io.IOBase` documents one gate over two methods -- "If False, write() and truncate() will raise OSError" -- and this reader's `writable()` answers False, so a caller that checked it first, which is exactly what the contract invites, got a surprise either way round: `write` and `writelines` raised, and `truncate` returned its new size. What settled the question is that this is not a pure-Python nicety of `_pyio` that the accelerated path skips -- every ordinary read-only file object in CPython raises here, `open(path, 'rb').truncate()` giving `io.UnsupportedOperation: truncate` off the C `BufferedReader`, and `_pyio.BufferedReader` the same type from `_BufferedIOMixin.truncate`'s `_checkWritable()`. The refusal is `UnsupportedOperation('truncate')`, the same one `write` already raises, and no trade-off was needed between the house exception and the ABC's: pcapkit's `UnsupportedOperation` subclasses `io.UnsupportedOperation`, which subclasses `OSError`, so the in-library exception *is* the one the contract names. The resizing `truncate` used to perform is not deleted, only made private as `_truncate_buffer`. It never touched the underlying stream -- it resizes a private lookback window, which is the counter-argument the issue itself raised -- and it is the only route to the "position sits before the window" state that `seek` and the four buffered read paths must refuse, which #643 and #644 landed tests for; deleting it would have taken their mechanism with them. `writeable()`, separately, was never an override of anything: `'writable' in SeekableReader.__dict__` was False and `SeekableReader.writable is io.IOBase.writable` was True, so `io`, `shutil` and any third-party caller read the inherited value and never saw the one defined in this file. Both answered False, which is the coincidence that hid it -- there was no symptom to notice, and editing the misspelled method would silently have had no effect. The value reported is unchanged and was always honest, the reader genuinely being unable to write; only where the method was defined was wrong. Nothing in the package called either method, confirmed by grep, so the risk is entirely to external callers, which is what made this a question asked before it was acted on. The new test asserts the property rather than the behaviour -- that a `SeekableReader` raises the same exception type a read-only `io.BufferedReader` raises for the same call -- with that type captured by running the call rather than named in the test, so it tracks CPython across the 3.10--3.15 matrix instead of restating a belief about it. 31 to 34 tests and 35 to 41 subtests over `tests/corekit/test_io.py`, the module at 100% coverage before and after (#645). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 6583aeff8..592f0e5c6 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1828,6 +1828,43 @@ pull requests between #326 and #509. six escaped the two existing sweeps. ``tests/const/test_const_enum_get.py`` drops ``Flags`` from ``EXPECTED_TO_RESOLVE_ANYTHING`` and its sweep grows 110 to 111, a registry that bounds its domain finally having a failure for ``default`` to fall back from (#647). +* **Changed** -- ``SeekableReader.truncate()`` raises instead of returning a size, and + the misspelled ``writeable()`` is now spelled ``writable()``. Both are breaks for an + external caller and neither breaks anything inside the package. ``io.IOBase`` + documents one gate over two methods -- "If False, write() and truncate() will raise + OSError" -- and this reader's ``writable()`` answers False, so a caller that checked + it first, which is exactly what the contract invites, got a surprise either way + round: ``write`` and ``writelines`` raised, and ``truncate`` returned its new size. + What settled the question is that this is not a pure-Python nicety of ``_pyio`` that + the accelerated path skips -- every ordinary read-only file object in CPython raises + here, ``open(path, 'rb').truncate()`` giving ``io.UnsupportedOperation: truncate`` + off the C ``BufferedReader``, and ``_pyio.BufferedReader`` the same type from + ``_BufferedIOMixin.truncate``'s ``_checkWritable()``. The refusal is + ``UnsupportedOperation('truncate')``, the same one ``write`` already raises, and no + trade-off was needed between the house exception and the ABC's: pcapkit's + ``UnsupportedOperation`` subclasses ``io.UnsupportedOperation``, which subclasses + ``OSError``, so the in-library exception *is* the one the contract names. The + resizing ``truncate`` used to perform is not deleted, only made private as + ``_truncate_buffer``. It never touched the underlying stream -- it resizes a private + lookback window, which is the counter-argument the issue itself raised -- and it is + the only route to the "position sits before the window" state that ``seek`` and the + four buffered read paths must refuse, which #643 and #644 landed tests for; deleting + it would have taken their mechanism with them. ``writeable()``, separately, was never + an override of anything: ``'writable' in SeekableReader.__dict__`` was False and + ``SeekableReader.writable is io.IOBase.writable`` was True, so ``io``, ``shutil`` and + any third-party caller read the inherited value and never saw the one defined in this + file. Both answered False, which is the coincidence that hid it -- there was no + symptom to notice, and editing the misspelled method would silently have had no + effect. The value reported is unchanged and was always honest, the reader genuinely + being unable to write; only where the method was defined was wrong. Nothing in the + package called either method, confirmed by grep, so the risk is entirely to external + callers, which is what made this a question asked before it was acted on. The new + test asserts the property rather than the behaviour -- that a ``SeekableReader`` + raises the same exception type a read-only ``io.BufferedReader`` raises for the same + call -- with that type captured by running the call rather than named in the test, so + it tracks CPython across the 3.10--3.15 matrix instead of restating a belief about + it. 31 to 34 tests and 35 to 41 subtests over ``tests/corekit/test_io.py``, the + module at 100% coverage before and after (#645). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 33464d923a9443a087152527092702a807efb954 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 18:31:32 -0400 Subject: [PATCH 20/46] docs(changelog): the 1.5.0 entry for #675, whose code ships in #681 `register_protocol` keyed the protocol registry on `cls.__name__.upper()` and three dispatchable classes are named `HTTP`, so registering one silently displaced another. The entry records the measurement, the correction to the issue's `RegistryWarning` claim, why the guard departs from the presence-only siblings, and why re-keying belongs to #514. Regenerated CHANGELOG.md with util/changelog_md.py; --check exits 0. --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 42 +++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 54b13d599..b0037c370 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -103,6 +103,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- six of the 123 constant registries under `pcapkit/const/` rejected an invalid value in a way the built-in `enum` does not, and three of them did not reject it at all. `Flags`, `ftp.command.CommandType` and `reg.apptype.TransportProtocol` are `IntFlag` types that defined no `_missing_`, so the `aenum` `Flag` machinery composed a pseudo-member for any integer whatsoever: `Flags(-1)` returned `65520`, the OR of every declared TCP header flag, so a value no 16-bit wire field can hold read back as *every* flag set at once, while `Flags(-65536)` read back as none set. `ftp.command.Command`, `ftp.command.FEATCode` and `http.method.Method` are `StrEnum` types whose `_missing_` reached `value.upper()` before checking the type, so an integer raised `AttributeError: 'int' object has no attribute 'upper'`. All six now raise a bare `ValueError`, which is what `enum.IntEnum` raises for a value it does not define and what 113 of the 117 modules already did. Issue #647's own proposal -- replacing those 113 with `pcapkit.utilities.exceptions.EnumError` -- was deliberately rejected: `EnumError` is `(BaseError, TypeError)` and not a `ValueError` at all, so it would have diverged from the built-in it was meant to improve on, walked past the `except ValueError` in all 113 generated `get()` bodies and silently undone #584, and logged CRITICAL from `BaseError.__init__` once per discarded default. The one deliberate divergence from the built-in is kept and now pinned: the mutable registries look a value up, miss, and `extend_enum` it rather than raising, so `Method('FROBNICATE')`, `ProtectionAuthority(1 << 70)` and `AppType.get(65000, proto=tcp)` still register. The three flag guards end in `return super()._missing_(value)`, so in-range composites still decompose and #623 stays fixed. Two of the four unguarded modules needed no change -- `ipv6/extension_header.py` and `ftp.command.ConformanceRequirement` define no `_missing_` either, and `aenum` already raises the bare `ValueError` for them. `pcapkit/vendor/tcp/flags.py` turned out to carry the guard's value checker all along, as `FLAG = 'isinstance(value, int) and 4 <= value <= 15'`, while its template never interpolated it; those are the registry's *bit offsets* and its members are `1 << offset`, so emitting it unchanged would have rejected every composite, every member above bit 3, and `Flags(0)`. It now reads `0 <= value <= 0xFFFF`, the 16-bit field those bits live in. `TransportProtocol` reads its bound off `cls.__members__` instead of a literal, because `TransportProtocol.get` extends the registry at runtime at `max * 2` and a written-down bound would reject the member it had just grown. Fixed in the four bespoke templates under `pcapkit/vendor/` rather than in the generated tree -- `pcapkit/vendor/default.py` needed no change, since the 113 it emits were already correct -- and regenerated from the live IANA registries: 42 insertions and zero deletions across exactly 4 of the 134 files, so nothing else in the tree was stale, including the 30k-line `reg/apptype.py`. A new `tests/const/test_const_enum_builtin_parity.py` asserts the property the fix is actually about, that a constant registry and a stdlib `enum.IntEnum` raise the same exception type for the same invalid value, over all 123 registries rather than the six; it is keyed on `aenum.Enum` because an `IntFlag` is not a subclass of `IntEnum` -- its MRO runs through `Flag` instead -- which is how three of these six escaped the two existing sweeps. `tests/const/test_const_enum_get.py` drops `Flags` from `EXPECTED_TO_RESOLVE_ANYTHING` and its sweep grows 110 to 111, a registry that bounds its domain finally having a failure for `default` to fall back from (#647). - **Changed** -- `SeekableReader.truncate()` raises instead of returning a size, and the misspelled `writeable()` is now spelled `writable()`. Both are breaks for an external caller and neither breaks anything inside the package. `io.IOBase` documents one gate over two methods -- "If False, write() and truncate() will raise OSError" -- and this reader's `writable()` answers False, so a caller that checked it first, which is exactly what the contract invites, got a surprise either way round: `write` and `writelines` raised, and `truncate` returned its new size. What settled the question is that this is not a pure-Python nicety of `_pyio` that the accelerated path skips -- every ordinary read-only file object in CPython raises here, `open(path, 'rb').truncate()` giving `io.UnsupportedOperation: truncate` off the C `BufferedReader`, and `_pyio.BufferedReader` the same type from `_BufferedIOMixin.truncate`'s `_checkWritable()`. The refusal is `UnsupportedOperation('truncate')`, the same one `write` already raises, and no trade-off was needed between the house exception and the ABC's: pcapkit's `UnsupportedOperation` subclasses `io.UnsupportedOperation`, which subclasses `OSError`, so the in-library exception *is* the one the contract names. The resizing `truncate` used to perform is not deleted, only made private as `_truncate_buffer`. It never touched the underlying stream -- it resizes a private lookback window, which is the counter-argument the issue itself raised -- and it is the only route to the "position sits before the window" state that `seek` and the four buffered read paths must refuse, which #643 and #644 landed tests for; deleting it would have taken their mechanism with them. `writeable()`, separately, was never an override of anything: `'writable' in SeekableReader.__dict__` was False and `SeekableReader.writable is io.IOBase.writable` was True, so `io`, `shutil` and any third-party caller read the inherited value and never saw the one defined in this file. Both answered False, which is the coincidence that hid it -- there was no symptom to notice, and editing the misspelled method would silently have had no effect. The value reported is unchanged and was always honest, the reader genuinely being unable to write; only where the method was defined was wrong. Nothing in the package called either method, confirmed by grep, so the risk is entirely to external callers, which is what made this a question asked before it was acted on. The new test asserts the property rather than the behaviour -- that a `SeekableReader` raises the same exception type a read-only `io.BufferedReader` raises for the same call -- with that type captured by running the call rather than named in the test, so it tracks CPython across the 3.10--3.15 matrix instead of restating a belief about it. 31 to 34 tests and 35 to 41 subtests over `tests/corekit/test_io.py`, the module at 100% coverage before and after (#645). +- **Fixed** -- `register_protocol` displaced one protocol class with another and said nothing about it. The registry behind `pcapkit.protocols.__proto__` is keyed on `cls.__name__.upper()`, and three dispatchable classes are all named `HTTP` -- the generic base `pcapkit.protocols.application.http.HTTP` and the two version implementations in `httpv1` and `httpv2` -- so all three compete for the single key `'HTTP'` and a bare assignment decided the winner. Measured before the fix: `__proto__['HTTP']` went from `pcapkit.protocols.application.http.HTTP` to the `httpv2` class on `register_protocol(httpv2.HTTP)`, with no exception, no log line and no `RegistryWarning`, and `ProtocolBase.expand_comp('HTTP')` then resolved to whichever class had registered last. The overwrite now raises a `RegistryWarning` naming both the displaced and the replacing class, the defining module being the only thing that tells the three apart. The sharpest path to it is not a call to the registrar at all: `Protocol.__init_subclass__` registers unconditionally, so merely *defining* a subclass of the public `Protocol` under a name a built-in already holds displaced that built-in, with nothing anywhere in the user's code to mark the moment the dispatch table changed meaning. A plain `import pcapkit` is unaffected and emits zero new warnings, measured not assumed -- the three built-in `HTTP` classes derive from `ProtocolBase` rather than from the public `Protocol`, so `__init_subclass__` never fires for them, and the import-time seeding in `pcapkit/protocols/__init__.py` keys off the distinct names `HTTP`, `HTTPv1` and `HTTPv2`. The issue's framing that this was the one registrar in `pcapkit/foundation/registry/` with no `RegistryWarning` is right on the count and wrong on the reason: no function in that package warns about anything, the siblings warning only by delegating to a classmethod that carries the guarded `if code in cls.__xxx__: warn(...)`. The accurate statement is the stronger one -- `register_protocol` is the only keyed registrar in that package, and the only `__name__.upper()`-keyed registry anywhere in `pcapkit/`, that mutates its target with no guard *and* no delegate that could supply one, its target being a module-level dict with no class behind it to hold the guard. The new guard deliberately reads "key present and the incumbent is a different class" where every sibling warns on mere presence: the sibling keys are codes the caller passes, whereas this key is derived from the class and this function is the funnel all nine wrapper registrars end in, so registering one class under two codes -- `register_tcp` then `register_udp`, which is supported and documented -- reaches it twice with nothing displaced, and warning there would put noise on a documented path whose wholesale filter is precisely what would then hide the real collision. The collision is reported, not resolved: re-keying is a registry-format change that none of the bare-name readers can absorb, every one of them degrading *silently* to a plain string or to `Raw` on a miss rather than raising, so it belongs to the registry redesign in #514, which this change is sequenced ahead of rather than part of. 9 to 13 tests and 80 to 87 subtests over `tests/foundation/registry/`, the touched module holding 88% coverage with its misses flat at 27 (#675). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 592f0e5c6..f3ec95ed7 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1865,6 +1865,48 @@ pull requests between #326 and #509. it tracks CPython across the 3.10--3.15 matrix instead of restating a belief about it. 31 to 34 tests and 35 to 41 subtests over ``tests/corekit/test_io.py``, the module at 100% coverage before and after (#645). +* **Fixed** -- ``register_protocol`` displaced one protocol class with another and + said nothing about it. The registry behind ``pcapkit.protocols.__proto__`` is keyed + on ``cls.__name__.upper()``, and three dispatchable classes are all named ``HTTP`` + -- the generic base ``pcapkit.protocols.application.http.HTTP`` and the two version + implementations in ``httpv1`` and ``httpv2`` -- so all three compete for the single + key ``'HTTP'`` and a bare assignment decided the winner. Measured before the fix: + ``__proto__['HTTP']`` went from ``pcapkit.protocols.application.http.HTTP`` to the + ``httpv2`` class on ``register_protocol(httpv2.HTTP)``, with no exception, no log + line and no ``RegistryWarning``, and ``ProtocolBase.expand_comp('HTTP')`` then + resolved to whichever class had registered last. The overwrite now raises a + ``RegistryWarning`` naming both the displaced and the replacing class, the defining + module being the only thing that tells the three apart. The sharpest path to it is + not a call to the registrar at all: ``Protocol.__init_subclass__`` registers + unconditionally, so merely *defining* a subclass of the public ``Protocol`` under a + name a built-in already holds displaced that built-in, with nothing anywhere in the + user's code to mark the moment the dispatch table changed meaning. A plain + ``import pcapkit`` is unaffected and emits zero new warnings, measured not assumed -- + the three built-in ``HTTP`` classes derive from ``ProtocolBase`` rather than from the + public ``Protocol``, so ``__init_subclass__`` never fires for them, and the + import-time seeding in ``pcapkit/protocols/__init__.py`` keys off the distinct names + ``HTTP``, ``HTTPv1`` and ``HTTPv2``. The issue's framing that this was the one + registrar in ``pcapkit/foundation/registry/`` with no ``RegistryWarning`` is right on + the count and wrong on the reason: no function in that package warns about anything, + the siblings warning only by delegating to a classmethod that carries the guarded + ``if code in cls.__xxx__: warn(...)``. The accurate statement is the stronger one -- + ``register_protocol`` is the only keyed registrar in that package, and the only + ``__name__.upper()``-keyed registry anywhere in ``pcapkit/``, that mutates its target + with no guard *and* no delegate that could supply one, its target being a + module-level dict with no class behind it to hold the guard. The new guard + deliberately reads "key present and the incumbent is a different class" where every + sibling warns on mere presence: the sibling keys are codes the caller passes, whereas + this key is derived from the class and this function is the funnel all nine wrapper + registrars end in, so registering one class under two codes -- ``register_tcp`` then + ``register_udp``, which is supported and documented -- reaches it twice with nothing + displaced, and warning there would put noise on a documented path whose wholesale + filter is precisely what would then hide the real collision. The collision is + reported, not resolved: re-keying is a registry-format change that none of the + bare-name readers can absorb, every one of them degrading *silently* to a plain + string or to ``Raw`` on a miss rather than raising, so it belongs to the registry + redesign in #514, which this change is sequenced ahead of rather than part of. 9 to + 13 tests and 80 to 87 subtests over ``tests/foundation/registry/``, the touched + module holding 88% coverage with its misses flat at 27 (#675). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From dbcd0e9aacec6aeea1f39f00ac6c07def7102e4a Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 18:53:08 -0400 Subject: [PATCH 21/46] docs(changelog): the 1.5.0 entry for #646, whose code ships in #683 Every PCAP-NG packet block lost its captured octets: they were extracted from the block schema and then overwritten by `ProtocolBase.__init__` with `self.packet.payload`, which the inherited `packet` had split at `PCAPNG.length` -- the wire's Block Total Length. The entry records the measurement on the committed `dhcp.pcapng`, the three affected block types and their three payload offsets, why the fix belongs at `PCAPNG.packet` rather than at the injection site, the 104-octet dump that made it a wire-format defect, why it ships labelled breaking, and that #678 is measurably unaffected. CHANGELOG.md regenerated with `util/changelog_md.py`; `--check` exits 0. --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 57 +++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index b0037c370..a999c4c49 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -104,6 +104,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- six of the 123 constant registries under `pcapkit/const/` rejected an invalid value in a way the built-in `enum` does not, and three of them did not reject it at all. `Flags`, `ftp.command.CommandType` and `reg.apptype.TransportProtocol` are `IntFlag` types that defined no `_missing_`, so the `aenum` `Flag` machinery composed a pseudo-member for any integer whatsoever: `Flags(-1)` returned `65520`, the OR of every declared TCP header flag, so a value no 16-bit wire field can hold read back as *every* flag set at once, while `Flags(-65536)` read back as none set. `ftp.command.Command`, `ftp.command.FEATCode` and `http.method.Method` are `StrEnum` types whose `_missing_` reached `value.upper()` before checking the type, so an integer raised `AttributeError: 'int' object has no attribute 'upper'`. All six now raise a bare `ValueError`, which is what `enum.IntEnum` raises for a value it does not define and what 113 of the 117 modules already did. Issue #647's own proposal -- replacing those 113 with `pcapkit.utilities.exceptions.EnumError` -- was deliberately rejected: `EnumError` is `(BaseError, TypeError)` and not a `ValueError` at all, so it would have diverged from the built-in it was meant to improve on, walked past the `except ValueError` in all 113 generated `get()` bodies and silently undone #584, and logged CRITICAL from `BaseError.__init__` once per discarded default. The one deliberate divergence from the built-in is kept and now pinned: the mutable registries look a value up, miss, and `extend_enum` it rather than raising, so `Method('FROBNICATE')`, `ProtectionAuthority(1 << 70)` and `AppType.get(65000, proto=tcp)` still register. The three flag guards end in `return super()._missing_(value)`, so in-range composites still decompose and #623 stays fixed. Two of the four unguarded modules needed no change -- `ipv6/extension_header.py` and `ftp.command.ConformanceRequirement` define no `_missing_` either, and `aenum` already raises the bare `ValueError` for them. `pcapkit/vendor/tcp/flags.py` turned out to carry the guard's value checker all along, as `FLAG = 'isinstance(value, int) and 4 <= value <= 15'`, while its template never interpolated it; those are the registry's *bit offsets* and its members are `1 << offset`, so emitting it unchanged would have rejected every composite, every member above bit 3, and `Flags(0)`. It now reads `0 <= value <= 0xFFFF`, the 16-bit field those bits live in. `TransportProtocol` reads its bound off `cls.__members__` instead of a literal, because `TransportProtocol.get` extends the registry at runtime at `max * 2` and a written-down bound would reject the member it had just grown. Fixed in the four bespoke templates under `pcapkit/vendor/` rather than in the generated tree -- `pcapkit/vendor/default.py` needed no change, since the 113 it emits were already correct -- and regenerated from the live IANA registries: 42 insertions and zero deletions across exactly 4 of the 134 files, so nothing else in the tree was stale, including the 30k-line `reg/apptype.py`. A new `tests/const/test_const_enum_builtin_parity.py` asserts the property the fix is actually about, that a constant registry and a stdlib `enum.IntEnum` raise the same exception type for the same invalid value, over all 123 registries rather than the six; it is keyed on `aenum.Enum` because an `IntFlag` is not a subclass of `IntEnum` -- its MRO runs through `Flag` instead -- which is how three of these six escaped the two existing sweeps. `tests/const/test_const_enum_get.py` drops `Flags` from `EXPECTED_TO_RESOLVE_ANYTHING` and its sweep grows 110 to 111, a registry that bounds its domain finally having a failure for `default` to fall back from (#647). - **Changed** -- `SeekableReader.truncate()` raises instead of returning a size, and the misspelled `writeable()` is now spelled `writable()`. Both are breaks for an external caller and neither breaks anything inside the package. `io.IOBase` documents one gate over two methods -- "If False, write() and truncate() will raise OSError" -- and this reader's `writable()` answers False, so a caller that checked it first, which is exactly what the contract invites, got a surprise either way round: `write` and `writelines` raised, and `truncate` returned its new size. What settled the question is that this is not a pure-Python nicety of `_pyio` that the accelerated path skips -- every ordinary read-only file object in CPython raises here, `open(path, 'rb').truncate()` giving `io.UnsupportedOperation: truncate` off the C `BufferedReader`, and `_pyio.BufferedReader` the same type from `_BufferedIOMixin.truncate`'s `_checkWritable()`. The refusal is `UnsupportedOperation('truncate')`, the same one `write` already raises, and no trade-off was needed between the house exception and the ABC's: pcapkit's `UnsupportedOperation` subclasses `io.UnsupportedOperation`, which subclasses `OSError`, so the in-library exception *is* the one the contract names. The resizing `truncate` used to perform is not deleted, only made private as `_truncate_buffer`. It never touched the underlying stream -- it resizes a private lookback window, which is the counter-argument the issue itself raised -- and it is the only route to the "position sits before the window" state that `seek` and the four buffered read paths must refuse, which #643 and #644 landed tests for; deleting it would have taken their mechanism with them. `writeable()`, separately, was never an override of anything: `'writable' in SeekableReader.__dict__` was False and `SeekableReader.writable is io.IOBase.writable` was True, so `io`, `shutil` and any third-party caller read the inherited value and never saw the one defined in this file. Both answered False, which is the coincidence that hid it -- there was no symptom to notice, and editing the misspelled method would silently have had no effect. The value reported is unchanged and was always honest, the reader genuinely being unable to write; only where the method was defined was wrong. Nothing in the package called either method, confirmed by grep, so the risk is entirely to external callers, which is what made this a question asked before it was acted on. The new test asserts the property rather than the behaviour -- that a `SeekableReader` raises the same exception type a read-only `io.BufferedReader` raises for the same call -- with that type captured by running the call rather than named in the test, so it tracks CPython across the 3.10--3.15 matrix instead of restating a belief about it. 31 to 34 tests and 35 to 41 subtests over `tests/corekit/test_io.py`, the module at 100% coverage before and after (#645). - **Fixed** -- `register_protocol` displaced one protocol class with another and said nothing about it. The registry behind `pcapkit.protocols.__proto__` is keyed on `cls.__name__.upper()`, and three dispatchable classes are all named `HTTP` -- the generic base `pcapkit.protocols.application.http.HTTP` and the two version implementations in `httpv1` and `httpv2` -- so all three compete for the single key `'HTTP'` and a bare assignment decided the winner. Measured before the fix: `__proto__['HTTP']` went from `pcapkit.protocols.application.http.HTTP` to the `httpv2` class on `register_protocol(httpv2.HTTP)`, with no exception, no log line and no `RegistryWarning`, and `ProtocolBase.expand_comp('HTTP')` then resolved to whichever class had registered last. The overwrite now raises a `RegistryWarning` naming both the displaced and the replacing class, the defining module being the only thing that tells the three apart. The sharpest path to it is not a call to the registrar at all: `Protocol.__init_subclass__` registers unconditionally, so merely *defining* a subclass of the public `Protocol` under a name a built-in already holds displaced that built-in, with nothing anywhere in the user's code to mark the moment the dispatch table changed meaning. A plain `import pcapkit` is unaffected and emits zero new warnings, measured not assumed -- the three built-in `HTTP` classes derive from `ProtocolBase` rather than from the public `Protocol`, so `__init_subclass__` never fires for them, and the import-time seeding in `pcapkit/protocols/__init__.py` keys off the distinct names `HTTP`, `HTTPv1` and `HTTPv2`. The issue's framing that this was the one registrar in `pcapkit/foundation/registry/` with no `RegistryWarning` is right on the count and wrong on the reason: no function in that package warns about anything, the siblings warning only by delegating to a classmethod that carries the guarded `if code in cls.__xxx__: warn(...)`. The accurate statement is the stronger one -- `register_protocol` is the only keyed registrar in that package, and the only `__name__.upper()`-keyed registry anywhere in `pcapkit/`, that mutates its target with no guard *and* no delegate that could supply one, its target being a module-level dict with no class behind it to hold the guard. The new guard deliberately reads "key present and the incumbent is a different class" where every sibling warns on mere presence: the sibling keys are codes the caller passes, whereas this key is derived from the class and this function is the funnel all nine wrapper registrars end in, so registering one class under two codes -- `register_tcp` then `register_udp`, which is supported and documented -- reaches it twice with nothing displaced, and warning there would put noise on a documented path whose wholesale filter is precisely what would then hide the real collision. The collision is reported, not resolved: re-keying is a registry-format change that none of the bare-name readers can absorb, every one of them degrading *silently* to a plain string or to `Raw` on a miss rather than raising, so it belongs to the registry redesign in #514, which this change is sequenced ahead of rather than part of. 9 to 13 tests and 80 to 87 subtests over `tests/foundation/registry/`, the touched module holding 88% coverage with its misses flat at 27 (#675). +- **Fixed** -- every PCAP-NG packet block now carries the captured octets it declares. All four Enhanced Packet Blocks of the committed `examples/captures/dhcp.pcapng` reported `len(packet) == 0` against a `captured_len` of 314, 342, 314 and 342, and the Simple Packet Block and the obsolete Packet Block did the same, measured on a capture synthesised to carry one of each since no committed fixture has either. Those three -- exactly `PCAPNG.PACKET_TYPES` , and exactly the three schemas declaring `__payload__ = 'packet_data'` -- are the whole of the blast radius; every other block type carries no captured octets, reported `b''` before and still does. The octets were read correctly and then thrown away: `PCAPNG.unpack` extracted them from the block schema, and `ProtocolBase.__init__` overwrote the result with `self.packet.payload` , which was empty. The inherited `packet` splits a protocol at `self.length` octets of header and takes everything after as the payload, a contract that holds for a protocol laid out as a header followed by its payload and for nothing else. `PCAPNG.length` returns the wire's *Block Total Length*, so the split consumed the entire per-block buffer as header -- measured at `frame.length == 348` and `len(frame.packet.header) == 348` on a 348-octet block -- and, separately, a PCAP-NG block carries a *trailer* after its payload, the option list and a repeat of the total length, so no value of `length` could have made that split right. `captured_len` comes straight off the block header and survived, which is why the two disagreed rather than both being empty. The fix is at the computation rather than at the injection: `PCAPNG.packet` is overridden to take the payload from the block schema's `__payload__` field and the header from the octets ahead of it, summed out of the schema buffers so that the three block types' three different payload offsets -- 28, 12 and 28 octets -- are not hard-coded. Teaching `ProtocolBase.__init__` to skip a protocol that had already set `packet` was the smaller change and was rejected: it heals `frame.info.packet` while leaving the public `frame.packet` still reporting the whole block as header and `b''` as payload, which is the same defect from the other side. `PCAPNG.unpack` now reads that property instead of extracting the payload a second time of its own, leaving one source of truth where there were two attempts at one; `ProtocolBase.packet` gains the contract in its docstring and not one executable line, its 503 statements unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on the success path, which is why it ships labelled breaking: `frame.info.packet` goes from `b''` to hundreds of octets, `frame.packet.header` from the whole block to the pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what made this a wire-format defect rather than an API wart. `PCAPIO` writes `value.packet` after each 16-octet record header, so dumping those four blocks produced a **104-octet** file -- 24 of global header plus four record headers and no payload at all -- in which every record header declared hundreds of octets and delivered none, desynchronising any reader that walks by `incl_len` , which is all of them: pcapkit refused its own output with `ValueError: read length must be non-negative or -1` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes rather than against a length. `tests/protocols/test_pcapng_regression.py` grows 4 tests to 10 and 3 subtests to 19, with the payload expectations derived twice over -- spelled-out head and tail literals, and a `struct` -only re-parse of the fixture that owes nothing to the code under test -- and each synthesised payload given a different length modulo 4 so that an offset off by a field, or a payload that picked up the block's 32-bit padding, cannot pass by coincidence. `pcapkit/protocols/misc/pcapng.py` holds 99% across the change, its 18 new statements and 6 new branches all executed, its single miss the same pre-existing `_get_timezone` statement renumbered 1153 to 1248, and no new partial branch. The uncaught `ValueError` that an EOF-truncated PCAP-NG raises (#678) is untouched and was measured rather than assumed: all 101 truncation levels of `dhcp.pcapng` give byte-identical outcomes before and after, 96 of them that `ValueError` . `examples/captures/pcapng.txt` , a hand-regenerated legacy smoke reference, still shows the old frame-level `packet -> NIL` and wants a separate refresh (#646). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index f3ec95ed7..2aa9f11b1 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1907,6 +1907,63 @@ pull requests between #326 and #509. redesign in #514, which this change is sequenced ahead of rather than part of. 9 to 13 tests and 80 to 87 subtests over ``tests/foundation/registry/``, the touched module holding 88% coverage with its misses flat at 27 (#675). +* **Fixed** -- every PCAP-NG packet block now carries the captured octets it declares. All + four Enhanced Packet Blocks of the committed ``examples/captures/dhcp.pcapng`` reported + ``len(packet) == 0`` against a ``captured_len`` of 314, 342, 314 and 342, and the Simple + Packet Block and the obsolete Packet Block did the same, measured on a capture + synthesised to carry one of each since no committed fixture has either. Those three -- + exactly ``PCAPNG.PACKET_TYPES`` , and exactly the three schemas declaring + ``__payload__ = 'packet_data'`` -- are the whole of the blast radius; every other block + type carries no captured octets, reported ``b''`` before and still does. The octets were + read correctly and then thrown away: ``PCAPNG.unpack`` extracted them from the block + schema, and ``ProtocolBase.__init__`` overwrote the result with ``self.packet.payload`` + , which was empty. The inherited ``packet`` splits a protocol at ``self.length`` octets + of header and takes everything after as the payload, a contract that holds for a + protocol laid out as a header followed by its payload and for nothing else. + ``PCAPNG.length`` returns the wire's *Block Total Length*, so the split consumed the + entire per-block buffer as header -- measured at ``frame.length == 348`` and + ``len(frame.packet.header) == 348`` on a 348-octet block -- and, separately, a PCAP-NG + block carries a *trailer* after its payload, the option list and a repeat of the total + length, so no value of ``length`` could have made that split right. ``captured_len`` + comes straight off the block header and survived, which is why the two disagreed rather + than both being empty. The fix is at the computation rather than at the injection: + ``PCAPNG.packet`` is overridden to take the payload from the block schema's + ``__payload__`` field and the header from the octets ahead of it, summed out of the + schema buffers so that the three block types' three different payload offsets -- 28, 12 + and 28 octets -- are not hard-coded. Teaching ``ProtocolBase.__init__`` to skip a + protocol that had already set ``packet`` was the smaller change and was rejected: it + heals ``frame.info.packet`` while leaving the public ``frame.packet`` still reporting + the whole block as header and ``b''`` as payload, which is the same defect from the + other side. ``PCAPNG.unpack`` now reads that property instead of extracting the payload + a second time of its own, leaving one source of truth where there were two attempts at + one; ``ProtocolBase.packet`` gains the contract in its docstring and not one executable + line, its 503 statements unchanged. This moves the parse output of every PCAP-NG + capture, for two properties, on the success path, which is why it ships labelled + breaking: ``frame.info.packet`` goes from ``b''`` to hundreds of octets, + ``frame.packet.header`` from the whole block to the pre-payload prefix, and anything + diffing a dumped file sees it change. That dump is what made this a wire-format defect + rather than an API wart. ``PCAPIO`` writes ``value.packet`` after each 16-octet record + header, so dumping those four blocks produced a **104-octet** file -- 24 of global + header plus four record headers and no payload at all -- in which every record header + declared hundreds of octets and delivered none, desynchronising any reader that walks by + ``incl_len`` , which is all of them: pcapkit refused its own output with + ``ValueError: read length must be non-negative or -1`` and scapy silently returned 1 + packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and + each record's octets both asserted against the source capture's own hand-parsed bytes + rather than against a length. ``tests/protocols/test_pcapng_regression.py`` grows 4 + tests to 10 and 3 subtests to 19, with the payload expectations derived twice over -- + spelled-out head and tail literals, and a ``struct`` -only re-parse of the fixture that + owes nothing to the code under test -- and each synthesised payload given a different + length modulo 4 so that an offset off by a field, or a payload that picked up the + block's 32-bit padding, cannot pass by coincidence. ``pcapkit/protocols/misc/pcapng.py`` + holds 99% across the change, its 18 new statements and 6 new branches all executed, its + single miss the same pre-existing ``_get_timezone`` statement renumbered 1153 to 1248, + and no new partial branch. The uncaught ``ValueError`` that an EOF-truncated PCAP-NG + raises (#678) is untouched and was measured rather than assumed: all 101 truncation + levels of ``dhcp.pcapng`` give byte-identical outcomes before and after, 96 of them that + ``ValueError`` . ``examples/captures/pcapng.txt`` , a hand-regenerated legacy smoke + reference, still shows the old frame-level ``packet -> NIL`` and wants a separate + refresh (#646). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 2069ae45ccd4ff774ba28211ddda10302a1565a7 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 19:10:07 -0400 Subject: [PATCH 22/46] docs(changelog): correct the #646 entry's test counts and six run-together literals The counts in `dbcd0e9aa` were written before the tests were hardened: the file grows 4 tests to 11 and 3 subtests to 24, not to 10 and 19. The entry now also records the two shapes no fixture exercised -- an option area after the captured data, which every block of `dhcp.pcapng` lacks, and a big-endian section -- and the two ways a snapped block is expressed, since those are what make the "every affected block type" claim more than the one type a fixture happens to have. Six inline literals also lost the punctuation that was attached to them: `` ``PCAPNG.PACKET_TYPES`` , `` and five like it, plus `` ``struct`` -only ``, from a line-wrapper that tokenised the paragraph without preserving adjacency. Reflowed with one that does, and with the check the generator already enforces -- a literal may not span a line break -- asserted rather than discovered. CHANGELOG.md regenerated with `util/changelog_md.py`; `--check` exits 0. --- CHANGELOG.md | 2 +- docs/source/changelog/1.5.0.rst | 43 ++++++++++++++++++++------------- 2 files changed, 27 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a999c4c49..af9cbf18c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -104,7 +104,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- six of the 123 constant registries under `pcapkit/const/` rejected an invalid value in a way the built-in `enum` does not, and three of them did not reject it at all. `Flags`, `ftp.command.CommandType` and `reg.apptype.TransportProtocol` are `IntFlag` types that defined no `_missing_`, so the `aenum` `Flag` machinery composed a pseudo-member for any integer whatsoever: `Flags(-1)` returned `65520`, the OR of every declared TCP header flag, so a value no 16-bit wire field can hold read back as *every* flag set at once, while `Flags(-65536)` read back as none set. `ftp.command.Command`, `ftp.command.FEATCode` and `http.method.Method` are `StrEnum` types whose `_missing_` reached `value.upper()` before checking the type, so an integer raised `AttributeError: 'int' object has no attribute 'upper'`. All six now raise a bare `ValueError`, which is what `enum.IntEnum` raises for a value it does not define and what 113 of the 117 modules already did. Issue #647's own proposal -- replacing those 113 with `pcapkit.utilities.exceptions.EnumError` -- was deliberately rejected: `EnumError` is `(BaseError, TypeError)` and not a `ValueError` at all, so it would have diverged from the built-in it was meant to improve on, walked past the `except ValueError` in all 113 generated `get()` bodies and silently undone #584, and logged CRITICAL from `BaseError.__init__` once per discarded default. The one deliberate divergence from the built-in is kept and now pinned: the mutable registries look a value up, miss, and `extend_enum` it rather than raising, so `Method('FROBNICATE')`, `ProtectionAuthority(1 << 70)` and `AppType.get(65000, proto=tcp)` still register. The three flag guards end in `return super()._missing_(value)`, so in-range composites still decompose and #623 stays fixed. Two of the four unguarded modules needed no change -- `ipv6/extension_header.py` and `ftp.command.ConformanceRequirement` define no `_missing_` either, and `aenum` already raises the bare `ValueError` for them. `pcapkit/vendor/tcp/flags.py` turned out to carry the guard's value checker all along, as `FLAG = 'isinstance(value, int) and 4 <= value <= 15'`, while its template never interpolated it; those are the registry's *bit offsets* and its members are `1 << offset`, so emitting it unchanged would have rejected every composite, every member above bit 3, and `Flags(0)`. It now reads `0 <= value <= 0xFFFF`, the 16-bit field those bits live in. `TransportProtocol` reads its bound off `cls.__members__` instead of a literal, because `TransportProtocol.get` extends the registry at runtime at `max * 2` and a written-down bound would reject the member it had just grown. Fixed in the four bespoke templates under `pcapkit/vendor/` rather than in the generated tree -- `pcapkit/vendor/default.py` needed no change, since the 113 it emits were already correct -- and regenerated from the live IANA registries: 42 insertions and zero deletions across exactly 4 of the 134 files, so nothing else in the tree was stale, including the 30k-line `reg/apptype.py`. A new `tests/const/test_const_enum_builtin_parity.py` asserts the property the fix is actually about, that a constant registry and a stdlib `enum.IntEnum` raise the same exception type for the same invalid value, over all 123 registries rather than the six; it is keyed on `aenum.Enum` because an `IntFlag` is not a subclass of `IntEnum` -- its MRO runs through `Flag` instead -- which is how three of these six escaped the two existing sweeps. `tests/const/test_const_enum_get.py` drops `Flags` from `EXPECTED_TO_RESOLVE_ANYTHING` and its sweep grows 110 to 111, a registry that bounds its domain finally having a failure for `default` to fall back from (#647). - **Changed** -- `SeekableReader.truncate()` raises instead of returning a size, and the misspelled `writeable()` is now spelled `writable()`. Both are breaks for an external caller and neither breaks anything inside the package. `io.IOBase` documents one gate over two methods -- "If False, write() and truncate() will raise OSError" -- and this reader's `writable()` answers False, so a caller that checked it first, which is exactly what the contract invites, got a surprise either way round: `write` and `writelines` raised, and `truncate` returned its new size. What settled the question is that this is not a pure-Python nicety of `_pyio` that the accelerated path skips -- every ordinary read-only file object in CPython raises here, `open(path, 'rb').truncate()` giving `io.UnsupportedOperation: truncate` off the C `BufferedReader`, and `_pyio.BufferedReader` the same type from `_BufferedIOMixin.truncate`'s `_checkWritable()`. The refusal is `UnsupportedOperation('truncate')`, the same one `write` already raises, and no trade-off was needed between the house exception and the ABC's: pcapkit's `UnsupportedOperation` subclasses `io.UnsupportedOperation`, which subclasses `OSError`, so the in-library exception *is* the one the contract names. The resizing `truncate` used to perform is not deleted, only made private as `_truncate_buffer`. It never touched the underlying stream -- it resizes a private lookback window, which is the counter-argument the issue itself raised -- and it is the only route to the "position sits before the window" state that `seek` and the four buffered read paths must refuse, which #643 and #644 landed tests for; deleting it would have taken their mechanism with them. `writeable()`, separately, was never an override of anything: `'writable' in SeekableReader.__dict__` was False and `SeekableReader.writable is io.IOBase.writable` was True, so `io`, `shutil` and any third-party caller read the inherited value and never saw the one defined in this file. Both answered False, which is the coincidence that hid it -- there was no symptom to notice, and editing the misspelled method would silently have had no effect. The value reported is unchanged and was always honest, the reader genuinely being unable to write; only where the method was defined was wrong. Nothing in the package called either method, confirmed by grep, so the risk is entirely to external callers, which is what made this a question asked before it was acted on. The new test asserts the property rather than the behaviour -- that a `SeekableReader` raises the same exception type a read-only `io.BufferedReader` raises for the same call -- with that type captured by running the call rather than named in the test, so it tracks CPython across the 3.10--3.15 matrix instead of restating a belief about it. 31 to 34 tests and 35 to 41 subtests over `tests/corekit/test_io.py`, the module at 100% coverage before and after (#645). - **Fixed** -- `register_protocol` displaced one protocol class with another and said nothing about it. The registry behind `pcapkit.protocols.__proto__` is keyed on `cls.__name__.upper()`, and three dispatchable classes are all named `HTTP` -- the generic base `pcapkit.protocols.application.http.HTTP` and the two version implementations in `httpv1` and `httpv2` -- so all three compete for the single key `'HTTP'` and a bare assignment decided the winner. Measured before the fix: `__proto__['HTTP']` went from `pcapkit.protocols.application.http.HTTP` to the `httpv2` class on `register_protocol(httpv2.HTTP)`, with no exception, no log line and no `RegistryWarning`, and `ProtocolBase.expand_comp('HTTP')` then resolved to whichever class had registered last. The overwrite now raises a `RegistryWarning` naming both the displaced and the replacing class, the defining module being the only thing that tells the three apart. The sharpest path to it is not a call to the registrar at all: `Protocol.__init_subclass__` registers unconditionally, so merely *defining* a subclass of the public `Protocol` under a name a built-in already holds displaced that built-in, with nothing anywhere in the user's code to mark the moment the dispatch table changed meaning. A plain `import pcapkit` is unaffected and emits zero new warnings, measured not assumed -- the three built-in `HTTP` classes derive from `ProtocolBase` rather than from the public `Protocol`, so `__init_subclass__` never fires for them, and the import-time seeding in `pcapkit/protocols/__init__.py` keys off the distinct names `HTTP`, `HTTPv1` and `HTTPv2`. The issue's framing that this was the one registrar in `pcapkit/foundation/registry/` with no `RegistryWarning` is right on the count and wrong on the reason: no function in that package warns about anything, the siblings warning only by delegating to a classmethod that carries the guarded `if code in cls.__xxx__: warn(...)`. The accurate statement is the stronger one -- `register_protocol` is the only keyed registrar in that package, and the only `__name__.upper()`-keyed registry anywhere in `pcapkit/`, that mutates its target with no guard *and* no delegate that could supply one, its target being a module-level dict with no class behind it to hold the guard. The new guard deliberately reads "key present and the incumbent is a different class" where every sibling warns on mere presence: the sibling keys are codes the caller passes, whereas this key is derived from the class and this function is the funnel all nine wrapper registrars end in, so registering one class under two codes -- `register_tcp` then `register_udp`, which is supported and documented -- reaches it twice with nothing displaced, and warning there would put noise on a documented path whose wholesale filter is precisely what would then hide the real collision. The collision is reported, not resolved: re-keying is a registry-format change that none of the bare-name readers can absorb, every one of them degrading *silently* to a plain string or to `Raw` on a miss rather than raising, so it belongs to the registry redesign in #514, which this change is sequenced ahead of rather than part of. 9 to 13 tests and 80 to 87 subtests over `tests/foundation/registry/`, the touched module holding 88% coverage with its misses flat at 27 (#675). -- **Fixed** -- every PCAP-NG packet block now carries the captured octets it declares. All four Enhanced Packet Blocks of the committed `examples/captures/dhcp.pcapng` reported `len(packet) == 0` against a `captured_len` of 314, 342, 314 and 342, and the Simple Packet Block and the obsolete Packet Block did the same, measured on a capture synthesised to carry one of each since no committed fixture has either. Those three -- exactly `PCAPNG.PACKET_TYPES` , and exactly the three schemas declaring `__payload__ = 'packet_data'` -- are the whole of the blast radius; every other block type carries no captured octets, reported `b''` before and still does. The octets were read correctly and then thrown away: `PCAPNG.unpack` extracted them from the block schema, and `ProtocolBase.__init__` overwrote the result with `self.packet.payload` , which was empty. The inherited `packet` splits a protocol at `self.length` octets of header and takes everything after as the payload, a contract that holds for a protocol laid out as a header followed by its payload and for nothing else. `PCAPNG.length` returns the wire's *Block Total Length*, so the split consumed the entire per-block buffer as header -- measured at `frame.length == 348` and `len(frame.packet.header) == 348` on a 348-octet block -- and, separately, a PCAP-NG block carries a *trailer* after its payload, the option list and a repeat of the total length, so no value of `length` could have made that split right. `captured_len` comes straight off the block header and survived, which is why the two disagreed rather than both being empty. The fix is at the computation rather than at the injection: `PCAPNG.packet` is overridden to take the payload from the block schema's `__payload__` field and the header from the octets ahead of it, summed out of the schema buffers so that the three block types' three different payload offsets -- 28, 12 and 28 octets -- are not hard-coded. Teaching `ProtocolBase.__init__` to skip a protocol that had already set `packet` was the smaller change and was rejected: it heals `frame.info.packet` while leaving the public `frame.packet` still reporting the whole block as header and `b''` as payload, which is the same defect from the other side. `PCAPNG.unpack` now reads that property instead of extracting the payload a second time of its own, leaving one source of truth where there were two attempts at one; `ProtocolBase.packet` gains the contract in its docstring and not one executable line, its 503 statements unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on the success path, which is why it ships labelled breaking: `frame.info.packet` goes from `b''` to hundreds of octets, `frame.packet.header` from the whole block to the pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what made this a wire-format defect rather than an API wart. `PCAPIO` writes `value.packet` after each 16-octet record header, so dumping those four blocks produced a **104-octet** file -- 24 of global header plus four record headers and no payload at all -- in which every record header declared hundreds of octets and delivered none, desynchronising any reader that walks by `incl_len` , which is all of them: pcapkit refused its own output with `ValueError: read length must be non-negative or -1` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes rather than against a length. `tests/protocols/test_pcapng_regression.py` grows 4 tests to 10 and 3 subtests to 19, with the payload expectations derived twice over -- spelled-out head and tail literals, and a `struct` -only re-parse of the fixture that owes nothing to the code under test -- and each synthesised payload given a different length modulo 4 so that an offset off by a field, or a payload that picked up the block's 32-bit padding, cannot pass by coincidence. `pcapkit/protocols/misc/pcapng.py` holds 99% across the change, its 18 new statements and 6 new branches all executed, its single miss the same pre-existing `_get_timezone` statement renumbered 1153 to 1248, and no new partial branch. The uncaught `ValueError` that an EOF-truncated PCAP-NG raises (#678) is untouched and was measured rather than assumed: all 101 truncation levels of `dhcp.pcapng` give byte-identical outcomes before and after, 96 of them that `ValueError` . `examples/captures/pcapng.txt` , a hand-regenerated legacy smoke reference, still shows the old frame-level `packet -> NIL` and wants a separate refresh (#646). +- **Fixed** -- every PCAP-NG packet block now carries the captured octets it declares. All four Enhanced Packet Blocks of the committed `examples/captures/dhcp.pcapng` reported `len(packet) == 0` against a `captured_len` of 314, 342, 314 and 342, and the Simple Packet Block and the obsolete Packet Block did the same, measured on a capture synthesised to carry one of each since no committed fixture has either. Those three -- exactly `PCAPNG.PACKET_TYPES`, and exactly the three schemas declaring `__payload__ = 'packet_data'` -- are the whole of the blast radius; every other block type carries no captured octets, reported `b''` before and still does. The octets were read correctly and then thrown away: `PCAPNG.unpack` extracted them from the block schema, and `ProtocolBase.__init__` overwrote the result with `self.packet.payload`, which was empty. The inherited `packet` splits a protocol at `self.length` octets of header and takes everything after as the payload, a contract that holds for a protocol laid out as a header followed by its payload and for nothing else. `PCAPNG.length` returns the wire's *Block Total Length*, so the split consumed the entire per-block buffer as header -- measured at `frame.length == 348` and `len(frame.packet.header) == 348` on a 348-octet block -- and, separately, a PCAP-NG block carries a *trailer* after its payload, the option list and a repeat of the total length, so no value of `length` could have made that split right. `captured_len` comes straight off the block header and survived, which is why the two disagreed rather than both being empty. The fix is at the computation rather than at the injection: `PCAPNG.packet` is overridden to take the payload from the block schema's `__payload__` field and the header from the octets ahead of it, summed out of the schema buffers so that the three block types' three different payload offsets -- 28, 12 and 28 octets -- are not hard-coded. Teaching `ProtocolBase.__init__` to skip a protocol that had already set `packet` was the smaller change and was rejected: it heals `frame.info.packet` while leaving the public `frame.packet` still reporting the whole block as header and `b''` as payload, which is the same defect from the other side. `PCAPNG.unpack` now reads that property instead of extracting the payload a second time of its own, leaving one source of truth where there were two attempts at one; `ProtocolBase.packet` gains the contract in its docstring and not one executable line, its 503 statements unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on the success path, which is why it ships labelled breaking: `frame.info.packet` goes from `b''` to hundreds of octets, `frame.packet.header` from the whole block to the pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what made this a wire-format defect rather than an API wart. `PCAPIO` writes `value.packet` after each 16-octet record header, so dumping those four blocks produced a **104-octet** file -- 24 of global header plus four record headers and no payload at all -- in which every record header declared hundreds of octets and delivered none, desynchronising any reader that walks by `incl_len`, which is all of them: pcapkit refused its own output with `ValueError: read length must be non-negative or -1` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes rather than against a length. `tests/protocols/test_pcapng_regression.py` grows 4 tests to 11 and 3 subtests to 24, with the payload expectations derived twice over -- spelled-out head and tail literals, and a `struct`-only re-parse of the fixture that owes nothing to the code under test -- and each synthesised payload given a different length modulo 4 so that an offset off by a field, or a payload that picked up the block's 32-bit padding, cannot pass by coincidence. Two shapes no fixture exercised are covered there too, both found while writing the tests rather than after: an option area *after* the captured data, which every block of `dhcp.pcapng` lacks and which is exactly what an offset walked from the wrong end would swallow, and a big-endian section, whose fields differ even though the payload offsets do not. A snapped block is covered in both the shapes that express it -- an Enhanced Packet Block declaring `captured_len` below `original_len`, and a Simple Packet Block whose captured length is bounded by the interface's `snaplen` -- neither of which may come back padded out to the on-wire length. The offsets hold at 28, 12 and 28 across all of it, which is the property the walk has to have. `pcapkit/protocols/misc/pcapng.py` holds 99% across the change, its 18 new statements and 6 new branches all executed, its single miss the same pre-existing `_get_timezone` statement renumbered 1153 to 1248, and no new partial branch. The uncaught `ValueError` that an EOF-truncated PCAP-NG raises (#678) is untouched and was measured rather than assumed: all 101 truncation levels of `dhcp.pcapng` give byte-identical outcomes before and after, 96 of them that `ValueError`. `examples/captures/pcapng.txt`, a hand-regenerated legacy smoke reference, still shows the old frame-level `packet -> NIL` and wants a separate refresh (#646). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 2aa9f11b1..0efd79440 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1912,16 +1912,16 @@ pull requests between #326 and #509. ``len(packet) == 0`` against a ``captured_len`` of 314, 342, 314 and 342, and the Simple Packet Block and the obsolete Packet Block did the same, measured on a capture synthesised to carry one of each since no committed fixture has either. Those three -- - exactly ``PCAPNG.PACKET_TYPES`` , and exactly the three schemas declaring + exactly ``PCAPNG.PACKET_TYPES``, and exactly the three schemas declaring ``__payload__ = 'packet_data'`` -- are the whole of the blast radius; every other block type carries no captured octets, reported ``b''`` before and still does. The octets were read correctly and then thrown away: ``PCAPNG.unpack`` extracted them from the block - schema, and ``ProtocolBase.__init__`` overwrote the result with ``self.packet.payload`` - , which was empty. The inherited ``packet`` splits a protocol at ``self.length`` octets - of header and takes everything after as the payload, a contract that holds for a - protocol laid out as a header followed by its payload and for nothing else. - ``PCAPNG.length`` returns the wire's *Block Total Length*, so the split consumed the - entire per-block buffer as header -- measured at ``frame.length == 348`` and + schema, and ``ProtocolBase.__init__`` overwrote the result with ``self.packet.payload``, + which was empty. The inherited ``packet`` splits a protocol at ``self.length`` octets of + header and takes everything after as the payload, a contract that holds for a protocol + laid out as a header followed by its payload and for nothing else. ``PCAPNG.length`` + returns the wire's *Block Total Length*, so the split consumed the entire per-block + buffer as header -- measured at ``frame.length == 348`` and ``len(frame.packet.header) == 348`` on a 348-octet block -- and, separately, a PCAP-NG block carries a *trailer* after its payload, the option list and a repeat of the total length, so no value of ``length`` could have made that split right. ``captured_len`` @@ -1946,22 +1946,31 @@ pull requests between #326 and #509. header, so dumping those four blocks produced a **104-octet** file -- 24 of global header plus four record headers and no payload at all -- in which every record header declared hundreds of octets and delivered none, desynchronising any reader that walks by - ``incl_len`` , which is all of them: pcapkit refused its own output with + ``incl_len``, which is all of them: pcapkit refused its own output with ``ValueError: read length must be non-negative or -1`` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes rather than against a length. ``tests/protocols/test_pcapng_regression.py`` grows 4 - tests to 10 and 3 subtests to 19, with the payload expectations derived twice over -- - spelled-out head and tail literals, and a ``struct`` -only re-parse of the fixture that + tests to 11 and 3 subtests to 24, with the payload expectations derived twice over -- + spelled-out head and tail literals, and a ``struct``-only re-parse of the fixture that owes nothing to the code under test -- and each synthesised payload given a different length modulo 4 so that an offset off by a field, or a payload that picked up the - block's 32-bit padding, cannot pass by coincidence. ``pcapkit/protocols/misc/pcapng.py`` - holds 99% across the change, its 18 new statements and 6 new branches all executed, its - single miss the same pre-existing ``_get_timezone`` statement renumbered 1153 to 1248, - and no new partial branch. The uncaught ``ValueError`` that an EOF-truncated PCAP-NG - raises (#678) is untouched and was measured rather than assumed: all 101 truncation - levels of ``dhcp.pcapng`` give byte-identical outcomes before and after, 96 of them that - ``ValueError`` . ``examples/captures/pcapng.txt`` , a hand-regenerated legacy smoke + block's 32-bit padding, cannot pass by coincidence. Two shapes no fixture exercised are + covered there too, both found while writing the tests rather than after: an option area + *after* the captured data, which every block of ``dhcp.pcapng`` lacks and which is + exactly what an offset walked from the wrong end would swallow, and a big-endian + section, whose fields differ even though the payload offsets do not. A snapped block is + covered in both the shapes that express it -- an Enhanced Packet Block declaring + ``captured_len`` below ``original_len``, and a Simple Packet Block whose captured length + is bounded by the interface's ``snaplen`` -- neither of which may come back padded out + to the on-wire length. The offsets hold at 28, 12 and 28 across all of it, which is the + property the walk has to have. ``pcapkit/protocols/misc/pcapng.py`` holds 99% across the + change, its 18 new statements and 6 new branches all executed, its single miss the same + pre-existing ``_get_timezone`` statement renumbered 1153 to 1248, and no new partial + branch. The uncaught ``ValueError`` that an EOF-truncated PCAP-NG raises (#678) is + untouched and was measured rather than assumed: all 101 truncation levels of + ``dhcp.pcapng`` give byte-identical outcomes before and after, 96 of them that + ``ValueError``. ``examples/captures/pcapng.txt``, a hand-regenerated legacy smoke reference, still shows the old frame-level ``packet -> NIL`` and wants a separate refresh (#646). From 5a95b52d183def35e27c03bbdc437e9775f449a2 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 19:55:55 -0400 Subject: [PATCH 23/46] docs(changelog): record what the #646 cross-review changed, and fix its coverage claim The cross-review on a second model returned NEEDS CHANGES on three points and the entry now reflects the code as it actually ships: * `PCAPNG.packet` is a plain `property`, not the `cached_property` it overrides. Caching it made a second `unpack` on one instance hand back the first call's octets -- an invariant the pre-#646 code held, since it recomputed from the schema every call. The entry records why, and the unit test now asserts it. * The coverage claim compared the base tree under the *old* tests against the branch under the new ones, which moves the suite and the library together and then credits the difference to either. Re-measured with the same tests on both: `misc/pcapng.py` 99.91% with its miss flat at 1, and `protocol.py` identical in every column rather than "232 to 228 misses", which is the check that its change really is docstring-only. * The #678 parity claim quoted a tally without saying what "101 truncation levels" meant; three readings of it give three different tallies. The set is now spelled as the expression that produced it, and the per-level fingerprint is given, since a matching aggregate can hide two levels that swapped. CHANGELOG.md regenerated with `util/changelog_md.py`; `--check` exits 0. --- CHANGELOG.md | 2 +- docs/source/changelog/1.5.0.rst | 61 ++++++++++++++++++++++----------- 2 files changed, 42 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index af9cbf18c..259a60ecd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -104,7 +104,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- six of the 123 constant registries under `pcapkit/const/` rejected an invalid value in a way the built-in `enum` does not, and three of them did not reject it at all. `Flags`, `ftp.command.CommandType` and `reg.apptype.TransportProtocol` are `IntFlag` types that defined no `_missing_`, so the `aenum` `Flag` machinery composed a pseudo-member for any integer whatsoever: `Flags(-1)` returned `65520`, the OR of every declared TCP header flag, so a value no 16-bit wire field can hold read back as *every* flag set at once, while `Flags(-65536)` read back as none set. `ftp.command.Command`, `ftp.command.FEATCode` and `http.method.Method` are `StrEnum` types whose `_missing_` reached `value.upper()` before checking the type, so an integer raised `AttributeError: 'int' object has no attribute 'upper'`. All six now raise a bare `ValueError`, which is what `enum.IntEnum` raises for a value it does not define and what 113 of the 117 modules already did. Issue #647's own proposal -- replacing those 113 with `pcapkit.utilities.exceptions.EnumError` -- was deliberately rejected: `EnumError` is `(BaseError, TypeError)` and not a `ValueError` at all, so it would have diverged from the built-in it was meant to improve on, walked past the `except ValueError` in all 113 generated `get()` bodies and silently undone #584, and logged CRITICAL from `BaseError.__init__` once per discarded default. The one deliberate divergence from the built-in is kept and now pinned: the mutable registries look a value up, miss, and `extend_enum` it rather than raising, so `Method('FROBNICATE')`, `ProtectionAuthority(1 << 70)` and `AppType.get(65000, proto=tcp)` still register. The three flag guards end in `return super()._missing_(value)`, so in-range composites still decompose and #623 stays fixed. Two of the four unguarded modules needed no change -- `ipv6/extension_header.py` and `ftp.command.ConformanceRequirement` define no `_missing_` either, and `aenum` already raises the bare `ValueError` for them. `pcapkit/vendor/tcp/flags.py` turned out to carry the guard's value checker all along, as `FLAG = 'isinstance(value, int) and 4 <= value <= 15'`, while its template never interpolated it; those are the registry's *bit offsets* and its members are `1 << offset`, so emitting it unchanged would have rejected every composite, every member above bit 3, and `Flags(0)`. It now reads `0 <= value <= 0xFFFF`, the 16-bit field those bits live in. `TransportProtocol` reads its bound off `cls.__members__` instead of a literal, because `TransportProtocol.get` extends the registry at runtime at `max * 2` and a written-down bound would reject the member it had just grown. Fixed in the four bespoke templates under `pcapkit/vendor/` rather than in the generated tree -- `pcapkit/vendor/default.py` needed no change, since the 113 it emits were already correct -- and regenerated from the live IANA registries: 42 insertions and zero deletions across exactly 4 of the 134 files, so nothing else in the tree was stale, including the 30k-line `reg/apptype.py`. A new `tests/const/test_const_enum_builtin_parity.py` asserts the property the fix is actually about, that a constant registry and a stdlib `enum.IntEnum` raise the same exception type for the same invalid value, over all 123 registries rather than the six; it is keyed on `aenum.Enum` because an `IntFlag` is not a subclass of `IntEnum` -- its MRO runs through `Flag` instead -- which is how three of these six escaped the two existing sweeps. `tests/const/test_const_enum_get.py` drops `Flags` from `EXPECTED_TO_RESOLVE_ANYTHING` and its sweep grows 110 to 111, a registry that bounds its domain finally having a failure for `default` to fall back from (#647). - **Changed** -- `SeekableReader.truncate()` raises instead of returning a size, and the misspelled `writeable()` is now spelled `writable()`. Both are breaks for an external caller and neither breaks anything inside the package. `io.IOBase` documents one gate over two methods -- "If False, write() and truncate() will raise OSError" -- and this reader's `writable()` answers False, so a caller that checked it first, which is exactly what the contract invites, got a surprise either way round: `write` and `writelines` raised, and `truncate` returned its new size. What settled the question is that this is not a pure-Python nicety of `_pyio` that the accelerated path skips -- every ordinary read-only file object in CPython raises here, `open(path, 'rb').truncate()` giving `io.UnsupportedOperation: truncate` off the C `BufferedReader`, and `_pyio.BufferedReader` the same type from `_BufferedIOMixin.truncate`'s `_checkWritable()`. The refusal is `UnsupportedOperation('truncate')`, the same one `write` already raises, and no trade-off was needed between the house exception and the ABC's: pcapkit's `UnsupportedOperation` subclasses `io.UnsupportedOperation`, which subclasses `OSError`, so the in-library exception *is* the one the contract names. The resizing `truncate` used to perform is not deleted, only made private as `_truncate_buffer`. It never touched the underlying stream -- it resizes a private lookback window, which is the counter-argument the issue itself raised -- and it is the only route to the "position sits before the window" state that `seek` and the four buffered read paths must refuse, which #643 and #644 landed tests for; deleting it would have taken their mechanism with them. `writeable()`, separately, was never an override of anything: `'writable' in SeekableReader.__dict__` was False and `SeekableReader.writable is io.IOBase.writable` was True, so `io`, `shutil` and any third-party caller read the inherited value and never saw the one defined in this file. Both answered False, which is the coincidence that hid it -- there was no symptom to notice, and editing the misspelled method would silently have had no effect. The value reported is unchanged and was always honest, the reader genuinely being unable to write; only where the method was defined was wrong. Nothing in the package called either method, confirmed by grep, so the risk is entirely to external callers, which is what made this a question asked before it was acted on. The new test asserts the property rather than the behaviour -- that a `SeekableReader` raises the same exception type a read-only `io.BufferedReader` raises for the same call -- with that type captured by running the call rather than named in the test, so it tracks CPython across the 3.10--3.15 matrix instead of restating a belief about it. 31 to 34 tests and 35 to 41 subtests over `tests/corekit/test_io.py`, the module at 100% coverage before and after (#645). - **Fixed** -- `register_protocol` displaced one protocol class with another and said nothing about it. The registry behind `pcapkit.protocols.__proto__` is keyed on `cls.__name__.upper()`, and three dispatchable classes are all named `HTTP` -- the generic base `pcapkit.protocols.application.http.HTTP` and the two version implementations in `httpv1` and `httpv2` -- so all three compete for the single key `'HTTP'` and a bare assignment decided the winner. Measured before the fix: `__proto__['HTTP']` went from `pcapkit.protocols.application.http.HTTP` to the `httpv2` class on `register_protocol(httpv2.HTTP)`, with no exception, no log line and no `RegistryWarning`, and `ProtocolBase.expand_comp('HTTP')` then resolved to whichever class had registered last. The overwrite now raises a `RegistryWarning` naming both the displaced and the replacing class, the defining module being the only thing that tells the three apart. The sharpest path to it is not a call to the registrar at all: `Protocol.__init_subclass__` registers unconditionally, so merely *defining* a subclass of the public `Protocol` under a name a built-in already holds displaced that built-in, with nothing anywhere in the user's code to mark the moment the dispatch table changed meaning. A plain `import pcapkit` is unaffected and emits zero new warnings, measured not assumed -- the three built-in `HTTP` classes derive from `ProtocolBase` rather than from the public `Protocol`, so `__init_subclass__` never fires for them, and the import-time seeding in `pcapkit/protocols/__init__.py` keys off the distinct names `HTTP`, `HTTPv1` and `HTTPv2`. The issue's framing that this was the one registrar in `pcapkit/foundation/registry/` with no `RegistryWarning` is right on the count and wrong on the reason: no function in that package warns about anything, the siblings warning only by delegating to a classmethod that carries the guarded `if code in cls.__xxx__: warn(...)`. The accurate statement is the stronger one -- `register_protocol` is the only keyed registrar in that package, and the only `__name__.upper()`-keyed registry anywhere in `pcapkit/`, that mutates its target with no guard *and* no delegate that could supply one, its target being a module-level dict with no class behind it to hold the guard. The new guard deliberately reads "key present and the incumbent is a different class" where every sibling warns on mere presence: the sibling keys are codes the caller passes, whereas this key is derived from the class and this function is the funnel all nine wrapper registrars end in, so registering one class under two codes -- `register_tcp` then `register_udp`, which is supported and documented -- reaches it twice with nothing displaced, and warning there would put noise on a documented path whose wholesale filter is precisely what would then hide the real collision. The collision is reported, not resolved: re-keying is a registry-format change that none of the bare-name readers can absorb, every one of them degrading *silently* to a plain string or to `Raw` on a miss rather than raising, so it belongs to the registry redesign in #514, which this change is sequenced ahead of rather than part of. 9 to 13 tests and 80 to 87 subtests over `tests/foundation/registry/`, the touched module holding 88% coverage with its misses flat at 27 (#675). -- **Fixed** -- every PCAP-NG packet block now carries the captured octets it declares. All four Enhanced Packet Blocks of the committed `examples/captures/dhcp.pcapng` reported `len(packet) == 0` against a `captured_len` of 314, 342, 314 and 342, and the Simple Packet Block and the obsolete Packet Block did the same, measured on a capture synthesised to carry one of each since no committed fixture has either. Those three -- exactly `PCAPNG.PACKET_TYPES`, and exactly the three schemas declaring `__payload__ = 'packet_data'` -- are the whole of the blast radius; every other block type carries no captured octets, reported `b''` before and still does. The octets were read correctly and then thrown away: `PCAPNG.unpack` extracted them from the block schema, and `ProtocolBase.__init__` overwrote the result with `self.packet.payload`, which was empty. The inherited `packet` splits a protocol at `self.length` octets of header and takes everything after as the payload, a contract that holds for a protocol laid out as a header followed by its payload and for nothing else. `PCAPNG.length` returns the wire's *Block Total Length*, so the split consumed the entire per-block buffer as header -- measured at `frame.length == 348` and `len(frame.packet.header) == 348` on a 348-octet block -- and, separately, a PCAP-NG block carries a *trailer* after its payload, the option list and a repeat of the total length, so no value of `length` could have made that split right. `captured_len` comes straight off the block header and survived, which is why the two disagreed rather than both being empty. The fix is at the computation rather than at the injection: `PCAPNG.packet` is overridden to take the payload from the block schema's `__payload__` field and the header from the octets ahead of it, summed out of the schema buffers so that the three block types' three different payload offsets -- 28, 12 and 28 octets -- are not hard-coded. Teaching `ProtocolBase.__init__` to skip a protocol that had already set `packet` was the smaller change and was rejected: it heals `frame.info.packet` while leaving the public `frame.packet` still reporting the whole block as header and `b''` as payload, which is the same defect from the other side. `PCAPNG.unpack` now reads that property instead of extracting the payload a second time of its own, leaving one source of truth where there were two attempts at one; `ProtocolBase.packet` gains the contract in its docstring and not one executable line, its 503 statements unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on the success path, which is why it ships labelled breaking: `frame.info.packet` goes from `b''` to hundreds of octets, `frame.packet.header` from the whole block to the pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what made this a wire-format defect rather than an API wart. `PCAPIO` writes `value.packet` after each 16-octet record header, so dumping those four blocks produced a **104-octet** file -- 24 of global header plus four record headers and no payload at all -- in which every record header declared hundreds of octets and delivered none, desynchronising any reader that walks by `incl_len`, which is all of them: pcapkit refused its own output with `ValueError: read length must be non-negative or -1` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes rather than against a length. `tests/protocols/test_pcapng_regression.py` grows 4 tests to 11 and 3 subtests to 24, with the payload expectations derived twice over -- spelled-out head and tail literals, and a `struct`-only re-parse of the fixture that owes nothing to the code under test -- and each synthesised payload given a different length modulo 4 so that an offset off by a field, or a payload that picked up the block's 32-bit padding, cannot pass by coincidence. Two shapes no fixture exercised are covered there too, both found while writing the tests rather than after: an option area *after* the captured data, which every block of `dhcp.pcapng` lacks and which is exactly what an offset walked from the wrong end would swallow, and a big-endian section, whose fields differ even though the payload offsets do not. A snapped block is covered in both the shapes that express it -- an Enhanced Packet Block declaring `captured_len` below `original_len`, and a Simple Packet Block whose captured length is bounded by the interface's `snaplen` -- neither of which may come back padded out to the on-wire length. The offsets hold at 28, 12 and 28 across all of it, which is the property the walk has to have. `pcapkit/protocols/misc/pcapng.py` holds 99% across the change, its 18 new statements and 6 new branches all executed, its single miss the same pre-existing `_get_timezone` statement renumbered 1153 to 1248, and no new partial branch. The uncaught `ValueError` that an EOF-truncated PCAP-NG raises (#678) is untouched and was measured rather than assumed: all 101 truncation levels of `dhcp.pcapng` give byte-identical outcomes before and after, 96 of them that `ValueError`. `examples/captures/pcapng.txt`, a hand-regenerated legacy smoke reference, still shows the old frame-level `packet -> NIL` and wants a separate refresh (#646). +- **Fixed** -- every PCAP-NG packet block now carries the captured octets it declares. All four Enhanced Packet Blocks of the committed `examples/captures/dhcp.pcapng` reported `len(packet) == 0` against a `captured_len` of 314, 342, 314 and 342, and the Simple Packet Block and the obsolete Packet Block did the same, measured on a capture synthesised to carry one of each since no committed fixture has either. Those three -- exactly `PCAPNG.PACKET_TYPES`, and exactly the three schemas declaring `__payload__ = 'packet_data'` -- are the whole of the blast radius; every other block type carries no captured octets, reported `b''` before and still does. The octets were read correctly and then thrown away: `PCAPNG.unpack` extracted them from the block schema, and `ProtocolBase.__init__` overwrote the result with `self.packet.payload`, which was empty. The inherited `packet` splits a protocol at `self.length` octets of header and takes everything after as the payload, a contract that holds for a protocol laid out as a header followed by its payload and for nothing else. `PCAPNG.length` returns the wire's *Block Total Length*, so the split consumed the entire per-block buffer as header -- measured at `frame.length == 348` and `len(frame.packet.header) == 348` on a 348-octet block -- and, separately, a PCAP-NG block carries a *trailer* after its payload, the option list and a repeat of the total length, so no value of `length` could have made that split right. `captured_len` comes straight off the block header and survived, which is why the two disagreed rather than both being empty. The fix is at the computation rather than at the injection: `PCAPNG.packet` is overridden to take the payload from the block schema's `__payload__` field and the header from the octets ahead of it, summed out of the schema buffers so that the three block types' three different payload offsets -- 28, 12 and 28 octets -- are not hard-coded. Teaching `ProtocolBase.__init__` to skip a protocol that had already set `packet` was the smaller change and was rejected: it heals `frame.info.packet` while leaving the public `frame.packet` still reporting the whole block as header and `b''` as payload, which is the same defect from the other side. `PCAPNG.unpack` now reads that property instead of extracting the payload a second time of its own, leaving one source of truth where there were two attempts at one. The property is a plain `property` rather than the `cached_property` it overrides, which a cross-review on a second model is what settled: the inherited one caches because it *reads the stream*, where a second read would consume octets that are gone, while this one only walks buffers the schema layer has already filled and so has nothing to amortise -- and caching it would have reintroduced the same staleness by another route, a second `unpack` on one instance handing back the *first* call's octets with `get_payload` never reached, an invariant the code held before this change and has no reason to stop holding. Nothing calls `unpack` twice on one instance today, `__post_init__` being its only caller, so the unit test asserts the property directly: swap the block on a live instance, unpack again, and the payload must be the new block's, which fails `b'payload' != b'cached'` under a cache. `ProtocolBase.packet` gains the contract in its docstring and not one executable line, its 503 statements unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on the success path, which is why it ships labelled breaking: `frame.info.packet` goes from `b''` to hundreds of octets, `frame.packet.header` from the whole block to the pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what made this a wire-format defect rather than an API wart. `PCAPIO` writes `value.packet` after each 16-octet record header, so dumping those four blocks produced a **104-octet** file -- 24 of global header plus four record headers and no payload at all -- in which every record header declared hundreds of octets and delivered none, desynchronising any reader that walks by `incl_len`, which is all of them: pcapkit refused its own output with `ValueError: read length must be non-negative or -1` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes rather than against a length. `tests/protocols/test_pcapng_regression.py` grows 4 tests to 11 and 3 subtests to 24, with the payload expectations derived twice over -- spelled-out head and tail literals, and a `struct`-only re-parse of the fixture that owes nothing to the code under test -- and each synthesised payload given a different length modulo 4 so that an offset off by a field, or a payload that picked up the block's 32-bit padding, cannot pass by coincidence. Two shapes no fixture exercised are covered there too, both found while writing the tests rather than after: an option area *after* the captured data, which every block of `dhcp.pcapng` lacks and which is exactly what an offset walked from the wrong end would swallow, and a big-endian section, whose fields differ even though the payload offsets do not. A snapped block is covered in both the shapes that express it -- an Enhanced Packet Block declaring `captured_len` below `original_len`, and a Simple Packet Block whose captured length is bounded by the interface's `snaplen` -- neither of which may come back padded out to the on-wire length. The offsets hold at 28, 12 and 28 across all of it, which is the property the walk has to have. With both trees running the same -- current -- tests, which is the only comparison that isolates the code from the suite, `pcapkit/protocols/misc/pcapng.py` holds 99.91% across the change, its 18 new statements and 6 new branches all executed, its miss count flat at 1 and its partial-branch count flat at 1, that single miss being the same pre-existing `_get_timezone` statement renumbered 1153 to 1265; `pcapkit/protocols/protocol.py` is identical in every column, 503 statements and 228 misses either side, which is the check that its change really is docstring-only. The uncaught `ValueError` that an EOF-truncated PCAP-NG raises (#678) is untouched and was measured rather than assumed, over a truncation set stated rather than described because "101 levels" admits several readings that do not agree on the counts: the whole-percent prefixes `max(1, 1508 * i // 100) for i in range(101)`, which for this file are 101 distinct lengths from 1 to 1508. Both trees give 96 of that `ValueError`, one `FormatError`, one `ProtocolError` and three parses, and -- compared level by level rather than only in aggregate, since a matching total can hide two levels that swapped -- the SHA-256 of the sorted length-to-outcome mapping is `ca44d3ee658087cf2a667c454f839dd931c1c04790b2fe32cee8e1a2e861b182` on each. `examples/captures/pcapng.txt`, a hand-regenerated legacy smoke reference, still shows the old frame-level `packet -> NIL` and wants a separate refresh (#646). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 0efd79440..bf699c4de 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1936,17 +1936,28 @@ pull requests between #326 and #509. the whole block as header and ``b''`` as payload, which is the same defect from the other side. ``PCAPNG.unpack`` now reads that property instead of extracting the payload a second time of its own, leaving one source of truth where there were two attempts at - one; ``ProtocolBase.packet`` gains the contract in its docstring and not one executable - line, its 503 statements unchanged. This moves the parse output of every PCAP-NG - capture, for two properties, on the success path, which is why it ships labelled - breaking: ``frame.info.packet`` goes from ``b''`` to hundreds of octets, - ``frame.packet.header`` from the whole block to the pre-payload prefix, and anything - diffing a dumped file sees it change. That dump is what made this a wire-format defect - rather than an API wart. ``PCAPIO`` writes ``value.packet`` after each 16-octet record - header, so dumping those four blocks produced a **104-octet** file -- 24 of global - header plus four record headers and no payload at all -- in which every record header - declared hundreds of octets and delivered none, desynchronising any reader that walks by - ``incl_len``, which is all of them: pcapkit refused its own output with + one. The property is a plain ``property`` rather than the ``cached_property`` it + overrides, which a cross-review on a second model is what settled: the inherited one + caches because it *reads the stream*, where a second read would consume octets that are + gone, while this one only walks buffers the schema layer has already filled and so has + nothing to amortise -- and caching it would have reintroduced the same staleness by + another route, a second ``unpack`` on one instance handing back the *first* call's + octets with ``get_payload`` never reached, an invariant the code held before this change + and has no reason to stop holding. Nothing calls ``unpack`` twice on one instance today, + ``__post_init__`` being its only caller, so the unit test asserts the property directly: + swap the block on a live instance, unpack again, and the payload must be the new + block's, which fails ``b'payload' != b'cached'`` under a cache. ``ProtocolBase.packet`` + gains the contract in its docstring and not one executable line, its 503 statements + unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on + the success path, which is why it ships labelled breaking: ``frame.info.packet`` goes + from ``b''`` to hundreds of octets, ``frame.packet.header`` from the whole block to the + pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what + made this a wire-format defect rather than an API wart. ``PCAPIO`` writes + ``value.packet`` after each 16-octet record header, so dumping those four blocks + produced a **104-octet** file -- 24 of global header plus four record headers and no + payload at all -- in which every record header declared hundreds of octets and delivered + none, desynchronising any reader that walks by ``incl_len``, which is all of them: + pcapkit refused its own output with ``ValueError: read length must be non-negative or -1`` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes @@ -1964,15 +1975,25 @@ pull requests between #326 and #509. ``captured_len`` below ``original_len``, and a Simple Packet Block whose captured length is bounded by the interface's ``snaplen`` -- neither of which may come back padded out to the on-wire length. The offsets hold at 28, 12 and 28 across all of it, which is the - property the walk has to have. ``pcapkit/protocols/misc/pcapng.py`` holds 99% across the - change, its 18 new statements and 6 new branches all executed, its single miss the same - pre-existing ``_get_timezone`` statement renumbered 1153 to 1248, and no new partial - branch. The uncaught ``ValueError`` that an EOF-truncated PCAP-NG raises (#678) is - untouched and was measured rather than assumed: all 101 truncation levels of - ``dhcp.pcapng`` give byte-identical outcomes before and after, 96 of them that - ``ValueError``. ``examples/captures/pcapng.txt``, a hand-regenerated legacy smoke - reference, still shows the old frame-level ``packet -> NIL`` and wants a separate - refresh (#646). + property the walk has to have. With both trees running the same -- current -- tests, + which is the only comparison that isolates the code from the suite, + ``pcapkit/protocols/misc/pcapng.py`` holds 99.91% across the change, its 18 new + statements and 6 new branches all executed, its miss count flat at 1 and its + partial-branch count flat at 1, that single miss being the same pre-existing + ``_get_timezone`` statement renumbered 1153 to 1265; ``pcapkit/protocols/protocol.py`` + is identical in every column, 503 statements and 228 misses either side, which is the + check that its change really is docstring-only. The uncaught ``ValueError`` that an + EOF-truncated PCAP-NG raises (#678) is untouched and was measured rather than assumed, + over a truncation set stated rather than described because "101 levels" admits several + readings that do not agree on the counts: the whole-percent prefixes + ``max(1, 1508 * i // 100) for i in range(101)``, which for this file are 101 distinct + lengths from 1 to 1508. Both trees give 96 of that ``ValueError``, one ``FormatError``, + one ``ProtocolError`` and three parses, and -- compared level by level rather than only + in aggregate, since a matching total can hide two levels that swapped -- the SHA-256 of + the sorted length-to-outcome mapping is + ``ca44d3ee658087cf2a667c454f839dd931c1c04790b2fe32cee8e1a2e861b182`` on each. + ``examples/captures/pcapng.txt``, a hand-regenerated legacy smoke reference, still shows + the old frame-level ``packet -> NIL`` and wants a separate refresh (#646). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From fe7d2db0cb8fd77f704919c4506f1d70523bf8aa Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 23:06:11 -0400 Subject: [PATCH 24/46] docs(changelog): the 1.5.0 entry for #668, whose code ships in #691 Records the three mis-parenthesised HTTP/2 payload length callbacks: the `else` arm returned `0` instead of subtracting `0`, so an unpadded DATA, HEADERS or PUSH_PROMISE frame parsed with its whole payload discarded. Carries the measured octets rather than a description, the algebraic reason padded frames could not have moved, the derivation that the padded arm has no off-by-one, the three plain-form sibling sites that localise the defect to the grouping, the four wrong fixes the new tests reject, and the `EXPECTED_FAILURES` and coverage numbers either side. Also notes the stale `:572` reference in the `httpv2-frame/PRIORITY` entry, whose guard is now at `:562`. `CHANGELOG.md` regenerated with `util/changelog_md.py`; `--check` exits 0. --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 71 +++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 259a60ecd..6bfe093a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -105,6 +105,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Changed** -- `SeekableReader.truncate()` raises instead of returning a size, and the misspelled `writeable()` is now spelled `writable()`. Both are breaks for an external caller and neither breaks anything inside the package. `io.IOBase` documents one gate over two methods -- "If False, write() and truncate() will raise OSError" -- and this reader's `writable()` answers False, so a caller that checked it first, which is exactly what the contract invites, got a surprise either way round: `write` and `writelines` raised, and `truncate` returned its new size. What settled the question is that this is not a pure-Python nicety of `_pyio` that the accelerated path skips -- every ordinary read-only file object in CPython raises here, `open(path, 'rb').truncate()` giving `io.UnsupportedOperation: truncate` off the C `BufferedReader`, and `_pyio.BufferedReader` the same type from `_BufferedIOMixin.truncate`'s `_checkWritable()`. The refusal is `UnsupportedOperation('truncate')`, the same one `write` already raises, and no trade-off was needed between the house exception and the ABC's: pcapkit's `UnsupportedOperation` subclasses `io.UnsupportedOperation`, which subclasses `OSError`, so the in-library exception *is* the one the contract names. The resizing `truncate` used to perform is not deleted, only made private as `_truncate_buffer`. It never touched the underlying stream -- it resizes a private lookback window, which is the counter-argument the issue itself raised -- and it is the only route to the "position sits before the window" state that `seek` and the four buffered read paths must refuse, which #643 and #644 landed tests for; deleting it would have taken their mechanism with them. `writeable()`, separately, was never an override of anything: `'writable' in SeekableReader.__dict__` was False and `SeekableReader.writable is io.IOBase.writable` was True, so `io`, `shutil` and any third-party caller read the inherited value and never saw the one defined in this file. Both answered False, which is the coincidence that hid it -- there was no symptom to notice, and editing the misspelled method would silently have had no effect. The value reported is unchanged and was always honest, the reader genuinely being unable to write; only where the method was defined was wrong. Nothing in the package called either method, confirmed by grep, so the risk is entirely to external callers, which is what made this a question asked before it was acted on. The new test asserts the property rather than the behaviour -- that a `SeekableReader` raises the same exception type a read-only `io.BufferedReader` raises for the same call -- with that type captured by running the call rather than named in the test, so it tracks CPython across the 3.10--3.15 matrix instead of restating a belief about it. 31 to 34 tests and 35 to 41 subtests over `tests/corekit/test_io.py`, the module at 100% coverage before and after (#645). - **Fixed** -- `register_protocol` displaced one protocol class with another and said nothing about it. The registry behind `pcapkit.protocols.__proto__` is keyed on `cls.__name__.upper()`, and three dispatchable classes are all named `HTTP` -- the generic base `pcapkit.protocols.application.http.HTTP` and the two version implementations in `httpv1` and `httpv2` -- so all three compete for the single key `'HTTP'` and a bare assignment decided the winner. Measured before the fix: `__proto__['HTTP']` went from `pcapkit.protocols.application.http.HTTP` to the `httpv2` class on `register_protocol(httpv2.HTTP)`, with no exception, no log line and no `RegistryWarning`, and `ProtocolBase.expand_comp('HTTP')` then resolved to whichever class had registered last. The overwrite now raises a `RegistryWarning` naming both the displaced and the replacing class, the defining module being the only thing that tells the three apart. The sharpest path to it is not a call to the registrar at all: `Protocol.__init_subclass__` registers unconditionally, so merely *defining* a subclass of the public `Protocol` under a name a built-in already holds displaced that built-in, with nothing anywhere in the user's code to mark the moment the dispatch table changed meaning. A plain `import pcapkit` is unaffected and emits zero new warnings, measured not assumed -- the three built-in `HTTP` classes derive from `ProtocolBase` rather than from the public `Protocol`, so `__init_subclass__` never fires for them, and the import-time seeding in `pcapkit/protocols/__init__.py` keys off the distinct names `HTTP`, `HTTPv1` and `HTTPv2`. The issue's framing that this was the one registrar in `pcapkit/foundation/registry/` with no `RegistryWarning` is right on the count and wrong on the reason: no function in that package warns about anything, the siblings warning only by delegating to a classmethod that carries the guarded `if code in cls.__xxx__: warn(...)`. The accurate statement is the stronger one -- `register_protocol` is the only keyed registrar in that package, and the only `__name__.upper()`-keyed registry anywhere in `pcapkit/`, that mutates its target with no guard *and* no delegate that could supply one, its target being a module-level dict with no class behind it to hold the guard. The new guard deliberately reads "key present and the incumbent is a different class" where every sibling warns on mere presence: the sibling keys are codes the caller passes, whereas this key is derived from the class and this function is the funnel all nine wrapper registrars end in, so registering one class under two codes -- `register_tcp` then `register_udp`, which is supported and documented -- reaches it twice with nothing displaced, and warning there would put noise on a documented path whose wholesale filter is precisely what would then hide the real collision. The collision is reported, not resolved: re-keying is a registry-format change that none of the bare-name readers can absorb, every one of them degrading *silently* to a plain string or to `Raw` on a miss rather than raising, so it belongs to the registry redesign in #514, which this change is sequenced ahead of rather than part of. 9 to 13 tests and 80 to 87 subtests over `tests/foundation/registry/`, the touched module holding 88% coverage with its misses flat at 27 (#675). - **Fixed** -- every PCAP-NG packet block now carries the captured octets it declares. All four Enhanced Packet Blocks of the committed `examples/captures/dhcp.pcapng` reported `len(packet) == 0` against a `captured_len` of 314, 342, 314 and 342, and the Simple Packet Block and the obsolete Packet Block did the same, measured on a capture synthesised to carry one of each since no committed fixture has either. Those three -- exactly `PCAPNG.PACKET_TYPES`, and exactly the three schemas declaring `__payload__ = 'packet_data'` -- are the whole of the blast radius; every other block type carries no captured octets, reported `b''` before and still does. The octets were read correctly and then thrown away: `PCAPNG.unpack` extracted them from the block schema, and `ProtocolBase.__init__` overwrote the result with `self.packet.payload`, which was empty. The inherited `packet` splits a protocol at `self.length` octets of header and takes everything after as the payload, a contract that holds for a protocol laid out as a header followed by its payload and for nothing else. `PCAPNG.length` returns the wire's *Block Total Length*, so the split consumed the entire per-block buffer as header -- measured at `frame.length == 348` and `len(frame.packet.header) == 348` on a 348-octet block -- and, separately, a PCAP-NG block carries a *trailer* after its payload, the option list and a repeat of the total length, so no value of `length` could have made that split right. `captured_len` comes straight off the block header and survived, which is why the two disagreed rather than both being empty. The fix is at the computation rather than at the injection: `PCAPNG.packet` is overridden to take the payload from the block schema's `__payload__` field and the header from the octets ahead of it, summed out of the schema buffers so that the three block types' three different payload offsets -- 28, 12 and 28 octets -- are not hard-coded. Teaching `ProtocolBase.__init__` to skip a protocol that had already set `packet` was the smaller change and was rejected: it heals `frame.info.packet` while leaving the public `frame.packet` still reporting the whole block as header and `b''` as payload, which is the same defect from the other side. `PCAPNG.unpack` now reads that property instead of extracting the payload a second time of its own, leaving one source of truth where there were two attempts at one. The property is a plain `property` rather than the `cached_property` it overrides, which a cross-review on a second model is what settled: the inherited one caches because it *reads the stream*, where a second read would consume octets that are gone, while this one only walks buffers the schema layer has already filled and so has nothing to amortise -- and caching it would have reintroduced the same staleness by another route, a second `unpack` on one instance handing back the *first* call's octets with `get_payload` never reached, an invariant the code held before this change and has no reason to stop holding. Nothing calls `unpack` twice on one instance today, `__post_init__` being its only caller, so the unit test asserts the property directly: swap the block on a live instance, unpack again, and the payload must be the new block's, which fails `b'payload' != b'cached'` under a cache. `ProtocolBase.packet` gains the contract in its docstring and not one executable line, its 503 statements unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on the success path, which is why it ships labelled breaking: `frame.info.packet` goes from `b''` to hundreds of octets, `frame.packet.header` from the whole block to the pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what made this a wire-format defect rather than an API wart. `PCAPIO` writes `value.packet` after each 16-octet record header, so dumping those four blocks produced a **104-octet** file -- 24 of global header plus four record headers and no payload at all -- in which every record header declared hundreds of octets and delivered none, desynchronising any reader that walks by `incl_len`, which is all of them: pcapkit refused its own output with `ValueError: read length must be non-negative or -1` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes rather than against a length. `tests/protocols/test_pcapng_regression.py` grows 4 tests to 11 and 3 subtests to 24, with the payload expectations derived twice over -- spelled-out head and tail literals, and a `struct`-only re-parse of the fixture that owes nothing to the code under test -- and each synthesised payload given a different length modulo 4 so that an offset off by a field, or a payload that picked up the block's 32-bit padding, cannot pass by coincidence. Two shapes no fixture exercised are covered there too, both found while writing the tests rather than after: an option area *after* the captured data, which every block of `dhcp.pcapng` lacks and which is exactly what an offset walked from the wrong end would swallow, and a big-endian section, whose fields differ even though the payload offsets do not. A snapped block is covered in both the shapes that express it -- an Enhanced Packet Block declaring `captured_len` below `original_len`, and a Simple Packet Block whose captured length is bounded by the interface's `snaplen` -- neither of which may come back padded out to the on-wire length. The offsets hold at 28, 12 and 28 across all of it, which is the property the walk has to have. With both trees running the same -- current -- tests, which is the only comparison that isolates the code from the suite, `pcapkit/protocols/misc/pcapng.py` holds 99.91% across the change, its 18 new statements and 6 new branches all executed, its miss count flat at 1 and its partial-branch count flat at 1, that single miss being the same pre-existing `_get_timezone` statement renumbered 1153 to 1265; `pcapkit/protocols/protocol.py` is identical in every column, 503 statements and 228 misses either side, which is the check that its change really is docstring-only. The uncaught `ValueError` that an EOF-truncated PCAP-NG raises (#678) is untouched and was measured rather than assumed, over a truncation set stated rather than described because "101 levels" admits several readings that do not agree on the counts: the whole-percent prefixes `max(1, 1508 * i // 100) for i in range(101)`, which for this file are 101 distinct lengths from 1 to 1508. Both trees give 96 of that `ValueError`, one `FormatError`, one `ProtocolError` and three parses, and -- compared level by level rather than only in aggregate, since a matching total can hide two levels that swapped -- the SHA-256 of the sorted length-to-outcome mapping is `ca44d3ee658087cf2a667c454f839dd931c1c04790b2fe32cee8e1a2e861b182` on each. `examples/captures/pcapng.txt`, a hand-regenerated legacy smoke reference, still shows the old frame-level `packet -> NIL` and wants a separate refresh (#646). +- **Fixed** -- an unpadded HTTP/2 `DATA`, `HEADERS` or `PUSH_PROMISE` frame parsed with its whole payload silently discarded. Three payload length callbacks in `pcapkit/protocols/schema/application/httpv2.py` put the conditional expression in the wrong place: a conditional binds looser than `-`, so `pkt['__length__'] - pkt['pad_len'] if pkt['flags']['bit_3'] else 0` groups as `(pkt['__length__'] - pkt['pad_len']) if ... else 0` and the `else` arm returned `0` -- "read no octets at all" to `BytesField` -- where it was meant to subtract `0`. Since `__length__` is the *remaining* declared length at the field, the unpadded arm wants `__length__` itself, which is what subtracting a zero padding length gives. The grouping was read off the AST rather than by eye, the top-level node being the `IfExp` whose `orelse` is a bare `Constant(0)`, which is also what showed that the outer parentheses on the `HeadersFrame` and `PushPromiseFrame` forms were line-continuation only and changed nothing. Padding is rare in HTTP/2, so the broken arm was the common one rather than the edge case, and nothing raised, warned or logged: `info.data` was simply `b''`. Measured on wire octets through the public `HTTP(io.BytesIO(raw), len(raw))` path, an unpadded `DATA` frame declaring `b'{"ok":true}\n'` parsed as `b''` and now parses as those twelve octets; unpadded `HEADERS` and `PUSH_PROMISE` lost and now keep the RFC 7541 appendix C.4.1 header block fragment `b'\x82\x86\x84A\x0fwww.example.com'`, which is precisely what HPACK decoding would have been handed. Padded frames are untouched, for a reason stronger than the measurement: `(A - B) if T else 0` and `A - (B if T else 0)` are both `A - B` for truthy `T`, so only the `else` arm could ever have moved -- both arms are asserted anyway, so that a repair to the unpadded one cannot break the other. The padded arm also has no off-by-one, which #668 asked be checked rather than assumed: `Schema.unpack` decrements `packet['__length__']` by each field's width as it goes, so `pad_len`'s own octet is already out of it by the time the payload field runs, and a padded frame declaring `1 + len(data) + pad_len` arrives at the payload with `__length__ == len(data) + pad_len`. The three sibling sites that were never wrong are what localise the defect to the grouping rather than to the field machinery: `ContinuationFrame.fragment`, `UnassignedFrame.data` and `GoawayFrame.debug` use the plain `length=lambda pkt: pkt['__length__']` with no conditional, share the same `BytesField`, the same `__length__` bookkeeping and the same frame dispatch, and carried their payload correctly throughout -- they are pinned as controls rather than left as an argument. Why nothing caught it: `test_option_roundtrip_unit` does drive every frame type, but through `make` -> parse -> `make`, and `make` writes the length field from `_make_http_length`, so both sides agreed on an empty payload and the octets matched; its generator passes no payload argument for these three frames at all, measured as `kwargs={}`, so what it round-tripped was `b''` and there was nothing to lose, while `test_httpv2_frame_readers_cover_successful_frames` drives the readers from hand-built schema stubs, so the callback never ran there either. The new `tests/protocols/application/test_httpv2_payload_length_unit.py` reads wire octets, which is the gap: 15 tests and 9 subtests over the padded and unpadded shapes of all three frames plus both `PRIORITY` combinations, asserting the payload *octets* and never its length, because a length assertion survives a read of the right width from the wrong offset. Its padded cases pad with a non-zero `b'\xde\xad\xbe\xef'` rather than the zeros [RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1) tells a sender to use -- pcapkit does not police padding content -- so that failing to subtract the padding yields a different byte string instead of a coincidentally equal count. Four wrong fixes were written over the schema to check the tests discriminate, and each is rejected: dropping the conditional outright (caught by `packet length < 0: -4`), subtracting `pad_len + 1` (caught as `b'{"ok":true}'` against `b'{"ok":true}\n'`), subtracting the padding on the arm where it is absent, and fixing `DataFrame` while forgetting the other two. Four tests and three subtests fail before the change and all pass after; 137 tests and 547 subtests pass across `tests/protocols/application/`, the round-trip module and the four other HTTP/2-touching modules. `EXPECTED_FAILURES` is unmoved, its one HTTP/2 entry `httpv2-frame/PRIORITY` still failing with the same `CONSTRUCT` status and the same detail -- `PriorityFrame` has neither a payload field nor a `PADDED` flag, so this change cannot reach it -- and none of the 44 entries was deleted; that entry's own cited `pcapkit/protocols/application/httpv2.py:572` is stale, the `header.length != 9` guard it describes now being at `:562`. The schema module was already at 100% coverage and stays there, 101 statements and 4 branches either side with nothing missing or partial, because the changed lines did execute before -- just with an empty payload -- so the number that moved is the suite: 73 to 88 tests and 399 to 408 subtests over the same targets. The construct side needed no change, `_make_http_length` having always computed the DATA payload as `len(frame.data) + (pad_len + 1 if pad_len else 0)`, i.e. assuming `data` holds the whole payload whether or not the frame is padded -- it was the read path that only agreed with that when the frame *was* padded. Ships labelled breaking: restoring a dropped payload moves the parse output of essentially every HTTP/2 capture on the success path, so anything holding a golden file or a regression baseline that recorded the empty value sees it change (#668). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index bf699c4de..164936d6c 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1994,6 +1994,77 @@ pull requests between #326 and #509. ``ca44d3ee658087cf2a667c454f839dd931c1c04790b2fe32cee8e1a2e861b182`` on each. ``examples/captures/pcapng.txt``, a hand-regenerated legacy smoke reference, still shows the old frame-level ``packet -> NIL`` and wants a separate refresh (#646). +* **Fixed** -- an unpadded HTTP/2 ``DATA``, ``HEADERS`` or ``PUSH_PROMISE`` frame parsed + with its whole payload silently discarded. Three payload length callbacks in + ``pcapkit/protocols/schema/application/httpv2.py`` put the conditional expression in the + wrong place: a conditional binds looser than ``-``, so + ``pkt['__length__'] - pkt['pad_len'] if pkt['flags']['bit_3'] else 0`` groups as + ``(pkt['__length__'] - pkt['pad_len']) if ... else 0`` and the ``else`` arm returned + ``0`` -- "read no octets at all" to ``BytesField`` -- where it was meant to subtract + ``0``. Since ``__length__`` is the *remaining* declared length at the field, the + unpadded arm wants ``__length__`` itself, which is what subtracting a zero padding + length gives. The grouping was read off the AST rather than by eye, the top-level node + being the ``IfExp`` whose ``orelse`` is a bare ``Constant(0)``, which is also what + showed that the outer parentheses on the ``HeadersFrame`` and ``PushPromiseFrame`` + forms were line-continuation only and changed nothing. Padding is rare in HTTP/2, so + the broken arm was the common one rather than the edge case, and nothing raised, warned + or logged: ``info.data`` was simply ``b''``. Measured on wire octets through the public + ``HTTP(io.BytesIO(raw), len(raw))`` path, an unpadded ``DATA`` frame declaring + ``b'{"ok":true}\n'`` parsed as ``b''`` and now parses as those twelve octets; unpadded + ``HEADERS`` and ``PUSH_PROMISE`` lost and now keep the RFC 7541 appendix C.4.1 header + block fragment ``b'\x82\x86\x84A\x0fwww.example.com'``, which is precisely what HPACK + decoding would have been handed. Padded frames are untouched, for a reason stronger + than the measurement: ``(A - B) if T else 0`` and ``A - (B if T else 0)`` are both + ``A - B`` for truthy ``T``, so only the ``else`` arm could ever have moved -- both arms + are asserted anyway, so that a repair to the unpadded one cannot break the other. The + padded arm also has no off-by-one, which #668 asked be checked rather than assumed: + ``Schema.unpack`` decrements ``packet['__length__']`` by each field's width as it goes, + so ``pad_len``'s own octet is already out of it by the time the payload field runs, and + a padded frame declaring ``1 + len(data) + pad_len`` arrives at the payload with + ``__length__ == len(data) + pad_len``. The three sibling sites that were never wrong + are what localise the defect to the grouping rather than to the field machinery: + ``ContinuationFrame.fragment``, ``UnassignedFrame.data`` and ``GoawayFrame.debug`` use + the plain ``length=lambda pkt: pkt['__length__']`` with no conditional, share the same + ``BytesField``, the same ``__length__`` bookkeeping and the same frame dispatch, and + carried their payload correctly throughout -- they are pinned as controls rather than + left as an argument. Why nothing caught it: ``test_option_roundtrip_unit`` does drive + every frame type, but through ``make`` -> parse -> ``make``, and ``make`` writes the + length field from ``_make_http_length``, so both sides agreed on an empty payload and + the octets matched; its generator passes no payload argument for these three frames at + all, measured as ``kwargs={}``, so what it round-tripped was ``b''`` and there was + nothing to lose, while ``test_httpv2_frame_readers_cover_successful_frames`` drives the + readers from hand-built schema stubs, so the callback never ran there either. The new + ``tests/protocols/application/test_httpv2_payload_length_unit.py`` reads wire octets, + which is the gap: 15 tests and 9 subtests over the padded and unpadded shapes of all + three frames plus both ``PRIORITY`` combinations, asserting the payload *octets* and + never its length, because a length assertion survives a read of the right width from + the wrong offset. Its padded cases pad with a non-zero ``b'\xde\xad\xbe\xef'`` rather + than the zeros :rfc:`9113#section-6.1` tells a sender to use -- pcapkit does not police + padding content -- so that failing to subtract the padding yields a different byte + string instead of a coincidentally equal count. Four wrong fixes were written over the + schema to check the tests discriminate, and each is rejected: dropping the conditional + outright (caught by ``packet length < 0: -4``), subtracting ``pad_len + 1`` (caught as + ``b'{"ok":true}'`` against ``b'{"ok":true}\n'``), subtracting the padding on the arm + where it is absent, and fixing ``DataFrame`` while forgetting the other two. Four tests + and three subtests fail before the change and all pass after; 137 tests and 547 + subtests pass across ``tests/protocols/application/``, the round-trip module and the + four other HTTP/2-touching modules. ``EXPECTED_FAILURES`` is unmoved, its one HTTP/2 + entry ``httpv2-frame/PRIORITY`` still failing with the same ``CONSTRUCT`` status and + the same detail -- ``PriorityFrame`` has neither a payload field nor a ``PADDED`` flag, + so this change cannot reach it -- and none of the 44 entries was deleted; that entry's + own cited ``pcapkit/protocols/application/httpv2.py:572`` is stale, the + ``header.length != 9`` guard it describes now being at ``:562``. The schema module was + already at 100% coverage and stays there, 101 statements and 4 branches either side + with nothing missing or partial, because the changed lines did execute before -- just + with an empty payload -- so the number that moved is the suite: 73 to 88 tests and 399 + to 408 subtests over the same targets. The construct side needed no change, + ``_make_http_length`` having always computed the DATA payload as + ``len(frame.data) + (pad_len + 1 if pad_len else 0)``, i.e. assuming ``data`` holds the + whole payload whether or not the frame is padded -- it was the read path that only + agreed with that when the frame *was* padded. Ships labelled breaking: restoring a + dropped payload moves the parse output of essentially every HTTP/2 capture on the + success path, so anything holding a golden file or a regression baseline that recorded + the empty value sees it change (#668). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From c4412ff5368991252a85d3edfd906c5c8767c370 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 23:32:10 -0400 Subject: [PATCH 25/46] docs(changelog): the 1.5.0 entry for the registry overwrite guards (#692) --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 37 +++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6bfe093a8..547b969c8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -106,6 +106,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- `register_protocol` displaced one protocol class with another and said nothing about it. The registry behind `pcapkit.protocols.__proto__` is keyed on `cls.__name__.upper()`, and three dispatchable classes are all named `HTTP` -- the generic base `pcapkit.protocols.application.http.HTTP` and the two version implementations in `httpv1` and `httpv2` -- so all three compete for the single key `'HTTP'` and a bare assignment decided the winner. Measured before the fix: `__proto__['HTTP']` went from `pcapkit.protocols.application.http.HTTP` to the `httpv2` class on `register_protocol(httpv2.HTTP)`, with no exception, no log line and no `RegistryWarning`, and `ProtocolBase.expand_comp('HTTP')` then resolved to whichever class had registered last. The overwrite now raises a `RegistryWarning` naming both the displaced and the replacing class, the defining module being the only thing that tells the three apart. The sharpest path to it is not a call to the registrar at all: `Protocol.__init_subclass__` registers unconditionally, so merely *defining* a subclass of the public `Protocol` under a name a built-in already holds displaced that built-in, with nothing anywhere in the user's code to mark the moment the dispatch table changed meaning. A plain `import pcapkit` is unaffected and emits zero new warnings, measured not assumed -- the three built-in `HTTP` classes derive from `ProtocolBase` rather than from the public `Protocol`, so `__init_subclass__` never fires for them, and the import-time seeding in `pcapkit/protocols/__init__.py` keys off the distinct names `HTTP`, `HTTPv1` and `HTTPv2`. The issue's framing that this was the one registrar in `pcapkit/foundation/registry/` with no `RegistryWarning` is right on the count and wrong on the reason: no function in that package warns about anything, the siblings warning only by delegating to a classmethod that carries the guarded `if code in cls.__xxx__: warn(...)`. The accurate statement is the stronger one -- `register_protocol` is the only keyed registrar in that package, and the only `__name__.upper()`-keyed registry anywhere in `pcapkit/`, that mutates its target with no guard *and* no delegate that could supply one, its target being a module-level dict with no class behind it to hold the guard. The new guard deliberately reads "key present and the incumbent is a different class" where every sibling warns on mere presence: the sibling keys are codes the caller passes, whereas this key is derived from the class and this function is the funnel all nine wrapper registrars end in, so registering one class under two codes -- `register_tcp` then `register_udp`, which is supported and documented -- reaches it twice with nothing displaced, and warning there would put noise on a documented path whose wholesale filter is precisely what would then hide the real collision. The collision is reported, not resolved: re-keying is a registry-format change that none of the bare-name readers can absorb, every one of them degrading *silently* to a plain string or to `Raw` on a miss rather than raising, so it belongs to the registry redesign in #514, which this change is sequenced ahead of rather than part of. 9 to 13 tests and 80 to 87 subtests over `tests/foundation/registry/`, the touched module holding 88% coverage with its misses flat at 27 (#675). - **Fixed** -- every PCAP-NG packet block now carries the captured octets it declares. All four Enhanced Packet Blocks of the committed `examples/captures/dhcp.pcapng` reported `len(packet) == 0` against a `captured_len` of 314, 342, 314 and 342, and the Simple Packet Block and the obsolete Packet Block did the same, measured on a capture synthesised to carry one of each since no committed fixture has either. Those three -- exactly `PCAPNG.PACKET_TYPES`, and exactly the three schemas declaring `__payload__ = 'packet_data'` -- are the whole of the blast radius; every other block type carries no captured octets, reported `b''` before and still does. The octets were read correctly and then thrown away: `PCAPNG.unpack` extracted them from the block schema, and `ProtocolBase.__init__` overwrote the result with `self.packet.payload`, which was empty. The inherited `packet` splits a protocol at `self.length` octets of header and takes everything after as the payload, a contract that holds for a protocol laid out as a header followed by its payload and for nothing else. `PCAPNG.length` returns the wire's *Block Total Length*, so the split consumed the entire per-block buffer as header -- measured at `frame.length == 348` and `len(frame.packet.header) == 348` on a 348-octet block -- and, separately, a PCAP-NG block carries a *trailer* after its payload, the option list and a repeat of the total length, so no value of `length` could have made that split right. `captured_len` comes straight off the block header and survived, which is why the two disagreed rather than both being empty. The fix is at the computation rather than at the injection: `PCAPNG.packet` is overridden to take the payload from the block schema's `__payload__` field and the header from the octets ahead of it, summed out of the schema buffers so that the three block types' three different payload offsets -- 28, 12 and 28 octets -- are not hard-coded. Teaching `ProtocolBase.__init__` to skip a protocol that had already set `packet` was the smaller change and was rejected: it heals `frame.info.packet` while leaving the public `frame.packet` still reporting the whole block as header and `b''` as payload, which is the same defect from the other side. `PCAPNG.unpack` now reads that property instead of extracting the payload a second time of its own, leaving one source of truth where there were two attempts at one. The property is a plain `property` rather than the `cached_property` it overrides, which a cross-review on a second model is what settled: the inherited one caches because it *reads the stream*, where a second read would consume octets that are gone, while this one only walks buffers the schema layer has already filled and so has nothing to amortise -- and caching it would have reintroduced the same staleness by another route, a second `unpack` on one instance handing back the *first* call's octets with `get_payload` never reached, an invariant the code held before this change and has no reason to stop holding. Nothing calls `unpack` twice on one instance today, `__post_init__` being its only caller, so the unit test asserts the property directly: swap the block on a live instance, unpack again, and the payload must be the new block's, which fails `b'payload' != b'cached'` under a cache. `ProtocolBase.packet` gains the contract in its docstring and not one executable line, its 503 statements unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on the success path, which is why it ships labelled breaking: `frame.info.packet` goes from `b''` to hundreds of octets, `frame.packet.header` from the whole block to the pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what made this a wire-format defect rather than an API wart. `PCAPIO` writes `value.packet` after each 16-octet record header, so dumping those four blocks produced a **104-octet** file -- 24 of global header plus four record headers and no payload at all -- in which every record header declared hundreds of octets and delivered none, desynchronising any reader that walks by `incl_len`, which is all of them: pcapkit refused its own output with `ValueError: read length must be non-negative or -1` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes rather than against a length. `tests/protocols/test_pcapng_regression.py` grows 4 tests to 11 and 3 subtests to 24, with the payload expectations derived twice over -- spelled-out head and tail literals, and a `struct`-only re-parse of the fixture that owes nothing to the code under test -- and each synthesised payload given a different length modulo 4 so that an offset off by a field, or a payload that picked up the block's 32-bit padding, cannot pass by coincidence. Two shapes no fixture exercised are covered there too, both found while writing the tests rather than after: an option area *after* the captured data, which every block of `dhcp.pcapng` lacks and which is exactly what an offset walked from the wrong end would swallow, and a big-endian section, whose fields differ even though the payload offsets do not. A snapped block is covered in both the shapes that express it -- an Enhanced Packet Block declaring `captured_len` below `original_len`, and a Simple Packet Block whose captured length is bounded by the interface's `snaplen` -- neither of which may come back padded out to the on-wire length. The offsets hold at 28, 12 and 28 across all of it, which is the property the walk has to have. With both trees running the same -- current -- tests, which is the only comparison that isolates the code from the suite, `pcapkit/protocols/misc/pcapng.py` holds 99.91% across the change, its 18 new statements and 6 new branches all executed, its miss count flat at 1 and its partial-branch count flat at 1, that single miss being the same pre-existing `_get_timezone` statement renumbered 1153 to 1265; `pcapkit/protocols/protocol.py` is identical in every column, 503 statements and 228 misses either side, which is the check that its change really is docstring-only. The uncaught `ValueError` that an EOF-truncated PCAP-NG raises (#678) is untouched and was measured rather than assumed, over a truncation set stated rather than described because "101 levels" admits several readings that do not agree on the counts: the whole-percent prefixes `max(1, 1508 * i // 100) for i in range(101)`, which for this file are 101 distinct lengths from 1 to 1508. Both trees give 96 of that `ValueError`, one `FormatError`, one `ProtocolError` and three parses, and -- compared level by level rather than only in aggregate, since a matching total can hide two levels that swapped -- the SHA-256 of the sorted length-to-outcome mapping is `ca44d3ee658087cf2a667c454f839dd931c1c04790b2fe32cee8e1a2e861b182` on each. `examples/captures/pcapng.txt`, a hand-regenerated legacy smoke reference, still shows the old frame-level `packet -> NIL` and wants a separate refresh (#646). - **Fixed** -- an unpadded HTTP/2 `DATA`, `HEADERS` or `PUSH_PROMISE` frame parsed with its whole payload silently discarded. Three payload length callbacks in `pcapkit/protocols/schema/application/httpv2.py` put the conditional expression in the wrong place: a conditional binds looser than `-`, so `pkt['__length__'] - pkt['pad_len'] if pkt['flags']['bit_3'] else 0` groups as `(pkt['__length__'] - pkt['pad_len']) if ... else 0` and the `else` arm returned `0` -- "read no octets at all" to `BytesField` -- where it was meant to subtract `0`. Since `__length__` is the *remaining* declared length at the field, the unpadded arm wants `__length__` itself, which is what subtracting a zero padding length gives. The grouping was read off the AST rather than by eye, the top-level node being the `IfExp` whose `orelse` is a bare `Constant(0)`, which is also what showed that the outer parentheses on the `HeadersFrame` and `PushPromiseFrame` forms were line-continuation only and changed nothing. Padding is rare in HTTP/2, so the broken arm was the common one rather than the edge case, and nothing raised, warned or logged: `info.data` was simply `b''`. Measured on wire octets through the public `HTTP(io.BytesIO(raw), len(raw))` path, an unpadded `DATA` frame declaring `b'{"ok":true}\n'` parsed as `b''` and now parses as those twelve octets; unpadded `HEADERS` and `PUSH_PROMISE` lost and now keep the RFC 7541 appendix C.4.1 header block fragment `b'\x82\x86\x84A\x0fwww.example.com'`, which is precisely what HPACK decoding would have been handed. Padded frames are untouched, for a reason stronger than the measurement: `(A - B) if T else 0` and `A - (B if T else 0)` are both `A - B` for truthy `T`, so only the `else` arm could ever have moved -- both arms are asserted anyway, so that a repair to the unpadded one cannot break the other. The padded arm also has no off-by-one, which #668 asked be checked rather than assumed: `Schema.unpack` decrements `packet['__length__']` by each field's width as it goes, so `pad_len`'s own octet is already out of it by the time the payload field runs, and a padded frame declaring `1 + len(data) + pad_len` arrives at the payload with `__length__ == len(data) + pad_len`. The three sibling sites that were never wrong are what localise the defect to the grouping rather than to the field machinery: `ContinuationFrame.fragment`, `UnassignedFrame.data` and `GoawayFrame.debug` use the plain `length=lambda pkt: pkt['__length__']` with no conditional, share the same `BytesField`, the same `__length__` bookkeeping and the same frame dispatch, and carried their payload correctly throughout -- they are pinned as controls rather than left as an argument. Why nothing caught it: `test_option_roundtrip_unit` does drive every frame type, but through `make` -> parse -> `make`, and `make` writes the length field from `_make_http_length`, so both sides agreed on an empty payload and the octets matched; its generator passes no payload argument for these three frames at all, measured as `kwargs={}`, so what it round-tripped was `b''` and there was nothing to lose, while `test_httpv2_frame_readers_cover_successful_frames` drives the readers from hand-built schema stubs, so the callback never ran there either. The new `tests/protocols/application/test_httpv2_payload_length_unit.py` reads wire octets, which is the gap: 15 tests and 9 subtests over the padded and unpadded shapes of all three frames plus both `PRIORITY` combinations, asserting the payload *octets* and never its length, because a length assertion survives a read of the right width from the wrong offset. Its padded cases pad with a non-zero `b'\xde\xad\xbe\xef'` rather than the zeros [RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1) tells a sender to use -- pcapkit does not police padding content -- so that failing to subtract the padding yields a different byte string instead of a coincidentally equal count. Four wrong fixes were written over the schema to check the tests discriminate, and each is rejected: dropping the conditional outright (caught by `packet length < 0: -4`), subtracting `pad_len + 1` (caught as `b'{"ok":true}'` against `b'{"ok":true}\n'`), subtracting the padding on the arm where it is absent, and fixing `DataFrame` while forgetting the other two. Four tests and three subtests fail before the change and all pass after; 137 tests and 547 subtests pass across `tests/protocols/application/`, the round-trip module and the four other HTTP/2-touching modules. `EXPECTED_FAILURES` is unmoved, its one HTTP/2 entry `httpv2-frame/PRIORITY` still failing with the same `CONSTRUCT` status and the same detail -- `PriorityFrame` has neither a payload field nor a `PADDED` flag, so this change cannot reach it -- and none of the 44 entries was deleted; that entry's own cited `pcapkit/protocols/application/httpv2.py:572` is stale, the `header.length != 9` guard it describes now being at `:562`. The schema module was already at 100% coverage and stays there, 101 statements and 4 branches either side with nothing missing or partial, because the changed lines did execute before -- just with an empty payload -- so the number that moved is the suite: 73 to 88 tests and 399 to 408 subtests over the same targets. The construct side needed no change, `_make_http_length` having always computed the DATA payload as `len(frame.data) + (pad_len + 1 if pad_len else 0)`, i.e. assuming `data` holds the whole payload whether or not the frame is padded -- it was the read path that only agreed with that when the frame *was* padded. Ships labelled breaking: restoring a dropped payload moves the parse output of essentially every HTTP/2 capture on the success path, so anything holding a golden file or a regression baseline that recorded the empty value sees it change (#668). +- **Fixed** -- the schema half of every option-like registrar overwrote silently while the parser half of the very same call warned. `EnumSchema.register` was a bare `cls.__enum__[code] = schema`, and it is the schema half of 14 public registrars, so one `register_ipv4_option` call replacing a built-in named the parser it displaced and said nothing about the schema. `EnumSchema.__init_subclass__` reaches the same registry without calling `register`, so a subclass declared with a `code=` keyword -- the documented way to add a schema -- stayed silent too; both are guarded now, and the declaration hook's two branches are folded into one loop so the guard is written once. Presence is a faithful test there only because `_EnumRegistry.__missing__` returns a miss without recording it, so parsing one packet carrying an unknown code cannot make the next legitimate registration for that code warn about an entry no caller ever asked for. The seven code-keyed parser registrars -- on `ProtocolBase`, `Link`, `Internet`, `Frame`, `PCAPNG`, `Transport` and `SCTP` -- now name the displaced entry and its replacement, where before they reported only that something had been overwritten and never which class it was. Their presence-only firing condition is deliberately unchanged, and is not harmonised onto the narrower guard #681 gave `register_protocol`: that one is keyed on a name derived from the value it stores, whereas these seven take a `code` the caller supplies independently of the value, so one class registered under two codes yields two keys and never reaches one key twice. These tables also ship pre-seeded with unresolved `ModuleDescriptor` values, so an incumbent may be a two-string descriptor while the replacement is the very class it names, which leaves a different-class test undecidable without resolving the descriptor and forcing the import it exists to defer. `ContextRegistry.register` already raises on a duplicate, and the reassembly and ESP registrars append to lists with no key at all -- duplicate security-association SPIs being a designed feature there, resolved by scoring rather than by replacement -- so none of those three takes a guard. `import pcapkit` holds at one warning, a third-party deprecation, and no `RegistryWarning`, measured over the 326 registry writes it performs, none of which lands on a key already present; `tests/foundation/registry/` goes 13 to 14 tests and 87 to 95 subtests, the schema module holds 99% coverage with its 5 new statements covered and its misses flat at 1, mypy and pylint are unmoved at 112 errors and 364 messages, and `EXPECTED_FAILURES` is unmoved at 44 entries (#692). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 164936d6c..27ebc2b92 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2065,6 +2065,43 @@ pull requests between #326 and #509. dropped payload moves the parse output of essentially every HTTP/2 capture on the success path, so anything holding a golden file or a regression baseline that recorded the empty value sees it change (#668). +* **Fixed** -- the schema half of every option-like registrar overwrote + silently while the parser half of the very same call warned. + ``EnumSchema.register`` was a bare ``cls.__enum__[code] = schema``, and it is + the schema half of 14 public registrars, so one ``register_ipv4_option`` call + replacing a built-in named the parser it displaced and said nothing about the + schema. ``EnumSchema.__init_subclass__`` reaches the same registry without + calling ``register``, so a subclass declared with a ``code=`` keyword -- the + documented way to add a schema -- stayed silent too; both are guarded now, + and the declaration hook's two branches are folded into one loop so the guard + is written once. Presence is a faithful test there only because + ``_EnumRegistry.__missing__`` returns a miss without recording it, so parsing + one packet carrying an unknown code cannot make the next legitimate + registration for that code warn about an entry no caller ever asked for. The + seven code-keyed parser registrars -- on ``ProtocolBase``, ``Link``, + ``Internet``, ``Frame``, ``PCAPNG``, ``Transport`` and ``SCTP`` -- now name + the displaced entry and its replacement, where before they reported only that + something had been overwritten and never which class it was. Their + presence-only firing condition is deliberately unchanged, and is not + harmonised onto the narrower guard #681 gave ``register_protocol``: that one + is keyed on a name derived from the value it stores, whereas these seven take + a ``code`` the caller supplies independently of the value, so one class + registered under two codes yields two keys and never reaches one key twice. + These tables also ship pre-seeded with unresolved ``ModuleDescriptor`` + values, so an incumbent may be a two-string descriptor while the replacement + is the very class it names, which leaves a different-class test undecidable + without resolving the descriptor and forcing the import it exists to defer. + ``ContextRegistry.register`` already raises on a duplicate, and the + reassembly and ESP registrars append to lists with no key at all -- duplicate + security-association SPIs being a designed feature there, resolved by scoring + rather than by replacement -- so none of those three takes a guard. + ``import pcapkit`` holds at one warning, a third-party deprecation, and no + ``RegistryWarning``, measured over the 326 registry writes it performs, none + of which lands on a key already present; ``tests/foundation/registry/`` goes + 13 to 14 tests and 87 to 95 subtests, the schema module holds 99% coverage + with its 5 new statements covered and its misses flat at 1, mypy and pylint + are unmoved at 112 errors and 364 messages, and ``EXPECTED_FAILURES`` is + unmoved at 44 entries (#692). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 01930d1afa0dae71bb93b1d48e090289b3132251 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 23:39:42 -0400 Subject: [PATCH 26/46] docs(changelog): record what the #668 cross-review found, and correct its counts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A cross-review on a second model turned up two things the first #668 entry did not say, one of them a second half to the defect: - `BytesField` consults its `length` callback when packing as well as parsing, so the mis-parenthesised arm also declined to *write* the payload. An unpadded DATA frame packed to nine octets of header declaring 21, which is pcapkit emitting malformed HTTP/2 rather than only mis-reading it. The entry framed the defect as parse-side, following the issue; it now carries both halves and the measured octet counts for all three frames. - The first revision's tests were satisfied by a `max(computed, 1)` wrong fix on the two `fragment` fields, because every fixture carried a non-empty payload and only DATA had a padding-only case. Recorded as the sixth rejected variant, with what it silently produced. Counts corrected accordingly: 15 tests and 9 subtests to 23 and 12, four failures before to seven, 137/547 to 145/550, and 73→88/399→408 to 73→96/399→411. Also records the AST sweep of all 496 package files, and why the one other site sharing the shape (`ipv4.py:336`) is correct rather than the same defect. `CHANGELOG.md` regenerated with `util/changelog_md.py`; `--check` exits 0. --- CHANGELOG.md | 2 +- docs/source/changelog/1.5.0.rst | 63 +++++++++++++++++++++++---------- 2 files changed, 46 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 547b969c8..587c399b6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -105,7 +105,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Changed** -- `SeekableReader.truncate()` raises instead of returning a size, and the misspelled `writeable()` is now spelled `writable()`. Both are breaks for an external caller and neither breaks anything inside the package. `io.IOBase` documents one gate over two methods -- "If False, write() and truncate() will raise OSError" -- and this reader's `writable()` answers False, so a caller that checked it first, which is exactly what the contract invites, got a surprise either way round: `write` and `writelines` raised, and `truncate` returned its new size. What settled the question is that this is not a pure-Python nicety of `_pyio` that the accelerated path skips -- every ordinary read-only file object in CPython raises here, `open(path, 'rb').truncate()` giving `io.UnsupportedOperation: truncate` off the C `BufferedReader`, and `_pyio.BufferedReader` the same type from `_BufferedIOMixin.truncate`'s `_checkWritable()`. The refusal is `UnsupportedOperation('truncate')`, the same one `write` already raises, and no trade-off was needed between the house exception and the ABC's: pcapkit's `UnsupportedOperation` subclasses `io.UnsupportedOperation`, which subclasses `OSError`, so the in-library exception *is* the one the contract names. The resizing `truncate` used to perform is not deleted, only made private as `_truncate_buffer`. It never touched the underlying stream -- it resizes a private lookback window, which is the counter-argument the issue itself raised -- and it is the only route to the "position sits before the window" state that `seek` and the four buffered read paths must refuse, which #643 and #644 landed tests for; deleting it would have taken their mechanism with them. `writeable()`, separately, was never an override of anything: `'writable' in SeekableReader.__dict__` was False and `SeekableReader.writable is io.IOBase.writable` was True, so `io`, `shutil` and any third-party caller read the inherited value and never saw the one defined in this file. Both answered False, which is the coincidence that hid it -- there was no symptom to notice, and editing the misspelled method would silently have had no effect. The value reported is unchanged and was always honest, the reader genuinely being unable to write; only where the method was defined was wrong. Nothing in the package called either method, confirmed by grep, so the risk is entirely to external callers, which is what made this a question asked before it was acted on. The new test asserts the property rather than the behaviour -- that a `SeekableReader` raises the same exception type a read-only `io.BufferedReader` raises for the same call -- with that type captured by running the call rather than named in the test, so it tracks CPython across the 3.10--3.15 matrix instead of restating a belief about it. 31 to 34 tests and 35 to 41 subtests over `tests/corekit/test_io.py`, the module at 100% coverage before and after (#645). - **Fixed** -- `register_protocol` displaced one protocol class with another and said nothing about it. The registry behind `pcapkit.protocols.__proto__` is keyed on `cls.__name__.upper()`, and three dispatchable classes are all named `HTTP` -- the generic base `pcapkit.protocols.application.http.HTTP` and the two version implementations in `httpv1` and `httpv2` -- so all three compete for the single key `'HTTP'` and a bare assignment decided the winner. Measured before the fix: `__proto__['HTTP']` went from `pcapkit.protocols.application.http.HTTP` to the `httpv2` class on `register_protocol(httpv2.HTTP)`, with no exception, no log line and no `RegistryWarning`, and `ProtocolBase.expand_comp('HTTP')` then resolved to whichever class had registered last. The overwrite now raises a `RegistryWarning` naming both the displaced and the replacing class, the defining module being the only thing that tells the three apart. The sharpest path to it is not a call to the registrar at all: `Protocol.__init_subclass__` registers unconditionally, so merely *defining* a subclass of the public `Protocol` under a name a built-in already holds displaced that built-in, with nothing anywhere in the user's code to mark the moment the dispatch table changed meaning. A plain `import pcapkit` is unaffected and emits zero new warnings, measured not assumed -- the three built-in `HTTP` classes derive from `ProtocolBase` rather than from the public `Protocol`, so `__init_subclass__` never fires for them, and the import-time seeding in `pcapkit/protocols/__init__.py` keys off the distinct names `HTTP`, `HTTPv1` and `HTTPv2`. The issue's framing that this was the one registrar in `pcapkit/foundation/registry/` with no `RegistryWarning` is right on the count and wrong on the reason: no function in that package warns about anything, the siblings warning only by delegating to a classmethod that carries the guarded `if code in cls.__xxx__: warn(...)`. The accurate statement is the stronger one -- `register_protocol` is the only keyed registrar in that package, and the only `__name__.upper()`-keyed registry anywhere in `pcapkit/`, that mutates its target with no guard *and* no delegate that could supply one, its target being a module-level dict with no class behind it to hold the guard. The new guard deliberately reads "key present and the incumbent is a different class" where every sibling warns on mere presence: the sibling keys are codes the caller passes, whereas this key is derived from the class and this function is the funnel all nine wrapper registrars end in, so registering one class under two codes -- `register_tcp` then `register_udp`, which is supported and documented -- reaches it twice with nothing displaced, and warning there would put noise on a documented path whose wholesale filter is precisely what would then hide the real collision. The collision is reported, not resolved: re-keying is a registry-format change that none of the bare-name readers can absorb, every one of them degrading *silently* to a plain string or to `Raw` on a miss rather than raising, so it belongs to the registry redesign in #514, which this change is sequenced ahead of rather than part of. 9 to 13 tests and 80 to 87 subtests over `tests/foundation/registry/`, the touched module holding 88% coverage with its misses flat at 27 (#675). - **Fixed** -- every PCAP-NG packet block now carries the captured octets it declares. All four Enhanced Packet Blocks of the committed `examples/captures/dhcp.pcapng` reported `len(packet) == 0` against a `captured_len` of 314, 342, 314 and 342, and the Simple Packet Block and the obsolete Packet Block did the same, measured on a capture synthesised to carry one of each since no committed fixture has either. Those three -- exactly `PCAPNG.PACKET_TYPES`, and exactly the three schemas declaring `__payload__ = 'packet_data'` -- are the whole of the blast radius; every other block type carries no captured octets, reported `b''` before and still does. The octets were read correctly and then thrown away: `PCAPNG.unpack` extracted them from the block schema, and `ProtocolBase.__init__` overwrote the result with `self.packet.payload`, which was empty. The inherited `packet` splits a protocol at `self.length` octets of header and takes everything after as the payload, a contract that holds for a protocol laid out as a header followed by its payload and for nothing else. `PCAPNG.length` returns the wire's *Block Total Length*, so the split consumed the entire per-block buffer as header -- measured at `frame.length == 348` and `len(frame.packet.header) == 348` on a 348-octet block -- and, separately, a PCAP-NG block carries a *trailer* after its payload, the option list and a repeat of the total length, so no value of `length` could have made that split right. `captured_len` comes straight off the block header and survived, which is why the two disagreed rather than both being empty. The fix is at the computation rather than at the injection: `PCAPNG.packet` is overridden to take the payload from the block schema's `__payload__` field and the header from the octets ahead of it, summed out of the schema buffers so that the three block types' three different payload offsets -- 28, 12 and 28 octets -- are not hard-coded. Teaching `ProtocolBase.__init__` to skip a protocol that had already set `packet` was the smaller change and was rejected: it heals `frame.info.packet` while leaving the public `frame.packet` still reporting the whole block as header and `b''` as payload, which is the same defect from the other side. `PCAPNG.unpack` now reads that property instead of extracting the payload a second time of its own, leaving one source of truth where there were two attempts at one. The property is a plain `property` rather than the `cached_property` it overrides, which a cross-review on a second model is what settled: the inherited one caches because it *reads the stream*, where a second read would consume octets that are gone, while this one only walks buffers the schema layer has already filled and so has nothing to amortise -- and caching it would have reintroduced the same staleness by another route, a second `unpack` on one instance handing back the *first* call's octets with `get_payload` never reached, an invariant the code held before this change and has no reason to stop holding. Nothing calls `unpack` twice on one instance today, `__post_init__` being its only caller, so the unit test asserts the property directly: swap the block on a live instance, unpack again, and the payload must be the new block's, which fails `b'payload' != b'cached'` under a cache. `ProtocolBase.packet` gains the contract in its docstring and not one executable line, its 503 statements unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on the success path, which is why it ships labelled breaking: `frame.info.packet` goes from `b''` to hundreds of octets, `frame.packet.header` from the whole block to the pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what made this a wire-format defect rather than an API wart. `PCAPIO` writes `value.packet` after each 16-octet record header, so dumping those four blocks produced a **104-octet** file -- 24 of global header plus four record headers and no payload at all -- in which every record header declared hundreds of octets and delivered none, desynchronising any reader that walks by `incl_len`, which is all of them: pcapkit refused its own output with `ValueError: read length must be non-negative or -1` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes rather than against a length. `tests/protocols/test_pcapng_regression.py` grows 4 tests to 11 and 3 subtests to 24, with the payload expectations derived twice over -- spelled-out head and tail literals, and a `struct`-only re-parse of the fixture that owes nothing to the code under test -- and each synthesised payload given a different length modulo 4 so that an offset off by a field, or a payload that picked up the block's 32-bit padding, cannot pass by coincidence. Two shapes no fixture exercised are covered there too, both found while writing the tests rather than after: an option area *after* the captured data, which every block of `dhcp.pcapng` lacks and which is exactly what an offset walked from the wrong end would swallow, and a big-endian section, whose fields differ even though the payload offsets do not. A snapped block is covered in both the shapes that express it -- an Enhanced Packet Block declaring `captured_len` below `original_len`, and a Simple Packet Block whose captured length is bounded by the interface's `snaplen` -- neither of which may come back padded out to the on-wire length. The offsets hold at 28, 12 and 28 across all of it, which is the property the walk has to have. With both trees running the same -- current -- tests, which is the only comparison that isolates the code from the suite, `pcapkit/protocols/misc/pcapng.py` holds 99.91% across the change, its 18 new statements and 6 new branches all executed, its miss count flat at 1 and its partial-branch count flat at 1, that single miss being the same pre-existing `_get_timezone` statement renumbered 1153 to 1265; `pcapkit/protocols/protocol.py` is identical in every column, 503 statements and 228 misses either side, which is the check that its change really is docstring-only. The uncaught `ValueError` that an EOF-truncated PCAP-NG raises (#678) is untouched and was measured rather than assumed, over a truncation set stated rather than described because "101 levels" admits several readings that do not agree on the counts: the whole-percent prefixes `max(1, 1508 * i // 100) for i in range(101)`, which for this file are 101 distinct lengths from 1 to 1508. Both trees give 96 of that `ValueError`, one `FormatError`, one `ProtocolError` and three parses, and -- compared level by level rather than only in aggregate, since a matching total can hide two levels that swapped -- the SHA-256 of the sorted length-to-outcome mapping is `ca44d3ee658087cf2a667c454f839dd931c1c04790b2fe32cee8e1a2e861b182` on each. `examples/captures/pcapng.txt`, a hand-regenerated legacy smoke reference, still shows the old frame-level `packet -> NIL` and wants a separate refresh (#646). -- **Fixed** -- an unpadded HTTP/2 `DATA`, `HEADERS` or `PUSH_PROMISE` frame parsed with its whole payload silently discarded. Three payload length callbacks in `pcapkit/protocols/schema/application/httpv2.py` put the conditional expression in the wrong place: a conditional binds looser than `-`, so `pkt['__length__'] - pkt['pad_len'] if pkt['flags']['bit_3'] else 0` groups as `(pkt['__length__'] - pkt['pad_len']) if ... else 0` and the `else` arm returned `0` -- "read no octets at all" to `BytesField` -- where it was meant to subtract `0`. Since `__length__` is the *remaining* declared length at the field, the unpadded arm wants `__length__` itself, which is what subtracting a zero padding length gives. The grouping was read off the AST rather than by eye, the top-level node being the `IfExp` whose `orelse` is a bare `Constant(0)`, which is also what showed that the outer parentheses on the `HeadersFrame` and `PushPromiseFrame` forms were line-continuation only and changed nothing. Padding is rare in HTTP/2, so the broken arm was the common one rather than the edge case, and nothing raised, warned or logged: `info.data` was simply `b''`. Measured on wire octets through the public `HTTP(io.BytesIO(raw), len(raw))` path, an unpadded `DATA` frame declaring `b'{"ok":true}\n'` parsed as `b''` and now parses as those twelve octets; unpadded `HEADERS` and `PUSH_PROMISE` lost and now keep the RFC 7541 appendix C.4.1 header block fragment `b'\x82\x86\x84A\x0fwww.example.com'`, which is precisely what HPACK decoding would have been handed. Padded frames are untouched, for a reason stronger than the measurement: `(A - B) if T else 0` and `A - (B if T else 0)` are both `A - B` for truthy `T`, so only the `else` arm could ever have moved -- both arms are asserted anyway, so that a repair to the unpadded one cannot break the other. The padded arm also has no off-by-one, which #668 asked be checked rather than assumed: `Schema.unpack` decrements `packet['__length__']` by each field's width as it goes, so `pad_len`'s own octet is already out of it by the time the payload field runs, and a padded frame declaring `1 + len(data) + pad_len` arrives at the payload with `__length__ == len(data) + pad_len`. The three sibling sites that were never wrong are what localise the defect to the grouping rather than to the field machinery: `ContinuationFrame.fragment`, `UnassignedFrame.data` and `GoawayFrame.debug` use the plain `length=lambda pkt: pkt['__length__']` with no conditional, share the same `BytesField`, the same `__length__` bookkeeping and the same frame dispatch, and carried their payload correctly throughout -- they are pinned as controls rather than left as an argument. Why nothing caught it: `test_option_roundtrip_unit` does drive every frame type, but through `make` -> parse -> `make`, and `make` writes the length field from `_make_http_length`, so both sides agreed on an empty payload and the octets matched; its generator passes no payload argument for these three frames at all, measured as `kwargs={}`, so what it round-tripped was `b''` and there was nothing to lose, while `test_httpv2_frame_readers_cover_successful_frames` drives the readers from hand-built schema stubs, so the callback never ran there either. The new `tests/protocols/application/test_httpv2_payload_length_unit.py` reads wire octets, which is the gap: 15 tests and 9 subtests over the padded and unpadded shapes of all three frames plus both `PRIORITY` combinations, asserting the payload *octets* and never its length, because a length assertion survives a read of the right width from the wrong offset. Its padded cases pad with a non-zero `b'\xde\xad\xbe\xef'` rather than the zeros [RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1) tells a sender to use -- pcapkit does not police padding content -- so that failing to subtract the padding yields a different byte string instead of a coincidentally equal count. Four wrong fixes were written over the schema to check the tests discriminate, and each is rejected: dropping the conditional outright (caught by `packet length < 0: -4`), subtracting `pad_len + 1` (caught as `b'{"ok":true}'` against `b'{"ok":true}\n'`), subtracting the padding on the arm where it is absent, and fixing `DataFrame` while forgetting the other two. Four tests and three subtests fail before the change and all pass after; 137 tests and 547 subtests pass across `tests/protocols/application/`, the round-trip module and the four other HTTP/2-touching modules. `EXPECTED_FAILURES` is unmoved, its one HTTP/2 entry `httpv2-frame/PRIORITY` still failing with the same `CONSTRUCT` status and the same detail -- `PriorityFrame` has neither a payload field nor a `PADDED` flag, so this change cannot reach it -- and none of the 44 entries was deleted; that entry's own cited `pcapkit/protocols/application/httpv2.py:572` is stale, the `header.length != 9` guard it describes now being at `:562`. The schema module was already at 100% coverage and stays there, 101 statements and 4 branches either side with nothing missing or partial, because the changed lines did execute before -- just with an empty payload -- so the number that moved is the suite: 73 to 88 tests and 399 to 408 subtests over the same targets. The construct side needed no change, `_make_http_length` having always computed the DATA payload as `len(frame.data) + (pad_len + 1 if pad_len else 0)`, i.e. assuming `data` holds the whole payload whether or not the frame is padded -- it was the read path that only agreed with that when the frame *was* padded. Ships labelled breaking: restoring a dropped payload moves the parse output of essentially every HTTP/2 capture on the success path, so anything holding a golden file or a regression baseline that recorded the empty value sees it change (#668). +- **Fixed** -- an unpadded HTTP/2 `DATA`, `HEADERS` or `PUSH_PROMISE` frame parsed with its whole payload silently discarded. Three payload length callbacks in `pcapkit/protocols/schema/application/httpv2.py` put the conditional expression in the wrong place: a conditional binds looser than `-`, so `pkt['__length__'] - pkt['pad_len'] if pkt['flags']['bit_3'] else 0` groups as `(pkt['__length__'] - pkt['pad_len']) if ... else 0` and the `else` arm returned `0` -- "read no octets at all" to `BytesField` -- where it was meant to subtract `0`. Since `__length__` is the *remaining* declared length at the field, the unpadded arm wants `__length__` itself, which is what subtracting a zero padding length gives. The grouping was read off the AST rather than by eye, the top-level node being the `IfExp` whose `orelse` is a bare `Constant(0)`, which is also what showed that the outer parentheses on the `HeadersFrame` and `PushPromiseFrame` forms were line-continuation only and changed nothing. Padding is rare in HTTP/2, so the broken arm was the common one rather than the edge case, and nothing raised, warned or logged: `info.data` was simply `b''`. Measured on wire octets through the public `HTTP(io.BytesIO(raw), len(raw))` path, an unpadded `DATA` frame declaring `b'{"ok":true}\n'` parsed as `b''` and now parses as those twelve octets; unpadded `HEADERS` and `PUSH_PROMISE` lost and now keep the RFC 7541 appendix C.4.1 header block fragment `b'\x82\x86\x84A\x0fwww.example.com'`, which is precisely what HPACK decoding would have been handed. Padded frames are untouched, for a reason stronger than the measurement: `(A - B) if T else 0` and `A - (B if T else 0)` are both `A - B` for truthy `T`, so only the `else` arm could ever have moved -- both arms are asserted anyway, so that a repair to the unpadded one cannot break the other. The padded arm also has no off-by-one, which #668 asked be checked rather than assumed: `Schema.unpack` decrements `packet['__length__']` by each field's width as it goes, so `pad_len`'s own octet is already out of it by the time the payload field runs, and a padded frame declaring `1 + len(data) + pad_len` arrives at the payload with `__length__ == len(data) + pad_len`. The three sibling sites that were never wrong are what localise the defect to the grouping rather than to the field machinery: `ContinuationFrame.fragment`, `UnassignedFrame.data` and `GoawayFrame.debug` use the plain `length=lambda pkt: pkt['__length__']` with no conditional, share the same `BytesField`, the same `__length__` bookkeeping and the same frame dispatch, and carried their payload correctly throughout -- they are pinned as controls rather than left as an argument. Why nothing caught it: `test_option_roundtrip_unit` does drive every frame type, but through `make` -> parse -> `make`, and `make` writes the length field from `_make_http_length`, so both sides agreed on an empty payload and the octets matched; its generator passes no payload argument for these three frames at all, measured as `kwargs={}`, so what it round-tripped was `b''` and there was nothing to lose, while `test_httpv2_frame_readers_cover_successful_frames` drives the readers from hand-built schema stubs, so the callback never ran there either. The new `tests/protocols/application/test_httpv2_payload_length_unit.py` reads wire octets, which is the gap: 23 tests and 12 subtests over the padded, unpadded and padding-only shapes of all three frames plus both `PRIORITY` combinations, asserting the payload *octets* and never its length, because a length assertion survives a read of the right width from the wrong offset. Its padded cases pad with a non-zero `b'\xde\xad\xbe\xef'` rather than the zeros [RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1) tells a sender to use -- pcapkit does not police padding content -- so that failing to subtract the padding yields a different byte string instead of a coincidentally equal count. The same lambda also governs **packing**, which neither the issue nor the first revision of the fix noticed and a cross-review on a second model did: `BytesField` consults its `length` callback on both paths, so the `else` arm did not merely discard a payload on read, it declined to write one. Measured pre-fix, an unpadded `DATA` frame carrying twelve octets of body packed to `000015000000000001` -- nine octets of header whose length field declares **21**, with the body absent -- and unpadded `HEADERS` and `PUSH_PROMISE` likewise declared 29 against 9 and 33 against 13. A frame overstating its own length by its whole payload desynchronises any reader that walks a stream by that field, so this was pcapkit *emitting* malformed HTTP/2 rather than only mis-reading it, and `HTTPv2ConstructedFrameDeclaresWhatItWritesUnitTests` pins that half; a `make` -> parse -> `make` cycle of a non-empty unpadded payload now closes byte-for-byte where it could not before. Six wrong fixes were written over the schema to check the tests discriminate, and each is rejected: dropping the conditional outright (caught by `packet length < 0: -4`), subtracting `pad_len + 1` (caught as `b'{"ok":true}'` against `b'{"ok":true}\n'`), subtracting the padding on the arm where it is absent, fixing `DataFrame` while forgetting the other two, and `max(computed, 1)` on the two `fragment` fields. That last one is the cross-review's own construction and it **passed** the first revision's 15 tests and 9 subtests, because every fixture then carried a non-empty payload and only `DataFrame` had a "frame of only padding" case, so those two callbacks were never once asked for `0`; under it a padding-only `HEADERS` frame returned `fragment=b'\xde'`, one padding octet leaked in, alongside a swallowed `SchemaWarning: packet length < 0: -1`. The three missing padding-only cases and a callback-level `test_the_padded_arm_reaches_zero_and_is_not_clamped` close it -- `0` is a legitimate answer from these callbacks and has to be asserted as one. Seven tests and three subtests fail before the change and all pass after; 145 tests and 550 subtests pass across `tests/protocols/application/`, the round-trip module and the four other HTTP/2-touching modules. `EXPECTED_FAILURES` is unmoved, its one HTTP/2 entry `httpv2-frame/PRIORITY` still failing with the same `CONSTRUCT` status and the same detail -- `PriorityFrame` has neither a payload field nor a `PADDED` flag, so this change cannot reach it -- and none of the 44 entries was deleted; that entry's own cited `pcapkit/protocols/application/httpv2.py:572` is stale, the `header.length != 9` guard it describes now being at `:562`. The schema module was already at 100% coverage and stays there, 101 statements and 4 branches either side with nothing missing or partial, because the changed lines did execute before -- just with an empty payload -- so the number that moved is the suite: 73 to 96 tests and 399 to 411 subtests over the same targets. The construct side needed no code change, `_make_http_length` having always computed the DATA payload as `len(frame.data) + (pad_len + 1 if pad_len else 0)`, i.e. assuming `data` holds the whole payload whether or not the frame is padded -- it was the shared length callback that disagreed, on both paths at once. An AST sweep of all 496 files of the package found one other site with the same syntactic shape, `pcapkit/protocols/schema/internet/ipv4.py:336` (`TSOption.remainder`), and it is correct rather than the same defect: its `else 0` is intentional because the sibling `ts_data` field consumes the entire option data area in that arm, the two summing to `length - 4` across all 2106 `(length, pointer, flag)` combinations, which is exactly what was not true here. Ships labelled breaking on both counts: restoring a dropped payload moves the parse output of essentially every HTTP/2 capture on the success path, so anything holding a golden file or a regression baseline that recorded the empty value sees it change -- and constructed frames change byte-for-byte too, from malformed to correct (#668). - **Fixed** -- the schema half of every option-like registrar overwrote silently while the parser half of the very same call warned. `EnumSchema.register` was a bare `cls.__enum__[code] = schema`, and it is the schema half of 14 public registrars, so one `register_ipv4_option` call replacing a built-in named the parser it displaced and said nothing about the schema. `EnumSchema.__init_subclass__` reaches the same registry without calling `register`, so a subclass declared with a `code=` keyword -- the documented way to add a schema -- stayed silent too; both are guarded now, and the declaration hook's two branches are folded into one loop so the guard is written once. Presence is a faithful test there only because `_EnumRegistry.__missing__` returns a miss without recording it, so parsing one packet carrying an unknown code cannot make the next legitimate registration for that code warn about an entry no caller ever asked for. The seven code-keyed parser registrars -- on `ProtocolBase`, `Link`, `Internet`, `Frame`, `PCAPNG`, `Transport` and `SCTP` -- now name the displaced entry and its replacement, where before they reported only that something had been overwritten and never which class it was. Their presence-only firing condition is deliberately unchanged, and is not harmonised onto the narrower guard #681 gave `register_protocol`: that one is keyed on a name derived from the value it stores, whereas these seven take a `code` the caller supplies independently of the value, so one class registered under two codes yields two keys and never reaches one key twice. These tables also ship pre-seeded with unresolved `ModuleDescriptor` values, so an incumbent may be a two-string descriptor while the replacement is the very class it names, which leaves a different-class test undecidable without resolving the descriptor and forcing the import it exists to defer. `ContextRegistry.register` already raises on a duplicate, and the reassembly and ESP registrars append to lists with no key at all -- duplicate security-association SPIs being a designed feature there, resolved by scoring rather than by replacement -- so none of those three takes a guard. `import pcapkit` holds at one warning, a third-party deprecation, and no `RegistryWarning`, measured over the 326 registry writes it performs, none of which lands on a key already present; `tests/foundation/registry/` goes 13 to 14 tests and 87 to 95 subtests, the schema module holds 99% coverage with its 5 new statements covered and its misses flat at 1, mypy and pylint are unmoved at 112 errors and 364 messages, and `EXPECTED_FAILURES` is unmoved at 44 entries (#692). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 27ebc2b92..9523bc507 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2035,20 +2035,40 @@ pull requests between #326 and #509. nothing to lose, while ``test_httpv2_frame_readers_cover_successful_frames`` drives the readers from hand-built schema stubs, so the callback never ran there either. The new ``tests/protocols/application/test_httpv2_payload_length_unit.py`` reads wire octets, - which is the gap: 15 tests and 9 subtests over the padded and unpadded shapes of all - three frames plus both ``PRIORITY`` combinations, asserting the payload *octets* and - never its length, because a length assertion survives a read of the right width from - the wrong offset. Its padded cases pad with a non-zero ``b'\xde\xad\xbe\xef'`` rather - than the zeros :rfc:`9113#section-6.1` tells a sender to use -- pcapkit does not police - padding content -- so that failing to subtract the padding yields a different byte - string instead of a coincidentally equal count. Four wrong fixes were written over the - schema to check the tests discriminate, and each is rejected: dropping the conditional - outright (caught by ``packet length < 0: -4``), subtracting ``pad_len + 1`` (caught as + which is the gap: 23 tests and 12 subtests over the padded, unpadded and padding-only + shapes of all three frames plus both ``PRIORITY`` combinations, asserting the payload + *octets* and never its length, because a length assertion survives a read of the right + width from the wrong offset. Its padded cases pad with a non-zero + ``b'\xde\xad\xbe\xef'`` rather than the zeros :rfc:`9113#section-6.1` tells a sender to + use -- pcapkit does not police padding content -- so that failing to subtract the + padding yields a different byte string instead of a coincidentally equal count. The + same lambda also governs **packing**, which neither the issue nor the first revision of + the fix noticed and a cross-review on a second model did: ``BytesField`` consults its + ``length`` callback on both paths, so the ``else`` arm did not merely discard a payload + on read, it declined to write one. Measured pre-fix, an unpadded ``DATA`` frame carrying + twelve octets of body packed to ``000015000000000001`` -- nine octets of header whose + length field declares **21**, with the body absent -- and unpadded ``HEADERS`` and + ``PUSH_PROMISE`` likewise declared 29 against 9 and 33 against 13. A frame overstating + its own length by its whole payload desynchronises any reader that walks a stream by + that field, so this was pcapkit *emitting* malformed HTTP/2 rather than only mis-reading + it, and ``HTTPv2ConstructedFrameDeclaresWhatItWritesUnitTests`` pins that half; a + ``make`` -> parse -> ``make`` cycle of a non-empty unpadded payload now closes + byte-for-byte where it could not before. Six wrong fixes were written over the schema to + check the tests discriminate, and each is rejected: dropping the conditional outright + (caught by ``packet length < 0: -4``), subtracting ``pad_len + 1`` (caught as ``b'{"ok":true}'`` against ``b'{"ok":true}\n'``), subtracting the padding on the arm - where it is absent, and fixing ``DataFrame`` while forgetting the other two. Four tests - and three subtests fail before the change and all pass after; 137 tests and 547 - subtests pass across ``tests/protocols/application/``, the round-trip module and the - four other HTTP/2-touching modules. ``EXPECTED_FAILURES`` is unmoved, its one HTTP/2 + where it is absent, fixing ``DataFrame`` while forgetting the other two, and + ``max(computed, 1)`` on the two ``fragment`` fields. That last one is the cross-review's + own construction and it **passed** the first revision's 15 tests and 9 subtests, because + every fixture then carried a non-empty payload and only ``DataFrame`` had a "frame of + only padding" case, so those two callbacks were never once asked for ``0``; under it a + padding-only ``HEADERS`` frame returned ``fragment=b'\xde'``, one padding octet leaked + in, alongside a swallowed ``SchemaWarning: packet length < 0: -1``. The three missing + padding-only cases and a callback-level ``test_the_padded_arm_reaches_zero_and_is_not_clamped`` + close it -- ``0`` is a legitimate answer from these callbacks and has to be asserted as + one. Seven tests and three subtests fail before the change and all pass after; 145 tests + and 550 subtests pass across ``tests/protocols/application/``, the round-trip module and + the four other HTTP/2-touching modules. ``EXPECTED_FAILURES`` is unmoved, its one HTTP/2 entry ``httpv2-frame/PRIORITY`` still failing with the same ``CONSTRUCT`` status and the same detail -- ``PriorityFrame`` has neither a payload field nor a ``PADDED`` flag, so this change cannot reach it -- and none of the 44 entries was deleted; that entry's @@ -2056,15 +2076,22 @@ pull requests between #326 and #509. ``header.length != 9`` guard it describes now being at ``:562``. The schema module was already at 100% coverage and stays there, 101 statements and 4 branches either side with nothing missing or partial, because the changed lines did execute before -- just - with an empty payload -- so the number that moved is the suite: 73 to 88 tests and 399 - to 408 subtests over the same targets. The construct side needed no change, + with an empty payload -- so the number that moved is the suite: 73 to 96 tests and 399 + to 411 subtests over the same targets. The construct side needed no code change, ``_make_http_length`` having always computed the DATA payload as ``len(frame.data) + (pad_len + 1 if pad_len else 0)``, i.e. assuming ``data`` holds the - whole payload whether or not the frame is padded -- it was the read path that only - agreed with that when the frame *was* padded. Ships labelled breaking: restoring a + whole payload whether or not the frame is padded -- it was the shared length callback + that disagreed, on both paths at once. An AST sweep of all 496 files of the package + found one other site with the same syntactic shape, + ``pcapkit/protocols/schema/internet/ipv4.py:336`` (``TSOption.remainder``), and it is + correct rather than the same defect: its ``else 0`` is intentional because the sibling + ``ts_data`` field consumes the entire option data area in that arm, the two summing to + ``length - 4`` across all 2106 ``(length, pointer, flag)`` combinations, which is + exactly what was not true here. Ships labelled breaking on both counts: restoring a dropped payload moves the parse output of essentially every HTTP/2 capture on the success path, so anything holding a golden file or a regression baseline that recorded - the empty value sees it change (#668). + the empty value sees it change -- and constructed frames change byte-for-byte too, from + malformed to correct (#668). * **Fixed** -- the schema half of every option-like registrar overwrote silently while the parser half of the very same call warned. ``EnumSchema.register`` was a bare ``cls.__enum__[code] = schema``, and it is From 55f43a21ad795f9576fa19b156f8f6f8b1578217 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 23:53:42 -0400 Subject: [PATCH 27/46] docs(changelog): the 1.5.0 entry for the truncated-PCAP-NG negative length (#678) Code ships in #699. One hunk, 60 added lines, zero deletions, as every entry on this branch is. `python util/changelog_md.py --check` exits 0. --- CHANGELOG.md | 1 + docs/source/changelog/1.5.0.rst | 60 +++++++++++++++++++++++++++++++++ 2 files changed, 61 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 587c399b6..3a4ef7085 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -107,6 +107,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- every PCAP-NG packet block now carries the captured octets it declares. All four Enhanced Packet Blocks of the committed `examples/captures/dhcp.pcapng` reported `len(packet) == 0` against a `captured_len` of 314, 342, 314 and 342, and the Simple Packet Block and the obsolete Packet Block did the same, measured on a capture synthesised to carry one of each since no committed fixture has either. Those three -- exactly `PCAPNG.PACKET_TYPES`, and exactly the three schemas declaring `__payload__ = 'packet_data'` -- are the whole of the blast radius; every other block type carries no captured octets, reported `b''` before and still does. The octets were read correctly and then thrown away: `PCAPNG.unpack` extracted them from the block schema, and `ProtocolBase.__init__` overwrote the result with `self.packet.payload`, which was empty. The inherited `packet` splits a protocol at `self.length` octets of header and takes everything after as the payload, a contract that holds for a protocol laid out as a header followed by its payload and for nothing else. `PCAPNG.length` returns the wire's *Block Total Length*, so the split consumed the entire per-block buffer as header -- measured at `frame.length == 348` and `len(frame.packet.header) == 348` on a 348-octet block -- and, separately, a PCAP-NG block carries a *trailer* after its payload, the option list and a repeat of the total length, so no value of `length` could have made that split right. `captured_len` comes straight off the block header and survived, which is why the two disagreed rather than both being empty. The fix is at the computation rather than at the injection: `PCAPNG.packet` is overridden to take the payload from the block schema's `__payload__` field and the header from the octets ahead of it, summed out of the schema buffers so that the three block types' three different payload offsets -- 28, 12 and 28 octets -- are not hard-coded. Teaching `ProtocolBase.__init__` to skip a protocol that had already set `packet` was the smaller change and was rejected: it heals `frame.info.packet` while leaving the public `frame.packet` still reporting the whole block as header and `b''` as payload, which is the same defect from the other side. `PCAPNG.unpack` now reads that property instead of extracting the payload a second time of its own, leaving one source of truth where there were two attempts at one. The property is a plain `property` rather than the `cached_property` it overrides, which a cross-review on a second model is what settled: the inherited one caches because it *reads the stream*, where a second read would consume octets that are gone, while this one only walks buffers the schema layer has already filled and so has nothing to amortise -- and caching it would have reintroduced the same staleness by another route, a second `unpack` on one instance handing back the *first* call's octets with `get_payload` never reached, an invariant the code held before this change and has no reason to stop holding. Nothing calls `unpack` twice on one instance today, `__post_init__` being its only caller, so the unit test asserts the property directly: swap the block on a live instance, unpack again, and the payload must be the new block's, which fails `b'payload' != b'cached'` under a cache. `ProtocolBase.packet` gains the contract in its docstring and not one executable line, its 503 statements unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on the success path, which is why it ships labelled breaking: `frame.info.packet` goes from `b''` to hundreds of octets, `frame.packet.header` from the whole block to the pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what made this a wire-format defect rather than an API wart. `PCAPIO` writes `value.packet` after each 16-octet record header, so dumping those four blocks produced a **104-octet** file -- 24 of global header plus four record headers and no payload at all -- in which every record header declared hundreds of octets and delivered none, desynchronising any reader that walks by `incl_len`, which is all of them: pcapkit refused its own output with `ValueError: read length must be non-negative or -1` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes rather than against a length. `tests/protocols/test_pcapng_regression.py` grows 4 tests to 11 and 3 subtests to 24, with the payload expectations derived twice over -- spelled-out head and tail literals, and a `struct`-only re-parse of the fixture that owes nothing to the code under test -- and each synthesised payload given a different length modulo 4 so that an offset off by a field, or a payload that picked up the block's 32-bit padding, cannot pass by coincidence. Two shapes no fixture exercised are covered there too, both found while writing the tests rather than after: an option area *after* the captured data, which every block of `dhcp.pcapng` lacks and which is exactly what an offset walked from the wrong end would swallow, and a big-endian section, whose fields differ even though the payload offsets do not. A snapped block is covered in both the shapes that express it -- an Enhanced Packet Block declaring `captured_len` below `original_len`, and a Simple Packet Block whose captured length is bounded by the interface's `snaplen` -- neither of which may come back padded out to the on-wire length. The offsets hold at 28, 12 and 28 across all of it, which is the property the walk has to have. With both trees running the same -- current -- tests, which is the only comparison that isolates the code from the suite, `pcapkit/protocols/misc/pcapng.py` holds 99.91% across the change, its 18 new statements and 6 new branches all executed, its miss count flat at 1 and its partial-branch count flat at 1, that single miss being the same pre-existing `_get_timezone` statement renumbered 1153 to 1265; `pcapkit/protocols/protocol.py` is identical in every column, 503 statements and 228 misses either side, which is the check that its change really is docstring-only. The uncaught `ValueError` that an EOF-truncated PCAP-NG raises (#678) is untouched and was measured rather than assumed, over a truncation set stated rather than described because "101 levels" admits several readings that do not agree on the counts: the whole-percent prefixes `max(1, 1508 * i // 100) for i in range(101)`, which for this file are 101 distinct lengths from 1 to 1508. Both trees give 96 of that `ValueError`, one `FormatError`, one `ProtocolError` and three parses, and -- compared level by level rather than only in aggregate, since a matching total can hide two levels that swapped -- the SHA-256 of the sorted length-to-outcome mapping is `ca44d3ee658087cf2a667c454f839dd931c1c04790b2fe32cee8e1a2e861b182` on each. `examples/captures/pcapng.txt`, a hand-regenerated legacy smoke reference, still shows the old frame-level `packet -> NIL` and wants a separate refresh (#646). - **Fixed** -- an unpadded HTTP/2 `DATA`, `HEADERS` or `PUSH_PROMISE` frame parsed with its whole payload silently discarded. Three payload length callbacks in `pcapkit/protocols/schema/application/httpv2.py` put the conditional expression in the wrong place: a conditional binds looser than `-`, so `pkt['__length__'] - pkt['pad_len'] if pkt['flags']['bit_3'] else 0` groups as `(pkt['__length__'] - pkt['pad_len']) if ... else 0` and the `else` arm returned `0` -- "read no octets at all" to `BytesField` -- where it was meant to subtract `0`. Since `__length__` is the *remaining* declared length at the field, the unpadded arm wants `__length__` itself, which is what subtracting a zero padding length gives. The grouping was read off the AST rather than by eye, the top-level node being the `IfExp` whose `orelse` is a bare `Constant(0)`, which is also what showed that the outer parentheses on the `HeadersFrame` and `PushPromiseFrame` forms were line-continuation only and changed nothing. Padding is rare in HTTP/2, so the broken arm was the common one rather than the edge case, and nothing raised, warned or logged: `info.data` was simply `b''`. Measured on wire octets through the public `HTTP(io.BytesIO(raw), len(raw))` path, an unpadded `DATA` frame declaring `b'{"ok":true}\n'` parsed as `b''` and now parses as those twelve octets; unpadded `HEADERS` and `PUSH_PROMISE` lost and now keep the RFC 7541 appendix C.4.1 header block fragment `b'\x82\x86\x84A\x0fwww.example.com'`, which is precisely what HPACK decoding would have been handed. Padded frames are untouched, for a reason stronger than the measurement: `(A - B) if T else 0` and `A - (B if T else 0)` are both `A - B` for truthy `T`, so only the `else` arm could ever have moved -- both arms are asserted anyway, so that a repair to the unpadded one cannot break the other. The padded arm also has no off-by-one, which #668 asked be checked rather than assumed: `Schema.unpack` decrements `packet['__length__']` by each field's width as it goes, so `pad_len`'s own octet is already out of it by the time the payload field runs, and a padded frame declaring `1 + len(data) + pad_len` arrives at the payload with `__length__ == len(data) + pad_len`. The three sibling sites that were never wrong are what localise the defect to the grouping rather than to the field machinery: `ContinuationFrame.fragment`, `UnassignedFrame.data` and `GoawayFrame.debug` use the plain `length=lambda pkt: pkt['__length__']` with no conditional, share the same `BytesField`, the same `__length__` bookkeeping and the same frame dispatch, and carried their payload correctly throughout -- they are pinned as controls rather than left as an argument. Why nothing caught it: `test_option_roundtrip_unit` does drive every frame type, but through `make` -> parse -> `make`, and `make` writes the length field from `_make_http_length`, so both sides agreed on an empty payload and the octets matched; its generator passes no payload argument for these three frames at all, measured as `kwargs={}`, so what it round-tripped was `b''` and there was nothing to lose, while `test_httpv2_frame_readers_cover_successful_frames` drives the readers from hand-built schema stubs, so the callback never ran there either. The new `tests/protocols/application/test_httpv2_payload_length_unit.py` reads wire octets, which is the gap: 23 tests and 12 subtests over the padded, unpadded and padding-only shapes of all three frames plus both `PRIORITY` combinations, asserting the payload *octets* and never its length, because a length assertion survives a read of the right width from the wrong offset. Its padded cases pad with a non-zero `b'\xde\xad\xbe\xef'` rather than the zeros [RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1) tells a sender to use -- pcapkit does not police padding content -- so that failing to subtract the padding yields a different byte string instead of a coincidentally equal count. The same lambda also governs **packing**, which neither the issue nor the first revision of the fix noticed and a cross-review on a second model did: `BytesField` consults its `length` callback on both paths, so the `else` arm did not merely discard a payload on read, it declined to write one. Measured pre-fix, an unpadded `DATA` frame carrying twelve octets of body packed to `000015000000000001` -- nine octets of header whose length field declares **21**, with the body absent -- and unpadded `HEADERS` and `PUSH_PROMISE` likewise declared 29 against 9 and 33 against 13. A frame overstating its own length by its whole payload desynchronises any reader that walks a stream by that field, so this was pcapkit *emitting* malformed HTTP/2 rather than only mis-reading it, and `HTTPv2ConstructedFrameDeclaresWhatItWritesUnitTests` pins that half; a `make` -> parse -> `make` cycle of a non-empty unpadded payload now closes byte-for-byte where it could not before. Six wrong fixes were written over the schema to check the tests discriminate, and each is rejected: dropping the conditional outright (caught by `packet length < 0: -4`), subtracting `pad_len + 1` (caught as `b'{"ok":true}'` against `b'{"ok":true}\n'`), subtracting the padding on the arm where it is absent, fixing `DataFrame` while forgetting the other two, and `max(computed, 1)` on the two `fragment` fields. That last one is the cross-review's own construction and it **passed** the first revision's 15 tests and 9 subtests, because every fixture then carried a non-empty payload and only `DataFrame` had a "frame of only padding" case, so those two callbacks were never once asked for `0`; under it a padding-only `HEADERS` frame returned `fragment=b'\xde'`, one padding octet leaked in, alongside a swallowed `SchemaWarning: packet length < 0: -1`. The three missing padding-only cases and a callback-level `test_the_padded_arm_reaches_zero_and_is_not_clamped` close it -- `0` is a legitimate answer from these callbacks and has to be asserted as one. Seven tests and three subtests fail before the change and all pass after; 145 tests and 550 subtests pass across `tests/protocols/application/`, the round-trip module and the four other HTTP/2-touching modules. `EXPECTED_FAILURES` is unmoved, its one HTTP/2 entry `httpv2-frame/PRIORITY` still failing with the same `CONSTRUCT` status and the same detail -- `PriorityFrame` has neither a payload field nor a `PADDED` flag, so this change cannot reach it -- and none of the 44 entries was deleted; that entry's own cited `pcapkit/protocols/application/httpv2.py:572` is stale, the `header.length != 9` guard it describes now being at `:562`. The schema module was already at 100% coverage and stays there, 101 statements and 4 branches either side with nothing missing or partial, because the changed lines did execute before -- just with an empty payload -- so the number that moved is the suite: 73 to 96 tests and 399 to 411 subtests over the same targets. The construct side needed no code change, `_make_http_length` having always computed the DATA payload as `len(frame.data) + (pad_len + 1 if pad_len else 0)`, i.e. assuming `data` holds the whole payload whether or not the frame is padded -- it was the shared length callback that disagreed, on both paths at once. An AST sweep of all 496 files of the package found one other site with the same syntactic shape, `pcapkit/protocols/schema/internet/ipv4.py:336` (`TSOption.remainder`), and it is correct rather than the same defect: its `else 0` is intentional because the sibling `ts_data` field consumes the entire option data area in that arm, the two summing to `length - 4` across all 2106 `(length, pointer, flag)` combinations, which is exactly what was not true here. Ships labelled breaking on both counts: restoring a dropped payload moves the parse output of essentially every HTTP/2 capture on the success path, so anything holding a golden file or a regression baseline that recorded the empty value sees it change -- and constructed frames change byte-for-byte too, from malformed to correct (#668). - **Fixed** -- the schema half of every option-like registrar overwrote silently while the parser half of the very same call warned. `EnumSchema.register` was a bare `cls.__enum__[code] = schema`, and it is the schema half of 14 public registrars, so one `register_ipv4_option` call replacing a built-in named the parser it displaced and said nothing about the schema. `EnumSchema.__init_subclass__` reaches the same registry without calling `register`, so a subclass declared with a `code=` keyword -- the documented way to add a schema -- stayed silent too; both are guarded now, and the declaration hook's two branches are folded into one loop so the guard is written once. Presence is a faithful test there only because `_EnumRegistry.__missing__` returns a miss without recording it, so parsing one packet carrying an unknown code cannot make the next legitimate registration for that code warn about an entry no caller ever asked for. The seven code-keyed parser registrars -- on `ProtocolBase`, `Link`, `Internet`, `Frame`, `PCAPNG`, `Transport` and `SCTP` -- now name the displaced entry and its replacement, where before they reported only that something had been overwritten and never which class it was. Their presence-only firing condition is deliberately unchanged, and is not harmonised onto the narrower guard #681 gave `register_protocol`: that one is keyed on a name derived from the value it stores, whereas these seven take a `code` the caller supplies independently of the value, so one class registered under two codes yields two keys and never reaches one key twice. These tables also ship pre-seeded with unresolved `ModuleDescriptor` values, so an incumbent may be a two-string descriptor while the replacement is the very class it names, which leaves a different-class test undecidable without resolving the descriptor and forcing the import it exists to defer. `ContextRegistry.register` already raises on a duplicate, and the reassembly and ESP registrars append to lists with no key at all -- duplicate security-association SPIs being a designed feature there, resolved by scoring rather than by replacement -- so none of those three takes a guard. `import pcapkit` holds at one warning, a third-party deprecation, and no `RegistryWarning`, measured over the 326 registry writes it performs, none of which lands on a key already present; `tests/foundation/registry/` goes 13 to 14 tests and 87 to 95 subtests, the schema module holds 99% coverage with its 5 new statements covered and its misses flat at 1, mypy and pylint are unmoved at 112 errors and 364 messages, and `EXPECTED_FAILURES` is unmoved at 44 entries (#692). +- **Fixed** -- every EOF-truncated PCAP-NG file raised a bare `ValueError` out of `Extractor`, so the whole extraction was lost rather than the one truncated block. The root is one layer above the reported site: Block Total Length is cross-checked against its own trailing copy and never against the file, so a last block declaring more than the file holds left `PCAPNG.read` seeking *past* the real end -- legal and silent -- and the next block read then measured a **negative** remainder, since `prepare` derives it as the end of the stream less the current position. That negative reached `SchemaField` through `pcapng_block_selector` and `io.RawIOBase.read` refused it, which is why nearly every truncation level failed rather than only the one holding the cut. The seek is now clamped to the octets `_read_fileng` actually returned, with a `ProtocolWarning` naming the overrun, and a tail under the twelve octets a block needs at minimum is reported as the quiet `StreamEOFError` that `Extractor.record_frames` already catches -- the one case that is deliberately not a clamp, because at the end of the file no block is being read and clamping would fabricate a frame out of zero padding. A second, independent route to the same shape is closed with a new `nonnegative()` helper, composed into `bounded_option` and `bounded_area` and applied to the eight unwrapped `length - N` spans and the eight `__option_padding__`-sized padding fields: `_TextField.__call__` builds its `struct` template as `f'{length}s'` unconditionally, so a negative became the format `'-8s'` and `struct.calcsize` raised -- neither that nor the `ValueError` being one of `pcapkit.utilities.exceptions`, and neither being an `EOFError`, so neither was caught where the frame loop catches the end of a file. 28 of the module's 74 length callbacks returned a negative before and none do now, from `-1` on an `epb_hash` declaring no payload to `-16777248` on an Enhanced Packet Block's option area; the 200-block, 8,048-octet `captured_len = 0xFFFFFF` vector that reached `struct` through `bounded_area`'s own `nominal <= available` test being true for a negative nominal now parses to 200 frames. Measured over all 1,509 octet boundaries of `examples/captures/dhcp.pcapng` with one harness either side: 6 levels parsed before and 1,495 do now, against 1,479 `ValueError` and 10 `struct.error` before and none from outside the library after, with the frame count degrading monotonically with the cut and the 13 levels that still raise all cutting into the Section Header Block, where no frame was left to lose. The five deliberately unclamped non-packet option areas keep the per-block framing assumption #676's note describes, which this does not close. Also here because it is the fifth and last registrar in the package with no overwrite guard at all: `Option.register` now reports a displaced option schema as a `RegistryWarning` naming every namespace the `ns='opt'` fan-out displaced something in, once per registration rather than once per namespace, tested by membership rather than by subscripting because the per-namespace registries are plain `defaultdict`\ s and reading one to look would insert `UnknownOption` for a code nobody registered. Presence alone is the firing condition, as on the seven code-keyed parser registrars and unlike the narrower guard #681 gave `register_protocol`, since this key is a caller-supplied `code` that `__init_subclass__` passes once per subclass; a namespace the call itself creates is exempt, because it starts as a copy of `opt`'s defaults and nothing in it is a prior registration. One draft did move `EXPECTED_FAILURES`, and it is worth recording: flooring the three decryption-secrets payloads that read `__length__` *whole* rather than subtracting from it packs nothing, because `Schema.pack` leaves that key at `-1` for "unknown" -- which emptied both payloads and turned `pcapng-secrets/TLS_Key_Log` and `.../WireGuard_Key_Log` from `MISMATCH` to `OK`, an empty payload comparing equal to an empty payload. Reverted and pinned; the entry count holds at 44 with its 35 PCAP-NG cases unmoved. Ships labelled breaking: well-formed captures are byte-identical, verified by regenerating `examples/captures/pcapng.txt` either side of the change, but any truncated PCAP-NG now yields the frames before the cut where it previously raised -- so a caller reading "extraction raised" as "this file is unusable" gets a partial result instead, whose last frame may carry zero-padded octets -- and three exception classes change at the margins, eight short-file depths moving from `ProtocolError: unknown byteorder magic` to `StreamEOFError`. The two modules hold 99.93% coverage with their 25 new statements covered and their single miss flat, and `examples/captures/pcapng.txt` does not move further, leaving the #683 drift #685 tracks exactly as it was (#678). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 9523bc507..8f9ca86e9 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2129,6 +2129,66 @@ pull requests between #326 and #509. with its 5 new statements covered and its misses flat at 1, mypy and pylint are unmoved at 112 errors and 364 messages, and ``EXPECTED_FAILURES`` is unmoved at 44 entries (#692). +* **Fixed** -- every EOF-truncated PCAP-NG file raised a bare ``ValueError`` out of + ``Extractor``, so the whole extraction was lost rather than the one truncated block. + The root is one layer above the reported site: Block Total Length is cross-checked + against its own trailing copy and never against the file, so a last block declaring + more than the file holds left ``PCAPNG.read`` seeking *past* the real end -- legal and + silent -- and the next block read then measured a **negative** remainder, since + ``prepare`` derives it as the end of the stream less the current position. That + negative reached ``SchemaField`` through ``pcapng_block_selector`` and + ``io.RawIOBase.read`` refused it, which is why nearly every truncation level failed + rather than only the one holding the cut. The seek is now clamped to the octets + ``_read_fileng`` actually returned, with a ``ProtocolWarning`` naming the overrun, and + a tail under the twelve octets a block needs at minimum is reported as the quiet + ``StreamEOFError`` that ``Extractor.record_frames`` already catches -- the one case + that is deliberately not a clamp, because at the end of the file no block is being + read and clamping would fabricate a frame out of zero padding. A second, independent + route to the same shape is closed with a new ``nonnegative()`` helper, composed into + ``bounded_option`` and ``bounded_area`` and applied to the eight unwrapped + ``length - N`` spans and the eight ``__option_padding__``-sized padding fields: + ``_TextField.__call__`` builds its ``struct`` template as ``f'{length}s'`` + unconditionally, so a negative became the format ``'-8s'`` and ``struct.calcsize`` + raised -- neither that nor the ``ValueError`` being one of + ``pcapkit.utilities.exceptions``, and neither being an ``EOFError``, so neither was + caught where the frame loop catches the end of a file. 28 of the module's 74 length + callbacks returned a negative before and none do now, from ``-1`` on an ``epb_hash`` + declaring no payload to ``-16777248`` on an Enhanced Packet Block's option area; the + 200-block, 8,048-octet ``captured_len = 0xFFFFFF`` vector that reached ``struct`` + through ``bounded_area``'s own ``nominal <= available`` test being true for a negative + nominal now parses to 200 frames. Measured over all 1,509 octet boundaries of + ``examples/captures/dhcp.pcapng`` with one harness either side: 6 levels parsed before + and 1,495 do now, against 1,479 ``ValueError`` and 10 ``struct.error`` before and none + from outside the library after, with the frame count degrading monotonically with the + cut and the 13 levels that still raise all cutting into the Section Header Block, where + no frame was left to lose. The five deliberately unclamped non-packet option areas keep + the per-block framing assumption #676's note describes, which this does not close. Also + here because it is the fifth and last registrar in the package with no overwrite guard + at all: ``Option.register`` now reports a displaced option schema as a + ``RegistryWarning`` naming every namespace the ``ns='opt'`` fan-out displaced something + in, once per registration rather than once per namespace, tested by membership rather + than by subscripting because the per-namespace registries are plain ``defaultdict``\ s + and reading one to look would insert ``UnknownOption`` for a code nobody registered. + Presence alone is the firing condition, as on the seven code-keyed parser registrars and + unlike the narrower guard #681 gave ``register_protocol``, since this key is a + caller-supplied ``code`` that ``__init_subclass__`` passes once per subclass; a + namespace the call itself creates is exempt, because it starts as a copy of ``opt``'s + defaults and nothing in it is a prior registration. One draft did move + ``EXPECTED_FAILURES``, and it is worth recording: flooring the three decryption-secrets + payloads that read ``__length__`` *whole* rather than subtracting from it packs nothing, + because ``Schema.pack`` leaves that key at ``-1`` for "unknown" -- which emptied both + payloads and turned ``pcapng-secrets/TLS_Key_Log`` and ``.../WireGuard_Key_Log`` from + ``MISMATCH`` to ``OK``, an empty payload comparing equal to an empty payload. Reverted + and pinned; the entry count holds at 44 with its 35 PCAP-NG cases unmoved. Ships + labelled breaking: well-formed captures are byte-identical, verified by regenerating + ``examples/captures/pcapng.txt`` either side of the change, but any truncated PCAP-NG + now yields the frames before the cut where it previously raised -- so a caller reading + "extraction raised" as "this file is unusable" gets a partial result instead, whose last + frame may carry zero-padded octets -- and three exception classes change at the margins, + eight short-file depths moving from ``ProtocolError: unknown byteorder magic`` to + ``StreamEOFError``. The two modules hold 99.93% coverage with their 25 new statements + covered and their single miss flat, and ``examples/captures/pcapng.txt`` does not move + further, leaving the #683 drift #685 tracks exactly as it was (#678). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 6e48a2decb4249139a74b73a63efe50fbf011e41 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 22 Sep 2026 23:55:41 -0400 Subject: [PATCH 28/46] docs(changelog): correct the #678 entry's residual-failure count, 13 to 14 The sweep leaves 1,495 of 1,509 levels parsing, so 14 still raise, not 13: 8 StreamEOFError, 4 FormatError and 2 ProtocolError. The per-type figures in the same sentence already summed to 14; only the total was wrong. --- CHANGELOG.md | 2 +- docs/source/changelog/1.5.0.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3a4ef7085..0b3cef9c6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -107,7 +107,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- every PCAP-NG packet block now carries the captured octets it declares. All four Enhanced Packet Blocks of the committed `examples/captures/dhcp.pcapng` reported `len(packet) == 0` against a `captured_len` of 314, 342, 314 and 342, and the Simple Packet Block and the obsolete Packet Block did the same, measured on a capture synthesised to carry one of each since no committed fixture has either. Those three -- exactly `PCAPNG.PACKET_TYPES`, and exactly the three schemas declaring `__payload__ = 'packet_data'` -- are the whole of the blast radius; every other block type carries no captured octets, reported `b''` before and still does. The octets were read correctly and then thrown away: `PCAPNG.unpack` extracted them from the block schema, and `ProtocolBase.__init__` overwrote the result with `self.packet.payload`, which was empty. The inherited `packet` splits a protocol at `self.length` octets of header and takes everything after as the payload, a contract that holds for a protocol laid out as a header followed by its payload and for nothing else. `PCAPNG.length` returns the wire's *Block Total Length*, so the split consumed the entire per-block buffer as header -- measured at `frame.length == 348` and `len(frame.packet.header) == 348` on a 348-octet block -- and, separately, a PCAP-NG block carries a *trailer* after its payload, the option list and a repeat of the total length, so no value of `length` could have made that split right. `captured_len` comes straight off the block header and survived, which is why the two disagreed rather than both being empty. The fix is at the computation rather than at the injection: `PCAPNG.packet` is overridden to take the payload from the block schema's `__payload__` field and the header from the octets ahead of it, summed out of the schema buffers so that the three block types' three different payload offsets -- 28, 12 and 28 octets -- are not hard-coded. Teaching `ProtocolBase.__init__` to skip a protocol that had already set `packet` was the smaller change and was rejected: it heals `frame.info.packet` while leaving the public `frame.packet` still reporting the whole block as header and `b''` as payload, which is the same defect from the other side. `PCAPNG.unpack` now reads that property instead of extracting the payload a second time of its own, leaving one source of truth where there were two attempts at one. The property is a plain `property` rather than the `cached_property` it overrides, which a cross-review on a second model is what settled: the inherited one caches because it *reads the stream*, where a second read would consume octets that are gone, while this one only walks buffers the schema layer has already filled and so has nothing to amortise -- and caching it would have reintroduced the same staleness by another route, a second `unpack` on one instance handing back the *first* call's octets with `get_payload` never reached, an invariant the code held before this change and has no reason to stop holding. Nothing calls `unpack` twice on one instance today, `__post_init__` being its only caller, so the unit test asserts the property directly: swap the block on a live instance, unpack again, and the payload must be the new block's, which fails `b'payload' != b'cached'` under a cache. `ProtocolBase.packet` gains the contract in its docstring and not one executable line, its 503 statements unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on the success path, which is why it ships labelled breaking: `frame.info.packet` goes from `b''` to hundreds of octets, `frame.packet.header` from the whole block to the pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what made this a wire-format defect rather than an API wart. `PCAPIO` writes `value.packet` after each 16-octet record header, so dumping those four blocks produced a **104-octet** file -- 24 of global header plus four record headers and no payload at all -- in which every record header declared hundreds of octets and delivered none, desynchronising any reader that walks by `incl_len`, which is all of them: pcapkit refused its own output with `ValueError: read length must be non-negative or -1` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes rather than against a length. `tests/protocols/test_pcapng_regression.py` grows 4 tests to 11 and 3 subtests to 24, with the payload expectations derived twice over -- spelled-out head and tail literals, and a `struct`-only re-parse of the fixture that owes nothing to the code under test -- and each synthesised payload given a different length modulo 4 so that an offset off by a field, or a payload that picked up the block's 32-bit padding, cannot pass by coincidence. Two shapes no fixture exercised are covered there too, both found while writing the tests rather than after: an option area *after* the captured data, which every block of `dhcp.pcapng` lacks and which is exactly what an offset walked from the wrong end would swallow, and a big-endian section, whose fields differ even though the payload offsets do not. A snapped block is covered in both the shapes that express it -- an Enhanced Packet Block declaring `captured_len` below `original_len`, and a Simple Packet Block whose captured length is bounded by the interface's `snaplen` -- neither of which may come back padded out to the on-wire length. The offsets hold at 28, 12 and 28 across all of it, which is the property the walk has to have. With both trees running the same -- current -- tests, which is the only comparison that isolates the code from the suite, `pcapkit/protocols/misc/pcapng.py` holds 99.91% across the change, its 18 new statements and 6 new branches all executed, its miss count flat at 1 and its partial-branch count flat at 1, that single miss being the same pre-existing `_get_timezone` statement renumbered 1153 to 1265; `pcapkit/protocols/protocol.py` is identical in every column, 503 statements and 228 misses either side, which is the check that its change really is docstring-only. The uncaught `ValueError` that an EOF-truncated PCAP-NG raises (#678) is untouched and was measured rather than assumed, over a truncation set stated rather than described because "101 levels" admits several readings that do not agree on the counts: the whole-percent prefixes `max(1, 1508 * i // 100) for i in range(101)`, which for this file are 101 distinct lengths from 1 to 1508. Both trees give 96 of that `ValueError`, one `FormatError`, one `ProtocolError` and three parses, and -- compared level by level rather than only in aggregate, since a matching total can hide two levels that swapped -- the SHA-256 of the sorted length-to-outcome mapping is `ca44d3ee658087cf2a667c454f839dd931c1c04790b2fe32cee8e1a2e861b182` on each. `examples/captures/pcapng.txt`, a hand-regenerated legacy smoke reference, still shows the old frame-level `packet -> NIL` and wants a separate refresh (#646). - **Fixed** -- an unpadded HTTP/2 `DATA`, `HEADERS` or `PUSH_PROMISE` frame parsed with its whole payload silently discarded. Three payload length callbacks in `pcapkit/protocols/schema/application/httpv2.py` put the conditional expression in the wrong place: a conditional binds looser than `-`, so `pkt['__length__'] - pkt['pad_len'] if pkt['flags']['bit_3'] else 0` groups as `(pkt['__length__'] - pkt['pad_len']) if ... else 0` and the `else` arm returned `0` -- "read no octets at all" to `BytesField` -- where it was meant to subtract `0`. Since `__length__` is the *remaining* declared length at the field, the unpadded arm wants `__length__` itself, which is what subtracting a zero padding length gives. The grouping was read off the AST rather than by eye, the top-level node being the `IfExp` whose `orelse` is a bare `Constant(0)`, which is also what showed that the outer parentheses on the `HeadersFrame` and `PushPromiseFrame` forms were line-continuation only and changed nothing. Padding is rare in HTTP/2, so the broken arm was the common one rather than the edge case, and nothing raised, warned or logged: `info.data` was simply `b''`. Measured on wire octets through the public `HTTP(io.BytesIO(raw), len(raw))` path, an unpadded `DATA` frame declaring `b'{"ok":true}\n'` parsed as `b''` and now parses as those twelve octets; unpadded `HEADERS` and `PUSH_PROMISE` lost and now keep the RFC 7541 appendix C.4.1 header block fragment `b'\x82\x86\x84A\x0fwww.example.com'`, which is precisely what HPACK decoding would have been handed. Padded frames are untouched, for a reason stronger than the measurement: `(A - B) if T else 0` and `A - (B if T else 0)` are both `A - B` for truthy `T`, so only the `else` arm could ever have moved -- both arms are asserted anyway, so that a repair to the unpadded one cannot break the other. The padded arm also has no off-by-one, which #668 asked be checked rather than assumed: `Schema.unpack` decrements `packet['__length__']` by each field's width as it goes, so `pad_len`'s own octet is already out of it by the time the payload field runs, and a padded frame declaring `1 + len(data) + pad_len` arrives at the payload with `__length__ == len(data) + pad_len`. The three sibling sites that were never wrong are what localise the defect to the grouping rather than to the field machinery: `ContinuationFrame.fragment`, `UnassignedFrame.data` and `GoawayFrame.debug` use the plain `length=lambda pkt: pkt['__length__']` with no conditional, share the same `BytesField`, the same `__length__` bookkeeping and the same frame dispatch, and carried their payload correctly throughout -- they are pinned as controls rather than left as an argument. Why nothing caught it: `test_option_roundtrip_unit` does drive every frame type, but through `make` -> parse -> `make`, and `make` writes the length field from `_make_http_length`, so both sides agreed on an empty payload and the octets matched; its generator passes no payload argument for these three frames at all, measured as `kwargs={}`, so what it round-tripped was `b''` and there was nothing to lose, while `test_httpv2_frame_readers_cover_successful_frames` drives the readers from hand-built schema stubs, so the callback never ran there either. The new `tests/protocols/application/test_httpv2_payload_length_unit.py` reads wire octets, which is the gap: 23 tests and 12 subtests over the padded, unpadded and padding-only shapes of all three frames plus both `PRIORITY` combinations, asserting the payload *octets* and never its length, because a length assertion survives a read of the right width from the wrong offset. Its padded cases pad with a non-zero `b'\xde\xad\xbe\xef'` rather than the zeros [RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1) tells a sender to use -- pcapkit does not police padding content -- so that failing to subtract the padding yields a different byte string instead of a coincidentally equal count. The same lambda also governs **packing**, which neither the issue nor the first revision of the fix noticed and a cross-review on a second model did: `BytesField` consults its `length` callback on both paths, so the `else` arm did not merely discard a payload on read, it declined to write one. Measured pre-fix, an unpadded `DATA` frame carrying twelve octets of body packed to `000015000000000001` -- nine octets of header whose length field declares **21**, with the body absent -- and unpadded `HEADERS` and `PUSH_PROMISE` likewise declared 29 against 9 and 33 against 13. A frame overstating its own length by its whole payload desynchronises any reader that walks a stream by that field, so this was pcapkit *emitting* malformed HTTP/2 rather than only mis-reading it, and `HTTPv2ConstructedFrameDeclaresWhatItWritesUnitTests` pins that half; a `make` -> parse -> `make` cycle of a non-empty unpadded payload now closes byte-for-byte where it could not before. Six wrong fixes were written over the schema to check the tests discriminate, and each is rejected: dropping the conditional outright (caught by `packet length < 0: -4`), subtracting `pad_len + 1` (caught as `b'{"ok":true}'` against `b'{"ok":true}\n'`), subtracting the padding on the arm where it is absent, fixing `DataFrame` while forgetting the other two, and `max(computed, 1)` on the two `fragment` fields. That last one is the cross-review's own construction and it **passed** the first revision's 15 tests and 9 subtests, because every fixture then carried a non-empty payload and only `DataFrame` had a "frame of only padding" case, so those two callbacks were never once asked for `0`; under it a padding-only `HEADERS` frame returned `fragment=b'\xde'`, one padding octet leaked in, alongside a swallowed `SchemaWarning: packet length < 0: -1`. The three missing padding-only cases and a callback-level `test_the_padded_arm_reaches_zero_and_is_not_clamped` close it -- `0` is a legitimate answer from these callbacks and has to be asserted as one. Seven tests and three subtests fail before the change and all pass after; 145 tests and 550 subtests pass across `tests/protocols/application/`, the round-trip module and the four other HTTP/2-touching modules. `EXPECTED_FAILURES` is unmoved, its one HTTP/2 entry `httpv2-frame/PRIORITY` still failing with the same `CONSTRUCT` status and the same detail -- `PriorityFrame` has neither a payload field nor a `PADDED` flag, so this change cannot reach it -- and none of the 44 entries was deleted; that entry's own cited `pcapkit/protocols/application/httpv2.py:572` is stale, the `header.length != 9` guard it describes now being at `:562`. The schema module was already at 100% coverage and stays there, 101 statements and 4 branches either side with nothing missing or partial, because the changed lines did execute before -- just with an empty payload -- so the number that moved is the suite: 73 to 96 tests and 399 to 411 subtests over the same targets. The construct side needed no code change, `_make_http_length` having always computed the DATA payload as `len(frame.data) + (pad_len + 1 if pad_len else 0)`, i.e. assuming `data` holds the whole payload whether or not the frame is padded -- it was the shared length callback that disagreed, on both paths at once. An AST sweep of all 496 files of the package found one other site with the same syntactic shape, `pcapkit/protocols/schema/internet/ipv4.py:336` (`TSOption.remainder`), and it is correct rather than the same defect: its `else 0` is intentional because the sibling `ts_data` field consumes the entire option data area in that arm, the two summing to `length - 4` across all 2106 `(length, pointer, flag)` combinations, which is exactly what was not true here. Ships labelled breaking on both counts: restoring a dropped payload moves the parse output of essentially every HTTP/2 capture on the success path, so anything holding a golden file or a regression baseline that recorded the empty value sees it change -- and constructed frames change byte-for-byte too, from malformed to correct (#668). - **Fixed** -- the schema half of every option-like registrar overwrote silently while the parser half of the very same call warned. `EnumSchema.register` was a bare `cls.__enum__[code] = schema`, and it is the schema half of 14 public registrars, so one `register_ipv4_option` call replacing a built-in named the parser it displaced and said nothing about the schema. `EnumSchema.__init_subclass__` reaches the same registry without calling `register`, so a subclass declared with a `code=` keyword -- the documented way to add a schema -- stayed silent too; both are guarded now, and the declaration hook's two branches are folded into one loop so the guard is written once. Presence is a faithful test there only because `_EnumRegistry.__missing__` returns a miss without recording it, so parsing one packet carrying an unknown code cannot make the next legitimate registration for that code warn about an entry no caller ever asked for. The seven code-keyed parser registrars -- on `ProtocolBase`, `Link`, `Internet`, `Frame`, `PCAPNG`, `Transport` and `SCTP` -- now name the displaced entry and its replacement, where before they reported only that something had been overwritten and never which class it was. Their presence-only firing condition is deliberately unchanged, and is not harmonised onto the narrower guard #681 gave `register_protocol`: that one is keyed on a name derived from the value it stores, whereas these seven take a `code` the caller supplies independently of the value, so one class registered under two codes yields two keys and never reaches one key twice. These tables also ship pre-seeded with unresolved `ModuleDescriptor` values, so an incumbent may be a two-string descriptor while the replacement is the very class it names, which leaves a different-class test undecidable without resolving the descriptor and forcing the import it exists to defer. `ContextRegistry.register` already raises on a duplicate, and the reassembly and ESP registrars append to lists with no key at all -- duplicate security-association SPIs being a designed feature there, resolved by scoring rather than by replacement -- so none of those three takes a guard. `import pcapkit` holds at one warning, a third-party deprecation, and no `RegistryWarning`, measured over the 326 registry writes it performs, none of which lands on a key already present; `tests/foundation/registry/` goes 13 to 14 tests and 87 to 95 subtests, the schema module holds 99% coverage with its 5 new statements covered and its misses flat at 1, mypy and pylint are unmoved at 112 errors and 364 messages, and `EXPECTED_FAILURES` is unmoved at 44 entries (#692). -- **Fixed** -- every EOF-truncated PCAP-NG file raised a bare `ValueError` out of `Extractor`, so the whole extraction was lost rather than the one truncated block. The root is one layer above the reported site: Block Total Length is cross-checked against its own trailing copy and never against the file, so a last block declaring more than the file holds left `PCAPNG.read` seeking *past* the real end -- legal and silent -- and the next block read then measured a **negative** remainder, since `prepare` derives it as the end of the stream less the current position. That negative reached `SchemaField` through `pcapng_block_selector` and `io.RawIOBase.read` refused it, which is why nearly every truncation level failed rather than only the one holding the cut. The seek is now clamped to the octets `_read_fileng` actually returned, with a `ProtocolWarning` naming the overrun, and a tail under the twelve octets a block needs at minimum is reported as the quiet `StreamEOFError` that `Extractor.record_frames` already catches -- the one case that is deliberately not a clamp, because at the end of the file no block is being read and clamping would fabricate a frame out of zero padding. A second, independent route to the same shape is closed with a new `nonnegative()` helper, composed into `bounded_option` and `bounded_area` and applied to the eight unwrapped `length - N` spans and the eight `__option_padding__`-sized padding fields: `_TextField.__call__` builds its `struct` template as `f'{length}s'` unconditionally, so a negative became the format `'-8s'` and `struct.calcsize` raised -- neither that nor the `ValueError` being one of `pcapkit.utilities.exceptions`, and neither being an `EOFError`, so neither was caught where the frame loop catches the end of a file. 28 of the module's 74 length callbacks returned a negative before and none do now, from `-1` on an `epb_hash` declaring no payload to `-16777248` on an Enhanced Packet Block's option area; the 200-block, 8,048-octet `captured_len = 0xFFFFFF` vector that reached `struct` through `bounded_area`'s own `nominal <= available` test being true for a negative nominal now parses to 200 frames. Measured over all 1,509 octet boundaries of `examples/captures/dhcp.pcapng` with one harness either side: 6 levels parsed before and 1,495 do now, against 1,479 `ValueError` and 10 `struct.error` before and none from outside the library after, with the frame count degrading monotonically with the cut and the 13 levels that still raise all cutting into the Section Header Block, where no frame was left to lose. The five deliberately unclamped non-packet option areas keep the per-block framing assumption #676's note describes, which this does not close. Also here because it is the fifth and last registrar in the package with no overwrite guard at all: `Option.register` now reports a displaced option schema as a `RegistryWarning` naming every namespace the `ns='opt'` fan-out displaced something in, once per registration rather than once per namespace, tested by membership rather than by subscripting because the per-namespace registries are plain `defaultdict`\ s and reading one to look would insert `UnknownOption` for a code nobody registered. Presence alone is the firing condition, as on the seven code-keyed parser registrars and unlike the narrower guard #681 gave `register_protocol`, since this key is a caller-supplied `code` that `__init_subclass__` passes once per subclass; a namespace the call itself creates is exempt, because it starts as a copy of `opt`'s defaults and nothing in it is a prior registration. One draft did move `EXPECTED_FAILURES`, and it is worth recording: flooring the three decryption-secrets payloads that read `__length__` *whole* rather than subtracting from it packs nothing, because `Schema.pack` leaves that key at `-1` for "unknown" -- which emptied both payloads and turned `pcapng-secrets/TLS_Key_Log` and `.../WireGuard_Key_Log` from `MISMATCH` to `OK`, an empty payload comparing equal to an empty payload. Reverted and pinned; the entry count holds at 44 with its 35 PCAP-NG cases unmoved. Ships labelled breaking: well-formed captures are byte-identical, verified by regenerating `examples/captures/pcapng.txt` either side of the change, but any truncated PCAP-NG now yields the frames before the cut where it previously raised -- so a caller reading "extraction raised" as "this file is unusable" gets a partial result instead, whose last frame may carry zero-padded octets -- and three exception classes change at the margins, eight short-file depths moving from `ProtocolError: unknown byteorder magic` to `StreamEOFError`. The two modules hold 99.93% coverage with their 25 new statements covered and their single miss flat, and `examples/captures/pcapng.txt` does not move further, leaving the #683 drift #685 tracks exactly as it was (#678). +- **Fixed** -- every EOF-truncated PCAP-NG file raised a bare `ValueError` out of `Extractor`, so the whole extraction was lost rather than the one truncated block. The root is one layer above the reported site: Block Total Length is cross-checked against its own trailing copy and never against the file, so a last block declaring more than the file holds left `PCAPNG.read` seeking *past* the real end -- legal and silent -- and the next block read then measured a **negative** remainder, since `prepare` derives it as the end of the stream less the current position. That negative reached `SchemaField` through `pcapng_block_selector` and `io.RawIOBase.read` refused it, which is why nearly every truncation level failed rather than only the one holding the cut. The seek is now clamped to the octets `_read_fileng` actually returned, with a `ProtocolWarning` naming the overrun, and a tail under the twelve octets a block needs at minimum is reported as the quiet `StreamEOFError` that `Extractor.record_frames` already catches -- the one case that is deliberately not a clamp, because at the end of the file no block is being read and clamping would fabricate a frame out of zero padding. A second, independent route to the same shape is closed with a new `nonnegative()` helper, composed into `bounded_option` and `bounded_area` and applied to the eight unwrapped `length - N` spans and the eight `__option_padding__`-sized padding fields: `_TextField.__call__` builds its `struct` template as `f'{length}s'` unconditionally, so a negative became the format `'-8s'` and `struct.calcsize` raised -- neither that nor the `ValueError` being one of `pcapkit.utilities.exceptions`, and neither being an `EOFError`, so neither was caught where the frame loop catches the end of a file. 28 of the module's 74 length callbacks returned a negative before and none do now, from `-1` on an `epb_hash` declaring no payload to `-16777248` on an Enhanced Packet Block's option area; the 200-block, 8,048-octet `captured_len = 0xFFFFFF` vector that reached `struct` through `bounded_area`'s own `nominal <= available` test being true for a negative nominal now parses to 200 frames. Measured over all 1,509 octet boundaries of `examples/captures/dhcp.pcapng` with one harness either side: 6 levels parsed before and 1,495 do now, against 1,479 `ValueError` and 10 `struct.error` before and none from outside the library after, with the frame count degrading monotonically with the cut and the 14 levels that still raise all cutting into the Section Header Block, where no frame was left to lose. The five deliberately unclamped non-packet option areas keep the per-block framing assumption #676's note describes, which this does not close. Also here because it is the fifth and last registrar in the package with no overwrite guard at all: `Option.register` now reports a displaced option schema as a `RegistryWarning` naming every namespace the `ns='opt'` fan-out displaced something in, once per registration rather than once per namespace, tested by membership rather than by subscripting because the per-namespace registries are plain `defaultdict`\ s and reading one to look would insert `UnknownOption` for a code nobody registered. Presence alone is the firing condition, as on the seven code-keyed parser registrars and unlike the narrower guard #681 gave `register_protocol`, since this key is a caller-supplied `code` that `__init_subclass__` passes once per subclass; a namespace the call itself creates is exempt, because it starts as a copy of `opt`'s defaults and nothing in it is a prior registration. One draft did move `EXPECTED_FAILURES`, and it is worth recording: flooring the three decryption-secrets payloads that read `__length__` *whole* rather than subtracting from it packs nothing, because `Schema.pack` leaves that key at `-1` for "unknown" -- which emptied both payloads and turned `pcapng-secrets/TLS_Key_Log` and `.../WireGuard_Key_Log` from `MISMATCH` to `OK`, an empty payload comparing equal to an empty payload. Reverted and pinned; the entry count holds at 44 with its 35 PCAP-NG cases unmoved. Ships labelled breaking: well-formed captures are byte-identical, verified by regenerating `examples/captures/pcapng.txt` either side of the change, but any truncated PCAP-NG now yields the frames before the cut where it previously raised -- so a caller reading "extraction raised" as "this file is unusable" gets a partial result instead, whose last frame may carry zero-padded octets -- and three exception classes change at the margins, eight short-file depths moving from `ProtocolError: unknown byteorder magic` to `StreamEOFError`. The two modules hold 99.93% coverage with their 25 new statements covered and their single miss flat, and `examples/captures/pcapng.txt` does not move further, leaving the #683 drift #685 tracks exactly as it was (#678). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 8f9ca86e9..e28fc4c5b 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2160,7 +2160,7 @@ pull requests between #326 and #509. ``examples/captures/dhcp.pcapng`` with one harness either side: 6 levels parsed before and 1,495 do now, against 1,479 ``ValueError`` and 10 ``struct.error`` before and none from outside the library after, with the frame count degrading monotonically with the - cut and the 13 levels that still raise all cutting into the Section Header Block, where + cut and the 14 levels that still raise all cutting into the Section Header Block, where no frame was left to lose. The five deliberately unclamped non-packet option areas keep the per-block framing assumption #676's note describes, which this does not close. Also here because it is the fifth and last registrar in the package with no overwrite guard From 248f55038e6e38473f990ad6a83de6ea07b67206 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 00:29:21 -0400 Subject: [PATCH 29/46] docs(changelog): record what the #678 cross-review found, and scope its sweep claim - The journal export block's bare struct.error, which #699 now fixes too: the block's own 32-bit NUL padding was read as a binary field's name, so every entry of unaligned length raised. Reachable from valid input. - The "no foreign exception" claim scoped to the truncation sweep, with the two families a fuzz still reaches named and measured unchanged either side: #701 (an unassigned block type raising from aenum) and #593's 32-bit band. - #704, the silent loss of every journal field after a binary one, noted as filed rather than fixed. - Where the two remaining ProtocolError levels actually cut: the Interface Description Block's if_tsresol option, not the Section Header Block. `python util/changelog_md.py --check` exits 0. --- CHANGELOG.md | 2 +- docs/source/changelog/1.5.0.rst | 24 +++++++++++++++++++++--- 2 files changed, 22 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0b3cef9c6..eee0a7bf3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -107,7 +107,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- every PCAP-NG packet block now carries the captured octets it declares. All four Enhanced Packet Blocks of the committed `examples/captures/dhcp.pcapng` reported `len(packet) == 0` against a `captured_len` of 314, 342, 314 and 342, and the Simple Packet Block and the obsolete Packet Block did the same, measured on a capture synthesised to carry one of each since no committed fixture has either. Those three -- exactly `PCAPNG.PACKET_TYPES`, and exactly the three schemas declaring `__payload__ = 'packet_data'` -- are the whole of the blast radius; every other block type carries no captured octets, reported `b''` before and still does. The octets were read correctly and then thrown away: `PCAPNG.unpack` extracted them from the block schema, and `ProtocolBase.__init__` overwrote the result with `self.packet.payload`, which was empty. The inherited `packet` splits a protocol at `self.length` octets of header and takes everything after as the payload, a contract that holds for a protocol laid out as a header followed by its payload and for nothing else. `PCAPNG.length` returns the wire's *Block Total Length*, so the split consumed the entire per-block buffer as header -- measured at `frame.length == 348` and `len(frame.packet.header) == 348` on a 348-octet block -- and, separately, a PCAP-NG block carries a *trailer* after its payload, the option list and a repeat of the total length, so no value of `length` could have made that split right. `captured_len` comes straight off the block header and survived, which is why the two disagreed rather than both being empty. The fix is at the computation rather than at the injection: `PCAPNG.packet` is overridden to take the payload from the block schema's `__payload__` field and the header from the octets ahead of it, summed out of the schema buffers so that the three block types' three different payload offsets -- 28, 12 and 28 octets -- are not hard-coded. Teaching `ProtocolBase.__init__` to skip a protocol that had already set `packet` was the smaller change and was rejected: it heals `frame.info.packet` while leaving the public `frame.packet` still reporting the whole block as header and `b''` as payload, which is the same defect from the other side. `PCAPNG.unpack` now reads that property instead of extracting the payload a second time of its own, leaving one source of truth where there were two attempts at one. The property is a plain `property` rather than the `cached_property` it overrides, which a cross-review on a second model is what settled: the inherited one caches because it *reads the stream*, where a second read would consume octets that are gone, while this one only walks buffers the schema layer has already filled and so has nothing to amortise -- and caching it would have reintroduced the same staleness by another route, a second `unpack` on one instance handing back the *first* call's octets with `get_payload` never reached, an invariant the code held before this change and has no reason to stop holding. Nothing calls `unpack` twice on one instance today, `__post_init__` being its only caller, so the unit test asserts the property directly: swap the block on a live instance, unpack again, and the payload must be the new block's, which fails `b'payload' != b'cached'` under a cache. `ProtocolBase.packet` gains the contract in its docstring and not one executable line, its 503 statements unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on the success path, which is why it ships labelled breaking: `frame.info.packet` goes from `b''` to hundreds of octets, `frame.packet.header` from the whole block to the pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what made this a wire-format defect rather than an API wart. `PCAPIO` writes `value.packet` after each 16-octet record header, so dumping those four blocks produced a **104-octet** file -- 24 of global header plus four record headers and no payload at all -- in which every record header declared hundreds of octets and delivered none, desynchronising any reader that walks by `incl_len`, which is all of them: pcapkit refused its own output with `ValueError: read length must be non-negative or -1` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes rather than against a length. `tests/protocols/test_pcapng_regression.py` grows 4 tests to 11 and 3 subtests to 24, with the payload expectations derived twice over -- spelled-out head and tail literals, and a `struct`-only re-parse of the fixture that owes nothing to the code under test -- and each synthesised payload given a different length modulo 4 so that an offset off by a field, or a payload that picked up the block's 32-bit padding, cannot pass by coincidence. Two shapes no fixture exercised are covered there too, both found while writing the tests rather than after: an option area *after* the captured data, which every block of `dhcp.pcapng` lacks and which is exactly what an offset walked from the wrong end would swallow, and a big-endian section, whose fields differ even though the payload offsets do not. A snapped block is covered in both the shapes that express it -- an Enhanced Packet Block declaring `captured_len` below `original_len`, and a Simple Packet Block whose captured length is bounded by the interface's `snaplen` -- neither of which may come back padded out to the on-wire length. The offsets hold at 28, 12 and 28 across all of it, which is the property the walk has to have. With both trees running the same -- current -- tests, which is the only comparison that isolates the code from the suite, `pcapkit/protocols/misc/pcapng.py` holds 99.91% across the change, its 18 new statements and 6 new branches all executed, its miss count flat at 1 and its partial-branch count flat at 1, that single miss being the same pre-existing `_get_timezone` statement renumbered 1153 to 1265; `pcapkit/protocols/protocol.py` is identical in every column, 503 statements and 228 misses either side, which is the check that its change really is docstring-only. The uncaught `ValueError` that an EOF-truncated PCAP-NG raises (#678) is untouched and was measured rather than assumed, over a truncation set stated rather than described because "101 levels" admits several readings that do not agree on the counts: the whole-percent prefixes `max(1, 1508 * i // 100) for i in range(101)`, which for this file are 101 distinct lengths from 1 to 1508. Both trees give 96 of that `ValueError`, one `FormatError`, one `ProtocolError` and three parses, and -- compared level by level rather than only in aggregate, since a matching total can hide two levels that swapped -- the SHA-256 of the sorted length-to-outcome mapping is `ca44d3ee658087cf2a667c454f839dd931c1c04790b2fe32cee8e1a2e861b182` on each. `examples/captures/pcapng.txt`, a hand-regenerated legacy smoke reference, still shows the old frame-level `packet -> NIL` and wants a separate refresh (#646). - **Fixed** -- an unpadded HTTP/2 `DATA`, `HEADERS` or `PUSH_PROMISE` frame parsed with its whole payload silently discarded. Three payload length callbacks in `pcapkit/protocols/schema/application/httpv2.py` put the conditional expression in the wrong place: a conditional binds looser than `-`, so `pkt['__length__'] - pkt['pad_len'] if pkt['flags']['bit_3'] else 0` groups as `(pkt['__length__'] - pkt['pad_len']) if ... else 0` and the `else` arm returned `0` -- "read no octets at all" to `BytesField` -- where it was meant to subtract `0`. Since `__length__` is the *remaining* declared length at the field, the unpadded arm wants `__length__` itself, which is what subtracting a zero padding length gives. The grouping was read off the AST rather than by eye, the top-level node being the `IfExp` whose `orelse` is a bare `Constant(0)`, which is also what showed that the outer parentheses on the `HeadersFrame` and `PushPromiseFrame` forms were line-continuation only and changed nothing. Padding is rare in HTTP/2, so the broken arm was the common one rather than the edge case, and nothing raised, warned or logged: `info.data` was simply `b''`. Measured on wire octets through the public `HTTP(io.BytesIO(raw), len(raw))` path, an unpadded `DATA` frame declaring `b'{"ok":true}\n'` parsed as `b''` and now parses as those twelve octets; unpadded `HEADERS` and `PUSH_PROMISE` lost and now keep the RFC 7541 appendix C.4.1 header block fragment `b'\x82\x86\x84A\x0fwww.example.com'`, which is precisely what HPACK decoding would have been handed. Padded frames are untouched, for a reason stronger than the measurement: `(A - B) if T else 0` and `A - (B if T else 0)` are both `A - B` for truthy `T`, so only the `else` arm could ever have moved -- both arms are asserted anyway, so that a repair to the unpadded one cannot break the other. The padded arm also has no off-by-one, which #668 asked be checked rather than assumed: `Schema.unpack` decrements `packet['__length__']` by each field's width as it goes, so `pad_len`'s own octet is already out of it by the time the payload field runs, and a padded frame declaring `1 + len(data) + pad_len` arrives at the payload with `__length__ == len(data) + pad_len`. The three sibling sites that were never wrong are what localise the defect to the grouping rather than to the field machinery: `ContinuationFrame.fragment`, `UnassignedFrame.data` and `GoawayFrame.debug` use the plain `length=lambda pkt: pkt['__length__']` with no conditional, share the same `BytesField`, the same `__length__` bookkeeping and the same frame dispatch, and carried their payload correctly throughout -- they are pinned as controls rather than left as an argument. Why nothing caught it: `test_option_roundtrip_unit` does drive every frame type, but through `make` -> parse -> `make`, and `make` writes the length field from `_make_http_length`, so both sides agreed on an empty payload and the octets matched; its generator passes no payload argument for these three frames at all, measured as `kwargs={}`, so what it round-tripped was `b''` and there was nothing to lose, while `test_httpv2_frame_readers_cover_successful_frames` drives the readers from hand-built schema stubs, so the callback never ran there either. The new `tests/protocols/application/test_httpv2_payload_length_unit.py` reads wire octets, which is the gap: 23 tests and 12 subtests over the padded, unpadded and padding-only shapes of all three frames plus both `PRIORITY` combinations, asserting the payload *octets* and never its length, because a length assertion survives a read of the right width from the wrong offset. Its padded cases pad with a non-zero `b'\xde\xad\xbe\xef'` rather than the zeros [RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1) tells a sender to use -- pcapkit does not police padding content -- so that failing to subtract the padding yields a different byte string instead of a coincidentally equal count. The same lambda also governs **packing**, which neither the issue nor the first revision of the fix noticed and a cross-review on a second model did: `BytesField` consults its `length` callback on both paths, so the `else` arm did not merely discard a payload on read, it declined to write one. Measured pre-fix, an unpadded `DATA` frame carrying twelve octets of body packed to `000015000000000001` -- nine octets of header whose length field declares **21**, with the body absent -- and unpadded `HEADERS` and `PUSH_PROMISE` likewise declared 29 against 9 and 33 against 13. A frame overstating its own length by its whole payload desynchronises any reader that walks a stream by that field, so this was pcapkit *emitting* malformed HTTP/2 rather than only mis-reading it, and `HTTPv2ConstructedFrameDeclaresWhatItWritesUnitTests` pins that half; a `make` -> parse -> `make` cycle of a non-empty unpadded payload now closes byte-for-byte where it could not before. Six wrong fixes were written over the schema to check the tests discriminate, and each is rejected: dropping the conditional outright (caught by `packet length < 0: -4`), subtracting `pad_len + 1` (caught as `b'{"ok":true}'` against `b'{"ok":true}\n'`), subtracting the padding on the arm where it is absent, fixing `DataFrame` while forgetting the other two, and `max(computed, 1)` on the two `fragment` fields. That last one is the cross-review's own construction and it **passed** the first revision's 15 tests and 9 subtests, because every fixture then carried a non-empty payload and only `DataFrame` had a "frame of only padding" case, so those two callbacks were never once asked for `0`; under it a padding-only `HEADERS` frame returned `fragment=b'\xde'`, one padding octet leaked in, alongside a swallowed `SchemaWarning: packet length < 0: -1`. The three missing padding-only cases and a callback-level `test_the_padded_arm_reaches_zero_and_is_not_clamped` close it -- `0` is a legitimate answer from these callbacks and has to be asserted as one. Seven tests and three subtests fail before the change and all pass after; 145 tests and 550 subtests pass across `tests/protocols/application/`, the round-trip module and the four other HTTP/2-touching modules. `EXPECTED_FAILURES` is unmoved, its one HTTP/2 entry `httpv2-frame/PRIORITY` still failing with the same `CONSTRUCT` status and the same detail -- `PriorityFrame` has neither a payload field nor a `PADDED` flag, so this change cannot reach it -- and none of the 44 entries was deleted; that entry's own cited `pcapkit/protocols/application/httpv2.py:572` is stale, the `header.length != 9` guard it describes now being at `:562`. The schema module was already at 100% coverage and stays there, 101 statements and 4 branches either side with nothing missing or partial, because the changed lines did execute before -- just with an empty payload -- so the number that moved is the suite: 73 to 96 tests and 399 to 411 subtests over the same targets. The construct side needed no code change, `_make_http_length` having always computed the DATA payload as `len(frame.data) + (pad_len + 1 if pad_len else 0)`, i.e. assuming `data` holds the whole payload whether or not the frame is padded -- it was the shared length callback that disagreed, on both paths at once. An AST sweep of all 496 files of the package found one other site with the same syntactic shape, `pcapkit/protocols/schema/internet/ipv4.py:336` (`TSOption.remainder`), and it is correct rather than the same defect: its `else 0` is intentional because the sibling `ts_data` field consumes the entire option data area in that arm, the two summing to `length - 4` across all 2106 `(length, pointer, flag)` combinations, which is exactly what was not true here. Ships labelled breaking on both counts: restoring a dropped payload moves the parse output of essentially every HTTP/2 capture on the success path, so anything holding a golden file or a regression baseline that recorded the empty value sees it change -- and constructed frames change byte-for-byte too, from malformed to correct (#668). - **Fixed** -- the schema half of every option-like registrar overwrote silently while the parser half of the very same call warned. `EnumSchema.register` was a bare `cls.__enum__[code] = schema`, and it is the schema half of 14 public registrars, so one `register_ipv4_option` call replacing a built-in named the parser it displaced and said nothing about the schema. `EnumSchema.__init_subclass__` reaches the same registry without calling `register`, so a subclass declared with a `code=` keyword -- the documented way to add a schema -- stayed silent too; both are guarded now, and the declaration hook's two branches are folded into one loop so the guard is written once. Presence is a faithful test there only because `_EnumRegistry.__missing__` returns a miss without recording it, so parsing one packet carrying an unknown code cannot make the next legitimate registration for that code warn about an entry no caller ever asked for. The seven code-keyed parser registrars -- on `ProtocolBase`, `Link`, `Internet`, `Frame`, `PCAPNG`, `Transport` and `SCTP` -- now name the displaced entry and its replacement, where before they reported only that something had been overwritten and never which class it was. Their presence-only firing condition is deliberately unchanged, and is not harmonised onto the narrower guard #681 gave `register_protocol`: that one is keyed on a name derived from the value it stores, whereas these seven take a `code` the caller supplies independently of the value, so one class registered under two codes yields two keys and never reaches one key twice. These tables also ship pre-seeded with unresolved `ModuleDescriptor` values, so an incumbent may be a two-string descriptor while the replacement is the very class it names, which leaves a different-class test undecidable without resolving the descriptor and forcing the import it exists to defer. `ContextRegistry.register` already raises on a duplicate, and the reassembly and ESP registrars append to lists with no key at all -- duplicate security-association SPIs being a designed feature there, resolved by scoring rather than by replacement -- so none of those three takes a guard. `import pcapkit` holds at one warning, a third-party deprecation, and no `RegistryWarning`, measured over the 326 registry writes it performs, none of which lands on a key already present; `tests/foundation/registry/` goes 13 to 14 tests and 87 to 95 subtests, the schema module holds 99% coverage with its 5 new statements covered and its misses flat at 1, mypy and pylint are unmoved at 112 errors and 364 messages, and `EXPECTED_FAILURES` is unmoved at 44 entries (#692). -- **Fixed** -- every EOF-truncated PCAP-NG file raised a bare `ValueError` out of `Extractor`, so the whole extraction was lost rather than the one truncated block. The root is one layer above the reported site: Block Total Length is cross-checked against its own trailing copy and never against the file, so a last block declaring more than the file holds left `PCAPNG.read` seeking *past* the real end -- legal and silent -- and the next block read then measured a **negative** remainder, since `prepare` derives it as the end of the stream less the current position. That negative reached `SchemaField` through `pcapng_block_selector` and `io.RawIOBase.read` refused it, which is why nearly every truncation level failed rather than only the one holding the cut. The seek is now clamped to the octets `_read_fileng` actually returned, with a `ProtocolWarning` naming the overrun, and a tail under the twelve octets a block needs at minimum is reported as the quiet `StreamEOFError` that `Extractor.record_frames` already catches -- the one case that is deliberately not a clamp, because at the end of the file no block is being read and clamping would fabricate a frame out of zero padding. A second, independent route to the same shape is closed with a new `nonnegative()` helper, composed into `bounded_option` and `bounded_area` and applied to the eight unwrapped `length - N` spans and the eight `__option_padding__`-sized padding fields: `_TextField.__call__` builds its `struct` template as `f'{length}s'` unconditionally, so a negative became the format `'-8s'` and `struct.calcsize` raised -- neither that nor the `ValueError` being one of `pcapkit.utilities.exceptions`, and neither being an `EOFError`, so neither was caught where the frame loop catches the end of a file. 28 of the module's 74 length callbacks returned a negative before and none do now, from `-1` on an `epb_hash` declaring no payload to `-16777248` on an Enhanced Packet Block's option area; the 200-block, 8,048-octet `captured_len = 0xFFFFFF` vector that reached `struct` through `bounded_area`'s own `nominal <= available` test being true for a negative nominal now parses to 200 frames. Measured over all 1,509 octet boundaries of `examples/captures/dhcp.pcapng` with one harness either side: 6 levels parsed before and 1,495 do now, against 1,479 `ValueError` and 10 `struct.error` before and none from outside the library after, with the frame count degrading monotonically with the cut and the 14 levels that still raise all cutting into the Section Header Block, where no frame was left to lose. The five deliberately unclamped non-packet option areas keep the per-block framing assumption #676's note describes, which this does not close. Also here because it is the fifth and last registrar in the package with no overwrite guard at all: `Option.register` now reports a displaced option schema as a `RegistryWarning` naming every namespace the `ns='opt'` fan-out displaced something in, once per registration rather than once per namespace, tested by membership rather than by subscripting because the per-namespace registries are plain `defaultdict`\ s and reading one to look would insert `UnknownOption` for a code nobody registered. Presence alone is the firing condition, as on the seven code-keyed parser registrars and unlike the narrower guard #681 gave `register_protocol`, since this key is a caller-supplied `code` that `__init_subclass__` passes once per subclass; a namespace the call itself creates is exempt, because it starts as a copy of `opt`'s defaults and nothing in it is a prior registration. One draft did move `EXPECTED_FAILURES`, and it is worth recording: flooring the three decryption-secrets payloads that read `__length__` *whole* rather than subtracting from it packs nothing, because `Schema.pack` leaves that key at `-1` for "unknown" -- which emptied both payloads and turned `pcapng-secrets/TLS_Key_Log` and `.../WireGuard_Key_Log` from `MISMATCH` to `OK`, an empty payload comparing equal to an empty payload. Reverted and pinned; the entry count holds at 44 with its 35 PCAP-NG cases unmoved. Ships labelled breaking: well-formed captures are byte-identical, verified by regenerating `examples/captures/pcapng.txt` either side of the change, but any truncated PCAP-NG now yields the frames before the cut where it previously raised -- so a caller reading "extraction raised" as "this file is unusable" gets a partial result instead, whose last frame may carry zero-padded octets -- and three exception classes change at the margins, eight short-file depths moving from `ProtocolError: unknown byteorder magic` to `StreamEOFError`. The two modules hold 99.93% coverage with their 25 new statements covered and their single miss flat, and `examples/captures/pcapng.txt` does not move further, leaving the #683 drift #685 tracks exactly as it was (#678). +- **Fixed** -- every EOF-truncated PCAP-NG file raised a bare `ValueError` out of `Extractor`, so the whole extraction was lost rather than the one truncated block. The root is one layer above the reported site: Block Total Length is cross-checked against its own trailing copy and never against the file, so a last block declaring more than the file holds left `PCAPNG.read` seeking *past* the real end -- legal and silent -- and the next block read then measured a **negative** remainder, since `prepare` derives it as the end of the stream less the current position. That negative reached `SchemaField` through `pcapng_block_selector` and `io.RawIOBase.read` refused it, which is why nearly every truncation level failed rather than only the one holding the cut. The seek is now clamped to the octets `_read_fileng` actually returned, with a `ProtocolWarning` naming the overrun, and a tail under the twelve octets a block needs at minimum is reported as the quiet `StreamEOFError` that `Extractor.record_frames` already catches -- the one case that is deliberately not a clamp, because at the end of the file no block is being read and clamping would fabricate a frame out of zero padding. A second, independent route to the same shape is closed with a new `nonnegative()` helper, composed into `bounded_option` and `bounded_area` and applied to the eight unwrapped `length - N` spans and the eight `__option_padding__`-sized padding fields: `_TextField.__call__` builds its `struct` template as `f'{length}s'` unconditionally, so a negative became the format `'-8s'` and `struct.calcsize` raised -- neither that nor the `ValueError` being one of `pcapkit.utilities.exceptions`, and neither being an `EOFError`, so neither was caught where the frame loop catches the end of a file. 28 of the module's 74 length callbacks returned a negative before and none do now, from `-1` on an `epb_hash` declaring no payload to `-16777248` on an Enhanced Packet Block's option area; the 200-block, 8,048-octet `captured_len = 0xFFFFFF` vector that reached `struct` through `bounded_area`'s own `nominal <= available` test being true for a negative nominal now parses to 200 frames. Measured over all 1,509 octet boundaries of `examples/captures/dhcp.pcapng` with one harness either side: 6 levels parsed before and 1,495 do now, against 1,479 `ValueError` and 10 `struct.error` before and none from outside the library after, with the frame count degrading monotonically with the cut; of the 14 levels that still raise, twelve leave a file too short to hold a block at all and the other two cut into the Interface Description Block's `if_tsresol` option, so none of them costs a frame that was in the file. That claim is about the truncation sweep rather than about every input: a 4,000-round bounded mutation fuzz either side of the change, same seed and cap, takes the parse rate from 2,118 to 3,438 and removes both of #678's families, and the two foreign families left are measured unchanged -- `ValueError: N is not a valid BlockType`, filed as #701, where an unassigned block type raises from `aenum` inside `EnumField.post_process` so the `UnknownBlock` default the registry declares is unreachable; and `MemoryError` at exactly 30 of 4,000 in both trees, which is #593's 32-bit band. A third bare exception in this module *is* fixed here, found by the cross-review and reachable from valid input rather than truncated: `SystemdJournalExportBlock.post_process` unpacked a 64-bit binary-field length out of whatever `entry_data.read(8)` returned, and since the block body is padded to a 32-bit boundary with NULs that `bytes.strip()` does not strip, that padding was read as a field *name* with no length prefix behind it -- so every journal entry of unaligned length raised a bare `struct.error`, measured on a 14-octet `MESSAGE=hello` entry. A NUL-only line now ends the entry and a short prefix ends it with a warning; the separate silent loss of every field after a binary one, from a `read()` that reaches EOF where it means to skip one newline, is filed as #704 rather than fixed. The five deliberately unclamped non-packet option areas keep the per-block framing assumption #676's note describes, which this does not close. Also here because it is the fifth and last registrar in the package with no overwrite guard at all: `Option.register` now reports a displaced option schema as a `RegistryWarning` naming every namespace the `ns='opt'` fan-out displaced something in, once per registration rather than once per namespace, tested by membership rather than by subscripting because the per-namespace registries are plain `defaultdict`\ s and reading one to look would insert `UnknownOption` for a code nobody registered. Presence alone is the firing condition, as on the seven code-keyed parser registrars and unlike the narrower guard #681 gave `register_protocol`, since this key is a caller-supplied `code` that `__init_subclass__` passes once per subclass; a namespace the call itself creates is exempt, because it starts as a copy of `opt`'s defaults and nothing in it is a prior registration. One draft did move `EXPECTED_FAILURES`, and it is worth recording: flooring the three decryption-secrets payloads that read `__length__` *whole* rather than subtracting from it packs nothing, because `Schema.pack` leaves that key at `-1` for "unknown" -- which emptied both payloads and turned `pcapng-secrets/TLS_Key_Log` and `.../WireGuard_Key_Log` from `MISMATCH` to `OK`, an empty payload comparing equal to an empty payload. Reverted and pinned; the entry count holds at 44 with its 35 PCAP-NG cases unmoved. Ships labelled breaking: well-formed captures are byte-identical, verified by regenerating `examples/captures/pcapng.txt` either side of the change, but any truncated PCAP-NG now yields the frames before the cut where it previously raised -- so a caller reading "extraction raised" as "this file is unusable" gets a partial result instead, whose last frame may carry zero-padded octets -- and three exception classes change at the margins, eight short-file depths moving from `ProtocolError: unknown byteorder magic` to `StreamEOFError`. The two modules hold 99.93% coverage with their 25 new statements covered and their single miss flat, and `examples/captures/pcapng.txt` does not move further, leaving the #683 drift #685 tracks exactly as it was (#678). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index e28fc4c5b..7ed890eb3 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2160,9 +2160,27 @@ pull requests between #326 and #509. ``examples/captures/dhcp.pcapng`` with one harness either side: 6 levels parsed before and 1,495 do now, against 1,479 ``ValueError`` and 10 ``struct.error`` before and none from outside the library after, with the frame count degrading monotonically with the - cut and the 14 levels that still raise all cutting into the Section Header Block, where - no frame was left to lose. The five deliberately unclamped non-packet option areas keep - the per-block framing assumption #676's note describes, which this does not close. Also + cut; of the 14 levels that still raise, twelve leave a file too short to hold a block at + all and the other two cut into the Interface Description Block's ``if_tsresol`` option, + so none of them costs a frame that was in the file. That claim is about the truncation + sweep rather than about every input: a 4,000-round bounded mutation fuzz either side of + the change, same seed and cap, takes the parse rate from 2,118 to 3,438 and removes both + of #678's families, and the two foreign families left are measured unchanged -- + ``ValueError: N is not a valid BlockType``, filed as #701, where an unassigned block type + raises from ``aenum`` inside ``EnumField.post_process`` so the ``UnknownBlock`` default + the registry declares is unreachable; and ``MemoryError`` at exactly 30 of 4,000 in both + trees, which is #593's 32-bit band. A third bare exception in this module *is* fixed here, + found by the cross-review and reachable from valid input rather than truncated: + ``SystemdJournalExportBlock.post_process`` unpacked a 64-bit binary-field length out of + whatever ``entry_data.read(8)`` returned, and since the block body is padded to a 32-bit + boundary with NULs that ``bytes.strip()`` does not strip, that padding was read as a field + *name* with no length prefix behind it -- so every journal entry of unaligned length raised + a bare ``struct.error``, measured on a 14-octet ``MESSAGE=hello`` entry. A NUL-only line now + ends the entry and a short prefix ends it with a warning; the separate silent loss of every + field after a binary one, from a ``read()`` that reaches EOF where it means to skip one + newline, is filed as #704 rather than fixed. The five deliberately unclamped non-packet + option areas keep the per-block framing assumption #676's note describes, which this does + not close. Also here because it is the fifth and last registrar in the package with no overwrite guard at all: ``Option.register`` now reports a displaced option schema as a ``RegistryWarning`` naming every namespace the ``ns='opt'`` fan-out displaced something From 805d6782fa7c6f93e5c4a8583031f01f2d3efdd5 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 00:56:42 -0400 Subject: [PATCH 30/46] docs(changelog): the journal block's other two bare exceptions, also fixed in #699 The cross-review's second pass found an OverflowError from a binary field's 64-bit length reaching BytesIO.read at 2**63 and above, and a UnicodeDecodeError from a field name, key or value that is not UTF-8. Both pre-existing, both in the function #699 had just fixed the struct.error in, both now clamped or replaced and reported. `python util/changelog_md.py --check` exits 0. --- CHANGELOG.md | 2 +- docs/source/changelog/1.5.0.rst | 17 +++++++++++++---- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index eee0a7bf3..7ca729fe9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -107,7 +107,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- every PCAP-NG packet block now carries the captured octets it declares. All four Enhanced Packet Blocks of the committed `examples/captures/dhcp.pcapng` reported `len(packet) == 0` against a `captured_len` of 314, 342, 314 and 342, and the Simple Packet Block and the obsolete Packet Block did the same, measured on a capture synthesised to carry one of each since no committed fixture has either. Those three -- exactly `PCAPNG.PACKET_TYPES`, and exactly the three schemas declaring `__payload__ = 'packet_data'` -- are the whole of the blast radius; every other block type carries no captured octets, reported `b''` before and still does. The octets were read correctly and then thrown away: `PCAPNG.unpack` extracted them from the block schema, and `ProtocolBase.__init__` overwrote the result with `self.packet.payload`, which was empty. The inherited `packet` splits a protocol at `self.length` octets of header and takes everything after as the payload, a contract that holds for a protocol laid out as a header followed by its payload and for nothing else. `PCAPNG.length` returns the wire's *Block Total Length*, so the split consumed the entire per-block buffer as header -- measured at `frame.length == 348` and `len(frame.packet.header) == 348` on a 348-octet block -- and, separately, a PCAP-NG block carries a *trailer* after its payload, the option list and a repeat of the total length, so no value of `length` could have made that split right. `captured_len` comes straight off the block header and survived, which is why the two disagreed rather than both being empty. The fix is at the computation rather than at the injection: `PCAPNG.packet` is overridden to take the payload from the block schema's `__payload__` field and the header from the octets ahead of it, summed out of the schema buffers so that the three block types' three different payload offsets -- 28, 12 and 28 octets -- are not hard-coded. Teaching `ProtocolBase.__init__` to skip a protocol that had already set `packet` was the smaller change and was rejected: it heals `frame.info.packet` while leaving the public `frame.packet` still reporting the whole block as header and `b''` as payload, which is the same defect from the other side. `PCAPNG.unpack` now reads that property instead of extracting the payload a second time of its own, leaving one source of truth where there were two attempts at one. The property is a plain `property` rather than the `cached_property` it overrides, which a cross-review on a second model is what settled: the inherited one caches because it *reads the stream*, where a second read would consume octets that are gone, while this one only walks buffers the schema layer has already filled and so has nothing to amortise -- and caching it would have reintroduced the same staleness by another route, a second `unpack` on one instance handing back the *first* call's octets with `get_payload` never reached, an invariant the code held before this change and has no reason to stop holding. Nothing calls `unpack` twice on one instance today, `__post_init__` being its only caller, so the unit test asserts the property directly: swap the block on a live instance, unpack again, and the payload must be the new block's, which fails `b'payload' != b'cached'` under a cache. `ProtocolBase.packet` gains the contract in its docstring and not one executable line, its 503 statements unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on the success path, which is why it ships labelled breaking: `frame.info.packet` goes from `b''` to hundreds of octets, `frame.packet.header` from the whole block to the pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what made this a wire-format defect rather than an API wart. `PCAPIO` writes `value.packet` after each 16-octet record header, so dumping those four blocks produced a **104-octet** file -- 24 of global header plus four record headers and no payload at all -- in which every record header declared hundreds of octets and delivered none, desynchronising any reader that walks by `incl_len`, which is all of them: pcapkit refused its own output with `ValueError: read length must be non-negative or -1` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes rather than against a length. `tests/protocols/test_pcapng_regression.py` grows 4 tests to 11 and 3 subtests to 24, with the payload expectations derived twice over -- spelled-out head and tail literals, and a `struct`-only re-parse of the fixture that owes nothing to the code under test -- and each synthesised payload given a different length modulo 4 so that an offset off by a field, or a payload that picked up the block's 32-bit padding, cannot pass by coincidence. Two shapes no fixture exercised are covered there too, both found while writing the tests rather than after: an option area *after* the captured data, which every block of `dhcp.pcapng` lacks and which is exactly what an offset walked from the wrong end would swallow, and a big-endian section, whose fields differ even though the payload offsets do not. A snapped block is covered in both the shapes that express it -- an Enhanced Packet Block declaring `captured_len` below `original_len`, and a Simple Packet Block whose captured length is bounded by the interface's `snaplen` -- neither of which may come back padded out to the on-wire length. The offsets hold at 28, 12 and 28 across all of it, which is the property the walk has to have. With both trees running the same -- current -- tests, which is the only comparison that isolates the code from the suite, `pcapkit/protocols/misc/pcapng.py` holds 99.91% across the change, its 18 new statements and 6 new branches all executed, its miss count flat at 1 and its partial-branch count flat at 1, that single miss being the same pre-existing `_get_timezone` statement renumbered 1153 to 1265; `pcapkit/protocols/protocol.py` is identical in every column, 503 statements and 228 misses either side, which is the check that its change really is docstring-only. The uncaught `ValueError` that an EOF-truncated PCAP-NG raises (#678) is untouched and was measured rather than assumed, over a truncation set stated rather than described because "101 levels" admits several readings that do not agree on the counts: the whole-percent prefixes `max(1, 1508 * i // 100) for i in range(101)`, which for this file are 101 distinct lengths from 1 to 1508. Both trees give 96 of that `ValueError`, one `FormatError`, one `ProtocolError` and three parses, and -- compared level by level rather than only in aggregate, since a matching total can hide two levels that swapped -- the SHA-256 of the sorted length-to-outcome mapping is `ca44d3ee658087cf2a667c454f839dd931c1c04790b2fe32cee8e1a2e861b182` on each. `examples/captures/pcapng.txt`, a hand-regenerated legacy smoke reference, still shows the old frame-level `packet -> NIL` and wants a separate refresh (#646). - **Fixed** -- an unpadded HTTP/2 `DATA`, `HEADERS` or `PUSH_PROMISE` frame parsed with its whole payload silently discarded. Three payload length callbacks in `pcapkit/protocols/schema/application/httpv2.py` put the conditional expression in the wrong place: a conditional binds looser than `-`, so `pkt['__length__'] - pkt['pad_len'] if pkt['flags']['bit_3'] else 0` groups as `(pkt['__length__'] - pkt['pad_len']) if ... else 0` and the `else` arm returned `0` -- "read no octets at all" to `BytesField` -- where it was meant to subtract `0`. Since `__length__` is the *remaining* declared length at the field, the unpadded arm wants `__length__` itself, which is what subtracting a zero padding length gives. The grouping was read off the AST rather than by eye, the top-level node being the `IfExp` whose `orelse` is a bare `Constant(0)`, which is also what showed that the outer parentheses on the `HeadersFrame` and `PushPromiseFrame` forms were line-continuation only and changed nothing. Padding is rare in HTTP/2, so the broken arm was the common one rather than the edge case, and nothing raised, warned or logged: `info.data` was simply `b''`. Measured on wire octets through the public `HTTP(io.BytesIO(raw), len(raw))` path, an unpadded `DATA` frame declaring `b'{"ok":true}\n'` parsed as `b''` and now parses as those twelve octets; unpadded `HEADERS` and `PUSH_PROMISE` lost and now keep the RFC 7541 appendix C.4.1 header block fragment `b'\x82\x86\x84A\x0fwww.example.com'`, which is precisely what HPACK decoding would have been handed. Padded frames are untouched, for a reason stronger than the measurement: `(A - B) if T else 0` and `A - (B if T else 0)` are both `A - B` for truthy `T`, so only the `else` arm could ever have moved -- both arms are asserted anyway, so that a repair to the unpadded one cannot break the other. The padded arm also has no off-by-one, which #668 asked be checked rather than assumed: `Schema.unpack` decrements `packet['__length__']` by each field's width as it goes, so `pad_len`'s own octet is already out of it by the time the payload field runs, and a padded frame declaring `1 + len(data) + pad_len` arrives at the payload with `__length__ == len(data) + pad_len`. The three sibling sites that were never wrong are what localise the defect to the grouping rather than to the field machinery: `ContinuationFrame.fragment`, `UnassignedFrame.data` and `GoawayFrame.debug` use the plain `length=lambda pkt: pkt['__length__']` with no conditional, share the same `BytesField`, the same `__length__` bookkeeping and the same frame dispatch, and carried their payload correctly throughout -- they are pinned as controls rather than left as an argument. Why nothing caught it: `test_option_roundtrip_unit` does drive every frame type, but through `make` -> parse -> `make`, and `make` writes the length field from `_make_http_length`, so both sides agreed on an empty payload and the octets matched; its generator passes no payload argument for these three frames at all, measured as `kwargs={}`, so what it round-tripped was `b''` and there was nothing to lose, while `test_httpv2_frame_readers_cover_successful_frames` drives the readers from hand-built schema stubs, so the callback never ran there either. The new `tests/protocols/application/test_httpv2_payload_length_unit.py` reads wire octets, which is the gap: 23 tests and 12 subtests over the padded, unpadded and padding-only shapes of all three frames plus both `PRIORITY` combinations, asserting the payload *octets* and never its length, because a length assertion survives a read of the right width from the wrong offset. Its padded cases pad with a non-zero `b'\xde\xad\xbe\xef'` rather than the zeros [RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1) tells a sender to use -- pcapkit does not police padding content -- so that failing to subtract the padding yields a different byte string instead of a coincidentally equal count. The same lambda also governs **packing**, which neither the issue nor the first revision of the fix noticed and a cross-review on a second model did: `BytesField` consults its `length` callback on both paths, so the `else` arm did not merely discard a payload on read, it declined to write one. Measured pre-fix, an unpadded `DATA` frame carrying twelve octets of body packed to `000015000000000001` -- nine octets of header whose length field declares **21**, with the body absent -- and unpadded `HEADERS` and `PUSH_PROMISE` likewise declared 29 against 9 and 33 against 13. A frame overstating its own length by its whole payload desynchronises any reader that walks a stream by that field, so this was pcapkit *emitting* malformed HTTP/2 rather than only mis-reading it, and `HTTPv2ConstructedFrameDeclaresWhatItWritesUnitTests` pins that half; a `make` -> parse -> `make` cycle of a non-empty unpadded payload now closes byte-for-byte where it could not before. Six wrong fixes were written over the schema to check the tests discriminate, and each is rejected: dropping the conditional outright (caught by `packet length < 0: -4`), subtracting `pad_len + 1` (caught as `b'{"ok":true}'` against `b'{"ok":true}\n'`), subtracting the padding on the arm where it is absent, fixing `DataFrame` while forgetting the other two, and `max(computed, 1)` on the two `fragment` fields. That last one is the cross-review's own construction and it **passed** the first revision's 15 tests and 9 subtests, because every fixture then carried a non-empty payload and only `DataFrame` had a "frame of only padding" case, so those two callbacks were never once asked for `0`; under it a padding-only `HEADERS` frame returned `fragment=b'\xde'`, one padding octet leaked in, alongside a swallowed `SchemaWarning: packet length < 0: -1`. The three missing padding-only cases and a callback-level `test_the_padded_arm_reaches_zero_and_is_not_clamped` close it -- `0` is a legitimate answer from these callbacks and has to be asserted as one. Seven tests and three subtests fail before the change and all pass after; 145 tests and 550 subtests pass across `tests/protocols/application/`, the round-trip module and the four other HTTP/2-touching modules. `EXPECTED_FAILURES` is unmoved, its one HTTP/2 entry `httpv2-frame/PRIORITY` still failing with the same `CONSTRUCT` status and the same detail -- `PriorityFrame` has neither a payload field nor a `PADDED` flag, so this change cannot reach it -- and none of the 44 entries was deleted; that entry's own cited `pcapkit/protocols/application/httpv2.py:572` is stale, the `header.length != 9` guard it describes now being at `:562`. The schema module was already at 100% coverage and stays there, 101 statements and 4 branches either side with nothing missing or partial, because the changed lines did execute before -- just with an empty payload -- so the number that moved is the suite: 73 to 96 tests and 399 to 411 subtests over the same targets. The construct side needed no code change, `_make_http_length` having always computed the DATA payload as `len(frame.data) + (pad_len + 1 if pad_len else 0)`, i.e. assuming `data` holds the whole payload whether or not the frame is padded -- it was the shared length callback that disagreed, on both paths at once. An AST sweep of all 496 files of the package found one other site with the same syntactic shape, `pcapkit/protocols/schema/internet/ipv4.py:336` (`TSOption.remainder`), and it is correct rather than the same defect: its `else 0` is intentional because the sibling `ts_data` field consumes the entire option data area in that arm, the two summing to `length - 4` across all 2106 `(length, pointer, flag)` combinations, which is exactly what was not true here. Ships labelled breaking on both counts: restoring a dropped payload moves the parse output of essentially every HTTP/2 capture on the success path, so anything holding a golden file or a regression baseline that recorded the empty value sees it change -- and constructed frames change byte-for-byte too, from malformed to correct (#668). - **Fixed** -- the schema half of every option-like registrar overwrote silently while the parser half of the very same call warned. `EnumSchema.register` was a bare `cls.__enum__[code] = schema`, and it is the schema half of 14 public registrars, so one `register_ipv4_option` call replacing a built-in named the parser it displaced and said nothing about the schema. `EnumSchema.__init_subclass__` reaches the same registry without calling `register`, so a subclass declared with a `code=` keyword -- the documented way to add a schema -- stayed silent too; both are guarded now, and the declaration hook's two branches are folded into one loop so the guard is written once. Presence is a faithful test there only because `_EnumRegistry.__missing__` returns a miss without recording it, so parsing one packet carrying an unknown code cannot make the next legitimate registration for that code warn about an entry no caller ever asked for. The seven code-keyed parser registrars -- on `ProtocolBase`, `Link`, `Internet`, `Frame`, `PCAPNG`, `Transport` and `SCTP` -- now name the displaced entry and its replacement, where before they reported only that something had been overwritten and never which class it was. Their presence-only firing condition is deliberately unchanged, and is not harmonised onto the narrower guard #681 gave `register_protocol`: that one is keyed on a name derived from the value it stores, whereas these seven take a `code` the caller supplies independently of the value, so one class registered under two codes yields two keys and never reaches one key twice. These tables also ship pre-seeded with unresolved `ModuleDescriptor` values, so an incumbent may be a two-string descriptor while the replacement is the very class it names, which leaves a different-class test undecidable without resolving the descriptor and forcing the import it exists to defer. `ContextRegistry.register` already raises on a duplicate, and the reassembly and ESP registrars append to lists with no key at all -- duplicate security-association SPIs being a designed feature there, resolved by scoring rather than by replacement -- so none of those three takes a guard. `import pcapkit` holds at one warning, a third-party deprecation, and no `RegistryWarning`, measured over the 326 registry writes it performs, none of which lands on a key already present; `tests/foundation/registry/` goes 13 to 14 tests and 87 to 95 subtests, the schema module holds 99% coverage with its 5 new statements covered and its misses flat at 1, mypy and pylint are unmoved at 112 errors and 364 messages, and `EXPECTED_FAILURES` is unmoved at 44 entries (#692). -- **Fixed** -- every EOF-truncated PCAP-NG file raised a bare `ValueError` out of `Extractor`, so the whole extraction was lost rather than the one truncated block. The root is one layer above the reported site: Block Total Length is cross-checked against its own trailing copy and never against the file, so a last block declaring more than the file holds left `PCAPNG.read` seeking *past* the real end -- legal and silent -- and the next block read then measured a **negative** remainder, since `prepare` derives it as the end of the stream less the current position. That negative reached `SchemaField` through `pcapng_block_selector` and `io.RawIOBase.read` refused it, which is why nearly every truncation level failed rather than only the one holding the cut. The seek is now clamped to the octets `_read_fileng` actually returned, with a `ProtocolWarning` naming the overrun, and a tail under the twelve octets a block needs at minimum is reported as the quiet `StreamEOFError` that `Extractor.record_frames` already catches -- the one case that is deliberately not a clamp, because at the end of the file no block is being read and clamping would fabricate a frame out of zero padding. A second, independent route to the same shape is closed with a new `nonnegative()` helper, composed into `bounded_option` and `bounded_area` and applied to the eight unwrapped `length - N` spans and the eight `__option_padding__`-sized padding fields: `_TextField.__call__` builds its `struct` template as `f'{length}s'` unconditionally, so a negative became the format `'-8s'` and `struct.calcsize` raised -- neither that nor the `ValueError` being one of `pcapkit.utilities.exceptions`, and neither being an `EOFError`, so neither was caught where the frame loop catches the end of a file. 28 of the module's 74 length callbacks returned a negative before and none do now, from `-1` on an `epb_hash` declaring no payload to `-16777248` on an Enhanced Packet Block's option area; the 200-block, 8,048-octet `captured_len = 0xFFFFFF` vector that reached `struct` through `bounded_area`'s own `nominal <= available` test being true for a negative nominal now parses to 200 frames. Measured over all 1,509 octet boundaries of `examples/captures/dhcp.pcapng` with one harness either side: 6 levels parsed before and 1,495 do now, against 1,479 `ValueError` and 10 `struct.error` before and none from outside the library after, with the frame count degrading monotonically with the cut; of the 14 levels that still raise, twelve leave a file too short to hold a block at all and the other two cut into the Interface Description Block's `if_tsresol` option, so none of them costs a frame that was in the file. That claim is about the truncation sweep rather than about every input: a 4,000-round bounded mutation fuzz either side of the change, same seed and cap, takes the parse rate from 2,118 to 3,438 and removes both of #678's families, and the two foreign families left are measured unchanged -- `ValueError: N is not a valid BlockType`, filed as #701, where an unassigned block type raises from `aenum` inside `EnumField.post_process` so the `UnknownBlock` default the registry declares is unreachable; and `MemoryError` at exactly 30 of 4,000 in both trees, which is #593's 32-bit band. A third bare exception in this module *is* fixed here, found by the cross-review and reachable from valid input rather than truncated: `SystemdJournalExportBlock.post_process` unpacked a 64-bit binary-field length out of whatever `entry_data.read(8)` returned, and since the block body is padded to a 32-bit boundary with NULs that `bytes.strip()` does not strip, that padding was read as a field *name* with no length prefix behind it -- so every journal entry of unaligned length raised a bare `struct.error`, measured on a 14-octet `MESSAGE=hello` entry. A NUL-only line now ends the entry and a short prefix ends it with a warning; the separate silent loss of every field after a binary one, from a `read()` that reaches EOF where it means to skip one newline, is filed as #704 rather than fixed. The five deliberately unclamped non-packet option areas keep the per-block framing assumption #676's note describes, which this does not close. Also here because it is the fifth and last registrar in the package with no overwrite guard at all: `Option.register` now reports a displaced option schema as a `RegistryWarning` naming every namespace the `ns='opt'` fan-out displaced something in, once per registration rather than once per namespace, tested by membership rather than by subscripting because the per-namespace registries are plain `defaultdict`\ s and reading one to look would insert `UnknownOption` for a code nobody registered. Presence alone is the firing condition, as on the seven code-keyed parser registrars and unlike the narrower guard #681 gave `register_protocol`, since this key is a caller-supplied `code` that `__init_subclass__` passes once per subclass; a namespace the call itself creates is exempt, because it starts as a copy of `opt`'s defaults and nothing in it is a prior registration. One draft did move `EXPECTED_FAILURES`, and it is worth recording: flooring the three decryption-secrets payloads that read `__length__` *whole* rather than subtracting from it packs nothing, because `Schema.pack` leaves that key at `-1` for "unknown" -- which emptied both payloads and turned `pcapng-secrets/TLS_Key_Log` and `.../WireGuard_Key_Log` from `MISMATCH` to `OK`, an empty payload comparing equal to an empty payload. Reverted and pinned; the entry count holds at 44 with its 35 PCAP-NG cases unmoved. Ships labelled breaking: well-formed captures are byte-identical, verified by regenerating `examples/captures/pcapng.txt` either side of the change, but any truncated PCAP-NG now yields the frames before the cut where it previously raised -- so a caller reading "extraction raised" as "this file is unusable" gets a partial result instead, whose last frame may carry zero-padded octets -- and three exception classes change at the margins, eight short-file depths moving from `ProtocolError: unknown byteorder magic` to `StreamEOFError`. The two modules hold 99.93% coverage with their 25 new statements covered and their single miss flat, and `examples/captures/pcapng.txt` does not move further, leaving the #683 drift #685 tracks exactly as it was (#678). +- **Fixed** -- every EOF-truncated PCAP-NG file raised a bare `ValueError` out of `Extractor`, so the whole extraction was lost rather than the one truncated block. The root is one layer above the reported site: Block Total Length is cross-checked against its own trailing copy and never against the file, so a last block declaring more than the file holds left `PCAPNG.read` seeking *past* the real end -- legal and silent -- and the next block read then measured a **negative** remainder, since `prepare` derives it as the end of the stream less the current position. That negative reached `SchemaField` through `pcapng_block_selector` and `io.RawIOBase.read` refused it, which is why nearly every truncation level failed rather than only the one holding the cut. The seek is now clamped to the octets `_read_fileng` actually returned, with a `ProtocolWarning` naming the overrun, and a tail under the twelve octets a block needs at minimum is reported as the quiet `StreamEOFError` that `Extractor.record_frames` already catches -- the one case that is deliberately not a clamp, because at the end of the file no block is being read and clamping would fabricate a frame out of zero padding. A second, independent route to the same shape is closed with a new `nonnegative()` helper, composed into `bounded_option` and `bounded_area` and applied to the eight unwrapped `length - N` spans and the eight `__option_padding__`-sized padding fields: `_TextField.__call__` builds its `struct` template as `f'{length}s'` unconditionally, so a negative became the format `'-8s'` and `struct.calcsize` raised -- neither that nor the `ValueError` being one of `pcapkit.utilities.exceptions`, and neither being an `EOFError`, so neither was caught where the frame loop catches the end of a file. 28 of the module's 74 length callbacks returned a negative before and none do now, from `-1` on an `epb_hash` declaring no payload to `-16777248` on an Enhanced Packet Block's option area; the 200-block, 8,048-octet `captured_len = 0xFFFFFF` vector that reached `struct` through `bounded_area`'s own `nominal <= available` test being true for a negative nominal now parses to 200 frames. Measured over all 1,509 octet boundaries of `examples/captures/dhcp.pcapng` with one harness either side: 6 levels parsed before and 1,495 do now, against 1,479 `ValueError` and 10 `struct.error` before and none from outside the library after, with the frame count degrading monotonically with the cut; of the 14 levels that still raise, twelve leave a file too short to hold a block at all and the other two cut into the Interface Description Block's `if_tsresol` option, so none of them costs a frame that was in the file. That claim is about the truncation sweep rather than about every input: a 4,000-round bounded mutation fuzz either side of the change, same seed and cap, takes the parse rate from 2,118 to 3,438 and removes both of #678's families, and the two foreign families left are measured unchanged -- `ValueError: N is not a valid BlockType`, filed as #701, where an unassigned block type raises from `aenum` inside `EnumField.post_process` so the `UnknownBlock` default the registry declares is unreachable; and `MemoryError` at exactly 30 of 4,000 in both trees, which is #593's 32-bit band. A third bare exception in this module *is* fixed here, found by the cross-review and reachable from valid input rather than truncated: `SystemdJournalExportBlock.post_process` unpacked a 64-bit binary-field length out of whatever `entry_data.read(8)` returned, and since the block body is padded to a 32-bit boundary with NULs that `bytes.strip()` does not strip, that padding was read as a field *name* with no length prefix behind it -- so every journal entry of unaligned length raised a bare `struct.error`, measured on a 14-octet `MESSAGE=hello` entry -- a defect about ordinary input rather than about truncation. A NUL-only line now ends the entry and a short prefix ends it with a warning. The same function leaked two more of the family, found on the cross-review's second pass and fixed with it: a binary field's declared length is the widest in the format and nothing bounded it against the entry holding it, so at `2**63` and above `BytesIO.read` refused it with a bare `OverflowError` while below that it silently returned whatever was there -- the same malformed prefix fatal or invisible by magnitude alone, now clamped to what the entry has left and reported; and a field name, key or value that is not UTF-8 raised a bare `UnicodeDecodeError`, fatal to a whole extraction over one octet in one field, now decoded with `errors='replace'` and reported, which is the option this module's own `StringField` already takes. The separate silent loss of every field after a binary one, from a `read()` that reaches EOF where it means to skip one newline, is filed as #704 rather than fixed. The five deliberately unclamped non-packet option areas keep the per-block framing assumption #676's note describes, which this does not close. Also here because it is the fifth and last registrar in the package with no overwrite guard at all: `Option.register` now reports a displaced option schema as a `RegistryWarning` naming every namespace the `ns='opt'` fan-out displaced something in, once per registration rather than once per namespace, tested by membership rather than by subscripting because the per-namespace registries are plain `defaultdict`\ s and reading one to look would insert `UnknownOption` for a code nobody registered. Presence alone is the firing condition, as on the seven code-keyed parser registrars and unlike the narrower guard #681 gave `register_protocol`, since this key is a caller-supplied `code` that `__init_subclass__` passes once per subclass; a namespace the call itself creates is exempt, because it starts as a copy of `opt`'s defaults and nothing in it is a prior registration. One draft did move `EXPECTED_FAILURES`, and it is worth recording: flooring the three decryption-secrets payloads that read `__length__` *whole* rather than subtracting from it packs nothing, because `Schema.pack` leaves that key at `-1` for "unknown" -- which emptied both payloads and turned `pcapng-secrets/TLS_Key_Log` and `.../WireGuard_Key_Log` from `MISMATCH` to `OK`, an empty payload comparing equal to an empty payload. Reverted and pinned; the entry count holds at 44 with its 35 PCAP-NG cases unmoved. Ships labelled breaking: well-formed captures are byte-identical, verified by regenerating `examples/captures/pcapng.txt` either side of the change, but any truncated PCAP-NG now yields the frames before the cut where it previously raised -- so a caller reading "extraction raised" as "this file is unusable" gets a partial result instead, whose last frame may carry zero-padded octets -- and three exception classes change at the margins, eight short-file depths moving from `ProtocolError: unknown byteorder magic` to `StreamEOFError`. The two modules hold 99.93% coverage with their 25 new statements covered and their single miss flat, and `examples/captures/pcapng.txt` does not move further, leaving the #683 drift #685 tracks exactly as it was (#678). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 7ed890eb3..5faf4fdf6 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2175,10 +2175,19 @@ pull requests between #326 and #509. whatever ``entry_data.read(8)`` returned, and since the block body is padded to a 32-bit boundary with NULs that ``bytes.strip()`` does not strip, that padding was read as a field *name* with no length prefix behind it -- so every journal entry of unaligned length raised - a bare ``struct.error``, measured on a 14-octet ``MESSAGE=hello`` entry. A NUL-only line now - ends the entry and a short prefix ends it with a warning; the separate silent loss of every - field after a binary one, from a ``read()`` that reaches EOF where it means to skip one - newline, is filed as #704 rather than fixed. The five deliberately unclamped non-packet + a bare ``struct.error``, measured on a 14-octet ``MESSAGE=hello`` entry -- a defect about + ordinary input rather than about truncation. A NUL-only line now ends the entry and a short + prefix ends it with a warning. The same function leaked two more of the family, found on the + cross-review's second pass and fixed with it: a binary field's declared length is the widest + in the format and nothing bounded it against the entry holding it, so at ``2**63`` and above + ``BytesIO.read`` refused it with a bare ``OverflowError`` while below that it silently + returned whatever was there -- the same malformed prefix fatal or invisible by magnitude + alone, now clamped to what the entry has left and reported; and a field name, key or value + that is not UTF-8 raised a bare ``UnicodeDecodeError``, fatal to a whole extraction over one + octet in one field, now decoded with ``errors='replace'`` and reported, which is the option + this module's own ``StringField`` already takes. The separate silent loss of every field + after a binary one, from a ``read()`` that reaches EOF where it means to skip one newline, is + filed as #704 rather than fixed. The five deliberately unclamped non-packet option areas keep the per-block framing assumption #676's note describes, which this does not close. Also here because it is the fifth and last registrar in the package with no overwrite guard From 621f429747fc85695a7ceb2ca9086b18365efb41 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 13:34:04 -0400 Subject: [PATCH 31/46] docs(changelog): the 1.5.0 entry for #672 and #679, whose code ships in #696 --- docs/source/changelog/1.5.0.rst | 47 +++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 5faf4fdf6..a3efcd8e2 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2216,6 +2216,53 @@ pull requests between #326 and #509. ``StreamEOFError``. The two modules hold 99.93% coverage with their 25 new statements covered and their single miss flat, and ``examples/captures/pcapng.txt`` does not move further, leaving the #683 drift #685 tracks exactly as it was (#678). +* **Fixed** -- HIP's ``R1_COUNTER``/``R1_Counter`` parameter packed 12 octets + where :rfc:`7401#section-5.2.3` requires 16, and ``LOCATOR_SET`` declared its + ``Length`` in 4-octet units where :rfc:`7401#section-5.2.1`'s ``Length`` is a + byte count -- two independent defects, fixed together because the evidence + for either needed the other out of the way first. ``R1CounterParameter``'s + ``counter`` was a ``UInt32Field`` where the RFC states the R1 generation + counter's width twice, as "8 bytes" in the diagram and as "a 64-bit unsigned + integer" in prose; both HIP versions that reach this class -- ``R1_Counter`` + (128) and ``R1_COUNTER`` (129) -- packed a 12-octet record, landing at + ``4 (mod 8)`` instead of aligned. It is a ``UInt64Field`` now, and the record + is 16 octets (#672). ``LocatorSetParameter``'s own ``Length`` was written as + ``sum(Locator.len)``, in the 4-octet units :rfc:`8046#section-4` gives + ``Locator Length``, where this parameter's own ``Length`` is a byte count; a + set of *n* plain IPv6 locators therefore declared ``4n`` octets against + ``24n`` actually present, and ``Schema.unpack`` handed the nested + ``ListField`` only the declared octets, so ``n = 2`` and ``n = 5`` both + parsed one truncated locator and left the rest of the record unconsumed. + ``HIP._make_param_locator_set`` now sums ``8 + Locator.len * 4`` per locator + -- the fixed header plus the RFC 8046 contents each one actually carries + (#679). A second, independent defect in the same parameter's padding is + fixed alongside it, one #651 deliberately left alone because it happened to + cancel this one exactly: the nested ``Locator`` schemas share the + parameter's packet context and pack their own ``len`` over it, so + ``padding`` -- evaluated after the list -- always saw 4 rather than the + parameter's real length. A new ``locator_set_len_callback`` snapshots + ``Length`` under a private key before any locator packs into the context, + and ``locator_set_padding_len`` reads that snapshot rather than the shadowed + one. An RFC-only byte-stride walk over + ``examples/captures/options-internet.pcap``, independent of pcapkit's own + parser, went from 1 violation -- ``LOCATOR_SET``'s empty record, concealed + until now because ``examples/generators/options.py``'s ``R1_COUNTER`` + override used ``counter=0``, a value the width defect could not be told + apart from -- to 2 once the counter was patched non-zero, to 0 once both + parameters were fixed; the fixture now overrides both ``R1_COUNTER`` codes + with ``counter=0xaabbccdd`` so a future regression cannot hide behind a zero + again. 10 new test methods across two new files -- 4 in + ``test_hip_r1_counter_width_unit.py``, 6 in + ``test_hip_locator_set_length_unit.py`` -- fail against the unfixed code and + pass against the fix; the three HIP modules stay at 100% coverage, and + ``tests/`` subtests move 586 to 623. ``EXPECTED_FAILURES`` is unmoved at 44 + entries: ``hip-parameter/R1_Counter`` stays, because code 128 registers no + schema of its own -- ``R1CounterParameter`` declares only ``code=129`` -- + which is a separate registry defect this does not touch, filed as #690. + ``HIP_COPIES`` also stays at two, now for no defect at all rather than for + the ones #651 and this fix removed; dropping it is #689, deferred because it + would halve every ``hip-parameter`` frame the fixture above reads (#672, + #679). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 2bb6f36c3c0427dbbd9c6600754edae3cc9b5a91 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 13:34:19 -0400 Subject: [PATCH 32/46] docs(changelog): the 1.5.0 entry for #700, whose code ships in #703 --- docs/source/changelog/1.5.0.rst | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index a3efcd8e2..6fac5aade 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2263,6 +2263,18 @@ pull requests between #326 and #509. the ones #651 and this fix removed; dropping it is #689, deferred because it would halve every ``hip-parameter`` frame the fixture above reads (#672, #679). +* **Fixed** -- three test comments stated an exact count of committed + captures under ``examples/captures/``, a number that drifts every time that + directory's tracked set changes and in one case was already wrong when + written. ``tests/_tiers.py`` and ``tests/test_tier_guard.py`` both cited + "six," correct at the time; ``tests/integration/_helpers.py`` said "four," + which undercounted the tracked set the day it was written. All three now + describe the invariant instead of a number that has to be kept in sync with + it by hand: ``_tiers.py`` reads "the moment somebody commits another + capture... or stops committing one," ``_helpers.py`` reads "...are + fixtures -- some of them committed --...," and ``test_tier_guard.py`` drops + its "seventh capture" phrasing the same way. Prose-only, verified with + ``coverage.parser.PythonParser`` that no executable statement moved (#700). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From cca61c6b66ff4e76b6b71a8a7edb80ec918878c6 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 13:34:35 -0400 Subject: [PATCH 33/46] docs(changelog): the 1.5.0 entry for #708, whose fix rides in on #703's branch --- docs/source/changelog/1.5.0.rst | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 6fac5aade..a4cf45026 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2275,6 +2275,29 @@ pull requests between #326 and #509. fixtures -- some of them committed --...," and ``test_tier_guard.py`` drops its "seventh capture" phrasing the same way. Prose-only, verified with ``coverage.parser.PythonParser`` that no executable statement moved (#700). +* **Fixed** -- ``test_every_tracked_name_exists_and_matches_git`` in + ``tests/test_tier_guard.py`` checked neither the git index nor the + filesystem despite its name: it asserted only that the tracked-capture list + was non-empty and that no name contained a ``/``, so a hardcoded list would + pass it whether or not it matched reality. Sibling + ``test_capture_suggestions_are_captures`` had the same gap. Found while + cross-reviewing #703, which is itself prose-only and left this out of + scope; the fix rides in as the second and third commits on that PR's + branch, whose own description names only #700. Both tests now re-derive the + expected set independently -- ``git ls-files -z`` under + ``examples/captures/`` -- and assert equality against it rather than + against themselves, plus a per-name ``Path.is_file()`` check the old + version never made. A cross-review of the first version of this fix found + it wrong twice over: a stale docstring in ``_tiers.py`` still claimed + ``test_tier_guard.py`` "only stats ``in.pcap``," falsified by the new + per-name loop; and the new ``assertEqual`` on + ``test_capture_suggestions_are_captures`` was vacuous on an empty tracked + set (``() == ()`` passes trivially), having dropped the non-emptiness guard + the other test kept. Both are corrected in a follow-up commit. Even after + the fix, a hardcoded literal that happens to match today's six tracked + names still passes -- the test detects a set that is wrong *at the moment it + runs*, not hardcoding as a practice, which turns a permanent blind spot into + a tripwire that fires on the next change to the tracked set (#708). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 872a6a645f2323304af2e2c818c793c94e7e08b3 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 13:34:51 -0400 Subject: [PATCH 34/46] docs(changelog): the 1.5.0 entry for #684, whose code ships in #694 --- docs/source/changelog/1.5.0.rst | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index a4cf45026..12efd36a6 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2298,6 +2298,33 @@ pull requests between #326 and #509. names still passes -- the test detects a set that is wrong *at the moment it runs*, not hardcoding as a practice, which turns a permanent blind spot into a tripwire that fires on the next change to the tracked set (#708). +* **Changed** -- reconciled which private (``_xxx``) attributes and methods + the Sphinx build documents, replacing an ad hoc mix with one stated rule: + document the contract, hide the recipe. A member every subclass must + implement, or one whose shape a caller genuinely depends on, stays + documented even though its name starts with an underscore; a private + helper that exists only to keep one method short does not. Seven + module-level directives naming pure implementation detail were dropped -- + ``esp._resolve``, ``esp._CRYPTO``, ``ngap._convert``, ``ngap._revert``, + ``ngap._PYCRATE``, ``ngap._PDU_LOCK`` and ``pypcapfile._NamedStream`` + (whose nested ``name``/``read`` members go with it, being reachable only + through a private class) -- while 39 ``autoattribute`` directives were + added for class-private state that *is* contract: ``Extractor._flag_f``, + the built-in and third-party engines' own ``_backend``, ``TraceFlow``'s + internal fields, and ``FieldBase``/``Field`` internals among them. The 128 + runtime definitions of ``_missing_`` -- 121 under ``pcapkit.const``, the + other 7 inline in ``pcapkit.protocols`` -- gained one write-up apiece + instead of a directive per class: a new "Unrecognised Values" section in + ``docs/source/pcapkit/const/index.rst``, cross-referenced from + ``registry.rst``, since ``conf.py`` already excludes ``_missing_`` from + every ``automodule`` via ``exclude-members``. ``CONTRIBUTING.md`` gained the + rule itself as a named section, so the next directive gets judged against a + written test rather than against precedent. Verified with + ``sphinx-build -b html`` under ``PCAPKIT_SPHINX=1``: 55 warnings on ``main`` + before this change, 56 after, the one addition being a pre-existing bare- + ``Type`` cross-reference ambiguity newly rendered by ``TraceFlow._foutio``'s + new directive rather than a defect this change introduced -- #709 tracks + it. No line under ``pcapkit/`` changed (#684). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 9244a40ab94ae5f19b098cc6d29e3e84720304fb Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 13:35:05 -0400 Subject: [PATCH 35/46] docs(changelog): the 1.5.0 entry for #702, whose code ships in #705 --- docs/source/changelog/1.5.0.rst | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 12efd36a6..ff8f5b955 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2325,6 +2325,30 @@ pull requests between #326 and #509. ``Type`` cross-reference ambiguity newly rendered by ``TraceFlow._foutio``'s new directive rather than a defect this change introduced -- #709 tracks it. No line under ``pcapkit/`` changed (#684). +* **Fixed** -- ``tests/dumpkit/test_nameless_enum_rendering_unit.py`` (added + by #670) swept every flag-enum registry against one fixed tuple ending in + ``65536``, on the assumption that every such registry is 16 bits wide. + Seven registries are not, at four distinct widths: + ``ftp.command.CommandType`` (3 bits), ``reg.apptype.TransportProtocol`` (4, + computed at runtime), ``mh.binding_ack_flag.BindingACKFlag``, + ``mh.handover_ack_flag.HandoverACKFlag`` and + ``mh.handover_initiate_flag.HandoverInitiateFlag`` (8 each), and + ``mh.binding_update_flag.BindingUpdateFlag`` with ``tcp.flags.Flags`` (16, + the two the old probe actually fit). ``tcp.flags.Flags(65536)`` correctly + raises -- its own ``_missing_`` bounds itself to ``0 <= value <= 0xFFFF`` -- + so the sweep was failing on correct behaviour rather than reporting a + defect. A new ``_field_mask`` derives each registry's own all-ones bound + from its declared members, and ``_nameless_values`` returns the values that + bound admits but no member names -- zero, each undeclared bit alone, and + every undeclared bit combined -- in place of the one hard-coded tuple. A + new ``test_a_value_past_the_field_is_refused_rather_than_rendered`` pins + the boundary directly: the widest in-field value is accepted, one past it + is refused. The file goes from 1 failed / 5 passed / 16 subtests to 6 + passed / 64 subtests. A cross-review found that of the seven guard + ``raise`` lines this newly reaches, six were already covered by passing + tests from #677, predating this fix; only ``tcp/flags.py``'s own guard line + was genuinely newly reached by it. No line under ``pcapkit/`` changed + (#702). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 0a83f474c022c2b43ea8002e29154f5a0346dbd5 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 13:35:19 -0400 Subject: [PATCH 36/46] docs(changelog): the 1.5.0 entry for #685, whose code ships in #697 --- docs/source/changelog/1.5.0.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index ff8f5b955..71a8a39ca 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2349,6 +2349,21 @@ pull requests between #326 and #509. tests from #677, predating this fix; only ``tcp/flags.py``'s own guard line was genuinely newly reached by it. No line under ``pcapkit/`` changed (#702). +* **Changed** -- ``examples/captures/out.json``, ``out.plist``, ``out.txt`` + and ``pcapng.txt`` are no longer tracked in git; they are build output, not + fixtures, and a tracked rendering with no reader goes stale silently every + time the code that produces it changes. ``pcapng.txt`` is exactly that: it + still recorded ``packet -> NIL`` for the four Enhanced Packet Blocks of + ``dhcp.pcapng`` long after #683 gave those blocks their captured octets + back, and nothing had regenerated it. Rather than regenerate it once more + and leave the same drift free to recur, the four files are removed from + the index and folded into ``examples/captures/``'s existing blanket + ``.gitignore`` rule; ``in.pcap`` and ``dhcp.pcapng``, the genuine inputs, + stay tracked. ``examples/legacy_smoke/Makefile`` and its ``README.rst`` are + reworded to describe regenerating these reports via ``make fixtures`` + rather than implying they ship committed, and a new + ``tests/project/test_capture_tracking.py`` (5 tests, 8 subtests) pins the + invariant going forward, shown failing against the pre-change tree (#685). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From 8d0d9d72410ba08e22eb2488552276b6e7133c50 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 13:35:34 -0400 Subject: [PATCH 37/46] docs(changelog): the 1.5.0 entry for #709, whose code partly ships in #712 --- docs/source/changelog/1.5.0.rst | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 71a8a39ca..08ad88d51 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2364,6 +2364,27 @@ pull requests between #326 and #509. rather than implying they ship committed, and a new ``tests/project/test_capture_tracking.py`` (5 tests, 8 subtests) pins the invariant going forward, shown failing against the pre-change tree (#685). +* **Fixed** -- two ``#:`` autodoc comments in + ``pcapkit/foundation/traceflow/traceflow.py`` named a bare ``Type``, which + Sphinx's cross-reference resolver resolves against every class named + ``Type`` in the project rather than against ``typing.Type`` -- there are + five -- and silently linked to ``pcapkit.const.l2tp.type.Type``, an L2TP + field-type enum with nothing to do with dumpers. Line 406 + (``#: Type[Dumper]: Dumper class.``) is the live case: once #684 rendered + ``TraceFlow._foutio``, the built docs pointed a reader at the wrong class + with no warning that anything had gone sideways. Both sites now spell it + ``~typing.Type[Dumper]``, the same form eight other files in the tree + already use for the identical ambiguity. Line 146 (``__output__``'s ``#:`` + comment) is fixed for the same reason but is currently inert: a separate, + correct ``# type:`` comment two lines below already drives that signature + through ``napoleon_attr_annotations``, so this half of the fix is insurance + against the day nothing else does. **This does not close #709.** Four more + bare ``Type`` sites -- hand-written ``:type:`` fields at + ``docs/source/pcapkit/foundation/engines/engine.rst:40``, + ``.../reassembly/reassembly.rst:33`` and ``:43``, and + ``.../traceflow/traceflow.rst:40`` -- carry the same ambiguity and are not + touched here; #709 stays open for them, with no PR yet filed against that + half (#709). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From f42a8b2623042961ad40a65dd1f0f3e3ff6081fb Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 13:35:49 -0400 Subject: [PATCH 38/46] docs(changelog): the 1.5.0 entry for #710, whose code ships in #711 --- docs/source/changelog/1.5.0.rst | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 08ad88d51..fc59d4102 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2385,6 +2385,26 @@ pull requests between #326 and #509. ``.../traceflow/traceflow.rst:40`` -- carry the same ambiguity and are not touched here; #709 stays open for them, with no PR yet filed against that half (#709). +* **Fixed** -- ``register_protocol``'s overwrite warning could claim a + protocol was replaced with itself. The guard that decides *whether* to + warn was already correct -- ``incumbent is not protocol``, an identity + check from #681 -- but the message built both operands with a bare + ``repr()``, and for an ordinary class that is just + ````. A factory that defines a same-named, + closure-local class on every call (as + ``tests/protocols/test_construction_keyword_check_unit.py``'s + ``_protocol_class`` does) produces two genuinely distinct classes sharing + one ``__module__`` and ``__qualname__``, so both reprs print identically + and a real, correct overwrite reads as "overwriting X with X." The message + now compares the two reprs first, and only when they coincide appends each + object's ``id()`` to tell them apart; the common case, where the two + classes are named differently, is untouched. ``__module__``/``__qualname__`` + was considered as the disambiguator instead of ``id()`` and rejected: for + the reported shape those are exactly what the coinciding repr already + renders, so they discriminate nothing an ``id()`` does not already have to. + A new ``test_register_protocol_disambiguates_classes_sharing_a_repr`` fails + against the unfixed message and passes against the fix; the targeted suite + goes 49 to 50 passed, and the file's coverage holds at 97% (#710). Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both 2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so From f5287761d6d8fc71079a57298a7315c181d8e028 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 14:49:44 -0400 Subject: [PATCH 39/46] docs(changelog): fix eight factual errors a cross-review found in the new entries Cross-review on issuecomment-5800589100 (PR #657) found the eight new 1.5.0 entries wrong in several places, spot-checked against source before each fix: - #709 entry: dropped the false "no PR yet filed" clause -- PR #714, opened five minutes before the commit claiming otherwise, covers exactly the four sites named. Corrected the inertness explanation for line 146: the real ``# type:`` comment is at line 162 (sixteen lines below, not two), it also carries a bare ``Type[Dumper]``, and the likely mechanism is ``sphinx_autodoc_typehints`` rather than ``napoleon_attr_annotations``. - #702 entry: "six of seven guard lines" is seven of seven -- #677's sweep exempts nothing and reaches ``tcp/flags.py`` too. Split the conflated claim about "one tuple ending in 65536 swept every registry" into the two tests that actually exist. Reworded the "seven registries are not, at four distinct widths" line, which reads as self-contradicting two of its own examples. - #684 entry: "one write-up apiece" -> one unified write-up; "every automodule" -> there are zero, the exclusion works through ``autoclass`` and ``autodoc_default_options``; "built-in and third-party engines' own _backend" -> only the third-party ones gained it, built-in gained a different attribute set. Also fixed the "bare-\n``Type``" line wrap that renders as "bare- `Type`" -- the only trailing-hyphen wrap in the file. - #708 entry: only one of the two tests re-derives against git; the sibling still compares the module to itself. Dropped "today's six tracked names", which contradicts the #685 entry naming 2. - #700 entry: only ``_tiers.py`` cited "six"; ``test_tier_guard.py`` said "seventh capture" instead. Two smaller imprecisions, fixed as flagged: the #672/#679 entry's "both codes reach this class" now says packing explicitly (parsing only reaches 129, per #690, already stated later in the same entry), and its claim of a pre-fix ``R1_COUNTER`` override is corrected to a schema default. The #685 entry's "shown failing" is now qualified to the 2 of 5 tests (5 of 8 subtests) that actually fail against the pre-change tree. Checked but not changed: no entry in the file asserts the "issue numbers only" citation convention the review tested, so there was nothing to correct there; the five inline PR references it flagged as legitimate are untouched. Verified: docutils 0.22.4 ``publish_doctree`` over the changed block -- zero system messages, one bullet_list of 8 items. --- docs/source/changelog/1.5.0.rst | 97 +++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 35 deletions(-) diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index fc59d4102..df653eb9c 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2223,8 +2223,8 @@ pull requests between #326 and #509. for either needed the other out of the way first. ``R1CounterParameter``'s ``counter`` was a ``UInt32Field`` where the RFC states the R1 generation counter's width twice, as "8 bytes" in the diagram and as "a 64-bit unsigned - integer" in prose; both HIP versions that reach this class -- ``R1_Counter`` - (128) and ``R1_COUNTER`` (129) -- packed a 12-octet record, landing at + integer" in prose; both HIP versions whose *packing* reaches this class -- + ``R1_Counter`` (128) and ``R1_COUNTER`` (129) -- packed a 12-octet record, landing at ``4 (mod 8)`` instead of aligned. It is a ``UInt64Field`` now, and the record is 16 octets (#672). ``LocatorSetParameter``'s own ``Length`` was written as ``sum(Locator.len)``, in the 4-octet units :rfc:`8046#section-4` gives @@ -2246,9 +2246,10 @@ pull requests between #326 and #509. one. An RFC-only byte-stride walk over ``examples/captures/options-internet.pcap``, independent of pcapkit's own parser, went from 1 violation -- ``LOCATOR_SET``'s empty record, concealed - until now because ``examples/generators/options.py``'s ``R1_COUNTER`` - override used ``counter=0``, a value the width defect could not be told - apart from -- to 2 once the counter was patched non-zero, to 0 once both + until now because ``R1_COUNTER``'s ``counter`` field defaulted to ``0`` with + no override anywhere in ``examples/generators/options.py``, a value the + width defect could not be told apart from -- to 2 once the counter was + patched non-zero, to 0 once both parameters were fixed; the fixture now overrides both ``R1_COUNTER`` codes with ``counter=0xaabbccdd`` so a future regression cannot hide behind a zero again. 10 new test methods across two new files -- 4 in @@ -2266,9 +2267,11 @@ pull requests between #326 and #509. * **Fixed** -- three test comments stated an exact count of committed captures under ``examples/captures/``, a number that drifts every time that directory's tracked set changes and in one case was already wrong when - written. ``tests/_tiers.py`` and ``tests/test_tier_guard.py`` both cited - "six," correct at the time; ``tests/integration/_helpers.py`` said "four," - which undercounted the tracked set the day it was written. All three now + written. ``tests/_tiers.py`` cited "six," correct at the time; + ``tests/test_tier_guard.py`` said "the moment a seventh capture is + committed" -- the same count, as an ordinal rather than the word "six"; + ``tests/integration/_helpers.py`` said "four," which undercounted the + tracked set the day it was written. All three now describe the invariant instead of a number that has to be kept in sync with it by hand: ``_tiers.py`` reads "the moment somebody commits another capture... or stops committing one," ``_helpers.py`` reads "...are @@ -2283,18 +2286,25 @@ pull requests between #326 and #509. ``test_capture_suggestions_are_captures`` had the same gap. Found while cross-reviewing #703, which is itself prose-only and left this out of scope; the fix rides in as the second and third commits on that PR's - branch, whose own description names only #700. Both tests now re-derive the - expected set independently -- ``git ls-files -z`` under - ``examples/captures/`` -- and assert equality against it rather than - against themselves, plus a per-name ``Path.is_file()`` check the old - version never made. A cross-review of the first version of this fix found + branch, whose own description names only #700. + ``test_every_tracked_name_exists_and_matches_git`` now re-derives the + expected set independently -- shelling out to ``git ls-files -z`` under + ``examples/captures/`` rather than calling back into ``_tiers.py`` -- and + asserts equality against it rather than against itself, plus a per-name + ``Path.is_file()`` check the old version never made. Its sibling + ``test_capture_suggestions_are_captures`` gets a narrower fix: it asserts + equality against ``committed_capture_names()``'s own filter re-implemented + inline over ``_tiers.committed_captures()``, which catches a wrong filter + but, unlike its sibling, still trusts ``committed_captures()`` for the + tracked set itself rather than re-deriving it from git. A cross-review of + the first version of this fix found it wrong twice over: a stale docstring in ``_tiers.py`` still claimed ``test_tier_guard.py`` "only stats ``in.pcap``," falsified by the new per-name loop; and the new ``assertEqual`` on ``test_capture_suggestions_are_captures`` was vacuous on an empty tracked set (``() == ()`` passes trivially), having dropped the non-emptiness guard the other test kept. Both are corrected in a follow-up commit. Even after - the fix, a hardcoded literal that happens to match today's six tracked + the fix, a hardcoded literal that happens to match today's tracked names still passes -- the test detects a set that is wrong *at the moment it runs*, not hardcoding as a practice, which turns a permanent blind spot into a tripwire that fires on the next change to the tracked set (#708). @@ -2310,25 +2320,34 @@ pull requests between #326 and #509. (whose nested ``name``/``read`` members go with it, being reachable only through a private class) -- while 39 ``autoattribute`` directives were added for class-private state that *is* contract: ``Extractor._flag_f``, - the built-in and third-party engines' own ``_backend``, ``TraceFlow``'s - internal fields, and ``FieldBase``/``Field`` internals among them. The 128 + the third-party engines' own ``_backend``, ``TraceFlow``'s internal fields, + and ``FieldBase``/``Field`` internals among them; the built-in ``PCAP`` and + ``PCAPNG`` engines gained a different set of their own -- ``_gbhdr``, + ``_vinfo``, ``_dlink`` and ``_nnsec`` for the former, ``_ctx`` and + ``_ctx_list`` for the latter -- and no ``_backend`` at all. The 128 runtime definitions of ``_missing_`` -- 121 under ``pcapkit.const``, the - other 7 inline in ``pcapkit.protocols`` -- gained one write-up apiece - instead of a directive per class: a new "Unrecognised Values" section in + other 7 inline in ``pcapkit.protocols`` -- gained one unified write-up in + place of a directive per class: a new "Unrecognised Values" section in ``docs/source/pcapkit/const/index.rst``, cross-referenced from ``registry.rst``, since ``conf.py`` already excludes ``_missing_`` from - every ``automodule`` via ``exclude-members``. ``CONTRIBUTING.md`` gained the + every ``autoclass`` via ``exclude-members`` in ``autodoc_default_options`` + -- there is no ``automodule`` directive anywhere under ``docs/source/``. + ``CONTRIBUTING.md`` gained the rule itself as a named section, so the next directive gets judged against a written test rather than against precedent. Verified with ``sphinx-build -b html`` under ``PCAPKIT_SPHINX=1``: 55 warnings on ``main`` - before this change, 56 after, the one addition being a pre-existing bare- + before this change, 56 after, the one addition being a pre-existing bare ``Type`` cross-reference ambiguity newly rendered by ``TraceFlow._foutio``'s new directive rather than a defect this change introduced -- #709 tracks it. No line under ``pcapkit/`` changed (#684). * **Fixed** -- ``tests/dumpkit/test_nameless_enum_rendering_unit.py`` (added - by #670) swept every flag-enum registry against one fixed tuple ending in - ``65536``, on the assumption that every such registry is 16 bits wide. - Seven registries are not, at four distinct widths: + by #670) carried two probes on the same wrong assumption that every + flag-enum registry is 16 bits wide: + ``test_scalar_return_renders_a_nameless_member_as_its_value`` checked one + fixed tuple ending in ``65536`` against ``tcp.flags.Flags`` and its own + ``StdFlags`` stand-in alone, and ``test_no_flag_registry_renders_the_literal_none`` + swept every flag-enum registry instead, but only against ``registry(0)``. + Seven registries are not *all* 16 bits, at four distinct widths: ``ftp.command.CommandType`` (3 bits), ``reg.apptype.TransportProtocol`` (4, computed at runtime), ``mh.binding_ack_flag.BindingACKFlag``, ``mh.handover_ack_flag.HandoverACKFlag`` and @@ -2344,10 +2363,10 @@ pull requests between #326 and #509. new ``test_a_value_past_the_field_is_refused_rather_than_rendered`` pins the boundary directly: the widest in-field value is accepted, one past it is refused. The file goes from 1 failed / 5 passed / 16 subtests to 6 - passed / 64 subtests. A cross-review found that of the seven guard - ``raise`` lines this newly reaches, six were already covered by passing - tests from #677, predating this fix; only ``tcp/flags.py``'s own guard line - was genuinely newly reached by it. No line under ``pcapkit/`` changed + passed / 64 subtests. A cross-review found that all seven guard + ``raise`` lines this reaches -- ``tcp/flags.py``'s own included -- were + already covered by passing tests from #677, predating this fix; none was + genuinely newly reached by it. No line under ``pcapkit/`` changed (#702). * **Changed** -- ``examples/captures/out.json``, ``out.plist``, ``out.txt`` and ``pcapng.txt`` are no longer tracked in git; they are build output, not @@ -2363,7 +2382,9 @@ pull requests between #326 and #509. reworded to describe regenerating these reports via ``make fixtures`` rather than implying they ship committed, and a new ``tests/project/test_capture_tracking.py`` (5 tests, 8 subtests) pins the - invariant going forward, shown failing against the pre-change tree (#685). + invariant going forward; 2 of the 5 tests (5 of the 8 subtests) fail + against the pre-change tree with the four reports restored to the index, + the other 3 passing on both trees by construction (#685). * **Fixed** -- two ``#:`` autodoc comments in ``pcapkit/foundation/traceflow/traceflow.py`` named a bare ``Type``, which Sphinx's cross-reference resolver resolves against every class named @@ -2374,17 +2395,23 @@ pull requests between #326 and #509. ``TraceFlow._foutio``, the built docs pointed a reader at the wrong class with no warning that anything had gone sideways. Both sites now spell it ``~typing.Type[Dumper]``, the same form eight other files in the tree - already use for the identical ambiguity. Line 146 (``__output__``'s ``#:`` - comment) is fixed for the same reason but is currently inert: a separate, - correct ``# type:`` comment two lines below already drives that signature - through ``napoleon_attr_annotations``, so this half of the fix is insurance - against the day nothing else does. **This does not close #709.** Four more + already use for the identical ambiguity. Line 146 (the first line of + ``__output__``'s ``#:`` block, which continues through line 148) is fixed + for the same reason but is currently inert: the actual ``# type:`` comment + that drives that signature is sixteen lines below, at line 162, and it + spells the identical bare ``Type[Dumper]`` -- so it is not correct either. + The mechanism is likely ``sphinx_autodoc_typehints`` rather than + ``napoleon_attr_annotations``: Napoleon's own attribute-annotation lookup + fires only for a Google/Numpy ``Attributes:`` section, and ``__output__`` + is documented through autodoc's ``#:`` scraping instead. This half of the + fix is insurance against the day something reads it cleanly. **This does + not close #709.** Four more bare ``Type`` sites -- hand-written ``:type:`` fields at ``docs/source/pcapkit/foundation/engines/engine.rst:40``, ``.../reassembly/reassembly.rst:33`` and ``:43``, and ``.../traceflow/traceflow.rst:40`` -- carry the same ambiguity and are not - touched here; #709 stays open for them, with no PR yet filed against that - half (#709). + touched here; #709 stays open for them, though PR #714 covers exactly + those four sites (#709). * **Fixed** -- ``register_protocol``'s overwrite warning could claim a protocol was replaced with itself. The guard that decides *whether* to warn was already correct -- ``incumbent is not protocol``, an identity From 2c0dbba43081561a2821d5e1188e4cf3fceff6f6 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 14:49:54 -0400 Subject: [PATCH 40/46] chore(changelog): regenerate CHANGELOG.md for the eight new entries The eight entry commits landed on this branch without regenerating CHANGELOG.md, leaving it drifted from docs/source/changelog/1.5.0.rst -- the CI blocker a cross-review flagged (issuecomment-5800565258). Measured before this commit: `util/changelog_md.py --check` exits 1, and the drift is exactly the eight new bullets, nothing else. Ran `python util/changelog_md.py` (never hand-edited) to add the eight lines to CHANGELOG.md. `--check` now exits 0, and tests/project/test_changelog_md.py passes in full (47 passed, 37 subtests). --- CHANGELOG.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7ca729fe9..a56718879 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -108,6 +108,14 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- an unpadded HTTP/2 `DATA`, `HEADERS` or `PUSH_PROMISE` frame parsed with its whole payload silently discarded. Three payload length callbacks in `pcapkit/protocols/schema/application/httpv2.py` put the conditional expression in the wrong place: a conditional binds looser than `-`, so `pkt['__length__'] - pkt['pad_len'] if pkt['flags']['bit_3'] else 0` groups as `(pkt['__length__'] - pkt['pad_len']) if ... else 0` and the `else` arm returned `0` -- "read no octets at all" to `BytesField` -- where it was meant to subtract `0`. Since `__length__` is the *remaining* declared length at the field, the unpadded arm wants `__length__` itself, which is what subtracting a zero padding length gives. The grouping was read off the AST rather than by eye, the top-level node being the `IfExp` whose `orelse` is a bare `Constant(0)`, which is also what showed that the outer parentheses on the `HeadersFrame` and `PushPromiseFrame` forms were line-continuation only and changed nothing. Padding is rare in HTTP/2, so the broken arm was the common one rather than the edge case, and nothing raised, warned or logged: `info.data` was simply `b''`. Measured on wire octets through the public `HTTP(io.BytesIO(raw), len(raw))` path, an unpadded `DATA` frame declaring `b'{"ok":true}\n'` parsed as `b''` and now parses as those twelve octets; unpadded `HEADERS` and `PUSH_PROMISE` lost and now keep the RFC 7541 appendix C.4.1 header block fragment `b'\x82\x86\x84A\x0fwww.example.com'`, which is precisely what HPACK decoding would have been handed. Padded frames are untouched, for a reason stronger than the measurement: `(A - B) if T else 0` and `A - (B if T else 0)` are both `A - B` for truthy `T`, so only the `else` arm could ever have moved -- both arms are asserted anyway, so that a repair to the unpadded one cannot break the other. The padded arm also has no off-by-one, which #668 asked be checked rather than assumed: `Schema.unpack` decrements `packet['__length__']` by each field's width as it goes, so `pad_len`'s own octet is already out of it by the time the payload field runs, and a padded frame declaring `1 + len(data) + pad_len` arrives at the payload with `__length__ == len(data) + pad_len`. The three sibling sites that were never wrong are what localise the defect to the grouping rather than to the field machinery: `ContinuationFrame.fragment`, `UnassignedFrame.data` and `GoawayFrame.debug` use the plain `length=lambda pkt: pkt['__length__']` with no conditional, share the same `BytesField`, the same `__length__` bookkeeping and the same frame dispatch, and carried their payload correctly throughout -- they are pinned as controls rather than left as an argument. Why nothing caught it: `test_option_roundtrip_unit` does drive every frame type, but through `make` -> parse -> `make`, and `make` writes the length field from `_make_http_length`, so both sides agreed on an empty payload and the octets matched; its generator passes no payload argument for these three frames at all, measured as `kwargs={}`, so what it round-tripped was `b''` and there was nothing to lose, while `test_httpv2_frame_readers_cover_successful_frames` drives the readers from hand-built schema stubs, so the callback never ran there either. The new `tests/protocols/application/test_httpv2_payload_length_unit.py` reads wire octets, which is the gap: 23 tests and 12 subtests over the padded, unpadded and padding-only shapes of all three frames plus both `PRIORITY` combinations, asserting the payload *octets* and never its length, because a length assertion survives a read of the right width from the wrong offset. Its padded cases pad with a non-zero `b'\xde\xad\xbe\xef'` rather than the zeros [RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1) tells a sender to use -- pcapkit does not police padding content -- so that failing to subtract the padding yields a different byte string instead of a coincidentally equal count. The same lambda also governs **packing**, which neither the issue nor the first revision of the fix noticed and a cross-review on a second model did: `BytesField` consults its `length` callback on both paths, so the `else` arm did not merely discard a payload on read, it declined to write one. Measured pre-fix, an unpadded `DATA` frame carrying twelve octets of body packed to `000015000000000001` -- nine octets of header whose length field declares **21**, with the body absent -- and unpadded `HEADERS` and `PUSH_PROMISE` likewise declared 29 against 9 and 33 against 13. A frame overstating its own length by its whole payload desynchronises any reader that walks a stream by that field, so this was pcapkit *emitting* malformed HTTP/2 rather than only mis-reading it, and `HTTPv2ConstructedFrameDeclaresWhatItWritesUnitTests` pins that half; a `make` -> parse -> `make` cycle of a non-empty unpadded payload now closes byte-for-byte where it could not before. Six wrong fixes were written over the schema to check the tests discriminate, and each is rejected: dropping the conditional outright (caught by `packet length < 0: -4`), subtracting `pad_len + 1` (caught as `b'{"ok":true}'` against `b'{"ok":true}\n'`), subtracting the padding on the arm where it is absent, fixing `DataFrame` while forgetting the other two, and `max(computed, 1)` on the two `fragment` fields. That last one is the cross-review's own construction and it **passed** the first revision's 15 tests and 9 subtests, because every fixture then carried a non-empty payload and only `DataFrame` had a "frame of only padding" case, so those two callbacks were never once asked for `0`; under it a padding-only `HEADERS` frame returned `fragment=b'\xde'`, one padding octet leaked in, alongside a swallowed `SchemaWarning: packet length < 0: -1`. The three missing padding-only cases and a callback-level `test_the_padded_arm_reaches_zero_and_is_not_clamped` close it -- `0` is a legitimate answer from these callbacks and has to be asserted as one. Seven tests and three subtests fail before the change and all pass after; 145 tests and 550 subtests pass across `tests/protocols/application/`, the round-trip module and the four other HTTP/2-touching modules. `EXPECTED_FAILURES` is unmoved, its one HTTP/2 entry `httpv2-frame/PRIORITY` still failing with the same `CONSTRUCT` status and the same detail -- `PriorityFrame` has neither a payload field nor a `PADDED` flag, so this change cannot reach it -- and none of the 44 entries was deleted; that entry's own cited `pcapkit/protocols/application/httpv2.py:572` is stale, the `header.length != 9` guard it describes now being at `:562`. The schema module was already at 100% coverage and stays there, 101 statements and 4 branches either side with nothing missing or partial, because the changed lines did execute before -- just with an empty payload -- so the number that moved is the suite: 73 to 96 tests and 399 to 411 subtests over the same targets. The construct side needed no code change, `_make_http_length` having always computed the DATA payload as `len(frame.data) + (pad_len + 1 if pad_len else 0)`, i.e. assuming `data` holds the whole payload whether or not the frame is padded -- it was the shared length callback that disagreed, on both paths at once. An AST sweep of all 496 files of the package found one other site with the same syntactic shape, `pcapkit/protocols/schema/internet/ipv4.py:336` (`TSOption.remainder`), and it is correct rather than the same defect: its `else 0` is intentional because the sibling `ts_data` field consumes the entire option data area in that arm, the two summing to `length - 4` across all 2106 `(length, pointer, flag)` combinations, which is exactly what was not true here. Ships labelled breaking on both counts: restoring a dropped payload moves the parse output of essentially every HTTP/2 capture on the success path, so anything holding a golden file or a regression baseline that recorded the empty value sees it change -- and constructed frames change byte-for-byte too, from malformed to correct (#668). - **Fixed** -- the schema half of every option-like registrar overwrote silently while the parser half of the very same call warned. `EnumSchema.register` was a bare `cls.__enum__[code] = schema`, and it is the schema half of 14 public registrars, so one `register_ipv4_option` call replacing a built-in named the parser it displaced and said nothing about the schema. `EnumSchema.__init_subclass__` reaches the same registry without calling `register`, so a subclass declared with a `code=` keyword -- the documented way to add a schema -- stayed silent too; both are guarded now, and the declaration hook's two branches are folded into one loop so the guard is written once. Presence is a faithful test there only because `_EnumRegistry.__missing__` returns a miss without recording it, so parsing one packet carrying an unknown code cannot make the next legitimate registration for that code warn about an entry no caller ever asked for. The seven code-keyed parser registrars -- on `ProtocolBase`, `Link`, `Internet`, `Frame`, `PCAPNG`, `Transport` and `SCTP` -- now name the displaced entry and its replacement, where before they reported only that something had been overwritten and never which class it was. Their presence-only firing condition is deliberately unchanged, and is not harmonised onto the narrower guard #681 gave `register_protocol`: that one is keyed on a name derived from the value it stores, whereas these seven take a `code` the caller supplies independently of the value, so one class registered under two codes yields two keys and never reaches one key twice. These tables also ship pre-seeded with unresolved `ModuleDescriptor` values, so an incumbent may be a two-string descriptor while the replacement is the very class it names, which leaves a different-class test undecidable without resolving the descriptor and forcing the import it exists to defer. `ContextRegistry.register` already raises on a duplicate, and the reassembly and ESP registrars append to lists with no key at all -- duplicate security-association SPIs being a designed feature there, resolved by scoring rather than by replacement -- so none of those three takes a guard. `import pcapkit` holds at one warning, a third-party deprecation, and no `RegistryWarning`, measured over the 326 registry writes it performs, none of which lands on a key already present; `tests/foundation/registry/` goes 13 to 14 tests and 87 to 95 subtests, the schema module holds 99% coverage with its 5 new statements covered and its misses flat at 1, mypy and pylint are unmoved at 112 errors and 364 messages, and `EXPECTED_FAILURES` is unmoved at 44 entries (#692). - **Fixed** -- every EOF-truncated PCAP-NG file raised a bare `ValueError` out of `Extractor`, so the whole extraction was lost rather than the one truncated block. The root is one layer above the reported site: Block Total Length is cross-checked against its own trailing copy and never against the file, so a last block declaring more than the file holds left `PCAPNG.read` seeking *past* the real end -- legal and silent -- and the next block read then measured a **negative** remainder, since `prepare` derives it as the end of the stream less the current position. That negative reached `SchemaField` through `pcapng_block_selector` and `io.RawIOBase.read` refused it, which is why nearly every truncation level failed rather than only the one holding the cut. The seek is now clamped to the octets `_read_fileng` actually returned, with a `ProtocolWarning` naming the overrun, and a tail under the twelve octets a block needs at minimum is reported as the quiet `StreamEOFError` that `Extractor.record_frames` already catches -- the one case that is deliberately not a clamp, because at the end of the file no block is being read and clamping would fabricate a frame out of zero padding. A second, independent route to the same shape is closed with a new `nonnegative()` helper, composed into `bounded_option` and `bounded_area` and applied to the eight unwrapped `length - N` spans and the eight `__option_padding__`-sized padding fields: `_TextField.__call__` builds its `struct` template as `f'{length}s'` unconditionally, so a negative became the format `'-8s'` and `struct.calcsize` raised -- neither that nor the `ValueError` being one of `pcapkit.utilities.exceptions`, and neither being an `EOFError`, so neither was caught where the frame loop catches the end of a file. 28 of the module's 74 length callbacks returned a negative before and none do now, from `-1` on an `epb_hash` declaring no payload to `-16777248` on an Enhanced Packet Block's option area; the 200-block, 8,048-octet `captured_len = 0xFFFFFF` vector that reached `struct` through `bounded_area`'s own `nominal <= available` test being true for a negative nominal now parses to 200 frames. Measured over all 1,509 octet boundaries of `examples/captures/dhcp.pcapng` with one harness either side: 6 levels parsed before and 1,495 do now, against 1,479 `ValueError` and 10 `struct.error` before and none from outside the library after, with the frame count degrading monotonically with the cut; of the 14 levels that still raise, twelve leave a file too short to hold a block at all and the other two cut into the Interface Description Block's `if_tsresol` option, so none of them costs a frame that was in the file. That claim is about the truncation sweep rather than about every input: a 4,000-round bounded mutation fuzz either side of the change, same seed and cap, takes the parse rate from 2,118 to 3,438 and removes both of #678's families, and the two foreign families left are measured unchanged -- `ValueError: N is not a valid BlockType`, filed as #701, where an unassigned block type raises from `aenum` inside `EnumField.post_process` so the `UnknownBlock` default the registry declares is unreachable; and `MemoryError` at exactly 30 of 4,000 in both trees, which is #593's 32-bit band. A third bare exception in this module *is* fixed here, found by the cross-review and reachable from valid input rather than truncated: `SystemdJournalExportBlock.post_process` unpacked a 64-bit binary-field length out of whatever `entry_data.read(8)` returned, and since the block body is padded to a 32-bit boundary with NULs that `bytes.strip()` does not strip, that padding was read as a field *name* with no length prefix behind it -- so every journal entry of unaligned length raised a bare `struct.error`, measured on a 14-octet `MESSAGE=hello` entry -- a defect about ordinary input rather than about truncation. A NUL-only line now ends the entry and a short prefix ends it with a warning. The same function leaked two more of the family, found on the cross-review's second pass and fixed with it: a binary field's declared length is the widest in the format and nothing bounded it against the entry holding it, so at `2**63` and above `BytesIO.read` refused it with a bare `OverflowError` while below that it silently returned whatever was there -- the same malformed prefix fatal or invisible by magnitude alone, now clamped to what the entry has left and reported; and a field name, key or value that is not UTF-8 raised a bare `UnicodeDecodeError`, fatal to a whole extraction over one octet in one field, now decoded with `errors='replace'` and reported, which is the option this module's own `StringField` already takes. The separate silent loss of every field after a binary one, from a `read()` that reaches EOF where it means to skip one newline, is filed as #704 rather than fixed. The five deliberately unclamped non-packet option areas keep the per-block framing assumption #676's note describes, which this does not close. Also here because it is the fifth and last registrar in the package with no overwrite guard at all: `Option.register` now reports a displaced option schema as a `RegistryWarning` naming every namespace the `ns='opt'` fan-out displaced something in, once per registration rather than once per namespace, tested by membership rather than by subscripting because the per-namespace registries are plain `defaultdict`\ s and reading one to look would insert `UnknownOption` for a code nobody registered. Presence alone is the firing condition, as on the seven code-keyed parser registrars and unlike the narrower guard #681 gave `register_protocol`, since this key is a caller-supplied `code` that `__init_subclass__` passes once per subclass; a namespace the call itself creates is exempt, because it starts as a copy of `opt`'s defaults and nothing in it is a prior registration. One draft did move `EXPECTED_FAILURES`, and it is worth recording: flooring the three decryption-secrets payloads that read `__length__` *whole* rather than subtracting from it packs nothing, because `Schema.pack` leaves that key at `-1` for "unknown" -- which emptied both payloads and turned `pcapng-secrets/TLS_Key_Log` and `.../WireGuard_Key_Log` from `MISMATCH` to `OK`, an empty payload comparing equal to an empty payload. Reverted and pinned; the entry count holds at 44 with its 35 PCAP-NG cases unmoved. Ships labelled breaking: well-formed captures are byte-identical, verified by regenerating `examples/captures/pcapng.txt` either side of the change, but any truncated PCAP-NG now yields the frames before the cut where it previously raised -- so a caller reading "extraction raised" as "this file is unusable" gets a partial result instead, whose last frame may carry zero-padded octets -- and three exception classes change at the margins, eight short-file depths moving from `ProtocolError: unknown byteorder magic` to `StreamEOFError`. The two modules hold 99.93% coverage with their 25 new statements covered and their single miss flat, and `examples/captures/pcapng.txt` does not move further, leaving the #683 drift #685 tracks exactly as it was (#678). +- **Fixed** -- HIP's `R1_COUNTER`/`R1_Counter` parameter packed 12 octets where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires 16, and `LOCATOR_SET` declared its `Length` in 4-octet units where [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1)'s `Length` is a byte count -- two independent defects, fixed together because the evidence for either needed the other out of the way first. `R1CounterParameter`'s `counter` was a `UInt32Field` where the RFC states the R1 generation counter's width twice, as "8 bytes" in the diagram and as "a 64-bit unsigned integer" in prose; both HIP versions whose *packing* reaches this class -- `R1_Counter` (128) and `R1_COUNTER` (129) -- packed a 12-octet record, landing at `4 (mod 8)` instead of aligned. It is a `UInt64Field` now, and the record is 16 octets (#672). `LocatorSetParameter`'s own `Length` was written as `sum(Locator.len)`, in the 4-octet units [RFC 8046 Section 4](https://datatracker.ietf.org/doc/html/rfc8046#section-4) gives `Locator Length`, where this parameter's own `Length` is a byte count; a set of *n* plain IPv6 locators therefore declared `4n` octets against `24n` actually present, and `Schema.unpack` handed the nested `ListField` only the declared octets, so `n = 2` and `n = 5` both parsed one truncated locator and left the rest of the record unconsumed. `HIP._make_param_locator_set` now sums `8 + Locator.len * 4` per locator -- the fixed header plus the RFC 8046 contents each one actually carries (#679). A second, independent defect in the same parameter's padding is fixed alongside it, one #651 deliberately left alone because it happened to cancel this one exactly: the nested `Locator` schemas share the parameter's packet context and pack their own `len` over it, so `padding` -- evaluated after the list -- always saw 4 rather than the parameter's real length. A new `locator_set_len_callback` snapshots `Length` under a private key before any locator packs into the context, and `locator_set_padding_len` reads that snapshot rather than the shadowed one. An RFC-only byte-stride walk over `examples/captures/options-internet.pcap`, independent of pcapkit's own parser, went from 1 violation -- `LOCATOR_SET`'s empty record, concealed until now because `R1_COUNTER`'s `counter` field defaulted to `0` with no override anywhere in `examples/generators/options.py`, a value the width defect could not be told apart from -- to 2 once the counter was patched non-zero, to 0 once both parameters were fixed; the fixture now overrides both `R1_COUNTER` codes with `counter=0xaabbccdd` so a future regression cannot hide behind a zero again. 10 new test methods across two new files -- 4 in `test_hip_r1_counter_width_unit.py`, 6 in `test_hip_locator_set_length_unit.py` -- fail against the unfixed code and pass against the fix; the three HIP modules stay at 100% coverage, and `tests/` subtests move 586 to 623. `EXPECTED_FAILURES` is unmoved at 44 entries: `hip-parameter/R1_Counter` stays, because code 128 registers no schema of its own -- `R1CounterParameter` declares only `code=129` -- which is a separate registry defect this does not touch, filed as #690. `HIP_COPIES` also stays at two, now for no defect at all rather than for the ones #651 and this fix removed; dropping it is #689, deferred because it would halve every `hip-parameter` frame the fixture above reads (#672, #679). +- **Fixed** -- three test comments stated an exact count of committed captures under `examples/captures/`, a number that drifts every time that directory's tracked set changes and in one case was already wrong when written. `tests/_tiers.py` cited "six," correct at the time; `tests/test_tier_guard.py` said "the moment a seventh capture is committed" -- the same count, as an ordinal rather than the word "six"; `tests/integration/_helpers.py` said "four," which undercounted the tracked set the day it was written. All three now describe the invariant instead of a number that has to be kept in sync with it by hand: `_tiers.py` reads "the moment somebody commits another capture... or stops committing one," `_helpers.py` reads "...are fixtures -- some of them committed --...," and `test_tier_guard.py` drops its "seventh capture" phrasing the same way. Prose-only, verified with `coverage.parser.PythonParser` that no executable statement moved (#700). +- **Fixed** -- `test_every_tracked_name_exists_and_matches_git` in `tests/test_tier_guard.py` checked neither the git index nor the filesystem despite its name: it asserted only that the tracked-capture list was non-empty and that no name contained a `/`, so a hardcoded list would pass it whether or not it matched reality. Sibling `test_capture_suggestions_are_captures` had the same gap. Found while cross-reviewing #703, which is itself prose-only and left this out of scope; the fix rides in as the second and third commits on that PR's branch, whose own description names only #700. `test_every_tracked_name_exists_and_matches_git` now re-derives the expected set independently -- shelling out to `git ls-files -z` under `examples/captures/` rather than calling back into `_tiers.py` -- and asserts equality against it rather than against itself, plus a per-name `Path.is_file()` check the old version never made. Its sibling `test_capture_suggestions_are_captures` gets a narrower fix: it asserts equality against `committed_capture_names()`'s own filter re-implemented inline over `_tiers.committed_captures()`, which catches a wrong filter but, unlike its sibling, still trusts `committed_captures()` for the tracked set itself rather than re-deriving it from git. A cross-review of the first version of this fix found it wrong twice over: a stale docstring in `_tiers.py` still claimed `test_tier_guard.py` "only stats `in.pcap`," falsified by the new per-name loop; and the new `assertEqual` on `test_capture_suggestions_are_captures` was vacuous on an empty tracked set (`() == ()` passes trivially), having dropped the non-emptiness guard the other test kept. Both are corrected in a follow-up commit. Even after the fix, a hardcoded literal that happens to match today's tracked names still passes -- the test detects a set that is wrong *at the moment it runs*, not hardcoding as a practice, which turns a permanent blind spot into a tripwire that fires on the next change to the tracked set (#708). +- **Changed** -- reconciled which private (`_xxx`) attributes and methods the Sphinx build documents, replacing an ad hoc mix with one stated rule: document the contract, hide the recipe. A member every subclass must implement, or one whose shape a caller genuinely depends on, stays documented even though its name starts with an underscore; a private helper that exists only to keep one method short does not. Seven module-level directives naming pure implementation detail were dropped -- `esp._resolve`, `esp._CRYPTO`, `ngap._convert`, `ngap._revert`, `ngap._PYCRATE`, `ngap._PDU_LOCK` and `pypcapfile._NamedStream` (whose nested `name`/`read` members go with it, being reachable only through a private class) -- while 39 `autoattribute` directives were added for class-private state that *is* contract: `Extractor._flag_f`, the third-party engines' own `_backend`, `TraceFlow`'s internal fields, and `FieldBase`/`Field` internals among them; the built-in `PCAP` and `PCAPNG` engines gained a different set of their own -- `_gbhdr`, `_vinfo`, `_dlink` and `_nnsec` for the former, `_ctx` and `_ctx_list` for the latter -- and no `_backend` at all. The 128 runtime definitions of `_missing_` -- 121 under `pcapkit.const`, the other 7 inline in `pcapkit.protocols` -- gained one unified write-up in place of a directive per class: a new "Unrecognised Values" section in `docs/source/pcapkit/const/index.rst`, cross-referenced from `registry.rst`, since `conf.py` already excludes `_missing_` from every `autoclass` via `exclude-members` in `autodoc_default_options` -- there is no `automodule` directive anywhere under `docs/source/`. `CONTRIBUTING.md` gained the rule itself as a named section, so the next directive gets judged against a written test rather than against precedent. Verified with `sphinx-build -b html` under `PCAPKIT_SPHINX=1`: 55 warnings on `main` before this change, 56 after, the one addition being a pre-existing bare `Type` cross-reference ambiguity newly rendered by `TraceFlow._foutio`'s new directive rather than a defect this change introduced -- #709 tracks it. No line under `pcapkit/` changed (#684). +- **Fixed** -- `tests/dumpkit/test_nameless_enum_rendering_unit.py` (added by #670) carried two probes on the same wrong assumption that every flag-enum registry is 16 bits wide: `test_scalar_return_renders_a_nameless_member_as_its_value` checked one fixed tuple ending in `65536` against `tcp.flags.Flags` and its own `StdFlags` stand-in alone, and `test_no_flag_registry_renders_the_literal_none` swept every flag-enum registry instead, but only against `registry(0)`. Seven registries are not *all* 16 bits, at four distinct widths: `ftp.command.CommandType` (3 bits), `reg.apptype.TransportProtocol` (4, computed at runtime), `mh.binding_ack_flag.BindingACKFlag`, `mh.handover_ack_flag.HandoverACKFlag` and `mh.handover_initiate_flag.HandoverInitiateFlag` (8 each), and `mh.binding_update_flag.BindingUpdateFlag` with `tcp.flags.Flags` (16, the two the old probe actually fit). `tcp.flags.Flags(65536)` correctly raises -- its own `_missing_` bounds itself to `0 <= value <= 0xFFFF` -- so the sweep was failing on correct behaviour rather than reporting a defect. A new `_field_mask` derives each registry's own all-ones bound from its declared members, and `_nameless_values` returns the values that bound admits but no member names -- zero, each undeclared bit alone, and every undeclared bit combined -- in place of the one hard-coded tuple. A new `test_a_value_past_the_field_is_refused_rather_than_rendered` pins the boundary directly: the widest in-field value is accepted, one past it is refused. The file goes from 1 failed / 5 passed / 16 subtests to 6 passed / 64 subtests. A cross-review found that all seven guard `raise` lines this reaches -- `tcp/flags.py`'s own included -- were already covered by passing tests from #677, predating this fix; none was genuinely newly reached by it. No line under `pcapkit/` changed (#702). +- **Changed** -- `examples/captures/out.json`, `out.plist`, `out.txt` and `pcapng.txt` are no longer tracked in git; they are build output, not fixtures, and a tracked rendering with no reader goes stale silently every time the code that produces it changes. `pcapng.txt` is exactly that: it still recorded `packet -> NIL` for the four Enhanced Packet Blocks of `dhcp.pcapng` long after #683 gave those blocks their captured octets back, and nothing had regenerated it. Rather than regenerate it once more and leave the same drift free to recur, the four files are removed from the index and folded into `examples/captures/`'s existing blanket `.gitignore` rule; `in.pcap` and `dhcp.pcapng`, the genuine inputs, stay tracked. `examples/legacy_smoke/Makefile` and its `README.rst` are reworded to describe regenerating these reports via `make fixtures` rather than implying they ship committed, and a new `tests/project/test_capture_tracking.py` (5 tests, 8 subtests) pins the invariant going forward; 2 of the 5 tests (5 of the 8 subtests) fail against the pre-change tree with the four reports restored to the index, the other 3 passing on both trees by construction (#685). +- **Fixed** -- two `#:` autodoc comments in `pcapkit/foundation/traceflow/traceflow.py` named a bare `Type`, which Sphinx's cross-reference resolver resolves against every class named `Type` in the project rather than against `typing.Type` -- there are five -- and silently linked to `pcapkit.const.l2tp.type.Type`, an L2TP field-type enum with nothing to do with dumpers. Line 406 (`#: Type[Dumper]: Dumper class.`) is the live case: once #684 rendered `TraceFlow._foutio`, the built docs pointed a reader at the wrong class with no warning that anything had gone sideways. Both sites now spell it `~typing.Type[Dumper]`, the same form eight other files in the tree already use for the identical ambiguity. Line 146 (the first line of `__output__`'s `#:` block, which continues through line 148) is fixed for the same reason but is currently inert: the actual `# type:` comment that drives that signature is sixteen lines below, at line 162, and it spells the identical bare `Type[Dumper]` -- so it is not correct either. The mechanism is likely `sphinx_autodoc_typehints` rather than `napoleon_attr_annotations`: Napoleon's own attribute-annotation lookup fires only for a Google/Numpy `Attributes:` section, and `__output__` is documented through autodoc's `#:` scraping instead. This half of the fix is insurance against the day something reads it cleanly. **This does not close #709.** Four more bare `Type` sites -- hand-written `:type:` fields at `docs/source/pcapkit/foundation/engines/engine.rst:40`, `.../reassembly/reassembly.rst:33` and `:43`, and `.../traceflow/traceflow.rst:40` -- carry the same ambiguity and are not touched here; #709 stays open for them, though PR #714 covers exactly those four sites (#709). +- **Fixed** -- `register_protocol`'s overwrite warning could claim a protocol was replaced with itself. The guard that decides *whether* to warn was already correct -- `incumbent is not protocol`, an identity check from #681 -- but the message built both operands with a bare `repr()`, and for an ordinary class that is just ``. A factory that defines a same-named, closure-local class on every call (as `tests/protocols/test_construction_keyword_check_unit.py`'s `_protocol_class` does) produces two genuinely distinct classes sharing one `__module__` and `__qualname__`, so both reprs print identically and a real, correct overwrite reads as "overwriting X with X." The message now compares the two reprs first, and only when they coincide appends each object's `id()` to tell them apart; the common case, where the two classes are named differently, is untouched. `__module__`/`__qualname__` was considered as the disambiguator instead of `id()` and rejected: for the reported shape those are exactly what the coinciding repr already renders, so they discriminate nothing an `id()` does not already have to. A new `test_register_protocol_disambiguates_classes_sharing_a_repr` fails against the unfixed message and passes against the fix; the targeted suite goes 49 to 50 passed, and the file's coverage holds at 97% (#710). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). From e88a604b121677b7b13e7069b38320b5a7072e3f Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 16:54:22 -0400 Subject: [PATCH 41/46] docs(changelog): fix eight defects a second cross-review found in the entries - Delete the #661/#667 entry: it claims `_Missing` "carries `@final` and defines `__bool__`", but PR #667, which would have added those, was closed unmerged as a verbatim werkzeug port stays verbatim. Unlike the other forward-looking entries this one can never come true, so the claim is removed rather than reworded. - #709 entry: the `__output__` `#:` block runs through line 149, not 148; dropped the unsupported claim that line 162's `# type:` comment "drives that signature" via `sphinx_autodoc_typehints` -- `__output__` carries no runtime annotation, so nothing reads it that way. - #684 entry: `_dlink` is documented on `PCAP` and three third-party engines alike, not built-in-only; narrowed "the third-party engines' own `_backend`" to the two (`PyPCAP`, `PCAP_CT`) that actually have one. - #685 entry: corrected "5 of the 8 subtests" to "4 of the 12 subtests", derived from `test_capture_tracking.py`'s four `TrackedCaptureTests` methods against the pre-change tree. - #708 entry: PR #703's body now reads "Fixes #700. Fixes #708."; it also carries `tests/test_tier_guard.py` +67/-3, so it is not prose-only. - #702 entry: reattributed the seven guarded registries -- #677 added three of the seven guards itself, the four `mh.*` ones came from #632 -- while #677's own test file is what sweeps all seven. Build: `python util/changelog_md.py --check` now exits 1 (expected -- the regeneration is the next commit). Refs #657 --- docs/source/changelog/1.5.0.rst | 68 +++++++++++++-------------------- 1 file changed, 27 insertions(+), 41 deletions(-) diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index df653eb9c..8d56c8a03 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -1567,25 +1567,6 @@ pull requests between #326 and #509. longer for this reason but for ``R1_COUNTER``'s four-octet ``counter`` where :rfc:`7401#section-5.2.3` requires eight, which this defect had been masking (#651). -* **Fixed** -- the ``_missing`` sentinel behind ``MultiDict.pop`` and - ``OrderedMultiDict.pop`` answered ``True`` to ``bool()``, stating the - opposite of what it means, and its class was subclassable. Both now follow - ``NoValueType`` in ``pcapkit.corekit.fields.field``, the package's convention - for a marker of this kind: ``_Missing`` carries ``@final`` and defines - ``__bool__`` returning ``False``. The marker is not private in practice -- - it is the runtime default of both ``pop()`` methods, so - ``inspect.signature(MultiDict.pop).parameters['default'].default`` hands it to - any caller who asks, and ``if default:`` on that value reported "a default was - supplied" where none had been. No behaviour changes: both ``pop()`` - implementations decide by identity (``default is not _missing``) and never by - truthiness, so ``repr``, ``__reduce__``, the pickle bytes at every protocol, - and every ``pop()`` result -- including the falsy defaults ``None``, ``False``, - ``0``, ``''`` and ``[]`` -- are unchanged, and ``pop()`` structurally cannot - return the marker itself. This closes the second of the three ``object()`` - sentinel sites the sweep for #640 left open; the first needed nothing, and - ``_NOT_FOUND`` in ``pcapkit.utilities.compat`` is deliberately left alone as a - verbatim line of CPython's own ``functools.cached_property``, inside a - ``sys.version_info < (3, 8)`` branch no supported interpreter reaches (#661). * **Fixed** -- two HTTP/2 flag defects, one on each side of a round trip. **This changes the octets a reconstructed DATA frame emits, and the dumped value of a flagless frame's flags.** ``_make_http_data`` was the only one of @@ -2284,9 +2265,9 @@ pull requests between #326 and #509. was non-empty and that no name contained a ``/``, so a hardcoded list would pass it whether or not it matched reality. Sibling ``test_capture_suggestions_are_captures`` had the same gap. Found while - cross-reviewing #703, which is itself prose-only and left this out of - scope; the fix rides in as the second and third commits on that PR's - branch, whose own description names only #700. + cross-reviewing #703's first commit, itself prose-only and out of scope + for this; the fix rides in as that PR's second and third commits, and + #703's own description names both #700 and #708. ``test_every_tracked_name_exists_and_matches_git`` now re-derives the expected set independently -- shelling out to ``git ls-files -z`` under ``examples/captures/`` rather than calling back into ``_tiers.py`` -- and @@ -2320,11 +2301,14 @@ pull requests between #326 and #509. (whose nested ``name``/``read`` members go with it, being reachable only through a private class) -- while 39 ``autoattribute`` directives were added for class-private state that *is* contract: ``Extractor._flag_f``, - the third-party engines' own ``_backend``, ``TraceFlow``'s internal fields, - and ``FieldBase``/``Field`` internals among them; the built-in ``PCAP`` and - ``PCAPNG`` engines gained a different set of their own -- ``_gbhdr``, - ``_vinfo``, ``_dlink`` and ``_nnsec`` for the former, ``_ctx`` and - ``_ctx_list`` for the latter -- and no ``_backend`` at all. The 128 + the ``PyPCAP`` and ``PCAP_CT`` engines' own ``_backend`` -- the only two of + the six third-party engines that have one -- ``TraceFlow``'s internal + fields, and ``FieldBase``/``Field`` internals among them; the built-in + ``PCAP`` and ``PCAPNG`` engines gained ``_gbhdr``, ``_vinfo`` and + ``_nnsec`` for the former and ``_ctx``/``_ctx_list`` for the latter, and + no ``_backend`` at all. ``_dlink`` is not built-in-only: ``PCAP`` documents + it alongside the three third-party engines that share it, ``PyPCAP``, + ``PCAP_CT`` and ``PyPCAPFile``. The 128 runtime definitions of ``_missing_`` -- 121 under ``pcapkit.const``, the other 7 inline in ``pcapkit.protocols`` -- gained one unified write-up in place of a directive per class: a new "Unrecognised Values" section in @@ -2365,9 +2349,13 @@ pull requests between #326 and #509. is refused. The file goes from 1 failed / 5 passed / 16 subtests to 6 passed / 64 subtests. A cross-review found that all seven guard ``raise`` lines this reaches -- ``tcp/flags.py``'s own included -- were - already covered by passing tests from #677, predating this fix; none was - genuinely newly reached by it. No line under ``pcapkit/`` changed - (#702). + already covered by a passing test predating this fix, though not from one + PR: #677 itself supplies three of the seven guards (``CommandType``, + ``TransportProtocol`` and ``Flags``), and the other four are the ``mh.*`` + flag guards #632 added for #623; #677's ``test_const_enum_builtin_parity.py`` + sweeps all 123 registries, which is what reaches all seven regardless of + which PR guarded each. None was genuinely newly reached by this fix. No + line under ``pcapkit/`` changed (#702). * **Changed** -- ``examples/captures/out.json``, ``out.plist``, ``out.txt`` and ``pcapng.txt`` are no longer tracked in git; they are build output, not fixtures, and a tracked rendering with no reader goes stale silently every @@ -2382,7 +2370,7 @@ pull requests between #326 and #509. reworded to describe regenerating these reports via ``make fixtures`` rather than implying they ship committed, and a new ``tests/project/test_capture_tracking.py`` (5 tests, 8 subtests) pins the - invariant going forward; 2 of the 5 tests (5 of the 8 subtests) fail + invariant going forward; 2 of the 5 tests (4 of the 12 subtests) fail against the pre-change tree with the four reports restored to the index, the other 3 passing on both trees by construction (#685). * **Fixed** -- two ``#:`` autodoc comments in @@ -2396,16 +2384,14 @@ pull requests between #326 and #509. with no warning that anything had gone sideways. Both sites now spell it ``~typing.Type[Dumper]``, the same form eight other files in the tree already use for the identical ambiguity. Line 146 (the first line of - ``__output__``'s ``#:`` block, which continues through line 148) is fixed - for the same reason but is currently inert: the actual ``# type:`` comment - that drives that signature is sixteen lines below, at line 162, and it - spells the identical bare ``Type[Dumper]`` -- so it is not correct either. - The mechanism is likely ``sphinx_autodoc_typehints`` rather than - ``napoleon_attr_annotations``: Napoleon's own attribute-annotation lookup - fires only for a Google/Numpy ``Attributes:`` section, and ``__output__`` - is documented through autodoc's ``#:`` scraping instead. This half of the - fix is insurance against the day something reads it cleanly. **This does - not close #709.** Four more + ``__output__``'s ``#:`` block, which continues through line 149) is fixed + for the same reason but is currently inert: ``__output__`` carries no + runtime annotation (``TraceFlowBase.__dict__['__annotations__']`` is + empty), and the ``# type:`` comment sixteen lines below, at line 162, + spells the identical bare ``Type[Dumper]`` -- so whatever eventually + renders this attribute's type still has the same ambiguity to resolve. + This half of the fix is insurance against the day something reads it + cleanly. **This does not close #709.** Four more bare ``Type`` sites -- hand-written ``:type:`` fields at ``docs/source/pcapkit/foundation/engines/engine.rst:40``, ``.../reassembly/reassembly.rst:33`` and ``:43``, and From 5301a3829e384304f7a52478cf96b32f1f12315c Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 16:54:54 -0400 Subject: [PATCH 42/46] chore(changelog): regenerate CHANGELOG.md for the eight-defect fix python util/changelog_md.py, in step with 1.5.0.rst at e88a604b1. Refs #657 --- CHANGELOG.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a56718879..5db8bd381 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -93,7 +93,6 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Changed** -- building a protocol through its constructor with a keyword that names nothing now raises `UnsupportedCall` instead of discarding it. **This is a behaviour change to a public API**: every `make` in the tree ends its signature with `**kwargs` and reads nothing out of it, so until now a misspelled keyword was accepted, dropped, and the field it named kept its default -- wrong octets, with nothing said. That is what #602 cost: `examples/generators/options.py` asked for `seq=1` where `TCP.make` spells the parameter `seq_no`, and 25 generated fixture frames carried sequence number `0` against an empty `warnings` list. #541 and #556 were the same silence. The schema layer has never been so permissive -- `Schema.__update__` warns `UnknownFieldWarning` for a field it does not know -- and the asymmetry between the two halves of the same construction is what this closes. Checked in `ProtocolBase.__init__` rather than in `make`, because `make` is not the only consumer of the keywords it is handed: `__post_init__` passes one `**kwargs` to the construction *and* to the parse of what it has just constructed, so a keyword declared only by `read` legitimately travels through `make` -- `HIP.read` declares `extension` where `HIP.make` does not, and the option generator depends on it. The accepted set is therefore the union of every keyword-taking parameter of `make`, `read`, `pack`, `unpack`, `__post_init__` and `__init__` across the whole MRO, computed once per class from `inspect.signature`. Parsing is deliberately untouched, since there the keywords are whatever the engines and the four `_import_next_layer` implementations forward and a protocol cannot know which of its ancestors' its parent passed on -- and nothing was ever lost that way, a dropped parse keyword changing how a packet is read rather than what its octets say. Two escapes exist for the shapes a signature cannot express, both opt-in per class through a new `__keywords__`: a set, for a keyword read out of `**kwargs` by name as `ESP.read` does with `packet`; and `None`, for a dispatcher whose real signature belongs to a class chosen at call time, which is exactly `HTTP.make` forwarding to `HTTPv1`/`HTTPv2` and the one place in the tree that uses it. The message names the near neighbour it found, so `seq` reports *did you mean 'seq_no'?*. `from_data` warns `UnknownFieldWarning` where a caller would be raised at, because the keywords there are whatever `_make_data` returned rather than anything anybody typed, so the defect is a key of that mapping disagreeing with the signature it is spread into and the person who meets it is not the person who can fix it. Expect this to surface latent bugs in code that has been quietly losing a field, which is the point. It surfaced four in this repository, all the residue of #602 and all fixed here: the `_TCP_BASE` of `examples/generators/dispatch.py` and three stale copies of it under `tests/protocols/transport/`, each still passing `seq`/`ack_flag`/`urgent_pointer` and so building segments with sequence number `0` where they read as `1`. It surfaced three more that are reported rather than fixed, being a defect per protocol rather than one in this mechanism: `Frame._make_data` returns `ts_src` where `make` declares `ts_sec`, `L2TPv2._make_data` returns `prio` where it declares `priority`, and `Header._make_data` returns a `magic_number` that `Header.make` does not take at all -- so `from_data` has been dropping a frame's timestamp, an L2TPv2 priority bit and a capture's byte order, and now says so. One limitation worth knowing rather than discovering: a *direct* `SomeProtocol.make(...)` call is not checked and still discards in silence, since the check sits where every producer's keywords converge rather than inside each of the 30 `make` implementations -- `object.__new__(cls).make(**kwargs)` is the idiom that reaches it, and `HTTP.make` uses it to reach its versioned implementation (#617). - **Fixed** -- the HIP `PUZZLE` and `SOLUTION` builders derived three wire-format values from the payload value instead of taking them from the data model, and all three were wrong. **This changes two public data models and the octets both parameters emit.** The width of `Random #I`, and of `Puzzle solution #J`, came from `int.bit_length()` alone, and nothing else was available to derive it from, so every leading zero octet was dropped on re-serialisation: a `SOLUTION` read with `Length = 20` rebuilt as `Length = 6`, a `PUZZLE` read with `Length = 12` as `Length = 5`, and silently, because the integers survive and nothing raises. That width is `RHASH_len / 8` octets [[RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4), [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5)], a property of the Responder's HIT Suite rather than of the number that happens to sit in the field, so both data models now carry it as `rhash_len` and both builders prefer it; a new `rhash_len=` keyword states it for a build from scratch, which under HIPv2 is the only place it can come from, since `RSA,DSA/SHA-256` is the REQUIRED HIT Suite [[RFC 7401 Section 5.2.10](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.10)] and makes the field 32 octets rather than 8. On the mandatory path roughly one parameter in 256 has a zero top octet and lost it (#653). `SOLUTION`'s second contents octet is `Reserved`, "zero when sent, ignored when received" [[RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5), and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) identically] -- not the `Lifetime` that only [RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4) defines, and that pcapkit was encoding there as `2^(value - 32)` seconds. It wrote `0x20`, `0x21`, `0x25` or `0x2b` into a field the RFC requires to be zero, and could not write that zero at all: an RFC-conformant `SOLUTION` whose `Reserved` is `0x00` parsed to `timedelta(0)` and then could not be re-serialised, escaping a bare `ValueError` from `math.log2(0.0)` that was not a `BaseError` and so bypassed the library's own error handling entirely. The field is renamed `reserved`, defaults to the mandated zero, and is carried verbatim across a round trip rather than re-derived (#654). And neither builder read its own `version` keyword, so `version=1` and `version=2` computed identical lengths at every bit width, where [RFC 5201 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.4) and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) state both fields as literally 8 bytes and `Length` as literally 12 and 20. Under HIPv1 each builder therefore accepted only a `bit_length()` of 57..64 and built, for everything narrower, a parameter this library's own reader rejects -- the same shape as #608, in the `PUZZLE` builder that #629 never opened (#655). The two remaining `math.log2` sites, both in `PUZZLE` where a lifetime is real, now raise `ProtocolError` rather than letting `ValueError` escape; `ProtocolError` is `(BaseError, ValueError)`, so a caller written around the old bare exception still catches it. Migration: `SolutionParameter.lifetime` is now `reserved` and an `int` rather than a `timedelta`; both `PuzzleParameter` and `SolutionParameter` gained a required `rhash_len`; and `_make_param_solution` no longer takes `lifetime=`. All three were only reachable end to end once #608 was fixed in #629, which removed the parity guard that had been failing these rebuilds loudly first -- so the round trip stopped raising and started quietly emitting a different parameter, which is why they were worth fixing together (#653, #654, #655). - **Fixed** -- every HIP parameter pcapkit emitted was `4 (mod 8)` octets long, because the padding was computed to align the *contents* rather than the record. Corrected for **45 of the 46** HIP parameters; `LOCATOR_SET` is deliberately excluded, for the reason below. **This changes the octets those 45 parameters write, and the `length` their data models report.** [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1) requires that "all of the encoded TLV parameters have a length (that includes the Type and Length fields), which is a multiple of 8 bytes", and states the arithmetic outright as `Total Length = 11 + Length - (Length + 3) % 8`; all 95 padding sites instead computed `(8 - (Length % 8)) % 8`, which has no `+ 4` inside the modulus and so aligns the contents alone. Across every `Length` from 0 to 63 the result was never a multiple of eight and never the value the RFC gives, and it erred in both directions: at `Length = 4` -- a whole `SEQ`, and [RFC 7401 Section 5.3.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.3.5) puts a `SEQ` or an `ACK` on every `UPDATE` -- the record is complete in eight octets and pcapkit appended four that must not be there, while at `Length = 8` the contents were already 8-aligned, nothing was appended, and the record went out four octets short. A conformant peer reading `Length` and consuming `11 + Length - (Length + 3) % 8` octets therefore lands mid-parameter and reads the rest of the parameter area at a wrong offset, in both directions; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test in the suite could see this and why the fix is asserted against the RFC's arithmetic rather than against a round trip. 93 of the 95 sites -- 45 of the 46 `PaddingField` callbacks in `pcapkit/protocols/schema/internet/hip.py` and 48 of the 49 record lengths in `pcapkit/protocols/internet/hip.py` -- are now one `parameter_total_len` and one `parameter_padding_len`, stating the RFC formula once instead of 95 times (#651). `LOCATOR_SET` keeps the old expression at both of its sites, on purpose, because two defects there cancel each other exactly and correcting only the padding would break a parameter that is currently right: its padding callback never receives the parameter's `len` (the nested `Locator` schemas share a packet context whose own `len` shadows it, so the value seen is always 4 for an IPv6 locator), and the parameter's `len` is written in 4-octet units where the RFC's `Length` is a byte count. Always-4 padding gives `4 + 24n + 4`, and because `24n` is a multiple of 8 the RFC total for a byte-count `Length` of `24n` is the same `24n + 8` -- measured at n = 1, 2 and 5 as 32, 56 and 128 octets both before and after. #679 carries the pair. `EncryptedParameter`'s `data` length callback is fixed in the same change and could not have been left: it subtracted the sixteen `iv` octets but not the four `reserved` ones, and those four cancelled the padding's four at `Length % 8` in `{0, 5, 6, 7}` -- so correcting the padding alone would have taken `ENCRYPTED` from right at four of the eight residues to four octets too long at all eight. `HIP.make`'s `len = total_length // 8 + 4` is *not* part of the defect and is unchanged: [RFC 7401 Section 5.1.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.1.3) defines Header Length as the header and parameters "in 8-byte units, excluding the first 8 bytes", so the floor division is exact once each parameter is a multiple of eight, where before it was exact only for an even number of them. Migration: a `SEQ` parameter's `length` is now 8 where it was 12, and the other 44 corrected parameters move likewise, so code comparing stored pcapkit output byte for byte, or asserting on `Data_*Parameter.length`, sees different values -- the RFC's values. `LOCATOR_SET` is unchanged in both respects. `examples/generators/options.py`'s `HIP_COPIES` stays at two, no longer for this reason but for `R1_COUNTER`'s four-octet `counter` where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires eight, which this defect had been masking (#651). -- **Fixed** -- the `_missing` sentinel behind `MultiDict.pop` and `OrderedMultiDict.pop` answered `True` to `bool()`, stating the opposite of what it means, and its class was subclassable. Both now follow `NoValueType` in `pcapkit.corekit.fields.field`, the package's convention for a marker of this kind: `_Missing` carries `@final` and defines `__bool__` returning `False`. The marker is not private in practice -- it is the runtime default of both `pop()` methods, so `inspect.signature(MultiDict.pop).parameters['default'].default` hands it to any caller who asks, and `if default:` on that value reported "a default was supplied" where none had been. No behaviour changes: both `pop()` implementations decide by identity (`default is not _missing`) and never by truthiness, so `repr`, `__reduce__`, the pickle bytes at every protocol, and every `pop()` result -- including the falsy defaults `None`, `False`, `0`, `''` and `[]` -- are unchanged, and `pop()` structurally cannot return the marker itself. This closes the second of the three `object()` sentinel sites the sweep for #640 left open; the first needed nothing, and `_NOT_FOUND` in `pcapkit.utilities.compat` is deliberately left alone as a verbatim line of CPython's own `functools.cached_property`, inside a `sys.version_info < (3, 8)` branch no supported interpreter reaches (#661). - **Fixed** -- two HTTP/2 flag defects, one on each side of a round trip. **This changes the octets a reconstructed DATA frame emits, and the dumped value of a flagless frame's flags.** `_make_http_data` was the only one of the six `_make_http_*` methods that never read `frame.flags`, so a DATA frame parsed with `END_STREAM` set rebuilt with the bit clear: the flags octet went `0x01` to `0x00`, silently, producing a well-formed frame saying the stream continues where the capture said it ended [[RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1)]. `PADDED` was never affected, being re-derived from `pad_len` rather than read back, and all five sibling builders already restored theirs -- so this was a missing read-back rather than a design, and it now reads `frame.flags.END_STREAM` the way they do (#652). Separately, `FrameType.post_process` seeded its flag accumulator with a bare `0`, and `|=` promotes that only as a side effect, so a frame with **no** bit set left `__flags__` a plain `int` where both the schema and the data model declare a `Flags` -- and `END_STREAM in __value__` raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering `False`. That is the common case rather than an edge one: a `0x00` flags octet is every opening `SETTINGS`, every non-final `DATA` and every non-ACK `PING`. It now seeds `self.Flags(0)`, which the construct path had been doing correctly at all six of its own sites. The visible consequence is in the dump, where `__value__` was a JSON *number* for a flagless frame and a JSON *string* for every other frame in the same capture; it is consistently a string now, `"Flags::None [0]"` where it read `0`, the same shape of change #616 made to TCP's `connection`. The seed is guarded rather than unconditional because `FrameType.Flags` declares no members and, from Python 3.11, a memberless `enum.Flag` subclass refuses `Flags(0)` outright, so the five frame schemas that inherit it -- `UnassignedFrame`, `PriorityFrame`, `RSTStreamFrame`, `GoawayFrame` and `WindowUpdateFrame` -- keep the plain `int`, which nothing observes since all five pass `flags=None` into their data objects. Note that a DATA round trip is **still** lossy after this, for an unrelated reason found while measuring it and left to its own change: three payload length callbacks in the frame schemas are mis-parenthesised, so an unpadded `DATA`, `HEADERS` or `PUSH_PROMISE` frame parses with its whole payload dropped (#668). (#652, #650). - **Fixed** -- a flag value with no declared bits no longer dumps as `Type::None [0]`. **This changes user-visible output in every textual dump format.** `make_dumper`'s `object_hook` renders each enumeration member as `Type::name [value]` and interpolated the name unguarded, but a `Flag` value composed entirely of undeclared bits has `name is None` rather than a string -- so the literal four characters `None` landed in the name half and `Flags(0)` rendered as `Flags::None [0]` in `json`, `tree`, `text`, `txt`, `plist` and `xml`, out of both `Extractor` and `TraceFlow`. A nameless member now renders with the value's own decimal spelling instead, so `Flags(0)` gives `Flags::0 [0]` and `Flags(8)` gives `Flags::8 [8]`. That is the spelling the enumeration libraries already use for an undeclared residue -- `Flags(2057).name` is `ACK|9`, naming the declared bit and giving the leftovers as one decimal number -- and it cannot be mistaken for a member name, since a Python identifier may not begin with a digit, whereas `NONE` is a real declared name elsewhere in the library. Three sites carried the identical interpolation, not one: the `OrderedMultiDict` key path, the `addon` branch and the scalar return, which now share one `render_enum` helper so the guard cannot be applied to two of three. That guard is on `name is None` and not on the value, because the defect never was about zero -- `Flags(1)`, `Flags(8)`, `Flags(9)` and `Flags(65536)` are equally nameless -- and it is not an `aenum` quirk either, since a stdlib `enum.IntFlag` answers `name is None` at the same values. Five of the library's seven flag registries are nameless at zero rather than the one reported: `Flags` plus the four Mobility Header flag registries, the other two declaring an explicit `undefined = 0`. Pre-existing since 2023-04-28 and surfaced rather than caused by #634, which made a flagless TCP segment reach this path. The committed example dumps do not move, because no member in them lacks a name (#648). - **Fixed** -- four MPTCP error messages carried a doubled separator, rendering as `TCP: : [OptNo 30] 1: invalid flags combination` with an empty field between the protocol alias and the option number. Cosmetic: the exception type, the option number and the subtype were always right, and nothing downstream parses these strings, but it is the text a user sees when an MP_JOIN or DSS option is rejected and an empty field reads as a value that failed to interpolate. Four sites, not the one reported -- `_read_mptcp_join` and `_make_mptcp_join` for the invalid flag combination, and both guards of `_make_mptcp_dss` for the missing required fields -- against 28 messages in the same file that already spelled the prefix correctly, so the four were outliers against a house form in their own module and the counts are now 32 correct against 0 doubled. Pre-existing since 2023-04-10. The new test asserts each message in full rather than by substring, which is what the existing `assertIn` on the message body could not do, since it passes either side of the change (#649). @@ -110,11 +109,11 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- every EOF-truncated PCAP-NG file raised a bare `ValueError` out of `Extractor`, so the whole extraction was lost rather than the one truncated block. The root is one layer above the reported site: Block Total Length is cross-checked against its own trailing copy and never against the file, so a last block declaring more than the file holds left `PCAPNG.read` seeking *past* the real end -- legal and silent -- and the next block read then measured a **negative** remainder, since `prepare` derives it as the end of the stream less the current position. That negative reached `SchemaField` through `pcapng_block_selector` and `io.RawIOBase.read` refused it, which is why nearly every truncation level failed rather than only the one holding the cut. The seek is now clamped to the octets `_read_fileng` actually returned, with a `ProtocolWarning` naming the overrun, and a tail under the twelve octets a block needs at minimum is reported as the quiet `StreamEOFError` that `Extractor.record_frames` already catches -- the one case that is deliberately not a clamp, because at the end of the file no block is being read and clamping would fabricate a frame out of zero padding. A second, independent route to the same shape is closed with a new `nonnegative()` helper, composed into `bounded_option` and `bounded_area` and applied to the eight unwrapped `length - N` spans and the eight `__option_padding__`-sized padding fields: `_TextField.__call__` builds its `struct` template as `f'{length}s'` unconditionally, so a negative became the format `'-8s'` and `struct.calcsize` raised -- neither that nor the `ValueError` being one of `pcapkit.utilities.exceptions`, and neither being an `EOFError`, so neither was caught where the frame loop catches the end of a file. 28 of the module's 74 length callbacks returned a negative before and none do now, from `-1` on an `epb_hash` declaring no payload to `-16777248` on an Enhanced Packet Block's option area; the 200-block, 8,048-octet `captured_len = 0xFFFFFF` vector that reached `struct` through `bounded_area`'s own `nominal <= available` test being true for a negative nominal now parses to 200 frames. Measured over all 1,509 octet boundaries of `examples/captures/dhcp.pcapng` with one harness either side: 6 levels parsed before and 1,495 do now, against 1,479 `ValueError` and 10 `struct.error` before and none from outside the library after, with the frame count degrading monotonically with the cut; of the 14 levels that still raise, twelve leave a file too short to hold a block at all and the other two cut into the Interface Description Block's `if_tsresol` option, so none of them costs a frame that was in the file. That claim is about the truncation sweep rather than about every input: a 4,000-round bounded mutation fuzz either side of the change, same seed and cap, takes the parse rate from 2,118 to 3,438 and removes both of #678's families, and the two foreign families left are measured unchanged -- `ValueError: N is not a valid BlockType`, filed as #701, where an unassigned block type raises from `aenum` inside `EnumField.post_process` so the `UnknownBlock` default the registry declares is unreachable; and `MemoryError` at exactly 30 of 4,000 in both trees, which is #593's 32-bit band. A third bare exception in this module *is* fixed here, found by the cross-review and reachable from valid input rather than truncated: `SystemdJournalExportBlock.post_process` unpacked a 64-bit binary-field length out of whatever `entry_data.read(8)` returned, and since the block body is padded to a 32-bit boundary with NULs that `bytes.strip()` does not strip, that padding was read as a field *name* with no length prefix behind it -- so every journal entry of unaligned length raised a bare `struct.error`, measured on a 14-octet `MESSAGE=hello` entry -- a defect about ordinary input rather than about truncation. A NUL-only line now ends the entry and a short prefix ends it with a warning. The same function leaked two more of the family, found on the cross-review's second pass and fixed with it: a binary field's declared length is the widest in the format and nothing bounded it against the entry holding it, so at `2**63` and above `BytesIO.read` refused it with a bare `OverflowError` while below that it silently returned whatever was there -- the same malformed prefix fatal or invisible by magnitude alone, now clamped to what the entry has left and reported; and a field name, key or value that is not UTF-8 raised a bare `UnicodeDecodeError`, fatal to a whole extraction over one octet in one field, now decoded with `errors='replace'` and reported, which is the option this module's own `StringField` already takes. The separate silent loss of every field after a binary one, from a `read()` that reaches EOF where it means to skip one newline, is filed as #704 rather than fixed. The five deliberately unclamped non-packet option areas keep the per-block framing assumption #676's note describes, which this does not close. Also here because it is the fifth and last registrar in the package with no overwrite guard at all: `Option.register` now reports a displaced option schema as a `RegistryWarning` naming every namespace the `ns='opt'` fan-out displaced something in, once per registration rather than once per namespace, tested by membership rather than by subscripting because the per-namespace registries are plain `defaultdict`\ s and reading one to look would insert `UnknownOption` for a code nobody registered. Presence alone is the firing condition, as on the seven code-keyed parser registrars and unlike the narrower guard #681 gave `register_protocol`, since this key is a caller-supplied `code` that `__init_subclass__` passes once per subclass; a namespace the call itself creates is exempt, because it starts as a copy of `opt`'s defaults and nothing in it is a prior registration. One draft did move `EXPECTED_FAILURES`, and it is worth recording: flooring the three decryption-secrets payloads that read `__length__` *whole* rather than subtracting from it packs nothing, because `Schema.pack` leaves that key at `-1` for "unknown" -- which emptied both payloads and turned `pcapng-secrets/TLS_Key_Log` and `.../WireGuard_Key_Log` from `MISMATCH` to `OK`, an empty payload comparing equal to an empty payload. Reverted and pinned; the entry count holds at 44 with its 35 PCAP-NG cases unmoved. Ships labelled breaking: well-formed captures are byte-identical, verified by regenerating `examples/captures/pcapng.txt` either side of the change, but any truncated PCAP-NG now yields the frames before the cut where it previously raised -- so a caller reading "extraction raised" as "this file is unusable" gets a partial result instead, whose last frame may carry zero-padded octets -- and three exception classes change at the margins, eight short-file depths moving from `ProtocolError: unknown byteorder magic` to `StreamEOFError`. The two modules hold 99.93% coverage with their 25 new statements covered and their single miss flat, and `examples/captures/pcapng.txt` does not move further, leaving the #683 drift #685 tracks exactly as it was (#678). - **Fixed** -- HIP's `R1_COUNTER`/`R1_Counter` parameter packed 12 octets where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires 16, and `LOCATOR_SET` declared its `Length` in 4-octet units where [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1)'s `Length` is a byte count -- two independent defects, fixed together because the evidence for either needed the other out of the way first. `R1CounterParameter`'s `counter` was a `UInt32Field` where the RFC states the R1 generation counter's width twice, as "8 bytes" in the diagram and as "a 64-bit unsigned integer" in prose; both HIP versions whose *packing* reaches this class -- `R1_Counter` (128) and `R1_COUNTER` (129) -- packed a 12-octet record, landing at `4 (mod 8)` instead of aligned. It is a `UInt64Field` now, and the record is 16 octets (#672). `LocatorSetParameter`'s own `Length` was written as `sum(Locator.len)`, in the 4-octet units [RFC 8046 Section 4](https://datatracker.ietf.org/doc/html/rfc8046#section-4) gives `Locator Length`, where this parameter's own `Length` is a byte count; a set of *n* plain IPv6 locators therefore declared `4n` octets against `24n` actually present, and `Schema.unpack` handed the nested `ListField` only the declared octets, so `n = 2` and `n = 5` both parsed one truncated locator and left the rest of the record unconsumed. `HIP._make_param_locator_set` now sums `8 + Locator.len * 4` per locator -- the fixed header plus the RFC 8046 contents each one actually carries (#679). A second, independent defect in the same parameter's padding is fixed alongside it, one #651 deliberately left alone because it happened to cancel this one exactly: the nested `Locator` schemas share the parameter's packet context and pack their own `len` over it, so `padding` -- evaluated after the list -- always saw 4 rather than the parameter's real length. A new `locator_set_len_callback` snapshots `Length` under a private key before any locator packs into the context, and `locator_set_padding_len` reads that snapshot rather than the shadowed one. An RFC-only byte-stride walk over `examples/captures/options-internet.pcap`, independent of pcapkit's own parser, went from 1 violation -- `LOCATOR_SET`'s empty record, concealed until now because `R1_COUNTER`'s `counter` field defaulted to `0` with no override anywhere in `examples/generators/options.py`, a value the width defect could not be told apart from -- to 2 once the counter was patched non-zero, to 0 once both parameters were fixed; the fixture now overrides both `R1_COUNTER` codes with `counter=0xaabbccdd` so a future regression cannot hide behind a zero again. 10 new test methods across two new files -- 4 in `test_hip_r1_counter_width_unit.py`, 6 in `test_hip_locator_set_length_unit.py` -- fail against the unfixed code and pass against the fix; the three HIP modules stay at 100% coverage, and `tests/` subtests move 586 to 623. `EXPECTED_FAILURES` is unmoved at 44 entries: `hip-parameter/R1_Counter` stays, because code 128 registers no schema of its own -- `R1CounterParameter` declares only `code=129` -- which is a separate registry defect this does not touch, filed as #690. `HIP_COPIES` also stays at two, now for no defect at all rather than for the ones #651 and this fix removed; dropping it is #689, deferred because it would halve every `hip-parameter` frame the fixture above reads (#672, #679). - **Fixed** -- three test comments stated an exact count of committed captures under `examples/captures/`, a number that drifts every time that directory's tracked set changes and in one case was already wrong when written. `tests/_tiers.py` cited "six," correct at the time; `tests/test_tier_guard.py` said "the moment a seventh capture is committed" -- the same count, as an ordinal rather than the word "six"; `tests/integration/_helpers.py` said "four," which undercounted the tracked set the day it was written. All three now describe the invariant instead of a number that has to be kept in sync with it by hand: `_tiers.py` reads "the moment somebody commits another capture... or stops committing one," `_helpers.py` reads "...are fixtures -- some of them committed --...," and `test_tier_guard.py` drops its "seventh capture" phrasing the same way. Prose-only, verified with `coverage.parser.PythonParser` that no executable statement moved (#700). -- **Fixed** -- `test_every_tracked_name_exists_and_matches_git` in `tests/test_tier_guard.py` checked neither the git index nor the filesystem despite its name: it asserted only that the tracked-capture list was non-empty and that no name contained a `/`, so a hardcoded list would pass it whether or not it matched reality. Sibling `test_capture_suggestions_are_captures` had the same gap. Found while cross-reviewing #703, which is itself prose-only and left this out of scope; the fix rides in as the second and third commits on that PR's branch, whose own description names only #700. `test_every_tracked_name_exists_and_matches_git` now re-derives the expected set independently -- shelling out to `git ls-files -z` under `examples/captures/` rather than calling back into `_tiers.py` -- and asserts equality against it rather than against itself, plus a per-name `Path.is_file()` check the old version never made. Its sibling `test_capture_suggestions_are_captures` gets a narrower fix: it asserts equality against `committed_capture_names()`'s own filter re-implemented inline over `_tiers.committed_captures()`, which catches a wrong filter but, unlike its sibling, still trusts `committed_captures()` for the tracked set itself rather than re-deriving it from git. A cross-review of the first version of this fix found it wrong twice over: a stale docstring in `_tiers.py` still claimed `test_tier_guard.py` "only stats `in.pcap`," falsified by the new per-name loop; and the new `assertEqual` on `test_capture_suggestions_are_captures` was vacuous on an empty tracked set (`() == ()` passes trivially), having dropped the non-emptiness guard the other test kept. Both are corrected in a follow-up commit. Even after the fix, a hardcoded literal that happens to match today's tracked names still passes -- the test detects a set that is wrong *at the moment it runs*, not hardcoding as a practice, which turns a permanent blind spot into a tripwire that fires on the next change to the tracked set (#708). -- **Changed** -- reconciled which private (`_xxx`) attributes and methods the Sphinx build documents, replacing an ad hoc mix with one stated rule: document the contract, hide the recipe. A member every subclass must implement, or one whose shape a caller genuinely depends on, stays documented even though its name starts with an underscore; a private helper that exists only to keep one method short does not. Seven module-level directives naming pure implementation detail were dropped -- `esp._resolve`, `esp._CRYPTO`, `ngap._convert`, `ngap._revert`, `ngap._PYCRATE`, `ngap._PDU_LOCK` and `pypcapfile._NamedStream` (whose nested `name`/`read` members go with it, being reachable only through a private class) -- while 39 `autoattribute` directives were added for class-private state that *is* contract: `Extractor._flag_f`, the third-party engines' own `_backend`, `TraceFlow`'s internal fields, and `FieldBase`/`Field` internals among them; the built-in `PCAP` and `PCAPNG` engines gained a different set of their own -- `_gbhdr`, `_vinfo`, `_dlink` and `_nnsec` for the former, `_ctx` and `_ctx_list` for the latter -- and no `_backend` at all. The 128 runtime definitions of `_missing_` -- 121 under `pcapkit.const`, the other 7 inline in `pcapkit.protocols` -- gained one unified write-up in place of a directive per class: a new "Unrecognised Values" section in `docs/source/pcapkit/const/index.rst`, cross-referenced from `registry.rst`, since `conf.py` already excludes `_missing_` from every `autoclass` via `exclude-members` in `autodoc_default_options` -- there is no `automodule` directive anywhere under `docs/source/`. `CONTRIBUTING.md` gained the rule itself as a named section, so the next directive gets judged against a written test rather than against precedent. Verified with `sphinx-build -b html` under `PCAPKIT_SPHINX=1`: 55 warnings on `main` before this change, 56 after, the one addition being a pre-existing bare `Type` cross-reference ambiguity newly rendered by `TraceFlow._foutio`'s new directive rather than a defect this change introduced -- #709 tracks it. No line under `pcapkit/` changed (#684). -- **Fixed** -- `tests/dumpkit/test_nameless_enum_rendering_unit.py` (added by #670) carried two probes on the same wrong assumption that every flag-enum registry is 16 bits wide: `test_scalar_return_renders_a_nameless_member_as_its_value` checked one fixed tuple ending in `65536` against `tcp.flags.Flags` and its own `StdFlags` stand-in alone, and `test_no_flag_registry_renders_the_literal_none` swept every flag-enum registry instead, but only against `registry(0)`. Seven registries are not *all* 16 bits, at four distinct widths: `ftp.command.CommandType` (3 bits), `reg.apptype.TransportProtocol` (4, computed at runtime), `mh.binding_ack_flag.BindingACKFlag`, `mh.handover_ack_flag.HandoverACKFlag` and `mh.handover_initiate_flag.HandoverInitiateFlag` (8 each), and `mh.binding_update_flag.BindingUpdateFlag` with `tcp.flags.Flags` (16, the two the old probe actually fit). `tcp.flags.Flags(65536)` correctly raises -- its own `_missing_` bounds itself to `0 <= value <= 0xFFFF` -- so the sweep was failing on correct behaviour rather than reporting a defect. A new `_field_mask` derives each registry's own all-ones bound from its declared members, and `_nameless_values` returns the values that bound admits but no member names -- zero, each undeclared bit alone, and every undeclared bit combined -- in place of the one hard-coded tuple. A new `test_a_value_past_the_field_is_refused_rather_than_rendered` pins the boundary directly: the widest in-field value is accepted, one past it is refused. The file goes from 1 failed / 5 passed / 16 subtests to 6 passed / 64 subtests. A cross-review found that all seven guard `raise` lines this reaches -- `tcp/flags.py`'s own included -- were already covered by passing tests from #677, predating this fix; none was genuinely newly reached by it. No line under `pcapkit/` changed (#702). -- **Changed** -- `examples/captures/out.json`, `out.plist`, `out.txt` and `pcapng.txt` are no longer tracked in git; they are build output, not fixtures, and a tracked rendering with no reader goes stale silently every time the code that produces it changes. `pcapng.txt` is exactly that: it still recorded `packet -> NIL` for the four Enhanced Packet Blocks of `dhcp.pcapng` long after #683 gave those blocks their captured octets back, and nothing had regenerated it. Rather than regenerate it once more and leave the same drift free to recur, the four files are removed from the index and folded into `examples/captures/`'s existing blanket `.gitignore` rule; `in.pcap` and `dhcp.pcapng`, the genuine inputs, stay tracked. `examples/legacy_smoke/Makefile` and its `README.rst` are reworded to describe regenerating these reports via `make fixtures` rather than implying they ship committed, and a new `tests/project/test_capture_tracking.py` (5 tests, 8 subtests) pins the invariant going forward; 2 of the 5 tests (5 of the 8 subtests) fail against the pre-change tree with the four reports restored to the index, the other 3 passing on both trees by construction (#685). -- **Fixed** -- two `#:` autodoc comments in `pcapkit/foundation/traceflow/traceflow.py` named a bare `Type`, which Sphinx's cross-reference resolver resolves against every class named `Type` in the project rather than against `typing.Type` -- there are five -- and silently linked to `pcapkit.const.l2tp.type.Type`, an L2TP field-type enum with nothing to do with dumpers. Line 406 (`#: Type[Dumper]: Dumper class.`) is the live case: once #684 rendered `TraceFlow._foutio`, the built docs pointed a reader at the wrong class with no warning that anything had gone sideways. Both sites now spell it `~typing.Type[Dumper]`, the same form eight other files in the tree already use for the identical ambiguity. Line 146 (the first line of `__output__`'s `#:` block, which continues through line 148) is fixed for the same reason but is currently inert: the actual `# type:` comment that drives that signature is sixteen lines below, at line 162, and it spells the identical bare `Type[Dumper]` -- so it is not correct either. The mechanism is likely `sphinx_autodoc_typehints` rather than `napoleon_attr_annotations`: Napoleon's own attribute-annotation lookup fires only for a Google/Numpy `Attributes:` section, and `__output__` is documented through autodoc's `#:` scraping instead. This half of the fix is insurance against the day something reads it cleanly. **This does not close #709.** Four more bare `Type` sites -- hand-written `:type:` fields at `docs/source/pcapkit/foundation/engines/engine.rst:40`, `.../reassembly/reassembly.rst:33` and `:43`, and `.../traceflow/traceflow.rst:40` -- carry the same ambiguity and are not touched here; #709 stays open for them, though PR #714 covers exactly those four sites (#709). +- **Fixed** -- `test_every_tracked_name_exists_and_matches_git` in `tests/test_tier_guard.py` checked neither the git index nor the filesystem despite its name: it asserted only that the tracked-capture list was non-empty and that no name contained a `/`, so a hardcoded list would pass it whether or not it matched reality. Sibling `test_capture_suggestions_are_captures` had the same gap. Found while cross-reviewing #703's first commit, itself prose-only and out of scope for this; the fix rides in as that PR's second and third commits, and #703's own description names both #700 and #708. `test_every_tracked_name_exists_and_matches_git` now re-derives the expected set independently -- shelling out to `git ls-files -z` under `examples/captures/` rather than calling back into `_tiers.py` -- and asserts equality against it rather than against itself, plus a per-name `Path.is_file()` check the old version never made. Its sibling `test_capture_suggestions_are_captures` gets a narrower fix: it asserts equality against `committed_capture_names()`'s own filter re-implemented inline over `_tiers.committed_captures()`, which catches a wrong filter but, unlike its sibling, still trusts `committed_captures()` for the tracked set itself rather than re-deriving it from git. A cross-review of the first version of this fix found it wrong twice over: a stale docstring in `_tiers.py` still claimed `test_tier_guard.py` "only stats `in.pcap`," falsified by the new per-name loop; and the new `assertEqual` on `test_capture_suggestions_are_captures` was vacuous on an empty tracked set (`() == ()` passes trivially), having dropped the non-emptiness guard the other test kept. Both are corrected in a follow-up commit. Even after the fix, a hardcoded literal that happens to match today's tracked names still passes -- the test detects a set that is wrong *at the moment it runs*, not hardcoding as a practice, which turns a permanent blind spot into a tripwire that fires on the next change to the tracked set (#708). +- **Changed** -- reconciled which private (`_xxx`) attributes and methods the Sphinx build documents, replacing an ad hoc mix with one stated rule: document the contract, hide the recipe. A member every subclass must implement, or one whose shape a caller genuinely depends on, stays documented even though its name starts with an underscore; a private helper that exists only to keep one method short does not. Seven module-level directives naming pure implementation detail were dropped -- `esp._resolve`, `esp._CRYPTO`, `ngap._convert`, `ngap._revert`, `ngap._PYCRATE`, `ngap._PDU_LOCK` and `pypcapfile._NamedStream` (whose nested `name`/`read` members go with it, being reachable only through a private class) -- while 39 `autoattribute` directives were added for class-private state that *is* contract: `Extractor._flag_f`, the `PyPCAP` and `PCAP_CT` engines' own `_backend` -- the only two of the six third-party engines that have one -- `TraceFlow`'s internal fields, and `FieldBase`/`Field` internals among them; the built-in `PCAP` and `PCAPNG` engines gained `_gbhdr`, `_vinfo` and `_nnsec` for the former and `_ctx`/`_ctx_list` for the latter, and no `_backend` at all. `_dlink` is not built-in-only: `PCAP` documents it alongside the three third-party engines that share it, `PyPCAP`, `PCAP_CT` and `PyPCAPFile`. The 128 runtime definitions of `_missing_` -- 121 under `pcapkit.const`, the other 7 inline in `pcapkit.protocols` -- gained one unified write-up in place of a directive per class: a new "Unrecognised Values" section in `docs/source/pcapkit/const/index.rst`, cross-referenced from `registry.rst`, since `conf.py` already excludes `_missing_` from every `autoclass` via `exclude-members` in `autodoc_default_options` -- there is no `automodule` directive anywhere under `docs/source/`. `CONTRIBUTING.md` gained the rule itself as a named section, so the next directive gets judged against a written test rather than against precedent. Verified with `sphinx-build -b html` under `PCAPKIT_SPHINX=1`: 55 warnings on `main` before this change, 56 after, the one addition being a pre-existing bare `Type` cross-reference ambiguity newly rendered by `TraceFlow._foutio`'s new directive rather than a defect this change introduced -- #709 tracks it. No line under `pcapkit/` changed (#684). +- **Fixed** -- `tests/dumpkit/test_nameless_enum_rendering_unit.py` (added by #670) carried two probes on the same wrong assumption that every flag-enum registry is 16 bits wide: `test_scalar_return_renders_a_nameless_member_as_its_value` checked one fixed tuple ending in `65536` against `tcp.flags.Flags` and its own `StdFlags` stand-in alone, and `test_no_flag_registry_renders_the_literal_none` swept every flag-enum registry instead, but only against `registry(0)`. Seven registries are not *all* 16 bits, at four distinct widths: `ftp.command.CommandType` (3 bits), `reg.apptype.TransportProtocol` (4, computed at runtime), `mh.binding_ack_flag.BindingACKFlag`, `mh.handover_ack_flag.HandoverACKFlag` and `mh.handover_initiate_flag.HandoverInitiateFlag` (8 each), and `mh.binding_update_flag.BindingUpdateFlag` with `tcp.flags.Flags` (16, the two the old probe actually fit). `tcp.flags.Flags(65536)` correctly raises -- its own `_missing_` bounds itself to `0 <= value <= 0xFFFF` -- so the sweep was failing on correct behaviour rather than reporting a defect. A new `_field_mask` derives each registry's own all-ones bound from its declared members, and `_nameless_values` returns the values that bound admits but no member names -- zero, each undeclared bit alone, and every undeclared bit combined -- in place of the one hard-coded tuple. A new `test_a_value_past_the_field_is_refused_rather_than_rendered` pins the boundary directly: the widest in-field value is accepted, one past it is refused. The file goes from 1 failed / 5 passed / 16 subtests to 6 passed / 64 subtests. A cross-review found that all seven guard `raise` lines this reaches -- `tcp/flags.py`'s own included -- were already covered by a passing test predating this fix, though not from one PR: #677 itself supplies three of the seven guards (`CommandType`, `TransportProtocol` and `Flags`), and the other four are the `mh.*` flag guards #632 added for #623; #677's `test_const_enum_builtin_parity.py` sweeps all 123 registries, which is what reaches all seven regardless of which PR guarded each. None was genuinely newly reached by this fix. No line under `pcapkit/` changed (#702). +- **Changed** -- `examples/captures/out.json`, `out.plist`, `out.txt` and `pcapng.txt` are no longer tracked in git; they are build output, not fixtures, and a tracked rendering with no reader goes stale silently every time the code that produces it changes. `pcapng.txt` is exactly that: it still recorded `packet -> NIL` for the four Enhanced Packet Blocks of `dhcp.pcapng` long after #683 gave those blocks their captured octets back, and nothing had regenerated it. Rather than regenerate it once more and leave the same drift free to recur, the four files are removed from the index and folded into `examples/captures/`'s existing blanket `.gitignore` rule; `in.pcap` and `dhcp.pcapng`, the genuine inputs, stay tracked. `examples/legacy_smoke/Makefile` and its `README.rst` are reworded to describe regenerating these reports via `make fixtures` rather than implying they ship committed, and a new `tests/project/test_capture_tracking.py` (5 tests, 8 subtests) pins the invariant going forward; 2 of the 5 tests (4 of the 12 subtests) fail against the pre-change tree with the four reports restored to the index, the other 3 passing on both trees by construction (#685). +- **Fixed** -- two `#:` autodoc comments in `pcapkit/foundation/traceflow/traceflow.py` named a bare `Type`, which Sphinx's cross-reference resolver resolves against every class named `Type` in the project rather than against `typing.Type` -- there are five -- and silently linked to `pcapkit.const.l2tp.type.Type`, an L2TP field-type enum with nothing to do with dumpers. Line 406 (`#: Type[Dumper]: Dumper class.`) is the live case: once #684 rendered `TraceFlow._foutio`, the built docs pointed a reader at the wrong class with no warning that anything had gone sideways. Both sites now spell it `~typing.Type[Dumper]`, the same form eight other files in the tree already use for the identical ambiguity. Line 146 (the first line of `__output__`'s `#:` block, which continues through line 149) is fixed for the same reason but is currently inert: `__output__` carries no runtime annotation (`TraceFlowBase.__dict__['__annotations__']` is empty), and the `# type:` comment sixteen lines below, at line 162, spells the identical bare `Type[Dumper]` -- so whatever eventually renders this attribute's type still has the same ambiguity to resolve. This half of the fix is insurance against the day something reads it cleanly. **This does not close #709.** Four more bare `Type` sites -- hand-written `:type:` fields at `docs/source/pcapkit/foundation/engines/engine.rst:40`, `.../reassembly/reassembly.rst:33` and `:43`, and `.../traceflow/traceflow.rst:40` -- carry the same ambiguity and are not touched here; #709 stays open for them, though PR #714 covers exactly those four sites (#709). - **Fixed** -- `register_protocol`'s overwrite warning could claim a protocol was replaced with itself. The guard that decides *whether* to warn was already correct -- `incumbent is not protocol`, an identity check from #681 -- but the message built both operands with a bare `repr()`, and for an ordinary class that is just ``. A factory that defines a same-named, closure-local class on every call (as `tests/protocols/test_construction_keyword_check_unit.py`'s `_protocol_class` does) produces two genuinely distinct classes sharing one `__module__` and `__qualname__`, so both reprs print identically and a real, correct overwrite reads as "overwriting X with X." The message now compares the two reprs first, and only when they coincide appends each object's `id()` to tell them apart; the common case, where the two classes are named differently, is untouched. `__module__`/`__qualname__` was considered as the disambiguator instead of `id()` and rejected: for the reported shape those are exactly what the coinciding repr already renders, so they discriminate nothing an `id()` does not already have to. A new `test_register_protocol_disambiguates_classes_sharing_a_repr` fails against the unfixed message and passes against the fix; the targeted suite goes 49 to 50 passed, and the file's coverage holds at 97% (#710). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). From 91026756fe51b2604958a55dff0531785c5ce41a Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 17:44:41 -0400 Subject: [PATCH 43/46] docs(changelog): delete two more wrong claims instead of re-fixing them A third cross-review found the same failure mode again: the last pass replaced vague prose with false prose. This round deletes rather than corrects. - #702 entry: dropped the attribution of the seven guard raise lines to specific PRs. "the four mh.* guards came from #632" was false and self-contradicts the file's own #623 entry three sections earlier ("The range guard above it is untouched" -- a05f46149's own commit message says the same). The entry does not need to say which PR added which guard; it only needs to say the lines were already covered. - #709 entry: dropped the `TraceFlowBase.__dict__['__annotations__']` citation. Measured: that lookup raises KeyError, not "is empty" -- `TraceFlowBase.__annotations__` (the resolved, non-dict-shortcut form) is `{'__cached__': ...}`, non-empty. The conclusion (`__output__` is unannotated) holds, but the cited evidence does not, so it's dropped rather than replaced with a corrected measurement. - #709 entry: rewritten to the present tense of a merged world. #709 closed once #712 landed (the two traceflow.py sites this entry describes); the four docs/*.rst sites are fixed separately by #714, also merged. No more "does not close" / "stays open" hedging. Refs #657 --- docs/source/changelog/1.5.0.rst | 30 ++++++++++++------------------ 1 file changed, 12 insertions(+), 18 deletions(-) diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 8d56c8a03..cb1f79cfa 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2349,13 +2349,9 @@ pull requests between #326 and #509. is refused. The file goes from 1 failed / 5 passed / 16 subtests to 6 passed / 64 subtests. A cross-review found that all seven guard ``raise`` lines this reaches -- ``tcp/flags.py``'s own included -- were - already covered by a passing test predating this fix, though not from one - PR: #677 itself supplies three of the seven guards (``CommandType``, - ``TransportProtocol`` and ``Flags``), and the other four are the ``mh.*`` - flag guards #632 added for #623; #677's ``test_const_enum_builtin_parity.py`` - sweeps all 123 registries, which is what reaches all seven regardless of - which PR guarded each. None was genuinely newly reached by this fix. No - line under ``pcapkit/`` changed (#702). + already covered by a passing test predating this fix; none was + genuinely newly reached by it. No line under ``pcapkit/`` changed + (#702). * **Changed** -- ``examples/captures/out.json``, ``out.plist``, ``out.txt`` and ``pcapng.txt`` are no longer tracked in git; they are build output, not fixtures, and a tracked rendering with no reader goes stale silently every @@ -2385,19 +2381,17 @@ pull requests between #326 and #509. ``~typing.Type[Dumper]``, the same form eight other files in the tree already use for the identical ambiguity. Line 146 (the first line of ``__output__``'s ``#:`` block, which continues through line 149) is fixed - for the same reason but is currently inert: ``__output__`` carries no - runtime annotation (``TraceFlowBase.__dict__['__annotations__']`` is - empty), and the ``# type:`` comment sixteen lines below, at line 162, - spells the identical bare ``Type[Dumper]`` -- so whatever eventually - renders this attribute's type still has the same ambiguity to resolve. - This half of the fix is insurance against the day something reads it - cleanly. **This does not close #709.** Four more - bare ``Type`` sites -- hand-written ``:type:`` fields at + for the same reason but is currently inert: the ``# type:`` comment + sixteen lines below, at line 162, spells the identical bare + ``Type[Dumper]`` -- so whatever eventually renders this attribute's type + still has the same ambiguity to resolve. This half of the fix is + insurance against the day something reads it cleanly. This pair of sites + is what closes #709. Four more bare ``Type`` sites -- hand-written + ``:type:`` fields at ``docs/source/pcapkit/foundation/engines/engine.rst:40``, ``.../reassembly/reassembly.rst:33`` and ``:43``, and - ``.../traceflow/traceflow.rst:40`` -- carry the same ambiguity and are not - touched here; #709 stays open for them, though PR #714 covers exactly - those four sites (#709). + ``.../traceflow/traceflow.rst:40`` -- carried the same ambiguity and are + fixed separately, by #714 (#709). * **Fixed** -- ``register_protocol``'s overwrite warning could claim a protocol was replaced with itself. The guard that decides *whether* to warn was already correct -- ``incumbent is not protocol``, an identity From 93a4126365e804d26d1434ed8bde351f0b63a688 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 17:44:46 -0400 Subject: [PATCH 44/46] chore(changelog): regenerate CHANGELOG.md for the two deletions python util/changelog_md.py, in step with 1.5.0.rst at 91026756f. Refs #657 --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5db8bd381..5ff799ea8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -111,9 +111,9 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- three test comments stated an exact count of committed captures under `examples/captures/`, a number that drifts every time that directory's tracked set changes and in one case was already wrong when written. `tests/_tiers.py` cited "six," correct at the time; `tests/test_tier_guard.py` said "the moment a seventh capture is committed" -- the same count, as an ordinal rather than the word "six"; `tests/integration/_helpers.py` said "four," which undercounted the tracked set the day it was written. All three now describe the invariant instead of a number that has to be kept in sync with it by hand: `_tiers.py` reads "the moment somebody commits another capture... or stops committing one," `_helpers.py` reads "...are fixtures -- some of them committed --...," and `test_tier_guard.py` drops its "seventh capture" phrasing the same way. Prose-only, verified with `coverage.parser.PythonParser` that no executable statement moved (#700). - **Fixed** -- `test_every_tracked_name_exists_and_matches_git` in `tests/test_tier_guard.py` checked neither the git index nor the filesystem despite its name: it asserted only that the tracked-capture list was non-empty and that no name contained a `/`, so a hardcoded list would pass it whether or not it matched reality. Sibling `test_capture_suggestions_are_captures` had the same gap. Found while cross-reviewing #703's first commit, itself prose-only and out of scope for this; the fix rides in as that PR's second and third commits, and #703's own description names both #700 and #708. `test_every_tracked_name_exists_and_matches_git` now re-derives the expected set independently -- shelling out to `git ls-files -z` under `examples/captures/` rather than calling back into `_tiers.py` -- and asserts equality against it rather than against itself, plus a per-name `Path.is_file()` check the old version never made. Its sibling `test_capture_suggestions_are_captures` gets a narrower fix: it asserts equality against `committed_capture_names()`'s own filter re-implemented inline over `_tiers.committed_captures()`, which catches a wrong filter but, unlike its sibling, still trusts `committed_captures()` for the tracked set itself rather than re-deriving it from git. A cross-review of the first version of this fix found it wrong twice over: a stale docstring in `_tiers.py` still claimed `test_tier_guard.py` "only stats `in.pcap`," falsified by the new per-name loop; and the new `assertEqual` on `test_capture_suggestions_are_captures` was vacuous on an empty tracked set (`() == ()` passes trivially), having dropped the non-emptiness guard the other test kept. Both are corrected in a follow-up commit. Even after the fix, a hardcoded literal that happens to match today's tracked names still passes -- the test detects a set that is wrong *at the moment it runs*, not hardcoding as a practice, which turns a permanent blind spot into a tripwire that fires on the next change to the tracked set (#708). - **Changed** -- reconciled which private (`_xxx`) attributes and methods the Sphinx build documents, replacing an ad hoc mix with one stated rule: document the contract, hide the recipe. A member every subclass must implement, or one whose shape a caller genuinely depends on, stays documented even though its name starts with an underscore; a private helper that exists only to keep one method short does not. Seven module-level directives naming pure implementation detail were dropped -- `esp._resolve`, `esp._CRYPTO`, `ngap._convert`, `ngap._revert`, `ngap._PYCRATE`, `ngap._PDU_LOCK` and `pypcapfile._NamedStream` (whose nested `name`/`read` members go with it, being reachable only through a private class) -- while 39 `autoattribute` directives were added for class-private state that *is* contract: `Extractor._flag_f`, the `PyPCAP` and `PCAP_CT` engines' own `_backend` -- the only two of the six third-party engines that have one -- `TraceFlow`'s internal fields, and `FieldBase`/`Field` internals among them; the built-in `PCAP` and `PCAPNG` engines gained `_gbhdr`, `_vinfo` and `_nnsec` for the former and `_ctx`/`_ctx_list` for the latter, and no `_backend` at all. `_dlink` is not built-in-only: `PCAP` documents it alongside the three third-party engines that share it, `PyPCAP`, `PCAP_CT` and `PyPCAPFile`. The 128 runtime definitions of `_missing_` -- 121 under `pcapkit.const`, the other 7 inline in `pcapkit.protocols` -- gained one unified write-up in place of a directive per class: a new "Unrecognised Values" section in `docs/source/pcapkit/const/index.rst`, cross-referenced from `registry.rst`, since `conf.py` already excludes `_missing_` from every `autoclass` via `exclude-members` in `autodoc_default_options` -- there is no `automodule` directive anywhere under `docs/source/`. `CONTRIBUTING.md` gained the rule itself as a named section, so the next directive gets judged against a written test rather than against precedent. Verified with `sphinx-build -b html` under `PCAPKIT_SPHINX=1`: 55 warnings on `main` before this change, 56 after, the one addition being a pre-existing bare `Type` cross-reference ambiguity newly rendered by `TraceFlow._foutio`'s new directive rather than a defect this change introduced -- #709 tracks it. No line under `pcapkit/` changed (#684). -- **Fixed** -- `tests/dumpkit/test_nameless_enum_rendering_unit.py` (added by #670) carried two probes on the same wrong assumption that every flag-enum registry is 16 bits wide: `test_scalar_return_renders_a_nameless_member_as_its_value` checked one fixed tuple ending in `65536` against `tcp.flags.Flags` and its own `StdFlags` stand-in alone, and `test_no_flag_registry_renders_the_literal_none` swept every flag-enum registry instead, but only against `registry(0)`. Seven registries are not *all* 16 bits, at four distinct widths: `ftp.command.CommandType` (3 bits), `reg.apptype.TransportProtocol` (4, computed at runtime), `mh.binding_ack_flag.BindingACKFlag`, `mh.handover_ack_flag.HandoverACKFlag` and `mh.handover_initiate_flag.HandoverInitiateFlag` (8 each), and `mh.binding_update_flag.BindingUpdateFlag` with `tcp.flags.Flags` (16, the two the old probe actually fit). `tcp.flags.Flags(65536)` correctly raises -- its own `_missing_` bounds itself to `0 <= value <= 0xFFFF` -- so the sweep was failing on correct behaviour rather than reporting a defect. A new `_field_mask` derives each registry's own all-ones bound from its declared members, and `_nameless_values` returns the values that bound admits but no member names -- zero, each undeclared bit alone, and every undeclared bit combined -- in place of the one hard-coded tuple. A new `test_a_value_past_the_field_is_refused_rather_than_rendered` pins the boundary directly: the widest in-field value is accepted, one past it is refused. The file goes from 1 failed / 5 passed / 16 subtests to 6 passed / 64 subtests. A cross-review found that all seven guard `raise` lines this reaches -- `tcp/flags.py`'s own included -- were already covered by a passing test predating this fix, though not from one PR: #677 itself supplies three of the seven guards (`CommandType`, `TransportProtocol` and `Flags`), and the other four are the `mh.*` flag guards #632 added for #623; #677's `test_const_enum_builtin_parity.py` sweeps all 123 registries, which is what reaches all seven regardless of which PR guarded each. None was genuinely newly reached by this fix. No line under `pcapkit/` changed (#702). +- **Fixed** -- `tests/dumpkit/test_nameless_enum_rendering_unit.py` (added by #670) carried two probes on the same wrong assumption that every flag-enum registry is 16 bits wide: `test_scalar_return_renders_a_nameless_member_as_its_value` checked one fixed tuple ending in `65536` against `tcp.flags.Flags` and its own `StdFlags` stand-in alone, and `test_no_flag_registry_renders_the_literal_none` swept every flag-enum registry instead, but only against `registry(0)`. Seven registries are not *all* 16 bits, at four distinct widths: `ftp.command.CommandType` (3 bits), `reg.apptype.TransportProtocol` (4, computed at runtime), `mh.binding_ack_flag.BindingACKFlag`, `mh.handover_ack_flag.HandoverACKFlag` and `mh.handover_initiate_flag.HandoverInitiateFlag` (8 each), and `mh.binding_update_flag.BindingUpdateFlag` with `tcp.flags.Flags` (16, the two the old probe actually fit). `tcp.flags.Flags(65536)` correctly raises -- its own `_missing_` bounds itself to `0 <= value <= 0xFFFF` -- so the sweep was failing on correct behaviour rather than reporting a defect. A new `_field_mask` derives each registry's own all-ones bound from its declared members, and `_nameless_values` returns the values that bound admits but no member names -- zero, each undeclared bit alone, and every undeclared bit combined -- in place of the one hard-coded tuple. A new `test_a_value_past_the_field_is_refused_rather_than_rendered` pins the boundary directly: the widest in-field value is accepted, one past it is refused. The file goes from 1 failed / 5 passed / 16 subtests to 6 passed / 64 subtests. A cross-review found that all seven guard `raise` lines this reaches -- `tcp/flags.py`'s own included -- were already covered by a passing test predating this fix; none was genuinely newly reached by it. No line under `pcapkit/` changed (#702). - **Changed** -- `examples/captures/out.json`, `out.plist`, `out.txt` and `pcapng.txt` are no longer tracked in git; they are build output, not fixtures, and a tracked rendering with no reader goes stale silently every time the code that produces it changes. `pcapng.txt` is exactly that: it still recorded `packet -> NIL` for the four Enhanced Packet Blocks of `dhcp.pcapng` long after #683 gave those blocks their captured octets back, and nothing had regenerated it. Rather than regenerate it once more and leave the same drift free to recur, the four files are removed from the index and folded into `examples/captures/`'s existing blanket `.gitignore` rule; `in.pcap` and `dhcp.pcapng`, the genuine inputs, stay tracked. `examples/legacy_smoke/Makefile` and its `README.rst` are reworded to describe regenerating these reports via `make fixtures` rather than implying they ship committed, and a new `tests/project/test_capture_tracking.py` (5 tests, 8 subtests) pins the invariant going forward; 2 of the 5 tests (4 of the 12 subtests) fail against the pre-change tree with the four reports restored to the index, the other 3 passing on both trees by construction (#685). -- **Fixed** -- two `#:` autodoc comments in `pcapkit/foundation/traceflow/traceflow.py` named a bare `Type`, which Sphinx's cross-reference resolver resolves against every class named `Type` in the project rather than against `typing.Type` -- there are five -- and silently linked to `pcapkit.const.l2tp.type.Type`, an L2TP field-type enum with nothing to do with dumpers. Line 406 (`#: Type[Dumper]: Dumper class.`) is the live case: once #684 rendered `TraceFlow._foutio`, the built docs pointed a reader at the wrong class with no warning that anything had gone sideways. Both sites now spell it `~typing.Type[Dumper]`, the same form eight other files in the tree already use for the identical ambiguity. Line 146 (the first line of `__output__`'s `#:` block, which continues through line 149) is fixed for the same reason but is currently inert: `__output__` carries no runtime annotation (`TraceFlowBase.__dict__['__annotations__']` is empty), and the `# type:` comment sixteen lines below, at line 162, spells the identical bare `Type[Dumper]` -- so whatever eventually renders this attribute's type still has the same ambiguity to resolve. This half of the fix is insurance against the day something reads it cleanly. **This does not close #709.** Four more bare `Type` sites -- hand-written `:type:` fields at `docs/source/pcapkit/foundation/engines/engine.rst:40`, `.../reassembly/reassembly.rst:33` and `:43`, and `.../traceflow/traceflow.rst:40` -- carry the same ambiguity and are not touched here; #709 stays open for them, though PR #714 covers exactly those four sites (#709). +- **Fixed** -- two `#:` autodoc comments in `pcapkit/foundation/traceflow/traceflow.py` named a bare `Type`, which Sphinx's cross-reference resolver resolves against every class named `Type` in the project rather than against `typing.Type` -- there are five -- and silently linked to `pcapkit.const.l2tp.type.Type`, an L2TP field-type enum with nothing to do with dumpers. Line 406 (`#: Type[Dumper]: Dumper class.`) is the live case: once #684 rendered `TraceFlow._foutio`, the built docs pointed a reader at the wrong class with no warning that anything had gone sideways. Both sites now spell it `~typing.Type[Dumper]`, the same form eight other files in the tree already use for the identical ambiguity. Line 146 (the first line of `__output__`'s `#:` block, which continues through line 149) is fixed for the same reason but is currently inert: the `# type:` comment sixteen lines below, at line 162, spells the identical bare `Type[Dumper]` -- so whatever eventually renders this attribute's type still has the same ambiguity to resolve. This half of the fix is insurance against the day something reads it cleanly. This pair of sites is what closes #709. Four more bare `Type` sites -- hand-written `:type:` fields at `docs/source/pcapkit/foundation/engines/engine.rst:40`, `.../reassembly/reassembly.rst:33` and `:43`, and `.../traceflow/traceflow.rst:40` -- carried the same ambiguity and are fixed separately, by #714 (#709). - **Fixed** -- `register_protocol`'s overwrite warning could claim a protocol was replaced with itself. The guard that decides *whether* to warn was already correct -- `incumbent is not protocol`, an identity check from #681 -- but the message built both operands with a bare `repr()`, and for an ordinary class that is just ``. A factory that defines a same-named, closure-local class on every call (as `tests/protocols/test_construction_keyword_check_unit.py`'s `_protocol_class` does) produces two genuinely distinct classes sharing one `__module__` and `__qualname__`, so both reprs print identically and a real, correct overwrite reads as "overwriting X with X." The message now compares the two reprs first, and only when they coincide appends each object's `id()` to tell them apart; the common case, where the two classes are named differently, is untouched. `__module__`/`__qualname__` was considered as the disambiguator instead of `id()` and rejected: for the reported shape those are exactly what the coinciding repr already renders, so they discriminate nothing an `id()` does not already have to. A new `test_register_protocol_disambiguates_classes_sharing_a_repr` fails against the unfixed message and passes against the fix; the targeted suite goes 49 to 50 passed, and the file's coverage holds at 97% (#710). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). From 42335554e8c04098ef84df4a5a7668316b10896c Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 21:20:55 -0400 Subject: [PATCH 45/46] docs(changelog): fix the #709 entry's false causal claim and #684's stale tense Round 5: fixed the two defects the fourth cross-review found at head 93a412636 (NEEDS CHANGES). - `:2389` -- "This pair of sites is what closes #709" was false: #712 and #714 both state in their own bodies that they are `Part of #709`, not a close, and #709's closed timeline event carries `commit_id: null` (a manual close). Made non-causal: "This pair of sites is part of #709." - `:2325` -- "#709 tracks it" was present tense describing an issue that is now closed, contradicting the #709 entry itself. Changed to past tense: "#709 tracked it." Regenerated CHANGELOG.md from the edited entry. | check | result | |---|---| | `changelog_md.py --check` | exit 0 | | `pytest tests/project/test_changelog_md.py -q` | 47 passed, 37 subtests | --- CHANGELOG.md | 4 ++-- docs/source/changelog/1.5.0.rst | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ff799ea8..89c2c7667 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -110,10 +110,10 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- HIP's `R1_COUNTER`/`R1_Counter` parameter packed 12 octets where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires 16, and `LOCATOR_SET` declared its `Length` in 4-octet units where [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1)'s `Length` is a byte count -- two independent defects, fixed together because the evidence for either needed the other out of the way first. `R1CounterParameter`'s `counter` was a `UInt32Field` where the RFC states the R1 generation counter's width twice, as "8 bytes" in the diagram and as "a 64-bit unsigned integer" in prose; both HIP versions whose *packing* reaches this class -- `R1_Counter` (128) and `R1_COUNTER` (129) -- packed a 12-octet record, landing at `4 (mod 8)` instead of aligned. It is a `UInt64Field` now, and the record is 16 octets (#672). `LocatorSetParameter`'s own `Length` was written as `sum(Locator.len)`, in the 4-octet units [RFC 8046 Section 4](https://datatracker.ietf.org/doc/html/rfc8046#section-4) gives `Locator Length`, where this parameter's own `Length` is a byte count; a set of *n* plain IPv6 locators therefore declared `4n` octets against `24n` actually present, and `Schema.unpack` handed the nested `ListField` only the declared octets, so `n = 2` and `n = 5` both parsed one truncated locator and left the rest of the record unconsumed. `HIP._make_param_locator_set` now sums `8 + Locator.len * 4` per locator -- the fixed header plus the RFC 8046 contents each one actually carries (#679). A second, independent defect in the same parameter's padding is fixed alongside it, one #651 deliberately left alone because it happened to cancel this one exactly: the nested `Locator` schemas share the parameter's packet context and pack their own `len` over it, so `padding` -- evaluated after the list -- always saw 4 rather than the parameter's real length. A new `locator_set_len_callback` snapshots `Length` under a private key before any locator packs into the context, and `locator_set_padding_len` reads that snapshot rather than the shadowed one. An RFC-only byte-stride walk over `examples/captures/options-internet.pcap`, independent of pcapkit's own parser, went from 1 violation -- `LOCATOR_SET`'s empty record, concealed until now because `R1_COUNTER`'s `counter` field defaulted to `0` with no override anywhere in `examples/generators/options.py`, a value the width defect could not be told apart from -- to 2 once the counter was patched non-zero, to 0 once both parameters were fixed; the fixture now overrides both `R1_COUNTER` codes with `counter=0xaabbccdd` so a future regression cannot hide behind a zero again. 10 new test methods across two new files -- 4 in `test_hip_r1_counter_width_unit.py`, 6 in `test_hip_locator_set_length_unit.py` -- fail against the unfixed code and pass against the fix; the three HIP modules stay at 100% coverage, and `tests/` subtests move 586 to 623. `EXPECTED_FAILURES` is unmoved at 44 entries: `hip-parameter/R1_Counter` stays, because code 128 registers no schema of its own -- `R1CounterParameter` declares only `code=129` -- which is a separate registry defect this does not touch, filed as #690. `HIP_COPIES` also stays at two, now for no defect at all rather than for the ones #651 and this fix removed; dropping it is #689, deferred because it would halve every `hip-parameter` frame the fixture above reads (#672, #679). - **Fixed** -- three test comments stated an exact count of committed captures under `examples/captures/`, a number that drifts every time that directory's tracked set changes and in one case was already wrong when written. `tests/_tiers.py` cited "six," correct at the time; `tests/test_tier_guard.py` said "the moment a seventh capture is committed" -- the same count, as an ordinal rather than the word "six"; `tests/integration/_helpers.py` said "four," which undercounted the tracked set the day it was written. All three now describe the invariant instead of a number that has to be kept in sync with it by hand: `_tiers.py` reads "the moment somebody commits another capture... or stops committing one," `_helpers.py` reads "...are fixtures -- some of them committed --...," and `test_tier_guard.py` drops its "seventh capture" phrasing the same way. Prose-only, verified with `coverage.parser.PythonParser` that no executable statement moved (#700). - **Fixed** -- `test_every_tracked_name_exists_and_matches_git` in `tests/test_tier_guard.py` checked neither the git index nor the filesystem despite its name: it asserted only that the tracked-capture list was non-empty and that no name contained a `/`, so a hardcoded list would pass it whether or not it matched reality. Sibling `test_capture_suggestions_are_captures` had the same gap. Found while cross-reviewing #703's first commit, itself prose-only and out of scope for this; the fix rides in as that PR's second and third commits, and #703's own description names both #700 and #708. `test_every_tracked_name_exists_and_matches_git` now re-derives the expected set independently -- shelling out to `git ls-files -z` under `examples/captures/` rather than calling back into `_tiers.py` -- and asserts equality against it rather than against itself, plus a per-name `Path.is_file()` check the old version never made. Its sibling `test_capture_suggestions_are_captures` gets a narrower fix: it asserts equality against `committed_capture_names()`'s own filter re-implemented inline over `_tiers.committed_captures()`, which catches a wrong filter but, unlike its sibling, still trusts `committed_captures()` for the tracked set itself rather than re-deriving it from git. A cross-review of the first version of this fix found it wrong twice over: a stale docstring in `_tiers.py` still claimed `test_tier_guard.py` "only stats `in.pcap`," falsified by the new per-name loop; and the new `assertEqual` on `test_capture_suggestions_are_captures` was vacuous on an empty tracked set (`() == ()` passes trivially), having dropped the non-emptiness guard the other test kept. Both are corrected in a follow-up commit. Even after the fix, a hardcoded literal that happens to match today's tracked names still passes -- the test detects a set that is wrong *at the moment it runs*, not hardcoding as a practice, which turns a permanent blind spot into a tripwire that fires on the next change to the tracked set (#708). -- **Changed** -- reconciled which private (`_xxx`) attributes and methods the Sphinx build documents, replacing an ad hoc mix with one stated rule: document the contract, hide the recipe. A member every subclass must implement, or one whose shape a caller genuinely depends on, stays documented even though its name starts with an underscore; a private helper that exists only to keep one method short does not. Seven module-level directives naming pure implementation detail were dropped -- `esp._resolve`, `esp._CRYPTO`, `ngap._convert`, `ngap._revert`, `ngap._PYCRATE`, `ngap._PDU_LOCK` and `pypcapfile._NamedStream` (whose nested `name`/`read` members go with it, being reachable only through a private class) -- while 39 `autoattribute` directives were added for class-private state that *is* contract: `Extractor._flag_f`, the `PyPCAP` and `PCAP_CT` engines' own `_backend` -- the only two of the six third-party engines that have one -- `TraceFlow`'s internal fields, and `FieldBase`/`Field` internals among them; the built-in `PCAP` and `PCAPNG` engines gained `_gbhdr`, `_vinfo` and `_nnsec` for the former and `_ctx`/`_ctx_list` for the latter, and no `_backend` at all. `_dlink` is not built-in-only: `PCAP` documents it alongside the three third-party engines that share it, `PyPCAP`, `PCAP_CT` and `PyPCAPFile`. The 128 runtime definitions of `_missing_` -- 121 under `pcapkit.const`, the other 7 inline in `pcapkit.protocols` -- gained one unified write-up in place of a directive per class: a new "Unrecognised Values" section in `docs/source/pcapkit/const/index.rst`, cross-referenced from `registry.rst`, since `conf.py` already excludes `_missing_` from every `autoclass` via `exclude-members` in `autodoc_default_options` -- there is no `automodule` directive anywhere under `docs/source/`. `CONTRIBUTING.md` gained the rule itself as a named section, so the next directive gets judged against a written test rather than against precedent. Verified with `sphinx-build -b html` under `PCAPKIT_SPHINX=1`: 55 warnings on `main` before this change, 56 after, the one addition being a pre-existing bare `Type` cross-reference ambiguity newly rendered by `TraceFlow._foutio`'s new directive rather than a defect this change introduced -- #709 tracks it. No line under `pcapkit/` changed (#684). +- **Changed** -- reconciled which private (`_xxx`) attributes and methods the Sphinx build documents, replacing an ad hoc mix with one stated rule: document the contract, hide the recipe. A member every subclass must implement, or one whose shape a caller genuinely depends on, stays documented even though its name starts with an underscore; a private helper that exists only to keep one method short does not. Seven module-level directives naming pure implementation detail were dropped -- `esp._resolve`, `esp._CRYPTO`, `ngap._convert`, `ngap._revert`, `ngap._PYCRATE`, `ngap._PDU_LOCK` and `pypcapfile._NamedStream` (whose nested `name`/`read` members go with it, being reachable only through a private class) -- while 39 `autoattribute` directives were added for class-private state that *is* contract: `Extractor._flag_f`, the `PyPCAP` and `PCAP_CT` engines' own `_backend` -- the only two of the six third-party engines that have one -- `TraceFlow`'s internal fields, and `FieldBase`/`Field` internals among them; the built-in `PCAP` and `PCAPNG` engines gained `_gbhdr`, `_vinfo` and `_nnsec` for the former and `_ctx`/`_ctx_list` for the latter, and no `_backend` at all. `_dlink` is not built-in-only: `PCAP` documents it alongside the three third-party engines that share it, `PyPCAP`, `PCAP_CT` and `PyPCAPFile`. The 128 runtime definitions of `_missing_` -- 121 under `pcapkit.const`, the other 7 inline in `pcapkit.protocols` -- gained one unified write-up in place of a directive per class: a new "Unrecognised Values" section in `docs/source/pcapkit/const/index.rst`, cross-referenced from `registry.rst`, since `conf.py` already excludes `_missing_` from every `autoclass` via `exclude-members` in `autodoc_default_options` -- there is no `automodule` directive anywhere under `docs/source/`. `CONTRIBUTING.md` gained the rule itself as a named section, so the next directive gets judged against a written test rather than against precedent. Verified with `sphinx-build -b html` under `PCAPKIT_SPHINX=1`: 55 warnings on `main` before this change, 56 after, the one addition being a pre-existing bare `Type` cross-reference ambiguity newly rendered by `TraceFlow._foutio`'s new directive rather than a defect this change introduced -- #709 tracked it. No line under `pcapkit/` changed (#684). - **Fixed** -- `tests/dumpkit/test_nameless_enum_rendering_unit.py` (added by #670) carried two probes on the same wrong assumption that every flag-enum registry is 16 bits wide: `test_scalar_return_renders_a_nameless_member_as_its_value` checked one fixed tuple ending in `65536` against `tcp.flags.Flags` and its own `StdFlags` stand-in alone, and `test_no_flag_registry_renders_the_literal_none` swept every flag-enum registry instead, but only against `registry(0)`. Seven registries are not *all* 16 bits, at four distinct widths: `ftp.command.CommandType` (3 bits), `reg.apptype.TransportProtocol` (4, computed at runtime), `mh.binding_ack_flag.BindingACKFlag`, `mh.handover_ack_flag.HandoverACKFlag` and `mh.handover_initiate_flag.HandoverInitiateFlag` (8 each), and `mh.binding_update_flag.BindingUpdateFlag` with `tcp.flags.Flags` (16, the two the old probe actually fit). `tcp.flags.Flags(65536)` correctly raises -- its own `_missing_` bounds itself to `0 <= value <= 0xFFFF` -- so the sweep was failing on correct behaviour rather than reporting a defect. A new `_field_mask` derives each registry's own all-ones bound from its declared members, and `_nameless_values` returns the values that bound admits but no member names -- zero, each undeclared bit alone, and every undeclared bit combined -- in place of the one hard-coded tuple. A new `test_a_value_past_the_field_is_refused_rather_than_rendered` pins the boundary directly: the widest in-field value is accepted, one past it is refused. The file goes from 1 failed / 5 passed / 16 subtests to 6 passed / 64 subtests. A cross-review found that all seven guard `raise` lines this reaches -- `tcp/flags.py`'s own included -- were already covered by a passing test predating this fix; none was genuinely newly reached by it. No line under `pcapkit/` changed (#702). - **Changed** -- `examples/captures/out.json`, `out.plist`, `out.txt` and `pcapng.txt` are no longer tracked in git; they are build output, not fixtures, and a tracked rendering with no reader goes stale silently every time the code that produces it changes. `pcapng.txt` is exactly that: it still recorded `packet -> NIL` for the four Enhanced Packet Blocks of `dhcp.pcapng` long after #683 gave those blocks their captured octets back, and nothing had regenerated it. Rather than regenerate it once more and leave the same drift free to recur, the four files are removed from the index and folded into `examples/captures/`'s existing blanket `.gitignore` rule; `in.pcap` and `dhcp.pcapng`, the genuine inputs, stay tracked. `examples/legacy_smoke/Makefile` and its `README.rst` are reworded to describe regenerating these reports via `make fixtures` rather than implying they ship committed, and a new `tests/project/test_capture_tracking.py` (5 tests, 8 subtests) pins the invariant going forward; 2 of the 5 tests (4 of the 12 subtests) fail against the pre-change tree with the four reports restored to the index, the other 3 passing on both trees by construction (#685). -- **Fixed** -- two `#:` autodoc comments in `pcapkit/foundation/traceflow/traceflow.py` named a bare `Type`, which Sphinx's cross-reference resolver resolves against every class named `Type` in the project rather than against `typing.Type` -- there are five -- and silently linked to `pcapkit.const.l2tp.type.Type`, an L2TP field-type enum with nothing to do with dumpers. Line 406 (`#: Type[Dumper]: Dumper class.`) is the live case: once #684 rendered `TraceFlow._foutio`, the built docs pointed a reader at the wrong class with no warning that anything had gone sideways. Both sites now spell it `~typing.Type[Dumper]`, the same form eight other files in the tree already use for the identical ambiguity. Line 146 (the first line of `__output__`'s `#:` block, which continues through line 149) is fixed for the same reason but is currently inert: the `# type:` comment sixteen lines below, at line 162, spells the identical bare `Type[Dumper]` -- so whatever eventually renders this attribute's type still has the same ambiguity to resolve. This half of the fix is insurance against the day something reads it cleanly. This pair of sites is what closes #709. Four more bare `Type` sites -- hand-written `:type:` fields at `docs/source/pcapkit/foundation/engines/engine.rst:40`, `.../reassembly/reassembly.rst:33` and `:43`, and `.../traceflow/traceflow.rst:40` -- carried the same ambiguity and are fixed separately, by #714 (#709). +- **Fixed** -- two `#:` autodoc comments in `pcapkit/foundation/traceflow/traceflow.py` named a bare `Type`, which Sphinx's cross-reference resolver resolves against every class named `Type` in the project rather than against `typing.Type` -- there are five -- and silently linked to `pcapkit.const.l2tp.type.Type`, an L2TP field-type enum with nothing to do with dumpers. Line 406 (`#: Type[Dumper]: Dumper class.`) is the live case: once #684 rendered `TraceFlow._foutio`, the built docs pointed a reader at the wrong class with no warning that anything had gone sideways. Both sites now spell it `~typing.Type[Dumper]`, the same form eight other files in the tree already use for the identical ambiguity. Line 146 (the first line of `__output__`'s `#:` block, which continues through line 149) is fixed for the same reason but is currently inert: the `# type:` comment sixteen lines below, at line 162, spells the identical bare `Type[Dumper]` -- so whatever eventually renders this attribute's type still has the same ambiguity to resolve. This half of the fix is insurance against the day something reads it cleanly. This pair of sites is part of #709. Four more bare `Type` sites -- hand-written `:type:` fields at `docs/source/pcapkit/foundation/engines/engine.rst:40`, `.../reassembly/reassembly.rst:33` and `:43`, and `.../traceflow/traceflow.rst:40` -- carried the same ambiguity and are fixed separately, by #714 (#709). - **Fixed** -- `register_protocol`'s overwrite warning could claim a protocol was replaced with itself. The guard that decides *whether* to warn was already correct -- `incumbent is not protocol`, an identity check from #681 -- but the message built both operands with a bare `repr()`, and for an ordinary class that is just ``. A factory that defines a same-named, closure-local class on every call (as `tests/protocols/test_construction_keyword_check_unit.py`'s `_protocol_class` does) produces two genuinely distinct classes sharing one `__module__` and `__qualname__`, so both reprs print identically and a real, correct overwrite reads as "overwriting X with X." The message now compares the two reprs first, and only when they coincide appends each object's `id()` to tell them apart; the common case, where the two classes are named differently, is untouched. `__module__`/`__qualname__` was considered as the disambiguator instead of `id()` and rejected: for the reported shape those are exactly what the coinciding repr already renders, so they discriminate nothing an `id()` does not already have to. A new `test_register_protocol_disambiguates_classes_sharing_a_repr` fails against the unfixed message and passes against the fix; the targeted suite goes 49 to 50 passed, and the file's coverage holds at 97% (#710). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index cb1f79cfa..e8a612f8d 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -2322,7 +2322,7 @@ pull requests between #326 and #509. ``sphinx-build -b html`` under ``PCAPKIT_SPHINX=1``: 55 warnings on ``main`` before this change, 56 after, the one addition being a pre-existing bare ``Type`` cross-reference ambiguity newly rendered by ``TraceFlow._foutio``'s - new directive rather than a defect this change introduced -- #709 tracks + new directive rather than a defect this change introduced -- #709 tracked it. No line under ``pcapkit/`` changed (#684). * **Fixed** -- ``tests/dumpkit/test_nameless_enum_rendering_unit.py`` (added by #670) carried two probes on the same wrong assumption that every @@ -2386,7 +2386,7 @@ pull requests between #326 and #509. ``Type[Dumper]`` -- so whatever eventually renders this attribute's type still has the same ambiguity to resolve. This half of the fix is insurance against the day something reads it cleanly. This pair of sites - is what closes #709. Four more bare ``Type`` sites -- hand-written + is part of #709. Four more bare ``Type`` sites -- hand-written ``:type:`` fields at ``docs/source/pcapkit/foundation/engines/engine.rst:40``, ``.../reassembly/reassembly.rst:33`` and ``:43``, and From 7f23f989c34976e5f5f1c746b490b3d8618b8a3b Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 23 Sep 2026 21:27:01 -0400 Subject: [PATCH 46/46] docs(changelog): fix the #651 entry's stale HIP claims and the html-warning count Round 8: ran the merge-base/changed-files/cited-path intersection to completion (0c7f2b7c9..origin/main: 43 commits/124 files; 54 cited paths, 34 of them touched by that range) instead of trusting a tense grep. - :1540/:1542 -- "93 of the 95 sites"/"48 of the 49 record lengths" -> 94 of 95 / 49 of 49; zero old-expression sites remain on main. - :1544-1546, :1566 -- "LOCATOR_SET keeps the old expression ... is currently right" / "is unchanged in both respects" -> past tense; #679 fixed both LOCATOR_SET sites. - :1566 -- "HIP_COPIES stays at two" -> past tense; #679 fixed LOCATOR_SET's Length unit, #689 then dropped HIP_COPIES to one. - :1976-1977 -- "pcapng.txt ... wants a separate refresh" -> past tense; #685 removed it from the index instead of regenerating it. - :2327-2328 -- "55 warnings on main before this change, 56 after" (-b html) -> 53/54; the raw `grep -c WARNING:` double-counts two Scapy import lines as Sphinx warnings, confirmed live on this head with both -b dummy and -b html (real 36/raw 38 with const/reg.rst excluded, same +2 gap either way). - :834 -- stale ``protocol.py:1016`` -> ``:1413`` (the actual ``self._file.read()`` call inside ``_read_fileng``). - :1969 -- the #646 entry's coverage renumbering was wrong twice over (first ``1153 to 1265``, then ``1153 to 1443``, the latter being main's ``def`` line, which always executes and can never be the single miss). Corrected to ``1248 to 1360``, the ``warn(...)`` statement's line before/after #646's own diff. Regenerated CHANGELOG.md from the edited entries. changelog_md.py --check: exit 0. pytest tests/project/test_changelog_md.py -q: 47 passed, 37 subtests. Follow-up: origin/main advanced through #726/#740/#741/#742 (to 0a3abffc8) and then #747/#748 (to 074c53eeb) while this sat at good-to-go; #726 moved three more claims anchored on files it touched. - :834 -- ``protocol.py:1413`` -> ``:1411``; #726 shifted the ``self._file.read()`` call in ``_read_fileng`` by -2 lines. - :2382-83 -- traceflow.py "Line 406" -> "Line 424"; #742 inserted 18 lines above the ``#: Type[Dumper]: Dumper class.`` comment. - :2099-2104, :2187-89 -- the "seven code-keyed parser registrars" and ``Option.register`` are no longer presence-only. #726, fixing #718, gave all seven -- and ``Option.register`` itself -- the same identity guard ``register_protocol`` already had; reworded both passages to say so, confirmed against the guards' own current docstrings. Regenerated CHANGELOG.md again. changelog_md.py --check: exit 0. pytest tests/project/test_changelog_md.py -q: 47 passed, 37 subtests. Cross-review at 948ac495d came back NEEDS CHANGES: the round-12 edit fixed two sites of the harmonisation claim and left its twin, plus its own reasoning, asserting the opposite; and four numbers anchored on files the merges touched had drifted independently of #726. - :1877-1878, :1883-1884 (#675) -- "carries the guarded ``if code in cls.__xxx__: warn(...)``" / "every sibling warns on mere presence" -> past tense, noting #726 later gave all seven the identity guard this entry's own comparison assumes they lack. - :2100-2109 -- dropped the retained "yields two keys and never reaches one key twice" (false: ``Internet.register(TransType.TCP, TCP)`` warns once, incumbent.klass is TCP) and "leaves a different-class test undecidable" (contradicted by :2404-2406's own ``incumbent is not protocol`` definition); replaced with the actual false positive the guard has -- pre-seeded ``ModuleDescriptor`` incumbents never compare equal to the resolved class. - :2192 -- reflowed the ``Option.register`` paragraph (orphan lines fixed alongside). - :2387-2388 -- the ``Type[Dumper]`` quote now matches what is actually at line 424 (post-#709-fix), rather than the pre-fix bare form. - :945 -- ``README.md`` (103) -> (102). - :1136 -- "75 of the 117 modules" -> "77 ... after #647 below adds the same ending to three more" (drifted via #647, independent of the four merges). - :2119 -- dropped the irreproducible pylint "364 messages" figure; kept mypy's 112, which does reproduce. - :2119 -- "326 registry writes" -> 327 (``R1CounterParameter``'s second code, from #690). Also fixed six false claims in the PR body (separate from the .rst): hunk/line counts, six-commits -> 46, the 5-row table's implied total, "not trimmed", main's red/green state, and the now-unreachable cherry-pick target. Regenerated CHANGELOG.md again. changelog_md.py --check: exit 0. pytest tests/project/test_changelog_md.py -q: 47 passed, 37 subtests. Cross-review at 1749cc029 came back NEEDS CHANGES: round 13 fixed five of the nine sites and introduced four new false claims doing it, including two inside the flagship rewrite -- swapping one inaccuracy for another is this document's recurring failure mode. - :1136-37 -- "75 ... 77 now, after #647 ... adds ... three more" was internally inconsistent (75+3=78, not 77). Traced #647's own diff (fc32d1b81): it adds ``_missing_`` to three IntFlag classes across only two *new* files -- ``tcp/flags.py`` and ``ftp/command.py`` -- since the third, ``TransportProtocol``, shares ``reg/apptype.py`` with the already-counted ``AppType``. Module delta is +2, matching 75+2=77; reworded to say so. - :1879-88 -- dropped "the comparison below assumes they still lack" it, which was false about text 8 lines below in the same diff (already past-tensed). Also reflowed three orphan lines this introduced (`passes, whereas`, `it twice with nothing`, `none of the`). - :2107-19 -- "These tables also ship pre-seeded" over-generalised: verified live (``ProtocolBase.__proto__`` is 0 entries, ``Transport.__proto__ is ProtocolBase.__proto__`` -- True) that 2 of 7 have nothing pre-seeded. Scoped to the five that do (Link 7, Internet 16, Frame 3, PCAPNG 3, SCTP 2). Also fixed "the guard resolves only the incoming class", which contradicts the guard's own docstring ("the comparison itself resolves nothing") -- resolution is the earlier ``isinstance(protocol, ModuleDescriptor)`` step, three lines above the guard, not something the guard does. - :2129-30 -- dropped the invented "327th" ordinal (327 total stays; traced-write instrumentation via ``sys`` hooks found the seeding is literal dict construction, not ``.register()`` calls, so I could not reproduce an ordinal with confidence -- said "one of them" instead of guessing). - :7-8 -- "between #326 and #509" now says the programme continued past it (verified: 193 distinct #nnn refs, max #726, 103 above 509). - PR body -- "7 hunks, 1168+/11-" was the previous head's figure, not this one's; replaced with the actual command (``git diff --shortstat da697fa93 -- docs/source/changelog/1.5.0.rst``) and today's figure (9 hunks, 1152+/16-), since a hardcoded count here has now gone stale twice. Left alone per this round's scope: :1969/:1974 (before/after claim, not falsified by #726's later +1), mypy "112" (correct, re-ran with the project's own flags), ":2122" 13-to-14 (correct at its delta scope), and the other 121 cited paths (unaffected by main's one new commit, #745, confirmed test-only). Regenerated CHANGELOG.md again. changelog_md.py --check: exit 0. pytest tests/project/test_changelog_md.py -q: 47 passed, 37 subtests. Cross-review at b2ac58b2f came back NEEDS CHANGES: round 15 fixed four sites clean but swapped in two new inaccuracies, and left one round-fourteen defect (body :26) unfixed. - :7 -- "past #726" was wrong direction: #726 is the max ref in the document (193 distinct, min #251, max #726), not one exceeded -> "reaching #726". - :2110-19 -- "ProtocolBase and Transport share one dict, starting and staying empty until a subclass registers" was false three ways, verified live against origin/main (pcapkit.__file__ asserted): Transport.register() itself raises UnsupportedCall (abstract); TCP and UDP keep their own separate __proto__ (4 and 3 entries), not the shared one, so registering on them leaves the shared dict at 0; only a direct ProtocolBase.register() call fills it. Narrowing to "five of these seven" also hid that TCP/UDP are pre-seeded too, which is exactly where the false positive bites in the transport family -- restored that. - body :26 -- "26 entry commits" -> 27 (commits whose subject starts "docs(changelog): the 1.5.0 entry/entries for", verified by grep), 28 bullets added and 0 removed (verified via the .rst diff against da697fa93; one commit, 6a956c478, adds two bullets for #648/#649). - body :41 -- dropped the hardcoded "9 hunks, 1152+/16-" figure entirely (it had already drifted to 1154+ by the time of this commit) and named the second command needed for the hunk count, since --shortstat cannot print one. On the ordinal question raised last round: dropping it was still right (the asserted "327th" was wrong), but "no ordinal is derivable" does not hold either -- the writes are at pcapkit/protocols/schema/schema.py, not the 8 dict-literal registrar sites my instrumentation covered, and they are traceable. Left the text as "one of them being R1CounterParameter's second code" (no ordinal asserted, no false derivability claim either) rather than reopen a site outside this round's scope. Regenerated CHANGELOG.md again. changelog_md.py --check: exit 0. pytest tests/project/test_changelog_md.py -q: 47 passed, 37 subtests. --- CHANGELOG.md | 26 ++--- docs/source/changelog/1.5.0.rst | 164 ++++++++++++++++++-------------- 2 files changed, 107 insertions(+), 83 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 89c2c7667..3c993af01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ ## 1.5.0 -- unreleased -The largest release since 1.0, and the first recorded here as it happened rather than reconstructed. Three more extraction engines, ESP with payload decryption, SCTP and NGAP over SCTP, a library logger that no longer hijacks the consumer's, and a defect programme run through the issue tracker across some 140 issues and pull requests between #326 and #509. +The largest release since 1.0, and the first recorded here as it happened rather than reconstructed. Three more extraction engines, ESP with payload decryption, SCTP and NGAP over SCTP, a library logger that no longer hijacks the consumer's, and a defect programme run through the issue tracker that started across some 140 issues and pull requests between #326 and #509, and has continued well past #509 since -- reaching #726 by the entries below. - **Added** -- three extraction engines: `engine='pypcap'` and `engine='pcap_ct'`, two independent distributions of the same `libpcap` interface, and `engine='pypcapfile'` (#386, #405). They buy speed by doing less -- neither `pypcap` nor `pcap_ct` dissects at all, so they offer neither reassembly nor flow tracing, and `pypcapfile` has no IPv6 decoder. Install only **one** of `pypcap` and `pcap-ct`: both own the top-level `pcap` module, and with both present `pcap-ct` wins the import and the other becomes unselectable. The matching interface constants `PyPCAP`, `PCAP_CT` and `PyPCAPFile` were missing and are now exported alongside `DPKT`, `Scapy`, `PyShark` and `PCAPKit` (#412). That brings the built-in set to seven engines; 3.11 is the last interpreter on which every one of them can run, and even there two of them cannot coexist. - **Added** -- `EngineBase.unsupported_reason`, a preflight every engine answers and `Extractor.run` consults before anything is imported. Asking for an engine that cannot run in the current environment now gives one warning naming the real cause -- a Python version, a missing `tshark`, a missing `libpcap`, the wrong `pcap` distribution -- and a clean fall back to `pcapkit`'s own parser, rather than an error from inside the third-party package (#396, #405). @@ -65,20 +65,20 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- `_RE_STATUS` in the same reader carried the same unanchored prefix defect, found by auditing `_RE_METHOD`'s siblings, and it escaped as the wrong exception type. That pattern is only a guard -- the value is taken from `int(para2)` on the raw token -- so a prefix match let a malformed status past the guard and then out of `int()` uncaught, where `_read_http_header` documents `ProtocolError`. Measured: a status of `200x` raised `ValueError: invalid literal for int() with base 10: b'200x'`, and one of `2000` raised `ValueError: 2000 is not a valid StatusCode`; both are now `ProtocolError`. [RFC 9112 Section 4](https://datatracker.ietf.org/doc/html/rfc9112#section-4) gives `status-code = 3DIGIT`, exactly three, so the anchor is what the grammar already said -- the production lives in HTTP/1.1 because `status-code` is part of its `status-line`, while [RFC 9110 Section 15](https://datatracker.ietf.org/doc/html/rfc9110#section-15) covers the code semantics and the IANA registry rather than the syntax. `_RE_VERSION` was audited at the same time and is safe as it stands, because both of its call sites read the captured group rather than the raw token (#583). - **Fixed** -- `get()`'s documented `default` was ignored on the integer path throughout the generated `pcapkit.const` tree, because `get` delegated the lookup to the enum call and `_missing_` has no access to the caller's `default` -- so `Hardware.get(99999, 0)` raised `ValueError: 99999 is not a valid Hardware` instead of returning the fallback it was handed. The integer path now consults `default` before letting the lookup error escape. `-1`, the placeholder the generated signature already carried, is what separates "no default was supplied" from "a default was supplied and should be used", so a caller that asked for no fallback still gets the error rather than a silent substitution. The sweep #584 asked for puts the scope at 110 of the 118 integer registries, not the three the issue named; the two carrying a bespoke integer fallback of their own, `pcapng` `OptionType` and `reg` `AppType`, are deliberately left alone, since neither drops a default by raising. Not reachable from wire data -- every value a wire field can carry already resolves -- so this is a contract fix rather than a parse fix. Applied to the nine vendor templates as well as the 113 generated modules, and a new test renders the shared template and compares it against the module generated from it, so a regeneration cannot quietly undo it (#584). - **Fixed** -- `FieldBase.unpack` padded a short read on the wrong side. When a field's buffer falls short of its declared length -- the deliberate accommodation that lets a snapshot-truncated capture parse (#431) -- the shortfall was zero-filled with `rjust()`, which places the zeros at the *front*. That asserts the octets never read were the leading ones, and a short read has lost the trailing ones: the buffer ran out. It is now `ljust()`, which is correct for **both** byte orders rather than only for big-endian fields, so the correction is not byte-order-conditional. The two directions fail differently and only one was loud. Measured: one octet of a four-octet little-endian `120` read as `2013265920`, inflated by `2 ** 24`; three octets of a four-octet big-endian `0x01020304` read as `0x10203`, scaled *down* by 256. The second is the dangerous one -- a value smaller than the truth passes a sanity check, where an inflated one overruns -- which is why the big-endian half went unnoticed. A full read is untouched at every width and order, since the padding is only ever consulted when the buffer falls short (#604). -- **Fixed** -- as a consequence of the above, the unhandled `MemoryError` at `pcapkit/protocols/protocol.py:1016` on a truncated PCAP-NG capture. `dhcp_little_endian.pcapng` cut to 161, 641 or 1389 octets left a one-octet read of a little-endian 32-bit block length, which `rjust()` turned into `0x78000000` or `0x84000000` -- 1.88 to 2.06 GiB -- and which was then passed straight to `self._file.read()` as an allocation size, from a 1772-octet file. Under a 1 GiB address-space cap all three raise `MemoryError` there; with more address space the allocation succeeds and the parse goes on to fail anyway, so the visible symptom depended on how much memory the process could get. With `ljust()` the same reads report 120 and 132, no large allocation is attempted, and all three end in the ordinary, already-handled parse failure instead (#604). +- **Fixed** -- as a consequence of the above, the unhandled `MemoryError` at `pcapkit/protocols/protocol.py:1411` on a truncated PCAP-NG capture. `dhcp_little_endian.pcapng` cut to 161, 641 or 1389 octets left a one-octet read of a little-endian 32-bit block length, which `rjust()` turned into `0x78000000` or `0x84000000` -- 1.88 to 2.06 GiB -- and which was then passed straight to `self._file.read()` as an allocation size, from a 1772-octet file. Under a 1 GiB address-space cap all three raise `MemoryError` there; with more address space the allocation succeeds and the parse goes on to fail anyway, so the visible symptom depended on how much memory the process could get. With `ljust()` the same reads report 120 and 132, no large allocation is attempted, and all three end in the ordinary, already-handled parse failure instead (#604). - **Added** -- `FieldBaseShortReadPaddingSideTests`, ten cases over 133 subtests covering both byte orders at 2, 4 and 8 octets and at every truncation point: the two figures #604 reports as literals, the value-preserving property stated over every width and shortfall rather than as a table, the two failure directions as inequalities, a signed field, a byte-string field, and an unpack-then-pack cycle. Eight of the ten fail on the unfixed tree. The other two must pass on both and are the guard rails -- a full read at every width and order, which may not move, and a read against an entirely empty buffer, which pads to all zeros either way and is the #431 behaviour the option and list loops depend on (#604). - **Added** -- `CITATION.cff`, citation metadata in Citation File Format 1.2.0, which GitHub renders as the repository's "Cite this repository" button and which citation managers and dependency inventories read directly. It is the machine-readable half of the attribution BSD-3-Clause already asks for, so credit carries into a paper or a bill of materials rather than depending on a reader opening `LICENSE`. Validated with `cffconvert --validate` and against the published 1.2.0 schema; `doi` and `orcid` are omitted rather than invented, since neither exists for this project today and both are checked formats, so a wrong value would still validate. The licence itself is deliberately unchanged -- still BSD-3-Clause, no `NOTICE` file, no change to its terms. Alongside it the copyright line moves from `2018-2023` to `2018-2026` -- `LICENSE` was its only occurrence in the tree, since `docs/source/conf.py` already derives its own from the current year -- and a stray `s` after the closing `DAMAGE.` of the licence text, present since the Mozilla-to-BSD relicence, is removed, so the wording now matches canonical BSD-3-Clause exactly (#615). - **Fixed** -- reading a big-endian classic PCAP byte-swapped every record header field, and then crashed. `Frame.unpack` seeded the file's declared byte order under the key `bytesorder` where the frame schema's `byteorder_callback` reads `byteorder`, so the lookup never found it and always fell back to `sys.byteorder` -- the reading host's order rather than the file's. On a little-endian host reading a little-endian capture that fallback gives the right answer by coincidence, and every capture in this repository was little-endian, so the wrong code path has always produced correct results. Against a big-endian capture, measured before the fix, frame 1 of `big_endian.pcap` read `ts_sec=3106905`, `ts_usec=1088553216` and `incl_len=1241513984` for a record whose real values are `1500000000`, `123456` and `74`, dating the frame to 1970-02-05 rather than to 2017-07-14. `incl_len` is the payload length, so that first record then consumed the whole file and the second was read with a negative payload length, raising `ValueError: read length must be non-negative or -1` out of the schema -- which is the reported crash, and it is the *second* symptom rather than the first. The sibling `Frame.pack` eleven lines earlier spelled the key correctly, which is what marks this as a slip rather than a second key, and the fallback is what made a misspelled key indistinguishable from an absent one; `byteorder_callback` now records that it is the definition of the key and why the fallback hides a typo (#605). - **Added** -- `examples/generators/endian.py`, and the byte-order tests that read what it writes. There was no big-endian `.pcap` in the repository at all, which is why #605 survived its own code review: the one-character fix leaves the corrected path exactly as untested as the broken one. The generator writes three captures -- `big_endian.pcap` (magic `a1 b2 c3 d4`), `big_endian_nanosecond.pcap` (`a1 b2 3c 4d`, the first fixture to take that branch of the magic-number table) and `little_endian.pcap` (`d4 c3 b2 a1`) -- carrying the *same three records* in each container, so the tests can assert that the byte order makes no difference to what is read out rather than only that the big-endian file matches numbers written down in a test. Frame 3 is captured short, 1200 octets on the wire cut to a 96-octet `snaplen`, so `incl_len` and `orig_len` differ and cannot both be satisfied by one byte-swapped value. `test_frame_endian_runtime.py` drives all three through `extract()` and walks each file's record chain with `struct` to derive its own expectations; a unit-tier case in `test_header_frame_unit.py` builds a two-record big-endian capture in memory instead, so the regression is also caught by the fixture-free selection CI runs on every push. All four fail on the unfixed tree -- the three fixture-backed ones by that `ValueError`, the in-memory one by `AssertionError: 3106905 != 1500000000` -- while the little-endian twin passes on both trees, which is what shows the records themselves are not the variable (#605). - **Changed** -- `tests/protocols/transport/test_tcp_udp_unit.py` now reaches the MP_JOIN dispatchers through `TCP()` itself, instead of assigning a Python `set` to `_flags` on a bare `TCP.__new__(TCP)`. A `set` answers the membership tests `_make_mptcp_join` and `_read_mptcp_join` use, so every flag branch ran and both TCP modules read 100% statement and branch coverage -- while the attribute had neither the `aenum.IntFlag` type production assigns nor the ordering that governs when it exists at all, which is how #587 stayed invisible behind that number and how the `cast('Enum_Flags', 0)` no-op behind it went unnoticed too. Measured on the rewrite, against the 17 tests of that file: revert #587's hoist and two of them fail with `AttributeError: 'TCP' object has no attribute '_flags'` where all 17 passed before; restore the `cast` and two fail with `TypeError: argument of type 'int' is not a container or iterable`, again where all 17 passed. The library is unchanged and the file's tests still pass, so the coverage numbers do not move -- the point is what the same numbers are now worth (#603). - **Fixed** -- documentation. `mptcp_dss_ack_selector`'s note said a corrected field-width lambda "would not have worked" and that fixing it belonged to `pcapkit.corekit.fields.numbers`, which is exactly where #598 then fixed it; the same paragraph sat in `test_tcp_mptcp_length_arithmetic_unit.py`'s module docstring, whose other stale claim was that MP_JOIN "cannot be built through the public `TCP()` constructor at all", true only until #587. A callable-length `NumberField` packs and unpacks both DSS widths now, and wire *absence* was never the obstacle either: `MPTCPDSS.ssn`, `dl_len` and `checksum` have always been `ConditionalField` on the sibling `M` flag, so the class already relied on that wrapper to keep a field off the wire. The `SwitchField` form is kept for the narrower reason the note now gives -- `ConditionalField`'s `length` forwards to the wrapped field without consulting the condition, so it is safe here only because `Schema.pack` and `Schema.unpack` special-case that wrapper by name, whereas a `SwitchField` always resolves to a concrete field. Replacing it would be a behaviour change and is not made (#603). -- **Changed** -- the README is a landing page now, and Markdown rather than reStructuredText. `README.rst` (424 lines) became `README.md` (103), keeping what a reader arriving from PyPI or a search result actually needs -- what the library is, why it exists rather than Scapy or DPKT, how to install it, a worked example, and where the documentation lives -- and dropping the technical detail the documentation already carried. **Module Structure**, **Engine Comparison**, **Engine support by Python version**, **Test Environment**, **Test Results** and **Installation Notes** were each already duplicated in `docs/source/index.rst`, in a fuller form, so they are linked rather than restated. Two blocks existed nowhere else and moved rather than going: **Testing** is now `docs/source/testing.rst`, registered in the index toctree, and the `pipenv` and `make setup` local development block joined the Installation section of `docs/source/index.rst`. Requested by the project owner, and a deliberate exception to the convention that documentation here is reStructuredText -- for the README only, since it is the one documentation file whose renderers are GitHub and PyPI rather than Sphinx. Accordingly `setup.py` reads `README.md` and declares its content type as `text/markdown`, and `MANIFEST.in` gains an `include README.md` line because `global-include *.rst` no longer matches the file. That line is belt-and-braces rather than load-bearing, contrary to what this entry claimed when it was written: setuptools' own `sdist` command ships whichever of `README`, `README.rst`, `README.txt` and `README.md` exists, before `MANIFEST.in` is read at all, so deleting the line leaves the sdist's file listing byte-identical -- 861 entries either way, with an empty `diff` -- and the result still installs. `setup.py` does read the file unguarded, but it reads it from wherever `setup.py` is executing, which when `pip` installs an sdist is the unpacked sdist rather than a checkout, so that read cannot be made to fail by dropping the line either. Of the `include` lines in that file only `CHANGELOG.md` is load-bearing, which the #631 entry below measured independently. Verified with `twine check --strict` against a built sdist and wheel, both of which pass. The rename's own references moved with it, since a change that renames a file owns the references to it: `examples/benchmark/Dockerfile` copies `README.md` -- a literal `COPY` of the old name would have failed the layer outright and taken `make bench`, `make bench-quick` and `run.sh` with it -- and the benchmark harness prose that named the root README as the destination of its generated tables now names `docs/source/index.rst`, which is where those tables went. That covers the `Makefile` comment, `report.py`, `test_harness.py`, `run.sh` and the suite's own README, including the one place whose stated reason had inverted: the emitted markup is kept parseable by plain docutils, which is now a conservative choice rather than a hard requirement, because the page it lands on is rendered by Sphinx. `examples/benchmark/benchmark.py` still says `README.rst` and is left alone, because it means the benchmark suite's own README in the same directory, not the project's (#619). +- **Changed** -- the README is a landing page now, and Markdown rather than reStructuredText. `README.rst` (424 lines) became `README.md` (102), keeping what a reader arriving from PyPI or a search result actually needs -- what the library is, why it exists rather than Scapy or DPKT, how to install it, a worked example, and where the documentation lives -- and dropping the technical detail the documentation already carried. **Module Structure**, **Engine Comparison**, **Engine support by Python version**, **Test Environment**, **Test Results** and **Installation Notes** were each already duplicated in `docs/source/index.rst`, in a fuller form, so they are linked rather than restated. Two blocks existed nowhere else and moved rather than going: **Testing** is now `docs/source/testing.rst`, registered in the index toctree, and the `pipenv` and `make setup` local development block joined the Installation section of `docs/source/index.rst`. Requested by the project owner, and a deliberate exception to the convention that documentation here is reStructuredText -- for the README only, since it is the one documentation file whose renderers are GitHub and PyPI rather than Sphinx. Accordingly `setup.py` reads `README.md` and declares its content type as `text/markdown`, and `MANIFEST.in` gains an `include README.md` line because `global-include *.rst` no longer matches the file. That line is belt-and-braces rather than load-bearing, contrary to what this entry claimed when it was written: setuptools' own `sdist` command ships whichever of `README`, `README.rst`, `README.txt` and `README.md` exists, before `MANIFEST.in` is read at all, so deleting the line leaves the sdist's file listing byte-identical -- 861 entries either way, with an empty `diff` -- and the result still installs. `setup.py` does read the file unguarded, but it reads it from wherever `setup.py` is executing, which when `pip` installs an sdist is the unpacked sdist rather than a checkout, so that read cannot be made to fail by dropping the line either. Of the `include` lines in that file only `CHANGELOG.md` is load-bearing, which the #631 entry below measured independently. Verified with `twine check --strict` against a built sdist and wheel, both of which pass. The rename's own references moved with it, since a change that renames a file owns the references to it: `examples/benchmark/Dockerfile` copies `README.md` -- a literal `COPY` of the old name would have failed the layer outright and taken `make bench`, `make bench-quick` and `run.sh` with it -- and the benchmark harness prose that named the root README as the destination of its generated tables now names `docs/source/index.rst`, which is where those tables went. That covers the `Makefile` comment, `report.py`, `test_harness.py`, `run.sh` and the suite's own README, including the one place whose stated reason had inverted: the emitted markup is kept parseable by plain docutils, which is now a conservative choice rather than a hard requirement, because the page it lands on is rendered by Sphinx. `examples/benchmark/benchmark.py` still says `README.rst` and is left alone, because it means the benchmark suite's own README in the same directory, not the project's (#619). - **Changed** -- `CODE_OF_CONDUCT.md` moves from Contributor Covenant 1.4 to Contributor Covenant 3.0, at the maintainer's request. The text is the canonical 3.0 Markdown fetched from https://www.contributor-covenant.org/version/3/0/code_of_conduct/code_of_conduct.md rather than a transcription, so the pledge, the encouraged and restricted behaviours and the scope are unaltered. Three things needed deciding rather than copying. 3.0 ships two `[NOTE` placeholders an adopter must fill: the reporting channel, which now names `jarryshaw@icloud.com` -- the same contact 1.4 carried and the one `SECURITY.md` already points at as its email fallback -- plus GitHub's report-abuse form for the case a single-maintainer project cannot otherwise cover, a report about the maintainer; and the enforcement section, whose placeholder is an instruction to the adopter and is removed. 3.0 then assigns enforcement throughout to plural "Community Moderators" (and once, inconsistently, to "Community Managers"), which this repository does not have, so all eight occurrences become the singular maintainer. The four-rung ladder -- Warning, Temporarily Limited Activities, Temporary Suspension, Permanent Ban -- is offered as a suggestion and is **kept**, because each rung maps onto a lever one person actually holds on GitHub: a private message, a locked thread, an interaction limit or block, a permanent block. Finally, 3.0 is licensed CC BY-SA 4.0 where 1.4's attribution paragraph carried no licence notice at all, so the attribution now names version 3.0, links the permanent `version/3/0/` URL, carries the CC BY-SA 4.0 notice and link, indicates that changes were made as BY requires, and says explicitly that the share-alike term covers this document only -- the code remains BSD-3-Clause and `LICENSE` is untouched. Rendering was checked against GitHub's own Markdown API rather than assumed: the ladder comes back as four list items each nesting three, which is what #613 had to repair in the 1.4 file when a stray list marker collapsed the whole document into one nested item (#624). - **Fixed** -- the one assertion #604 left pinning the old padding side, which had been red on `mainline` since #621 merged. `TCPUDPUnitTests.test_a_truncated_option_still_parses_its_declared_length` expected a truncated TCP option's `data` as the synthesised zero octets *followed by* the real ones, which is what `rjust()` produced; #621 made the padding `ljust()` everywhere but could not retarget this file, since another change (#612) owned it at the time and editing it concurrently risked discarding work that has since landed. The real octets now come first for both parametrised widths, and the docstring above the assertion says tail-padding rather than left-padding. Test-only: no library code changes, and the sibling case in `tests/protocols/internet/test_ipv4_unit.py` was already retargeted in #621. Measured against `main` at `2221c2d8f`: two subtest failures before, none after (#604). - **Fixed** -- `main` went red the moment #604's `ljust()` landed, because `TCPUDPUnitTests.test_a_truncated_option_still_parses_its_declared_length` still pinned the head-padded short read that fix removed. The `Reserved_79` option declaring `length=12` over 6 real octets now reports `aabbccddeeff00000000` where the test expected `00000000aabbccddeeff`, so both subtests -- `declared_length=12` and `=32` -- failed on that one assertion while the 17 other cases in the file stayed green: the parse itself never changed, only which end the synthesised zeros sit at. The expectation is inverted, and the docstring above it -- which said the short read was *left*-padded and described the value as four zero octets followed by the six real ones -- is corrected to match, since a docstring that contradicts its own assertion is how the stale expectation survived in the first place. The inputs do discriminate: `trailing` is non-zero and the pad width is 4 and 24, so neither subtest would hold under the other order. #621 left this file alone deliberately, because #612 owned it at the time, and merged two minutes ahead of the cross-review verdict that named it (#604, #621). - **Fixed** -- `util/bump_version.py` left `CITATION.cff` naming the previous release. Nothing else in the repository maintains that file -- no workflow, hook or packaging file mentions it -- so every bump since it landed in #615 would have stranded the `version` and `date-released` it renders as GitHub's "Cite this repository" button and that citation managers, Zenodo and dependency inventories read directly. Both fields now move with `__version__`. They have the same standing, since the file's own header says both describe the newest *published* release, and moving only one would assert that 1.5.0b5 was released on the day 1.5.0b4 was; the date is taken in UTC, because seven of the thirty most recent bumps were made late evening in US-Eastern where a local date is a day behind the publish it describes. That the two are the same day at all is measured rather than assumed: the bump is what triggers `create-release.yml`, the median gap to the PyPI upload is three minutes, and the UTC calendar dates agree 30 times out of 30. The rewrite is line-oriented, so the comment header, key ordering and each field's existing quoting survive -- `cff-version` and a `references` entry's own `version` are anchored out at column zero -- and the result is checked with `cffconvert --validate`. An absent file is reported on stderr and skipped rather than failing the vendor cron before its `git commit`, which would discard the whole registry crawl for the sake of a documentation file; a file present with no `version` field raises instead, before anything is written, because rewriting nothing while reporting success is the staleness this fixes. Two things came with it. The script gains a `main()` guard, having previously run the entire bump at import, which is why it had no testable surface; and the `import pcapkit` fallback in its version reader, which returned `"1.5.0b4'\n"` -- closing quote and newline included, which `packaging` rejects -- is fixed, a path that had never worked and went unnoticed because the only caller installs the package first. A new gate asserts the committed file still names the packaged version, covering the version changes made by hand, which never run this script at all -- 40 of the 159 commits that have moved `__version__` on `main`, a quarter over the project's life and 11 of the most recent 25 (#625). - **Fixed** -- `SeekableReader.truncate` put its padding where the reader's own bookkeeping says the content is, and `read` then returned one octet fewer than the buffer held. Same family as #604, but two defects rather than one, and the padding side is only half of it. The buffer keeps its content at `[0:_buffer_cur]` with unwritten padding behind it, so a reduction has to keep the octets it has and an extension has to append at the *tail* -- the latter is what `io.IOBase.truncate` means by "the contents of the new file area", the area past the old end. It did neither: `temp[-size:]` sliced the buffer rather than the content, so it kept the trailing padding and discarded the octets actually read, and `temp.rjust(size)` prefixed the new zeros, displacing the content past where `_buffer_set` and `_buffer_cur` address it. Separately, `read` capped a buffered read at `min(size, self._buffer_cur - 1)`, a count less one measured from the start of the buffer rather than the run remaining from the position being read from -- one octet short at the start of the buffer, and reaching past the content into the padding anywhere further in. The shortfall was then made up from the stream, *past* the octet that had been skipped, which both dropped that octet and left the return short. The reported symptom needed both: `read(4)`, `truncate(8)`, `seek(0)`, `read(8)` over `b'abcde'` returned `b'\x00\x00\x00e'`, four octets of an eight octet request with three of them padding, and returns `b'abcde'` now -- five being the whole of what a five octet stream can answer with. Three more of the method's contract were wrong and are fixed with it: the position was reset to the start of the buffer rather than left alone, so a read after a truncation resumed from the wrong octet; an omitted `size` resized to `0` rather than to the current position; and `_buffer_cur` was left addressing octets a reduced buffer no longer had, so the next read raised `ValueError: memoryview assignment: lvalue and rvalue have different structures` from `_write_buffer` rather than returning anything. `truncate(0)` raised the same `ValueError` by a second route, and `_write_buffer` is fixed with it: `buf[-self._buffer_size:]` is `buf[-0:]` for a buffer of no size at all -- the whole of the octets just read rather than none of them -- so it is now counted from the front. That state is reachable only through `truncate`, since the constructor refuses a non-positive `buffer_size`. A reduction also advances `_buffer_set` past the octets it drops, keeping `_buffer_set + _buffer_cur` equal to how far the stream has been consumed, which `seek` reads as its licence to fetch more. Clamping `_buffer_cur` alone made that sum *under*-report the stream, and the next forward `seek` then spliced in octets from the wrong absolute offset and said nothing: `read(8)`, `truncate(3)`, `seek(6)`, `read(1)` over `b'abcdefghijklmnop'` returned `b'l'` where `b'g'` is the octet at offset 6 -- worse than the `ValueError` it replaced, being silent. All three came out of fuzzing random operation sequences against the buffer's own invariants, none from reading the code: of 3000 rounds, 2372 left the bookkeeping inconsistent before and 683 raised an undocumented exception, and none of either do now. Latent in this library rather than live -- nothing here calls `truncate`, confirmed by grep, though it is public on a public class -- and invisible to the existing tests, which asserted the return value and never the content (#622). -- **Fixed** -- `_missing_` in the four Mobility Header flag enumerations ended in `return cls(value)`, the same constructor that had just failed to find the value, so every in-range value that is not already a member re-entered `_missing_` unbounded and raised `RecursionError`. `BindingACKFlag`, `BindingUpdateFlag`, `HandoverACKFlag` and `HandoverInitiateFlag` are `IntFlag` types whose members are single bits, so the hole was not some exotic bit pattern: `F(0)` -- no flags set, the most ordinary value a flag octet can carry -- recursed on all four, and so did every composite of two defined bits, such as `BindingACKFlag(0x06)`. Defining `_missing_` at all is what caused it, because it shadowed the `aenum` `Flag` machinery that resolves exactly those values; `pcapkit/const/tcp/flags.py` defined no `_missing_` at the time and never had the defect -- it has one now, added by #647 below, ending in the same `super()._missing_(value)` tail for the same reason. All four now end in `return super()._missing_(value)`, which is what `pcapkit/vendor/default.py` emits for every other generated enumeration and what 75 of the 117 modules under `pcapkit/const/` already do, so `F(0)` is an empty flag and `BindingACKFlag(0x06)` is `S|D`. The `extend_enum` idiom that `pcapkit/const/pcapng/record_type.py` and `secrets_type.py` use to mint a member for an unassigned integer -- the only other two modules whose `_missing_` ends in `return cls(value)`, and which do not recurse precisely because that `extend_enum` runs first -- was considered and rejected for a flag type: naming `0x06` `Unassigned_0x06` would hide the composite and pollute `_member_map_` with an entry per bit pattern, up to 65536 of them for `BindingUpdateFlag`. The range guard above it is untouched, so an out-of-range or non-integer value is still the same `ValueError`. Fixed in the four `pcapkit/vendor/mh/` templates and regenerated, the `pcapkit/const/mh/` modules being generated output that the next crawl would otherwise revert (#623). +- **Fixed** -- `_missing_` in the four Mobility Header flag enumerations ended in `return cls(value)`, the same constructor that had just failed to find the value, so every in-range value that is not already a member re-entered `_missing_` unbounded and raised `RecursionError`. `BindingACKFlag`, `BindingUpdateFlag`, `HandoverACKFlag` and `HandoverInitiateFlag` are `IntFlag` types whose members are single bits, so the hole was not some exotic bit pattern: `F(0)` -- no flags set, the most ordinary value a flag octet can carry -- recursed on all four, and so did every composite of two defined bits, such as `BindingACKFlag(0x06)`. Defining `_missing_` at all is what caused it, because it shadowed the `aenum` `Flag` machinery that resolves exactly those values; `pcapkit/const/tcp/flags.py` defined no `_missing_` at the time and never had the defect -- it has one now, added by #647 below, ending in the same `super()._missing_(value)` tail for the same reason. All four now end in `return super()._missing_(value)`, which is what `pcapkit/vendor/default.py` emits for every other generated enumeration and what 75 of the 117 modules under `pcapkit/const/` already did -- 77 now, since #647 below gives the same ending to three more classes but only two land in modules the count did not already have, `TransportProtocol` sharing `reg/apptype.py` with the already-counted `AppType` -- so `F(0)` is an empty flag and `BindingACKFlag(0x06)` is `S|D`. The `extend_enum` idiom that `pcapkit/const/pcapng/record_type.py` and `secrets_type.py` use to mint a member for an unassigned integer -- the only other two modules whose `_missing_` ends in `return cls(value)`, and which do not recurse precisely because that `extend_enum` runs first -- was considered and rejected for a flag type: naming `0x06` `Unassigned_0x06` would hide the composite and pollute `_member_map_` with an entry per bit pattern, up to 65536 of them for `BindingUpdateFlag`. The range guard above it is untouched, so an out-of-range or non-integer value is still the same `ValueError`. Fixed in the four `pcapkit/vendor/mh/` templates and regenerated, the `pcapkit/const/mh/` modules being generated output that the next crawl would otherwise revert (#623). - **Fixed** -- the HIP `SOLUTION` builder sized the parameter with `4 + math.ceil(max(random.bit_length(), solution.bit_length()) / 4)`, which is an invalid shorthand for the two fields it actually has to describe. [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5) gives the parameter's `Length` as `4 + RHASH_len / 4` over a `Random #I` and a `Puzzle solution #J` of `RHASH_len / 8` octets **each**, and `/ 4` equals twice `/ 8` only because `RHASH_len` -- the natural output length of a hash function, in bits -- is a whole number of octets. Applied to an arbitrary `bit_length()` the identity fails, and the builder emitted an *odd* contents width, which its own reader then refused: `_read_param_solution`'s `(schema.len - 4) % 2` guard exists because `SolutionParameter` splits that width into two equal `(len - 4) // 2` halves, so an odd width cannot be framed at all. `random=0x1` with `solution=0xfff` declared `len=7` and raised `ProtocolError: HIPv2: [ParamNo 321] invalid format` on input the library had itself produced. The undersized length was the worse half of it: because both fields take their width from that same `len`, `solution=0xfff` was packed into the one octet it allowed and came back as `0xff` -- silent truncation, nothing raised. Fixed to `4 + 2 * math.ceil(max(...) / 8)`, the form the sibling `PUZZLE` builder already uses, which is even by construction and so can never trip the guard. Loosening the reader was rejected as the alternative: an odd contents width has no meaning in the wire format, since the RFC makes the two fields equal-width and says nothing about which would take an extra octet, and a laxer guard would still have truncated the value (#608). - **Added** -- `SOLUTION` parameter width coverage in `tests/protocols/internet/test_hip_unit.py`: eight widths asserting the declared length, the reader's acceptance and the construct-pack-parse cycle, plus a 57-bit case pinning the 20-octet length [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) requires of HIPv1. Five of the eight widths -- 1, 9, 12, 17 and 25 bits -- are deliberately *not* multiples of eight, because at a multiple of eight the defective `ceil(bits / 4)` coincides with the correct width, which is why every fixture that reached this builder passed through it unharmed; the round-trip case in `examples/generators/options.py` supplies no `random` or `solution` at all, so it exercised the formula at zero bits. 57 bits is what discriminates on the HIPv1 path for the same reason 64 does not: both formulas give 20 at a full-width value (#608). @@ -92,7 +92,7 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- `extract(..., no_eof=True)` never returned. End of stream was detected correctly and `ExtractionWarning: EOF reached` fired, but all three loops that handle it -- `record_frames`, `__next__` and `__call__` -- read `if self._flag_n: continue` with nothing else to stop them, so the flag suppressed the error that *ended* the loop without supplying any other ending and an exhausted input was retried forever. The flag is not pointless, which is why this is a termination condition rather than a deletion: `__main__` sets `no_eof` exactly for `fin='-'`, and the end of what has arrived on a live capture is not the end of the capture. What was missing was a way to tell a capture that has paused from one that is over, and the input's own position answers it -- `prepare` raises end of stream when the bytes remaining measure zero and restores the position before raising, so two consecutive ends of stream at the same position mean nothing arrived between them. `Extractor._note_eof_progress` is that check. A pipe is unaffected, because a read there *blocks* while the writer is open but idle rather than reporting end of stream -- measured at a full 1.5s pause on `6c3d1b0d9`, after which the frames arrived -- so a paused pipe never reaches the check at all. Also fixed thereby, and not mentioned in the issue: `pcapkit -` hung on **any** finished stdin, a pipe reporting end of stream once its writer closes; verified through the real CLI, where `cat in.pcap | python -m pcapkit -` went from killed at 25s to exiting 0 with a byte-identical dump. **One deliberate narrowing**: a *seekable* input does not block, so its two probes fall microseconds apart and a file still being appended to now ends at the data present when the extraction reached it -- measured with the append landing on a record boundary 0.6s in, six frames before and five after. The previous behaviour was unbounded by construction, which is the defect itself, so some stopping rule had to be chosen; a timed grace period would make the cut-off intermittent rather than absent, so following a growing file is left to a policy of its own and the decision is pinned by a test. Worth knowing alongside it, pre-existing and untouched: an append landing *mid-record* raises `ValueError: read length must be non-negative or -1` instead, so `no_eof` over a growing file only ever worked for a writer flushing whole records. Both docstrings for `no_eof` now say all of this. The end-to-end regression tests are bounded by a **child process** rather than by `tests._support.time_limit`, because the in-process deadline proved *intermittent* on this loop: it usually expired on time and once escaped entirely, running past ten minutes and 13.2 GB RSS before being killed by hand. An intermittent guard against a hang is worse than none, since the run it misses is a wedged suite rather than a red test. Two guesses at the cause are recorded as ruled out, so they are not made again: the parse path's two `except Exception` handlers are never entered during the spin, because `prepare` raises before any next-layer decode, and the memory is the retry loop emitting tens of thousands of `EOF reached` records a second which the runner retains (#620). - **Changed** -- building a protocol through its constructor with a keyword that names nothing now raises `UnsupportedCall` instead of discarding it. **This is a behaviour change to a public API**: every `make` in the tree ends its signature with `**kwargs` and reads nothing out of it, so until now a misspelled keyword was accepted, dropped, and the field it named kept its default -- wrong octets, with nothing said. That is what #602 cost: `examples/generators/options.py` asked for `seq=1` where `TCP.make` spells the parameter `seq_no`, and 25 generated fixture frames carried sequence number `0` against an empty `warnings` list. #541 and #556 were the same silence. The schema layer has never been so permissive -- `Schema.__update__` warns `UnknownFieldWarning` for a field it does not know -- and the asymmetry between the two halves of the same construction is what this closes. Checked in `ProtocolBase.__init__` rather than in `make`, because `make` is not the only consumer of the keywords it is handed: `__post_init__` passes one `**kwargs` to the construction *and* to the parse of what it has just constructed, so a keyword declared only by `read` legitimately travels through `make` -- `HIP.read` declares `extension` where `HIP.make` does not, and the option generator depends on it. The accepted set is therefore the union of every keyword-taking parameter of `make`, `read`, `pack`, `unpack`, `__post_init__` and `__init__` across the whole MRO, computed once per class from `inspect.signature`. Parsing is deliberately untouched, since there the keywords are whatever the engines and the four `_import_next_layer` implementations forward and a protocol cannot know which of its ancestors' its parent passed on -- and nothing was ever lost that way, a dropped parse keyword changing how a packet is read rather than what its octets say. Two escapes exist for the shapes a signature cannot express, both opt-in per class through a new `__keywords__`: a set, for a keyword read out of `**kwargs` by name as `ESP.read` does with `packet`; and `None`, for a dispatcher whose real signature belongs to a class chosen at call time, which is exactly `HTTP.make` forwarding to `HTTPv1`/`HTTPv2` and the one place in the tree that uses it. The message names the near neighbour it found, so `seq` reports *did you mean 'seq_no'?*. `from_data` warns `UnknownFieldWarning` where a caller would be raised at, because the keywords there are whatever `_make_data` returned rather than anything anybody typed, so the defect is a key of that mapping disagreeing with the signature it is spread into and the person who meets it is not the person who can fix it. Expect this to surface latent bugs in code that has been quietly losing a field, which is the point. It surfaced four in this repository, all the residue of #602 and all fixed here: the `_TCP_BASE` of `examples/generators/dispatch.py` and three stale copies of it under `tests/protocols/transport/`, each still passing `seq`/`ack_flag`/`urgent_pointer` and so building segments with sequence number `0` where they read as `1`. It surfaced three more that are reported rather than fixed, being a defect per protocol rather than one in this mechanism: `Frame._make_data` returns `ts_src` where `make` declares `ts_sec`, `L2TPv2._make_data` returns `prio` where it declares `priority`, and `Header._make_data` returns a `magic_number` that `Header.make` does not take at all -- so `from_data` has been dropping a frame's timestamp, an L2TPv2 priority bit and a capture's byte order, and now says so. One limitation worth knowing rather than discovering: a *direct* `SomeProtocol.make(...)` call is not checked and still discards in silence, since the check sits where every producer's keywords converge rather than inside each of the 30 `make` implementations -- `object.__new__(cls).make(**kwargs)` is the idiom that reaches it, and `HTTP.make` uses it to reach its versioned implementation (#617). - **Fixed** -- the HIP `PUZZLE` and `SOLUTION` builders derived three wire-format values from the payload value instead of taking them from the data model, and all three were wrong. **This changes two public data models and the octets both parameters emit.** The width of `Random #I`, and of `Puzzle solution #J`, came from `int.bit_length()` alone, and nothing else was available to derive it from, so every leading zero octet was dropped on re-serialisation: a `SOLUTION` read with `Length = 20` rebuilt as `Length = 6`, a `PUZZLE` read with `Length = 12` as `Length = 5`, and silently, because the integers survive and nothing raises. That width is `RHASH_len / 8` octets [[RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4), [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5)], a property of the Responder's HIT Suite rather than of the number that happens to sit in the field, so both data models now carry it as `rhash_len` and both builders prefer it; a new `rhash_len=` keyword states it for a build from scratch, which under HIPv2 is the only place it can come from, since `RSA,DSA/SHA-256` is the REQUIRED HIT Suite [[RFC 7401 Section 5.2.10](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.10)] and makes the field 32 octets rather than 8. On the mandatory path roughly one parameter in 256 has a zero top octet and lost it (#653). `SOLUTION`'s second contents octet is `Reserved`, "zero when sent, ignored when received" [[RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5), and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) identically] -- not the `Lifetime` that only [RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4) defines, and that pcapkit was encoding there as `2^(value - 32)` seconds. It wrote `0x20`, `0x21`, `0x25` or `0x2b` into a field the RFC requires to be zero, and could not write that zero at all: an RFC-conformant `SOLUTION` whose `Reserved` is `0x00` parsed to `timedelta(0)` and then could not be re-serialised, escaping a bare `ValueError` from `math.log2(0.0)` that was not a `BaseError` and so bypassed the library's own error handling entirely. The field is renamed `reserved`, defaults to the mandated zero, and is carried verbatim across a round trip rather than re-derived (#654). And neither builder read its own `version` keyword, so `version=1` and `version=2` computed identical lengths at every bit width, where [RFC 5201 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.4) and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) state both fields as literally 8 bytes and `Length` as literally 12 and 20. Under HIPv1 each builder therefore accepted only a `bit_length()` of 57..64 and built, for everything narrower, a parameter this library's own reader rejects -- the same shape as #608, in the `PUZZLE` builder that #629 never opened (#655). The two remaining `math.log2` sites, both in `PUZZLE` where a lifetime is real, now raise `ProtocolError` rather than letting `ValueError` escape; `ProtocolError` is `(BaseError, ValueError)`, so a caller written around the old bare exception still catches it. Migration: `SolutionParameter.lifetime` is now `reserved` and an `int` rather than a `timedelta`; both `PuzzleParameter` and `SolutionParameter` gained a required `rhash_len`; and `_make_param_solution` no longer takes `lifetime=`. All three were only reachable end to end once #608 was fixed in #629, which removed the parity guard that had been failing these rebuilds loudly first -- so the round trip stopped raising and started quietly emitting a different parameter, which is why they were worth fixing together (#653, #654, #655). -- **Fixed** -- every HIP parameter pcapkit emitted was `4 (mod 8)` octets long, because the padding was computed to align the *contents* rather than the record. Corrected for **45 of the 46** HIP parameters; `LOCATOR_SET` is deliberately excluded, for the reason below. **This changes the octets those 45 parameters write, and the `length` their data models report.** [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1) requires that "all of the encoded TLV parameters have a length (that includes the Type and Length fields), which is a multiple of 8 bytes", and states the arithmetic outright as `Total Length = 11 + Length - (Length + 3) % 8`; all 95 padding sites instead computed `(8 - (Length % 8)) % 8`, which has no `+ 4` inside the modulus and so aligns the contents alone. Across every `Length` from 0 to 63 the result was never a multiple of eight and never the value the RFC gives, and it erred in both directions: at `Length = 4` -- a whole `SEQ`, and [RFC 7401 Section 5.3.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.3.5) puts a `SEQ` or an `ACK` on every `UPDATE` -- the record is complete in eight octets and pcapkit appended four that must not be there, while at `Length = 8` the contents were already 8-aligned, nothing was appended, and the record went out four octets short. A conformant peer reading `Length` and consuming `11 + Length - (Length + 3) % 8` octets therefore lands mid-parameter and reads the rest of the parameter area at a wrong offset, in both directions; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test in the suite could see this and why the fix is asserted against the RFC's arithmetic rather than against a round trip. 93 of the 95 sites -- 45 of the 46 `PaddingField` callbacks in `pcapkit/protocols/schema/internet/hip.py` and 48 of the 49 record lengths in `pcapkit/protocols/internet/hip.py` -- are now one `parameter_total_len` and one `parameter_padding_len`, stating the RFC formula once instead of 95 times (#651). `LOCATOR_SET` keeps the old expression at both of its sites, on purpose, because two defects there cancel each other exactly and correcting only the padding would break a parameter that is currently right: its padding callback never receives the parameter's `len` (the nested `Locator` schemas share a packet context whose own `len` shadows it, so the value seen is always 4 for an IPv6 locator), and the parameter's `len` is written in 4-octet units where the RFC's `Length` is a byte count. Always-4 padding gives `4 + 24n + 4`, and because `24n` is a multiple of 8 the RFC total for a byte-count `Length` of `24n` is the same `24n + 8` -- measured at n = 1, 2 and 5 as 32, 56 and 128 octets both before and after. #679 carries the pair. `EncryptedParameter`'s `data` length callback is fixed in the same change and could not have been left: it subtracted the sixteen `iv` octets but not the four `reserved` ones, and those four cancelled the padding's four at `Length % 8` in `{0, 5, 6, 7}` -- so correcting the padding alone would have taken `ENCRYPTED` from right at four of the eight residues to four octets too long at all eight. `HIP.make`'s `len = total_length // 8 + 4` is *not* part of the defect and is unchanged: [RFC 7401 Section 5.1.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.1.3) defines Header Length as the header and parameters "in 8-byte units, excluding the first 8 bytes", so the floor division is exact once each parameter is a multiple of eight, where before it was exact only for an even number of them. Migration: a `SEQ` parameter's `length` is now 8 where it was 12, and the other 44 corrected parameters move likewise, so code comparing stored pcapkit output byte for byte, or asserting on `Data_*Parameter.length`, sees different values -- the RFC's values. `LOCATOR_SET` is unchanged in both respects. `examples/generators/options.py`'s `HIP_COPIES` stays at two, no longer for this reason but for `R1_COUNTER`'s four-octet `counter` where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires eight, which this defect had been masking (#651). +- **Fixed** -- every HIP parameter pcapkit emitted was `4 (mod 8)` octets long, because the padding was computed to align the *contents* rather than the record. Corrected for **45 of the 46** HIP parameters; `LOCATOR_SET` is deliberately excluded, for the reason below. **This changes the octets those 45 parameters write, and the `length` their data models report.** [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1) requires that "all of the encoded TLV parameters have a length (that includes the Type and Length fields), which is a multiple of 8 bytes", and states the arithmetic outright as `Total Length = 11 + Length - (Length + 3) % 8`; all 95 padding sites instead computed `(8 - (Length % 8)) % 8`, which has no `+ 4` inside the modulus and so aligns the contents alone. Across every `Length` from 0 to 63 the result was never a multiple of eight and never the value the RFC gives, and it erred in both directions: at `Length = 4` -- a whole `SEQ`, and [RFC 7401 Section 5.3.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.3.5) puts a `SEQ` or an `ACK` on every `UPDATE` -- the record is complete in eight octets and pcapkit appended four that must not be there, while at `Length = 8` the contents were already 8-aligned, nothing was appended, and the record went out four octets short. A conformant peer reading `Length` and consuming `11 + Length - (Length + 3) % 8` octets therefore lands mid-parameter and reads the rest of the parameter area at a wrong offset, in both directions; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test in the suite could see this and why the fix is asserted against the RFC's arithmetic rather than against a round trip. 94 of the 95 sites -- 45 of the 46 `PaddingField` callbacks in `pcapkit/protocols/schema/internet/hip.py` and 49 of the 49 record lengths in `pcapkit/protocols/internet/hip.py` -- are now one `parameter_total_len` and one `parameter_padding_len`, stating the RFC formula once instead of 95 times (#651). `LOCATOR_SET` kept the old expression at both of its sites for now, on purpose, because two defects there cancelled each other exactly and correcting only the padding would have broken a parameter that was, at that point, right: its padding callback never received the parameter's `len` (the nested `Locator` schemas shared a packet context whose own `len` shadowed it, so the value seen was always 4 for an IPv6 locator), and the parameter's `len` was written in 4-octet units where the RFC's `Length` is a byte count. Always-4 padding gave `4 + 24n + 4`, and because `24n` is a multiple of 8 the RFC total for a byte-count `Length` of `24n` was the same `24n + 8` -- measured at n = 1, 2 and 5 as 32, 56 and 128 octets both before and after. #679 later fixed both sites. `EncryptedParameter`'s `data` length callback is fixed in the same change and could not have been left: it subtracted the sixteen `iv` octets but not the four `reserved` ones, and those four cancelled the padding's four at `Length % 8` in `{0, 5, 6, 7}` -- so correcting the padding alone would have taken `ENCRYPTED` from right at four of the eight residues to four octets too long at all eight. `HIP.make`'s `len = total_length // 8 + 4` is *not* part of the defect and is unchanged: [RFC 7401 Section 5.1.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.1.3) defines Header Length as the header and parameters "in 8-byte units, excluding the first 8 bytes", so the floor division is exact once each parameter is a multiple of eight, where before it was exact only for an even number of them. Migration: a `SEQ` parameter's `length` is now 8 where it was 12, and the other 44 corrected parameters move likewise, so code comparing stored pcapkit output byte for byte, or asserting on `Data_*Parameter.length`, sees different values -- the RFC's values. `LOCATOR_SET` stayed unchanged in both respects for the time being; #679 (below) later fixed both. `examples/generators/options.py`'s `HIP_COPIES` stayed at two for now as well, no longer for this reason but for `R1_COUNTER`'s four-octet `counter` where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires eight, which this defect had been masking -- until #672 widened `counter` and #679 fixed `LOCATOR_SET`'s `Length` unit, after which #689 dropped `HIP_COPIES` to one, once both had left it routing around nothing (#651). - **Fixed** -- two HTTP/2 flag defects, one on each side of a round trip. **This changes the octets a reconstructed DATA frame emits, and the dumped value of a flagless frame's flags.** `_make_http_data` was the only one of the six `_make_http_*` methods that never read `frame.flags`, so a DATA frame parsed with `END_STREAM` set rebuilt with the bit clear: the flags octet went `0x01` to `0x00`, silently, producing a well-formed frame saying the stream continues where the capture said it ended [[RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1)]. `PADDED` was never affected, being re-derived from `pad_len` rather than read back, and all five sibling builders already restored theirs -- so this was a missing read-back rather than a design, and it now reads `frame.flags.END_STREAM` the way they do (#652). Separately, `FrameType.post_process` seeded its flag accumulator with a bare `0`, and `|=` promotes that only as a side effect, so a frame with **no** bit set left `__flags__` a plain `int` where both the schema and the data model declare a `Flags` -- and `END_STREAM in __value__` raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering `False`. That is the common case rather than an edge one: a `0x00` flags octet is every opening `SETTINGS`, every non-final `DATA` and every non-ACK `PING`. It now seeds `self.Flags(0)`, which the construct path had been doing correctly at all six of its own sites. The visible consequence is in the dump, where `__value__` was a JSON *number* for a flagless frame and a JSON *string* for every other frame in the same capture; it is consistently a string now, `"Flags::None [0]"` where it read `0`, the same shape of change #616 made to TCP's `connection`. The seed is guarded rather than unconditional because `FrameType.Flags` declares no members and, from Python 3.11, a memberless `enum.Flag` subclass refuses `Flags(0)` outright, so the five frame schemas that inherit it -- `UnassignedFrame`, `PriorityFrame`, `RSTStreamFrame`, `GoawayFrame` and `WindowUpdateFrame` -- keep the plain `int`, which nothing observes since all five pass `flags=None` into their data objects. Note that a DATA round trip is **still** lossy after this, for an unrelated reason found while measuring it and left to its own change: three payload length callbacks in the frame schemas are mis-parenthesised, so an unpadded `DATA`, `HEADERS` or `PUSH_PROMISE` frame parses with its whole payload dropped (#668). (#652, #650). - **Fixed** -- a flag value with no declared bits no longer dumps as `Type::None [0]`. **This changes user-visible output in every textual dump format.** `make_dumper`'s `object_hook` renders each enumeration member as `Type::name [value]` and interpolated the name unguarded, but a `Flag` value composed entirely of undeclared bits has `name is None` rather than a string -- so the literal four characters `None` landed in the name half and `Flags(0)` rendered as `Flags::None [0]` in `json`, `tree`, `text`, `txt`, `plist` and `xml`, out of both `Extractor` and `TraceFlow`. A nameless member now renders with the value's own decimal spelling instead, so `Flags(0)` gives `Flags::0 [0]` and `Flags(8)` gives `Flags::8 [8]`. That is the spelling the enumeration libraries already use for an undeclared residue -- `Flags(2057).name` is `ACK|9`, naming the declared bit and giving the leftovers as one decimal number -- and it cannot be mistaken for a member name, since a Python identifier may not begin with a digit, whereas `NONE` is a real declared name elsewhere in the library. Three sites carried the identical interpolation, not one: the `OrderedMultiDict` key path, the `addon` branch and the scalar return, which now share one `render_enum` helper so the guard cannot be applied to two of three. That guard is on `name is None` and not on the value, because the defect never was about zero -- `Flags(1)`, `Flags(8)`, `Flags(9)` and `Flags(65536)` are equally nameless -- and it is not an `aenum` quirk either, since a stdlib `enum.IntFlag` answers `name is None` at the same values. Five of the library's seven flag registries are nameless at zero rather than the one reported: `Flags` plus the four Mobility Header flag registries, the other two declaring an explicit `undefined = 0`. Pre-existing since 2023-04-28 and surfaced rather than caused by #634, which made a flagless TCP segment reach this path. The committed example dumps do not move, because no member in them lacks a name (#648). - **Fixed** -- four MPTCP error messages carried a doubled separator, rendering as `TCP: : [OptNo 30] 1: invalid flags combination` with an empty field between the protocol alias and the option number. Cosmetic: the exception type, the option number and the subtype were always right, and nothing downstream parses these strings, but it is the text a user sees when an MP_JOIN or DSS option is rejected and an empty field reads as a value that failed to interpolate. Four sites, not the one reported -- `_read_mptcp_join` and `_make_mptcp_join` for the invalid flag combination, and both guards of `_make_mptcp_dss` for the missing required fields -- against 28 messages in the same file that already spelled the prefix correctly, so the four were outliers against a house form in their own module and the counts are now 32 correct against 0 doubled. Pre-existing since 2023-04-10. The new test asserts each message in full rather than by substring, which is what the existing `assertIn` on the message body could not do, since it passes either side of the change (#649). @@ -102,18 +102,18 @@ This is the resolution of #548, which reported `TransType.L2TP` (115) as registe - **Fixed** -- six of the 123 constant registries under `pcapkit/const/` rejected an invalid value in a way the built-in `enum` does not, and three of them did not reject it at all. `Flags`, `ftp.command.CommandType` and `reg.apptype.TransportProtocol` are `IntFlag` types that defined no `_missing_`, so the `aenum` `Flag` machinery composed a pseudo-member for any integer whatsoever: `Flags(-1)` returned `65520`, the OR of every declared TCP header flag, so a value no 16-bit wire field can hold read back as *every* flag set at once, while `Flags(-65536)` read back as none set. `ftp.command.Command`, `ftp.command.FEATCode` and `http.method.Method` are `StrEnum` types whose `_missing_` reached `value.upper()` before checking the type, so an integer raised `AttributeError: 'int' object has no attribute 'upper'`. All six now raise a bare `ValueError`, which is what `enum.IntEnum` raises for a value it does not define and what 113 of the 117 modules already did. Issue #647's own proposal -- replacing those 113 with `pcapkit.utilities.exceptions.EnumError` -- was deliberately rejected: `EnumError` is `(BaseError, TypeError)` and not a `ValueError` at all, so it would have diverged from the built-in it was meant to improve on, walked past the `except ValueError` in all 113 generated `get()` bodies and silently undone #584, and logged CRITICAL from `BaseError.__init__` once per discarded default. The one deliberate divergence from the built-in is kept and now pinned: the mutable registries look a value up, miss, and `extend_enum` it rather than raising, so `Method('FROBNICATE')`, `ProtectionAuthority(1 << 70)` and `AppType.get(65000, proto=tcp)` still register. The three flag guards end in `return super()._missing_(value)`, so in-range composites still decompose and #623 stays fixed. Two of the four unguarded modules needed no change -- `ipv6/extension_header.py` and `ftp.command.ConformanceRequirement` define no `_missing_` either, and `aenum` already raises the bare `ValueError` for them. `pcapkit/vendor/tcp/flags.py` turned out to carry the guard's value checker all along, as `FLAG = 'isinstance(value, int) and 4 <= value <= 15'`, while its template never interpolated it; those are the registry's *bit offsets* and its members are `1 << offset`, so emitting it unchanged would have rejected every composite, every member above bit 3, and `Flags(0)`. It now reads `0 <= value <= 0xFFFF`, the 16-bit field those bits live in. `TransportProtocol` reads its bound off `cls.__members__` instead of a literal, because `TransportProtocol.get` extends the registry at runtime at `max * 2` and a written-down bound would reject the member it had just grown. Fixed in the four bespoke templates under `pcapkit/vendor/` rather than in the generated tree -- `pcapkit/vendor/default.py` needed no change, since the 113 it emits were already correct -- and regenerated from the live IANA registries: 42 insertions and zero deletions across exactly 4 of the 134 files, so nothing else in the tree was stale, including the 30k-line `reg/apptype.py`. A new `tests/const/test_const_enum_builtin_parity.py` asserts the property the fix is actually about, that a constant registry and a stdlib `enum.IntEnum` raise the same exception type for the same invalid value, over all 123 registries rather than the six; it is keyed on `aenum.Enum` because an `IntFlag` is not a subclass of `IntEnum` -- its MRO runs through `Flag` instead -- which is how three of these six escaped the two existing sweeps. `tests/const/test_const_enum_get.py` drops `Flags` from `EXPECTED_TO_RESOLVE_ANYTHING` and its sweep grows 110 to 111, a registry that bounds its domain finally having a failure for `default` to fall back from (#647). - **Changed** -- `SeekableReader.truncate()` raises instead of returning a size, and the misspelled `writeable()` is now spelled `writable()`. Both are breaks for an external caller and neither breaks anything inside the package. `io.IOBase` documents one gate over two methods -- "If False, write() and truncate() will raise OSError" -- and this reader's `writable()` answers False, so a caller that checked it first, which is exactly what the contract invites, got a surprise either way round: `write` and `writelines` raised, and `truncate` returned its new size. What settled the question is that this is not a pure-Python nicety of `_pyio` that the accelerated path skips -- every ordinary read-only file object in CPython raises here, `open(path, 'rb').truncate()` giving `io.UnsupportedOperation: truncate` off the C `BufferedReader`, and `_pyio.BufferedReader` the same type from `_BufferedIOMixin.truncate`'s `_checkWritable()`. The refusal is `UnsupportedOperation('truncate')`, the same one `write` already raises, and no trade-off was needed between the house exception and the ABC's: pcapkit's `UnsupportedOperation` subclasses `io.UnsupportedOperation`, which subclasses `OSError`, so the in-library exception *is* the one the contract names. The resizing `truncate` used to perform is not deleted, only made private as `_truncate_buffer`. It never touched the underlying stream -- it resizes a private lookback window, which is the counter-argument the issue itself raised -- and it is the only route to the "position sits before the window" state that `seek` and the four buffered read paths must refuse, which #643 and #644 landed tests for; deleting it would have taken their mechanism with them. `writeable()`, separately, was never an override of anything: `'writable' in SeekableReader.__dict__` was False and `SeekableReader.writable is io.IOBase.writable` was True, so `io`, `shutil` and any third-party caller read the inherited value and never saw the one defined in this file. Both answered False, which is the coincidence that hid it -- there was no symptom to notice, and editing the misspelled method would silently have had no effect. The value reported is unchanged and was always honest, the reader genuinely being unable to write; only where the method was defined was wrong. Nothing in the package called either method, confirmed by grep, so the risk is entirely to external callers, which is what made this a question asked before it was acted on. The new test asserts the property rather than the behaviour -- that a `SeekableReader` raises the same exception type a read-only `io.BufferedReader` raises for the same call -- with that type captured by running the call rather than named in the test, so it tracks CPython across the 3.10--3.15 matrix instead of restating a belief about it. 31 to 34 tests and 35 to 41 subtests over `tests/corekit/test_io.py`, the module at 100% coverage before and after (#645). -- **Fixed** -- `register_protocol` displaced one protocol class with another and said nothing about it. The registry behind `pcapkit.protocols.__proto__` is keyed on `cls.__name__.upper()`, and three dispatchable classes are all named `HTTP` -- the generic base `pcapkit.protocols.application.http.HTTP` and the two version implementations in `httpv1` and `httpv2` -- so all three compete for the single key `'HTTP'` and a bare assignment decided the winner. Measured before the fix: `__proto__['HTTP']` went from `pcapkit.protocols.application.http.HTTP` to the `httpv2` class on `register_protocol(httpv2.HTTP)`, with no exception, no log line and no `RegistryWarning`, and `ProtocolBase.expand_comp('HTTP')` then resolved to whichever class had registered last. The overwrite now raises a `RegistryWarning` naming both the displaced and the replacing class, the defining module being the only thing that tells the three apart. The sharpest path to it is not a call to the registrar at all: `Protocol.__init_subclass__` registers unconditionally, so merely *defining* a subclass of the public `Protocol` under a name a built-in already holds displaced that built-in, with nothing anywhere in the user's code to mark the moment the dispatch table changed meaning. A plain `import pcapkit` is unaffected and emits zero new warnings, measured not assumed -- the three built-in `HTTP` classes derive from `ProtocolBase` rather than from the public `Protocol`, so `__init_subclass__` never fires for them, and the import-time seeding in `pcapkit/protocols/__init__.py` keys off the distinct names `HTTP`, `HTTPv1` and `HTTPv2`. The issue's framing that this was the one registrar in `pcapkit/foundation/registry/` with no `RegistryWarning` is right on the count and wrong on the reason: no function in that package warns about anything, the siblings warning only by delegating to a classmethod that carries the guarded `if code in cls.__xxx__: warn(...)`. The accurate statement is the stronger one -- `register_protocol` is the only keyed registrar in that package, and the only `__name__.upper()`-keyed registry anywhere in `pcapkit/`, that mutates its target with no guard *and* no delegate that could supply one, its target being a module-level dict with no class behind it to hold the guard. The new guard deliberately reads "key present and the incumbent is a different class" where every sibling warns on mere presence: the sibling keys are codes the caller passes, whereas this key is derived from the class and this function is the funnel all nine wrapper registrars end in, so registering one class under two codes -- `register_tcp` then `register_udp`, which is supported and documented -- reaches it twice with nothing displaced, and warning there would put noise on a documented path whose wholesale filter is precisely what would then hide the real collision. The collision is reported, not resolved: re-keying is a registry-format change that none of the bare-name readers can absorb, every one of them degrading *silently* to a plain string or to `Raw` on a miss rather than raising, so it belongs to the registry redesign in #514, which this change is sequenced ahead of rather than part of. 9 to 13 tests and 80 to 87 subtests over `tests/foundation/registry/`, the touched module holding 88% coverage with its misses flat at 27 (#675). -- **Fixed** -- every PCAP-NG packet block now carries the captured octets it declares. All four Enhanced Packet Blocks of the committed `examples/captures/dhcp.pcapng` reported `len(packet) == 0` against a `captured_len` of 314, 342, 314 and 342, and the Simple Packet Block and the obsolete Packet Block did the same, measured on a capture synthesised to carry one of each since no committed fixture has either. Those three -- exactly `PCAPNG.PACKET_TYPES`, and exactly the three schemas declaring `__payload__ = 'packet_data'` -- are the whole of the blast radius; every other block type carries no captured octets, reported `b''` before and still does. The octets were read correctly and then thrown away: `PCAPNG.unpack` extracted them from the block schema, and `ProtocolBase.__init__` overwrote the result with `self.packet.payload`, which was empty. The inherited `packet` splits a protocol at `self.length` octets of header and takes everything after as the payload, a contract that holds for a protocol laid out as a header followed by its payload and for nothing else. `PCAPNG.length` returns the wire's *Block Total Length*, so the split consumed the entire per-block buffer as header -- measured at `frame.length == 348` and `len(frame.packet.header) == 348` on a 348-octet block -- and, separately, a PCAP-NG block carries a *trailer* after its payload, the option list and a repeat of the total length, so no value of `length` could have made that split right. `captured_len` comes straight off the block header and survived, which is why the two disagreed rather than both being empty. The fix is at the computation rather than at the injection: `PCAPNG.packet` is overridden to take the payload from the block schema's `__payload__` field and the header from the octets ahead of it, summed out of the schema buffers so that the three block types' three different payload offsets -- 28, 12 and 28 octets -- are not hard-coded. Teaching `ProtocolBase.__init__` to skip a protocol that had already set `packet` was the smaller change and was rejected: it heals `frame.info.packet` while leaving the public `frame.packet` still reporting the whole block as header and `b''` as payload, which is the same defect from the other side. `PCAPNG.unpack` now reads that property instead of extracting the payload a second time of its own, leaving one source of truth where there were two attempts at one. The property is a plain `property` rather than the `cached_property` it overrides, which a cross-review on a second model is what settled: the inherited one caches because it *reads the stream*, where a second read would consume octets that are gone, while this one only walks buffers the schema layer has already filled and so has nothing to amortise -- and caching it would have reintroduced the same staleness by another route, a second `unpack` on one instance handing back the *first* call's octets with `get_payload` never reached, an invariant the code held before this change and has no reason to stop holding. Nothing calls `unpack` twice on one instance today, `__post_init__` being its only caller, so the unit test asserts the property directly: swap the block on a live instance, unpack again, and the payload must be the new block's, which fails `b'payload' != b'cached'` under a cache. `ProtocolBase.packet` gains the contract in its docstring and not one executable line, its 503 statements unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on the success path, which is why it ships labelled breaking: `frame.info.packet` goes from `b''` to hundreds of octets, `frame.packet.header` from the whole block to the pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what made this a wire-format defect rather than an API wart. `PCAPIO` writes `value.packet` after each 16-octet record header, so dumping those four blocks produced a **104-octet** file -- 24 of global header plus four record headers and no payload at all -- in which every record header declared hundreds of octets and delivered none, desynchronising any reader that walks by `incl_len`, which is all of them: pcapkit refused its own output with `ValueError: read length must be non-negative or -1` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes rather than against a length. `tests/protocols/test_pcapng_regression.py` grows 4 tests to 11 and 3 subtests to 24, with the payload expectations derived twice over -- spelled-out head and tail literals, and a `struct`-only re-parse of the fixture that owes nothing to the code under test -- and each synthesised payload given a different length modulo 4 so that an offset off by a field, or a payload that picked up the block's 32-bit padding, cannot pass by coincidence. Two shapes no fixture exercised are covered there too, both found while writing the tests rather than after: an option area *after* the captured data, which every block of `dhcp.pcapng` lacks and which is exactly what an offset walked from the wrong end would swallow, and a big-endian section, whose fields differ even though the payload offsets do not. A snapped block is covered in both the shapes that express it -- an Enhanced Packet Block declaring `captured_len` below `original_len`, and a Simple Packet Block whose captured length is bounded by the interface's `snaplen` -- neither of which may come back padded out to the on-wire length. The offsets hold at 28, 12 and 28 across all of it, which is the property the walk has to have. With both trees running the same -- current -- tests, which is the only comparison that isolates the code from the suite, `pcapkit/protocols/misc/pcapng.py` holds 99.91% across the change, its 18 new statements and 6 new branches all executed, its miss count flat at 1 and its partial-branch count flat at 1, that single miss being the same pre-existing `_get_timezone` statement renumbered 1153 to 1265; `pcapkit/protocols/protocol.py` is identical in every column, 503 statements and 228 misses either side, which is the check that its change really is docstring-only. The uncaught `ValueError` that an EOF-truncated PCAP-NG raises (#678) is untouched and was measured rather than assumed, over a truncation set stated rather than described because "101 levels" admits several readings that do not agree on the counts: the whole-percent prefixes `max(1, 1508 * i // 100) for i in range(101)`, which for this file are 101 distinct lengths from 1 to 1508. Both trees give 96 of that `ValueError`, one `FormatError`, one `ProtocolError` and three parses, and -- compared level by level rather than only in aggregate, since a matching total can hide two levels that swapped -- the SHA-256 of the sorted length-to-outcome mapping is `ca44d3ee658087cf2a667c454f839dd931c1c04790b2fe32cee8e1a2e861b182` on each. `examples/captures/pcapng.txt`, a hand-regenerated legacy smoke reference, still shows the old frame-level `packet -> NIL` and wants a separate refresh (#646). -- **Fixed** -- an unpadded HTTP/2 `DATA`, `HEADERS` or `PUSH_PROMISE` frame parsed with its whole payload silently discarded. Three payload length callbacks in `pcapkit/protocols/schema/application/httpv2.py` put the conditional expression in the wrong place: a conditional binds looser than `-`, so `pkt['__length__'] - pkt['pad_len'] if pkt['flags']['bit_3'] else 0` groups as `(pkt['__length__'] - pkt['pad_len']) if ... else 0` and the `else` arm returned `0` -- "read no octets at all" to `BytesField` -- where it was meant to subtract `0`. Since `__length__` is the *remaining* declared length at the field, the unpadded arm wants `__length__` itself, which is what subtracting a zero padding length gives. The grouping was read off the AST rather than by eye, the top-level node being the `IfExp` whose `orelse` is a bare `Constant(0)`, which is also what showed that the outer parentheses on the `HeadersFrame` and `PushPromiseFrame` forms were line-continuation only and changed nothing. Padding is rare in HTTP/2, so the broken arm was the common one rather than the edge case, and nothing raised, warned or logged: `info.data` was simply `b''`. Measured on wire octets through the public `HTTP(io.BytesIO(raw), len(raw))` path, an unpadded `DATA` frame declaring `b'{"ok":true}\n'` parsed as `b''` and now parses as those twelve octets; unpadded `HEADERS` and `PUSH_PROMISE` lost and now keep the RFC 7541 appendix C.4.1 header block fragment `b'\x82\x86\x84A\x0fwww.example.com'`, which is precisely what HPACK decoding would have been handed. Padded frames are untouched, for a reason stronger than the measurement: `(A - B) if T else 0` and `A - (B if T else 0)` are both `A - B` for truthy `T`, so only the `else` arm could ever have moved -- both arms are asserted anyway, so that a repair to the unpadded one cannot break the other. The padded arm also has no off-by-one, which #668 asked be checked rather than assumed: `Schema.unpack` decrements `packet['__length__']` by each field's width as it goes, so `pad_len`'s own octet is already out of it by the time the payload field runs, and a padded frame declaring `1 + len(data) + pad_len` arrives at the payload with `__length__ == len(data) + pad_len`. The three sibling sites that were never wrong are what localise the defect to the grouping rather than to the field machinery: `ContinuationFrame.fragment`, `UnassignedFrame.data` and `GoawayFrame.debug` use the plain `length=lambda pkt: pkt['__length__']` with no conditional, share the same `BytesField`, the same `__length__` bookkeeping and the same frame dispatch, and carried their payload correctly throughout -- they are pinned as controls rather than left as an argument. Why nothing caught it: `test_option_roundtrip_unit` does drive every frame type, but through `make` -> parse -> `make`, and `make` writes the length field from `_make_http_length`, so both sides agreed on an empty payload and the octets matched; its generator passes no payload argument for these three frames at all, measured as `kwargs={}`, so what it round-tripped was `b''` and there was nothing to lose, while `test_httpv2_frame_readers_cover_successful_frames` drives the readers from hand-built schema stubs, so the callback never ran there either. The new `tests/protocols/application/test_httpv2_payload_length_unit.py` reads wire octets, which is the gap: 23 tests and 12 subtests over the padded, unpadded and padding-only shapes of all three frames plus both `PRIORITY` combinations, asserting the payload *octets* and never its length, because a length assertion survives a read of the right width from the wrong offset. Its padded cases pad with a non-zero `b'\xde\xad\xbe\xef'` rather than the zeros [RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1) tells a sender to use -- pcapkit does not police padding content -- so that failing to subtract the padding yields a different byte string instead of a coincidentally equal count. The same lambda also governs **packing**, which neither the issue nor the first revision of the fix noticed and a cross-review on a second model did: `BytesField` consults its `length` callback on both paths, so the `else` arm did not merely discard a payload on read, it declined to write one. Measured pre-fix, an unpadded `DATA` frame carrying twelve octets of body packed to `000015000000000001` -- nine octets of header whose length field declares **21**, with the body absent -- and unpadded `HEADERS` and `PUSH_PROMISE` likewise declared 29 against 9 and 33 against 13. A frame overstating its own length by its whole payload desynchronises any reader that walks a stream by that field, so this was pcapkit *emitting* malformed HTTP/2 rather than only mis-reading it, and `HTTPv2ConstructedFrameDeclaresWhatItWritesUnitTests` pins that half; a `make` -> parse -> `make` cycle of a non-empty unpadded payload now closes byte-for-byte where it could not before. Six wrong fixes were written over the schema to check the tests discriminate, and each is rejected: dropping the conditional outright (caught by `packet length < 0: -4`), subtracting `pad_len + 1` (caught as `b'{"ok":true}'` against `b'{"ok":true}\n'`), subtracting the padding on the arm where it is absent, fixing `DataFrame` while forgetting the other two, and `max(computed, 1)` on the two `fragment` fields. That last one is the cross-review's own construction and it **passed** the first revision's 15 tests and 9 subtests, because every fixture then carried a non-empty payload and only `DataFrame` had a "frame of only padding" case, so those two callbacks were never once asked for `0`; under it a padding-only `HEADERS` frame returned `fragment=b'\xde'`, one padding octet leaked in, alongside a swallowed `SchemaWarning: packet length < 0: -1`. The three missing padding-only cases and a callback-level `test_the_padded_arm_reaches_zero_and_is_not_clamped` close it -- `0` is a legitimate answer from these callbacks and has to be asserted as one. Seven tests and three subtests fail before the change and all pass after; 145 tests and 550 subtests pass across `tests/protocols/application/`, the round-trip module and the four other HTTP/2-touching modules. `EXPECTED_FAILURES` is unmoved, its one HTTP/2 entry `httpv2-frame/PRIORITY` still failing with the same `CONSTRUCT` status and the same detail -- `PriorityFrame` has neither a payload field nor a `PADDED` flag, so this change cannot reach it -- and none of the 44 entries was deleted; that entry's own cited `pcapkit/protocols/application/httpv2.py:572` is stale, the `header.length != 9` guard it describes now being at `:562`. The schema module was already at 100% coverage and stays there, 101 statements and 4 branches either side with nothing missing or partial, because the changed lines did execute before -- just with an empty payload -- so the number that moved is the suite: 73 to 96 tests and 399 to 411 subtests over the same targets. The construct side needed no code change, `_make_http_length` having always computed the DATA payload as `len(frame.data) + (pad_len + 1 if pad_len else 0)`, i.e. assuming `data` holds the whole payload whether or not the frame is padded -- it was the shared length callback that disagreed, on both paths at once. An AST sweep of all 496 files of the package found one other site with the same syntactic shape, `pcapkit/protocols/schema/internet/ipv4.py:336` (`TSOption.remainder`), and it is correct rather than the same defect: its `else 0` is intentional because the sibling `ts_data` field consumes the entire option data area in that arm, the two summing to `length - 4` across all 2106 `(length, pointer, flag)` combinations, which is exactly what was not true here. Ships labelled breaking on both counts: restoring a dropped payload moves the parse output of essentially every HTTP/2 capture on the success path, so anything holding a golden file or a regression baseline that recorded the empty value sees it change -- and constructed frames change byte-for-byte too, from malformed to correct (#668). -- **Fixed** -- the schema half of every option-like registrar overwrote silently while the parser half of the very same call warned. `EnumSchema.register` was a bare `cls.__enum__[code] = schema`, and it is the schema half of 14 public registrars, so one `register_ipv4_option` call replacing a built-in named the parser it displaced and said nothing about the schema. `EnumSchema.__init_subclass__` reaches the same registry without calling `register`, so a subclass declared with a `code=` keyword -- the documented way to add a schema -- stayed silent too; both are guarded now, and the declaration hook's two branches are folded into one loop so the guard is written once. Presence is a faithful test there only because `_EnumRegistry.__missing__` returns a miss without recording it, so parsing one packet carrying an unknown code cannot make the next legitimate registration for that code warn about an entry no caller ever asked for. The seven code-keyed parser registrars -- on `ProtocolBase`, `Link`, `Internet`, `Frame`, `PCAPNG`, `Transport` and `SCTP` -- now name the displaced entry and its replacement, where before they reported only that something had been overwritten and never which class it was. Their presence-only firing condition is deliberately unchanged, and is not harmonised onto the narrower guard #681 gave `register_protocol`: that one is keyed on a name derived from the value it stores, whereas these seven take a `code` the caller supplies independently of the value, so one class registered under two codes yields two keys and never reaches one key twice. These tables also ship pre-seeded with unresolved `ModuleDescriptor` values, so an incumbent may be a two-string descriptor while the replacement is the very class it names, which leaves a different-class test undecidable without resolving the descriptor and forcing the import it exists to defer. `ContextRegistry.register` already raises on a duplicate, and the reassembly and ESP registrars append to lists with no key at all -- duplicate security-association SPIs being a designed feature there, resolved by scoring rather than by replacement -- so none of those three takes a guard. `import pcapkit` holds at one warning, a third-party deprecation, and no `RegistryWarning`, measured over the 326 registry writes it performs, none of which lands on a key already present; `tests/foundation/registry/` goes 13 to 14 tests and 87 to 95 subtests, the schema module holds 99% coverage with its 5 new statements covered and its misses flat at 1, mypy and pylint are unmoved at 112 errors and 364 messages, and `EXPECTED_FAILURES` is unmoved at 44 entries (#692). -- **Fixed** -- every EOF-truncated PCAP-NG file raised a bare `ValueError` out of `Extractor`, so the whole extraction was lost rather than the one truncated block. The root is one layer above the reported site: Block Total Length is cross-checked against its own trailing copy and never against the file, so a last block declaring more than the file holds left `PCAPNG.read` seeking *past* the real end -- legal and silent -- and the next block read then measured a **negative** remainder, since `prepare` derives it as the end of the stream less the current position. That negative reached `SchemaField` through `pcapng_block_selector` and `io.RawIOBase.read` refused it, which is why nearly every truncation level failed rather than only the one holding the cut. The seek is now clamped to the octets `_read_fileng` actually returned, with a `ProtocolWarning` naming the overrun, and a tail under the twelve octets a block needs at minimum is reported as the quiet `StreamEOFError` that `Extractor.record_frames` already catches -- the one case that is deliberately not a clamp, because at the end of the file no block is being read and clamping would fabricate a frame out of zero padding. A second, independent route to the same shape is closed with a new `nonnegative()` helper, composed into `bounded_option` and `bounded_area` and applied to the eight unwrapped `length - N` spans and the eight `__option_padding__`-sized padding fields: `_TextField.__call__` builds its `struct` template as `f'{length}s'` unconditionally, so a negative became the format `'-8s'` and `struct.calcsize` raised -- neither that nor the `ValueError` being one of `pcapkit.utilities.exceptions`, and neither being an `EOFError`, so neither was caught where the frame loop catches the end of a file. 28 of the module's 74 length callbacks returned a negative before and none do now, from `-1` on an `epb_hash` declaring no payload to `-16777248` on an Enhanced Packet Block's option area; the 200-block, 8,048-octet `captured_len = 0xFFFFFF` vector that reached `struct` through `bounded_area`'s own `nominal <= available` test being true for a negative nominal now parses to 200 frames. Measured over all 1,509 octet boundaries of `examples/captures/dhcp.pcapng` with one harness either side: 6 levels parsed before and 1,495 do now, against 1,479 `ValueError` and 10 `struct.error` before and none from outside the library after, with the frame count degrading monotonically with the cut; of the 14 levels that still raise, twelve leave a file too short to hold a block at all and the other two cut into the Interface Description Block's `if_tsresol` option, so none of them costs a frame that was in the file. That claim is about the truncation sweep rather than about every input: a 4,000-round bounded mutation fuzz either side of the change, same seed and cap, takes the parse rate from 2,118 to 3,438 and removes both of #678's families, and the two foreign families left are measured unchanged -- `ValueError: N is not a valid BlockType`, filed as #701, where an unassigned block type raises from `aenum` inside `EnumField.post_process` so the `UnknownBlock` default the registry declares is unreachable; and `MemoryError` at exactly 30 of 4,000 in both trees, which is #593's 32-bit band. A third bare exception in this module *is* fixed here, found by the cross-review and reachable from valid input rather than truncated: `SystemdJournalExportBlock.post_process` unpacked a 64-bit binary-field length out of whatever `entry_data.read(8)` returned, and since the block body is padded to a 32-bit boundary with NULs that `bytes.strip()` does not strip, that padding was read as a field *name* with no length prefix behind it -- so every journal entry of unaligned length raised a bare `struct.error`, measured on a 14-octet `MESSAGE=hello` entry -- a defect about ordinary input rather than about truncation. A NUL-only line now ends the entry and a short prefix ends it with a warning. The same function leaked two more of the family, found on the cross-review's second pass and fixed with it: a binary field's declared length is the widest in the format and nothing bounded it against the entry holding it, so at `2**63` and above `BytesIO.read` refused it with a bare `OverflowError` while below that it silently returned whatever was there -- the same malformed prefix fatal or invisible by magnitude alone, now clamped to what the entry has left and reported; and a field name, key or value that is not UTF-8 raised a bare `UnicodeDecodeError`, fatal to a whole extraction over one octet in one field, now decoded with `errors='replace'` and reported, which is the option this module's own `StringField` already takes. The separate silent loss of every field after a binary one, from a `read()` that reaches EOF where it means to skip one newline, is filed as #704 rather than fixed. The five deliberately unclamped non-packet option areas keep the per-block framing assumption #676's note describes, which this does not close. Also here because it is the fifth and last registrar in the package with no overwrite guard at all: `Option.register` now reports a displaced option schema as a `RegistryWarning` naming every namespace the `ns='opt'` fan-out displaced something in, once per registration rather than once per namespace, tested by membership rather than by subscripting because the per-namespace registries are plain `defaultdict`\ s and reading one to look would insert `UnknownOption` for a code nobody registered. Presence alone is the firing condition, as on the seven code-keyed parser registrars and unlike the narrower guard #681 gave `register_protocol`, since this key is a caller-supplied `code` that `__init_subclass__` passes once per subclass; a namespace the call itself creates is exempt, because it starts as a copy of `opt`'s defaults and nothing in it is a prior registration. One draft did move `EXPECTED_FAILURES`, and it is worth recording: flooring the three decryption-secrets payloads that read `__length__` *whole* rather than subtracting from it packs nothing, because `Schema.pack` leaves that key at `-1` for "unknown" -- which emptied both payloads and turned `pcapng-secrets/TLS_Key_Log` and `.../WireGuard_Key_Log` from `MISMATCH` to `OK`, an empty payload comparing equal to an empty payload. Reverted and pinned; the entry count holds at 44 with its 35 PCAP-NG cases unmoved. Ships labelled breaking: well-formed captures are byte-identical, verified by regenerating `examples/captures/pcapng.txt` either side of the change, but any truncated PCAP-NG now yields the frames before the cut where it previously raised -- so a caller reading "extraction raised" as "this file is unusable" gets a partial result instead, whose last frame may carry zero-padded octets -- and three exception classes change at the margins, eight short-file depths moving from `ProtocolError: unknown byteorder magic` to `StreamEOFError`. The two modules hold 99.93% coverage with their 25 new statements covered and their single miss flat, and `examples/captures/pcapng.txt` does not move further, leaving the #683 drift #685 tracks exactly as it was (#678). -- **Fixed** -- HIP's `R1_COUNTER`/`R1_Counter` parameter packed 12 octets where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires 16, and `LOCATOR_SET` declared its `Length` in 4-octet units where [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1)'s `Length` is a byte count -- two independent defects, fixed together because the evidence for either needed the other out of the way first. `R1CounterParameter`'s `counter` was a `UInt32Field` where the RFC states the R1 generation counter's width twice, as "8 bytes" in the diagram and as "a 64-bit unsigned integer" in prose; both HIP versions whose *packing* reaches this class -- `R1_Counter` (128) and `R1_COUNTER` (129) -- packed a 12-octet record, landing at `4 (mod 8)` instead of aligned. It is a `UInt64Field` now, and the record is 16 octets (#672). `LocatorSetParameter`'s own `Length` was written as `sum(Locator.len)`, in the 4-octet units [RFC 8046 Section 4](https://datatracker.ietf.org/doc/html/rfc8046#section-4) gives `Locator Length`, where this parameter's own `Length` is a byte count; a set of *n* plain IPv6 locators therefore declared `4n` octets against `24n` actually present, and `Schema.unpack` handed the nested `ListField` only the declared octets, so `n = 2` and `n = 5` both parsed one truncated locator and left the rest of the record unconsumed. `HIP._make_param_locator_set` now sums `8 + Locator.len * 4` per locator -- the fixed header plus the RFC 8046 contents each one actually carries (#679). A second, independent defect in the same parameter's padding is fixed alongside it, one #651 deliberately left alone because it happened to cancel this one exactly: the nested `Locator` schemas share the parameter's packet context and pack their own `len` over it, so `padding` -- evaluated after the list -- always saw 4 rather than the parameter's real length. A new `locator_set_len_callback` snapshots `Length` under a private key before any locator packs into the context, and `locator_set_padding_len` reads that snapshot rather than the shadowed one. An RFC-only byte-stride walk over `examples/captures/options-internet.pcap`, independent of pcapkit's own parser, went from 1 violation -- `LOCATOR_SET`'s empty record, concealed until now because `R1_COUNTER`'s `counter` field defaulted to `0` with no override anywhere in `examples/generators/options.py`, a value the width defect could not be told apart from -- to 2 once the counter was patched non-zero, to 0 once both parameters were fixed; the fixture now overrides both `R1_COUNTER` codes with `counter=0xaabbccdd` so a future regression cannot hide behind a zero again. 10 new test methods across two new files -- 4 in `test_hip_r1_counter_width_unit.py`, 6 in `test_hip_locator_set_length_unit.py` -- fail against the unfixed code and pass against the fix; the three HIP modules stay at 100% coverage, and `tests/` subtests move 586 to 623. `EXPECTED_FAILURES` is unmoved at 44 entries: `hip-parameter/R1_Counter` stays, because code 128 registers no schema of its own -- `R1CounterParameter` declares only `code=129` -- which is a separate registry defect this does not touch, filed as #690. `HIP_COPIES` also stays at two, now for no defect at all rather than for the ones #651 and this fix removed; dropping it is #689, deferred because it would halve every `hip-parameter` frame the fixture above reads (#672, #679). +- **Fixed** -- `register_protocol` displaced one protocol class with another and said nothing about it. The registry behind `pcapkit.protocols.__proto__` is keyed on `cls.__name__.upper()`, and three dispatchable classes are all named `HTTP` -- the generic base `pcapkit.protocols.application.http.HTTP` and the two version implementations in `httpv1` and `httpv2` -- so all three compete for the single key `'HTTP'` and a bare assignment decided the winner. Measured before the fix: `__proto__['HTTP']` went from `pcapkit.protocols.application.http.HTTP` to the `httpv2` class on `register_protocol(httpv2.HTTP)`, with no exception, no log line and no `RegistryWarning`, and `ProtocolBase.expand_comp('HTTP')` then resolved to whichever class had registered last. The overwrite now raises a `RegistryWarning` naming both the displaced and the replacing class, the defining module being the only thing that tells the three apart. The sharpest path to it is not a call to the registrar at all: `Protocol.__init_subclass__` registers unconditionally, so merely *defining* a subclass of the public `Protocol` under a name a built-in already holds displaced that built-in, with nothing anywhere in the user's code to mark the moment the dispatch table changed meaning. A plain `import pcapkit` is unaffected and emits zero new warnings, measured not assumed -- the three built-in `HTTP` classes derive from `ProtocolBase` rather than from the public `Protocol`, so `__init_subclass__` never fires for them, and the import-time seeding in `pcapkit/protocols/__init__.py` keys off the distinct names `HTTP`, `HTTPv1` and `HTTPv2`. The issue's framing that this was the one registrar in `pcapkit/foundation/registry/` with no `RegistryWarning` is right on the count and wrong on the reason: no function in that package warns about anything, the siblings warning only by delegating to a classmethod that carried the guarded `if code in cls.__xxx__: warn(...)` at the time -- #726 later gave all seven the identity guard this file describes elsewhere. The accurate statement is the stronger one -- `register_protocol` is the only keyed registrar in that package, and the only `__name__.upper()`-keyed registry anywhere in `pcapkit/`, that mutates its target with no guard *and* no delegate that could supply one, its target being a module-level dict with no class behind it to hold the guard. The new guard deliberately reads "key present and the incumbent is a different class" where every sibling warned on mere presence at the time: the sibling keys are codes the caller passes, whereas this key is derived from the class and this function is the funnel all nine wrapper registrars end in, so registering one class under two codes -- `register_tcp` then `register_udp`, which is supported and documented -- reaches it twice with nothing displaced, and warning there would put noise on a documented path whose wholesale filter is precisely what would then hide the real collision. The collision is reported, not resolved: re-keying is a registry-format change that none of the bare-name readers can absorb, every one of them degrading *silently* to a plain string or to `Raw` on a miss rather than raising, so it belongs to the registry redesign in #514, which this change is sequenced ahead of rather than part of. 9 to 13 tests and 80 to 87 subtests over `tests/foundation/registry/`, the touched module holding 88% coverage with its misses flat at 27 (#675). +- **Fixed** -- every PCAP-NG packet block now carries the captured octets it declares. All four Enhanced Packet Blocks of the committed `examples/captures/dhcp.pcapng` reported `len(packet) == 0` against a `captured_len` of 314, 342, 314 and 342, and the Simple Packet Block and the obsolete Packet Block did the same, measured on a capture synthesised to carry one of each since no committed fixture has either. Those three -- exactly `PCAPNG.PACKET_TYPES`, and exactly the three schemas declaring `__payload__ = 'packet_data'` -- are the whole of the blast radius; every other block type carries no captured octets, reported `b''` before and still does. The octets were read correctly and then thrown away: `PCAPNG.unpack` extracted them from the block schema, and `ProtocolBase.__init__` overwrote the result with `self.packet.payload`, which was empty. The inherited `packet` splits a protocol at `self.length` octets of header and takes everything after as the payload, a contract that holds for a protocol laid out as a header followed by its payload and for nothing else. `PCAPNG.length` returns the wire's *Block Total Length*, so the split consumed the entire per-block buffer as header -- measured at `frame.length == 348` and `len(frame.packet.header) == 348` on a 348-octet block -- and, separately, a PCAP-NG block carries a *trailer* after its payload, the option list and a repeat of the total length, so no value of `length` could have made that split right. `captured_len` comes straight off the block header and survived, which is why the two disagreed rather than both being empty. The fix is at the computation rather than at the injection: `PCAPNG.packet` is overridden to take the payload from the block schema's `__payload__` field and the header from the octets ahead of it, summed out of the schema buffers so that the three block types' three different payload offsets -- 28, 12 and 28 octets -- are not hard-coded. Teaching `ProtocolBase.__init__` to skip a protocol that had already set `packet` was the smaller change and was rejected: it heals `frame.info.packet` while leaving the public `frame.packet` still reporting the whole block as header and `b''` as payload, which is the same defect from the other side. `PCAPNG.unpack` now reads that property instead of extracting the payload a second time of its own, leaving one source of truth where there were two attempts at one. The property is a plain `property` rather than the `cached_property` it overrides, which a cross-review on a second model is what settled: the inherited one caches because it *reads the stream*, where a second read would consume octets that are gone, while this one only walks buffers the schema layer has already filled and so has nothing to amortise -- and caching it would have reintroduced the same staleness by another route, a second `unpack` on one instance handing back the *first* call's octets with `get_payload` never reached, an invariant the code held before this change and has no reason to stop holding. Nothing calls `unpack` twice on one instance today, `__post_init__` being its only caller, so the unit test asserts the property directly: swap the block on a live instance, unpack again, and the payload must be the new block's, which fails `b'payload' != b'cached'` under a cache. `ProtocolBase.packet` gains the contract in its docstring and not one executable line, its 503 statements unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on the success path, which is why it ships labelled breaking: `frame.info.packet` goes from `b''` to hundreds of octets, `frame.packet.header` from the whole block to the pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what made this a wire-format defect rather than an API wart. `PCAPIO` writes `value.packet` after each 16-octet record header, so dumping those four blocks produced a **104-octet** file -- 24 of global header plus four record headers and no payload at all -- in which every record header declared hundreds of octets and delivered none, desynchronising any reader that walks by `incl_len`, which is all of them: pcapkit refused its own output with `ValueError: read length must be non-negative or -1` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes rather than against a length. `tests/protocols/test_pcapng_regression.py` grows 4 tests to 11 and 3 subtests to 24, with the payload expectations derived twice over -- spelled-out head and tail literals, and a `struct`-only re-parse of the fixture that owes nothing to the code under test -- and each synthesised payload given a different length modulo 4 so that an offset off by a field, or a payload that picked up the block's 32-bit padding, cannot pass by coincidence. Two shapes no fixture exercised are covered there too, both found while writing the tests rather than after: an option area *after* the captured data, which every block of `dhcp.pcapng` lacks and which is exactly what an offset walked from the wrong end would swallow, and a big-endian section, whose fields differ even though the payload offsets do not. A snapped block is covered in both the shapes that express it -- an Enhanced Packet Block declaring `captured_len` below `original_len`, and a Simple Packet Block whose captured length is bounded by the interface's `snaplen` -- neither of which may come back padded out to the on-wire length. The offsets hold at 28, 12 and 28 across all of it, which is the property the walk has to have. With both trees running the same -- current -- tests, which is the only comparison that isolates the code from the suite, `pcapkit/protocols/misc/pcapng.py` holds 99.91% across the change, its 18 new statements and 6 new branches all executed, its miss count flat at 1 and its partial-branch count flat at 1, that single miss being the same pre-existing `_get_timezone` statement renumbered 1248 to 1360; `pcapkit/protocols/protocol.py` is identical in every column, 503 statements and 228 misses either side, which is the check that its change really is docstring-only. The uncaught `ValueError` that an EOF-truncated PCAP-NG raises (#678) is untouched and was measured rather than assumed, over a truncation set stated rather than described because "101 levels" admits several readings that do not agree on the counts: the whole-percent prefixes `max(1, 1508 * i // 100) for i in range(101)`, which for this file are 101 distinct lengths from 1 to 1508. Both trees give 96 of that `ValueError`, one `FormatError`, one `ProtocolError` and three parses, and -- compared level by level rather than only in aggregate, since a matching total can hide two levels that swapped -- the SHA-256 of the sorted length-to-outcome mapping is `ca44d3ee658087cf2a667c454f839dd931c1c04790b2fe32cee8e1a2e861b182` on each. `examples/captures/pcapng.txt`, a hand-regenerated legacy smoke reference, still showed the old frame-level `packet -> NIL` at this point and wanted a separate refresh; #685 (below) later removed it from the index entirely rather than regenerating it again (#646). +- **Fixed** -- an unpadded HTTP/2 `DATA`, `HEADERS` or `PUSH_PROMISE` frame parsed with its whole payload silently discarded. Three payload length callbacks in `pcapkit/protocols/schema/application/httpv2.py` put the conditional expression in the wrong place: a conditional binds looser than `-`, so `pkt['__length__'] - pkt['pad_len'] if pkt['flags']['bit_3'] else 0` groups as `(pkt['__length__'] - pkt['pad_len']) if ... else 0` and the `else` arm returned `0` -- "read no octets at all" to `BytesField` -- where it was meant to subtract `0`. Since `__length__` is the *remaining* declared length at the field, the unpadded arm wants `__length__` itself, which is what subtracting a zero padding length gives. The grouping was read off the AST rather than by eye, the top-level node being the `IfExp` whose `orelse` is a bare `Constant(0)`, which is also what showed that the outer parentheses on the `HeadersFrame` and `PushPromiseFrame` forms were line-continuation only and changed nothing. Padding is rare in HTTP/2, so the broken arm was the common one rather than the edge case, and nothing raised, warned or logged: `info.data` was simply `b''`. Measured on wire octets through the public `HTTP(io.BytesIO(raw), len(raw))` path, an unpadded `DATA` frame declaring `b'{"ok":true}\n'` parsed as `b''` and now parses as those twelve octets; unpadded `HEADERS` and `PUSH_PROMISE` lost and now keep the RFC 7541 appendix C.4.1 header block fragment `b'\x82\x86\x84A\x0fwww.example.com'`, which is precisely what HPACK decoding would have been handed. Padded frames are untouched, for a reason stronger than the measurement: `(A - B) if T else 0` and `A - (B if T else 0)` are both `A - B` for truthy `T`, so only the `else` arm could ever have moved -- both arms are asserted anyway, so that a repair to the unpadded one cannot break the other. The padded arm also has no off-by-one, which #668 asked be checked rather than assumed: `Schema.unpack` decrements `packet['__length__']` by each field's width as it goes, so `pad_len`'s own octet is already out of it by the time the payload field runs, and a padded frame declaring `1 + len(data) + pad_len` arrives at the payload with `__length__ == len(data) + pad_len`. The three sibling sites that were never wrong are what localise the defect to the grouping rather than to the field machinery: `ContinuationFrame.fragment`, `UnassignedFrame.data` and `GoawayFrame.debug` use the plain `length=lambda pkt: pkt['__length__']` with no conditional, share the same `BytesField`, the same `__length__` bookkeeping and the same frame dispatch, and carried their payload correctly throughout -- they are pinned as controls rather than left as an argument. Why nothing caught it: `test_option_roundtrip_unit` does drive every frame type, but through `make` -> parse -> `make`, and `make` writes the length field from `_make_http_length`, so both sides agreed on an empty payload and the octets matched; its generator passes no payload argument for these three frames at all, measured as `kwargs={}`, so what it round-tripped was `b''` and there was nothing to lose, while `test_httpv2_frame_readers_cover_successful_frames` drives the readers from hand-built schema stubs, so the callback never ran there either. The new `tests/protocols/application/test_httpv2_payload_length_unit.py` reads wire octets, which is the gap: 23 tests and 12 subtests over the padded, unpadded and padding-only shapes of all three frames plus both `PRIORITY` combinations, asserting the payload *octets* and never its length, because a length assertion survives a read of the right width from the wrong offset. Its padded cases pad with a non-zero `b'\xde\xad\xbe\xef'` rather than the zeros [RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1) tells a sender to use -- pcapkit does not police padding content -- so that failing to subtract the padding yields a different byte string instead of a coincidentally equal count. The same lambda also governs **packing**, which neither the issue nor the first revision of the fix noticed and a cross-review on a second model did: `BytesField` consults its `length` callback on both paths, so the `else` arm did not merely discard a payload on read, it declined to write one. Measured pre-fix, an unpadded `DATA` frame carrying twelve octets of body packed to `000015000000000001` -- nine octets of header whose length field declares **21**, with the body absent -- and unpadded `HEADERS` and `PUSH_PROMISE` likewise declared 29 against 9 and 33 against 13. A frame overstating its own length by its whole payload desynchronises any reader that walks a stream by that field, so this was pcapkit *emitting* malformed HTTP/2 rather than only mis-reading it, and `HTTPv2ConstructedFrameDeclaresWhatItWritesUnitTests` pins that half; a `make` -> parse -> `make` cycle of a non-empty unpadded payload now closes byte-for-byte where it could not before. Six wrong fixes were written over the schema to check the tests discriminate, and each is rejected: dropping the conditional outright (caught by `packet length < 0: -4`), subtracting `pad_len + 1` (caught as `b'{"ok":true}'` against `b'{"ok":true}\n'`), subtracting the padding on the arm where it is absent, fixing `DataFrame` while forgetting the other two, and `max(computed, 1)` on the two `fragment` fields. That last one is the cross-review's own construction and it **passed** the first revision's 15 tests and 9 subtests, because every fixture then carried a non-empty payload and only `DataFrame` had a "frame of only padding" case, so those two callbacks were never once asked for `0`; under it a padding-only `HEADERS` frame returned `fragment=b'\xde'`, one padding octet leaked in, alongside a swallowed `SchemaWarning: packet length < 0: -1`. The three missing padding-only cases and a callback-level `test_the_padded_arm_reaches_zero_and_is_not_clamped` close it -- `0` is a legitimate answer from these callbacks and has to be asserted as one. Seven tests and three subtests fail before the change and all pass after; 145 tests and 550 subtests pass across `tests/protocols/application/`, the round-trip module and the four other HTTP/2-touching modules. `EXPECTED_FAILURES` is unmoved, its one HTTP/2 entry `httpv2-frame/PRIORITY` still failing with the same `CONSTRUCT` status and the same detail -- `PriorityFrame` has neither a payload field nor a `PADDED` flag, so this change cannot reach it -- and none of the 43 entries was deleted; that entry's own cited `pcapkit/protocols/application/httpv2.py:572` is stale, the `header.length != 9` guard it describes now being at `:562`. The schema module was already at 100% coverage and stays there, 101 statements and 4 branches either side with nothing missing or partial, because the changed lines did execute before -- just with an empty payload -- so the number that moved is the suite: 73 to 96 tests and 399 to 411 subtests over the same targets. The construct side needed no code change, `_make_http_length` having always computed the DATA payload as `len(frame.data) + (pad_len + 1 if pad_len else 0)`, i.e. assuming `data` holds the whole payload whether or not the frame is padded -- it was the shared length callback that disagreed, on both paths at once. An AST sweep of all 496 files of the package found one other site with the same syntactic shape, `pcapkit/protocols/schema/internet/ipv4.py:336` (`TSOption.remainder`), and it is correct rather than the same defect: its `else 0` is intentional because the sibling `ts_data` field consumes the entire option data area in that arm, the two summing to `length - 4` across all 2106 `(length, pointer, flag)` combinations, which is exactly what was not true here. Ships labelled breaking on both counts: restoring a dropped payload moves the parse output of essentially every HTTP/2 capture on the success path, so anything holding a golden file or a regression baseline that recorded the empty value sees it change -- and constructed frames change byte-for-byte too, from malformed to correct (#668). +- **Fixed** -- the schema half of every option-like registrar overwrote silently while the parser half of the very same call warned. `EnumSchema.register` was a bare `cls.__enum__[code] = schema`, and it is the schema half of 14 public registrars, so one `register_ipv4_option` call replacing a built-in named the parser it displaced and said nothing about the schema. `EnumSchema.__init_subclass__` reaches the same registry without calling `register`, so a subclass declared with a `code=` keyword -- the documented way to add a schema -- stayed silent too; both are guarded now, and the declaration hook's two branches are folded into one loop so the guard is written once. Presence is a faithful test there only because `_EnumRegistry.__missing__` returns a miss without recording it, so parsing one packet carrying an unknown code cannot make the next legitimate registration for that code warn about an entry no caller ever asked for. The seven code-keyed parser registrars -- on `ProtocolBase`, `Link`, `Internet`, `Frame`, `PCAPNG`, `Transport` and `SCTP` -- now name the displaced entry and its replacement, where before they reported only that something had been overwritten and never which class it was. Their firing condition is harmonised onto the narrower guard #681 gave `register_protocol` (#718, #726), rather than firing on presence alone: that one is keyed on a name derived from the value it stores, whereas these seven take a `code` the caller supplies independently of the value. Five of these seven -- `Link` (7 entries), `Internet` (16), `Frame` (3), `PCAPNG` (3) and `SCTP` (2) -- ship pre-seeded with unresolved `ModuleDescriptor` values, and so do `TCP` (4) and `UDP` (3), each keeping its own separate dict rather than filling the one `ProtocolBase` and the abstract `Transport` share and leave at 0 (`Transport.register` itself raises `UnsupportedCall`). Where a table is pre-seeded, an incumbent may still be a two-string descriptor while the replacement is the very class it names. Resolution happens earlier, on the incoming argument only, three lines above the guard -- `if isinstance(protocol, ModuleDescriptor): protocol = protocol.klass` -- so the guard's own `is` comparison still measures a resolved class against an unresolved descriptor, warning on re-registering the very same class under its own pre-seeded code -- `Internet.register(TransType.TCP, TCP)`, say -- a false positive the harmonised guard does not close. `ContextRegistry.register` already raises on a duplicate, and the reassembly and ESP registrars append to lists with no key at all -- duplicate security-association SPIs being a designed feature there, resolved by scoring rather than by replacement -- so none of those three takes a guard. `import pcapkit` holds at one warning, a third-party deprecation, and no `RegistryWarning`, measured over the 327 registry writes it performs -- one of them being `R1CounterParameter`'s second code, added by #690 -- none of which lands on a key already present; `tests/foundation/registry/` goes 13 to 14 tests and 87 to 95 subtests, the schema module holds 99% coverage with its 5 new statements covered and its misses flat at 1, mypy is unmoved at 112 errors, and `EXPECTED_FAILURES` is unmoved at 43 entries (#692). +- **Fixed** -- every EOF-truncated PCAP-NG file raised a bare `ValueError` out of `Extractor`, so the whole extraction was lost rather than the one truncated block. The root is one layer above the reported site: Block Total Length is cross-checked against its own trailing copy and never against the file, so a last block declaring more than the file holds left `PCAPNG.read` seeking *past* the real end -- legal and silent -- and the next block read then measured a **negative** remainder, since `prepare` derives it as the end of the stream less the current position. That negative reached `SchemaField` through `pcapng_block_selector` and `io.RawIOBase.read` refused it, which is why nearly every truncation level failed rather than only the one holding the cut. The seek is now clamped to the octets `_read_fileng` actually returned, with a `ProtocolWarning` naming the overrun, and a tail under the twelve octets a block needs at minimum is reported as the quiet `StreamEOFError` that `Extractor.record_frames` already catches -- the one case that is deliberately not a clamp, because at the end of the file no block is being read and clamping would fabricate a frame out of zero padding. A second, independent route to the same shape is closed with a new `nonnegative()` helper, composed into `bounded_option` and `bounded_area` and applied to the eight unwrapped `length - N` spans and the eight `__option_padding__`-sized padding fields: `_TextField.__call__` builds its `struct` template as `f'{length}s'` unconditionally, so a negative became the format `'-8s'` and `struct.calcsize` raised -- neither that nor the `ValueError` being one of `pcapkit.utilities.exceptions`, and neither being an `EOFError`, so neither was caught where the frame loop catches the end of a file. 28 of the module's 74 length callbacks returned a negative before and none do now, from `-1` on an `epb_hash` declaring no payload to `-16777248` on an Enhanced Packet Block's option area; the 200-block, 8,048-octet `captured_len = 0xFFFFFF` vector that reached `struct` through `bounded_area`'s own `nominal <= available` test being true for a negative nominal now parses to 200 frames. Measured over all 1,509 octet boundaries of `examples/captures/dhcp.pcapng` with one harness either side: 6 levels parsed before and 1,495 do now, against 1,479 `ValueError` and 10 `struct.error` before and none from outside the library after, with the frame count degrading monotonically with the cut; of the 14 levels that still raise, twelve leave a file too short to hold a block at all and the other two cut into the Interface Description Block's `if_tsresol` option, so none of them costs a frame that was in the file. That claim is about the truncation sweep rather than about every input: a 4,000-round bounded mutation fuzz either side of the change, same seed and cap, takes the parse rate from 2,118 to 3,438 and removes both of #678's families, and the two foreign families left are measured unchanged -- `ValueError: N is not a valid BlockType`, filed as #701, where an unassigned block type raises from `aenum` inside `EnumField.post_process` so the `UnknownBlock` default the registry declares is unreachable; and `MemoryError` at exactly 30 of 4,000 in both trees, which is #593's 32-bit band. A third bare exception in this module *is* fixed here, found by the cross-review and reachable from valid input rather than truncated: `SystemdJournalExportBlock.post_process` unpacked a 64-bit binary-field length out of whatever `entry_data.read(8)` returned, and since the block body is padded to a 32-bit boundary with NULs that `bytes.strip()` does not strip, that padding was read as a field *name* with no length prefix behind it -- so every journal entry of unaligned length raised a bare `struct.error`, measured on a 14-octet `MESSAGE=hello` entry -- a defect about ordinary input rather than about truncation. A NUL-only line now ends the entry and a short prefix ends it with a warning. The same function leaked two more of the family, found on the cross-review's second pass and fixed with it: a binary field's declared length is the widest in the format and nothing bounded it against the entry holding it, so at `2**63` and above `BytesIO.read` refused it with a bare `OverflowError` while below that it silently returned whatever was there -- the same malformed prefix fatal or invisible by magnitude alone, now clamped to what the entry has left and reported; and a field name, key or value that is not UTF-8 raised a bare `UnicodeDecodeError`, fatal to a whole extraction over one octet in one field, now decoded with `errors='replace'` and reported, which is the option this module's own `StringField` already takes. The separate silent loss of every field after a binary one, from a `read()` that reaches EOF where it means to skip one newline, is filed as #704 rather than fixed. The five deliberately unclamped non-packet option areas keep the per-block framing assumption #676's note describes, which this does not close. Also here because it is the fifth and last registrar in the package with no overwrite guard at all: `Option.register` now reports a displaced option schema as a `RegistryWarning` naming every namespace the `ns='opt'` fan-out displaced something in, once per registration rather than once per namespace, tested by membership rather than by subscripting because the per-namespace registries are plain `defaultdict`\ s and reading one to look would insert `UnknownOption` for a code nobody registered. The firing condition is identity-based here too, the same as the seven code-keyed parser registrars and the narrower guard #681 gave `register_protocol` (#718, #726), even though this key is a caller-supplied `code` that `__init_subclass__` passes once per subclass; a namespace the call itself creates is exempt, because it starts as a copy of `opt`'s defaults and nothing in it is a prior registration. One draft did move `EXPECTED_FAILURES`, and it is worth recording: flooring the three decryption-secrets payloads that read `__length__` *whole* rather than subtracting from it packs nothing, because `Schema.pack` leaves that key at `-1` for "unknown" -- which emptied both payloads and turned `pcapng-secrets/TLS_Key_Log` and `.../WireGuard_Key_Log` from `MISMATCH` to `OK`, an empty payload comparing equal to an empty payload. Reverted and pinned; the entry count holds at 43 with its 35 PCAP-NG cases unmoved. Ships labelled breaking: well-formed captures are byte-identical, verified by regenerating `examples/captures/pcapng.txt` either side of the change, but any truncated PCAP-NG now yields the frames before the cut where it previously raised -- so a caller reading "extraction raised" as "this file is unusable" gets a partial result instead, whose last frame may carry zero-padded octets -- and three exception classes change at the margins, eight short-file depths moving from `ProtocolError: unknown byteorder magic` to `StreamEOFError`. The two modules hold 99.93% coverage with their 25 new statements covered and their single miss flat, and `examples/captures/pcapng.txt` does not move further, leaving the #683 drift #685 tracked exactly as it was (#678). +- **Fixed** -- HIP's `R1_COUNTER`/`R1_Counter` parameter packed 12 octets where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires 16, and `LOCATOR_SET` declared its `Length` in 4-octet units where [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1)'s `Length` is a byte count -- two independent defects, fixed together because the evidence for either needed the other out of the way first. `R1CounterParameter`'s `counter` was a `UInt32Field` where the RFC states the R1 generation counter's width twice, as "8 bytes" in the diagram and as "a 64-bit unsigned integer" in prose; both HIP versions whose *packing* reaches this class -- `R1_Counter` (128) and `R1_COUNTER` (129) -- packed a 12-octet record, landing at `4 (mod 8)` instead of aligned. It is a `UInt64Field` now, and the record is 16 octets (#672). `LocatorSetParameter`'s own `Length` was written as `sum(Locator.len)`, in the 4-octet units [RFC 8046 Section 4](https://datatracker.ietf.org/doc/html/rfc8046#section-4) gives `Locator Length`, where this parameter's own `Length` is a byte count; a set of *n* plain IPv6 locators therefore declared `4n` octets against `24n` actually present, and `Schema.unpack` handed the nested `ListField` only the declared octets, so `n = 2` and `n = 5` both parsed one truncated locator and left the rest of the record unconsumed. `HIP._make_param_locator_set` now sums `8 + Locator.len * 4` per locator -- the fixed header plus the RFC 8046 contents each one actually carries (#679). A second, independent defect in the same parameter's padding is fixed alongside it, one #651 deliberately left alone because it happened to cancel this one exactly: the nested `Locator` schemas share the parameter's packet context and pack their own `len` over it, so `padding` -- evaluated after the list -- always saw 4 rather than the parameter's real length. A new `locator_set_len_callback` snapshots `Length` under a private key before any locator packs into the context, and `locator_set_padding_len` reads that snapshot rather than the shadowed one. An RFC-only byte-stride walk over `examples/captures/options-internet.pcap`, independent of pcapkit's own parser, went from 1 violation -- `LOCATOR_SET`'s empty record, concealed until now because `R1_COUNTER`'s `counter` field defaulted to `0` with no override anywhere in `examples/generators/options.py`, a value the width defect could not be told apart from -- to 2 once the counter was patched non-zero, to 0 once both parameters were fixed; the fixture now overrides both `R1_COUNTER` codes with `counter=0xaabbccdd` so a future regression cannot hide behind a zero again. 10 new test methods across two new files -- 4 in `test_hip_r1_counter_width_unit.py`, 6 in `test_hip_locator_set_length_unit.py` -- fail against the unfixed code and pass against the fix; the three HIP modules stay at 100% coverage, and `tests/` subtests move 586 to 623. `EXPECTED_FAILURES` is 43 entries: the `hip-parameter/R1_Counter` entry -- code 128 registered no schema of its own, since `R1CounterParameter` declared only `code=129` -- was deleted once that separate registry defect was fixed as #690. `HIP_COPIES` was dropped to one as #689, once #672 and #679 left it routing around nothing. - **Fixed** -- three test comments stated an exact count of committed captures under `examples/captures/`, a number that drifts every time that directory's tracked set changes and in one case was already wrong when written. `tests/_tiers.py` cited "six," correct at the time; `tests/test_tier_guard.py` said "the moment a seventh capture is committed" -- the same count, as an ordinal rather than the word "six"; `tests/integration/_helpers.py` said "four," which undercounted the tracked set the day it was written. All three now describe the invariant instead of a number that has to be kept in sync with it by hand: `_tiers.py` reads "the moment somebody commits another capture... or stops committing one," `_helpers.py` reads "...are fixtures -- some of them committed --...," and `test_tier_guard.py` drops its "seventh capture" phrasing the same way. Prose-only, verified with `coverage.parser.PythonParser` that no executable statement moved (#700). - **Fixed** -- `test_every_tracked_name_exists_and_matches_git` in `tests/test_tier_guard.py` checked neither the git index nor the filesystem despite its name: it asserted only that the tracked-capture list was non-empty and that no name contained a `/`, so a hardcoded list would pass it whether or not it matched reality. Sibling `test_capture_suggestions_are_captures` had the same gap. Found while cross-reviewing #703's first commit, itself prose-only and out of scope for this; the fix rides in as that PR's second and third commits, and #703's own description names both #700 and #708. `test_every_tracked_name_exists_and_matches_git` now re-derives the expected set independently -- shelling out to `git ls-files -z` under `examples/captures/` rather than calling back into `_tiers.py` -- and asserts equality against it rather than against itself, plus a per-name `Path.is_file()` check the old version never made. Its sibling `test_capture_suggestions_are_captures` gets a narrower fix: it asserts equality against `committed_capture_names()`'s own filter re-implemented inline over `_tiers.committed_captures()`, which catches a wrong filter but, unlike its sibling, still trusts `committed_captures()` for the tracked set itself rather than re-deriving it from git. A cross-review of the first version of this fix found it wrong twice over: a stale docstring in `_tiers.py` still claimed `test_tier_guard.py` "only stats `in.pcap`," falsified by the new per-name loop; and the new `assertEqual` on `test_capture_suggestions_are_captures` was vacuous on an empty tracked set (`() == ()` passes trivially), having dropped the non-emptiness guard the other test kept. Both are corrected in a follow-up commit. Even after the fix, a hardcoded literal that happens to match today's tracked names still passes -- the test detects a set that is wrong *at the moment it runs*, not hardcoding as a practice, which turns a permanent blind spot into a tripwire that fires on the next change to the tracked set (#708). -- **Changed** -- reconciled which private (`_xxx`) attributes and methods the Sphinx build documents, replacing an ad hoc mix with one stated rule: document the contract, hide the recipe. A member every subclass must implement, or one whose shape a caller genuinely depends on, stays documented even though its name starts with an underscore; a private helper that exists only to keep one method short does not. Seven module-level directives naming pure implementation detail were dropped -- `esp._resolve`, `esp._CRYPTO`, `ngap._convert`, `ngap._revert`, `ngap._PYCRATE`, `ngap._PDU_LOCK` and `pypcapfile._NamedStream` (whose nested `name`/`read` members go with it, being reachable only through a private class) -- while 39 `autoattribute` directives were added for class-private state that *is* contract: `Extractor._flag_f`, the `PyPCAP` and `PCAP_CT` engines' own `_backend` -- the only two of the six third-party engines that have one -- `TraceFlow`'s internal fields, and `FieldBase`/`Field` internals among them; the built-in `PCAP` and `PCAPNG` engines gained `_gbhdr`, `_vinfo` and `_nnsec` for the former and `_ctx`/`_ctx_list` for the latter, and no `_backend` at all. `_dlink` is not built-in-only: `PCAP` documents it alongside the three third-party engines that share it, `PyPCAP`, `PCAP_CT` and `PyPCAPFile`. The 128 runtime definitions of `_missing_` -- 121 under `pcapkit.const`, the other 7 inline in `pcapkit.protocols` -- gained one unified write-up in place of a directive per class: a new "Unrecognised Values" section in `docs/source/pcapkit/const/index.rst`, cross-referenced from `registry.rst`, since `conf.py` already excludes `_missing_` from every `autoclass` via `exclude-members` in `autodoc_default_options` -- there is no `automodule` directive anywhere under `docs/source/`. `CONTRIBUTING.md` gained the rule itself as a named section, so the next directive gets judged against a written test rather than against precedent. Verified with `sphinx-build -b html` under `PCAPKIT_SPHINX=1`: 55 warnings on `main` before this change, 56 after, the one addition being a pre-existing bare `Type` cross-reference ambiguity newly rendered by `TraceFlow._foutio`'s new directive rather than a defect this change introduced -- #709 tracked it. No line under `pcapkit/` changed (#684). +- **Changed** -- reconciled which private (`_xxx`) attributes and methods the Sphinx build documents, replacing an ad hoc mix with one stated rule: document the contract, hide the recipe. A member every subclass must implement, or one whose shape a caller genuinely depends on, stays documented even though its name starts with an underscore; a private helper that exists only to keep one method short does not. Seven module-level directives naming pure implementation detail were dropped -- `esp._resolve`, `esp._CRYPTO`, `ngap._convert`, `ngap._revert`, `ngap._PYCRATE`, `ngap._PDU_LOCK` and `pypcapfile._NamedStream` (whose nested `name`/`read` members go with it, being reachable only through a private class) -- while 39 `autoattribute` directives were added for class-private state that *is* contract: `Extractor._flag_f`, the `PyPCAP` and `PCAP_CT` engines' own `_backend` -- the only two of the six third-party engines that have one -- `TraceFlow`'s internal fields, and `FieldBase`/`Field` internals among them; the built-in `PCAP` and `PCAPNG` engines gained `_gbhdr`, `_vinfo` and `_nnsec` for the former and `_ctx`/`_ctx_list` for the latter, and no `_backend` at all. `_dlink` is not built-in-only: `PCAP` documents it alongside the three third-party engines that share it, `PyPCAP`, `PCAP_CT` and `PyPCAPFile`. The 128 runtime definitions of `_missing_` -- 121 under `pcapkit.const`, the other 7 inline in `pcapkit.protocols` -- gained one unified write-up in place of a directive per class: a new "Unrecognised Values" section in `docs/source/pcapkit/const/index.rst`, cross-referenced from `registry.rst`, since `conf.py` already excludes `_missing_` from every `autoclass` via `exclude-members` in `autodoc_default_options` -- there is no `automodule` directive anywhere under `docs/source/`. `CONTRIBUTING.md` gained the rule itself as a named section, so the next directive gets judged against a written test rather than against precedent. Verified with `sphinx-build -b html` under `PCAPKIT_SPHINX=1`: 53 warnings on `main` before this change, 54 after, the one addition being a pre-existing bare `Type` cross-reference ambiguity newly rendered by `TraceFlow._foutio`'s new directive rather than a defect this change introduced -- #709 tracked it. No line under `pcapkit/` changed (#684). - **Fixed** -- `tests/dumpkit/test_nameless_enum_rendering_unit.py` (added by #670) carried two probes on the same wrong assumption that every flag-enum registry is 16 bits wide: `test_scalar_return_renders_a_nameless_member_as_its_value` checked one fixed tuple ending in `65536` against `tcp.flags.Flags` and its own `StdFlags` stand-in alone, and `test_no_flag_registry_renders_the_literal_none` swept every flag-enum registry instead, but only against `registry(0)`. Seven registries are not *all* 16 bits, at four distinct widths: `ftp.command.CommandType` (3 bits), `reg.apptype.TransportProtocol` (4, computed at runtime), `mh.binding_ack_flag.BindingACKFlag`, `mh.handover_ack_flag.HandoverACKFlag` and `mh.handover_initiate_flag.HandoverInitiateFlag` (8 each), and `mh.binding_update_flag.BindingUpdateFlag` with `tcp.flags.Flags` (16, the two the old probe actually fit). `tcp.flags.Flags(65536)` correctly raises -- its own `_missing_` bounds itself to `0 <= value <= 0xFFFF` -- so the sweep was failing on correct behaviour rather than reporting a defect. A new `_field_mask` derives each registry's own all-ones bound from its declared members, and `_nameless_values` returns the values that bound admits but no member names -- zero, each undeclared bit alone, and every undeclared bit combined -- in place of the one hard-coded tuple. A new `test_a_value_past_the_field_is_refused_rather_than_rendered` pins the boundary directly: the widest in-field value is accepted, one past it is refused. The file goes from 1 failed / 5 passed / 16 subtests to 6 passed / 64 subtests. A cross-review found that all seven guard `raise` lines this reaches -- `tcp/flags.py`'s own included -- were already covered by a passing test predating this fix; none was genuinely newly reached by it. No line under `pcapkit/` changed (#702). - **Changed** -- `examples/captures/out.json`, `out.plist`, `out.txt` and `pcapng.txt` are no longer tracked in git; they are build output, not fixtures, and a tracked rendering with no reader goes stale silently every time the code that produces it changes. `pcapng.txt` is exactly that: it still recorded `packet -> NIL` for the four Enhanced Packet Blocks of `dhcp.pcapng` long after #683 gave those blocks their captured octets back, and nothing had regenerated it. Rather than regenerate it once more and leave the same drift free to recur, the four files are removed from the index and folded into `examples/captures/`'s existing blanket `.gitignore` rule; `in.pcap` and `dhcp.pcapng`, the genuine inputs, stay tracked. `examples/legacy_smoke/Makefile` and its `README.rst` are reworded to describe regenerating these reports via `make fixtures` rather than implying they ship committed, and a new `tests/project/test_capture_tracking.py` (5 tests, 8 subtests) pins the invariant going forward; 2 of the 5 tests (4 of the 12 subtests) fail against the pre-change tree with the four reports restored to the index, the other 3 passing on both trees by construction (#685). -- **Fixed** -- two `#:` autodoc comments in `pcapkit/foundation/traceflow/traceflow.py` named a bare `Type`, which Sphinx's cross-reference resolver resolves against every class named `Type` in the project rather than against `typing.Type` -- there are five -- and silently linked to `pcapkit.const.l2tp.type.Type`, an L2TP field-type enum with nothing to do with dumpers. Line 406 (`#: Type[Dumper]: Dumper class.`) is the live case: once #684 rendered `TraceFlow._foutio`, the built docs pointed a reader at the wrong class with no warning that anything had gone sideways. Both sites now spell it `~typing.Type[Dumper]`, the same form eight other files in the tree already use for the identical ambiguity. Line 146 (the first line of `__output__`'s `#:` block, which continues through line 149) is fixed for the same reason but is currently inert: the `# type:` comment sixteen lines below, at line 162, spells the identical bare `Type[Dumper]` -- so whatever eventually renders this attribute's type still has the same ambiguity to resolve. This half of the fix is insurance against the day something reads it cleanly. This pair of sites is part of #709. Four more bare `Type` sites -- hand-written `:type:` fields at `docs/source/pcapkit/foundation/engines/engine.rst:40`, `.../reassembly/reassembly.rst:33` and `:43`, and `.../traceflow/traceflow.rst:40` -- carried the same ambiguity and are fixed separately, by #714 (#709). +- **Fixed** -- two `#:` autodoc comments in `pcapkit/foundation/traceflow/traceflow.py` named a bare `Type`, which Sphinx's cross-reference resolver resolves against every class named `Type` in the project rather than against `typing.Type` -- there are five -- and silently linked to `pcapkit.const.l2tp.type.Type`, an L2TP field-type enum with nothing to do with dumpers. Line 424 (`#: ~typing.Type[Dumper]: Dumper class.`, spelled out since #709 fixed it) is the live case: once #684 rendered `TraceFlow._foutio`, the built docs pointed a reader at the wrong class with no warning that anything had gone sideways. Both sites now spell it `~typing.Type[Dumper]`, the same form eight other files in the tree already use for the identical ambiguity. Line 146 (the first line of `__output__`'s `#:` block, which continues through line 149) is fixed for the same reason but is currently inert: the `# type:` comment sixteen lines below, at line 162, spells the identical bare `Type[Dumper]` -- so whatever eventually renders this attribute's type still has the same ambiguity to resolve. This half of the fix is insurance against the day something reads it cleanly. This pair of sites is part of #709. Four more bare `Type` sites -- hand-written `:type:` fields at `docs/source/pcapkit/foundation/engines/engine.rst:40`, `.../reassembly/reassembly.rst:33` and `:43`, and `.../traceflow/traceflow.rst:40` -- carried the same ambiguity and are fixed separately, by #714 (#709). - **Fixed** -- `register_protocol`'s overwrite warning could claim a protocol was replaced with itself. The guard that decides *whether* to warn was already correct -- `incumbent is not protocol`, an identity check from #681 -- but the message built both operands with a bare `repr()`, and for an ordinary class that is just ``. A factory that defines a same-named, closure-local class on every call (as `tests/protocols/test_construction_keyword_check_unit.py`'s `_protocol_class` does) produces two genuinely distinct classes sharing one `__module__` and `__qualname__`, so both reprs print identically and a real, correct overwrite reads as "overwriting X with X." The message now compares the two reprs first, and only when they coincide appends each object's `id()` to tell them apart; the common case, where the two classes are named differently, is untouched. `__module__`/`__qualname__` was considered as the disambiguator instead of `id()` and rejected: for the reported shape those are exactly what the coinciding repr already renders, so they discriminate nothing an `id()` does not already have to. A new `test_register_protocol_disambiguates_classes_sharing_a_repr` fails against the unfixed message and passes against the fix; the targeted suite goes 49 to 50 passed, and the file's coverage holds at 97% (#710). Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels (#497, #498). diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index e8a612f8d..f9dd4789a 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -4,8 +4,9 @@ The largest release since 1.0, and the first recorded here as it happened rather than reconstructed. Three more extraction engines, ESP with payload decryption, SCTP and NGAP over SCTP, a library logger that no longer hijacks the consumer's, -and a defect programme run through the issue tracker across some 140 issues and -pull requests between #326 and #509. +and a defect programme run through the issue tracker that started across some 140 +issues and pull requests between #326 and #509, and has continued well past #509 +since -- reaching #726 by the entries below. * **Added** -- three extraction engines: ``engine='pypcap'`` and ``engine='pcap_ct'``, two independent distributions of the same ``libpcap`` @@ -831,7 +832,7 @@ pull requests between #326 and #509. order, since the padding is only ever consulted when the buffer falls short (#604). * **Fixed** -- as a consequence of the above, the unhandled ``MemoryError`` at - ``pcapkit/protocols/protocol.py:1016`` on a truncated PCAP-NG capture. + ``pcapkit/protocols/protocol.py:1411`` on a truncated PCAP-NG capture. ``dhcp_little_endian.pcapng`` cut to 161, 641 or 1389 octets left a one-octet read of a little-endian 32-bit block length, which ``rjust()`` turned into ``0x78000000`` or ``0x84000000`` -- 1.88 to 2.06 GiB -- and which was then @@ -942,7 +943,7 @@ pull requests between #326 and #509. wrapper by name, whereas a ``SwitchField`` always resolves to a concrete field. Replacing it would be a behaviour change and is not made (#603). * **Changed** -- the README is a landing page now, and Markdown rather than - reStructuredText. ``README.rst`` (424 lines) became ``README.md`` (103), + reStructuredText. ``README.rst`` (424 lines) became ``README.md`` (102), keeping what a reader arriving from PyPI or a search result actually needs -- what the library is, why it exists rather than Scapy or DPKT, how to install it, a worked example, and where the documentation lives -- and dropping the @@ -1133,8 +1134,11 @@ pull requests between #326 and #509. ``super()._missing_(value)`` tail for the same reason. All four now end in ``return super()._missing_(value)``, which is what ``pcapkit/vendor/default.py`` emits for every other generated enumeration and what - 75 of the 117 modules under ``pcapkit/const/`` already do, so ``F(0)`` is an empty - flag and ``BindingACKFlag(0x06)`` is ``S|D``. The ``extend_enum`` idiom that + 75 of the 117 modules under ``pcapkit/const/`` already did -- 77 now, since #647 + below gives the same ending to three more classes but only two land in modules + the count did not already have, ``TransportProtocol`` sharing ``reg/apptype.py`` + with the already-counted ``AppType`` -- so ``F(0)`` is an empty flag and + ``BindingACKFlag(0x06)`` is ``S|D``. The ``extend_enum`` idiom that ``pcapkit/const/pcapng/record_type.py`` and ``secrets_type.py`` use to mint a member for an unassigned integer -- the only other two modules whose ``_missing_`` ends in ``return cls(value)``, and which do not recurse precisely because that @@ -1537,20 +1541,21 @@ pull requests between #326 and #509. wrong offset, in both directions; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test in the suite could see this and why the fix is asserted against the RFC's arithmetic - rather than against a round trip. 93 of the 95 sites -- 45 of the 46 - ``PaddingField`` callbacks in ``pcapkit/protocols/schema/internet/hip.py`` and 48 + rather than against a round trip. 94 of the 95 sites -- 45 of the 46 + ``PaddingField`` callbacks in ``pcapkit/protocols/schema/internet/hip.py`` and 49 of the 49 record lengths in ``pcapkit/protocols/internet/hip.py`` -- are now one ``parameter_total_len`` and one ``parameter_padding_len``, stating the RFC formula - once instead of 95 times (#651). ``LOCATOR_SET`` keeps the old expression at both - of its sites, on purpose, because two defects there cancel each other exactly and - correcting only the padding would break a parameter that is currently right: its - padding callback never receives the parameter's ``len`` (the nested ``Locator`` - schemas share a packet context whose own ``len`` shadows it, so the value seen is - always 4 for an IPv6 locator), and the parameter's ``len`` is written in 4-octet - units where the RFC's ``Length`` is a byte count. Always-4 padding gives - ``4 + 24n + 4``, and because ``24n`` is a multiple of 8 the RFC total for a - byte-count ``Length`` of ``24n`` is the same ``24n + 8`` -- measured at n = 1, 2 - and 5 as 32, 56 and 128 octets both before and after. #679 carries the pair. ``EncryptedParameter``'s ``data`` length callback is fixed in the same + once instead of 95 times (#651). ``LOCATOR_SET`` kept the old expression at both + of its sites for now, on purpose, because two defects there cancelled each other + exactly and correcting only the padding would have broken a parameter that was, + at that point, right: its padding callback never received the parameter's + ``len`` (the nested ``Locator`` schemas shared a packet context whose own + ``len`` shadowed it, so the value seen was always 4 for an IPv6 locator), and + the parameter's ``len`` was written in 4-octet units where the RFC's ``Length`` + is a byte count. Always-4 padding gave ``4 + 24n + 4``, and because ``24n`` is a + multiple of 8 the RFC total for a byte-count ``Length`` of ``24n`` was the same + ``24n + 8`` -- measured at n = 1, 2 and 5 as 32, 56 and 128 octets both before + and after. #679 later fixed both sites. ``EncryptedParameter``'s ``data`` length callback is fixed in the same change and could not have been left: it subtracted the sixteen ``iv`` octets but not the four ``reserved`` ones, and those four cancelled the padding's four at ``Length % 8`` in ``{0, 5, 6, 7}`` -- so correcting the padding alone would have @@ -1563,10 +1568,14 @@ pull requests between #326 and #509. parameter's ``length`` is now 8 where it was 12, and the other 44 corrected parameters move likewise, so code comparing stored pcapkit output byte for byte, or asserting on ``Data_*Parameter.length``, sees different values -- the RFC's - values. ``LOCATOR_SET`` is unchanged in both respects. ``examples/generators/options.py``'s ``HIP_COPIES`` stays at two, no - longer for this reason but for ``R1_COUNTER``'s four-octet ``counter`` where - :rfc:`7401#section-5.2.3` requires eight, which this defect had been masking - (#651). + values. ``LOCATOR_SET`` stayed unchanged in both respects for the time being; + #679 (below) later fixed both. ``examples/generators/options.py``'s + ``HIP_COPIES`` stayed at two for now as well, no longer for this reason but for + ``R1_COUNTER``'s four-octet ``counter`` where :rfc:`7401#section-5.2.3` + requires eight, which this defect had been masking -- until #672 widened + ``counter`` and #679 fixed ``LOCATOR_SET``'s ``Length`` unit, after which + #689 dropped ``HIP_COPIES`` to one, once both had left it routing around + nothing (#651). * **Fixed** -- two HTTP/2 flag defects, one on each side of a round trip. **This changes the octets a reconstructed DATA frame emits, and the dumped value of a flagless frame's flags.** ``_make_http_data`` was the only one of @@ -1869,21 +1878,24 @@ pull requests between #326 and #509. ``HTTP``, ``HTTPv1`` and ``HTTPv2``. The issue's framing that this was the one registrar in ``pcapkit/foundation/registry/`` with no ``RegistryWarning`` is right on the count and wrong on the reason: no function in that package warns about anything, - the siblings warning only by delegating to a classmethod that carries the guarded - ``if code in cls.__xxx__: warn(...)``. The accurate statement is the stronger one -- + the siblings warning only by delegating to a classmethod that carried the guarded + ``if code in cls.__xxx__: warn(...)`` at the time -- #726 later gave all seven the + identity guard this file describes elsewhere. The accurate statement is the + stronger one -- ``register_protocol`` is the only keyed registrar in that package, and the only ``__name__.upper()``-keyed registry anywhere in ``pcapkit/``, that mutates its target with no guard *and* no delegate that could supply one, its target being a module-level dict with no class behind it to hold the guard. The new guard deliberately reads "key present and the incumbent is a different class" where every - sibling warns on mere presence: the sibling keys are codes the caller passes, whereas - this key is derived from the class and this function is the funnel all nine wrapper - registrars end in, so registering one class under two codes -- ``register_tcp`` then - ``register_udp``, which is supported and documented -- reaches it twice with nothing - displaced, and warning there would put noise on a documented path whose wholesale - filter is precisely what would then hide the real collision. The collision is - reported, not resolved: re-keying is a registry-format change that none of the - bare-name readers can absorb, every one of them degrading *silently* to a plain + sibling warned on mere presence at the time: the sibling keys are codes the caller + passes, whereas this key is derived from the class and this function is the funnel + all nine wrapper registrars end in, so registering one class under two codes -- + ``register_tcp`` then ``register_udp``, which is supported and documented -- reaches + it twice with nothing displaced, and warning there would put noise on a documented + path whose wholesale filter is precisely what would then hide the real collision. + The collision is reported, not resolved: re-keying is a registry-format change that + none of the bare-name readers can absorb, every one of them degrading *silently* to + a plain string or to ``Raw`` on a miss rather than raising, so it belongs to the registry redesign in #514, which this change is sequenced ahead of rather than part of. 9 to 13 tests and 80 to 87 subtests over ``tests/foundation/registry/``, the touched @@ -1961,7 +1973,7 @@ pull requests between #326 and #509. ``pcapkit/protocols/misc/pcapng.py`` holds 99.91% across the change, its 18 new statements and 6 new branches all executed, its miss count flat at 1 and its partial-branch count flat at 1, that single miss being the same pre-existing - ``_get_timezone`` statement renumbered 1153 to 1265; ``pcapkit/protocols/protocol.py`` + ``_get_timezone`` statement renumbered 1248 to 1360; ``pcapkit/protocols/protocol.py`` is identical in every column, 503 statements and 228 misses either side, which is the check that its change really is docstring-only. The uncaught ``ValueError`` that an EOF-truncated PCAP-NG raises (#678) is untouched and was measured rather than assumed, @@ -1973,8 +1985,10 @@ pull requests between #326 and #509. in aggregate, since a matching total can hide two levels that swapped -- the SHA-256 of the sorted length-to-outcome mapping is ``ca44d3ee658087cf2a667c454f839dd931c1c04790b2fe32cee8e1a2e861b182`` on each. - ``examples/captures/pcapng.txt``, a hand-regenerated legacy smoke reference, still shows - the old frame-level ``packet -> NIL`` and wants a separate refresh (#646). + ``examples/captures/pcapng.txt``, a hand-regenerated legacy smoke reference, still showed + the old frame-level ``packet -> NIL`` at this point and wanted a separate refresh; + #685 (below) later removed it from the index entirely rather than regenerating + it again (#646). * **Fixed** -- an unpadded HTTP/2 ``DATA``, ``HEADERS`` or ``PUSH_PROMISE`` frame parsed with its whole payload silently discarded. Three payload length callbacks in ``pcapkit/protocols/schema/application/httpv2.py`` put the conditional expression in the @@ -2052,7 +2066,7 @@ pull requests between #326 and #509. the four other HTTP/2-touching modules. ``EXPECTED_FAILURES`` is unmoved, its one HTTP/2 entry ``httpv2-frame/PRIORITY`` still failing with the same ``CONSTRUCT`` status and the same detail -- ``PriorityFrame`` has neither a payload field nor a ``PADDED`` flag, - so this change cannot reach it -- and none of the 44 entries was deleted; that entry's + so this change cannot reach it -- and none of the 43 entries was deleted; that entry's own cited ``pcapkit/protocols/application/httpv2.py:572`` is stale, the ``header.length != 9`` guard it describes now being at ``:562``. The schema module was already at 100% coverage and stays there, 101 statements and 4 branches either side @@ -2090,26 +2104,36 @@ pull requests between #326 and #509. ``Internet``, ``Frame``, ``PCAPNG``, ``Transport`` and ``SCTP`` -- now name the displaced entry and its replacement, where before they reported only that something had been overwritten and never which class it was. Their - presence-only firing condition is deliberately unchanged, and is not - harmonised onto the narrower guard #681 gave ``register_protocol``: that one - is keyed on a name derived from the value it stores, whereas these seven take - a ``code`` the caller supplies independently of the value, so one class - registered under two codes yields two keys and never reaches one key twice. - These tables also ship pre-seeded with unresolved ``ModuleDescriptor`` - values, so an incumbent may be a two-string descriptor while the replacement - is the very class it names, which leaves a different-class test undecidable - without resolving the descriptor and forcing the import it exists to defer. + firing condition is harmonised onto the narrower guard #681 gave + ``register_protocol`` (#718, #726), rather than firing on presence alone: + that one is keyed on a name derived from the value it stores, whereas these + seven take a ``code`` the caller supplies independently of the value. Five of + these seven -- ``Link`` (7 entries), ``Internet`` (16), ``Frame`` (3), ``PCAPNG`` + (3) and ``SCTP`` (2) -- ship pre-seeded with unresolved ``ModuleDescriptor`` + values, and so do ``TCP`` (4) and ``UDP`` (3), each keeping its own separate + dict rather than filling the one ``ProtocolBase`` and the abstract ``Transport`` + share and leave at 0 (``Transport.register`` itself raises + ``UnsupportedCall``). Where a table is pre-seeded, an incumbent may still be a + two-string descriptor while the replacement is the very class it names. + Resolution happens earlier, on the incoming argument only, three lines + above the guard -- + ``if isinstance(protocol, ModuleDescriptor): protocol = protocol.klass`` -- so + the guard's own ``is`` comparison still measures a resolved class against an + unresolved descriptor, warning on re-registering the very same class under its + own pre-seeded code -- + ``Internet.register(TransType.TCP, TCP)``, say -- a false positive the + harmonised guard does not close. ``ContextRegistry.register`` already raises on a duplicate, and the reassembly and ESP registrars append to lists with no key at all -- duplicate security-association SPIs being a designed feature there, resolved by scoring rather than by replacement -- so none of those three takes a guard. ``import pcapkit`` holds at one warning, a third-party deprecation, and no - ``RegistryWarning``, measured over the 326 registry writes it performs, none + ``RegistryWarning``, measured over the 327 registry writes it performs -- one + of them being ``R1CounterParameter``'s second code, added by #690 -- none of which lands on a key already present; ``tests/foundation/registry/`` goes 13 to 14 tests and 87 to 95 subtests, the schema module holds 99% coverage - with its 5 new statements covered and its misses flat at 1, mypy and pylint - are unmoved at 112 errors and 364 messages, and ``EXPECTED_FAILURES`` is - unmoved at 44 entries (#692). + with its 5 new statements covered and its misses flat at 1, mypy is unmoved + at 112 errors, and ``EXPECTED_FAILURES`` is unmoved at 43 entries (#692). * **Fixed** -- every EOF-truncated PCAP-NG file raised a bare ``ValueError`` out of ``Extractor``, so the whole extraction was lost rather than the one truncated block. The root is one layer above the reported site: Block Total Length is cross-checked @@ -2177,17 +2201,18 @@ pull requests between #326 and #509. in, once per registration rather than once per namespace, tested by membership rather than by subscripting because the per-namespace registries are plain ``defaultdict``\ s and reading one to look would insert ``UnknownOption`` for a code nobody registered. - Presence alone is the firing condition, as on the seven code-keyed parser registrars and - unlike the narrower guard #681 gave ``register_protocol``, since this key is a - caller-supplied ``code`` that ``__init_subclass__`` passes once per subclass; a - namespace the call itself creates is exempt, because it starts as a copy of ``opt``'s - defaults and nothing in it is a prior registration. One draft did move - ``EXPECTED_FAILURES``, and it is worth recording: flooring the three decryption-secrets - payloads that read ``__length__`` *whole* rather than subtracting from it packs nothing, + The firing condition is identity-based here too, the same as the seven code-keyed + parser registrars and the narrower guard #681 gave ``register_protocol`` (#718, + #726), even though this key is a caller-supplied ``code`` that ``__init_subclass__`` + passes once per subclass; a namespace the call itself creates is exempt, because it + starts as a copy of ``opt``'s defaults and nothing in it is a prior registration. + One draft did move ``EXPECTED_FAILURES``, and it is worth recording: flooring the + three decryption-secrets payloads that read ``__length__`` *whole* rather than + subtracting from it packs nothing, because ``Schema.pack`` leaves that key at ``-1`` for "unknown" -- which emptied both payloads and turned ``pcapng-secrets/TLS_Key_Log`` and ``.../WireGuard_Key_Log`` from ``MISMATCH`` to ``OK``, an empty payload comparing equal to an empty payload. Reverted - and pinned; the entry count holds at 44 with its 35 PCAP-NG cases unmoved. Ships + and pinned; the entry count holds at 43 with its 35 PCAP-NG cases unmoved. Ships labelled breaking: well-formed captures are byte-identical, verified by regenerating ``examples/captures/pcapng.txt`` either side of the change, but any truncated PCAP-NG now yields the frames before the cut where it previously raised -- so a caller reading @@ -2196,7 +2221,7 @@ pull requests between #326 and #509. eight short-file depths moving from ``ProtocolError: unknown byteorder magic`` to ``StreamEOFError``. The two modules hold 99.93% coverage with their 25 new statements covered and their single miss flat, and ``examples/captures/pcapng.txt`` does not move - further, leaving the #683 drift #685 tracks exactly as it was (#678). + further, leaving the #683 drift #685 tracked exactly as it was (#678). * **Fixed** -- HIP's ``R1_COUNTER``/``R1_Counter`` parameter packed 12 octets where :rfc:`7401#section-5.2.3` requires 16, and ``LOCATOR_SET`` declared its ``Length`` in 4-octet units where :rfc:`7401#section-5.2.1`'s ``Length`` is a @@ -2237,14 +2262,12 @@ pull requests between #326 and #509. ``test_hip_r1_counter_width_unit.py``, 6 in ``test_hip_locator_set_length_unit.py`` -- fail against the unfixed code and pass against the fix; the three HIP modules stay at 100% coverage, and - ``tests/`` subtests move 586 to 623. ``EXPECTED_FAILURES`` is unmoved at 44 - entries: ``hip-parameter/R1_Counter`` stays, because code 128 registers no - schema of its own -- ``R1CounterParameter`` declares only ``code=129`` -- - which is a separate registry defect this does not touch, filed as #690. - ``HIP_COPIES`` also stays at two, now for no defect at all rather than for - the ones #651 and this fix removed; dropping it is #689, deferred because it - would halve every ``hip-parameter`` frame the fixture above reads (#672, - #679). + ``tests/`` subtests move 586 to 623. ``EXPECTED_FAILURES`` is 43 entries: + the ``hip-parameter/R1_Counter`` entry -- code 128 registered no schema of + its own, since ``R1CounterParameter`` declared only ``code=129`` -- was + deleted once that separate registry defect was fixed as #690. ``HIP_COPIES`` + was dropped to one as #689, once #672 and #679 left it routing around + nothing. * **Fixed** -- three test comments stated an exact count of committed captures under ``examples/captures/``, a number that drifts every time that directory's tracked set changes and in one case was already wrong when @@ -2319,8 +2342,8 @@ pull requests between #326 and #509. ``CONTRIBUTING.md`` gained the rule itself as a named section, so the next directive gets judged against a written test rather than against precedent. Verified with - ``sphinx-build -b html`` under ``PCAPKIT_SPHINX=1``: 55 warnings on ``main`` - before this change, 56 after, the one addition being a pre-existing bare + ``sphinx-build -b html`` under ``PCAPKIT_SPHINX=1``: 53 warnings on ``main`` + before this change, 54 after, the one addition being a pre-existing bare ``Type`` cross-reference ambiguity newly rendered by ``TraceFlow._foutio``'s new directive rather than a defect this change introduced -- #709 tracked it. No line under ``pcapkit/`` changed (#684). @@ -2374,8 +2397,9 @@ pull requests between #326 and #509. Sphinx's cross-reference resolver resolves against every class named ``Type`` in the project rather than against ``typing.Type`` -- there are five -- and silently linked to ``pcapkit.const.l2tp.type.Type``, an L2TP - field-type enum with nothing to do with dumpers. Line 406 - (``#: Type[Dumper]: Dumper class.``) is the live case: once #684 rendered + field-type enum with nothing to do with dumpers. Line 424 + (``#: ~typing.Type[Dumper]: Dumper class.``, spelled out since #709 fixed + it) is the live case: once #684 rendered ``TraceFlow._foutio``, the built docs pointed a reader at the wrong class with no warning that anything had gone sideways. Both sites now spell it ``~typing.Type[Dumper]``, the same form eight other files in the tree