diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a13eb545..0e8bc02ad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,185 +2,185 @@ The largest release since 1.0, and the first recorded here as it happened rather than reconstructed. Three more extraction engines, ESP with payload decryption, SCTP and NGAP over SCTP, a library logger that no longer hijacks the consumer's, and a defect programme run through the issue tracker spanning 254 issues and pull requests, from [#121](https://github.com/JarryShaw/PyPCAPKit/pull/121) to [#883](https://github.com/JarryShaw/PyPCAPKit/pull/883) as of this revision. The programme is still running, so those figures are a snapshot rather than a total -- they are re-derived from the entries below each time this file is regenerated. -Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` is what reshipped it -- the release workflow is version-driven, so an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels ([#497](https://github.com/JarryShaw/PyPCAPKit/pull/497), [#498](https://github.com/JarryShaw/PyPCAPKit/pull/498)). +Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` reshipped it -- an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels ([#497](https://github.com/JarryShaw/PyPCAPKit/pull/497), [#498](https://github.com/JarryShaw/PyPCAPKit/pull/498)). ### pcapkit.const #### Changed -- **a breaking change to** `pcapkit.const.reg.apptype`. `AppType` becomes a memberless base over one registry per transport protocol -- `TCP`, `UDP`, `SCTP` and `DCCP` -- because `aenum` refuses to subclass an enumeration that has members. `isinstance(TCP.http, AppType)` still holds and `AppType.get(port, proto=...)` proxies to the registry, so nothing under `pcapkit.protocols` changes. **The 1,004 rows IANA assigns no port, and the 704 it assigns no transport protocol, stop being members**; they become list-table docstrings on the registry that owns them, which autodoc publishes. `__members_proto__` is now a per-registry `MultiDict` keyed on port, so all three services IANA registers on port 80 survive and `extend_enum` on an occupied port appends rather than displacing what a lookup returned. `get()` answers with the canonical service, curated from `/etc/services` because IANA names no precedence among them, and `get_all()` reaches the aliases; `get()`'s dead `str` branch is removed rather than carried over ([#754](https://github.com/JarryShaw/PyPCAPKit/pull/754)). -- the six `raise ValueError(...)` calls still using `%` formatting in the generated `pcapkit.const.reg.apptype.apptype`, left that way by [#759](https://github.com/JarryShaw/PyPCAPKit/issues/759) so its own diff over a 12,391-member file stayed reviewable, are f-strings, matching the other 605 tree-wide ([#792](https://github.com/JarryShaw/PyPCAPKit/issues/792)). The remainder followed: the three other `_missing_` guards still raising with `%` -- in `pcapkit.vendor.tcp.flags`, `pcapkit.vendor.ftp.command` and `pcapkit.vendor.http.method` -- `AppType`'s `__new__`, `__repr__` and `__str__`, and the `extend_enum` mint fallbacks in `get()` and in the span-handling tail. `__new__`'s format sets every real member's underlying `StrEnum` value, so that one is proven byte-identical member by member across all 12,391 -- TCP 6,147, UDP 6,143, SCTP 91, DCCP 10 -- rather than spot-checked. The module-level `consider-using-f-string` disable drops from `tcp.flags` and `reg.apptype.apptype`, which carry no `%`-formatted code left, and stays on `ftp.command` and `http.method`, whose `__repr__` still renders with one. Both rounds are edited in the vendor templates, braces doubled since `BASE` and `LINE` are f-strings emitting f-strings, and applied to the generated const files by hand rather than by a crawl, which would fetch IANA's live registry and rewrite unrelated member rows ([#798](https://github.com/JarryShaw/PyPCAPKit/issues/798)). -- the three `__repr__` methods [#798](https://github.com/JarryShaw/PyPCAPKit/issues/798) left on `%` formatting -- `pcapkit/const/ftp/command.py:40,119` and `pcapkit/const/http/method.py:42` -- are f-strings now, so the `consider-using-f-string` disable drops from both files and from both `pcapkit/vendor/` templates; each generator's own `%`-formatted `wrap_comment` argument went with them, dropping their two inline disables. The other bespoke templates in these directories -- `{const,vendor}/ftp/return_code.py`, `{const,vendor}/http/status_code.py` and `vendor/http/frame.py` -- and the shared `vendor/default.py` one still carry the disable, so [#804](https://github.com/JarryShaw/PyPCAPKit/issues/804)'s claim that nothing in `pcapkit/{const,vendor}/{ftp,http}/` needs it holds for this pair only. Generated, so both sides changed: rendering each `LINE` template with the real arguments reproduces the committed const module byte-for-byte, and the render-against-committed diff is 6 changed lines for `ftp/command` and 4 for `http/method`, the intended pairs and nothing else. `test_the_disable_drops_only_where_nothing_else_needs_percent_formatting` asserted the opposite for this pair -- that the disable was retained -- so it is flipped rather than added to; `tests/const/test_const_enum_builtin_parity.py` goes 22 to 27 tests / 701 subtests, with 8 subtests across the 3 converted methods failing against the unfixed pair. `repr()` is asserted member by member against the old `%` expression's own output for all of `Command`, `FEATCode` and `Method`, including a `desc` of `None` ([#804](https://github.com/JarryShaw/PyPCAPKit/issues/804)). -- `test_const_enum_no_mint.py`'s `is_manufactured()` exemption for [#878](https://github.com/JarryShaw/PyPCAPKit/pull/878)'s (above) 53 preserved hex-suffixed names moves from a per-*file* exemption to a per-*name-shape* one. Keying it on `HEX_SUFFIXED_NAME_EXEMPT_PATHS` made the exemption shape-blind inside those two files: rewriting a branch to `'Xyplex_%d' % value` -- exactly the value-derived shape the predicate exists to flag -- still left the sweep green, since `exempt_hits` stayed at 53 regardless of what the name actually was. The check now does an AST structural comparison instead: a `%`-`BinOp` whose left side is a string constant ending in `'_0x%s'` and whose right side is structurally identical to a parsed `hex(value)[2:].upper().zfill(4)` template, matched via `ast.dump()` equality rather than a source-text or per-prefix pattern -- there are 53 distinct prefixes, `Xyplex_` through `Registered by Xerox_`, and the AST match is immune to quote-style or whitespace drift in generated source. `HEX_SUFFIXED_NAME_EXEMPT_PATHS` is dropped entirely. Verified independently: mutating `pcapkit/const/reg/ethertype.py`'s `Xyplex_0x%s`-style branch to the bare value-derived form now fails with `52 != 53: expected exactly 53 hex-suffixed-name exemptions to fire`, where it used to pass; the unmutated file stays green, and a value-derived name in an unrelated file (mutating `pcapkit/const/arp/hardware.py` to `'Unassigned_%d' % value`) is still flagged, naming that path. The exact `exempt_hits == 53` bound is kept, so adding or removing a branch still forces a human look. One file changed, `tests/const/test_const_enum_no_mint.py`; nothing under `pcapkit/` or `docs/`. Closes [#879](https://github.com/JarryShaw/PyPCAPKit/issues/879) ([#881](https://github.com/JarryShaw/PyPCAPKit/pull/881)). +- **a breaking change to** `pcapkit.const.reg.apptype`. `AppType` becomes a memberless base over one registry per transport protocol -- `TCP`, `UDP`, `SCTP` and `DCCP` -- because `aenum` refuses to subclass an enumeration that has members. `isinstance(TCP.http, AppType)` still holds and `AppType.get(port, proto=...)` proxies to the registry, so nothing under `pcapkit.protocols` changes. **The 1,004 rows IANA assigns no port, and the 704 it assigns no transport protocol, stop being members**; they become list-table docstrings on the registry that owns them. `__members_proto__` is now a per-registry `MultiDict` keyed on port, so all three services IANA registers on port 80 survive. `get()` answers with the canonical service, curated from `/etc/services` because IANA names no precedence, and `get_all()` reaches the aliases ([#754](https://github.com/JarryShaw/PyPCAPKit/pull/754)). +- the six `%`-formatted `raise ValueError(...)` calls in the generated `pcapkit.const.reg.apptype.apptype`, left by [#759](https://github.com/JarryShaw/PyPCAPKit/issues/759) so its diff over a 12,391-member file stayed reviewable, are f-strings, matching the other 605 tree-wide ([#792](https://github.com/JarryShaw/PyPCAPKit/issues/792)). The remainder followed: three `_missing_` guards (`pcapkit.vendor.tcp.flags`, `.ftp.command`, `.http.method`), `AppType`'s `__new__`/`__repr__`/`__str__`, and the `extend_enum` fallbacks in `get()` and in the span-handling tail. The `consider-using-f-string` disable drops from `tcp.flags` and `reg.apptype.apptype` and stays on `ftp.command` and `http.method`, whose `__repr__` still used `%`. Edited in the vendor templates and applied to the generated const files by hand, since a crawl would fetch IANA's live registry and rewrite unrelated rows ([#798](https://github.com/JarryShaw/PyPCAPKit/issues/798)). +- the three `__repr__` methods [#798](https://github.com/JarryShaw/PyPCAPKit/issues/798) left on `%` (`pcapkit/const/ftp/command.py:40,119`, `pcapkit/const/http/method.py:42`) are f-strings, so the disable drops from both files and both `pcapkit/vendor/` templates. The other bespoke templates (`{const,vendor}/ftp/return_code.py`, `{const,vendor}/http/status_code.py`, `vendor/http/frame.py`) and `vendor/default.py` still carry it, so [#804](https://github.com/JarryShaw/PyPCAPKit/issues/804)'s claim that nothing in `pcapkit/{const,vendor}/{ftp,http}/` needs it holds for this pair only. `test_the_disable_drops_only_where_nothing_else_needs_percent_formatting` asserted the opposite for this pair and is flipped; `repr()` is asserted member by member against the old `%` output ([#804](https://github.com/JarryShaw/PyPCAPKit/issues/804)). +- `test_const_enum_no_mint.py`'s `is_manufactured()` exemption for [#878](https://github.com/JarryShaw/PyPCAPKit/pull/878)'s (above) 53 preserved hex-suffixed names moves from a per-*file* to a per-*name-shape* one. The per-file `HEX_SUFFIXED_NAME_EXEMPT_PATHS` was shape-blind: rewriting a branch to `'Xyplex_%d' % value` -- exactly the value-derived shape the predicate exists to flag -- left the sweep green. The check now matches an AST `%`-`BinOp` whose left side is a string ending in `'_0x%s'` and whose right side equals a parsed `hex(value)[2:].upper().zfill(4)` (via `ast.dump()`, immune to quote-style or whitespace drift). The exact `exempt_hits == 53` bound stays, so adding or removing a branch still forces a human look. Only `tests/const/test_const_enum_no_mint.py` changed. Closes [#879](https://github.com/JarryShaw/PyPCAPKit/issues/879) ([#881](https://github.com/JarryShaw/PyPCAPKit/pull/881)). #### Fixed - constant lookups that rejected a value the registry defines. `RouterAlert(0)` is the only value [RFC 2113](https://datatracker.ietf.org/doc/html/rfc2113) defines and the one IGMP, RSVP and MLD actually send, and it was discarded because the vendor crawler skipped a header row IANA's CSV does not have; IPX `Socket(0)` is that protocol's own default, so `bytes(IPX(...))` crashed on its own defaults; and two FTP `_missing_` overrides were plain methods rather than classmethods, so every unregistered value raised `TypeError` instead of extending the enumeration ([#492](https://github.com/JarryShaw/PyPCAPKit/issues/492), [#503](https://github.com/JarryShaw/PyPCAPKit/pull/503)). -- the string-keyed `get()` in `pcapkit.const.ftp.command` and `pcapkit.const.http.method` tested membership with the raw key but registered `key.upper()`, so the *first* lowercase or mixed-case token raised `TypeError: 'RETR' already in use` rather than resolving. Reachable from wire data for FTP: `pcapkit.protocols.application.ftp` compiles its request pattern with `re.I` and passes the match verbatim, and [RFC 959 Section 5](https://datatracker.ietf.org/doc/html/rfc959#section-5) makes FTP commands case-insensitive -- "Upper and lower case alphabetic characters are to be treated identically", listing `RETR Retr retr ReTr rETr` as the same command -- so `retr file.txt` was a valid request this library could not parse. Both `get()` and `_missing_` now look the key up under the same canonical upper-case name they register it under, so every casing resolves to the one member that already exists instead of colliding with it. Resolving rather than registering a second member matters beyond not crashing -- a duplicate `GET` would carry neither the `safe` nor the `idempotent` attribute of the real one ([#582](https://github.com/JarryShaw/PyPCAPKit/issues/582), [#583](https://github.com/JarryShaw/PyPCAPKit/issues/583)). -- `get()`'s documented `default` was ignored on the integer path throughout the generated `pcapkit.const` tree, because `get` delegated the lookup to the enum call and `_missing_` has no access to the caller's `default` -- so `Hardware.get(99999, 0)` raised `ValueError: 99999 is not a valid Hardware` instead of returning the fallback it was handed. The integer path now consults `default` before letting the lookup error escape. `-1`, the placeholder the generated signature already carried, was what separated "no default was supplied" from "a default was supplied and should be used" at the time, so a caller that asked for no fallback got the error rather than a silent substitution. That `-1` convention is being replaced by an identity sentinel object in [#859](https://github.com/JarryShaw/PyPCAPKit/pull/859), not yet landed, once the shared base class [#858](https://github.com/JarryShaw/PyPCAPKit/pull/858) (below) puts `get()` behind one call site for most of these registries instead of many. The sweep [#584](https://github.com/JarryShaw/PyPCAPKit/issues/584) asked for puts the scope at 110 of the 118 integer registries, not the three the issue named; the two carrying a bespoke integer fallback of their own, `pcapng` `OptionType` and `reg` `AppType`, are deliberately left alone, since neither drops a default by raising. Not reachable from wire data -- every value a wire field can carry already resolves -- so this is a contract fix rather than a parse fix. Applied to the nine vendor templates as well as the 113 generated modules, and a new test renders the shared template and compares it against the module generated from it, so a regeneration cannot quietly undo it ([#584](https://github.com/JarryShaw/PyPCAPKit/issues/584)). -- `_missing_` in the four Mobility Header flag enumerations ended in `return cls(value)`, the same constructor that had just failed to find the value, so every in-range value that is not already a member re-entered `_missing_` unbounded and raised `RecursionError`. `BindingACKFlag`, `BindingUpdateFlag`, `HandoverACKFlag` and `HandoverInitiateFlag` are `IntFlag` types whose members are single bits, so the hole was not some exotic bit pattern: `F(0)` -- no flags set, the most ordinary value a flag octet can carry -- recursed on all four, and so did every composite of two defined bits, such as `BindingACKFlag(0x06)`. Defining `_missing_` at all is what caused it, because it shadowed the `aenum` `Flag` machinery that resolves exactly those values; `pcapkit/const/tcp/flags.py` defined no `_missing_` at the time and never had the defect -- it has one now, added by [#647](https://github.com/JarryShaw/PyPCAPKit/issues/647) below, ending in the same `super()._missing_(value)` tail for the same reason. All four now end in `return super()._missing_(value)`, which is what `pcapkit/vendor/default.py` emits for every other generated enumeration and what 75 of the 117 modules under `pcapkit/const/` already did -- 77 now, since [#647](https://github.com/JarryShaw/PyPCAPKit/issues/647) below gives the same ending to three more classes but only two land in modules the count did not already have, `TransportProtocol` sharing `reg/apptype.py` with the already-counted `AppType` -- so `F(0)` is an empty flag and `BindingACKFlag(0x06)` is `S|D`. The `extend_enum` idiom that `pcapkit/const/pcapng/record_type.py` and `secrets_type.py` use to mint a member for an unassigned integer -- the only other two modules whose `_missing_` ends in `return cls(value)`, and which do not recurse precisely because that `extend_enum` runs first -- was considered and rejected for a flag type: naming `0x06` `Unassigned_0x06` would hide the composite and pollute `_member_map_` with an entry per bit pattern, up to 65536 of them for `BindingUpdateFlag`. The range guard above it is untouched, so an out-of-range or non-integer value is still the same `ValueError`. Fixed in the four `pcapkit/vendor/mh/` templates and regenerated, the `pcapkit/const/mh/` modules being generated output that the next crawl would otherwise revert ([#623](https://github.com/JarryShaw/PyPCAPKit/issues/623)). -- six of the 123 constant registries under `pcapkit/const/` rejected an invalid value in a way the built-in `enum` does not, and three of them did not reject it at all. `Flags`, `ftp.command.CommandType` and `reg.apptype.TransportProtocol` are `IntFlag` types that defined no `_missing_`, so the `aenum` `Flag` machinery composed a pseudo-member for any integer whatsoever: `Flags(-1)` returned `65520`, the OR of every declared TCP header flag, so a value no 16-bit wire field can hold read back as *every* flag set at once, while `Flags(-65536)` read back as none set. `ftp.command.Command`, `ftp.command.FEATCode` and `http.method.Method` are `StrEnum` types whose `_missing_` reached `value.upper()` before checking the type, so an integer raised `AttributeError: 'int' object has no attribute 'upper'`. All six now raise a bare `ValueError`, which is what `enum.IntEnum` raises for a value it does not define and what 113 of the 117 modules already did. Issue [#647](https://github.com/JarryShaw/PyPCAPKit/issues/647)'s own proposal -- replacing those 113 with `pcapkit.utilities.exceptions.EnumError` -- was deliberately rejected: `EnumError` is `(BaseError, TypeError)` and not a `ValueError` at all, so it would have diverged from the built-in it was meant to improve on, walked past the `except ValueError` in all 113 generated `get()` bodies and silently undone [#584](https://github.com/JarryShaw/PyPCAPKit/issues/584), and logged CRITICAL from `BaseError.__init__` once per discarded default. The one deliberate divergence from the built-in is kept and now pinned: the mutable registries look a value up, miss, and `extend_enum` it rather than raising, so `Method('FROBNICATE')`, `ProtectionAuthority(1 << 70)` and `AppType.get(65000, proto=tcp)` still register. The three flag guards end in `return super()._missing_(value)`, so in-range composites still decompose and [#623](https://github.com/JarryShaw/PyPCAPKit/issues/623) stays fixed. Two of the four unguarded modules needed no change -- `ipv6/extension_header.py` and `ftp.command.ConformanceRequirement` define no `_missing_` either, and `aenum` already raises the bare `ValueError` for them. `pcapkit/vendor/tcp/flags.py` turned out to carry the guard's value checker all along, as `FLAG = 'isinstance(value, int) and 4 <= value <= 15'`, while its template never interpolated it; those are the registry's *bit offsets* and its members are `1 << offset`, so emitting it unchanged would have rejected every composite, every member above bit 3, and `Flags(0)`. It now reads `0 <= value <= 0xFFFF`, the 16-bit field those bits live in. `TransportProtocol` reads its bound off `cls.__members__` instead of a literal, because `TransportProtocol.get` extends the registry at runtime at `max * 2` and a written-down bound would reject the member it had just grown. Fixed in the four bespoke templates under `pcapkit/vendor/` rather than in the generated tree -- `pcapkit/vendor/default.py` needed no change, since the 113 it emits were already correct -- and regenerated from the live IANA registries: 42 insertions and zero deletions across exactly 4 of the 134 files, so nothing else in the tree was stale, including the 30k-line `reg/apptype.py`. A new `tests/const/test_const_enum_builtin_parity.py` asserts the property the fix is actually about, that a constant registry and a stdlib `enum.IntEnum` raise the same exception type for the same invalid value, over all 123 registries rather than the six; it is keyed on `aenum.Enum` because an `IntFlag` is not a subclass of `IntEnum` -- its MRO runs through `Flag` instead -- which is how three of these six escaped the two existing sweeps. `tests/const/test_const_enum_get.py` drops `Flags` from `EXPECTED_TO_RESOLVE_ANYTHING` and its sweep grows 110 to 111, a registry that bounds its domain finally having a failure for `default` to fall back from ([#647](https://github.com/JarryShaw/PyPCAPKit/issues/647)). -- **a breaking change to** `AppType.get`: an out-of-range port is refused rather than minted. Its `except ValueError` caught the range rejection `_missing_` raises and minted regardless, so `AppType.get(-1, proto='tcp')` returned `PORT_-1_tcp` -- a `__members__` key no attribute access can reach -- while `TCP(-1)` raised. It now tests `_missing_` for `None`, which is its "no row for this port" answer, and lets the rejection through, so both entry points raise the same `ValueError`; a valid but unassigned port still mints, which is what `get` is for. **`Transport._make_port` passes an unvalidated `int`, so `TCP.make(srcport=99999)` now raises where it minted junk.** Separately, the crawler rendered one `_missing_` branch per registry row, so the two spans IANA registers on more than one transport emitted the same condition twice and every registry answered with the first: `AppType.get(6010, proto='udp')` gave a UDP member whose `proto` read `tcp`, `6666/udp` answered `ircu` rather than `reserved`, and `proto='sctp'` minted `x11` where IANA assigns nothing. A branch naming a transport now tests `cls.__transport__`; the 762 naming none answer every registry as before ([#764](https://github.com/JarryShaw/PyPCAPKit/pull/764)). -- **a breaking change to** `AppType._dispatch`: it resolved a `proto` naming several transport protocols by taking its lowest set bit. `show_flag_values` iterates LSB-first and `tcp` is the lowest declared bit, so every composite containing it dispatched into the TCP registry whatever else it named -- silently, and undetectably to a caller checking equality, since `AppType.__eq__` compares on `port` alone. `TransportProtocol` is an `aenum.IntFlag` and a member carries the whole set IANA assigned the service, so `tcp | udp` is an ordinary value to read off one and an ordinary thing to pass back in. Sweeping all 10,625 multi-transport members through `get(m.port, proto=m.proto)` gave 0 exceptions, 0 mints and 46 answers naming another service, at 20 distinct ports; 44 of those differ only in that the member is not its port's canonical, which a single-bit lookup does too and which `get` documents, leaving two real defects -- port 888, where `accessbuilder` resolved to TCP's `cddbp`, and port 999, where `puprouter` resolved to TCP's `garcon` against UDP's `applix`. It now raises `ProtocolError`, a `ValueError` subclass, so the documented contract of `get` and `get_all` still holds. Refusing costs no caller: the ten library call sites into the lookup each name one transport protocol, and `register_apptype`, the one place that reads a member's composite `proto`, tests it with `in` and never reaches the lookup ([#759](https://github.com/JarryShaw/PyPCAPKit/issues/759)). -- `get()`'s string-key miss and every generated registry's `_missing_` bounded-range branch both called `aenum.extend_enum()` on an unrecognised value, so e.g. `Hardware(40)` minted a permanent member on first call, forever. Fixed at the two shared sites in `pcapkit/vendor/default.py`, plus a new `register(value, name)` base classmethod for the caller-named path that is still meant to mint. All 105 generated `pcapkit/const/` registries are hand-edited to match and independently proved by regenerating the 21 changed ones from live IANA data -- byte-identical to the hand-edit; a further seven regenerate with no change at all. `get()`'s no-longer-minting miss applies to all 105 registries; the `_missing_` fix only changes runtime behaviour for the **21** whose `_missing_` carries a bounded-range `extend_enum` branch -- the other 84 have no such branch to fix: 83 override `process()` with their own `extend_enum` call, and one, `hip.transport.Transport`, inherits the unmodified template but has no unassigned gap for its FLAG-bound range to mint against. Two toolkit call sites, `pcapkit/toolkit/scapy.py` and `pcapkit/toolkit/pyshark.py`, fed `get()` a string key with no default and now raise `MissingKeyError` where they used to get back a minted placeholder; four protocol reader sites in `hopopt.py`/`ipv6_opts.py` convert the same `KeyError` into `ProtocolError`. Riding along: `pcapkit/toolkit/pyshark.py` gains a two-entry `FILTER_NAME_TO_LINKTYPE` (`eth` to `ETHERNET`, `tr` to `IEEE802_5`), consulted ahead of `LinkType.get` since pyshark reports a filter name, not a DLT name. An AST census puts what remains out of scope at 1026 minting call sites -- 1007 in `_missing_` and 7 in `get` across `pcapkit/const/`, plus 12 hand-written `extend_enum` calls in `pcapkit/protocols/internet/mh.py` and `pcapkit/protocols/application/ngap.py` -- all deferred to a later tier; `register`/`register_alias` account for a further 106 sites that are the by-design caller-named path and stay untouched ([#838](https://github.com/JarryShaw/PyPCAPKit/pull/838)). -- **a breaking change to** `pcapkit.const.ipx.socket.Socket`'s `_missing_`: its five archived range branches were tested in table order, and three were strict subsets of a range tested earlier -- `(0x0020, 0x003F)` after `(0x0001, 0x0BB8)`, and both `(0x4000, 0x4FFF)` and `(0x8000, 0xFFFF)` after `(0x0BB9, 0xFFFF)` -- so they never fired. Fixed in the generator, `pcapkit/vendor/ipx/socket.py`, which now lists each subset range ahead of the range containing it; `pcapkit/const/ipx/socket.py` changes only as the regenerated, byte-identical consequence. **36891 of the 65521 `_missing_`-resolved values (56.3%) change name** -- 32 from `Registered by Xerox` to `Experimental`, 4095 from `Dynamically Assigned` to `Dynamically Assigned Socket Numbers` and 32764 from `Dynamically Assigned` to `Statically Assigned Socket Numbers`. No defined member changes and every value still resolves; the shift is entirely in which archived row's name a previously-shadowed value receives. The prior tests had pinned the shadowing as intended behaviour, with comments reading `# masks 'Experimental'`; those assertions are corrected here rather than dropped, and a new test pins all three ranges as reachable ([#847](https://github.com/JarryShaw/PyPCAPKit/pull/847)). -- `LinkType(209).name` returned `'IPMB_LINUX'`, the name tcpdump's own table (and pcapkit's generated comment) marks "Legacy names (do not use)", because that row was emitted above the current `I2C_LINUX = 209` and `aenum` gives a value to whichever member is defined first -- introduced by a 2024-05-04 regeneration that inserted the legacy row above the current one. Fixed in the generator: `pcapkit/vendor/reg/linktype.py` now sinks any row whose note mentions "legacy" into a bucket appended after every current row, so the current name is always defined first for a value the table double-assigns; today that is value 209 alone. `pcapkit/const/reg/linktype.py` changes by a 3-line move, verified byte-reproducible from the generator. `IPMB_LINUX` is not removed -- it stays a reachable alias, so `LinkType['IPMB_LINUX']` still resolves to 209 -- only the value-to-name direction changes. Not breaking: no name disappears, no value changes and no membership changes; `LinkType.IPMB_LINUX is LinkType.I2C_LINUX` holds both before and after, and only the canonical name for the shared value moves ([#848](https://github.com/JarryShaw/PyPCAPKit/pull/848)). -- **a breaking change to** 82 more const registries' `_missing_`: applies the owner's mint/unmint ruling (settled on [#847](https://github.com/JarryShaw/PyPCAPKit/pull/847), confirmed on [#775](https://github.com/JarryShaw/PyPCAPKit/issues/775) -- *"a final concrete assigned name -> mint; a notation for the readers -> unmint"*) across every one of the 89 `_missing_` files still calling `extend_enum` on this tree. 80 registries convert wholly (164 branches) -- each mints a bare status word (`Unassigned`, `Reserved [...]`, `Reserved for Private/Experimental Use`, `Unspecified in the IANA registry`, `Deprecated` and similar), which the ruling treats as a notation for the reader rather than an assignment. Two are mixed: `EtherType`'s `DEC Unassigned` (3 branches) and its "Old Xerox Experimental values. Invalid as an Ethertype since 1983." branch (1) convert, while 46 attributed vendor-company names across 50 range blocks (Xyplex, Motorola, Walker Richer & Quinn and 43 more) keep minting, since a proprietary protocol has no public name beyond the company's -- two more of the surviving branches, `IEEE802_3_Length_Field` and `Berkeley_Trailer_encap_IP`, keep minting too but are not company names and so are not part of that count; `Socket` (ipx)'s `Experimental`, `Dynamically Assigned Socket Numbers`, `Statically Assigned Socket Numbers` and `Dynamically Assigned` convert -- each names an allocation policy, not a specific assignment -- while `Registered by Xerox` keeps minting, a real ownership fact. 7 of the 89 are left alone: `CGAType`'s `Tag_` mint is not an IANA-style range at all, and the other 6 files sit on 9 classes not yet on `EnumRegistry` (`AppType`, `StatusCode`, `ftp.return_code`, `ftp.command`, `Method`, `OptionType`), blocked on [#859](https://github.com/JarryShaw/PyPCAPKit/pull/859)/[#860](https://github.com/JarryShaw/PyPCAPKit/issues/860). Each conversion touches both the vendor crawler and the generated `const` file, per [#838](https://github.com/JarryShaw/PyPCAPKit/pull/838)'s and [#858](https://github.com/JarryShaw/PyPCAPKit/pull/858)'s precedent; verified byte-identical regeneration on a 5-file sample spanning both crawler shapes and both mixed registries (`ipv4/tos_del.py`, `ipx/socket.py`, `reg/ethertype.py`, `mh/access_type.py`, `mh/status_code.py`). Fixed collateral breakage this caused in three sibling test files that had pinned the pre-ruling mint behaviour for `ProtectionAuthority`, `TransType` and four of `Socket`'s ranges. `docs/source/conventions.rst`'s worked examples already agreed with keeping `Registered by Xerox` under Mint by the time this PR landed: two prior commits corrected them, `dbae27548` and `6b333d7e4`, the second of which is this PR's own merge-commit parent ([#861](https://github.com/JarryShaw/PyPCAPKit/pull/861)). -- **a breaking change to** `EtherType`'s `_missing_` ordering: values `0x0101`-`0x01FF` now resolve to their Old Xerox Experimental name instead of an `IEEE802_3_Length_Field_0x0101`-style one. The generated method tested `0x0000 <= value <= 0x05DC` (IEEE802.3 Length Field) before `0x0101 <= value <= 0x01FF` (Old Xerox Experimental); the second range sits wholly inside the first, and the method returns on its first matching `if`, so the Old Xerox branch was unreachable for every value it covers. Unlike [#841](https://github.com/JarryShaw/PyPCAPKit/issues/841)'s IPX `Socket` fix, these bounds come from the live IANA CSV rather than a literal table, so the fix is general rather than a manual swap: `pcapkit/vendor/reg/ethertype.py`'s `process()` now collects each range row, and a new `EtherType._insert_range` places each range ahead of the first already-placed range that fully contains it, leaving every non-overlapping pair in the CSV's own row order. `pcapkit/const/reg/ethertype.py` is regenerated; the diff is exactly the two branches swapping places. Swept all 56 range tests in the generated `_missing_`: this pair is the only containment among them, so no other subsumed range exists today. `tests/const/test_const_ethertype_862_unit.py` pins both the symptom against the committed const file and the root cause against `process()` fed a reproduction of IANA's own CSV rows, plus the general ordering rule against synthetic nested ranges -- each verified to fail against the pre-fix generator and const file. Closes [#862](https://github.com/JarryShaw/PyPCAPKit/issues/862) ([#865](https://github.com/JarryShaw/PyPCAPKit/pull/865)). -- **a breaking change to** `Method`: `__new__` called `str.__new__(cls)` with no argument, so every one of the 40 registered members' `str` payload was permanently empty regardless of its declared value -- `str(Method.GET) == ''`, and `Method.GET == 'GET'` was `False`; `.value` was always correct. [#869](https://github.com/JarryShaw/PyPCAPKit/pull/869) (above) had fixed this class's *unregistered* path only, exposing the asymmetry: `str(Method('frob')) == 'frob'` but `str(Method.GET) == ''`. Fixed to `str.__new__(cls, value)`, mirroring `Command.__new__` (never had this defect), in both the generator (`pcapkit/vendor/http/method.py`) and the generated module; a second regeneration is byte-identical. A survey of every hand-rolled `__new__` under `pcapkit/const/` found no sibling with the same defect: `Command` already passes its value, `FEATCode` has no custom `__new__`, and `OptionType`/`AppType` deliberately store a formatted display string as their value. The consequences, all in the correcting direction and all measured: `Method.GET == 'GET'` is `True` for all 40 members where it was `False`; `bool(Method.GET)` flips `False` to `True`, since an empty payload made every member falsy; `json.dumps(Method.GET)` now emits `"GET"` where it emitted `""`; `sorted()` over members is now lexicographic rather than input-order-stable; and a member-keyed `dict`/`set` no longer collapses -- `{m: ... for m in Method}` gains 40 entries where it had 1, because `str.__hash__` wins the MRO and all 40 members previously hashed as `''` and compared equal to one another. Nothing under `pcapkit/protocols` relied on the old behaviour; the one caller in `httpv1.py`/`httpv2.py` only reads `.value` via `Method.get()`. Closes [#870](https://github.com/JarryShaw/PyPCAPKit/issues/870) ([#871](https://github.com/JarryShaw/PyPCAPKit/pull/871)). -- **a breaking change to** `TransportProtocol` and `AppType`: step 2 of [#860](https://github.com/JarryShaw/PyPCAPKit/issues/860), PR 2 of 2 (the 8 non-`AppType` bespoke registries were [#869](https://github.com/JarryShaw/PyPCAPKit/pull/869), above). Brings `AppType` and its four transport subclasses onto `EnumRegistry` and stops both of its mint sites -- `_missing_`'s 766 range branches, and `get()`'s own second, independent mint (`PORT_{port}_{transport}` -> `'unknown'`) -- both now building through a new `_unregistered_member` override that reconstructs the `svc`/`port`/`proto` attributes the base's generic helper does not know about, per the owner's ruling that *"only IANA registered ones are legit values... get will not have sufficient information to create new ones"*. All 766+1 branches convert uniformly, including the 8 that name a real service assigned to a whole port span rather than declared individually (7 distinct names, `x11` appearing twice: `x11`, `active-net`, `satvid-datalnk`, `vrml-multi-use`, `ircu`, `swx`, `flex-lm`); unlike `FEATCode`'s fix ([#869](https://github.com/JarryShaw/PyPCAPKit/pull/869), above) there is no import-time self-mutation defect to address by declaring members statically, so these stay deliberately unregistered lookups. `AppType` also gains a working `register()`, previously absent -- the base's generic one would have built a member with `svc=''` the moment this class mixed in `EnumRegistry` unoverridden, since it calls `__new__` with only one positional argument. **The breaking part**: `TransportProtocol` moves from power-of-two values (`undefined=0, tcp=1, udp=2, sctp=4, dccp=8`) to sequential `auto()` (`undefined=0, tcp=1, udp=2, sctp=3, dccp=4`), per the owner's follow-up ruling once `|`-joined composite values stopped being parsed at all. `AppType.get`'s `proto` parameter treats whatever `int` it is given as a whole rather than decoding it, on either numbering, so the renumbering changes what specific integers *mean*, not only what hand-composed ones do -- measured on `AppType.get(80, proto=...)`: a bare `4` resolved to `SCTP.http` on `main` and now **silently** resolves to `DCCP.unknown`, no exception, because a real member sits at `4` under the new numbering too, just a different one; a bare `8` used to mint `DCCP.PORT_80_dccp` and now raises `ValueError`. The silent-reroute case is the one that matters and is disclosed here rather than only in a code comment, though deliberately left untested for the composed-value case specifically, per the owner's own ruling that it is "an obsoleted path from a breaking change". All five crawlers (`apptype`, `tcp`, `udp`, `sctp`, `dccp`) regenerate byte-identically on a second run. Fixed 9 existing tests whose bodies pinned the old minting or power-of-two behaviour and added 18 new tests to `test_const_enum_no_mint.py` against 1 removed (net +17); 403 test methods pass across `tests/const/` and `tests/protocols/transport/`, plus a further 142 across `tests/vendor/` and the other `TransportProtocol`-consuming suites ([#874](https://github.com/JarryShaw/PyPCAPKit/pull/874)). -- closes the remaining scope of [#775](https://github.com/JarryShaw/PyPCAPKit/issues/775): `EtherType` (52 branches) and `Socket` (1) were the last two registries whose `_missing_` still minted a permanent member for an unrecognised value; both now return a non-registering `cls._unregistered_member(value, name)`. The same AST walk the issue originally used, counting `extend_enum` inside `_missing_`/`get` across every class in `pcapkit/const/`, goes from 54 sites (52 in `reg/ethertype.py`, 1 each in `ipx/socket.py` and `mh/cga_type.py`) to the single `CGAType` carve-out that remains a deliberate exception (its mint is not an IANA-style range at all) -- taking the issue from its original 1,169 sites to that one. Measured both directions: before, `EtherType(0x0888)` grew `__members__` from 160 to 161 and a repeat lookup returned the identical object; after, it returns `` with `__members__` unchanged, absent from `_value2member_map_`, and equal-but-not-identical across lookups -- the same shape for `Socket(0x0010)`. Two asymmetries worth knowing: `pcapkit/vendor/ipx/socket.py`'s regeneration was verified byte-identical offline, since `Socket.LINK is None` means that crawler never touches the network, but `pcapkit/vendor/reg/ethertype.py`'s live-IANA-CSV fetch is off-limits here, so its correctness rests on a mechanical substitution checked to match exactly 52 lines plus the generator run against the already-committed CSV fixture -- weaker than byte-identity, and disclosed as such. Preserving each branch's existing hex-suffixed name reintroduces the value-derived name shape `test_const_enum_no_mint.py`'s `is_manufactured()` sweep forbids; rather than weakening that predicate, a narrow `HEX_SUFFIXED_NAME_EXEMPT_PATHS` covers exactly these two files, on the grounds that the naming collision the check guards against cannot occur once `_unregistered_member` never registers. `pcapkit/corekit/enum.py`'s docstring is corrected from naming three still-minting registries (`EtherType`, `Socket`, `CGAType`) to naming `CGAType` alone. `tests/vendor/test_ipx_socket_unit.py` is not among the tests this falsifies -- its code tokens are unchanged and its `EXPECTED_MISSING_NAMES` still carries `0x0010: 'Registered by Xerox_0x0010'` and still passes, since the name itself is preserved ([#878](https://github.com/JarryShaw/PyPCAPKit/pull/878)). +- the string-keyed `get()` in `pcapkit.const.ftp.command` and `pcapkit.const.http.method` tested membership with the raw key but registered `key.upper()`, so the *first* lowercase or mixed-case token raised `TypeError: 'RETR' already in use`. Reachable from wire data: `pcapkit.protocols.application.ftp` matches with `re.I` and passes the match verbatim, and [RFC 959 Section 5](https://datatracker.ietf.org/doc/html/rfc959#section-5) makes FTP commands case-insensitive, so `retr file.txt` was a valid request the library could not parse. `get()` and `_missing_` now look the key up under the canonical upper-case name, so every casing resolves to the existing member; a duplicate `GET` would have lacked the real one's `safe` and `idempotent` attributes ([#582](https://github.com/JarryShaw/PyPCAPKit/issues/582), [#583](https://github.com/JarryShaw/PyPCAPKit/issues/583)). +- `get()`'s documented `default` was ignored on the integer path across the generated `pcapkit.const` tree, because `_missing_` cannot see the caller's `default` -- `Hardware.get(99999, 0)` raised `ValueError` instead of returning `0`. The integer path now consults `default` before letting the error escape. `-1`, the placeholder the generated signature already carried, still separates "no default" from "a default"; it is to be replaced by an identity sentinel in [#859](https://github.com/JarryShaw/PyPCAPKit/pull/859) (not yet landed) once the shared base [#858](https://github.com/JarryShaw/PyPCAPKit/pull/858) (below) puts `get()` behind one call site. The sweep covers 110 of the 118 integer registries, not the three [#584](https://github.com/JarryShaw/PyPCAPKit/issues/584) named; `pcapng` `OptionType` and `reg` `AppType` carry a bespoke integer fallback and are deliberately left alone, since neither drops a default by raising. Not reachable from wire data, so a contract fix rather than a parse fix. Applied to the nine vendor templates and the 113 generated modules; a new test renders the shared template and compares it with the generated module, so a regeneration cannot undo it ([#584](https://github.com/JarryShaw/PyPCAPKit/issues/584)). +- `_missing_` in the four Mobility Header flag enumerations (`BindingACKFlag`, `BindingUpdateFlag`, `HandoverACKFlag`, `HandoverInitiateFlag`) ended in `return cls(value)`, the constructor that had just failed, so every in-range value that is not already a member re-entered `_missing_` unbounded and raised `RecursionError` -- `F(0)`, no flags set, and every composite such as `BindingACKFlag(0x06)`. Defining `_missing_` at all caused it, by shadowing the `aenum` `Flag` machinery that resolves exactly those values; `pcapkit/const/tcp/flags.py` had none and so never had the defect (it has one now, from [#647](https://github.com/JarryShaw/PyPCAPKit/issues/647) below, with the same tail). All four now end in `return super()._missing_(value)`, as `pcapkit/vendor/default.py` emits and 75 of the 117 modules under `pcapkit/const/` already did (77 with [#647](https://github.com/JarryShaw/PyPCAPKit/issues/647)'s two new modules). `F(0)` is an empty flag and `BindingACKFlag(0x06)` is `S|D`. The `extend_enum` mint used by `pcapng/record_type.py` and `secrets_type.py` was rejected for a flag type: naming `0x06` `Unassigned_0x06` would hide the composite and add a `_member_map_` entry per bit pattern, up to 65536 for `BindingUpdateFlag`. The range guard is untouched. Fixed in the four `pcapkit/vendor/mh/` templates and regenerated ([#623](https://github.com/JarryShaw/PyPCAPKit/issues/623)). +- six of the 123 constant registries rejected an invalid value differently from the built-in `enum`, and three did not reject it at all. `Flags`, `ftp.command.CommandType` and `reg.apptype.TransportProtocol` are `IntFlag` types with no `_missing_`, so `aenum` composed a pseudo-member for any integer: `Flags(-1)` returned `65520` (every TCP flag set, from a value no 16-bit field can hold) and `Flags(-65536)` none. `ftp.command.Command`, `FEATCode` and `http.method.Method` are `StrEnum` types whose `_missing_` called `value.upper()` before checking the type, so an integer raised `AttributeError`. All six now raise a bare `ValueError`, as `enum.IntEnum` does and 113 of the 117 modules already did. [#647](https://github.com/JarryShaw/PyPCAPKit/issues/647)'s proposal to replace those 113 with `pcapkit.utilities.exceptions.EnumError` was deliberately rejected: `EnumError` is `(BaseError, TypeError)`, not a `ValueError`, so it would diverge from the built-in, walk past the `except ValueError` in all 113 generated `get()` bodies (silently undoing [#584](https://github.com/JarryShaw/PyPCAPKit/issues/584)), and log CRITICAL from `BaseError.__init__` once per discarded default. The one deliberate divergence is kept and pinned: mutable registries look up, miss, and `extend_enum` rather than raising, so `Method('FROBNICATE')`, `ProtectionAuthority(1 << 70)` and `AppType.get(65000, proto=tcp)` still register. The three flag guards end in `return super()._missing_(value)`, so in-range composites still decompose and [#623](https://github.com/JarryShaw/PyPCAPKit/issues/623) stays fixed. `pcapkit/vendor/tcp/flags.py` carried the value checker all along as `FLAG = 'isinstance(value, int) and 4 <= value <= 15'` without interpolating it; those are *bit offsets*, so emitting it unchanged would have rejected every composite and `Flags(0)`. It now reads `0 <= value <= 0xFFFF`. `TransportProtocol` reads its bound from `cls.__members__`, because `get` grows the registry at runtime and a literal would reject the member it had just added. Fixed in the four bespoke templates under `pcapkit/vendor/` and regenerated from live IANA data (42 insertions, no deletions, in 4 of 134 files). A new `tests/const/test_const_enum_builtin_parity.py` asserts that every one of the 123 registries raises the same exception type as a stdlib `enum.IntEnum`; it keys on `aenum.Enum` because an `IntFlag` is not an `IntEnum` subclass, which is how three of the six escaped the existing sweeps ([#647](https://github.com/JarryShaw/PyPCAPKit/issues/647)). +- **a breaking change to** `AppType.get`: an out-of-range port is refused rather than minted. Its `except ValueError` caught the range rejection from `_missing_` and minted regardless, so `AppType.get(-1, proto='tcp')` returned an unreachable `PORT_-1_tcp` while `TCP(-1)` raised. It now tests `_missing_` for `None` ("no row for this port") and lets the rejection through; a valid but unassigned port still mints. **`Transport._make_port` passes an unvalidated `int`, so `TCP.make(srcport=99999)` now raises where it minted junk.** Separately, the crawler rendered one `_missing_` branch per registry row, so the two spans IANA registers on more than one transport emitted the same condition twice and every registry answered with the first: `AppType.get(6010, proto='udp')` gave a member whose `proto` read `tcp`, `6666/udp` answered `ircu` rather than `reserved`, and `proto='sctp'` minted `x11` where IANA assigns nothing. A branch naming a transport now tests `cls.__transport__`; the 762 naming none are unchanged ([#764](https://github.com/JarryShaw/PyPCAPKit/pull/764)). +- **a breaking change to** `AppType._dispatch`: it resolved a `proto` naming several transports by its lowest set bit, and `tcp` is lowest, so every composite containing it dispatched into the TCP registry -- silently, and undetectably to a caller checking equality, since `AppType.__eq__` compares on `port` alone. `TransportProtocol` is an `aenum.IntFlag` and a member carries the whole set IANA assigned the service, so `tcp | udp` is an ordinary value to pass back in. Sweeping all 10,625 multi-transport members through `get(m.port, proto=m.proto)` gave 46 answers naming another service at 20 ports; 44 differ only in not being the port's canonical, which a single-bit lookup does too, leaving two real defects: port 888 (`accessbuilder` resolved to TCP's `cddbp`) and port 999 (`puprouter` resolved to TCP's `garcon`). It now raises `ProtocolError`, a `ValueError` subclass, so the documented contract holds. No caller is affected: the ten library call sites each name one transport, and `register_apptype`, the one reader of a composite `proto`, tests it with `in` ([#759](https://github.com/JarryShaw/PyPCAPKit/issues/759)). +- `get()`'s string-key miss and every generated registry's `_missing_` bounded-range branch called `aenum.extend_enum()` on an unrecognised value, so e.g. `Hardware(40)` minted a permanent member. Fixed at the two shared sites in `pcapkit/vendor/default.py`, plus a new `register(value, name)` base classmethod for the caller-named path that still mints. All 105 generated registries are updated; the non-minting `get()` miss applies to all 105, but the `_missing_` fix changes behaviour only for the **21** with a bounded-range `extend_enum` branch. `pcapkit/toolkit/scapy.py` and `pcapkit/toolkit/pyshark.py` fed `get()` a string key with no default and now raise `MissingKeyError` instead of receiving a placeholder; four reader sites in `hopopt.py`/`ipv6_opts.py` convert the `KeyError` into `ProtocolError`. Riding along: `pcapkit/toolkit/pyshark.py` gains a two-entry `FILTER_NAME_TO_LINKTYPE` (`eth` to `ETHERNET`, `tr` to `IEEE802_5`), consulted ahead of `LinkType.get` since pyshark reports a filter name, not a DLT name. Out of scope and deferred: 1026 minting call sites (1007 in `_missing_`, 7 in `get`, 12 hand-written in `pcapkit/protocols/internet/mh.py` and `pcapkit/protocols/application/ngap.py`); the 106 `register`/`register_alias` sites are the by-design caller-named path ([#838](https://github.com/JarryShaw/PyPCAPKit/pull/838)). +- **a breaking change to** `pcapkit.const.ipx.socket.Socket`'s `_missing_`: three of its five archived range branches were strict subsets of a range tested earlier -- `(0x0020, 0x003F)` after `(0x0001, 0x0BB8)`, and `(0x4000, 0x4FFF)` and `(0x8000, 0xFFFF)` after `(0x0BB9, 0xFFFF)` -- so they never fired. The generator `pcapkit/vendor/ipx/socket.py` now lists each subset range ahead of the range containing it. **36891 of the 65521 `_missing_`-resolved values (56.3%) change name**: 32 from `Registered by Xerox` to `Experimental`, 4095 from `Dynamically Assigned` to `Dynamically Assigned Socket Numbers` and 32764 from `Dynamically Assigned` to `Statically Assigned Socket Numbers`. No defined member changes and every value still resolves. The prior tests had pinned the shadowing as intended (`# masks 'Experimental'`); they are corrected rather than dropped, and a new test pins all three ranges as reachable ([#847](https://github.com/JarryShaw/PyPCAPKit/pull/847)). +- `LinkType(209).name` returned `'IPMB_LINUX'`, which tcpdump's own table marks "Legacy names (do not use)", because a 2024-05-04 regeneration emitted that row above the current `I2C_LINUX = 209` and `aenum` gives a value to the member defined first. `pcapkit/vendor/reg/linktype.py` now sinks any row whose note mentions "legacy" after every current row, so the current name is defined first; today that affects value 209 alone. `IPMB_LINUX` stays a reachable alias (`LinkType['IPMB_LINUX']` still resolves to 209). Not breaking: no name disappears, no value or membership changes, and `LinkType.IPMB_LINUX is LinkType.I2C_LINUX` holds before and after; only the canonical name for the shared value moves ([#848](https://github.com/JarryShaw/PyPCAPKit/pull/848)). +- **a breaking change to** 82 more const registries' `_missing_`: applies the owner's mint/unmint ruling (settled on [#847](https://github.com/JarryShaw/PyPCAPKit/pull/847), confirmed on [#775](https://github.com/JarryShaw/PyPCAPKit/issues/775) -- *"a final concrete assigned name -> mint; a notation for the readers -> unmint"*) across the 89 `_missing_` files still calling `extend_enum`. 80 registries convert wholly (164 branches), each minting a bare status word (`Unassigned`, `Reserved [...]`, `Reserved for Private/Experimental Use`, `Unspecified in the IANA registry`, `Deprecated`), which the ruling treats as a notation rather than an assignment. Two are mixed: `EtherType`'s `DEC Unassigned` (3 branches) and "Old Xerox Experimental values. Invalid as an Ethertype since 1983." (1) convert, while 46 vendor-company names across 50 range blocks (Xyplex, Motorola, Walker Richer & Quinn and 43 more) keep minting, since a proprietary protocol has no public name beyond the company's; `IEEE802_3_Length_Field` and `Berkeley_Trailer_encap_IP` also keep minting but are not company names. `Socket` (ipx)'s allocation-policy names (`Experimental`, `Dynamically Assigned Socket Numbers`, `Statically Assigned Socket Numbers`, `Dynamically Assigned`) convert, while `Registered by Xerox` keeps minting, a real ownership fact. 7 of the 89 are left alone: `CGAType`'s `Tag_` mint is not an IANA-style range, and the other 6 files sit on 9 classes not yet on `EnumRegistry` (`AppType`, `StatusCode`, `ftp.return_code`, `ftp.command`, `Method`, `OptionType`), blocked on [#859](https://github.com/JarryShaw/PyPCAPKit/pull/859)/[#860](https://github.com/JarryShaw/PyPCAPKit/issues/860). Each conversion changes both the vendor crawler and the generated file, per [#838](https://github.com/JarryShaw/PyPCAPKit/pull/838)'s and [#858](https://github.com/JarryShaw/PyPCAPKit/pull/858)'s precedent. `docs/source/conventions.rst`'s worked examples already agreed with keeping `Registered by Xerox` under Mint ([#861](https://github.com/JarryShaw/PyPCAPKit/pull/861)). +- **a breaking change to** `EtherType`'s `_missing_` ordering: values `0x0101`-`0x01FF` now resolve to their Old Xerox Experimental name instead of an `IEEE802_3_Length_Field_0x0101`-style one. The generated method tested `0x0000 <= value <= 0x05DC` (IEEE802.3 Length Field) first, and the Old Xerox range sits wholly inside it, so that branch was unreachable. Unlike [#841](https://github.com/JarryShaw/PyPCAPKit/issues/841)'s IPX fix, these bounds come from the live IANA CSV, so the fix is general: `pcapkit/vendor/reg/ethertype.py`'s `process()` collects range rows and a new `EtherType._insert_range` places each ahead of the first already-placed range that fully contains it, leaving non-overlapping pairs in CSV order. Of all 56 range tests, this pair is the only containment. `tests/const/test_const_ethertype_862_unit.py` pins the symptom, the root cause and the ordering rule. Closes [#862](https://github.com/JarryShaw/PyPCAPKit/issues/862) ([#865](https://github.com/JarryShaw/PyPCAPKit/pull/865)). +- **a breaking change to** `Method`: `__new__` called `str.__new__(cls)` with no argument, so all 40 registered members' `str` payload was empty regardless of value -- `str(Method.GET) == ''` and `Method.GET == 'GET'` was `False`; `.value` was always correct. [#869](https://github.com/JarryShaw/PyPCAPKit/pull/869) (above) had fixed the *unregistered* path only, leaving `str(Method('frob')) == 'frob'` beside `str(Method.GET) == ''`. Fixed to `str.__new__(cls, value)`, mirroring `Command.__new__`, in the generator (`pcapkit/vendor/http/method.py`) and the generated module. No sibling hand-rolled `__new__` has the defect (`OptionType`/`AppType` deliberately store a display string). Consequences, all corrections: `Method.GET == 'GET'` is `True`; `bool(Method.GET)` flips `False` to `True`; `json.dumps(Method.GET)` emits `"GET"` not `""`; `sorted()` is lexicographic rather than input-order-stable; and `{m: ... for m in Method}` gains 40 entries where it had 1, because all 40 members previously hashed as `''` and compared equal. Nothing under `pcapkit/protocols` relied on the old behaviour (`httpv1.py`/`httpv2.py` read only `.value`). Closes [#870](https://github.com/JarryShaw/PyPCAPKit/issues/870) ([#871](https://github.com/JarryShaw/PyPCAPKit/pull/871)). +- **a breaking change to** `TransportProtocol` and `AppType`: step 2 of [#860](https://github.com/JarryShaw/PyPCAPKit/issues/860), PR 2 of 2 (the 8 non-`AppType` bespoke registries were [#869](https://github.com/JarryShaw/PyPCAPKit/pull/869), above). `AppType` and its four transport subclasses move onto `EnumRegistry` and both mint sites -- `_missing_`'s 766 range branches and `get()`'s second mint (`PORT_{port}_{transport}` -> `'unknown'`) -- now build through a new `_unregistered_member` override that reconstructs `svc`/`port`/`proto`, per the owner's ruling that *"only IANA registered ones are legit values... get will not have sufficient information to create new ones"*. All 766+1 branches convert, including 8 naming a real service assigned to a whole port span (7 distinct names: `x11` twice, `active-net`, `satvid-datalnk`, `vrml-multi-use`, `ircu`, `swx`, `flex-lm`); unlike `FEATCode`'s fix ([#869](https://github.com/JarryShaw/PyPCAPKit/pull/869), above) there is no import-time self-mutation to cure by declaring members statically, so these stay deliberately unregistered lookups. `AppType` gains a working `register()`; the base's would have built a member with `svc=''`. **The breaking part**: `TransportProtocol` moves from power-of-two values (`tcp=1, udp=2, sctp=4, dccp=8`) to sequential `auto()` (`sctp=3, dccp=4`), per the owner's follow-up ruling once `|`-joined composites stopped being parsed. `AppType.get` treats whatever `int` it is given as a whole, so the renumbering changes what specific integers *mean*: on `AppType.get(80, proto=...)`, a bare `4` resolved to `SCTP.http` on `main` and now **silently** resolves to `DCCP.unknown`, because a real member sits at `4` under both numberings; a bare `8` used to mint `DCCP.PORT_80_dccp` and now raises `ValueError`. The silent reroute is disclosed here, and the composed-value case is deliberately untested, per the owner's ruling that it is "an obsoleted path from a breaking change". ([#874](https://github.com/JarryShaw/PyPCAPKit/pull/874)). +- closes the remaining scope of [#775](https://github.com/JarryShaw/PyPCAPKit/issues/775): `EtherType` (52 branches) and `Socket` (1), the last registries whose `_missing_` minted a permanent member, now return a non-registering `cls._unregistered_member(value, name)`. The issue's AST census of `extend_enum` in `_missing_`/`get` goes from 54 sites to the single `CGAType` carve-out (its mint is not an IANA-style range), from the original 1,169. `EtherType(0x0888)` used to grow `__members__` from 160 to 161 and return the identical object on a repeat; it now returns `` with `__members__` unchanged, absent from `_value2member_map_`, equal but not identical across lookups -- likewise `Socket(0x0010)`. Weaker verification, disclosed: `vendor/ipx/socket.py` regenerates byte-identically offline, but `vendor/reg/ethertype.py` needs the live IANA CSV, so it rests on a mechanical substitution matching exactly 52 lines plus a run against the committed CSV fixture. Keeping each branch's hex-suffixed name reintroduces the value-derived shape `test_const_enum_no_mint.py`'s `is_manufactured()` sweep forbids; rather than weaken the predicate, a narrow `HEX_SUFFIXED_NAME_EXEMPT_PATHS` covers exactly these two files, since the collision the check guards against cannot occur once `_unregistered_member` never registers. `pcapkit/corekit/enum.py`'s docstring now names `CGAType` alone as still minting. `tests/vendor/test_ipx_socket_unit.py` is unaffected (`EXPECTED_MISSING_NAMES` still carries `0x0010: 'Registered by Xerox_0x0010'`) ([#878](https://github.com/JarryShaw/PyPCAPKit/pull/878)). ### pcapkit.corekit #### Added -- `tests/corekit/test_fields_numbers_width_repair.py`, covering each octet boundary in its own method rather than one parametrised sweep, since the defect is a pattern and a single case would pass against a fix that special-cased the reported width. Each boundary is asserted as a pair -- the value below it, which always packed, and the value above it, which did not -- so that a width shifted by one in the other direction fails too. Also swept over all eight boundaries, pinned as the `ceil(bit_length / 8)` invariant, checked for the smallest mis-sized value being `1`, round-tripped through pack and unpack, and given controls for the bit lengths that divide by eight and for the reachability of the repair at all. The suite deliberately asserts widths and octets rather than exception types, because the exception depends on whether the mis-sized width happens to have a native `struct` code ([#599](https://github.com/JarryShaw/PyPCAPKit/issues/599)). -- `FieldBaseShortReadPaddingSideTests`, ten cases over 133 subtests covering both byte orders at 2, 4 and 8 octets and at every truncation point: the two figures [#604](https://github.com/JarryShaw/PyPCAPKit/issues/604) reports as literals, the value-preserving property stated over every width and shortfall rather than as a table, the two failure directions as inequalities, a signed field, a byte-string field, and an unpack-then-pack cycle. Eight of the ten fail on the unfixed tree. The other two must pass on both and are the guard rails -- a full read at every width and order, which may not move, and a read against an entirely empty buffer, which pads to all zeros either way and is the [#431](https://github.com/JarryShaw/PyPCAPKit/issues/431) behaviour the option and list loops depend on ([#604](https://github.com/JarryShaw/PyPCAPKit/issues/604)). +- `tests/corekit/test_fields_numbers_width_repair.py`, covering each octet boundary in its own method, since the defect is a pattern and a single case would pass against a fix that special-cased the reported width. Each boundary is asserted as a pair (the value below, which always packed, and the value above, which did not), so a width shifted by one in the other direction fails too. It asserts widths and octets rather than exception types, because the exception depends on whether the mis-sized width has a native `struct` code ([#599](https://github.com/JarryShaw/PyPCAPKit/issues/599)). +- `FieldBaseShortReadPaddingSideTests`, ten cases over 133 subtests covering both byte orders at 2, 4 and 8 octets and every truncation point, including the two figures [#604](https://github.com/JarryShaw/PyPCAPKit/issues/604) reports as literals. Eight fail on the unfixed tree; the other two are guard rails -- a full read at every width and order, which may not move, and a read against an empty buffer, which pads to zeros either way and is the [#431](https://github.com/JarryShaw/PyPCAPKit/issues/431) behaviour the option and list loops depend on ([#604](https://github.com/JarryShaw/PyPCAPKit/issues/604)). #### Changed - `Probe`, `CipherSuite` and `IntegritySuite` are `Info` subclasses rather than `typing.NamedTuple`, and no `NamedTuple` remains in the package. They are Mappings now, so `len()` and iteration yield field names rather than values. -- `ModuleDescriptor.klass` reads an already-imported module out of `sys.modules` rather than re-entering `importlib.import_module`, which matters because next layer dispatch resolves a descriptor there on a per-frame path. A registry *hit* holding a `ModuleDescriptor` is resolved once and written back, but a *miss* deliberately is not -- recording a miss in a class-level `collections.defaultdict` is the defect [#421](https://github.com/JarryShaw/PyPCAPKit/issues/421)/[#426](https://github.com/JarryShaw/PyPCAPKit/pull/426) fixed at this layer, [#425](https://github.com/JarryShaw/PyPCAPKit/issues/425)/[#428](https://github.com/JarryShaw/PyPCAPKit/pull/428) fixed for the option, chunk and block registries, and [#555](https://github.com/JarryShaw/PyPCAPKit/issues/555)/[#560](https://github.com/JarryShaw/PyPCAPKit/pull/560) fixed at the schema layer -- so every unrecognised frame resolved the same fallback descriptor again: 48 of the 52 `ModuleDescriptor.klass` resolutions an extraction of `many_interfaces.pcapng` performs, and 4 of the 7 on `ipv4.pcap`. `import_module` keeps real per-call work for a module `sys.modules` already holds, so that resolution now costs ~117 ns rather than ~436 ns and the whole miss path ~526 ns rather than ~883 ns, on CPython 3.14.7. **The scale is worth stating plainly: this is not measurable in** `extract()` **wall clock.** 48 avoided calls is ~17 us against a ~37 ms extraction, two orders of magnitude inside this host's run-to-run variance, and the "~40% of cumulative time" reading that prompted the work was an artifact of `_import_next_layer` being a recursive-descent dispatcher -- its *self* time is 0.58%, while `aenum.extend_enum` is 16.7%. What the change is taken for is its shape rather than its speed: nothing memoises the resolved class, anywhere, so `sys.modules` stays the only module cache in play and its invalidation is the interpreter's. A memo of the class would serve the pre-reload class after an `importlib.reload` forever, and an instance of it fails `isinstance` against the live one. Proposed by `@Ts-Boom` in [#563](https://github.com/JarryShaw/PyPCAPKit/pull/563), whose profiling found the miss path; the implementation differs because that one added a second, never-invalidated cache of resolved classes ([#574](https://github.com/JarryShaw/PyPCAPKit/issues/574)). -- `SeekableReader.truncate()` raises instead of returning a size, and the misspelled `writeable()` is now spelled `writable()`. Both are breaks for an external caller and neither breaks anything inside the package. `io.IOBase` documents one gate over two methods -- "If False, write() and truncate() will raise OSError" -- and this reader's `writable()` answers False, so a caller that checked it first, which is exactly what the contract invites, got a surprise either way round: `write` and `writelines` raised, and `truncate` returned its new size. What settled the question is that this is not a pure-Python nicety of `_pyio` that the accelerated path skips -- every ordinary read-only file object in CPython raises here, `open(path, 'rb').truncate()` giving `io.UnsupportedOperation: truncate` off the C `BufferedReader`, and `_pyio.BufferedReader` the same type from `_BufferedIOMixin.truncate`'s `_checkWritable()`. The refusal is `UnsupportedOperation('truncate')`, the same one `write` already raises, and no trade-off was needed between the house exception and the ABC's: pcapkit's `UnsupportedOperation` subclasses `io.UnsupportedOperation`, which subclasses `OSError`, so the in-library exception *is* the one the contract names. The resizing `truncate` used to perform is not deleted, only made private as `_truncate_buffer`. It never touched the underlying stream -- it resizes a private lookback window, which is the counter-argument the issue itself raised -- and it is the only route to the "position sits before the window" state that `seek` and the four buffered read paths must refuse, which [#643](https://github.com/JarryShaw/PyPCAPKit/issues/643) and [#644](https://github.com/JarryShaw/PyPCAPKit/issues/644) landed tests for; deleting it would have taken their mechanism with them. `writeable()`, separately, was never an override of anything: `'writable' in SeekableReader.__dict__` was False and `SeekableReader.writable is io.IOBase.writable` was True, so `io`, `shutil` and any third-party caller read the inherited value and never saw the one defined in this file. Both answered False, which is the coincidence that hid it -- there was no symptom to notice, and editing the misspelled method would silently have had no effect. The value reported is unchanged and was always honest, the reader genuinely being unable to write; only where the method was defined was wrong. Nothing in the package called either method, confirmed by grep, so the risk is entirely to external callers, which is what made this a question asked before it was acted on. The new test asserts the property rather than the behaviour -- that a `SeekableReader` raises the same exception type a read-only `io.BufferedReader` raises for the same call -- with that type captured by running the call rather than named in the test, so it tracks CPython across the 3.10--3.15 matrix instead of restating a belief about it. 31 to 34 tests and 35 to 41 subtests over `tests/corekit/test_io.py`, the module at 100% coverage before and after ([#645](https://github.com/JarryShaw/PyPCAPKit/issues/645)). -- **a breaking change to** `@final`, now enforced at runtime on `Info` and `Schema`. `info_final` and `schema_final` both end `return final(cls)`, so every finalised class already carried `typing.final`'s `__final__` marker and nothing read it: the decorator was a promise to the type checker that the interpreter was free to ignore. **`@final` alone on an `Info` or `Schema` subclass -- without `@info_final` or `@schema_final` -- now raises `InfoError` or `SchemaError` at first construction**, the class being marked final but never finalised and so carrying no generated `__init__` and nothing usable. `__init_subclass__` cannot catch that, `final` being applied after class creation, so the check is nested inside the existing one-shot `FinalisedState.NONE` branch and a finalised class pays nothing for it. **Deriving from a finalised class raises too**, from new `Info.__init_subclass__` and `Schema.__init_subclass__` hooks, where before only `EnumSchema` had one. **`SchemaError` is a `ValueError`**, so code catching `TypeError` around a subclass declaration will not see it. `@info_final @final` and `@final @info_final` both finalise silently, order being unable to matter, which is why the re-entry check keys on `__finalised__` rather than `__final__` -- only the former records *the decorator* having run, and decorators apply bottom-up, so a `__final__` test reads a class marked by `final` an instant earlier as already finalised and skips the generation. `@info_final` twice warns and hands back the finalised class unchanged. Every check reads its marker out of the class's own `__dict__`, both markers being ordinary class attributes and so inheriting, and a class that merely descends from a finalised one has not been mismarked by anybody. `pcapkit.utilities.compat` takes `final` from `typing_extensions` below 3.11 rather than below 3.8, because `typing.final` only records `__final__` from 3.11 on and the guards were otherwise a silent no-op on the 3.10 leg. `EnumSchema.__init_subclass__` calls the base hook first so a refused declaration cannot leave `__enum__` pointing at a discarded class, and `pcapkit.protocols.schema.misc.pcapng`'s `Option.__init_subclass__` is reordered to match, having otherwise let a refused `Option` subclass displace a built-in schema first. Additive in tree: of `Info`'s 488 descendants 455 carry `__final__` and none is subclassed, of `Schema`'s 445, 408 do and none is, and no class carries `__final__` without `FinalisedState.FINAL`, so the bare-`@final` guard cannot fire on the library itself ([#778](https://github.com/JarryShaw/PyPCAPKit/issues/778)). +- `ModuleDescriptor.klass` reads an already-imported module out of `sys.modules` rather than re-entering `importlib.import_module`, since next layer dispatch resolves a descriptor there on a per-frame path. A registry *hit* holding a `ModuleDescriptor` is resolved once and written back; a *miss* deliberately is not -- recording a miss in a class-level `collections.defaultdict` is the defect [#421](https://github.com/JarryShaw/PyPCAPKit/issues/421)/[#426](https://github.com/JarryShaw/PyPCAPKit/pull/426) fixed at this layer, [#425](https://github.com/JarryShaw/PyPCAPKit/issues/425)/[#428](https://github.com/JarryShaw/PyPCAPKit/pull/428) for the option, chunk and block registries, and [#555](https://github.com/JarryShaw/PyPCAPKit/issues/555)/[#560](https://github.com/JarryShaw/PyPCAPKit/pull/560) at the schema layer. So every unrecognised frame resolved the fallback descriptor again (48 of the 52 resolutions in an extraction of `many_interfaces.pcapng`); a resolution now costs ~117 ns rather than ~436 ns. **This is not measurable in** `extract()` **wall clock**: 48 avoided calls is ~17 us against a ~37 ms extraction, inside run-to-run variance, and the "~40% of cumulative time" reading that prompted the work was an artifact of `_import_next_layer` being a recursive dispatcher. The change is taken for its shape: `sys.modules` stays the only module cache, so its invalidation is the interpreter's, whereas a memo of the class would serve the pre-reload class after an `importlib.reload` forever and fail `isinstance` against the live one. Proposed by `@Ts-Boom` in [#563](https://github.com/JarryShaw/PyPCAPKit/pull/563), whose profiling found the miss path; the implementation differs because that one added a second, never-invalidated cache of resolved classes ([#574](https://github.com/JarryShaw/PyPCAPKit/issues/574)). +- `SeekableReader.truncate()` raises instead of returning a size, and the misspelled `writeable()` is now `writable()`. Both break external callers; nothing inside the package calls either. `io.IOBase` documents one gate over two methods -- "If False, write() and truncate() will raise OSError" -- and this reader's `writable()` answers False, yet `truncate` returned its new size. Every ordinary read-only file object in CPython raises here (`open(path, 'rb').truncate()` gives `io.UnsupportedOperation`), so this is not a pure-Python nicety that the C path skips. The refusal is `UnsupportedOperation('truncate')`, the same one `write` raises; pcapkit's subclasses `io.UnsupportedOperation`, so the house exception *is* the one the contract names. The old resizing is not deleted, only made private as `_truncate_buffer`: it resizes a private lookback window, not the stream (the issue's own counter-argument), and it is the only route to the "position sits before the window" state that `seek` and the four buffered read paths must refuse, for which [#643](https://github.com/JarryShaw/PyPCAPKit/issues/643) and [#644](https://github.com/JarryShaw/PyPCAPKit/issues/644) landed tests. `writeable()` was never an override of anything -- `SeekableReader.writable is io.IOBase.writable` was True -- so `io`, `shutil` and third parties never saw it; both answered False, which hid it, and the value reported is unchanged. The new test asserts the property, that a `SeekableReader` raises the same exception type as a read-only `io.BufferedReader`, capturing that type by running the call so it tracks CPython across 3.10--3.15. ([#645](https://github.com/JarryShaw/PyPCAPKit/issues/645)). +- **a breaking change to** `@final`, now enforced at runtime on `Info` and `Schema`. `info_final` and `schema_final` both end `return final(cls)`, so every finalised class carried `typing.final`'s `__final__` marker and nothing read it. **`@final` alone on an `Info` or `Schema` subclass -- without `@info_final` or `@schema_final` -- now raises `InfoError` or `SchemaError` at first construction**, the class being marked final but never finalised and so carrying no generated `__init__`. `__init_subclass__` cannot catch that, since `final` applies after class creation, so the check sits in the existing one-shot `FinalisedState.NONE` branch and a finalised class pays nothing. **Deriving from a finalised class raises too**, from new `Info.__init_subclass__` and `Schema.__init_subclass__` hooks (before, only `EnumSchema` had one). **`SchemaError` is a `ValueError`**, so code catching `TypeError` around a subclass declaration will not see it. `@info_final @final` and `@final @info_final` both finalise silently; the re-entry check keys on `__finalised__` rather than `__final__`, because only the former records *the decorator* having run and decorators apply bottom-up, so a `__final__` test would read a class marked an instant earlier as already finalised. `@info_final` twice warns and returns the class unchanged. Every check reads its marker from the class's own `__dict__`, since the markers are ordinary inheriting attributes. `pcapkit.utilities.compat` takes `final` from `typing_extensions` below 3.11 rather than 3.8, because `typing.final` only records `__final__` from 3.11 and the guards were otherwise a silent no-op on the 3.10 leg. `EnumSchema.__init_subclass__` calls the base hook first so a refused declaration cannot leave `__enum__` pointing at a discarded class, and `pcapkit.protocols.schema.misc.pcapng`'s `Option.__init_subclass__` is reordered to match. Additive in tree: no `Info` or `Schema` class is subclassed after finalising or carries `__final__` without `FinalisedState.FINAL`, so the guard cannot fire on the library itself ([#778](https://github.com/JarryShaw/PyPCAPKit/issues/778)). #### Fixed - stdlib exceptions leaking out where the library's own were promised: a malformed IP field value raised a bare `ValueError` instead of `FieldValueError` ([#465](https://github.com/JarryShaw/PyPCAPKit/issues/465)); a `bool` address was silently packed as `0.0.0.1` or `0.0.0.0`, `bool` being an `int` subclass ([#491](https://github.com/JarryShaw/PyPCAPKit/issues/491), [#500](https://github.com/JarryShaw/PyPCAPKit/pull/500)); and `@prepare` discarded extra arguments silently and treated a *declared* zero length as end of stream, which is now distinguished from a genuinely exhausted one and raises `StreamEOFError` ([#454](https://github.com/JarryShaw/PyPCAPKit/issues/454), [#458](https://github.com/JarryShaw/PyPCAPKit/issues/458)). -- `FieldBase.unpack` zero-padded straight up to a field's declared `length` with `rjust()`, regardless of how little data `buffer` actually held; a ~40-octet PCAP-NG Decryption Secrets Block with a bogus inner length was enough to force a multi-gigabyte allocation, since `length` is frequently wire-derived and so attacker-controlled. A declared length past 262144 octets -- libpcap's own `MAXIMUM_SNAPLEN`, and this package's own default `snaplen` -- that the buffer cannot back now raises `FieldValueError` instead of padding for it; the option and list loops' own tolerance for a short read past a truncated area ([#431](https://github.com/JarryShaw/PyPCAPKit/issues/431)) is far under that ceiling and is untouched ([#554](https://github.com/JarryShaw/PyPCAPKit/issues/554)). -- that ceiling bounds one field, and a packet holds many, so the *sum* of a parse's zero padding was still unbounded: a declared length just under 262144 octets is honoured however often it is declared. 200 minimal PCAP-NG Decryption Secrets Blocks -- 4,800 wire octets, each declaring a `secrets_length` of 262,142 against two supplied octets -- retained 50.0 MiB, an amplification of 10,922x per block, with the per-field guard never firing because every individual field was within it. `FieldBase.unpack` now keeps a running per-context ledger of octets supplied against octets synthesised, and raises `FieldValueError` once the total of shortfalls *past 65,536 octets* passes `262144 + 16 * supplied`. The same 200 blocks now retain 0.2 MiB, 54.6x rather than 10,922x. A shortfall of 65,536 octets or fewer is padded unconditionally and charged to nothing, and that band is the load-bearing part rather than a concession. 65,536 is the whole span of a 16-bit wire length -- how an IP header, an IPv6 payload, a TCP or IPv4 option and a PCAP-NG option all declare their size -- so no shortfall a capture cut short by its snapshot length can produce is subject to the budget at all, on the first frame or the ten-thousandth. Without that band, a running budget alone made the *same* legitimate 54-octet frame declaring an IPv4 total length of 65,535 parse to one result on 37 of 40 identical calls and to another on calls 26, 33 and 39, since whether it fit depended on what had been parsed before it; a guard whose answer moves with history is worse than the amplification it bounds. The worst legitimate single shortfall measured anywhere was 65,495 octets, from exactly that frame, and the worst from a truncated PCAP-NG option was 64,750. **What this does not close, measured rather than assumed**: the 16-bit band is deliberately untouched, so a length declared by a 16-bit wire field can still be repeated without limit. A crafted 80,048-octet PCAP-NG file of 2,000 Enhanced Packet Blocks, each carrying one option declaring 65,535 octets against four real ones, parses end to end through `Extractor(store=True)` and retains 125.00 MiB of synthesised zeros for 216.88 MiB of RSS -- 1,637x its own size, linear in the block count -- both before this change and after it. That is not an oversight in the bound: parsing a bare 40-octet IPv4 header declaring a total length of 65,535, which is what a legitimate capture of offload-sized segments truncated to its snapshot length looks like, amplifies by **the same 1,637x**. The two are not separable by any budget at this layer. Separating them needs the frame's own `incl_len`/`orig_len` -- a crafted block claims nothing was truncated while declaring more than it holds, and a snapshot-truncated frame says so on the wire -- which is knowable at the protocol layer and not here. Nor is the budget scoped per file: nothing in the package resets the ledger, so as shipped the bound is over everything a context has parsed rather than over one `Extractor` run. That is still proportionate to the octets that context was genuinely given, and tightening it is a one-line change at whichever layer owns a run. Verified against every capture in `examples/captures/`, every one of them truncated at some 8,700 offsets, 190 snapshot-length rewrites, and the synthetic offload shapes above. The comparison is of the instrumented padding and supplied-octet tallies *and* of each parse's outcome -- frame count, or exception type and message -- so a cut that changed from parsing to crashing would show rather than be swallowed; every one is identical to before ([#573](https://github.com/JarryShaw/PyPCAPKit/issues/573)). -- which exception a malformed TCP SACK option raised depended on unrelated process state: a clean interpreter raised `ProtocolError` as documented, but a process that had already popped `pcapkit.corekit.fields.misc` from `sys.modules` -- which the `#439` ABC-cache regression tests do in every case's `setUp`/`tearDown` -- raised `FieldValueError` instead, from a different layer entirely, before the documented check was even reached ([#525](https://github.com/JarryShaw/PyPCAPKit/issues/525)). The cause was `ListField.unpack` resolving `SchemaField` through a function-local import re-run on every call; a module popped and reimported mid-process comes back as a second, distinct class, so `isinstance` against it silently misclassified the field and billed each item by its declared length instead of by what it actually consumed. **Any caller relying on the previously-observed** `FieldValueError` **for this case now gets** `ProtocolError` **instead, deterministically**, matching the method's own docstring. Fixed by importing at module level instead. -- a `NumberField` whose `length` was a callable could not pack or parse at any width `struct` has a native integer code for. `length` is a placeholder of `-1` until the callable is resolved, `-1` has no native code, and the template builder raised `_need_process` for it and never put it back -- so the flag was a latch. Resolving the real width rebuilt the template and left the latch set, and `pre_process` then handed `bytes` to a template that had become `>Q`, raising `struct.error: required argument is not an integer`; parsing failed in the mirror direction, calling `int.from_bytes` on the integer that `struct.unpack` had already produced. The flag is now recomputed from the width actually in force rather than only ever raised, which is also what keeps a callable resolving to a width with no native code -- 3 octets, say -- byte-packed as it must be. **All four native widths were affected, not only the 8 that was reported**: the latch has nothing to do with the width it latches into, so 1, 2 and 4 failed identically, on `NumberField` and on `EnumField`, both of which leave `__template__` unset. This is what made every extended 8-octet MPTCP DSS form unbuildable, since those widths are chosen at runtime from the DSS flags and so must come from a callable; [#585](https://github.com/JarryShaw/PyPCAPKit/pull/585) worked around it in the TCP schema alone, leaving every other caller exposed ([#591](https://github.com/JarryShaw/PyPCAPKit/issues/591)). -- `NumberField.pre_process` sized a value with floor division dressed up as a ceiling. When a field is packed while its `length` is still the `-1` placeholder, the width is derived from the value, and it was derived with `math.ceil(value.bit_length() // 8)`. `math.ceil` of an integer is that integer, so the `//` had already floored the quotient and the outer call did nothing at all; the expression was plain floor division and the width came out one octet short. `256` was sized at one octet, `65536` at two, `16777216` at three, and both `int.to_bytes` and `struct.pack` refuse a value that does not fit the width they are given. **The reach is wider than "just past a boundary"**: floor division is wrong for every bit length that is not an exact multiple of eight, so `1` -- bit length 1, floored to *zero* octets -- failed too, and every value from 1 to 127 with it. Now written as the ceiling it was meant to be, matching the `math.ceil(n / 8)` idiom used elsewhere in the package. The repair is reached only by packing a field the caller never resolved, since a schema resolves every field before packing it and `__call__` installs a real width; that narrowness is why the defect survived the suite added for [#591](https://github.com/JarryShaw/PyPCAPKit/issues/591), whose five repair-path values -- `0xFF`, `0xFFFF`, `0xFFFFFFFF`, `0xFFFFFFFFFFFFFFFF` and `0x800001` -- have bit lengths of 8, 16, 32, 64 and 24 and so sat exactly where floor division and the ceiling agree. [#591](https://github.com/JarryShaw/PyPCAPKit/issues/591)'s own fix neither caused nor masked this, but it did change what the failure looks like: with `_need_process` now recomputed from the width in force, a mis-sized 1, 2 or 4 octets surfaces from `struct.pack` as `'B' format requires 0 <= number <= 255` where it used to surface from `int.to_bytes` as `OverflowError`, which is why the exception named in the report is no longer the one a mis-sized octet boundary raises. Two things on this path are deliberately left alone, both independent of the arithmetic: a signed field is sized without room for its sign bit, so an unresolved signed field still cannot pack `128`; and an unresolved field's bit mask is `-1`, which makes the masking and the sign remap above no-ops. The identical `math.ceil(x.bit_length() // 8)` expression also survives at the two ILNP nonce option builders in `hopopt.py` and `ipv6_opts.py`, which are a separate change ([#599](https://github.com/JarryShaw/PyPCAPKit/issues/599)). -- `FieldBase.unpack` padded a short read on the wrong side. When a field's buffer falls short of its declared length -- the deliberate accommodation that lets a snapshot-truncated capture parse ([#431](https://github.com/JarryShaw/PyPCAPKit/issues/431)) -- the shortfall was zero-filled with `rjust()`, which places the zeros at the *front*. That asserts the octets never read were the leading ones, and a short read has lost the trailing ones: the buffer ran out. It is now `ljust()`, which is correct for **both** byte orders rather than only for big-endian fields, so the correction is not byte-order-conditional. The two directions fail differently and only one was loud. Measured: one octet of a four-octet little-endian `120` read as `2013265920`, inflated by `2 ** 24`; three octets of a four-octet big-endian `0x01020304` read as `0x10203`, scaled *down* by 256. The second is the dangerous one -- a value smaller than the truth passes a sanity check, where an inflated one overruns -- which is why the big-endian half went unnoticed. A full read is untouched at every width and order, since the padding is only ever consulted when the buffer falls short ([#604](https://github.com/JarryShaw/PyPCAPKit/issues/604)). -- `SeekableReader.truncate` put its padding where the reader's own bookkeeping says the content is, and `read` then returned one octet fewer than the buffer held. Same family as [#604](https://github.com/JarryShaw/PyPCAPKit/issues/604), but two defects rather than one, and the padding side is only half of it. The buffer keeps its content at `[0:_buffer_cur]` with unwritten padding behind it, so a reduction has to keep the octets it has and an extension has to append at the *tail* -- the latter is what `io.IOBase.truncate` means by "the contents of the new file area", the area past the old end. It did neither: `temp[-size:]` sliced the buffer rather than the content, so it kept the trailing padding and discarded the octets actually read, and `temp.rjust(size)` prefixed the new zeros, displacing the content past where `_buffer_set` and `_buffer_cur` address it. Separately, `read` capped a buffered read at `min(size, self._buffer_cur - 1)`, a count less one measured from the start of the buffer rather than the run remaining from the position being read from -- one octet short at the start of the buffer, and reaching past the content into the padding anywhere further in. The shortfall was then made up from the stream, *past* the octet that had been skipped, which both dropped that octet and left the return short. The reported symptom needed both: `read(4)`, `truncate(8)`, `seek(0)`, `read(8)` over `b'abcde'` returned `b'\x00\x00\x00e'`, four octets of an eight octet request with three of them padding, and returns `b'abcde'` now -- five being the whole of what a five octet stream can answer with. Three more of the method's contract were wrong and are fixed with it: the position was reset to the start of the buffer rather than left alone, so a read after a truncation resumed from the wrong octet; an omitted `size` resized to `0` rather than to the current position; and `_buffer_cur` was left addressing octets a reduced buffer no longer had, so the next read raised `ValueError: memoryview assignment: lvalue and rvalue have different structures` from `_write_buffer` rather than returning anything. `truncate(0)` raised the same `ValueError` by a second route, and `_write_buffer` is fixed with it: `buf[-self._buffer_size:]` is `buf[-0:]` for a buffer of no size at all -- the whole of the octets just read rather than none of them -- so it is now counted from the front. That state is reachable only through `truncate`, since the constructor refuses a non-positive `buffer_size`. A reduction also advances `_buffer_set` past the octets it drops, keeping `_buffer_set + _buffer_cur` equal to how far the stream has been consumed, which `seek` reads as its licence to fetch more. Clamping `_buffer_cur` alone made that sum *under*-report the stream, and the next forward `seek` then spliced in octets from the wrong absolute offset and said nothing: `read(8)`, `truncate(3)`, `seek(6)`, `read(1)` over `b'abcdefghijklmnop'` returned `b'l'` where `b'g'` is the octet at offset 6 -- worse than the `ValueError` it replaced, being silent. All three came out of fuzzing random operation sequences against the buffer's own invariants, none from reading the code: of 3000 rounds, 2372 left the bookkeeping inconsistent before and 683 raised an undocumented exception, and none of either do now. Latent in this library rather than live -- nothing here calls `truncate`, confirmed by grep, though it is public on a public class -- and invisible to the existing tests, which asserted the return value and never the content ([#622](https://github.com/JarryShaw/PyPCAPKit/issues/622)). -- the 16-bit band the [#573](https://github.com/JarryShaw/PyPCAPKit/issues/573) entry above records as deliberately left open. `FieldBase.unpack` pads a shortfall of 65,536 octets or fewer unconditionally and charges it to nothing, which is load-bearing for a capture cut short by its snapshot length, so a length declared by a 16-bit wire field could still be repeated without limit. The crafted 80,048-octet PCAP-NG of 2,000 Enhanced Packet Blocks that entry describes, each carrying one option declaring 65,535 octets against none present, synthesised 131,070,000 octets of zeros -- 1637.393x its own size, linear in the block count and so unbounded in the input -- and now synthesises none, with all 2,000 frames and all 2,000 options still parsed. The bound is taken one layer up, where the information the field layer lacks already exists: a PCAP-NG block's Block Total Length is authoritative and cross-checked against its own trailing copy, so the option area is that length less the fixed fields, `captured_len` and `captured_len`'s padding, and an option declaring more payload than that area has left is malformed however complete the file behind it is. A snapshot-truncated capture says so through `captured_len` instead and leaves its options whole, so it never trips this -- which is exactly what the [#571](https://github.com/JarryShaw/PyPCAPKit/pull/571) `len(buffer) < length` rejection could not distinguish, and what it was declined for. A new `bounded_option` clamps the payload to the octets the area has left at that field and warns with `SchemaWarning`; it is applied to all fifteen variable-width option and record payloads in the PCAP-NG schema, and deliberately not to the block-level payload fields, which are the 32-bit band the running ledger already budgets. A second, `bounded_area`, clamps a packet block's option area to the octets the block itself holds, less the trailing Block Total Length: the area is otherwise sized from a declared length that nothing checks against the file -- `BlockType.post_process` compares it only with its own trailing copy -- so a block declaring 1,000,000 octets while holding 36 sized its area at 999,964, an option inside it declaring 65,535 was under that and went unclamped, and 65,535 octets of zeros were synthesised from 36 regardless, 1,820x with no warning at all. That came out of the change's cross-review rather than from writing it, and it is a no-op on a well-formed block, where the octets left of the block are exactly the area plus the trailing length's four. The five non-packet blocks' option areas keep the framing assumption, each computing its span with a different offset, and the general fix for a declared length reaching a read at all is [#678](https://github.com/JarryShaw/PyPCAPKit/issues/678). Clamping rather than refusing is what keeps the [#431](https://github.com/JarryShaw/PyPCAPKit/issues/431) accommodation, since a PCAP-NG block read has no catch point above `FieldBase.unpack` and one refusal would abort a whole extraction rather than one block; and the clamp reads only the block's own declared framing, never a running total, so byte-identical input answers identically whatever preceded it. The skip for a remainder already past zero is load-bearing on the unpacking path, where a ten-octet area leaves `__length__` at -2 before a payload sizes itself and clamping to it would hand the field a `'-2s'` struct template; it is *not* what protects the packing path, since `BytesField` and `StringField` both repair a negative width to `len(value)` in `pre_process`. Zero clamps fired across all six PCAP-NG sample captures, 338 options between them, and 501 truncation levels of `dhcp.pcapng` swept an octet at a time gave byte-identical results including the failures -- 497 of which are the uncaught `ValueError` that [#678](https://github.com/JarryShaw/PyPCAPKit/issues/678) records, two a `struct.error`, and two of which parse. The bound is pinned as a property rather than as one input: every combination of one to three options against nine declared lengths asserts that the octets a block's options report holding never exceed the area, and the amplification ratio is asserted flat across 1, 8, 64 and 512 blocks, which is what separates a bound from a smaller constant. The worst ratio reachable over five adversarial shapes afterwards is 0.862x, against 1,637x, 960x and 224x for the same three before ([#594](https://github.com/JarryShaw/PyPCAPKit/issues/594)). -- **a breaking change to** `FieldBase.length`: it called `struct.calcsize` on a template built from a negative resolved field length -- e.g. `'-5s'`, from a `length=` callback whose running counter had gone negative once overdrawn -- which raised a bare `struct.error`, uncatchable by ordinary caller code. The one chokepoint every affected schema module shares now catches that and raises `ProtocolError` instead, covering `application/{ftp,httpv1,httpv2,ngap}.py`, `internet/{hip,ipv6_route,mh}.py`, `link/ethernet.py`, `misc/pcapng.py` and `transport/sctp.py` with no other file changed; a non-negative length still returns the same value, so well-formed input is unaffected. This closes the follow-on the `httpv2` length-guard entry above filed as out of scope rather than fixing directly ([#805](https://github.com/JarryShaw/PyPCAPKit/issues/805)). -- **a breaking change to** `EnumRegistry.register()`: it now raises `ValueError` naming the existing member and pointing at `register_alias()`, where it used to silently alias an already-registered value under the caller's new name -- `aenum` treats a taken value as an alias request, so `register(existing_value, 'TOTALLY_NEW_NAME')` returned the existing member unchanged, minted nothing and raised nothing, contradicting the method's own docstring. `pcapkit.corekit.enums.EnumRegistry` is a new mixin implementing `get`, `get_all`, `register`, `register_alias`, `register_aliases` and `_unregistered_member` in one place, per the maintainer's ruling on [#842](https://github.com/JarryShaw/PyPCAPKit/issues/842); it converts the first six registries with no bespoke `__new__` -- `ipv6.extension_header.ExtensionHeader`, `tcp.flags.Flags` and four `mh.*_flag` modules -- and the one base serves `IntEnum`, `IntFlag` and `StrEnum` alike. The new guard reaches only those six: the other 105 const registries still carry the old generator's bare `extend_enum(cls, name, value)`, so `pcapkit.const` ships two contradictory `register` contracts until [#858](https://github.com/JarryShaw/PyPCAPKit/pull/858) (below) migrates the rest. Riding along: the six also gain the generated template's `KeyError`-catching on a missing name, which the hand-copied `get` never had, so e.g. `Flags.get('NOPE', 0)` returns `0` now rather than raising -- a widening, not a regression, though undisclosed until this PR; and a key that is neither `int` nor `str` used to raise `TypeError` on the five `IntFlag` registries' hand-copied `get` -- e.g. `Flags.get(3.5)` -- since `cls[key]` is a containment test on a `Flag` subclass rather than a name lookup there; all five now raise `ValueError` instead, tried as a value by the base. `ExtensionHeader`, the sixth and the only `IntEnum` among them, already raised `KeyError` on the same input before this PR -- the shift [#858](https://github.com/JarryShaw/PyPCAPKit/pull/858)'s entry attributes to the other 105 -- so for it this is not a new divergence at all, only the same one arriving six registries early ([#855](https://github.com/JarryShaw/PyPCAPKit/pull/855)). -- **a breaking change to** `EnumRegistry.register()`, extended: the 105 const registries still on the generated template's bare `extend_enum(cls, name, value)` now inherit the guard [#855](https://github.com/JarryShaw/PyPCAPKit/pull/855) (above) added, so e.g. `TransType.register(6, 'TOTALLY_NEW_NAME')` now raises `ValueError`, naming the value and the member that already holds it and pointing at `register_alias()`, where it used to mint nothing and raise nothing. `pcapkit/vendor/default.py`'s generated template now emits `class {NAME}(EnumRegistry, IntEnum)` and drops its own `get`/`register`/`_unregistered_member` entirely, so every const-enum class it produces inherits them from the shared base instead. Census across the 121 const modules holding 127 enum classes: 6 classes, one module each, already used `EnumRegistry` (tier 1, [#855](https://github.com/JarryShaw/PyPCAPKit/pull/855)); of the other 115 modules, 105 share the generated template byte-for-byte and are converted here, and the remaining 10 -- three of which define more than one class -- keep their own bespoke `__new__` and are left alone: `ftp/command` (4 classes), `ftp/return_code` (3), `http/method`, `http/status_code`, `pcapng/option_type`, `reg/apptype/apptype.py` (2, `AppType` and `TransportProtocol`) and its four transport subclasses `dccp.DCCP`/`sctp.SCTP`/`tcp.TCP`/`udp.UDP`. That brings the total inheriting `EnumRegistry` to **111 of the 127 classes**, up from 6. `get()`'s dispatch matrix shifts only for a key that is neither `int` nor `str`: `KeyError` before (tried as a name), `ValueError` now (tried as a value) -- a key type nothing in the library passes. Also renames `pcapkit/corekit/enums.py` to **`enum.py`** (no `-s`), the maintainer's ruling. Five of the 105 hand-edited files were independently regenerated live and came back byte-identical ([#858](https://github.com/JarryShaw/PyPCAPKit/pull/858)). -- `EnumRegistry.get()`'s `str`-key branch tried only a name lookup (`cls._member_map_[key]`) and, on a miss, either raised or returned `cls(default)` -- it never tried the *value* path at all, so on a `StrEnum` registry a key that resolved fine through the constructor (`_Str('known-value')`) raised `KeyError` through `get()` (`_Str.get('known-value')`), and a supplied `default` compounded it by overriding a good value match the code never attempted. Fixed by falling back to a plain `_value2member_map_` lookup once the name lookup misses and before consulting `default`. Deliberately not `cls(key)`: `FEATCode._missing_` (`pcapkit/const/ftp/command.py`) mints directly via `extend_enum` for any unrecognised string, so routing the fallback through the constructor would let a failed name lookup mint a permanent member the moment a registry like it converts onto this base -- a dict lookup never reaches `_missing_` and so cannot mint, on any registry. A name still wins when a string is both a name and a different member's value, matching the existing behaviour for names that already resolve. Step 1 of [#860](https://github.com/JarryShaw/PyPCAPKit/issues/860) only; step 2, converting the bespoke registries themselves, follows separately ([#869](https://github.com/JarryShaw/PyPCAPKit/pull/869), [#874](https://github.com/JarryShaw/PyPCAPKit/pull/874), below). Not touched: the `IntEnum`/`IntFlag` non-`str` path, measured identical before and after on both `TransType` and `HandoverACKFlag`; nothing under `pcapkit/const/` or `pcapkit/vendor/`. Widens rather than breaks -- a call that raised `KeyError` before now resolves, and no caller that got a correct result before gets a different one now ([#863](https://github.com/JarryShaw/PyPCAPKit/pull/863)). -- **a breaking change to** `EnumRegistry.get()`: its docstring claimed "It never mints", but both of its `cls(default)` call sites reached `_missing_` for a `default` that fell inside a still-minting registry's own range, growing the registry as a side effect of resolving `default` rather than `key`. Per the owner's ruling (*"Only register can mint. get should not mint unless it falls through the _missing_'s minted ranges"*, endorsing resolving `default` through a plain `_value2member_map_` lookup only), both sites are replaced; `key` resolution is unchanged, so `EtherType.get(0x0888)` still mints `Xyplex_0x0888` through its own `_missing_` exactly as [#775](https://github.com/JarryShaw/PyPCAPKit/issues/775)'s deliberate exception permits, landing in both lookup tables the way `register` would. A `default` naming no registered member now falls through to the same lookup error `key` itself would have raised, rather than a fresh error about the default -- the accepted cost of the ruling. The docstring's "It never mints" is now qualified to `default` only, since `key` may still mint on the three still-minting registries. New `GetDefaultNoMintTests` pin the issue's repro on the real `EtherType` registry with member count asserted unchanged, the preserved key-path mint as a no-change guard, and default resolution on both an `int` and a `str` registry. Four pre-existing assertions in `tests/const/test_const_enum_get.py` and `tests/const/test_const_enum_builtin_parity.py` that encoded the old contract (an unregistered default failing with its own error, or resolving into a declared-but-unassigned range) are retargeted rather than weakened. A fifth case -- `Hardware.get('Definitely-Not-A-Member', 40)`, where `40` sits in `Hardware`'s declared-but-unassigned range and now raises `KeyError` naming the original key rather than resolving -- was deferred a round because its test file was contended with [#865](https://github.com/JarryShaw/PyPCAPKit/pull/865), then applied once [#865](https://github.com/JarryShaw/PyPCAPKit/pull/865) (above) merged, re-measured with `Hardware`'s member count unchanged at 42 before and after. Closes [#864](https://github.com/JarryShaw/PyPCAPKit/issues/864) ([#868](https://github.com/JarryShaw/PyPCAPKit/pull/868)). +- `FieldBase.unpack` zero-padded up to a field's declared `length` with `rjust()` however little data `buffer` held; a ~40-octet PCAP-NG Decryption Secrets Block with a bogus inner length forced a multi-gigabyte allocation, since `length` is often wire-derived. A declared length past 262144 octets -- libpcap's `MAXIMUM_SNAPLEN` and this package's default `snaplen` -- that the buffer cannot back now raises `FieldValueError`; the option and list loops' tolerance for a short read past a truncated area ([#431](https://github.com/JarryShaw/PyPCAPKit/issues/431)) is far under that ceiling and untouched ([#554](https://github.com/JarryShaw/PyPCAPKit/issues/554)). +- that ceiling bounds one field, and a packet holds many, so the *sum* of a parse's zero padding was still unbounded: 200 minimal PCAP-NG Decryption Secrets Blocks (4,800 wire octets, each declaring `secrets_length` 262,142 against two supplied octets) retained 50.0 MiB, 10,922x per block, with the per-field guard never firing. `FieldBase.unpack` now keeps a per-context ledger of octets supplied against octets synthesised, and raises `FieldValueError` once the total of shortfalls *past 65,536 octets* passes `262144 + 16 * supplied`. The same 200 blocks retain 0.2 MiB (54.6x). A shortfall of 65,536 octets or fewer is padded unconditionally and charged to nothing, and that band is load-bearing. 65,536 is the span of a 16-bit wire length (IP header, IPv6 payload, TCP/IPv4 option, PCAP-NG option), so no shortfall from a snapshot-truncated capture is subject to the budget. Without the band, a running budget alone made the *same* legitimate 54-octet frame declaring an IPv4 total length of 65,535 parse one way on 37 of 40 identical calls and another on the rest, depending on what had been parsed before; a guard whose answer moves with history is worse than the amplification it bounds. **What this does not close**: a length declared by a 16-bit field can still be repeated without limit. A crafted 80,048-octet PCAP-NG file of 2,000 Enhanced Packet Blocks, each with one option declaring 65,535 octets against four real ones, retains 125.00 MiB of zeros (1,637x its size), before and after. That is not an oversight: a bare 40-octet IPv4 header declaring total length 65,535, which is what a legitimate capture of offload-sized segments truncated to its snapshot length looks like, amplifies by **the same 1,637x**, and no budget at this layer separates them. Separating them needs the frame's own `incl_len`/`orig_len`, knowable at the protocol layer and not here. Nor is the budget per file: nothing resets the ledger, so it bounds everything a context has parsed, which is still proportionate to the octets that context was given; tightening it is a one-line change at whichever layer owns a run. Padding tallies and each parse's outcome are identical to before across every capture in `examples/captures/` truncated at some 8,700 offsets ([#573](https://github.com/JarryShaw/PyPCAPKit/issues/573)). +- which exception a malformed TCP SACK option raised depended on unrelated process state: a clean interpreter raised `ProtocolError` as documented, but a process that had popped `pcapkit.corekit.fields.misc` from `sys.modules` (as the `#439` ABC-cache regression tests do in every `setUp`/`tearDown`) raised `FieldValueError` from a different layer, before the documented check ([#525](https://github.com/JarryShaw/PyPCAPKit/issues/525)). `ListField.unpack` resolved `SchemaField` through a function-local import re-run on every call; a module reimported mid-process is a second, distinct class, so `isinstance` misclassified the field and billed each item by its declared length instead of what it consumed. **Any caller relying on the previously-observed** `FieldValueError` **now gets** `ProtocolError`, deterministically, matching the docstring. Fixed by importing at module level. +- a `NumberField` whose `length` was a callable could not pack or parse at any width `struct` has a native integer code for. `length` is `-1` until the callable resolves, `-1` has no native code, and the template builder raised `_need_process` for it and never cleared it -- a latch. Resolving the real width rebuilt the template but left the latch set, so `pre_process` handed `bytes` to a template that had become `>Q` (`struct.error: required argument is not an integer`), and parsing failed in mirror by calling `int.from_bytes` on an integer `struct.unpack` had produced. The flag is now recomputed from the width in force, which also keeps a callable resolving to a width with no native code (3 octets) byte-packed. **All four native widths were affected, not only the 8 reported**, on `NumberField` and `EnumField` alike. This made every extended 8-octet MPTCP DSS form unbuildable, those widths being chosen at runtime from the DSS flags; [#585](https://github.com/JarryShaw/PyPCAPKit/pull/585) worked around it in the TCP schema alone ([#591](https://github.com/JarryShaw/PyPCAPKit/issues/591)). +- `NumberField.pre_process` sized a value with floor division dressed as a ceiling. When a field is packed while `length` is still the `-1` placeholder, the width came from `math.ceil(value.bit_length() // 8)`; the `//` had already floored, so `math.ceil` did nothing and the width was one octet short: `256` at one octet, `65536` at two, `16777216` at three. **The reach is wider than "just past a boundary"**: every bit length that is not a multiple of eight is wrong, so `1` (bit length 1, floored to *zero* octets) and every value from 1 to 127 failed too. It is now the ceiling it was meant to be, matching the `math.ceil(n / 8)` idiom used elsewhere. The repair is reached only by packing a field the caller never resolved (a schema resolves every field first), which is why it survived [#591](https://github.com/JarryShaw/PyPCAPKit/issues/591)'s suite: its five repair-path values (`0xFF`, `0xFFFF`, `0xFFFFFFFF`, `0xFFFFFFFFFFFFFFFF`, `0x800001`) have bit lengths of 8, 16, 32, 64 and 24, where floor and ceiling agree. [#591](https://github.com/JarryShaw/PyPCAPKit/issues/591)'s fix neither caused nor masked this, but it changed the failure: a mis-sized 1, 2 or 4 octets now surfaces from `struct.pack` as `'B' format requires 0 <= number <= 255` rather than `OverflowError` from `int.to_bytes`. Deliberately left alone: a signed field is sized without room for its sign bit, so an unresolved signed field still cannot pack `128`; and an unresolved field's bit mask is `-1`, making the masking and sign remap no-ops. The same `math.ceil(x.bit_length() // 8)` survives at the two ILNP nonce option builders in `hopopt.py` and `ipv6_opts.py`, a separate change ([#599](https://github.com/JarryShaw/PyPCAPKit/issues/599)). +- `FieldBase.unpack` padded a short read on the wrong side. When a buffer falls short of the declared length -- the accommodation that lets a snapshot-truncated capture parse ([#431](https://github.com/JarryShaw/PyPCAPKit/issues/431)) -- the shortfall was zero-filled with `rjust()`, placing the zeros at the *front*, as if the octets never read were the leading ones; a short read has lost the trailing ones. It is now `ljust()`, correct for **both** byte orders. The two directions failed differently and only one was loud: one octet of a four-octet little-endian `120` read as `2013265920` (inflated by `2 ** 24`), while three octets of a four-octet big-endian `0x01020304` read as `0x10203` (scaled *down* by 256). The second is the dangerous one, since a value smaller than the truth passes a sanity check where an inflated one overruns, which is why the big-endian half went unnoticed. A full read is untouched ([#604](https://github.com/JarryShaw/PyPCAPKit/issues/604)). +- `SeekableReader.truncate` put its padding where the reader's bookkeeping says the content is, and `read` then returned one octet fewer than the buffer held. Same family as [#604](https://github.com/JarryShaw/PyPCAPKit/issues/604), but two defects. The buffer keeps content at `[0:_buffer_cur]` with padding behind it, so a reduction must keep the octets it has and an extension must append at the *tail*; `temp[-size:]` sliced the buffer rather than the content, keeping trailing padding and discarding octets read, and `temp.rjust(size)` prefixed the zeros. Separately, `read` capped at `min(size, self._buffer_cur - 1)`, one short at the start and reaching into padding further in, and made up the shortfall from the stream *past* the skipped octet. The reported symptom needed both: `read(4)`, `truncate(8)`, `seek(0)`, `read(8)` over `b'abcde'` returned `b'\x00\x00\x00e'` and now returns `b'abcde'`. Three more contract faults are fixed with it: the position was reset to the buffer start rather than left alone; an omitted `size` resized to `0` rather than the current position; and `_buffer_cur` was left addressing octets a reduced buffer no longer had, so the next read raised `ValueError: memoryview assignment: lvalue and rvalue have different structures` from `_write_buffer`. `truncate(0)` raised the same by a second route (`buf[-self._buffer_size:]` is `buf[-0:]`, the whole buffer), so `_write_buffer` now counts from the front; that state is reachable only through `truncate`. A reduction also advances `_buffer_set` past the dropped octets, keeping `_buffer_set + _buffer_cur` equal to how far the stream has been consumed, which `seek` reads as its licence to fetch more; clamping `_buffer_cur` alone made that sum under-report, and the next forward `seek` silently spliced in octets from the wrong offset (`read(8)`, `truncate(3)`, `seek(6)`, `read(1)` over `b'abcdefghijklmnop'` returned `b'l'` where `b'g'` sits at offset 6). All three came from fuzzing random operation sequences against the buffer's invariants: of 3000 rounds, 2372 left the bookkeeping inconsistent and 683 raised an undocumented exception, and none do now. Latent rather than live -- nothing here calls `truncate`, though it is public -- and invisible to existing tests, which asserted the return value and never the content ([#622](https://github.com/JarryShaw/PyPCAPKit/issues/622)). +- the 16-bit band the [#573](https://github.com/JarryShaw/PyPCAPKit/issues/573) entry above records as deliberately left open: a length declared by a 16-bit wire field could still be repeated without limit. The crafted 80,048-octet PCAP-NG of 2,000 Enhanced Packet Blocks that entry describes synthesised 131,070,000 octets of zeros (1637.393x its size) and now synthesises none, with all 2,000 frames and options still parsed. The bound is taken one layer up, where the missing information exists: a PCAP-NG block's Block Total Length is authoritative and cross-checked against its trailing copy, so the option area is that length less the fixed fields, `captured_len` and its padding, and an option declaring more payload than the area has left is malformed. A snapshot-truncated capture says so through `captured_len` and leaves its options whole, so it never trips this -- which the [#571](https://github.com/JarryShaw/PyPCAPKit/pull/571) `len(buffer) < length` rejection could not distinguish, and why it was declined. A new `bounded_option` clamps the payload to the octets the area has left and warns with `SchemaWarning`; it is applied to all fifteen variable-width option and record payloads in the PCAP-NG schema, and deliberately not to the block-level payload fields, the 32-bit band the running ledger already budgets. A second, `bounded_area`, clamps a packet block's option area to the octets the block itself holds less the trailing Block Total Length: the area was otherwise sized from a declared length nothing checks against the file, so a block declaring 1,000,000 octets while holding 36 had a 999,964-octet area and 65,535 zeros were synthesised from 36 (1,820x) with no warning. The cross-review found that one. Both are no-ops on a well-formed block. The five non-packet blocks' option areas keep the framing assumption; the general fix for a declared length reaching a read is [#678](https://github.com/JarryShaw/PyPCAPKit/issues/678). Clamping rather than refusing keeps the [#431](https://github.com/JarryShaw/PyPCAPKit/issues/431) accommodation, since a PCAP-NG block read has no catch point above `FieldBase.unpack` and one refusal would abort a whole extraction rather than one block; and the clamp reads only the block's own declared framing, never a running total, so identical input answers identically whatever preceded it. The skip for a remainder already past zero is load-bearing on the unpacking path (a ten-octet area leaves `__length__` at -2 before a payload sizes itself, and clamping would give the field a `'-2s'` template); it is *not* what protects packing, since `BytesField` and `StringField` repair a negative width in `pre_process`. No clamp fires across the six PCAP-NG sample captures (338 options), and 501 truncation levels of `dhcp.pcapng` give byte-identical outcomes, most being the uncaught `ValueError` that [#678](https://github.com/JarryShaw/PyPCAPKit/issues/678) records. The bound is pinned as a property, not one input: across every combination of one to three options and nine declared lengths the options never report more octets than the area, and the ratio is flat across 1, 8, 64 and 512 blocks, which separates a bound from a smaller constant. The worst ratio over five adversarial shapes is now 0.862x, against 1,637x, 960x and 224x for the same three before ([#594](https://github.com/JarryShaw/PyPCAPKit/issues/594)). +- **a breaking change to** `FieldBase.length`: it called `struct.calcsize` on a template built from a negative resolved length (e.g. `'-5s'`, from a `length=` callback whose running counter had been overdrawn), raising a bare `struct.error` that ordinary caller code does not catch. The chokepoint every affected schema shares now raises `ProtocolError`, covering `application/{ftp,httpv1,httpv2,ngap}.py`, `internet/{hip,ipv6_route,mh}.py`, `link/ethernet.py`, `misc/pcapng.py` and `transport/sctp.py` with no other file changed; a non-negative length is unaffected. This closes the follow-on the `httpv2` length-guard entry above filed as out of scope ([#805](https://github.com/JarryShaw/PyPCAPKit/issues/805)). +- **a breaking change to** `EnumRegistry.register()`: it now raises `ValueError` naming the existing member and pointing at `register_alias()`, where it used to silently alias an already-registered value under the caller's new name -- `aenum` treats a taken value as an alias request, so `register(existing_value, 'TOTALLY_NEW_NAME')` returned the existing member, minted nothing and raised nothing, contradicting its docstring. `pcapkit.corekit.enums.EnumRegistry` is a new mixin implementing `get`, `get_all`, `register`, `register_alias`, `register_aliases` and `_unregistered_member` once, per the maintainer's ruling on [#842](https://github.com/JarryShaw/PyPCAPKit/issues/842); it converts the first six registries with no bespoke `__new__` (`ipv6.extension_header.ExtensionHeader`, `tcp.flags.Flags` and four `mh.*_flag` modules), and serves `IntEnum`, `IntFlag` and `StrEnum` alike. The guard reaches only those six: the other 105 const registries still carry the old generator's bare `extend_enum(cls, name, value)`, so `pcapkit.const` ships two contradictory `register` contracts until [#858](https://github.com/JarryShaw/PyPCAPKit/pull/858) (below). The six also gain the generated template's `KeyError` catching on a missing name, so `Flags.get('NOPE', 0)` returns `0` rather than raising (a widening, undisclosed until this PR); and a key that is neither `int` nor `str`, which raised `TypeError` on the five `IntFlag` registries' hand-copied `get` (`Flags.get(3.5)`, since `cls[key]` is a containment test on a `Flag`), now raises `ValueError`, tried as a value. `ExtensionHeader`, the only `IntEnum` among them, already raised `KeyError` there, the shift [#858](https://github.com/JarryShaw/PyPCAPKit/pull/858)'s entry attributes to the other 105 ([#855](https://github.com/JarryShaw/PyPCAPKit/pull/855)). +- **a breaking change to** `EnumRegistry.register()`, extended: the 105 const registries still on the generated template's bare `extend_enum` now inherit the guard [#855](https://github.com/JarryShaw/PyPCAPKit/pull/855) (above) added, so `TransType.register(6, 'TOTALLY_NEW_NAME')` raises `ValueError` naming the value and the member that holds it and pointing at `register_alias()`, where it used to mint and raise nothing. `pcapkit/vendor/default.py`'s template now emits `class {NAME}(EnumRegistry, IntEnum)` and drops its own `get`/`register`/`_unregistered_member`. Across 121 const modules holding 127 enum classes: 6 already used `EnumRegistry` ([#855](https://github.com/JarryShaw/PyPCAPKit/pull/855)); of the other 115 modules, 105 share the template byte-for-byte and are converted, and the remaining 10 keep their bespoke `__new__` and are left alone: `ftp/command` (4 classes), `ftp/return_code` (3), `http/method`, `http/status_code`, `pcapng/option_type`, `reg/apptype/apptype.py` (`AppType` and `TransportProtocol`) and its four transport subclasses. That makes **111 of the 127 classes** inherit `EnumRegistry`, up from 6. `get()`'s dispatch shifts only for a key that is neither `int` nor `str` (`KeyError`, tried as a name; now `ValueError`, tried as a value), a key type nothing in the library passes. `pcapkit/corekit/enums.py` is renamed **`enum.py`**, the maintainer's ruling ([#858](https://github.com/JarryShaw/PyPCAPKit/pull/858)). +- `EnumRegistry.get()`'s `str`-key branch tried only a name lookup (`cls._member_map_[key]`) and on a miss raised or returned `cls(default)`, never the *value* path, so on a `StrEnum` registry a key that resolved through the constructor (`_Str('known-value')`) raised `KeyError` through `get()`, and a supplied `default` overrode a good value match the code never attempted. It now falls back to a plain `_value2member_map_` lookup once the name lookup misses and before consulting `default`. Deliberately not `cls(key)`: `FEATCode._missing_` (`pcapkit/const/ftp/command.py`) mints via `extend_enum` for any unrecognised string, so routing through the constructor would let a failed name lookup mint a permanent member once a registry like it converts; a dict lookup never reaches `_missing_`. A name still wins when a string is both a name and a different member's value. Step 1 of [#860](https://github.com/JarryShaw/PyPCAPKit/issues/860); step 2, converting the bespoke registries, follows ([#869](https://github.com/JarryShaw/PyPCAPKit/pull/869), [#874](https://github.com/JarryShaw/PyPCAPKit/pull/874), below). The `IntEnum`/`IntFlag` non-`str` path is unchanged, measured on `TransType` and `HandoverACKFlag`. Widens rather than breaks: a call that raised `KeyError` now resolves, and no caller that got a correct result gets a different one ([#863](https://github.com/JarryShaw/PyPCAPKit/pull/863)). +- **a breaking change to** `EnumRegistry.get()`: its docstring claimed "It never mints", but both `cls(default)` call sites reached `_missing_` for a `default` inside a still-minting registry's range, growing the registry as a side effect of resolving `default` rather than `key`. Per the owner's ruling (*"Only register can mint. get should not mint unless it falls through the _missing_'s minted ranges"*), both sites now resolve `default` through a plain `_value2member_map_` lookup; `key` resolution is unchanged, so `EtherType.get(0x0888)` still mints `Xyplex_0x0888` through its own `_missing_`, as [#775](https://github.com/JarryShaw/PyPCAPKit/issues/775)'s deliberate exception permits. A `default` naming no registered member now falls through to the error `key` itself would have raised, rather than a fresh error about the default -- the accepted cost of the ruling. The docstring's "It never mints" is qualified to `default` only. New `GetDefaultNoMintTests` pin the issue's repro on the real `EtherType` registry, the preserved key-path mint, and default resolution on an `int` and a `str` registry. Four assertions in `tests/const/test_const_enum_get.py` and `tests/const/test_const_enum_builtin_parity.py` that encoded the old contract are retargeted rather than weakened; a fifth, `Hardware.get('Definitely-Not-A-Member', 40)` (`40` sits in `Hardware`'s declared-but-unassigned range and now raises `KeyError` naming the original key), waited on [#865](https://github.com/JarryShaw/PyPCAPKit/pull/865) because its test file was contended, then landed once [#865](https://github.com/JarryShaw/PyPCAPKit/pull/865) (above) merged, with `Hardware`'s member count unchanged at 42. Closes [#864](https://github.com/JarryShaw/PyPCAPKit/issues/864) ([#868](https://github.com/JarryShaw/PyPCAPKit/pull/868)). ### pcapkit.dumpkit #### Fixed -- a flag value with no declared bits no longer dumps as `Type::None [0]`. **This changes user-visible output in every textual dump format.** `make_dumper`'s `object_hook` renders each enumeration member as `Type::name [value]` and interpolated the name unguarded, but a `Flag` value composed entirely of undeclared bits has `name is None` rather than a string -- so the literal four characters `None` landed in the name half and `Flags(0)` rendered as `Flags::None [0]` in `json`, `tree`, `text`, `txt`, `plist` and `xml`, out of both `Extractor` and `TraceFlow`. A nameless member now renders with the value's own decimal spelling instead, so `Flags(0)` gives `Flags::0 [0]` and `Flags(8)` gives `Flags::8 [8]`. That is the spelling the enumeration libraries already use for an undeclared residue -- `Flags(2057).name` is `ACK|9`, naming the declared bit and giving the leftovers as one decimal number -- and it cannot be mistaken for a member name, since a Python identifier may not begin with a digit, whereas `NONE` is a real declared name elsewhere in the library. Three sites carried the identical interpolation, not one: the `OrderedMultiDict` key path, the `addon` branch and the scalar return, which now share one `render_enum` helper so the guard cannot be applied to two of three. That guard is on `name is None` and not on the value, because the defect never was about zero -- `Flags(1)`, `Flags(8)`, `Flags(9)` and `Flags(65536)` are equally nameless -- and it is not an `aenum` quirk either, since a stdlib `enum.IntFlag` answers `name is None` at the same values. Five of the library's seven flag registries are nameless at zero rather than the one reported: `Flags` plus the four Mobility Header flag registries, the other two declaring an explicit `undefined = 0`. Pre-existing since 2023-04-28 and surfaced rather than caused by [#634](https://github.com/JarryShaw/PyPCAPKit/pull/634), which made a flagless TCP segment reach this path. The committed example dumps do not move, because no member in them lacks a name ([#648](https://github.com/JarryShaw/PyPCAPKit/issues/648)). -- `tests/dumpkit/test_nameless_enum_rendering_unit.py` (added by [#670](https://github.com/JarryShaw/PyPCAPKit/pull/670)) carried two probes on the same wrong assumption that every flag-enum registry is 16 bits wide: `test_scalar_return_renders_a_nameless_member_as_its_value` checked one fixed tuple ending in `65536` against `tcp.flags.Flags` and its own `StdFlags` stand-in alone, and `test_no_flag_registry_renders_the_literal_none` swept every flag-enum registry instead, but only against `registry(0)`. Seven registries are not *all* 16 bits, at four distinct widths: `ftp.command.CommandType` (3 bits), `reg.apptype.TransportProtocol` (4, computed at runtime), `mh.binding_ack_flag.BindingACKFlag`, `mh.handover_ack_flag.HandoverACKFlag` and `mh.handover_initiate_flag.HandoverInitiateFlag` (8 each), and `mh.binding_update_flag.BindingUpdateFlag` with `tcp.flags.Flags` (16, the two the old probe actually fit). `tcp.flags.Flags(65536)` correctly raises -- its own `_missing_` bounds itself to `0 <= value <= 0xFFFF` -- so the sweep was failing on correct behaviour rather than reporting a defect. A new `_field_mask` derives each registry's own all-ones bound from its declared members, and `_nameless_values` returns the values that bound admits but no member names -- zero, each undeclared bit alone, and every undeclared bit combined -- in place of the one hard-coded tuple. A new `test_a_value_past_the_field_is_refused_rather_than_rendered` pins the boundary directly: the widest in-field value is accepted, one past it is refused. The file goes from 1 failed / 5 passed / 16 subtests to 6 passed / 64 subtests. A cross-review found that all seven guard `raise` lines this reaches -- `tcp/flags.py`'s own included -- were already covered by a passing test predating this fix; none was genuinely newly reached by it. No line under `pcapkit/` changed ([#702](https://github.com/JarryShaw/PyPCAPKit/issues/702)). -- the plist writer emitted mapping keys unescaped, so a key carrying `&`, `<` or `>` produced a report that is not well-formed XML. `dictdumper`'s `_append_dict` interpolates the key into `'{item}'` and calls `_encode_value` on the value only, so both branches of `pcapkit.dumpkit.common` that build a mapping now escape their own keys through one `escape_key` helper -- a `MultiDict`, where [#575](https://github.com/JarryShaw/PyPCAPKit/issues/575) escaped an enum-derived key inline and nothing else, and a plain dict, where nothing was escaped at all. The json, tree and text writers are untouched: `escape_key` is a no-op for them and a plain dict is not even rebuilt, so each still receives the caller's own mapping with the caller's own key objects in it. Measured across 6 captures and 5 formats, exactly 2 of the 30 reports changed -- `test.pcapng`'s plist and xml, one writer under two names -- and by one line each, leaving the other 28 byte-identical; that capture is why a non-`str` key is handled at all, its decryption secrets block keying the TLS key log entries by a raw bytes client random whose repr carries all three characters. The upstream half is `JarryShaw/DictDumper#125`, where `_append_dict` should call `_encode_value` on a key; this does not wait on it. `PcapngUnescapedKeyTests`' docstrings, which claimed both the json and the plist report come out unparseable, are reworded to match -- only the json writer's quoting defect still stands, upstream as `JarryShaw/DictDumper#121` ([#772](https://github.com/JarryShaw/PyPCAPKit/issues/772)). +- a flag value with no declared bits no longer dumps as `Type::None [0]`. **This changes user-visible output in every textual dump format.** `make_dumper`'s `object_hook` renders each enumeration member as `Type::name [value]` and interpolated the name unguarded, but a `Flag` value composed entirely of undeclared bits has `name is None`, so the literal `None` landed in the name half: `Flags(0)` rendered as `Flags::None [0]` in `json`, `tree`, `text`, `txt`, `plist` and `xml`, out of both `Extractor` and `TraceFlow`. A nameless member now renders with the value's decimal spelling, so `Flags(0)` gives `Flags::0 [0]` and `Flags(8)` gives `Flags::8 [8]`. That is the spelling the enumeration libraries use for an undeclared residue (`Flags(2057).name` is `ACK|9`), and it cannot be mistaken for a member name, since an identifier may not begin with a digit, whereas `NONE` is a real declared name elsewhere. Three sites carried the identical interpolation (the `OrderedMultiDict` key path, the `addon` branch and the scalar return) and now share one `render_enum` helper. The guard is on `name is None` and not on the value, because the defect was never about zero (`Flags(1)`, `Flags(8)`, `Flags(9)` and `Flags(65536)` are equally nameless) and is not an `aenum` quirk: stdlib `enum.IntFlag` answers `name is None` at the same values. Five of the seven flag registries are nameless at zero (`Flags` and the four Mobility Header flags; the other two declare `undefined = 0`). Pre-existing since 2023-04-28 and surfaced by [#634](https://github.com/JarryShaw/PyPCAPKit/pull/634), which made a flagless TCP segment reach this path. The committed example dumps do not move ([#648](https://github.com/JarryShaw/PyPCAPKit/issues/648)). +- `tests/dumpkit/test_nameless_enum_rendering_unit.py` (added by [#670](https://github.com/JarryShaw/PyPCAPKit/pull/670)) carried two probes assuming every flag-enum registry is 16 bits wide: `test_scalar_return_renders_a_nameless_member_as_its_value` used one fixed tuple ending in `65536`, and `test_no_flag_registry_renders_the_literal_none` swept every registry but only against `registry(0)`. Widths differ: `ftp.command.CommandType` 3 bits, `reg.apptype.TransportProtocol` 4 (computed at runtime), the three `mh` ack/initiate flags 8, and `mh.binding_update_flag.BindingUpdateFlag` and `tcp.flags.Flags` 16. `tcp.flags.Flags(65536)` correctly raises (`0 <= value <= 0xFFFF`), so the sweep was failing on correct behaviour. A new `_field_mask` derives each registry's own bound from its declared members, `_nameless_values` returns the values that bound admits but no member names (zero, each undeclared bit alone, all combined), and `test_a_value_past_the_field_is_refused_rather_than_rendered` pins the boundary: the widest in-field value is accepted, one past it is refused. A cross-review found all seven guard `raise` lines it reaches (`tcp/flags.py`'s own included) were already covered by a test predating the fix, so none was newly reached. Nothing under `pcapkit/` changed ([#702](https://github.com/JarryShaw/PyPCAPKit/issues/702)). +- the plist writer emitted mapping keys unescaped, so a key carrying `&`, `<` or `>` produced a report that is not well-formed XML. `dictdumper`'s `_append_dict` interpolates the key into `'{item}'` and calls `_encode_value` on the value only, so both branches of `pcapkit.dumpkit.common` that build a mapping now escape their keys through one `escape_key` helper -- a `MultiDict`, where [#575](https://github.com/JarryShaw/PyPCAPKit/issues/575) escaped an enum-derived key inline and nothing else, and a plain dict, where nothing was escaped. The json, tree and text writers are untouched (`escape_key` is a no-op for them and a plain dict is not rebuilt). Across 6 captures and 5 formats, exactly 2 of 30 reports changed (`test.pcapng`'s plist and xml, one writer under two names), by one line each; that capture is why a non-`str` key is handled at all, its decryption secrets block keying TLS key log entries by a raw bytes client random whose repr carries all three characters. The upstream half is `JarryShaw/DictDumper#125` (`_append_dict` should call `_encode_value` on a key); this does not wait on it. `PcapngUnescapedKeyTests`' docstrings, which claimed both the json and plist reports were unparseable, are reworded: only the json writer's quoting defect stands, upstream as `JarryShaw/DictDumper#121` ([#772](https://github.com/JarryShaw/PyPCAPKit/issues/772)). ### pcapkit.foundation #### Added -- three extraction engines: `engine='pypcap'` and `engine='pcap_ct'`, two independent distributions of the same `libpcap` interface, and `engine='pypcapfile'` ([#386](https://github.com/JarryShaw/PyPCAPKit/pull/386), [#405](https://github.com/JarryShaw/PyPCAPKit/pull/405)). They buy speed by doing less -- neither `pypcap` nor `pcap_ct` dissects at all, so they offer neither reassembly nor flow tracing, and `pypcapfile` has no IPv6 decoder. Install only **one** of `pypcap` and `pcap-ct`: both own the top-level `pcap` module, and with both present `pcap-ct` wins the import and the other becomes unselectable. The matching interface constants `PyPCAP`, `PCAP_CT` and `PyPCAPFile` were missing and are now exported alongside `DPKT`, `Scapy`, `PyShark` and `PCAPKit` ([#412](https://github.com/JarryShaw/PyPCAPKit/pull/412)). That brings the built-in set to seven engines; 3.11 is the last interpreter on which every one of them can run, and even there two of them cannot coexist. -- `EngineBase.unsupported_reason`, a preflight every engine answers and `Extractor.run` consults before anything is imported. Asking for an engine that cannot run in the current environment now gives one warning naming the real cause -- a Python version, a missing `tshark`, a missing `libpcap`, the wrong `pcap` distribution -- and a clean fall back to `pcapkit`'s own parser, rather than an error from inside the third-party package ([#396](https://github.com/JarryShaw/PyPCAPKit/pull/396), [#405](https://github.com/JarryShaw/PyPCAPKit/pull/405)). -- `conflict` on the reassembly data models: absolute, inclusive ranges where two fragments claimed the same span with different bytes, which was previously lost silently on both the IP ([#482](https://github.com/JarryShaw/PyPCAPKit/pull/482)) and TCP ([#443](https://github.com/JarryShaw/PyPCAPKit/issues/443), [#478](https://github.com/JarryShaw/PyPCAPKit/pull/478)) paths. +- three extraction engines: `engine='pypcap'` and `engine='pcap_ct'`, two independent distributions of the same `libpcap` interface, and `engine='pypcapfile'` ([#386](https://github.com/JarryShaw/PyPCAPKit/pull/386), [#405](https://github.com/JarryShaw/PyPCAPKit/pull/405)). They buy speed by doing less -- neither `pypcap` nor `pcap_ct` dissects at all, so they offer no reassembly or flow tracing, and `pypcapfile` has no IPv6 decoder. Install only **one** of `pypcap` and `pcap-ct`: both own the top-level `pcap` module, and with both present `pcap-ct` wins the import and the other becomes unselectable. The missing interface constants `PyPCAP`, `PCAP_CT` and `PyPCAPFile` are now exported alongside `DPKT`, `Scapy`, `PyShark` and `PCAPKit` ([#412](https://github.com/JarryShaw/PyPCAPKit/pull/412)). That makes seven built-in engines; 3.11 is the last interpreter on which every one can run, and even there two cannot coexist. +- `EngineBase.unsupported_reason`, a preflight every engine answers and `Extractor.run` consults before anything is imported. An engine that cannot run in the current environment now gives one warning naming the real cause (a Python version, a missing `tshark` or `libpcap`, the wrong `pcap` distribution) and a clean fall back to `pcapkit`'s own parser, rather than an error from inside the third-party package ([#396](https://github.com/JarryShaw/PyPCAPKit/pull/396), [#405](https://github.com/JarryShaw/PyPCAPKit/pull/405)). +- `conflict` on the reassembly data models: absolute, inclusive ranges where two fragments claimed the same span with different bytes, previously lost silently on both the IP ([#482](https://github.com/JarryShaw/PyPCAPKit/pull/482)) and TCP ([#443](https://github.com/JarryShaw/PyPCAPKit/issues/443), [#478](https://github.com/JarryShaw/PyPCAPKit/pull/478)) paths. #### Changed -- `layer=` and `protocol=` are honoured rather than inert. Both were read under the wrong names, so every value a caller passed was dropped into `**kwargs` and discarded; the CLI's `-L` also now validates its argument instead of accepting anything. The packet context reaches the schema layer for the first time as well, so a field the wire elides can be resolved from its enclosing packet ([#404](https://github.com/JarryShaw/PyPCAPKit/pull/404)). `follow_tcp_stream` dispatches on the engine type, where both branches of the old test were dead and the native adapter ran against every engine's frames ([#402](https://github.com/JarryShaw/PyPCAPKit/pull/402)). -- two reassembly and flow-tracing defaults moved ([#435](https://github.com/JarryShaw/PyPCAPKit/pull/435)), and both are visible to a caller. `Datagram.completed` widened from `bool` to a `Completion` enumeration (`COMPLETE`, `PARTIAL`, `TIMEOUT`); only `COMPLETE` is truthy, so `if datagram.completed:` is unaffected but `datagram.completed == True` no longer holds. TCP flow tracing is **bidirectional by default**, which merges each flow's two halves and closes one only once both have FINed -- 331 flows become 111 on the sample HTTP capture, the difference being single-frame stray tails; pass `trace_bidirectional=False` for the old behaviour. IP reassembly also gained the 60-second timeout [[RFC 1122](https://datatracker.ietf.org/doc/html/rfc1122), [RFC 8200](https://datatracker.ietf.org/doc/html/rfc8200)], clocked off the capture's own timestamps rather than the wall clock, tunable with `reasm_timeout=`; TCP reassembly gets no timeout by default. `trace_analyse=` is new, and reassembles each traced flow's application layer. -- conflicting TCP overlaps resolve first-write-wins, per [RFC 9293](https://datatracker.ietf.org/doc/html/rfc9293) section 3.10, where they had silently resolved last-write-wins ([#443](https://github.com/JarryShaw/PyPCAPKit/issues/443), [#478](https://github.com/JarryShaw/PyPCAPKit/pull/478)). A deliberate behaviour break, and a narrow one: a conforming retransmission carries identical bytes, so nothing changes for it. IP fragment reassembly keeps last-write-wins, because [RFC 791](https://datatracker.ietf.org/doc/html/rfc791) specifies the opposite resolution, and records the disagreement instead ([#482](https://github.com/JarryShaw/PyPCAPKit/pull/482)). +- `layer=` and `protocol=` are honoured rather than inert. Both were read under the wrong names, so every value a caller passed was dropped into `**kwargs`; the CLI's `-L` also now validates its argument. The packet context reaches the schema layer for the first time, so a field the wire elides can be resolved from its enclosing packet ([#404](https://github.com/JarryShaw/PyPCAPKit/pull/404)). `follow_tcp_stream` dispatches on the engine type, where both branches of the old test were dead and the native adapter ran against every engine's frames ([#402](https://github.com/JarryShaw/PyPCAPKit/pull/402)). +- two reassembly and flow-tracing defaults moved ([#435](https://github.com/JarryShaw/PyPCAPKit/pull/435)), both visible to a caller. `Datagram.completed` widened from `bool` to a `Completion` enumeration (`COMPLETE`, `PARTIAL`, `TIMEOUT`); only `COMPLETE` is truthy, so `if datagram.completed:` is unaffected but `datagram.completed == True` no longer holds. TCP flow tracing is **bidirectional by default**, which merges each flow's two halves and closes one only once both have FINed -- 331 flows become 111 on the sample HTTP capture, the difference being single-frame stray tails; pass `trace_bidirectional=False` for the old behaviour. IP reassembly also gained the 60-second timeout [[RFC 1122](https://datatracker.ietf.org/doc/html/rfc1122), [RFC 8200](https://datatracker.ietf.org/doc/html/rfc8200)], clocked off the capture's own timestamps, tunable with `reasm_timeout=`; TCP reassembly gets no timeout by default. `trace_analyse=` is new, and reassembles each traced flow's application layer. +- conflicting TCP overlaps resolve first-write-wins, per [RFC 9293](https://datatracker.ietf.org/doc/html/rfc9293) section 3.10, where they had silently resolved last-write-wins ([#443](https://github.com/JarryShaw/PyPCAPKit/issues/443), [#478](https://github.com/JarryShaw/PyPCAPKit/pull/478)). A deliberate, narrow behaviour break: a conforming retransmission carries identical bytes. IP fragment reassembly keeps last-write-wins, because [RFC 791](https://datatracker.ietf.org/doc/html/rfc791) specifies the opposite resolution, and records the disagreement instead ([#482](https://github.com/JarryShaw/PyPCAPKit/pull/482)). - renames with no compatibility alias left behind: `HoleDiscriptor` is spelled `HoleDescriptor` and its package alias `TCP_HoleDiscriptor` is `TCP_HoleDescriptor` ([#350](https://github.com/JarryShaw/PyPCAPKit/pull/350)). -- subclass registration is **opt-in** for `Engine`, `Reassembly`, `TraceFlow` and `dumpkit`'s `Dumper` ([#514](https://github.com/JarryShaw/PyPCAPKit/issues/514)). Each registers if and only if its registry keyword is given -- `engine=` for `Engine`, `protocol=` for `Reassembly` and `TraceFlow`, `fmt=` for `Dumper`. Previously an absent keyword fell back to the class' own name, so *every* subclass of the public class was registered, and declining meant subclassing the parallel `*Base` class under an alias -- which is what every built-in does, and why the public classes had **0** subclasses between them against the `*Base` classes' 9, 5, 2 and 3. **This breaks out-of-tree code that subclasses one of the four and relies on the derived key**; pass the keyword, or call the matching `register_*` function. Nothing the library ships is affected, and the `*Base` classes remain importable. Two things that were silent are now loud: an unrecognised class keyword raises `UnsupportedCall` instead of being swallowed by `**kwargs` -- which used to register the class under its own name, so passing `name=` to a `Reassembly` subclass silently ignored the key it was given, `protocol=` being the real one -- and `Dumper`'s `ext=` without `fmt=` likewise. A class attribute is not an opt-in: `__engine_name__` and `__protocol_name__` still set the name a class reports, registered or not. Each metaclass also gained a class-level `registry` property mirroring `EnumSchema.registry`. As a side effect a `Dumper` subclass no longer touches the filesystem while its `class` statement runs: inferring `fmt` from the `kind` property meant instantiating the class against a `NamedTemporaryFile` mid-definition. `Engine`'s keyword is `engine=` rather than the `name=` this first shipped with, because `name` cannot be passed as a class keyword at all on Python 3.10: `mcls`, `name`, `bases` and `namespace` collide with `abc.ABCMeta.__new__`'s own parameters, which are positional-or-keyword before 3.11 and positional-only from 3.11, so a class statement naming any of the four raises `TypeError` from the metaclass before the hook is reached. Those four are the whole of the `ABCMeta.__new__` collision surface, measured on 3.10.21, 3.11.15 and 3.14.7; `engine=`, `protocol=` and `fmt=` are all outside it, so the documented registration path works on every supported version. There is no `name=` alias -- a keyword that worked on some interpreters and not others is the trap being removed, not a compatibility measure. -- extraction is around 46% faster on a 1,117-frame HTTP capture, with byte-identical output ([#420](https://github.com/JarryShaw/PyPCAPKit/pull/420)). A reassembled datagram's payload is now analysed on first read rather than eagerly, which cuts IP reassembly's own cost by 90.7% and TCP's by 23.7% -- IP reassembly submits a datagram for every frame, fragmented or not ([#424](https://github.com/JarryShaw/PyPCAPKit/pull/424)). Flow tracing over the same capture went from 1416.6 ms to 744.0 ms, because the flow dumper had been handing each record to a `Frame` constructor that re-dissected the whole protocol stack to return bytes it had just been given; options are no longer parsed twice either ([#427](https://github.com/JarryShaw/PyPCAPKit/pull/427)). All output compared byte-for-byte across the sample captures in each case. +- subclass registration is **opt-in** for `Engine`, `Reassembly`, `TraceFlow` and `dumpkit`'s `Dumper` ([#514](https://github.com/JarryShaw/PyPCAPKit/issues/514)). Each registers if and only if its registry keyword is given -- `engine=` for `Engine`, `protocol=` for `Reassembly` and `TraceFlow`, `fmt=` for `Dumper`. Previously an absent keyword fell back to the class' own name, so *every* subclass of the public class was registered, and declining meant subclassing the parallel `*Base` class under an alias, which every built-in does (hence the public classes had **0** subclasses against the `*Base` classes' 9, 5, 2 and 3). **This breaks out-of-tree code that subclasses one of the four and relies on the derived key**; pass the keyword, or call the matching `register_*` function. Nothing the library ships is affected, and the `*Base` classes remain importable. Two silent failures are now loud: an unrecognised class keyword raises `UnsupportedCall` instead of being swallowed by `**kwargs` (passing `name=` to a `Reassembly` subclass silently ignored the key, since `protocol=` is the real one), and so does `Dumper`'s `ext=` without `fmt=`. A class attribute is not an opt-in: `__engine_name__` and `__protocol_name__` still set the name a class reports, registered or not. Each metaclass gained a class-level `registry` property mirroring `EnumSchema.registry`, and a `Dumper` subclass no longer touches the filesystem while its `class` statement runs (inferring `fmt` from `kind` meant instantiating it against a `NamedTemporaryFile`). `Engine`'s keyword is `engine=` rather than the `name=` first shipped, because `name` cannot be a class keyword on Python 3.10: `mcls`, `name`, `bases` and `namespace` collide with `abc.ABCMeta.__new__`'s parameters (positional-or-keyword before 3.11, positional-only from 3.11), raising `TypeError` before the hook runs. Those four are the whole collision surface (measured on 3.10.21, 3.11.15 and 3.14.7), and `engine=`, `protocol=` and `fmt=` are outside it. There is no `name=` alias: a keyword that works on some interpreters and not others is the trap being removed. +- extraction is around 46% faster on a 1,117-frame HTTP capture, with byte-identical output ([#420](https://github.com/JarryShaw/PyPCAPKit/pull/420)). A reassembled datagram's payload is analysed on first read rather than eagerly, cutting IP reassembly's own cost by 90.7% and TCP's by 23.7% (IP reassembly submits a datagram for every frame, fragmented or not) ([#424](https://github.com/JarryShaw/PyPCAPKit/pull/424)). Flow tracing over the same capture went from 1416.6 ms to 744.0 ms, because the flow dumper handed each record to a `Frame` constructor that re-dissected the whole stack to return the bytes it had just been given; options are no longer parsed twice either ([#427](https://github.com/JarryShaw/PyPCAPKit/pull/427)). #### Fixed - TCP reassembly mixed absolute sequence numbers with buffer-relative slicing, so on any capture carrying a SYN with a realistic initial sequence number incomplete datagrams were dropped silently, and the `completed=False` branch of the public API was unreachable ([#349](https://github.com/JarryShaw/PyPCAPKit/issues/349), [#376](https://github.com/JarryShaw/PyPCAPKit/pull/376)). - `format='text'` raised `AttributeError` before writing anything, naming a `dictdumper.Text` that has never existed. It now points at `Tree`, as the `'txt'` alias beside it already did. -- `Extractor` had the ownership of its input stream inverted, so it did both halves of the wrong thing at once: a handle it opened itself, from `fin` given as a path, was **never closed**, while a stream the *caller* supplied and still needed **was**. `_cleanup` closed under `not self._flag_s` where `self._flag_s` is the flag that gated the `open()`, and the `SeekableReader` wrapping a non-seekable input asked for `stream_closing=not self._flag_s` in the same wrong direction. The leak was one descriptor per extraction for the life of the process, and the production root cause of the `ResourceWarning` that reddened [#577](https://github.com/JarryShaw/PyPCAPKit/pull/577), [#596](https://github.com/JarryShaw/PyPCAPKit/pull/596) and [#600](https://github.com/JarryShaw/PyPCAPKit/pull/600) from three unrelated pull requests -- [#606](https://github.com/JarryShaw/PyPCAPKit/issues/606) made the assertion in `tests/utilities/test_stacklevel.py` immune to foreign warnings, which repaired the CI signal but could not stop the leak, because the leak was in library code. Closing the caller's stream was the more dangerous half: silent data loss for anyone passing an open file they meant to keep reading, with no warning and no error to notice it by. Both conditions now follow ownership, via a single `Extractor._owns_input`, and a finaliser releases the handle of an extraction *abandoned* before end of file -- `auto=False`, iterated part way and dropped -- which reaches `_cleanup` by no route at all and was the second of the two warnings. Measured on `6c3d1b0d9`: a path-given extraction left 1 descriptor open and a stream-given one came back closed and unreadable; the two files [#610](https://github.com/JarryShaw/PyPCAPKit/issues/610) names now emit 0 such warnings where they emitted 2. `__exit__` is deliberately left closing unconditionally, since a caller who scopes an `Extractor` with `with` has asked for exactly that ([#610](https://github.com/JarryShaw/PyPCAPKit/issues/610)). -- `extract(..., no_eof=True)` never returned. End of stream was detected correctly and `ExtractionWarning: EOF reached` fired, but all three loops that handle it -- `record_frames`, `__next__` and `__call__` -- read `if self._flag_n: continue` with nothing else to stop them, so the flag suppressed the error that *ended* the loop without supplying any other ending and an exhausted input was retried forever. The flag is not pointless, which is why this is a termination condition rather than a deletion: `__main__` sets `no_eof` exactly for `fin='-'`, and the end of what has arrived on a live capture is not the end of the capture. What was missing was a way to tell a capture that has paused from one that is over, and the input's own position answers it -- `prepare` raises end of stream when the bytes remaining measure zero and restores the position before raising, so two consecutive ends of stream at the same position mean nothing arrived between them. `Extractor._note_eof_progress` is that check. A pipe is unaffected, because a read there *blocks* while the writer is open but idle rather than reporting end of stream -- measured at a full 1.5s pause on `6c3d1b0d9`, after which the frames arrived -- so a paused pipe never reaches the check at all. Also fixed thereby, and not mentioned in the issue: `pcapkit -` hung on **any** finished stdin, a pipe reporting end of stream once its writer closes; verified through the real CLI, where `cat in.pcap | python -m pcapkit -` went from killed at 25s to exiting 0 with a byte-identical dump. **One deliberate narrowing**: a *seekable* input does not block, so its two probes fall microseconds apart and a file still being appended to now ends at the data present when the extraction reached it -- measured with the append landing on a record boundary 0.6s in, six frames before and five after. The previous behaviour was unbounded by construction, which is the defect itself, so some stopping rule had to be chosen; a timed grace period would make the cut-off intermittent rather than absent, so following a growing file is left to a policy of its own and the decision is pinned by a test. Worth knowing alongside it, pre-existing and untouched: an append landing *mid-record* raises `ValueError: read length must be non-negative or -1` instead, so `no_eof` over a growing file only ever worked for a writer flushing whole records. Both docstrings for `no_eof` now say all of this. The end-to-end regression tests are bounded by a **child process** rather than by `tests._support.time_limit`, because the in-process deadline proved *intermittent* on this loop: it usually expired on time and once escaped entirely, running past ten minutes and 13.2 GB RSS before being killed by hand. An intermittent guard against a hang is worse than none, since the run it misses is a wedged suite rather than a red test. Two guesses at the cause are recorded as ruled out, so they are not made again: the parse path's two `except Exception` handlers are never entered during the spin, because `prepare` raises before any next-layer decode, and the memory is the retry loop emitting tens of thousands of `EOF reached` records a second which the runner retains ([#620](https://github.com/JarryShaw/PyPCAPKit/issues/620)). -- `register_protocol` displaced one protocol class with another and said nothing about it. The registry behind `pcapkit.protocols.__proto__` is keyed on `cls.__name__.upper()`, and three dispatchable classes are all named `HTTP` -- the generic base `pcapkit.protocols.application.http.HTTP` and the two version implementations in `httpv1` and `httpv2` -- so all three compete for the single key `'HTTP'` and a bare assignment decided the winner. Measured before the fix: `__proto__['HTTP']` went from `pcapkit.protocols.application.http.HTTP` to the `httpv2` class on `register_protocol(httpv2.HTTP)`, with no exception, no log line and no `RegistryWarning`, and `ProtocolBase.expand_comp('HTTP')` then resolved to whichever class had registered last. The overwrite now raises a `RegistryWarning` naming both the displaced and the replacing class, the defining module being the only thing that tells the three apart. The sharpest path to it is not a call to the registrar at all: `Protocol.__init_subclass__` registers unconditionally, so merely *defining* a subclass of the public `Protocol` under a name a built-in already holds displaced that built-in, with nothing anywhere in the user's code to mark the moment the dispatch table changed meaning. A plain `import pcapkit` is unaffected and emits zero new warnings, measured not assumed -- the three built-in `HTTP` classes derive from `ProtocolBase` rather than from the public `Protocol`, so `__init_subclass__` never fires for them, and the import-time seeding in `pcapkit/protocols/__init__.py` keys off the distinct names `HTTP`, `HTTPv1` and `HTTPv2`. The issue's framing that this was the one registrar in `pcapkit/foundation/registry/` with no `RegistryWarning` is right on the count and wrong on the reason: no function in that package warns about anything, the siblings warning only by delegating to a classmethod that carried the guarded `if code in cls.__xxx__: warn(...)` at the time -- [#726](https://github.com/JarryShaw/PyPCAPKit/pull/726) later gave all seven the identity guard this file describes elsewhere. The accurate statement is the stronger one -- `register_protocol` is the only keyed registrar in that package, and the only `__name__.upper()`-keyed registry anywhere in `pcapkit/`, that mutates its target with no guard *and* no delegate that could supply one, its target being a module-level dict with no class behind it to hold the guard. The new guard deliberately reads "key present and the incumbent is a different class" where every sibling warned on mere presence at the time: the sibling keys are codes the caller passes, whereas this key is derived from the class and this function is the funnel all nine wrapper registrars end in, so registering one class under two codes -- `register_tcp` then `register_udp`, which is supported and documented -- reaches it twice with nothing displaced, and warning there would put noise on a documented path whose wholesale filter is precisely what would then hide the real collision. The collision is reported, not resolved: re-keying is a registry-format change that none of the bare-name readers can absorb, every one of them degrading *silently* to a plain string or to `Raw` on a miss rather than raising, so it belongs to the registry redesign in [#514](https://github.com/JarryShaw/PyPCAPKit/issues/514), which this change is sequenced ahead of rather than part of. 9 to 13 tests and 80 to 87 subtests over `tests/foundation/registry/`, the touched module holding 88% coverage with its misses flat at 27 ([#675](https://github.com/JarryShaw/PyPCAPKit/issues/675)). -- two `#:` autodoc comments in `pcapkit/foundation/traceflow/traceflow.py` named a bare `Type`, which Sphinx's cross-reference resolver resolves against every class named `Type` in the project rather than against `typing.Type` -- there are five -- and silently linked to `pcapkit.const.l2tp.type.Type`, an L2TP field-type enum with nothing to do with dumpers. Line 424 (`#: ~typing.Type[Dumper]: Dumper class.`, spelled out since [#709](https://github.com/JarryShaw/PyPCAPKit/issues/709) fixed it) is the live case: once [#684](https://github.com/JarryShaw/PyPCAPKit/issues/684) rendered `TraceFlow._foutio`, the built docs pointed a reader at the wrong class with no warning that anything had gone sideways. Both sites now spell it `~typing.Type[Dumper]`, the same form eight other files in the tree already use for the identical ambiguity. Line 146 (the first line of `__output__`'s `#:` block, which continues through line 149) is fixed for the same reason but is currently inert: the `# type:` comment sixteen lines below, at line 162, spells the identical bare `Type[Dumper]` -- so whatever eventually renders this attribute's type still has the same ambiguity to resolve. This half of the fix is insurance against the day something reads it cleanly. This pair of sites is part of [#709](https://github.com/JarryShaw/PyPCAPKit/issues/709). Four more bare `Type` sites -- hand-written `:type:` fields at `docs/source/pcapkit/foundation/engines/engine.rst:40`, `.../reassembly/reassembly.rst:33` and `:43`, and `.../traceflow/traceflow.rst:40` -- carried the same ambiguity and are fixed separately, by [#714](https://github.com/JarryShaw/PyPCAPKit/pull/714) ([#709](https://github.com/JarryShaw/PyPCAPKit/issues/709)). -- `register_protocol`'s overwrite warning could claim a protocol was replaced with itself. The guard that decides *whether* to warn was already correct -- `incumbent is not protocol`, an identity check from [#681](https://github.com/JarryShaw/PyPCAPKit/pull/681) -- but the message built both operands with a bare `repr()`, and for an ordinary class that is just ``. A factory that defines a same-named, closure-local class on every call (as `tests/protocols/test_construction_keyword_check_unit.py`'s `_protocol_class` does) produces two genuinely distinct classes sharing one `__module__` and `__qualname__`, so both reprs print identically and a real, correct overwrite reads as "overwriting X with X." The message now compares the two reprs first, and only when they coincide appends each object's `id()` to tell them apart; the common case, where the two classes are named differently, is untouched. `__module__`/`__qualname__` was considered as the disambiguator instead of `id()` and rejected: for the reported shape those are exactly what the coinciding repr already renders, so they discriminate nothing an `id()` does not already have to. A new `test_register_protocol_disambiguates_classes_sharing_a_repr` fails against the unfixed message and passes against the fix; the targeted suite goes 49 to 50 passed, and the file's coverage holds at 97% ([#710](https://github.com/JarryShaw/PyPCAPKit/issues/710)). -- five more registrars warned on mere key presence rather than on an actual overwrite, outside the wording of [#718](https://github.com/JarryShaw/PyPCAPKit/issues/718)'s identity guard (`incumbent is not None and incumbent is not new`, landed by [#726](https://github.com/JarryShaw/PyPCAPKit/pull/726)), whose issue named only code-keyed registrars: `register_engine`, `register_reassembly` and `register_traceflow` on `Extractor`, and `register_dumper` on both `Extractor` and `TraceFlow`. All five now compare the incumbent by identity before warning; both `register_dumper` sites compare only the stored dumper, so a re-registration that changes just the file extension stays silent too, judged defensible rather than comparing the full `(dumper, ext)` pair. Non-breaking: a correct caller sees strictly fewer warnings and no change to return value or exception. Five new test methods pin the silent/warns-anyway split, each failing with `AssertionError: Expected 'warn' to not have been called` against the reverted guard alone ([#739](https://github.com/JarryShaw/PyPCAPKit/issues/739)). +- `Extractor` had the ownership of its input stream inverted: a handle it opened itself, from `fin` given as a path, was **never closed**, while a stream the *caller* supplied and still needed **was**. `_cleanup` closed under `not self._flag_s` (the flag that gated the `open()`), and the `SeekableReader` wrapping a non-seekable input asked for `stream_closing=not self._flag_s` in the same wrong direction. The leak was one descriptor per extraction for the life of the process, and the production root cause of the `ResourceWarning` that reddened [#577](https://github.com/JarryShaw/PyPCAPKit/pull/577), [#596](https://github.com/JarryShaw/PyPCAPKit/pull/596) and [#600](https://github.com/JarryShaw/PyPCAPKit/pull/600); [#606](https://github.com/JarryShaw/PyPCAPKit/issues/606) made `tests/utilities/test_stacklevel.py` immune to foreign warnings, which repaired the CI signal but not the leak. Closing the caller's stream was the more dangerous half: silent data loss for anyone passing an open file they meant to keep reading. Both conditions now follow ownership via a single `Extractor._owns_input`, and a finaliser releases the handle of an extraction *abandoned* before end of file (`auto=False`, iterated part way and dropped), which reaches `_cleanup` by no route and was the second warning. A path-given extraction used to leave 1 descriptor open and a stream-given one came back closed and unreadable; the two files [#610](https://github.com/JarryShaw/PyPCAPKit/issues/610) names emit 0 such warnings where they emitted 2. `__exit__` deliberately still closes unconditionally, since a caller who scopes an `Extractor` with `with` asked for that ([#610](https://github.com/JarryShaw/PyPCAPKit/issues/610)). +- `extract(..., no_eof=True)` never returned. End of stream was detected and `ExtractionWarning: EOF reached` fired, but all three loops that handle it (`record_frames`, `__next__`, `__call__`) read `if self._flag_n: continue`, so the flag suppressed the error that *ended* the loop without supplying another ending and an exhausted input was retried forever. The flag is not pointless -- `__main__` sets `no_eof` for `fin='-'`, and the end of what has arrived on a live capture is not the end of the capture -- so this is a termination condition rather than a deletion. The input's own position tells a paused capture from a finished one: `prepare` raises end of stream when zero bytes remain and restores the position first, so two consecutive ends of stream at the same position mean nothing arrived between them (`Extractor._note_eof_progress`). A pipe is unaffected, since a read there *blocks* while the writer is open but idle. Also fixed, not in the issue: `pcapkit -` hung on **any** finished stdin; `cat in.pcap | python -m pcapkit -` went from killed at 25s to exiting 0 with a byte-identical dump. **One deliberate narrowing**: a *seekable* input does not block, so its two probes fall microseconds apart and a file still being appended to ends at the data present when the extraction reached it. The previous behaviour was unbounded, which is the defect; a timed grace period would make the cut-off intermittent rather than absent, so following a growing file is left to a policy of its own and the decision is pinned by a test. Pre-existing and untouched: an append landing *mid-record* raises `ValueError: read length must be non-negative or -1`, so `no_eof` over a growing file only ever worked for a writer flushing whole records. Both `no_eof` docstrings say all of this. The end-to-end regression tests are bounded by a **child process** rather than `tests._support.time_limit`, because the in-process deadline proved *intermittent* on this loop (once it escaped entirely, past ten minutes and 13.2 GB RSS), and an intermittent guard against a hang leaves a wedged suite rather than a red test. Two guesses at the cause are ruled out: the parse path's two `except Exception` handlers are never entered during the spin (`prepare` raises before any next-layer decode), and the memory is the retry loop emitting tens of thousands of `EOF reached` records a second, which the runner retains ([#620](https://github.com/JarryShaw/PyPCAPKit/issues/620)). +- `register_protocol` displaced one protocol class with another and said nothing. The registry behind `pcapkit.protocols.__proto__` is keyed on `cls.__name__.upper()`, and three dispatchable classes are all named `HTTP` (the base `pcapkit.protocols.application.http.HTTP` and the `httpv1` and `httpv2` implementations), so a bare assignment decided the winner: `register_protocol(httpv2.HTTP)` moved `__proto__['HTTP']` to the `httpv2` class with no exception, log line or `RegistryWarning`, and `ProtocolBase.expand_comp('HTTP')` then resolved to whichever registered last. The overwrite now raises a `RegistryWarning` naming the displaced and the replacing class. The sharpest path is not a call to the registrar: `Protocol.__init_subclass__` registers unconditionally, so merely *defining* a subclass of the public `Protocol` under a name a built-in holds displaced that built-in. A plain `import pcapkit` emits zero new warnings: the three built-in `HTTP` classes derive from `ProtocolBase`, so `__init_subclass__` never fires for them, and import-time seeding keys off the distinct names `HTTP`, `HTTPv1` and `HTTPv2`. The issue called this the one registrar in `pcapkit/foundation/registry/` with no warning; more accurately, none warned by itself, the siblings only by delegating to a classmethod with an `if code in cls.__xxx__: warn(...)` guard ([#726](https://github.com/JarryShaw/PyPCAPKit/pull/726) later gave all seven the identity guard described elsewhere), and this is the only `__name__.upper()`-keyed registry in `pcapkit/` that mutates its target with no guard and no delegate able to supply one. The new guard deliberately reads "key present and the incumbent is a different class" where siblings warned on mere presence then: this key is derived from the class, and this function is the funnel all nine wrapper registrars end in, so registering one class under two codes (`register_tcp` then `register_udp`, supported and documented) reaches it twice with nothing displaced, and warning there would put noise on a documented path whose wholesale filter would then hide the real collision. The collision is reported, not resolved: re-keying is a registry-format change that none of the bare-name readers can absorb (each degrades *silently* to a string or `Raw` on a miss), so it belongs to the registry redesign in [#514](https://github.com/JarryShaw/PyPCAPKit/issues/514), which this is sequenced ahead of ([#675](https://github.com/JarryShaw/PyPCAPKit/issues/675)). +- two `#:` autodoc comments in `pcapkit/foundation/traceflow/traceflow.py` named a bare `Type`, which Sphinx resolves against every class named `Type` in the project (five) rather than `typing.Type`, and silently linked to `pcapkit.const.l2tp.type.Type`, an L2TP field-type enum. Line 424 (`#: ~typing.Type[Dumper]: Dumper class.`, spelled out since [#709](https://github.com/JarryShaw/PyPCAPKit/issues/709) fixed it) is the live case: once [#684](https://github.com/JarryShaw/PyPCAPKit/issues/684) rendered `TraceFlow._foutio`, the built docs pointed at the wrong class with no warning. Both sites now spell it `~typing.Type[Dumper]`, as eight other files already do. Line 146 (the first line of `__output__`'s `#:` block) is fixed for the same reason but is currently inert, since the `# type:` comment at line 162 spells the same bare `Type[Dumper]`; it is insurance for whatever eventually renders that type. This pair is part of [#709](https://github.com/JarryShaw/PyPCAPKit/issues/709). Four more bare `Type` sites, the hand-written `:type:` fields at `docs/source/pcapkit/foundation/engines/engine.rst:40`, `.../reassembly/reassembly.rst:33` and `:43`, and `.../traceflow/traceflow.rst:40`, are fixed separately by [#714](https://github.com/JarryShaw/PyPCAPKit/pull/714) ([#709](https://github.com/JarryShaw/PyPCAPKit/issues/709)). +- `register_protocol`'s overwrite warning could claim a protocol was replaced with itself. The guard was correct (`incumbent is not protocol`, an identity check from [#681](https://github.com/JarryShaw/PyPCAPKit/pull/681)), but the message built both operands with a bare `repr()`. A factory defining a same-named closure-local class on every call (as `tests/protocols/test_construction_keyword_check_unit.py`'s `_protocol_class` does) produces two distinct classes sharing one `__module__` and `__qualname__`, so a real overwrite read "overwriting X with X." The message now compares the two reprs and, only when they coincide, appends each object's `id()`. `__module__`/`__qualname__` was rejected as the disambiguator: for the reported shape they are exactly what the coinciding repr already renders. The new `test_register_protocol_disambiguates_classes_sharing_a_repr` fails against the unfixed message ([#710](https://github.com/JarryShaw/PyPCAPKit/issues/710)). +- five more registrars warned on mere key presence rather than an actual overwrite, outside the wording of [#718](https://github.com/JarryShaw/PyPCAPKit/issues/718)'s identity guard (`incumbent is not None and incumbent is not new`, landed by [#726](https://github.com/JarryShaw/PyPCAPKit/pull/726)), whose issue named only code-keyed registrars: `register_engine`, `register_reassembly` and `register_traceflow` on `Extractor`, and `register_dumper` on both `Extractor` and `TraceFlow`. All five now compare the incumbent by identity before warning; both `register_dumper` sites compare only the stored dumper, so re-registering with just a new file extension stays silent too, judged defensible rather than comparing the full `(dumper, ext)` pair. Non-breaking: a correct caller sees strictly fewer warnings and no change to return value or exception. Five new tests pin the silent/warns-anyway split ([#739](https://github.com/JarryShaw/PyPCAPKit/issues/739)). ### pcapkit.protocols #### Added -- ESP parsing and construction [[RFC 4303](https://datatracker.ietf.org/doc/html/rfc4303)], with optional payload decryption and ICV verification through `cryptography` (`pip install pypcapkit[crypto]`) ([#378](https://github.com/JarryShaw/PyPCAPKit/pull/378)). Keys reach the dissector through a new caller-state channel, `pcapkit.corekit.context`, surfaced as the `context=` keyword on `extract()` and `Extractor`, carrying an `esp.SecurityAssociation`. Nothing here raises: with no association the SPI and sequence number are still reported and the ciphertext is left opaque, with `status=NO_SA`, and a failed decryption or ICV check is recorded on the parsed result the same way. Extended Sequence Numbers, TFC padding and anti-replay are not implemented, and an unsupported cipher or MAC is refused with a clear error rather than half-processed. -- SCTP as a transport protocol [[RFC 9260](https://datatracker.ietf.org/doc/html/rfc9260)]: all 13 chunk types, 8 chunk parameters and 13 error causes, with the CRC32c both recorded and verifiable -- it covers the SCTP packet alone, with no IP pseudo-header, so it can be checked from the SCTP bytes. Upper layers register on the DATA chunk's Payload Protocol Identifier through `register_sctp`, not on a port ([#379](https://github.com/JarryShaw/PyPCAPKit/pull/379)). +- ESP parsing and construction [[RFC 4303](https://datatracker.ietf.org/doc/html/rfc4303)], with optional payload decryption and ICV verification through `cryptography` (`pip install pypcapkit[crypto]`) ([#378](https://github.com/JarryShaw/PyPCAPKit/pull/378)). Keys reach the dissector through a new caller-state channel, `pcapkit.corekit.context`, surfaced as the `context=` keyword on `extract()` and `Extractor`, carrying an `esp.SecurityAssociation`. Nothing here raises: with no association the SPI and sequence number are still reported and the ciphertext is left opaque with `status=NO_SA`, and a failed decryption or ICV check is recorded on the parsed result the same way. Extended Sequence Numbers, TFC padding and anti-replay are not implemented, and an unsupported cipher or MAC is refused with a clear error rather than half-processed. +- SCTP as a transport protocol [[RFC 9260](https://datatracker.ietf.org/doc/html/rfc9260)]: all 13 chunk types, 8 chunk parameters and 13 error causes, with the CRC32c both recorded and verifiable (it covers the SCTP packet alone, with no IP pseudo-header). Upper layers register on the DATA chunk's Payload Protocol Identifier through `register_sctp`, not on a port ([#379](https://github.com/JarryShaw/PyPCAPKit/pull/379)). - NGAP over SCTP (3GPP TS 38.413), decoding aligned PER through `pycrate` (`pip install pypcapkit[NGAP]`) ([#251](https://github.com/JarryShaw/PyPCAPKit/discussions/251), [#417](https://github.com/JarryShaw/PyPCAPKit/pull/417)). Decoding is generic by ASN.1 shape rather than per-procedure, so all 81 elementary procedures and 438 protocol IEs work and a new 3GPP release needs no code change. Registered as a default on PPID 60 and 66, but only 60 decodes: PPID 66 is an NGAP PDU inside a DTLS record, and there is no DTLS dissector. `pycrate` is deliberately excluded from the `all` extra -- it is LGPL-2.1+ where this package is BSD-3-Clause, and lands some 238 MB to obtain one module. -- the Mobility Header registry, completed ([#383](https://github.com/JarryShaw/PyPCAPKit/pull/383), [#437](https://github.com/JarryShaw/PyPCAPKit/pull/437)). The [RFC 5568](https://datatracker.ietf.org/doc/html/rfc5568) fast-handover messages and options first, then all 24 registered message data types, all 71 registered options -- with nested sub-option registries for the flow identification, access network identifier, quality-of-service and LMA-controlled MAG parameter families -- and all 4 CGA extensions. The CGA Parameters option was the last to come off the generic handler, which is what let the four CGA extensions round-trip end to end, since it is the only option that can carry one on the wire. +- the Mobility Header registry, completed ([#383](https://github.com/JarryShaw/PyPCAPKit/pull/383), [#437](https://github.com/JarryShaw/PyPCAPKit/pull/437)): the [RFC 5568](https://datatracker.ietf.org/doc/html/rfc5568) fast-handover messages and options, then all 24 registered message data types, all 71 registered options (with nested sub-option registries for the flow identification, access network identifier, quality-of-service and LMA-controlled MAG parameter families) and all 4 CGA extensions. The CGA Parameters option, the only option that can carry a CGA extension on the wire, was the last to come off the generic handler, which is what let the four extensions round-trip end to end. - dispatch entries for dissectors that existed but were reachable from no registry ([#436](https://github.com/JarryShaw/PyPCAPKit/pull/436)): FTP-DATA on TCP 20, HTTP/1 on TCP 8080, HTTP on UDP 8080, `L2TPv2` on UDP 1701 and OSPF at `TransType` 89. `VLAN` became an abstract base with `C_Tag` (802.1Q) and `S_Tag` (802.1ad) as concrete subclasses, so a Q-in-Q frame no longer collapses into one opaque `Raw`; `L2TP` likewise became a base, with `L2TPv2` carrying the [RFC 2661](https://datatracker.ietf.org/doc/html/rfc2661) implementation. -- `Protocol`/`ProtocolBase` gain a `code=` class keyword for the next-layer dispatch registries -- `Link.__proto__`, `Internet.__proto__`, `TCP.__proto__`, `UDP.__proto__`, `SCTP.__proto__`, `Frame.__proto__` and `PCAPNG.__proto__` -- the same opt-in treatment [#514](https://github.com/JarryShaw/PyPCAPKit/issues/514) gave `Engine`, `Reassembly`, `TraceFlow` and `Dumper` above, extended to the one family that needed a registration key invented rather than merely un-guarded. Omitting `code` leaves a subclass unregistered, exactly as before this keyword existed: the built-in dispatch tables are still populated by literal assignment in each layer module, not by `__init_subclass__`, so nothing the library ships moves. `code` accepts a bare enum member, whose *type* infers the destination -- `EtherType` means `Link`, `TransType` means `Internet`, `PayloadProtocolIdentifier` means `SCTP`, and `LinkType` means *both* `Frame` **and** `PCAPNG`, deterministically, mirroring what `register_linktype` already does by hand -- or a `{destination: key}` mapping, required for a raw `int` such as a TCP/UDP port number, which cannot say by itself which transport it belongs to. Either form may appear in an iterable, so one declaration can register a class into several registries at once, e.g. a `L2TP` subclass reachable both by IP protocol number and by a UDP port. The explicit mapping form is accepted even for a key whose type could be inferred -- being more explicit than required is never an error. Inference refuses rather than guesses: an enum member whose type names no known destination raises `RegistryError` instead of silently doing nothing or picking an arbitrary registry, and an unrecognised class keyword raises `UnsupportedCall`, matching the other four families. Backed by the new `pcapkit.foundation.registry.protocols.register_protocol_code`, which can also be called directly to register a class that declined at class-definition time. This was written up as the mechanism [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548) (`TransType.L2TP` registered nowhere) needs, with fixing that issue described here as "now a one-declaration change". Investigating [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548) found otherwise -- 115 is an [RFC 3931](https://datatracker.ietf.org/doc/html/rfc3931) L2TPv3-over-IP header with no class to dispatch to, so the declaration would have pointed the [RFC 2661](https://datatracker.ietf.org/doc/html/rfc2661) parser at it. See the corresponding **Fixed** entry below; the mechanism itself is unaffected, and its worked example now names `L2TPv3` rather than `L2TPv2` ([#570](https://github.com/JarryShaw/PyPCAPKit/pull/570)). -- `tests/protocols/test_dispatch_reachability_unit.py`, the coverage [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548) asked for: every `ProtocolBase` descendant whose `__index__` returns an enum member is checked to be reachable under that code in the registry its enum *type* designates, read from the same `_CODE_DESTINATIONS` table backing `code=` so the two cannot drift. Where `test_dispatch_registry_unit.py` walks the 38 entries that exist and checks each parses, this walks the classes and catches one nothing registered at all -- the shape in which `OSPF` once shipped reachable from no table. 23 claims verified, no gaps; a companion case injects a gap and confirms the audit reports it, so the guard cannot rot into a permanently green no-op ([#548](https://github.com/JarryShaw/PyPCAPKit/issues/548)). -- `tests/protocols/transport/test_tcp_mptcp_join_flag_ordering_unit.py`, covering all three MP_JOIN layouts through the *public* constructor, the construct-pack-parse cycle for each, the stale-flags case that rules out a zero-valued default, the statement order itself, and controls that the parse path and the flag-independent options are unaffected. The gap it closes is why 100% statement and branch coverage of the two changed modules coexisted with a completely broken public path: the pre-existing cases reach `_make_mptcp_join` by assigning a Python `set` to `_flags` on a bare `TCP.__new__(TCP)`, which executes every branch while bypassing both the ordering and the accumulator's type. The now-stale `tcp-mptcp/MP_JOIN` entry is deleted from `EXPECTED_FAILURES`, and the MP_JOIN exclusion in `test_tcp_mptcp_subtype_unit.py` is lifted ([#587](https://github.com/JarryShaw/PyPCAPKit/issues/587)). -- `tests/protocols/test_option_generator_tcp_base_unit.py`, asserting that every `TCP_BASE` key is a parameter `TCP.make` declares -- derived from `inspect.signature`, so it catches a fourth misspelling nobody has made yet -- and that the segment the mapping builds carries the stated header in both the data model and the packed octets. Two of the three wrong names were invisible to any assertion about a *value*, the value asked for being equal to the default that was used instead, which is what the signature check is for ([#602](https://github.com/JarryShaw/PyPCAPKit/issues/602)). -- ILNP nonce sizing coverage in `tests/protocols/internet/test_ipv6_extension_unit.py`, one test per protocol, asserting the declared length, the exact packed octets and the construct-pack-parse cycle over ten nonces. The reason the existing suite missed this is that the only ILNP nonce it ever exercised was `0xFFFFFF` -- bit length 24, an exact multiple of eight, precisely where floor division and the ceiling agree -- the same blind spot that hid the identical typo in `numbers.py` behind bit lengths 8, 16, 24, 32 and 64. Every new case bar two deliberate controls therefore has a bit length that is *not* a multiple of eight, several of them below 256. The table also guards itself: the test asserts that at least six of its own values stay non-byte-aligned and that one stays below 256, so rounding them off to convenient constants later cannot quietly disarm the regression ([#601](https://github.com/JarryShaw/PyPCAPKit/issues/601)). -- `SOLUTION` parameter width coverage in `tests/protocols/internet/test_hip_unit.py`: eight widths asserting the declared length, the reader's acceptance and the construct-pack-parse cycle, plus a 57-bit case pinning the 20-octet length [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) requires of HIPv1. Five of the eight widths -- 1, 9, 12, 17 and 25 bits -- are deliberately *not* multiples of eight, because at a multiple of eight the defective `ceil(bits / 4)` coincides with the correct width, which is why every fixture that reached this builder passed through it unharmed; the round-trip case in `examples/generators/options.py` supplies no `random` or `solution` at all, so it exercised the formula at zero bits. 57 bits is what discriminates on the HIPv1 path for the same reason 64 does not: both formulas give 20 at a full-width value ([#608](https://github.com/JarryShaw/PyPCAPKit/issues/608)). +- `Protocol`/`ProtocolBase` gain a `code=` class keyword for the next-layer dispatch registries (`Link.__proto__`, `Internet.__proto__`, `TCP.__proto__`, `UDP.__proto__`, `SCTP.__proto__`, `Frame.__proto__`, `PCAPNG.__proto__`), the opt-in treatment [#514](https://github.com/JarryShaw/PyPCAPKit/issues/514) gave `Engine`, `Reassembly`, `TraceFlow` and `Dumper` above, extended to the one family that needed a registration key invented. Omitting `code` leaves a subclass unregistered, as before; the built-in tables are still filled by literal assignment in each layer module, so nothing the library ships moves. `code` accepts a bare enum member, whose *type* infers the destination (`EtherType` means `Link`, `TransType` means `Internet`, `PayloadProtocolIdentifier` means `SCTP`, and `LinkType` means *both* `Frame` **and** `PCAPNG`, mirroring `register_linktype`), or a `{destination: key}` mapping, required for a raw `int` such as a port number, which cannot say which transport it belongs to. Either form may appear in an iterable, so one declaration can register a class into several registries (e.g. a `L2TP` subclass reachable by IP protocol number and by a UDP port), and the mapping form is accepted even where inference would work. Inference refuses rather than guesses: an enum member whose type names no known destination raises `RegistryError`, and an unrecognised class keyword raises `UnsupportedCall`, matching the other four families. Backed by `pcapkit.foundation.registry.protocols.register_protocol_code`, also callable directly for a class that declined at definition time. It was written up as the mechanism [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548) (`TransType.L2TP` registered nowhere) needs, "now a one-declaration change"; investigating [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548) found otherwise -- 115 is an [RFC 3931](https://datatracker.ietf.org/doc/html/rfc3931) L2TPv3-over-IP header with no class to dispatch to, so the declaration would have pointed the [RFC 2661](https://datatracker.ietf.org/doc/html/rfc2661) parser at it (see the corresponding **Fixed** entry below). The mechanism is unaffected, and its worked example now names `L2TPv3` rather than `L2TPv2` ([#570](https://github.com/JarryShaw/PyPCAPKit/pull/570)). +- `tests/protocols/test_dispatch_reachability_unit.py`, the coverage [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548) asked for: every `ProtocolBase` descendant whose `__index__` returns an enum member must be reachable under that code in the registry its enum *type* designates, read from the same `_CODE_DESTINATIONS` table backing `code=` so the two cannot drift. Where `test_dispatch_registry_unit.py` walks the 38 entries that exist and checks each parses, this walks the classes and catches one nothing registered, the shape in which `OSPF` once shipped reachable from no table. 23 claims verified, no gaps; a companion case injects a gap and confirms the audit reports it ([#548](https://github.com/JarryShaw/PyPCAPKit/issues/548)). +- `tests/protocols/transport/test_tcp_mptcp_join_flag_ordering_unit.py`, covering all three MP_JOIN layouts through the *public* constructor, the construct-pack-parse cycle for each, the stale-flags case that rules out a zero-valued default, the statement order, and controls for the parse path and the flag-independent options. It closes the gap that let 100% statement and branch coverage of the two changed modules coexist with a broken public path: the old cases reached `_make_mptcp_join` by assigning a Python `set` to `_flags` on a bare `TCP.__new__(TCP)`, which executes every branch while bypassing both the ordering and the accumulator's type. The stale `tcp-mptcp/MP_JOIN` entry is deleted from `EXPECTED_FAILURES` and the MP_JOIN exclusion in `test_tcp_mptcp_subtype_unit.py` is lifted ([#587](https://github.com/JarryShaw/PyPCAPKit/issues/587)). +- `tests/protocols/test_option_generator_tcp_base_unit.py`, asserting that every `TCP_BASE` key is a parameter `TCP.make` declares (derived from `inspect.signature`, so it catches a fourth misspelling nobody has made yet) and that the built segment carries the stated header in both data model and packed octets. Two of the three wrong names were invisible to any assertion about a *value*, the value asked for equalling the default used instead ([#602](https://github.com/JarryShaw/PyPCAPKit/issues/602)). +- ILNP nonce sizing coverage in `tests/protocols/internet/test_ipv6_extension_unit.py`, one test per protocol, asserting the declared length, the exact packed octets and the construct-pack-parse cycle over ten nonces. The existing suite missed the defect because its only ILNP nonce was `0xFFFFFF` (bit length 24, a multiple of eight, where floor division and the ceiling agree), the blind spot that hid the same typo in `numbers.py` behind bit lengths 8, 16, 24, 32 and 64. Every new case bar two controls therefore has a bit length that is *not* a multiple of eight, several below 256, and the test asserts that at least six stay non-byte-aligned and one stays below 256, so rounding them off later cannot disarm it ([#601](https://github.com/JarryShaw/PyPCAPKit/issues/601)). +- `SOLUTION` parameter width coverage in `tests/protocols/internet/test_hip_unit.py`: eight widths asserting the declared length, the reader's acceptance and the construct-pack-parse cycle, plus a 57-bit case pinning the 20-octet length [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) requires of HIPv1. Five widths (1, 9, 12, 17 and 25 bits) are deliberately *not* multiples of eight, because at a multiple of eight the defective `ceil(bits / 4)` coincides with the correct width, which is why every fixture reaching this builder passed; the round-trip case in `examples/generators/options.py` supplies no `random` or `solution`, so it exercised the formula at zero bits. 57 bits discriminates on the HIPv1 path where 64 does not: both formulas give 20 at a full-width value ([#608](https://github.com/JarryShaw/PyPCAPKit/issues/608)). #### Changed - renames with no compatibility alias left behind: PCAP-NG `Option` subclasses spell the namespace class keyword `ns=` instead of `namespace=` ([#439](https://github.com/JarryShaw/PyPCAPKit/issues/439)). -- `tests/protocols/transport/test_tcp_udp_unit.py` now reaches the MP_JOIN dispatchers through `TCP()` itself, instead of assigning a Python `set` to `_flags` on a bare `TCP.__new__(TCP)`. A `set` answers the membership tests `_make_mptcp_join` and `_read_mptcp_join` use, so every flag branch ran and both TCP modules read 100% statement and branch coverage -- while the attribute had neither the `aenum.IntFlag` type production assigns nor the ordering that governs when it exists at all, which is how [#587](https://github.com/JarryShaw/PyPCAPKit/issues/587) stayed invisible behind that number and how the `cast('Enum_Flags', 0)` no-op behind it went unnoticed too. Measured on the rewrite, against the 17 tests of that file: revert [#587](https://github.com/JarryShaw/PyPCAPKit/issues/587)'s hoist and two of them fail with `AttributeError: 'TCP' object has no attribute '_flags'` where all 17 passed before; restore the `cast` and two fail with `TypeError: argument of type 'int' is not a container or iterable`, again where all 17 passed. The library is unchanged and the file's tests still pass, so the coverage numbers do not move -- the point is what the same numbers are now worth ([#603](https://github.com/JarryShaw/PyPCAPKit/issues/603)). -- building a protocol through its constructor with a keyword that names nothing now raises `UnsupportedCall` instead of discarding it. **This is a behaviour change to a public API**: every `make` in the tree ends its signature with `**kwargs` and reads nothing out of it, so until now a misspelled keyword was accepted, dropped, and the field it named kept its default -- wrong octets, with nothing said. That is what [#602](https://github.com/JarryShaw/PyPCAPKit/issues/602) cost: `examples/generators/options.py` asked for `seq=1` where `TCP.make` spells the parameter `seq_no`, and 25 generated fixture frames carried sequence number `0` against an empty `warnings` list. [#541](https://github.com/JarryShaw/PyPCAPKit/issues/541) and [#556](https://github.com/JarryShaw/PyPCAPKit/issues/556) were the same silence. The schema layer has never been so permissive -- `Schema.__update__` warns `UnknownFieldWarning` for a field it does not know -- and the asymmetry between the two halves of the same construction is what this closes. Checked in `ProtocolBase.__init__` rather than in `make`, because `make` is not the only consumer of the keywords it is handed: `__post_init__` passes one `**kwargs` to the construction *and* to the parse of what it has just constructed, so a keyword declared only by `read` legitimately travels through `make` -- `HIP.read` declares `extension` where `HIP.make` does not, and the option generator depends on it. The accepted set is therefore the union of every keyword-taking parameter of `make`, `read`, `pack`, `unpack`, `__post_init__` and `__init__` across the whole MRO, computed once per class from `inspect.signature`. Parsing is deliberately untouched, since there the keywords are whatever the engines and the four `_import_next_layer` implementations forward and a protocol cannot know which of its ancestors' its parent passed on -- and nothing was ever lost that way, a dropped parse keyword changing how a packet is read rather than what its octets say. Two escapes exist for the shapes a signature cannot express, both opt-in per class through a new `__keywords__`: a set, for a keyword read out of `**kwargs` by name as `ESP.read` does with `packet`; and `None`, for a dispatcher whose real signature belongs to a class chosen at call time, which is exactly `HTTP.make` forwarding to `HTTPv1`/`HTTPv2` and the one place in the tree that uses it. The message names the near neighbour it found, so `seq` reports *did you mean 'seq_no'?*. `from_data` warns `UnknownFieldWarning` where a caller would be raised at, because the keywords there are whatever `_make_data` returned rather than anything anybody typed, so the defect is a key of that mapping disagreeing with the signature it is spread into and the person who meets it is not the person who can fix it. Expect this to surface latent bugs in code that has been quietly losing a field, which is the point. It surfaced four in this repository, all the residue of [#602](https://github.com/JarryShaw/PyPCAPKit/issues/602) and all fixed here: the `_TCP_BASE` of `examples/generators/dispatch.py` and three stale copies of it under `tests/protocols/transport/`, each still passing `seq`/`ack_flag`/`urgent_pointer` and so building segments with sequence number `0` where they read as `1`. It surfaced three more that are reported rather than fixed, being a defect per protocol rather than one in this mechanism: `Frame._make_data` returns `ts_src` where `make` declares `ts_sec`, `L2TPv2._make_data` returns `prio` where it declares `priority`, and `Header._make_data` returns a `magic_number` that `Header.make` does not take at all -- so `from_data` has been dropping a frame's timestamp, an L2TPv2 priority bit and a capture's byte order, and now says so. One limitation worth knowing rather than discovering: a *direct* `SomeProtocol.make(...)` call is not checked and still discards in silence, since the check sits where every producer's keywords converge rather than inside each of the 30 `make` implementations -- `object.__new__(cls).make(**kwargs)` is the idiom that reaches it, and `HTTP.make` uses it to reach its versioned implementation ([#617](https://github.com/JarryShaw/PyPCAPKit/issues/617)). +- `tests/protocols/transport/test_tcp_udp_unit.py` now reaches the MP_JOIN dispatchers through `TCP()` itself, instead of assigning a Python `set` to `_flags` on a bare `TCP.__new__(TCP)`. A `set` answers the membership tests `_make_mptcp_join` and `_read_mptcp_join` use, so every flag branch ran and both TCP modules read 100% coverage, while the attribute had neither the `aenum.IntFlag` type production assigns nor the ordering that governs when it exists -- how [#587](https://github.com/JarryShaw/PyPCAPKit/issues/587) stayed invisible behind that number, and the `cast('Enum_Flags', 0)` no-op behind it. Reverting [#587](https://github.com/JarryShaw/PyPCAPKit/issues/587)'s hoist now fails two of the file's 17 tests with `AttributeError: 'TCP' object has no attribute '_flags'`, and restoring the `cast` fails two with `TypeError: argument of type 'int' is not a container or iterable`; all 17 passed before. The library is unchanged and coverage does not move -- the point is what the same numbers are now worth ([#603](https://github.com/JarryShaw/PyPCAPKit/issues/603)). +- building a protocol through its constructor with a keyword that names nothing now raises `UnsupportedCall` instead of discarding it. **This is a behaviour change to a public API**: every `make` in the tree ends its signature with `**kwargs` and reads nothing out of it, so a misspelled keyword was accepted, dropped, and the field it named kept its default -- wrong octets, nothing said. That is what [#602](https://github.com/JarryShaw/PyPCAPKit/issues/602) cost: `examples/generators/options.py` asked for `seq=1` where `TCP.make` spells it `seq_no`, and 25 generated fixture frames carried sequence number `0`. [#541](https://github.com/JarryShaw/PyPCAPKit/issues/541) and [#556](https://github.com/JarryShaw/PyPCAPKit/issues/556) were the same silence. The schema layer was never so permissive (`Schema.__update__` warns `UnknownFieldWarning`), and that asymmetry is what this closes. The check sits in `ProtocolBase.__init__` rather than `make`, because `__post_init__` passes one `**kwargs` to the construction *and* to the parse of what it has just constructed, so a keyword declared only by `read` legitimately travels through `make` (`HIP.read` declares `extension`, which `HIP.make` does not, and the option generator depends on it). The accepted set is the union of every keyword-taking parameter of `make`, `read`, `pack`, `unpack`, `__post_init__` and `__init__` across the MRO, computed once per class from `inspect.signature`. Parsing is deliberately untouched: there the keywords are whatever the engines and the four `_import_next_layer` implementations forward, a protocol cannot know which its parent passed on, and a dropped parse keyword changes how a packet is read, not what its octets say. Two opt-in escapes exist per class through a new `__keywords__`: a set, for a keyword read out of `**kwargs` by name (`ESP.read` with `packet`); and `None`, for a dispatcher whose real signature belongs to a class chosen at call time (only `HTTP.make` forwarding to `HTTPv1`/`HTTPv2`). The message names the near neighbour (`seq` reports *did you mean 'seq_no'?*). `from_data` warns `UnknownFieldWarning` rather than raising, because its keywords are whatever `_make_data` returned, so the defect is a key of that mapping disagreeing with the signature it is spread into, and the person who meets it cannot fix it. This surfaced four latent bugs in this repository, all residue of [#602](https://github.com/JarryShaw/PyPCAPKit/issues/602) and fixed here: the `_TCP_BASE` of `examples/generators/dispatch.py` and three stale copies under `tests/protocols/transport/`, each building segments with sequence number `0`. Three more are reported, not fixed, being a defect per protocol: `Frame._make_data` returns `ts_src` where `make` declares `ts_sec`, `L2TPv2._make_data` returns `prio` where it declares `priority`, and `Header._make_data` returns a `magic_number` that `Header.make` does not take, so `from_data` has been dropping a frame's timestamp, an L2TPv2 priority bit and a capture's byte order, and now says so. One limitation: a *direct* `SomeProtocol.make(...)` call is not checked and still discards silently, since the check sits where every producer's keywords converge rather than inside each of the 30 `make` implementations; `object.__new__(cls).make(**kwargs)`, the idiom `HTTP.make` uses, reaches it ([#617](https://github.com/JarryShaw/PyPCAPKit/issues/617)). #### Fixed -- next-layer, option, chunk, block and parameter dispatch all read `defaultdict` registries, so a lookup miss inserted the key into class-level state shared by every later instance, after which a legitimate `register_*` call warned that the code was already registered. Every read now goes through a lookup that does not grow the table, and `IPv4.__option__` and `HIP.__parameter__` became inspectable class attributes rather than names assembled at call time ([#426](https://github.com/JarryShaw/PyPCAPKit/pull/426), [#428](https://github.com/JarryShaw/PyPCAPKit/pull/428), [#429](https://github.com/JarryShaw/PyPCAPKit/issues/429), [#434](https://github.com/JarryShaw/PyPCAPKit/pull/434)). One break comes with it: a tuple-registered handler pair written to the documented `OptionParser`/`OptionConstructor` signature now works where it could previously never be called at all, and a pair written with an explicit leading `self` -- the only shape that used to work -- now does not. -- the identical defect one layer up, in the schema layer's own `EnumSchema.registry`: `Option.registry[code]` for an unregistered `code` inserted the default schema under that code, so a single lookup made an unassigned TCP option number, e.g. `156`, read back as registered for the rest of the process. `EnumSchema.__enum__` is now built (or, when a subclass seeds it manually in its own class body -- `PCAPNG.Option`'s namespaced mapping, `TCP.MPTCP`'s plain one) as a retention-safe mapping that still returns the registered default on a miss, it just stops recording it; `.registry` keeps returning the same object it always did, so nothing that held a reference to it is affected ([#555](https://github.com/JarryShaw/PyPCAPKit/issues/555)). -- on Python 3.10 and older, no `Schema` subclass got its own `_abc_impl`: all of them fell through to `collections.abc.Mapping`'s, so a single `isinstance` or `issubclass` answer poisoned every later question about that class for the rest of the process. A terminating PCAP-NG `EndRecord` tested `True` as an `IPv4Record` ([#439](https://github.com/JarryShaw/PyPCAPKit/issues/439)). -- construction, which was broken in several places at once: the generated typed `__init__` was never installed, so `__post_init__` did not run and a schema built from a subset of its fields could not be packed at all -- `UDP(srcport=53, dstport=5353)` now packs ([#430](https://github.com/JarryShaw/PyPCAPKit/pull/430)); IPv6 and Mobility Header option padding was wrong, leaving construction wholly broken ([#398](https://github.com/JarryShaw/PyPCAPKit/pull/398)); `HTTP.make` called the versioned `make` unbound, so every real call raised `TypeError` ([#452](https://github.com/JarryShaw/PyPCAPKit/issues/452), [#462](https://github.com/JarryShaw/PyPCAPKit/pull/462)); and `IPv4._make_data` returned the fragment offset in octets where the wire wants 8-octet units, and read `data.options` on a packet that has none ([#494](https://github.com/JarryShaw/PyPCAPKit/issues/494), [#499](https://github.com/JarryShaw/PyPCAPKit/pull/499)). -- a truncated or under-declared area no longer parses "successfully", and no longer wedges the process. The option and list loops could spin forever with no exception on a truncated area, reachable from untrusted input through HOPOPT, IPv6-Opts, MH, HIP and SCTP; each iteration must now advance the stream by at least one octet, and the error names the option, the offset and the octets remaining ([#431](https://github.com/JarryShaw/PyPCAPKit/issues/431), [#432](https://github.com/JarryShaw/PyPCAPKit/pull/432)). Separately, wire-derived lengths in `ipv6_opts`, CALIPSO, MPL, REG_INFO and four HIP list callbacks underflowed below zero, which `ListField`'s own `while length > 0` then turned into a silent empty list; they are floored and raise instead ([#449](https://github.com/JarryShaw/PyPCAPKit/pull/449), [#456](https://github.com/JarryShaw/PyPCAPKit/pull/456), [#460](https://github.com/JarryShaw/PyPCAPKit/pull/460), [#463](https://github.com/JarryShaw/PyPCAPKit/issues/463)). -- field widths and units, each measured against the specification rather than inferred: HIP's `TRANSPORT_FORMAT_LIST`, `NAT_TRAVERSAL_MODE` and `ESP_TRANSFORM` list entries are two octets, not one [[RFC 7401](https://datatracker.ietf.org/doc/html/rfc7401), [RFC 5770](https://datatracker.ietf.org/doc/html/rfc5770), [RFC 7402](https://datatracker.ietf.org/doc/html/rfc7402)] ([#463](https://github.com/JarryShaw/PyPCAPKit/issues/463), [#472](https://github.com/JarryShaw/PyPCAPKit/issues/472)); the MN-ID option sizes from its subtype, not from `identifier`'s Python type ([#448](https://github.com/JarryShaw/PyPCAPKit/issues/448), [#464](https://github.com/JarryShaw/PyPCAPKit/pull/464), [#467](https://github.com/JarryShaw/PyPCAPKit/issues/467)); IPv6-Route's `Hdr Ext Len` is computed in 8-octet units on both sides [[RFC 8200](https://datatracker.ietf.org/doc/html/rfc8200)] ([#487](https://github.com/JarryShaw/PyPCAPKit/issues/487), [#489](https://github.com/JarryShaw/PyPCAPKit/pull/489)); and the Fast Binding Update and Acknowledgment Lifetimes are plain seconds [[RFC 5568](https://datatracker.ietf.org/doc/html/rfc5568)], not [RFC 6275](https://datatracker.ietf.org/doc/html/rfc6275)'s four-second units ([#502](https://github.com/JarryShaw/PyPCAPKit/pull/502)). -- the last four call sites where a `_make_*` method converts an address itself, ahead of the schema, so the same `bool`-as-`int` mistake reached them too, unguarded by the schema-layer fix above: `ARP._make_proto_resolve`, `IPv6_Route._make_data_type_rpl` (which also derives its RPL compression lengths from the laundered value, corrupting the address list alongside the exception it should have raised), the `dst` parameter of `IPv6_Route.make`, and the latent `OSPF._make_id_numbers`, which nothing calls yet but would have inherited the defect regardless. All four now go through `parse_ip_address`, the helper [#539](https://github.com/JarryShaw/PyPCAPKit/pull/539) and [#552](https://github.com/JarryShaw/PyPCAPKit/issues/552) route their own sites through, completing the programme [#481](https://github.com/JarryShaw/PyPCAPKit/pull/481) and [#508](https://github.com/JarryShaw/PyPCAPKit/issues/508) started. Pinning `version=6` on the two `IPv6_Route` sites is a second, smaller behaviour change beyond bool rejection: both previously converted a plain integer through the bare, family-inferring `ipaddress.ip_address`, which let `ip=[258]` pack a 4-octet IPv4 address (`0.0.1.2`) inside an IPv6-only header; it now packs the 16-octet IPv6 form (`::102`) instead, which is what an IPv6-only header should hold regardless of what an integer argument happens to fit as an IPv4 address ([#540](https://github.com/JarryShaw/PyPCAPKit/issues/540)). +- next-layer, option, chunk, block and parameter dispatch all read `defaultdict` registries, so a lookup miss inserted the key into class-level state shared by every later instance, after which a legitimate `register_*` call warned that the code was already registered. Every read now goes through a lookup that does not grow the table, and `IPv4.__option__` and `HIP.__parameter__` became inspectable class attributes rather than names assembled at call time ([#426](https://github.com/JarryShaw/PyPCAPKit/pull/426), [#428](https://github.com/JarryShaw/PyPCAPKit/pull/428), [#429](https://github.com/JarryShaw/PyPCAPKit/issues/429), [#434](https://github.com/JarryShaw/PyPCAPKit/pull/434)). One break: a tuple-registered handler pair written to the documented `OptionParser`/`OptionConstructor` signature now works where it could never be called, and a pair written with an explicit leading `self` -- the only shape that used to work -- now does not. +- the identical defect one layer up, in the schema layer's `EnumSchema.registry`: `Option.registry[code]` for an unregistered `code` inserted the default schema under that code, so a single lookup made an unassigned TCP option number such as `156` read back as registered for the rest of the process. `EnumSchema.__enum__` is now built (or, when a subclass seeds it manually, as `PCAPNG.Option`'s namespaced mapping and `TCP.MPTCP`'s plain one do) as a retention-safe mapping that still returns the registered default on a miss but stops recording it; `.registry` returns the same object as before, so nothing holding a reference is affected ([#555](https://github.com/JarryShaw/PyPCAPKit/issues/555)). +- on Python 3.10 and older, no `Schema` subclass got its own `_abc_impl`: all fell through to `collections.abc.Mapping`'s, so a single `isinstance` or `issubclass` answer poisoned every later question about that class for the rest of the process. A terminating PCAP-NG `EndRecord` tested `True` as an `IPv4Record` ([#439](https://github.com/JarryShaw/PyPCAPKit/issues/439)). +- construction, broken in several places at once: the generated typed `__init__` was never installed, so `__post_init__` did not run and a schema built from a subset of its fields could not be packed -- `UDP(srcport=53, dstport=5353)` now packs ([#430](https://github.com/JarryShaw/PyPCAPKit/pull/430)); IPv6 and Mobility Header option padding was wrong, leaving construction wholly broken ([#398](https://github.com/JarryShaw/PyPCAPKit/pull/398)); `HTTP.make` called the versioned `make` unbound, so every real call raised `TypeError` ([#452](https://github.com/JarryShaw/PyPCAPKit/issues/452), [#462](https://github.com/JarryShaw/PyPCAPKit/pull/462)); and `IPv4._make_data` returned the fragment offset in octets where the wire wants 8-octet units, and read `data.options` on a packet that has none ([#494](https://github.com/JarryShaw/PyPCAPKit/issues/494), [#499](https://github.com/JarryShaw/PyPCAPKit/pull/499)). +- a truncated or under-declared area no longer parses "successfully", and no longer wedges the process. The option and list loops could spin forever with no exception on a truncated area, reachable from untrusted input through HOPOPT, IPv6-Opts, MH, HIP and SCTP; each iteration must now advance the stream by at least one octet, and the error names the option, the offset and the octets remaining ([#431](https://github.com/JarryShaw/PyPCAPKit/issues/431), [#432](https://github.com/JarryShaw/PyPCAPKit/pull/432)). Separately, wire-derived lengths in `ipv6_opts`, CALIPSO, MPL, REG_INFO and four HIP list callbacks underflowed below zero, which `ListField`'s own `while length > 0` turned into a silent empty list; they are floored and raise instead ([#449](https://github.com/JarryShaw/PyPCAPKit/pull/449), [#456](https://github.com/JarryShaw/PyPCAPKit/pull/456), [#460](https://github.com/JarryShaw/PyPCAPKit/pull/460), [#463](https://github.com/JarryShaw/PyPCAPKit/issues/463)). +- field widths and units, each measured against the specification: HIP's `TRANSPORT_FORMAT_LIST`, `NAT_TRAVERSAL_MODE` and `ESP_TRANSFORM` list entries are two octets, not one [[RFC 7401](https://datatracker.ietf.org/doc/html/rfc7401), [RFC 5770](https://datatracker.ietf.org/doc/html/rfc5770), [RFC 7402](https://datatracker.ietf.org/doc/html/rfc7402)] ([#463](https://github.com/JarryShaw/PyPCAPKit/issues/463), [#472](https://github.com/JarryShaw/PyPCAPKit/issues/472)); the MN-ID option sizes from its subtype, not from `identifier`'s Python type ([#448](https://github.com/JarryShaw/PyPCAPKit/issues/448), [#464](https://github.com/JarryShaw/PyPCAPKit/pull/464), [#467](https://github.com/JarryShaw/PyPCAPKit/issues/467)); IPv6-Route's `Hdr Ext Len` is computed in 8-octet units on both sides [[RFC 8200](https://datatracker.ietf.org/doc/html/rfc8200)] ([#487](https://github.com/JarryShaw/PyPCAPKit/issues/487), [#489](https://github.com/JarryShaw/PyPCAPKit/pull/489)); and the Fast Binding Update and Acknowledgment Lifetimes are plain seconds [[RFC 5568](https://datatracker.ietf.org/doc/html/rfc5568)], not [RFC 6275](https://datatracker.ietf.org/doc/html/rfc6275)'s four-second units ([#502](https://github.com/JarryShaw/PyPCAPKit/pull/502)). +- the last four call sites where a `_make_*` method converts an address itself, ahead of the schema, so the same `bool`-as-`int` mistake reached them, unguarded by the schema-layer fix above: `ARP._make_proto_resolve`, `IPv6_Route._make_data_type_rpl` (which also derived its RPL compression lengths from the laundered value, corrupting the address list), the `dst` parameter of `IPv6_Route.make`, and the latent `OSPF._make_id_numbers`. All four now go through `parse_ip_address`, the helper [#539](https://github.com/JarryShaw/PyPCAPKit/pull/539) and [#552](https://github.com/JarryShaw/PyPCAPKit/issues/552) route their own sites through, completing the programme [#481](https://github.com/JarryShaw/PyPCAPKit/pull/481) and [#508](https://github.com/JarryShaw/PyPCAPKit/issues/508) started. Pinning `version=6` on the two `IPv6_Route` sites is a second, smaller behaviour change: both converted a plain integer through the family-inferring `ipaddress.ip_address`, so `ip=[258]` packed a 4-octet IPv4 address (`0.0.1.2`) inside an IPv6-only header; it now packs the 16-octet `::102` ([#540](https://github.com/JarryShaw/PyPCAPKit/issues/540)). - PCAP and PCAP-NG output and parsing: `bytes(frame)` returned the *next* frame's octets, `files=True` wrote names like `Frame 1..json`, and a PCAP-NG `timestamp_epoch` was shifted by the reading host's timezone ([#403](https://github.com/JarryShaw/PyPCAPKit/pull/403)); seven further parser defects ([#341](https://github.com/JarryShaw/PyPCAPKit/issues/341)--[#347](https://github.com/JarryShaw/PyPCAPKit/issues/347), [#371](https://github.com/JarryShaw/PyPCAPKit/pull/371)) and, on the write path, four more block-parsing ones ([#388](https://github.com/JarryShaw/PyPCAPKit/pull/388)); `BitField` packed every named bit as set ([#359](https://github.com/JarryShaw/PyPCAPKit/issues/359), [#374](https://github.com/JarryShaw/PyPCAPKit/pull/374)); the extension-header walk failed to advance past the last IPv6 extension header ([#348](https://github.com/JarryShaw/PyPCAPKit/issues/348), [#373](https://github.com/JarryShaw/PyPCAPKit/pull/373)); and IPv6 fragment offsets went unscaled, with reassembly keyed on the flow label -- optional, and routinely zero, so distinct datagrams collapsed together -- rather than on the fragment identification ([#389](https://github.com/JarryShaw/PyPCAPKit/pull/389)). -- `TCP._make_mptcp_addaddr` could not build an `ADD_ADDR` option end to end: its `kind=`/`length=` arguments were rejected with `UnknownFieldWarning` and silently dropped, and `.pack()` then raised `KeyError: 'length'` from `port`'s own condition, `pkt['length'] in (10, 22)`. The cause was one layer up -- `MPTCP`, the base class every Multipath TCP subtype schema inherits, declared `kind` and `length` only under `typing.TYPE_CHECKING` rather than as real fields, unlike `Option`, which every non-Multipath TCP option schema inherits instead. That silently dropped `kind=`/`length=` for every `_make_mptcp_*` constructor, not only `ADD_ADDR`'s, so `MPTCP` now declares both for real, the same way `Option` already did ([#541](https://github.com/JarryShaw/PyPCAPKit/issues/541)). The same missing fields broke parsing too: with no `kind`/`length` fields ahead of it, a Multipath TCP subtype schema's own leading field read the `kind` octet itself rather than the octet meant for it, an off-by-two in field alignment rather than a wire-format change -- a correct sender's octets were always right, only this library's reading of them was shifted. Spec-correct `ADD_ADDR` and `MP_PRIO` options failed to parse with `FieldError: TCP: [OptNo 30] 3 invalid IP version` and `KeyError: 'length'` respectively; both parse correctly now. -- two dropped-keyword/wrong-cast defects flagged in review during this release and never filed until now: HIP's `_make_param_encrypted` passed `cipher=` to a schema with no such field, so the value was silently dropped and an AES-cipher `ENCRYPTED` parameter built through `make` packed without its IV; and IPv6-Route's `RPL.post_process`, which runs on every `Schema.pack` and not only after a parse, assumed `self.addresses` was still the concatenated `bytes` a parse leaves it as, and raised slicing the `list[bytes]` a `make`-built multi-address header actually holds there ([#556](https://github.com/JarryShaw/PyPCAPKit/issues/556)). -- two more defects [#541](https://github.com/JarryShaw/PyPCAPKit/issues/541) exposed rather than caused, both since it let construction reach code that had never run before. `MPTCP.subtype` was still `typing.TYPE_CHECKING`-only, an annotation rather than a field, so `TCP(options=[(Enum_Option.Multipath_TCP, ...)])` raised `AttributeError: ... has no attribute 'subtype'` for every subtype but `MP_JOIN`: the convenience constructor builds a schema in memory and reads it straight back through `_read_mptcp_*` with no byte round trip, so `_MPTCP.post_process` -- the only code that ever set `subtype` -- never ran. Fixed on the construction path (`TCP._make_mode_mp`) rather than by adding a third real field the way `kind`/`length` got in [#541](https://github.com/JarryShaw/PyPCAPKit/issues/541): unlike those two, `subtype` is already packed as 4 bits of each subtype's own `test` bitfield, and a second, independent field for the same bits would either double-encode them or need a "derive, don't pack" field kind this library does not have ([#566](https://github.com/JarryShaw/PyPCAPKit/issues/566)). Separately, `_make_mptcp_capable` wrote `length=20 if rkey is None else 32` where [RFC 8684](https://datatracker.ietf.org/doc/html/rfc8684) section 3.1 gives 12 and 20, and `MPTCPCapable.rkey`'s own condition (`pkt['length'] != 32`) dropped the receiver's key for exactly the length the maker used to mean "key present" -- so a spec-correct, key-absent MP_CAPABLE could not be built at all, and a key-present one silently lost its key on the wire. Both, and the matching guard in `_read_mptcp_capable`, now agree on 12/20. **This changes MP_CAPABLE's packed output**: a 20-octet, key-present option built or parsed under the old code becomes 12 octets with no key, or 20 octets with the key actually present, depending on which the caller meant ([#567](https://github.com/JarryShaw/PyPCAPKit/issues/567)). -- IPv6-Route's RPL routing data was five octets wide at the front where [RFC 6554](https://datatracker.ietf.org/doc/html/rfc6554) section 3 gives four, and three further defects sat stacked behind it. `CmprI`, `CmprE` and `Pad` are each a 4-bit field, sharing one 32-bit word with a 20-bit `Reserved`, but the schema declared `cmpr_i` and `cmpr_e` as whole octets -- so a constructed two-address header packed to 41 octets while the `Hdr Ext Len` of 5 derived from that inflated data area declared 48. Correcting the width is a **wire-format change**, and it reshapes the schema: `RPL(cmpr_i=..., cmpr_e=...)` is now `RPL(cmpr={'cmpr_i': ..., 'cmpr_e': ...})`, beside the `pad={'pad_len': ...}` that was already there. Behind it, the reader's `header.length % 16` guard read `Hdr Ext Len` as an octet count and assumed 16-octet addresses, which an SRH only carries when `CmprI` and `CmprE` are both 0 -- the unit confusion [#487](https://github.com/JarryShaw/PyPCAPKit/issues/487) fixed for Source Route and Type 2, flagged and deliberately left by [#489](https://github.com/JarryShaw/PyPCAPKit/pull/489) for want of a working RPL round trip to validate a replacement against. It is replaced by section 4.2's own address-count arithmetic, `n = (((Hdr Ext Len * 8) - Pad - (16 - CmprE)) / (16 - CmprI)) + 1`, which the reader now requires to close -- non-negative and whole. `RPL.post_process` subtracted `pad_len` a second time from a buffer whose own length callback had already taken it off, losing one address per `16 - CmprI` octets of padding, and set `ip` only when it had parsed octets -- so once the guard stopped rejecting every constructed header, `IPv6_Route(type=..., data={'ip': [...]})` raised a bare `AttributeError: 'RPL' object has no attribute 'ip'` from the reader. And `_make_data_type_rpl` computed `Pad` as `8 - length % 8` without the outer `% 8`, so an already-aligned address vector was handed a full 8 octets of padding where section 3 requires that when `CmprI` and `CmprE` are both 0, `Pad` MUST carry a value of 0. The narrower `cmpr_i` also removes a latent divide-by-zero: a whole octet could hold 16, making `16 - CmprI` zero, where a 4-bit field tops out at 15. `ipv6-route-type/RPL_Source_Route_Header` round-trips now and its `EXPECTED_FAILURES` entry is deleted; as with the guard it replaces, none of this has been checked against a real RPL capture ([#564](https://github.com/JarryShaw/PyPCAPKit/issues/564)). -- `L2TPv2` parsed any version nibble, so an L2TPv3 datagram was reported as v2 with a tunnel and session ID read out of v3's Control Connection ID. [RFC 2661](https://datatracker.ietf.org/doc/html/rfc2661) §3.1 fixes `Ver` at 2 and reserves 1 for L2F, and `L2TPv2.version` already documented that "a datagram carrying any other value is a different protocol reached through a different class" -- but nothing enforced it, so the hard-coded `Literal[2]` property and `info.version` disagreed on the same octets, answering 2 and 3. `read` now raises `ProtocolError`, which degrades the payload to `Raw` through the existing `beholder` path with the reason recorded. **This affects real captures**: [RFC 3931](https://datatracker.ietf.org/doc/html/rfc3931) §4.1.2 puts L2TPv3 on port 1701 too, the port `UDP.__proto__` already binds, so `Ethernet:IPv4:UDP:L2TPv2:Raw` with invented field values becomes `Ethernet:IPv4:UDP:Raw` with the octets preserved. - -This is the resolution of [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548), which reported `TransType.L2TP` (115) as registered nowhere and proposed binding `L2TPv2` there. That binding is wrong rather than merely awkward: [RFC 3931](https://datatracker.ietf.org/doc/html/rfc3931) §4.1.1 gives 115 to *L2TPv3 over IP*, whose session header is "free of any restrictions imposed by coexistence with L2TPv2 and L2F" and carries **no version nibble at all**, so a v2 parser cannot even detect that the datagram is not its own. Measured, it produced `version=4`, `tunnelid=0x5678` and `sessionid=0xff03` from the top half of a Session ID and two octets of the PPP frame behind it. 115 is a missing *class*, not a missing registration, and stays unbound until an `L2TPv3` class exists; no dissector was invented here to fill it. The reasoning is now recorded in `pcapkit.protocols.link.l2tp` rather than only in a test, and `register_protocol_code`'s worked example -- which named `L2TPv2` at 115 -- names `L2TPv3` instead ([#548](https://github.com/JarryShaw/PyPCAPKit/issues/548)). -- building any `MP_JOIN` option raised `AttributeError: 'TCP' object has no attribute '_flags'`. `TCP.make` constructed the options *before* it assigned the `self._flags` that the option makers read, and `_make_mptcp_join` branches on that attribute to choose between the three layouts [RFC 8684](https://datatracker.ietf.org/doc/html/rfc8684) section 3.2 gives for MP_JOIN -- figure 5 for SYN at 12 octets, figure 6 for SYN/ACK at 16, figure 7 for ACK at 24. The parse path was never affected: `read` assigns the flags before it parses the options, so the identical branches in `_read_mptcp_join` always had them. Fixed by hoisting the flag resolution above the `_make_tcp_options` call, leaving only the header's data-offset computation -- which genuinely needs the options' total length -- after it. **Not** fixed by giving `_flags` a zero default, which would have been worse: `Protocol.pack` is public and calls `make`, so an instance that had already parsed a segment always had the attribute set, and it silently built the option for the segment it had *read* rather than the one it was asked to write. Measured pre-fix, a parsed MP_JOIN-SYN instance asked to pack an MP_JOIN-ACK segment emitted an ACK header carrying figure 5's 12-octet SYN option, with the caller's 20-octet HMAC -- the whole authentication payload of figure 7's form -- replaced by an all-zero phantom token and nonce, and nothing raised. The crash was the benign symptom; a default value fixes only that and leaves the silent corruption. Hoisting also made one branch reachable for the first time, an MP_JOIN asked for on a segment with neither SYN nor ACK set: the accumulator was seeded with `cast('Enum_Flags', 0)`, and `typing.cast` being a runtime no-op, `self._flags` stayed a plain `int` on which the first membership test raised `TypeError` rather than the `ProtocolError` the method documents. It is now seeded with `Enum_Flags(0)`, a real flagless member that compares equal to `0` and ORs identically. The read path's own seed is deliberately unchanged -- a flagless MP_JOIN is rejected by `mptcp_data_selector` before `_read_mptcp_join` runs, so it cannot reach those branches ([#587](https://github.com/JarryShaw/PyPCAPKit/issues/587)). -- the ILNP Nonce option builders in `HOPOPT` and `IPv6_Opts` sized the option with `math.ceil(nonce.bit_length() // 8)`, which is floor division dressed up as a ceiling: `//` floors, and `math.ceil` of an `int` is a no-op, so the ceiling was never actually taken. The nonce is packed by a `NumberField` whose width *is* that declared `len`, so an under-declared length did not merely mis-state the option -- it silently truncated the nonce on the wire, with nothing raised. Every nonce whose bit length was not an exact multiple of eight was affected, and **small values were the worst case rather than boundary values**: any nonce below 256 was declared as *zero* octets and dropped from the packet altogether, so `nonce=9` packed to `b'\x8b\x00'` and parsed back as `0`, while `nonce=256` and `nonce=65536` each truncated to `0` as well. Fixed to `max(1, math.ceil(nonce.bit_length() / 8))`, the form already used at five other sizing sites across `hip.py` and `mh.py`. The one-octet floor is the `mh.py` convention and is load-bearing here because `nonce` defaults to `0`, whose bit length is `0`: without it the default argument builds an ILNP Nonce option carrying no Nonce Value field at all, collapsing "the nonce is 0" into "there is no nonce" when [RFC 6744](https://datatracker.ietf.org/doc/html/rfc6744) gives the option that field. The read path was never affected, since it takes the width from the `len` octet on the wire rather than recomputing it ([#601](https://github.com/JarryShaw/PyPCAPKit/issues/601)). -- `httpv1`'s `_RE_METHOD` was unanchored and `re.match` anchors only at the start, so it prefix-matched, and the request-line reader then passed the whole `para1` to `Method.get` rather than the captured `method` group. Together those meant `b'Get'` matched on the single character `G`, satisfied the guard that decides a start-line is a request, and handed the entire mixed-case token to a lookup that raised on it. Fixing either half alone still gives a wrong answer -- normalising the lookup would parse `b'Get'` as `GET` off a one-character match, and passing the group would parse it as a method named `G`. The pattern is now anchored at both ends and the captured group is what is looked up, so a token that is not a method is a malformed request line rather than a mis-parsed one. Method tokens are case-sensitive per [RFC 9110 Section 9.1](https://datatracker.ietf.org/doc/html/rfc9110#section-9.1), so no `re.I` was added: `GET` parses, `Get` and `get` are rejected ([#583](https://github.com/JarryShaw/PyPCAPKit/issues/583)). -- `_RE_STATUS` in the same reader carried the same unanchored prefix defect, found by auditing `_RE_METHOD`'s siblings, and it escaped as the wrong exception type. That pattern is only a guard -- the value is taken from `int(para2)` on the raw token -- so a prefix match let a malformed status past the guard and then out of `int()` uncaught, where `_read_http_header` documents `ProtocolError`. Measured: a status of `200x` raised `ValueError: invalid literal for int() with base 10: b'200x'`, and one of `2000` raised `ValueError: 2000 is not a valid StatusCode`; both are now `ProtocolError`. [RFC 9112 Section 4](https://datatracker.ietf.org/doc/html/rfc9112#section-4) gives `status-code = 3DIGIT`, exactly three, so the anchor is what the grammar already said -- the production lives in HTTP/1.1 because `status-code` is part of its `status-line`, while [RFC 9110 Section 15](https://datatracker.ietf.org/doc/html/rfc9110#section-15) covers the code semantics and the IANA registry rather than the syntax. `_RE_VERSION` was audited at the same time and is safe as it stands, because both of its call sites read the captured group rather than the raw token ([#583](https://github.com/JarryShaw/PyPCAPKit/issues/583)). -- as a consequence of the above, the unhandled `MemoryError` at `pcapkit/protocols/protocol.py:1411` on a truncated PCAP-NG capture. `dhcp_little_endian.pcapng` cut to 161, 641 or 1389 octets left a one-octet read of a little-endian 32-bit block length, which `rjust()` turned into `0x78000000` or `0x84000000` -- 1.88 to 2.06 GiB -- and which was then passed straight to `self._file.read()` as an allocation size, from a 1772-octet file. Under a 1 GiB address-space cap all three raise `MemoryError` there; with more address space the allocation succeeds and the parse goes on to fail anyway, so the visible symptom depended on how much memory the process could get. With `ljust()` the same reads report 120 and 132, no large allocation is attempted, and all three end in the ordinary, already-handled parse failure instead ([#604](https://github.com/JarryShaw/PyPCAPKit/issues/604)). -- reading a big-endian classic PCAP byte-swapped every record header field, and then crashed. `Frame.unpack` seeded the file's declared byte order under the key `bytesorder` where the frame schema's `byteorder_callback` reads `byteorder`, so the lookup never found it and always fell back to `sys.byteorder` -- the reading host's order rather than the file's. On a little-endian host reading a little-endian capture that fallback gives the right answer by coincidence, and every capture in this repository was little-endian, so the wrong code path has always produced correct results. Against a big-endian capture, measured before the fix, frame 1 of `big_endian.pcap` read `ts_sec=3106905`, `ts_usec=1088553216` and `incl_len=1241513984` for a record whose real values are `1500000000`, `123456` and `74`, dating the frame to 1970-02-05 rather than to 2017-07-14. `incl_len` is the payload length, so that first record then consumed the whole file and the second was read with a negative payload length, raising `ValueError: read length must be non-negative or -1` out of the schema -- which is the reported crash, and it is the *second* symptom rather than the first. The sibling `Frame.pack` eleven lines earlier spelled the key correctly, which is what marks this as a slip rather than a second key, and the fallback is what made a misspelled key indistinguishable from an absent one; `byteorder_callback` now records that it is the definition of the key and why the fallback hides a typo ([#605](https://github.com/JarryShaw/PyPCAPKit/issues/605)). -- documentation. `mptcp_dss_ack_selector`'s note said a corrected field-width lambda "would not have worked" and that fixing it belonged to `pcapkit.corekit.fields.numbers`, which is exactly where [#598](https://github.com/JarryShaw/PyPCAPKit/pull/598) then fixed it; the same paragraph sat in `test_tcp_mptcp_length_arithmetic_unit.py`'s module docstring, whose other stale claim was that MP_JOIN "cannot be built through the public `TCP()` constructor at all", true only until [#587](https://github.com/JarryShaw/PyPCAPKit/issues/587). A callable-length `NumberField` packs and unpacks both DSS widths now, and wire *absence* was never the obstacle either: `MPTCPDSS.ssn`, `dl_len` and `checksum` have always been `ConditionalField` on the sibling `M` flag, so the class already relied on that wrapper to keep a field off the wire. The `SwitchField` form is kept for the narrower reason the note now gives -- `ConditionalField`'s `length` forwards to the wrapped field without consulting the condition, so it is safe here only because `Schema.pack` and `Schema.unpack` special-case that wrapper by name, whereas a `SwitchField` always resolves to a concrete field. Replacing it would be a behaviour change and is not made ([#603](https://github.com/JarryShaw/PyPCAPKit/issues/603)). -- the one assertion [#604](https://github.com/JarryShaw/PyPCAPKit/issues/604) left pinning the old padding side, which had been red on `mainline` since [#621](https://github.com/JarryShaw/PyPCAPKit/pull/621) merged. `TCPUDPUnitTests.test_a_truncated_option_still_parses_its_declared_length` expected a truncated TCP option's `data` as the synthesised zero octets *followed by* the real ones, which is what `rjust()` produced; [#621](https://github.com/JarryShaw/PyPCAPKit/pull/621) made the padding `ljust()` everywhere but could not retarget this file, since another change ([#612](https://github.com/JarryShaw/PyPCAPKit/pull/612)) owned it at the time and editing it concurrently risked discarding work that has since landed. The real octets now come first for both parametrised widths, and the docstring above the assertion says tail-padding rather than left-padding. Test-only: no library code changes, and the sibling case in `tests/protocols/internet/test_ipv4_unit.py` was already retargeted in [#621](https://github.com/JarryShaw/PyPCAPKit/pull/621). Measured against `main` at `2221c2d8f`: two subtest failures before, none after ([#604](https://github.com/JarryShaw/PyPCAPKit/issues/604)). -- `main` went red the moment [#604](https://github.com/JarryShaw/PyPCAPKit/issues/604)'s `ljust()` landed, because `TCPUDPUnitTests.test_a_truncated_option_still_parses_its_declared_length` still pinned the head-padded short read that fix removed. The `Reserved_79` option declaring `length=12` over 6 real octets now reports `aabbccddeeff00000000` where the test expected `00000000aabbccddeeff`, so both subtests -- `declared_length=12` and `=32` -- failed on that one assertion while the 17 other cases in the file stayed green: the parse itself never changed, only which end the synthesised zeros sit at. The expectation is inverted, and the docstring above it -- which said the short read was *left*-padded and described the value as four zero octets followed by the six real ones -- is corrected to match, since a docstring that contradicts its own assertion is how the stale expectation survived in the first place. The inputs do discriminate: `trailing` is non-zero and the pad width is 4 and 24, so neither subtest would hold under the other order. [#621](https://github.com/JarryShaw/PyPCAPKit/pull/621) left this file alone deliberately, because [#612](https://github.com/JarryShaw/PyPCAPKit/pull/612) owned it at the time, and merged two minutes ahead of the cross-review verdict that named it ([#604](https://github.com/JarryShaw/PyPCAPKit/issues/604), [#621](https://github.com/JarryShaw/PyPCAPKit/pull/621)). -- the HIP `SOLUTION` builder sized the parameter with `4 + math.ceil(max(random.bit_length(), solution.bit_length()) / 4)`, which is an invalid shorthand for the two fields it actually has to describe. [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5) gives the parameter's `Length` as `4 + RHASH_len / 4` over a `Random #I` and a `Puzzle solution #J` of `RHASH_len / 8` octets **each**, and `/ 4` equals twice `/ 8` only because `RHASH_len` -- the natural output length of a hash function, in bits -- is a whole number of octets. Applied to an arbitrary `bit_length()` the identity fails, and the builder emitted an *odd* contents width, which its own reader then refused: `_read_param_solution`'s `(schema.len - 4) % 2` guard exists because `SolutionParameter` splits that width into two equal `(len - 4) // 2` halves, so an odd width cannot be framed at all. `random=0x1` with `solution=0xfff` declared `len=7` and raised `ProtocolError: HIPv2: [ParamNo 321] invalid format` on input the library had itself produced. The undersized length was the worse half of it: because both fields take their width from that same `len`, `solution=0xfff` was packed into the one octet it allowed and came back as `0xff` -- silent truncation, nothing raised. Fixed to `4 + 2 * math.ceil(max(...) / 8)`, the form the sibling `PUZZLE` builder already uses, which is even by construction and so can never trip the guard. Loosening the reader was rejected as the alternative: an odd contents width has no meaning in the wire format, since the RFC makes the two fields equal-width and says nothing about which would take an extra octet, and a laxer guard would still have truncated the value ([#608](https://github.com/JarryShaw/PyPCAPKit/issues/608)). -- `TCP.read` seeded its connection-flag accumulator with `cast('Enum_Flags', 0)`. `typing.cast` is a runtime no-op -- it returns its second argument unchanged -- so the accumulator began life as the plain `int` `0`, and the `|=` that promotes it to a `pcapkit.const.tcp.flags.Flags` member is the only thing that ever did. A segment whose flags octet is all zero, an nmap NULL scan among them, promoted nothing and left `self._flags` an `int`, so a membership test against it raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering, and `TCP.connection` returned an `int` where both that property and `Data_TCP.connection` annotate `Flags`. Any flag at all masked it, which is how it survived a module at 100% statement and branch coverage. The seed is `Flags(0)` now, which is what [#597](https://github.com/JarryShaw/PyPCAPKit/pull/597) had already done to the sibling accumulator in `make` for the same reason; `Flags` declares no `_missing_` of its own, so `Flags(0)` is an ordinary `aenum.IntFlag` pseudo-member and does not meet the `RecursionError` that the same construction hits on the flag enumerations under `pcapkit.const.mh` ([#623](https://github.com/JarryShaw/PyPCAPKit/issues/623)). Nothing a caller can reach changed: the three MP_JOIN layouts of [RFC 8684](https://datatracker.ietf.org/doc/html/rfc8684) section 3.2 are chosen by `_read_mptcp_join` from these very membership tests, but `mptcp_data_selector` rejects a flagless MP_JOIN with a `FieldError` before the dispatcher runs -- measured identical either side of the fix -- so the `TypeError` was latent rather than live, and latent only by virtue of a guard in another file. Called directly on a flagless parsed segment the dispatcher now raises the library's own `ProtocolError` naming an invalid flags combination instead. One difference *is* observable, and it is a dump-format change rather than a value change: a flagless segment's `connection` renders as the string `Flags::None [0]` where it rendered as the number `0`, in all three of the JSON, tree and PLIST outputs. The value is numerically the same and the wire bytes are untouched; what changed is that `connection` no longer switches JSON type with the flags, having been a number for a flagless segment and a string -- `Flags::ACK [2048]` -- for every other one. The literal `None` inside it is a separate rendering defect: the hook `pcapkit.dumpkit.common.make_dumper` installs interpolates `o.name` without accounting for a nameless composite member, so it would spell any zero-valued flag enumeration in the library the same way. That is left to its own change and pinned by a test here so it cannot drift unnoticed. The committed example dumps are unaffected, `examples/captures/in.pcap` carrying no flagless segment. Coverage cannot see the fix itself, because the changed line already executed and `tcp.py` reads 100% statement and branch either side; the added subtests are the evidence ([#616](https://github.com/JarryShaw/PyPCAPKit/issues/616)). -- **a breaking change to a public attribute.** `Frame.len` and `Frame.cap_len` were filled from *opposite* wire fields depending on which container format was read, so `frame.len` meant the captured length out of a `.pcap` and the on-wire length out of a `.pcapng`. The PCAP reader is the one that moved, and now matches the PCAP-NG reader: `len` is the on-wire length (the record header's `orig_len`) and `cap_len` the octets actually stored (`incl_len`). **Code reading `frame.len` or `frame.cap_len` from a `.pcap` gets the other field's value than it did before**, and for a truncated frame that is a different number rather than a relabelling. Which reader to move was a real decision rather than the correction of a typo, and not settled by seniority: the PCAP spelling is the *older* of the two, dating to `c43892af` (2022-01-11) with the data model's docstrings agreeing with it a day later, while the opposite spelling in `pcapkit.toolkit.pcapng` arrived 15 months afterwards in `25f216f4` (2023-04-27). Both were internally consistent. What settles it is that the *names* are Wireshark's, and its `epan/dissectors/packet-frame.c` registers `frame.len` as "Frame length on the wire" and `frame.cap_len` as "Frame length stored into the capture file", and raises the `frame.len_lt_caplen` expert info, `PI_MALFORMED`/`PI_ERROR`, on `frame_len < cap_len` -- which could not be malformed if `len` were the smaller, captured one. So the later spelling is the one that matches the borrowed names. Both formats define the underlying fields the same way: `pcap-savefile(5)` gives `incl_len` as "the number of bytes of captured data that follow the per-packet header" and `orig_len` as "the number of bytes that would have been present had the packet not been truncated by the snapshot length", and `draft-ietf-opsawg-pcapng` gives Captured Packet Length as "the number of octets captured from the packet" against Original Packet Length's "number of octets of packet data that would have been provided had the packet not been truncated", which "SHOULD NOT be less than the Captured Packet Length". The internal consumer moved with it: `Frame.read` hands `_decode_next_layer` the octets that are *present*, which is now spelled `frame.cap_len` and was `frame.len` when that was the captured length -- handing over the on-wire length instead would be the declared-length-exceeds-available-octets fault of [#554](https://github.com/JarryShaw/PyPCAPKit/issues/554), [#573](https://github.com/JarryShaw/PyPCAPKit/issues/573) and [#594](https://github.com/JarryShaw/PyPCAPKit/issues/594) on every snapped frame. The dumpers are untouched, since `PCAPIO` packs its record header from `frame_info.incl_len` and `frame_info.orig_len` rather than from these two, and `frame_info` was already right in both readers. This went unnoticed because the two lengths differ only for a frame the snapshot length cut short, and no such fixture existed until [#614](https://github.com/JarryShaw/PyPCAPKit/pull/614) added one -- `big_endian.pcap`'s third frame, 1200 octets on the wire against 96 captured -- so every earlier assertion compared a value against itself ([#618](https://github.com/JarryShaw/PyPCAPKit/issues/618)). -- the HIP `PUZZLE` and `SOLUTION` builders derived three wire-format values from the payload value instead of taking them from the data model, and all three were wrong. **This changes two public data models and the octets both parameters emit.** The width of `Random #I`, and of `Puzzle solution #J`, came from `int.bit_length()` alone, and nothing else was available to derive it from, so every leading zero octet was dropped on re-serialisation: a `SOLUTION` read with `Length = 20` rebuilt as `Length = 6`, a `PUZZLE` read with `Length = 12` as `Length = 5`, and silently, because the integers survive and nothing raises. That width is `RHASH_len / 8` octets [[RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4), [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5)], a property of the Responder's HIT Suite rather than of the number that happens to sit in the field, so both data models now carry it as `rhash_len` and both builders prefer it; a new `rhash_len=` keyword states it for a build from scratch, which under HIPv2 is the only place it can come from, since `RSA,DSA/SHA-256` is the REQUIRED HIT Suite [[RFC 7401 Section 5.2.10](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.10)] and makes the field 32 octets rather than 8. On the mandatory path roughly one parameter in 256 has a zero top octet and lost it ([#653](https://github.com/JarryShaw/PyPCAPKit/issues/653)). `SOLUTION`'s second contents octet is `Reserved`, "zero when sent, ignored when received" [[RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5), and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) identically] -- not the `Lifetime` that only [RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4) defines, and that pcapkit was encoding there as `2^(value - 32)` seconds. It wrote `0x20`, `0x21`, `0x25` or `0x2b` into a field the RFC requires to be zero, and could not write that zero at all: an RFC-conformant `SOLUTION` whose `Reserved` is `0x00` parsed to `timedelta(0)` and then could not be re-serialised, escaping a bare `ValueError` from `math.log2(0.0)` that was not a `BaseError` and so bypassed the library's own error handling entirely. The field is renamed `reserved`, defaults to the mandated zero, and is carried verbatim across a round trip rather than re-derived ([#654](https://github.com/JarryShaw/PyPCAPKit/issues/654)). And neither builder read its own `version` keyword, so `version=1` and `version=2` computed identical lengths at every bit width, where [RFC 5201 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.4) and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) state both fields as literally 8 bytes and `Length` as literally 12 and 20. Under HIPv1 each builder therefore accepted only a `bit_length()` of 57..64 and built, for everything narrower, a parameter this library's own reader rejects -- the same shape as [#608](https://github.com/JarryShaw/PyPCAPKit/issues/608), in the `PUZZLE` builder that [#629](https://github.com/JarryShaw/PyPCAPKit/pull/629) never opened ([#655](https://github.com/JarryShaw/PyPCAPKit/issues/655)). The two remaining `math.log2` sites, both in `PUZZLE` where a lifetime is real, now raise `ProtocolError` rather than letting `ValueError` escape; `ProtocolError` is `(BaseError, ValueError)`, so a caller written around the old bare exception still catches it. Migration: `SolutionParameter.lifetime` is now `reserved` and an `int` rather than a `timedelta`; both `PuzzleParameter` and `SolutionParameter` gained a required `rhash_len`; and `_make_param_solution` no longer takes `lifetime=`. All three were only reachable end to end once [#608](https://github.com/JarryShaw/PyPCAPKit/issues/608) was fixed in [#629](https://github.com/JarryShaw/PyPCAPKit/pull/629), which removed the parity guard that had been failing these rebuilds loudly first -- so the round trip stopped raising and started quietly emitting a different parameter, which is why they were worth fixing together ([#653](https://github.com/JarryShaw/PyPCAPKit/issues/653), [#654](https://github.com/JarryShaw/PyPCAPKit/issues/654), [#655](https://github.com/JarryShaw/PyPCAPKit/issues/655)). -- every HIP parameter pcapkit emitted was `4 (mod 8)` octets long, because the padding was computed to align the *contents* rather than the record. Corrected for **45 of the 46** HIP parameters; `LOCATOR_SET` is deliberately excluded, for the reason below. **This changes the octets those 45 parameters write, and the `length` their data models report.** [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1) requires that "all of the encoded TLV parameters have a length (that includes the Type and Length fields), which is a multiple of 8 bytes", and states the arithmetic outright as `Total Length = 11 + Length - (Length + 3) % 8`; all 95 padding sites instead computed `(8 - (Length % 8)) % 8`, which has no `+ 4` inside the modulus and so aligns the contents alone. Across every `Length` from 0 to 63 the result was never a multiple of eight and never the value the RFC gives, and it erred in both directions: at `Length = 4` -- a whole `SEQ`, and [RFC 7401 Section 5.3.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.3.5) puts a `SEQ` or an `ACK` on every `UPDATE` -- the record is complete in eight octets and pcapkit appended four that must not be there, while at `Length = 8` the contents were already 8-aligned, nothing was appended, and the record went out four octets short. A conformant peer reading `Length` and consuming `11 + Length - (Length + 3) % 8` octets therefore lands mid-parameter and reads the rest of the parameter area at a wrong offset, in both directions; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test in the suite could see this and why the fix is asserted against the RFC's arithmetic rather than against a round trip. 94 of the 95 sites -- 45 of the 46 `PaddingField` callbacks in `pcapkit/protocols/schema/internet/hip.py` and 49 of the 49 record lengths in `pcapkit/protocols/internet/hip.py` -- are now one `parameter_total_len` and one `parameter_padding_len`, stating the RFC formula once instead of 95 times ([#651](https://github.com/JarryShaw/PyPCAPKit/issues/651)). `LOCATOR_SET` kept the old expression at both of its sites for now, on purpose, because two defects there cancelled each other exactly and correcting only the padding would have broken a parameter that was, at that point, right: its padding callback never received the parameter's `len` (the nested `Locator` schemas shared a packet context whose own `len` shadowed it, so the value seen was always 4 for an IPv6 locator), and the parameter's `len` was written in 4-octet units where the RFC's `Length` is a byte count. Always-4 padding gave `4 + 24n + 4`, and because `24n` is a multiple of 8 the RFC total for a byte-count `Length` of `24n` was the same `24n + 8` -- measured at n = 1, 2 and 5 as 32, 56 and 128 octets both before and after. [#679](https://github.com/JarryShaw/PyPCAPKit/issues/679) later fixed both sites. `EncryptedParameter`'s `data` length callback is fixed in the same change and could not have been left: it subtracted the sixteen `iv` octets but not the four `reserved` ones, and those four cancelled the padding's four at `Length % 8` in `{0, 5, 6, 7}` -- so correcting the padding alone would have taken `ENCRYPTED` from right at four of the eight residues to four octets too long at all eight. `HIP.make`'s `len = total_length // 8 + 4` is *not* part of the defect and is unchanged: [RFC 7401 Section 5.1.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.1.3) defines Header Length as the header and parameters "in 8-byte units, excluding the first 8 bytes", so the floor division is exact once each parameter is a multiple of eight, where before it was exact only for an even number of them. Migration: a `SEQ` parameter's `length` is now 8 where it was 12, and the other 44 corrected parameters move likewise, so code comparing stored pcapkit output byte for byte, or asserting on `Data_*Parameter.length`, sees different values -- the RFC's values. `LOCATOR_SET` stayed unchanged in both respects for the time being; [#679](https://github.com/JarryShaw/PyPCAPKit/issues/679) (below) later fixed both. `examples/generators/options.py`'s `HIP_COPIES` stayed at two for now as well, no longer for this reason but for `R1_COUNTER`'s four-octet `counter` where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires eight, which this defect had been masking -- until [#672](https://github.com/JarryShaw/PyPCAPKit/issues/672) widened `counter` and [#679](https://github.com/JarryShaw/PyPCAPKit/issues/679) fixed `LOCATOR_SET`'s `Length` unit, after which [#689](https://github.com/JarryShaw/PyPCAPKit/issues/689) dropped `HIP_COPIES` to one, once both had left it routing around nothing ([#651](https://github.com/JarryShaw/PyPCAPKit/issues/651)). -- two HTTP/2 flag defects, one on each side of a round trip. **This changes the octets a reconstructed DATA frame emits, and the dumped value of a flagless frame's flags.** `_make_http_data` was the only one of the six `_make_http_*` methods that never read `frame.flags`, so a DATA frame parsed with `END_STREAM` set rebuilt with the bit clear: the flags octet went `0x01` to `0x00`, silently, producing a well-formed frame saying the stream continues where the capture said it ended [[RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1)]. `PADDED` was never affected, being re-derived from `pad_len` rather than read back, and all five sibling builders already restored theirs -- so this was a missing read-back rather than a design, and it now reads `frame.flags.END_STREAM` the way they do ([#652](https://github.com/JarryShaw/PyPCAPKit/issues/652)). Separately, `FrameType.post_process` seeded its flag accumulator with a bare `0`, and `|=` promotes that only as a side effect, so a frame with **no** bit set left `__flags__` a plain `int` where both the schema and the data model declare a `Flags` -- and `END_STREAM in __value__` raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering `False`. That is the common case rather than an edge one: a `0x00` flags octet is every opening `SETTINGS`, every non-final `DATA` and every non-ACK `PING`. It now seeds `self.Flags(0)`, which the construct path had been doing correctly at all six of its own sites. The visible consequence is in the dump, where `__value__` was a JSON *number* for a flagless frame and a JSON *string* for every other frame in the same capture; it is consistently a string now, `"Flags::None [0]"` where it read `0`, the same shape of change [#616](https://github.com/JarryShaw/PyPCAPKit/issues/616) made to TCP's `connection`. The seed is guarded rather than unconditional because `FrameType.Flags` declares no members and, from Python 3.11, a memberless `enum.Flag` subclass refuses `Flags(0)` outright, so the five frame schemas that inherit it -- `UnassignedFrame`, `PriorityFrame`, `RSTStreamFrame`, `GoawayFrame` and `WindowUpdateFrame` -- keep the plain `int`, which nothing observes since all five pass `flags=None` into their data objects. Note that a DATA round trip is **still** lossy after this, for an unrelated reason found while measuring it and left to its own change: three payload length callbacks in the frame schemas are mis-parenthesised, so an unpadded `DATA`, `HEADERS` or `PUSH_PROMISE` frame parses with its whole payload dropped ([#668](https://github.com/JarryShaw/PyPCAPKit/issues/668)). ([#652](https://github.com/JarryShaw/PyPCAPKit/issues/652), [#650](https://github.com/JarryShaw/PyPCAPKit/issues/650)). -- four MPTCP error messages carried a doubled separator, rendering as `TCP: : [OptNo 30] 1: invalid flags combination` with an empty field between the protocol alias and the option number. Cosmetic: the exception type, the option number and the subtype were always right, and nothing downstream parses these strings, but it is the text a user sees when an MP_JOIN or DSS option is rejected and an empty field reads as a value that failed to interpolate. Four sites, not the one reported -- `_read_mptcp_join` and `_make_mptcp_join` for the invalid flag combination, and both guards of `_make_mptcp_dss` for the missing required fields -- against 28 messages in the same file that already spelled the prefix correctly, so the four were outliers against a house form in their own module and the counts are now 32 correct against 0 doubled. Pre-existing since 2023-04-10. The new test asserts each message in full rather than by substring, which is what the existing `assertIn` on the message body could not do, since it passes either side of the change ([#649](https://github.com/JarryShaw/PyPCAPKit/issues/649)). -- three names appeared in string annotations that their own module never imported, so a type checker or a documentation build could not resolve them while every test went on passing: `Any` in `pcapkit/utilities/logging.py`, and in `pcapkit/protocols/schema/internet/ipv6_route.py` both `Protocol` in a `payload:` stub and `Optional` inside a `typing.cast`. All three are now in their module's `TYPE_CHECKING` block, `Protocol` spelled `ProtocolBase as Protocol` the way twenty-one sibling schema modules already spell it in that same stub, which makes twenty-two. The `cast` case is the one worth naming: `typing.cast` never evaluates its first argument, so no test that runs the line can see a bad name in it. Nothing resolves at runtime that did not resolve before -- `TYPE_CHECKING` is `False` when the interpreter runs, so a name imported under it is absent from the module namespace either way, and making these annotations resolve at runtime is a separate decision about the idiom rather than a missing import. mypy 2.3.1 over `pcapkit` reported three `name-defined` errors before and none after, its total moving 115 to 112, so nothing else shifted. A new `tests/project/test_annotation_names.py` pins the invariant without needing a type checker installed: it resolves every string annotation in the package against the names its own module binds, following a nested forward reference such as `'list["Nested"]'` while treating `Literal` members and `Annotated` metadata as the values they are, and it reports exactly those three findings on the unfixed tree and none after. That module reaches the two PEP 695 node classes it needs through `getattr` rather than naming them, because `ast.TypeAlias` and `ast.TypeVar` arrived in Python 3.12 while the supported range starts at 3.10. The same change corrects the `MANIFEST.in` comment asserting that `include README.md` was the only thing putting the README in a source distribution and that an sdist without it could not be installed; both halves are false, and the [#619](https://github.com/JarryShaw/PyPCAPKit/pull/619) entry above carried the identical claim and is corrected with it ([#642](https://github.com/JarryShaw/PyPCAPKit/issues/642)). -- every PCAP-NG packet block now carries the captured octets it declares. All four Enhanced Packet Blocks of the committed `examples/captures/dhcp.pcapng` reported `len(packet) == 0` against a `captured_len` of 314, 342, 314 and 342, and the Simple Packet Block and the obsolete Packet Block did the same, measured on a capture synthesised to carry one of each since no committed fixture has either. Those three -- exactly `PCAPNG.PACKET_TYPES`, and exactly the three schemas declaring `__payload__ = 'packet_data'` -- are the whole of the blast radius; every other block type carries no captured octets, reported `b''` before and still does. The octets were read correctly and then thrown away: `PCAPNG.unpack` extracted them from the block schema, and `ProtocolBase.__init__` overwrote the result with `self.packet.payload`, which was empty. The inherited `packet` splits a protocol at `self.length` octets of header and takes everything after as the payload, a contract that holds for a protocol laid out as a header followed by its payload and for nothing else. `PCAPNG.length` returns the wire's *Block Total Length*, so the split consumed the entire per-block buffer as header -- measured at `frame.length == 348` and `len(frame.packet.header) == 348` on a 348-octet block -- and, separately, a PCAP-NG block carries a *trailer* after its payload, the option list and a repeat of the total length, so no value of `length` could have made that split right. `captured_len` comes straight off the block header and survived, which is why the two disagreed rather than both being empty. The fix is at the computation rather than at the injection: `PCAPNG.packet` is overridden to take the payload from the block schema's `__payload__` field and the header from the octets ahead of it, summed out of the schema buffers so that the three block types' three different payload offsets -- 28, 12 and 28 octets -- are not hard-coded. Teaching `ProtocolBase.__init__` to skip a protocol that had already set `packet` was the smaller change and was rejected: it heals `frame.info.packet` while leaving the public `frame.packet` still reporting the whole block as header and `b''` as payload, which is the same defect from the other side. `PCAPNG.unpack` now reads that property instead of extracting the payload a second time of its own, leaving one source of truth where there were two attempts at one. The property is a plain `property` rather than the `cached_property` it overrides, which a cross-review on a second model is what settled: the inherited one caches because it *reads the stream*, where a second read would consume octets that are gone, while this one only walks buffers the schema layer has already filled and so has nothing to amortise -- and caching it would have reintroduced the same staleness by another route, a second `unpack` on one instance handing back the *first* call's octets with `get_payload` never reached, an invariant the code held before this change and has no reason to stop holding. Nothing calls `unpack` twice on one instance today, `__post_init__` being its only caller, so the unit test asserts the property directly: swap the block on a live instance, unpack again, and the payload must be the new block's, which fails `b'payload' != b'cached'` under a cache. `ProtocolBase.packet` gains the contract in its docstring and not one executable line, its 503 statements unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on the success path, which is why it ships labelled breaking: `frame.info.packet` goes from `b''` to hundreds of octets, `frame.packet.header` from the whole block to the pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what made this a wire-format defect rather than an API wart. `PCAPIO` writes `value.packet` after each 16-octet record header, so dumping those four blocks produced a **104-octet** file -- 24 of global header plus four record headers and no payload at all -- in which every record header declared hundreds of octets and delivered none, desynchronising any reader that walks by `incl_len`, which is all of them: pcapkit refused its own output with `ValueError: read length must be non-negative or -1` and scapy silently returned 1 packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and each record's octets both asserted against the source capture's own hand-parsed bytes rather than against a length. `tests/protocols/test_pcapng_regression.py` grows 4 tests to 11 and 3 subtests to 24, with the payload expectations derived twice over -- spelled-out head and tail literals, and a `struct`-only re-parse of the fixture that owes nothing to the code under test -- and each synthesised payload given a different length modulo 4 so that an offset off by a field, or a payload that picked up the block's 32-bit padding, cannot pass by coincidence. Two shapes no fixture exercised are covered there too, both found while writing the tests rather than after: an option area *after* the captured data, which every block of `dhcp.pcapng` lacks and which is exactly what an offset walked from the wrong end would swallow, and a big-endian section, whose fields differ even though the payload offsets do not. A snapped block is covered in both the shapes that express it -- an Enhanced Packet Block declaring `captured_len` below `original_len`, and a Simple Packet Block whose captured length is bounded by the interface's `snaplen` -- neither of which may come back padded out to the on-wire length. The offsets hold at 28, 12 and 28 across all of it, which is the property the walk has to have. With both trees running the same -- current -- tests, which is the only comparison that isolates the code from the suite, `pcapkit/protocols/misc/pcapng.py` holds 99.91% across the change, its 18 new statements and 6 new branches all executed, its miss count flat at 1 and its partial-branch count flat at 1, that single miss being the same pre-existing `_get_timezone` statement renumbered 1248 to 1360; `pcapkit/protocols/protocol.py` is identical in every column, 503 statements and 228 misses either side, which is the check that its change really is docstring-only. The uncaught `ValueError` that an EOF-truncated PCAP-NG raises ([#678](https://github.com/JarryShaw/PyPCAPKit/issues/678)) is untouched and was measured rather than assumed, over a truncation set stated rather than described because "101 levels" admits several readings that do not agree on the counts: the whole-percent prefixes `max(1, 1508 * i // 100) for i in range(101)`, which for this file are 101 distinct lengths from 1 to 1508. Both trees give 96 of that `ValueError`, one `FormatError`, one `ProtocolError` and three parses, and -- compared level by level rather than only in aggregate, since a matching total can hide two levels that swapped -- the SHA-256 of the sorted length-to-outcome mapping is `ca44d3ee658087cf2a667c454f839dd931c1c04790b2fe32cee8e1a2e861b182` on each. `examples/captures/pcapng.txt`, a hand-regenerated legacy smoke reference, still showed the old frame-level `packet -> NIL` at this point and wanted a separate refresh; [#685](https://github.com/JarryShaw/PyPCAPKit/issues/685) (below) later removed it from the index entirely rather than regenerating it again ([#646](https://github.com/JarryShaw/PyPCAPKit/issues/646)). -- an unpadded HTTP/2 `DATA`, `HEADERS` or `PUSH_PROMISE` frame parsed with its whole payload silently discarded. Three payload length callbacks in `pcapkit/protocols/schema/application/httpv2.py` put the conditional expression in the wrong place: a conditional binds looser than `-`, so `pkt['__length__'] - pkt['pad_len'] if pkt['flags']['bit_3'] else 0` groups as `(pkt['__length__'] - pkt['pad_len']) if ... else 0` and the `else` arm returned `0` -- "read no octets at all" to `BytesField` -- where it was meant to subtract `0`. Since `__length__` is the *remaining* declared length at the field, the unpadded arm wants `__length__` itself, which is what subtracting a zero padding length gives. The grouping was read off the AST rather than by eye, the top-level node being the `IfExp` whose `orelse` is a bare `Constant(0)`, which is also what showed that the outer parentheses on the `HeadersFrame` and `PushPromiseFrame` forms were line-continuation only and changed nothing. Padding is rare in HTTP/2, so the broken arm was the common one rather than the edge case, and nothing raised, warned or logged: `info.data` was simply `b''`. Measured on wire octets through the public `HTTP(io.BytesIO(raw), len(raw))` path, an unpadded `DATA` frame declaring `b'{"ok":true}\n'` parsed as `b''` and now parses as those twelve octets; unpadded `HEADERS` and `PUSH_PROMISE` lost and now keep the RFC 7541 appendix C.4.1 header block fragment `b'\x82\x86\x84A\x0fwww.example.com'`, which is precisely what HPACK decoding would have been handed. Padded frames are untouched, for a reason stronger than the measurement: `(A - B) if T else 0` and `A - (B if T else 0)` are both `A - B` for truthy `T`, so only the `else` arm could ever have moved -- both arms are asserted anyway, so that a repair to the unpadded one cannot break the other. The padded arm also has no off-by-one, which [#668](https://github.com/JarryShaw/PyPCAPKit/issues/668) asked be checked rather than assumed: `Schema.unpack` decrements `packet['__length__']` by each field's width as it goes, so `pad_len`'s own octet is already out of it by the time the payload field runs, and a padded frame declaring `1 + len(data) + pad_len` arrives at the payload with `__length__ == len(data) + pad_len`. The three sibling sites that were never wrong are what localise the defect to the grouping rather than to the field machinery: `ContinuationFrame.fragment`, `UnassignedFrame.data` and `GoawayFrame.debug` use the plain `length=lambda pkt: pkt['__length__']` with no conditional, share the same `BytesField`, the same `__length__` bookkeeping and the same frame dispatch, and carried their payload correctly throughout -- they are pinned as controls rather than left as an argument. Why nothing caught it: `test_option_roundtrip_unit` does drive every frame type, but through `make` -> parse -> `make`, and `make` writes the length field from `_make_http_length`, so both sides agreed on an empty payload and the octets matched; its generator passes no payload argument for these three frames at all, measured as `kwargs={}`, so what it round-tripped was `b''` and there was nothing to lose, while `test_httpv2_frame_readers_cover_successful_frames` drives the readers from hand-built schema stubs, so the callback never ran there either. The new `tests/protocols/application/test_httpv2_payload_length_unit.py` reads wire octets, which is the gap: 23 tests and 12 subtests over the padded, unpadded and padding-only shapes of all three frames plus both `PRIORITY` combinations, asserting the payload *octets* and never its length, because a length assertion survives a read of the right width from the wrong offset. Its padded cases pad with a non-zero `b'\xde\xad\xbe\xef'` rather than the zeros [RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1) tells a sender to use -- pcapkit does not police padding content -- so that failing to subtract the padding yields a different byte string instead of a coincidentally equal count. The same lambda also governs **packing**, which neither the issue nor the first revision of the fix noticed and a cross-review on a second model did: `BytesField` consults its `length` callback on both paths, so the `else` arm did not merely discard a payload on read, it declined to write one. Measured pre-fix, an unpadded `DATA` frame carrying twelve octets of body packed to `000015000000000001` -- nine octets of header whose length field declares **21**, with the body absent -- and unpadded `HEADERS` and `PUSH_PROMISE` likewise declared 29 against 9 and 33 against 13. A frame overstating its own length by its whole payload desynchronises any reader that walks a stream by that field, so this was pcapkit *emitting* malformed HTTP/2 rather than only mis-reading it, and `HTTPv2ConstructedFrameDeclaresWhatItWritesUnitTests` pins that half; a `make` -> parse -> `make` cycle of a non-empty unpadded payload now closes byte-for-byte where it could not before. Six wrong fixes were written over the schema to check the tests discriminate, and each is rejected: dropping the conditional outright (caught by `packet length < 0: -4`), subtracting `pad_len + 1` (caught as `b'{"ok":true}'` against `b'{"ok":true}\n'`), subtracting the padding on the arm where it is absent, fixing `DataFrame` while forgetting the other two, and `max(computed, 1)` on the two `fragment` fields. That last one is the cross-review's own construction and it **passed** the first revision's 15 tests and 9 subtests, because every fixture then carried a non-empty payload and only `DataFrame` had a "frame of only padding" case, so those two callbacks were never once asked for `0`; under it a padding-only `HEADERS` frame returned `fragment=b'\xde'`, one padding octet leaked in, alongside a swallowed `SchemaWarning: packet length < 0: -1`. The three missing padding-only cases and a callback-level `test_the_padded_arm_reaches_zero_and_is_not_clamped` close it -- `0` is a legitimate answer from these callbacks and has to be asserted as one. Seven tests and three subtests fail before the change and all pass after; 145 tests and 550 subtests pass across `tests/protocols/application/`, the round-trip module and the four other HTTP/2-touching modules. `EXPECTED_FAILURES` is unmoved, its one HTTP/2 entry `httpv2-frame/PRIORITY` still failing with the same `CONSTRUCT` status and the same detail -- `PriorityFrame` has neither a payload field nor a `PADDED` flag, so this change cannot reach it -- and none of the 43 entries was deleted; that entry's own cited `pcapkit/protocols/application/httpv2.py:572` is stale, the `header.length != 9` guard it describes now being at `:650`. The schema module was already at 100% coverage and stays there, 101 statements and 4 branches either side with nothing missing or partial, because the changed lines did execute before -- just with an empty payload -- so the number that moved is the suite: 73 to 96 tests and 399 to 411 subtests over the same targets. The construct side needed no code change, `_make_http_length` having always computed the DATA payload as `len(frame.data) + (pad_len + 1 if pad_len else 0)`, i.e. assuming `data` holds the whole payload whether or not the frame is padded -- it was the shared length callback that disagreed, on both paths at once. An AST sweep of all 496 files of the package found one other site with the same syntactic shape, `pcapkit/protocols/schema/internet/ipv4.py:336` (`TSOption.remainder`), and it is correct rather than the same defect: its `else 0` is intentional because the sibling `ts_data` field consumes the entire option data area in that arm, the two summing to `length - 4` across all 2106 `(length, pointer, flag)` combinations, which is exactly what was not true here. Ships labelled breaking on both counts: restoring a dropped payload moves the parse output of essentially every HTTP/2 capture on the success path, so anything holding a golden file or a regression baseline that recorded the empty value sees it change -- and constructed frames change byte-for-byte too, from malformed to correct ([#668](https://github.com/JarryShaw/PyPCAPKit/issues/668)). -- the schema half of every option-like registrar overwrote silently while the parser half of the very same call warned. `EnumSchema.register` was a bare `cls.__enum__[code] = schema`, and it is the schema half of 14 public registrars, so one `register_ipv4_option` call replacing a built-in named the parser it displaced and said nothing about the schema. `EnumSchema.__init_subclass__` reaches the same registry without calling `register`, so a subclass declared with a `code=` keyword -- the documented way to add a schema -- stayed silent too; both are guarded now, and the declaration hook's two branches are folded into one loop so the guard is written once. Presence is a faithful test there only because `_EnumRegistry.__missing__` returns a miss without recording it, so parsing one packet carrying an unknown code cannot make the next legitimate registration for that code warn about an entry no caller ever asked for. The seven code-keyed parser registrars -- on `ProtocolBase`, `Link`, `Internet`, `Frame`, `PCAPNG`, `Transport` and `SCTP` -- now name the displaced entry and its replacement, where before they reported only that something had been overwritten and never which class it was. Their firing condition is harmonised onto the narrower guard [#681](https://github.com/JarryShaw/PyPCAPKit/pull/681) gave `register_protocol` ([#718](https://github.com/JarryShaw/PyPCAPKit/issues/718), [#726](https://github.com/JarryShaw/PyPCAPKit/pull/726)), rather than firing on presence alone: that one is keyed on a name derived from the value it stores, whereas these seven take a `code` the caller supplies independently of the value. Five of these seven -- `Link` (7 entries), `Internet` (16), `Frame` (3), `PCAPNG` (3) and `SCTP` (2) -- ship pre-seeded with unresolved `ModuleDescriptor` values, and so do `TCP` (4) and `UDP` (3), each keeping its own separate dict rather than filling the one `ProtocolBase` and the abstract `Transport` share and leave at 0 (`Transport.register` itself raises `UnsupportedCall`). Where a table is pre-seeded, an incumbent may still be a two-string descriptor while the replacement is the very class it names. Resolution happens earlier, on the incoming argument only, three lines above the guard -- `if isinstance(protocol, ModuleDescriptor): protocol = protocol.klass` -- so the guard's own `is` comparison still measures a resolved class against an unresolved descriptor, warning on re-registering the very same class under its own pre-seeded code -- `Internet.register(TransType.TCP, TCP)`, say -- a false positive the harmonised guard does not close. `ContextRegistry.register` already raises on a duplicate, and the reassembly and ESP registrars append to lists with no key at all -- duplicate security-association SPIs being a designed feature there, resolved by scoring rather than by replacement -- so none of those three takes a guard. `import pcapkit` holds at one warning, a third-party deprecation, and no `RegistryWarning`, measured over the 327 registry writes it performs -- one of them being `R1CounterParameter`'s second code, added by [#690](https://github.com/JarryShaw/PyPCAPKit/issues/690) -- none of which lands on a key already present; `tests/foundation/registry/` goes 13 to 14 tests and 87 to 95 subtests, the schema module holds 99% coverage with its 5 new statements covered and its misses flat at 1, mypy and pylint are unmoved at 112 errors and 364 messages, and `EXPECTED_FAILURES` is unmoved at 43 entries ([#692](https://github.com/JarryShaw/PyPCAPKit/issues/692)). -- every EOF-truncated PCAP-NG file raised a bare `ValueError` out of `Extractor`, so the whole extraction was lost rather than the one truncated block. The root is one layer above the reported site: Block Total Length is cross-checked against its own trailing copy and never against the file, so a last block declaring more than the file holds left `PCAPNG.read` seeking *past* the real end -- legal and silent -- and the next block read then measured a **negative** remainder, since `prepare` derives it as the end of the stream less the current position. That negative reached `SchemaField` through `pcapng_block_selector` and `io.RawIOBase.read` refused it, which is why nearly every truncation level failed rather than only the one holding the cut. The seek is now clamped to the octets `_read_fileng` actually returned, with a `ProtocolWarning` naming the overrun, and a tail under the twelve octets a block needs at minimum is reported as the quiet `StreamEOFError` that `Extractor.record_frames` already catches -- the one case that is deliberately not a clamp, because at the end of the file no block is being read and clamping would fabricate a frame out of zero padding. A second, independent route to the same shape is closed with a new `nonnegative()` helper, composed into `bounded_option` and `bounded_area` and applied to the eight unwrapped `length - N` spans and the eight `__option_padding__`-sized padding fields: `_TextField.__call__` builds its `struct` template as `f'{length}s'` unconditionally, so a negative became the format `'-8s'` and `struct.calcsize` raised -- neither that nor the `ValueError` being one of `pcapkit.utilities.exceptions`, and neither being an `EOFError`, so neither was caught where the frame loop catches the end of a file. 28 of the module's 74 length callbacks returned a negative before and none do now, from `-1` on an `epb_hash` declaring no payload to `-16777248` on an Enhanced Packet Block's option area; the 200-block, 8,048-octet `captured_len = 0xFFFFFF` vector that reached `struct` through `bounded_area`'s own `nominal <= available` test being true for a negative nominal now parses to 200 frames. Measured over all 1,509 octet boundaries of `examples/captures/dhcp.pcapng` with one harness either side: 6 levels parsed before and 1,495 do now, against 1,479 `ValueError` and 10 `struct.error` before and none from outside the library after, with the frame count degrading monotonically with the cut; of the 14 levels that still raise, twelve leave a file too short to hold a block at all and the other two cut into the Interface Description Block's `if_tsresol` option, so none of them costs a frame that was in the file. That claim is about the truncation sweep rather than about every input: a 4,000-round bounded mutation fuzz either side of the change, same seed and cap, takes the parse rate from 2,118 to 3,438 and removes both of [#678](https://github.com/JarryShaw/PyPCAPKit/issues/678)'s families, and the two foreign families left are measured unchanged -- `ValueError: N is not a valid BlockType`, filed as [#701](https://github.com/JarryShaw/PyPCAPKit/issues/701), where an unassigned block type raises from `aenum` inside `EnumField.post_process` so the `UnknownBlock` default the registry declares is unreachable; and `MemoryError` at exactly 30 of 4,000 in both trees, which is [#593](https://github.com/JarryShaw/PyPCAPKit/pull/593)'s 32-bit band. A third bare exception in this module *is* fixed here, found by the cross-review and reachable from valid input rather than truncated: `SystemdJournalExportBlock.post_process` unpacked a 64-bit binary-field length out of whatever `entry_data.read(8)` returned, and since the block body is padded to a 32-bit boundary with NULs that `bytes.strip()` does not strip, that padding was read as a field *name* with no length prefix behind it -- so every journal entry of unaligned length raised a bare `struct.error`, measured on a 14-octet `MESSAGE=hello` entry -- a defect about ordinary input rather than about truncation. A NUL-only line now ends the entry and a short prefix ends it with a warning. The same function leaked two more of the family, found on the cross-review's second pass and fixed with it: a binary field's declared length is the widest in the format and nothing bounded it against the entry holding it, so at `2**63` and above `BytesIO.read` refused it with a bare `OverflowError` while below that it silently returned whatever was there -- the same malformed prefix fatal or invisible by magnitude alone, now clamped to what the entry has left and reported; and a field name, key or value that is not UTF-8 raised a bare `UnicodeDecodeError`, fatal to a whole extraction over one octet in one field, now decoded with `errors='replace'` and reported, which is the option this module's own `StringField` already takes. The separate silent loss of every field after a binary one, from a `read()` that reaches EOF where it means to skip one newline, is filed as [#704](https://github.com/JarryShaw/PyPCAPKit/issues/704) rather than fixed. The five deliberately unclamped non-packet option areas keep the per-block framing assumption [#676](https://github.com/JarryShaw/PyPCAPKit/pull/676)'s note describes, which this does not close. Also here because it is the fifth and last registrar in the package with no overwrite guard at all: `Option.register` now reports a displaced option schema as a `RegistryWarning` naming every namespace the `ns='opt'` fan-out displaced something in, once per registration rather than once per namespace, tested by membership rather than by subscripting because the per-namespace registries are plain `defaultdict`\ s and reading one to look would insert `UnknownOption` for a code nobody registered. The firing condition is identity-based here too, the same as the seven code-keyed parser registrars and the narrower guard [#681](https://github.com/JarryShaw/PyPCAPKit/pull/681) gave `register_protocol` ([#718](https://github.com/JarryShaw/PyPCAPKit/issues/718), [#726](https://github.com/JarryShaw/PyPCAPKit/pull/726)), even though this key is a caller-supplied `code` that `__init_subclass__` passes once per subclass; a namespace the call itself creates is exempt, because it starts as a copy of `opt`'s defaults and nothing in it is a prior registration. One draft did move `EXPECTED_FAILURES`, and it is worth recording: flooring the three decryption-secrets payloads that read `__length__` *whole* rather than subtracting from it packs nothing, because `Schema.pack` leaves that key at `-1` for "unknown" -- which emptied both payloads and turned `pcapng-secrets/TLS_Key_Log` and `.../WireGuard_Key_Log` from `MISMATCH` to `OK`, an empty payload comparing equal to an empty payload. Reverted and pinned; the entry count holds at 43 with its 35 PCAP-NG cases unmoved. Ships labelled breaking: well-formed captures are byte-identical, verified by regenerating `examples/captures/pcapng.txt` either side of the change, but any truncated PCAP-NG now yields the frames before the cut where it previously raised -- so a caller reading "extraction raised" as "this file is unusable" gets a partial result instead, whose last frame may carry zero-padded octets -- and three exception classes change at the margins, eight short-file depths moving from `ProtocolError: unknown byteorder magic` to `StreamEOFError`. The two modules hold 99.93% coverage with their 25 new statements covered and their single miss flat, and `examples/captures/pcapng.txt` does not move further, leaving the [#683](https://github.com/JarryShaw/PyPCAPKit/pull/683) drift [#685](https://github.com/JarryShaw/PyPCAPKit/issues/685) tracked exactly as it was ([#678](https://github.com/JarryShaw/PyPCAPKit/issues/678)). -- HIP's `R1_COUNTER`/`R1_Counter` parameter packed 12 octets where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires 16, and `LOCATOR_SET` declared its `Length` in 4-octet units where [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1)'s `Length` is a byte count -- two independent defects, fixed together because the evidence for either needed the other out of the way first. `R1CounterParameter`'s `counter` was a `UInt32Field` where the RFC states the R1 generation counter's width twice, as "8 bytes" in the diagram and as "a 64-bit unsigned integer" in prose; both HIP versions whose *packing* reaches this class -- `R1_Counter` (128) and `R1_COUNTER` (129) -- packed a 12-octet record, landing at `4 (mod 8)` instead of aligned. It is a `UInt64Field` now, and the record is 16 octets ([#672](https://github.com/JarryShaw/PyPCAPKit/issues/672)). `LocatorSetParameter`'s own `Length` was written as `sum(Locator.len)`, in the 4-octet units [RFC 8046 Section 4](https://datatracker.ietf.org/doc/html/rfc8046#section-4) gives `Locator Length`, where this parameter's own `Length` is a byte count; a set of *n* plain IPv6 locators therefore declared `4n` octets against `24n` actually present, and `Schema.unpack` handed the nested `ListField` only the declared octets, so `n = 2` and `n = 5` both parsed one truncated locator and left the rest of the record unconsumed. `HIP._make_param_locator_set` now sums `8 + Locator.len * 4` per locator -- the fixed header plus the RFC 8046 contents each one actually carries ([#679](https://github.com/JarryShaw/PyPCAPKit/issues/679)). A second, independent defect in the same parameter's padding is fixed alongside it, one [#651](https://github.com/JarryShaw/PyPCAPKit/issues/651) deliberately left alone because it happened to cancel this one exactly: the nested `Locator` schemas share the parameter's packet context and pack their own `len` over it, so `padding` -- evaluated after the list -- always saw 4 rather than the parameter's real length. A new `locator_set_len_callback` snapshots `Length` under a private key before any locator packs into the context, and `locator_set_padding_len` reads that snapshot rather than the shadowed one. An RFC-only byte-stride walk over `examples/captures/options-internet.pcap`, independent of pcapkit's own parser, went from 1 violation -- `LOCATOR_SET`'s empty record, concealed until now because `R1_COUNTER`'s `counter` field defaulted to `0` with no override anywhere in `examples/generators/options.py`, a value the width defect could not be told apart from -- to 2 once the counter was patched non-zero, to 0 once both parameters were fixed; the fixture now overrides both `R1_COUNTER` codes with `counter=0xaabbccdd` so a future regression cannot hide behind a zero again. 10 new test methods across two new files -- 4 in `test_hip_r1_counter_width_unit.py`, 6 in `test_hip_locator_set_length_unit.py` -- fail against the unfixed code and pass against the fix; the three HIP modules stay at 100% coverage, and `tests/` subtests move 586 to 623. `EXPECTED_FAILURES` is 43 entries: the `hip-parameter/R1_Counter` entry -- code 128 registered no schema of its own, since `R1CounterParameter` declared only `code=129` -- was deleted once that separate registry defect was fixed as [#690](https://github.com/JarryShaw/PyPCAPKit/issues/690). `HIP_COPIES` was dropped to one as [#689](https://github.com/JarryShaw/PyPCAPKit/issues/689), once [#672](https://github.com/JarryShaw/PyPCAPKit/issues/672) and [#679](https://github.com/JarryShaw/PyPCAPKit/issues/679) left it routing around nothing. -- `httpv1.HTTP` raised a bare exception for any payload that is not an HTTP/1 message, at four sites: the header/body split in `read`, the header's CRLF split and the start-line split in `_read_http_header`, and `item[1]` on a field line with no colon. `HTTP._guess_version` falls through on `ProtocolError` alone, so the HTTP/1 attempt aborted the guess rather than failing it and the HTTP/2 arm below was dead code -- for every one of those four input classes, not just the first. All four now raise `ProtocolError`, which each method's `Raises:` section already documented, and a test pins the invariant rather than the list: nothing escapes `httpv1.HTTP` that is not a `ProtocolError`, so a fifth bare-raising site would fail there. An obs-fold continuation line [[RFC 9112 Section 5.2](https://datatracker.ietf.org/doc/html/rfc9112#section-5.2)] is legal HTTP/1 and is now unfolded, the RFC's own remedy, rather than read as a field of its own: with no colon it raised that `IndexError`, and with one it parsed in silence into a spurious extra field. Reaching the HTTP/2 arm exposed a payload under nine octets to `httpv2.HTTP` for the first time, where it usually fails inside the schema with a bare `struct.error` -- neither a `ValueError` nor a `ProtocolError` nor pcapkit's `StructError`, so no caller could catch it; `read` converts it and `_guess_version` suppresses it on the last arm only, since an arm that is not last hands the payload onward when it swallows. Separately, `PayloadField.protocol` looked a `str` up in `__proto__` as given while the registry is keyed on the upper-cased class name, so a name not already upper-case resolved to `None` and silently yielded `Raw`; it now folds case and warns with `RegistryWarning` for a genuinely unregistered name, and `__init__` assigns through that property instead of writing `_protocol` past it ([#787](https://github.com/JarryShaw/PyPCAPKit/issues/787)). -- a systemd journal entry whose last field lacks the format's mandatory trailing newline let the PCAP-NG block's own 32-bit alignment padding be read as part of that field's value, with no exception and no warning: `b'MESSAGE=hello'` plus three padding NULs returned `MESSAGE == 'hello\x00\x00\x00'`. The padding-only-line guard from [#678](https://github.com/JarryShaw/PyPCAPKit/issues/678) only catches padding that lands on a line of its own, which requires the preceding field to have ended with a real newline; without one `readline()` runs straight through the value and into the padding behind it. Reachable on text fields only -- `readline()` returns a line without its own newline at end of stream alone, so a binary field's length-prefixed value read is never in play, and a binary field *name* landing on such a line fails its own length-prefix read regardless. The fix is tolerant rather than refusing: block padding is 0-3 octets, so at most the trailing three octets of a terminator-less line are stripped as padding, with a `SchemaWarning`, and anything past that cannot be padding and is left as data. The gate reads the *raw*, unstripped line, since a real trailing newline -- or any other raw last octet, ordinary ASCII whitespace included -- proves the true padding is zero ([#794](https://github.com/JarryShaw/PyPCAPKit/issues/794)). `examples/generators/pcapng.py`'s `_journal_entry` cited a bare line number for the defect its alignment workaround exists to avoid, and mis-attributed the fix; it now names [#678](https://github.com/JarryShaw/PyPCAPKit/issues/678)'s own fix, the `not line.strip(b'\x00')` guard, and the mechanism instead, the parser rewrite having only moved that guard ([#791](https://github.com/JarryShaw/PyPCAPKit/issues/791)). -- `httpv2.HTTP.read` tested only the 24-bit declared length off the wire (`schema.length < 9`), never how many octets the buffer actually held, so a frame backed by far fewer real octets than it declared still parsed and reported the declared, attacker-controlled length. Sweeping the declared length against a four-octet buffer gave non-monotone accept/reject -- 9, 15, 65535 and 16777215 parsed, 10 and 100 did not -- an artifact of `_read_http_settings`'s own unrelated `(declared - 9) % 6` check rather than evidence the buffer held what was declared. `HTTP.unpack` now rejects a non-empty buffer under nine octets before the schema layer runs, preserving `StreamEOFError` for a genuinely exhausted implicit-length stream, and `read`'s guard also requires the declared length not to exceed the available buffer. A buffer that clears nine octets can still carry a frame type whose own fixed-width fields exceed what is left after the header -- GOAWAY at 9-16 octets, PUSH_PROMISE at 9-12, padded DATA, HEADERS and PUSH_PROMISE -- and still crash the schema layer with a bare `struct.error`; that root cause is in the generic `Schema.unpack`/`FieldBase.length` machinery shared by ten schema modules, out of scope there and filed as [#805](https://github.com/JarryShaw/PyPCAPKit/issues/805), which is why `_guess_version`'s last arm keeps suppressing `struct.error` alongside `ProtocolError` ([#802](https://github.com/JarryShaw/PyPCAPKit/pull/802)). -- `SystemdJournalExportBlock.post_process` skipped a binary field's one trailing newline with a bare `entry_data.read()`, which reads to **EOF** rather than past that one octet: the entry loop's next `readline()` then returned `b''` and ended the entry, discarding every field behind the first binary one, however well-formed. The skip is now bound to `entry_data.read(1)`; a missing or non-`b'\n'` octet ends the entry with a `SchemaWarning` instead of resynchronising at the wrong offset. `test_journal_fields_following_a_binary_field_are_not_discarded` fails against the unfixed code with `MissingKeyError: 'SECOND'` and passes after ([#704](https://github.com/JarryShaw/PyPCAPKit/issues/704)). -- the same block's entries were split on a bare `b'\n\n'`, which shreds a binary field whose value happens to contain that byte pair. Walking the entry instead of splitting it introduced its own regression on first review: a `malformed` flag broke the *outer* per-block loop on a bad terminator, ending the whole block rather than just the one entry and silently dropping every well-formed entry behind it. Fixed by dropping `malformed` entirely, leaving the existing per-entry `break` as the only effect. A second, independent gap closed in the same change: a trailing separator landing on the block's last octet was indistinguishable from EOF and swallowed, so a rebuild wrote `length` one octet short ([#723](https://github.com/JarryShaw/PyPCAPKit/issues/723)). -- **a breaking change to** four `.get()`-backed enum fields: TCP/UDP/SCTP's `PortEnumField` and PCAP-NG's `OptionEnumField` minted a member through `aenum.extend_enum` for every value no row or documented span covers -- 111 calls on `http.pcap`, all ephemeral ports. All four now peek at what `.get()` would consult and mint only once the value is actually unresolvable; a genuine miss gets `EnumField._unregistered_member` instead, a real member of the same registry built through its storage base's `__new__`, so `isinstance` still holds and the lookup table stops growing. `extend_enum` calls drop 111 to 0 on `http.pcap` and 15 to 2 on `many_interfaces.pcapng` (both from the untouched `_missing_`). Riding along: PCAP-NG option namespaces stop cross-contaminating -- on the unfixed tree, once a code was minted under `opt`, `OptionType.get` merged `opt` over the requested namespace and 6 of the other 7 namespaces all answered `opt_unknown` for it, where each now gets its own `_unknown`. **Breaking** because [#575](https://github.com/JarryShaw/PyPCAPKit/issues/575) asked for byte-identical output across the sample captures, as [#420](https://github.com/JarryShaw/PyPCAPKit/pull/420) and [#427](https://github.com/JarryShaw/PyPCAPKit/pull/427) did, and this deliberately misses that bar: 14 of 23 captures change, entirely from the new `` rendering plus the PLIST escaping above; normalising both leaves all 14 byte-identical. A `pickle` round-trip of a resolved unassigned member worked before this change (it was minted, so registered) and would have broken silently on read-back after; `_unregistered_member` now installs a `__reduce_ex__` rebuilding an equivalent unregistered member instead, verified on protocols 0-5 under both picklers and `copy`/`deepcopy` ([#575](https://github.com/JarryShaw/PyPCAPKit/issues/575)). -- `httpv2._guess_version` tried to parse a stream as HTTP/2 and read failure as "not HTTP/2," rather than identifying the version first, so a genuine connection preface followed by a real `SETTINGS` frame still raised `ProtocolError: unknown HTTP version` instead of parsing. `_guess_version` now identifies before it parses: a preface is recognised as HTTP/2 outright, skipped rather than fed to `httpv2`, and counted as header so `info` stays byte-identical to reading the following frame alone -- without which the injected `packet=self.packet.payload`, sliced from octet 9 of a buffer whose frame starts at 24, reported preface remnants as payload. A preface with no frame behind it now raises `ProtocolError: HTTP/2: connection preface with no frame`, naming the real cause; a bare mid-stream `SETTINGS` frame, an `Upgrade: h2c` request and non-HTTP input are all unchanged, the first two left undecidable on purpose -- `Upgrade: h2c` needs per-connection state a single payload cannot carry, and a mid-stream frame has no safe heuristic, since a `type <= 9` guess misfires on binary HTTP/1 bodies. Protochain over all 23 sample captures (1,604 frames, 231 HTTP-bearing) is byte-identical to the unfixed tree ([#800](https://github.com/JarryShaw/PyPCAPKit/issues/800)). -- `TCP.__proto__` bound `httpv1.HTTP` directly for ports 80 and 8080, so a segment on either port was HTTP/1 by assertion of the port number alone; both versions share those ports on the wire, so the port cannot decide the version and the payload has to. Repointed to the generic `pcapkit.protocols.application.http.HTTP` proxy, whose identification became positive only once [#800](https://github.com/JarryShaw/PyPCAPKit/issues/800) and [#814](https://github.com/JarryShaw/PyPCAPKit/pull/814) landed, which is why this waited. `udp.py` already bound the proxy for both ports; this removes the asymmetry its docstring and `docs/source/pep.rst` documented as an open request, prose-only on that side since its port rows already pointed at the proxy. Protochain over all 23 sample captures (1,604 frames) is *not* byte-identical, and that is the fix: all 231 HTTP/1.1 frames keep their chain, and nine frames in `options-transport.pcap` change `Ethernet:IPv4:TCP:Raw` to `Ethernet:IPv4:TCP:HTTP/2` -- genuine HTTP/2 frames this library's own `httpv2.HTTP.make` built, previously refused by the HTTP/1 parser. `_guess_version`'s entry count over the corpus goes 0 to 252: 231 HTTP/1.1, 9 HTTP/2 and 12 fall-throughs that stay `Raw`. Not labelled breaking, but not free: TCP:80/8080 traffic that is neither valid HTTP/1 nor preface-carrying is now exposed to `_guess_version`'s fall-through arm, where the direct `httpv1` binding used to leave it `Raw` outright regardless of shape -- the 12 fall-throughs above are that arm firing on this corpus, and a caller depending on the old direct bind for traffic that is neither is the one who would notice ([#682](https://github.com/JarryShaw/PyPCAPKit/issues/682)). -- **a breaking change to** 8 more bespoke registries: step 2 of [#860](https://github.com/JarryShaw/PyPCAPKit/issues/860), PR 1 of 2 (`AppType` is PR 2, [#874](https://github.com/JarryShaw/PyPCAPKit/pull/874) below), bringing `StatusCode`, `ReturnCode`, `ResponseKind`, `GroupingInformation`, `OptionType`, `FEATCode`, `Command` and `Method` onto `EnumRegistry` and applying [#775](https://github.com/JarryShaw/PyPCAPKit/issues/775)'s mint/unmint ruling to their `get()` as well as their `_missing_`. The first five convert 12 unambiguous placeholder branches (`Unassigned`, `Unknown`, `opt_unknown`) to `_unregistered_member`; each of the three with a custom `__new__` (`StatusCode`, `ReturnCode`, `OptionType`) gets its own override reconstructing the attributes the base's generic helper would otherwise leave unset, and `StatusCode`/`ReturnCode`'s hand-written `get()` -- still on the retired `default == -1` convention -- is replaced by the base's, with no caller in this tree found relying on the old form. `OptionType` keeps its own `get()` for its genuine multi-namespace dispatch, but a round-2 review found `get()` itself still minted on both its int/namespace path and its `str` path -- the live pcapng parse path (`PCAPNG._make_pcapng_options`) calls it with wire bytes, so parsing an undeclared option code was still registering a permanent member; both paths now build an unregistered member too. `FEATCode`, `Command` and `Method` mint the literal, unmodified wire value as its own name rather than any manufactured placeholder; per the owner's ruling (*"get will not have sufficient information to create new ones"* -- `Command` needs `feat`/`desc`/`type`/`conf` and `Method` needs `safe`/`idempotent`, neither of which a bare wire string carries), both `_missing_` and each class's own `get()` -- a second, independent mint site bypassing `_missing_` -- now build an unregistered member instead. `FEATCode`'s own crawler now declares all 15 real FEAT-code values from the live IANA table instead of minting 10 of them as a side effect of evaluating `Command`'s rows at import time, the same import-time-mutation shape [#861](https://github.com/JarryShaw/PyPCAPKit/pull/861) removed from `FilterType`; pinned count-agnostically so a future IANA update cannot fail a correct regeneration. Untouched, per explicit rulings: `CommandType` stays `IntFlag` (real `A|P` composites in the generated data), and `TransportProtocol`'s `auto()` renumbering plus `AppType` itself are PR 2's territory. All five crawlers regenerate byte-identically on a third run. 332 test methods pass across the seven touched test files plus `tests/protocols/misc/test_pcapng_unit.py`; touched const modules land at 92-100% coverage. `pcapkit/protocols/schema/misc/pcapng.py`'s `OptionEnumField.post_process` docstring is corrected to explain that it still bypasses `OptionType.get()` directly -- now for `pickle` round-trip safety, since [#860](https://github.com/JarryShaw/PyPCAPKit/issues/860) already stops the minting that used to be the reason, and neither of `OptionType`'s own two paths gets both pickling and correct rendering right at once ([#869](https://github.com/JarryShaw/PyPCAPKit/pull/869)). +- `TCP._make_mptcp_addaddr` could not build an `ADD_ADDR` option end to end: its `kind=`/`length=` arguments were rejected with `UnknownFieldWarning` and dropped, and `.pack()` then raised `KeyError: 'length'` from `port`'s own condition, `pkt['length'] in (10, 22)`. The cause was one layer up: `MPTCP`, the base of every Multipath TCP subtype schema, declared `kind` and `length` only under `typing.TYPE_CHECKING` rather than as real fields, unlike `Option`. That dropped `kind=`/`length=` for every `_make_mptcp_*` constructor, so `MPTCP` now declares both for real ([#541](https://github.com/JarryShaw/PyPCAPKit/issues/541)). The missing fields broke parsing too: a subtype schema's leading field read the `kind` octet itself, an off-by-two in field alignment rather than a wire-format change (a correct sender's octets were always right). Spec-correct `ADD_ADDR` and `MP_PRIO` options failed with `FieldError: TCP: [OptNo 30] 3 invalid IP version` and `KeyError: 'length'`; both parse now. +- two dropped-keyword/wrong-cast defects flagged in review during this release and never filed until now: HIP's `_make_param_encrypted` passed `cipher=` to a schema with no such field, so an AES-cipher `ENCRYPTED` parameter built through `make` packed without its IV; and IPv6-Route's `RPL.post_process`, which runs on every `Schema.pack`, assumed `self.addresses` was still the concatenated `bytes` a parse leaves, and raised slicing the `list[bytes]` a `make`-built multi-address header holds ([#556](https://github.com/JarryShaw/PyPCAPKit/issues/556)). +- two more defects [#541](https://github.com/JarryShaw/PyPCAPKit/issues/541) exposed by letting construction reach code that had never run. `MPTCP.subtype` was still `typing.TYPE_CHECKING`-only, so `TCP(options=[(Enum_Option.Multipath_TCP, ...)])` raised `AttributeError: ... has no attribute 'subtype'` for every subtype but `MP_JOIN`: the convenience constructor reads the schema straight back through `_read_mptcp_*` with no byte round trip, so `_MPTCP.post_process`, the only code that set `subtype`, never ran. Fixed on the construction path (`TCP._make_mode_mp`) rather than by adding a third real field the way `kind`/`length` got in [#541](https://github.com/JarryShaw/PyPCAPKit/issues/541): `subtype` is already packed as 4 bits of each subtype's own `test` bitfield, and a second field for the same bits would double-encode them or need a "derive, don't pack" field kind this library lacks ([#566](https://github.com/JarryShaw/PyPCAPKit/issues/566)). Separately, `_make_mptcp_capable` wrote `length=20 if rkey is None else 32` where [RFC 8684](https://datatracker.ietf.org/doc/html/rfc8684) section 3.1 gives 12 and 20, and `MPTCPCapable.rkey`'s condition (`pkt['length'] != 32`) dropped the receiver's key for exactly the length the maker used for "key present": a key-absent MP_CAPABLE could not be built, and a key-present one lost its key on the wire. Both, and the guard in `_read_mptcp_capable`, now agree on 12/20. **This changes MP_CAPABLE's packed output**: a 20-octet key-present option built or parsed under the old code becomes 12 octets with no key, or 20 octets with the key present, depending on which the caller meant ([#567](https://github.com/JarryShaw/PyPCAPKit/issues/567)). +- IPv6-Route's RPL routing data was five octets wide at the front where [RFC 6554](https://datatracker.ietf.org/doc/html/rfc6554) section 3 gives four, with three further defects behind it. `CmprI`, `CmprE` and `Pad` are each 4 bits sharing a 32-bit word with a 20-bit `Reserved`, but the schema declared `cmpr_i` and `cmpr_e` as whole octets, so a constructed two-address header packed to 41 octets while its `Hdr Ext Len` of 5 declared 48. Correcting the width is a **wire-format change** and reshapes the schema: `RPL(cmpr_i=..., cmpr_e=...)` is now `RPL(cmpr={'cmpr_i': ..., 'cmpr_e': ...})`, beside the existing `pad={'pad_len': ...}`. Behind it, the reader's `header.length % 16` guard read `Hdr Ext Len` as an octet count and assumed 16-octet addresses, which an SRH carries only when `CmprI` and `CmprE` are both 0 -- the unit confusion [#487](https://github.com/JarryShaw/PyPCAPKit/issues/487) fixed for Source Route and Type 2, left by [#489](https://github.com/JarryShaw/PyPCAPKit/pull/489) for want of a working RPL round trip to validate against. It is replaced by section 4.2's address-count arithmetic, `n = (((Hdr Ext Len * 8) - Pad - (16 - CmprE)) / (16 - CmprI)) + 1`, which the reader now requires to be non-negative and whole. `RPL.post_process` subtracted `pad_len` a second time from a buffer whose length callback had already taken it off, losing one address per `16 - CmprI` octets of padding, and set `ip` only when it had parsed octets, so `IPv6_Route(type=..., data={'ip': [...]})` raised a bare `AttributeError: 'RPL' object has no attribute 'ip'`. And `_make_data_type_rpl` computed `Pad` as `8 - length % 8` without the outer `% 8`, so an aligned address vector got a full 8 octets of padding where section 3 requires `Pad` to be 0 when `CmprI` and `CmprE` are both 0. The narrower `cmpr_i` also removes a latent divide-by-zero (a whole octet could hold 16, making `16 - CmprI` zero). `ipv6-route-type/RPL_Source_Route_Header` round-trips and its `EXPECTED_FAILURES` entry is deleted; none of this has been checked against a real RPL capture ([#564](https://github.com/JarryShaw/PyPCAPKit/issues/564)). +- `L2TPv2` parsed any version nibble, so an L2TPv3 datagram was reported as v2 with a tunnel and session ID read out of v3's Control Connection ID. [RFC 2661](https://datatracker.ietf.org/doc/html/rfc2661) §3.1 fixes `Ver` at 2 and reserves 1 for L2F, and `L2TPv2.version` documented that any other value "is a different protocol reached through a different class", but nothing enforced it, so the hard-coded `Literal[2]` property and `info.version` answered 2 and 3 on the same octets. `read` now raises `ProtocolError`, degrading the payload to `Raw` through the existing `beholder` path with the reason recorded. **This affects real captures**: [RFC 3931](https://datatracker.ietf.org/doc/html/rfc3931) §4.1.2 puts L2TPv3 on port 1701 too, the port `UDP.__proto__` binds, so `Ethernet:IPv4:UDP:L2TPv2:Raw` with invented field values becomes `Ethernet:IPv4:UDP:Raw` with the octets preserved. + +This resolves [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548), which reported `TransType.L2TP` (115) as registered nowhere and proposed binding `L2TPv2` there. That binding is wrong rather than awkward: [RFC 3931](https://datatracker.ietf.org/doc/html/rfc3931) §4.1.1 gives 115 to *L2TPv3 over IP*, whose session header carries **no version nibble at all**, so a v2 parser cannot detect that the datagram is not its own. 115 is a missing *class*, not a missing registration, and stays unbound until an `L2TPv3` class exists; no dissector was invented to fill it. The reasoning is recorded in `pcapkit.protocols.link.l2tp`, and `register_protocol_code`'s worked example, which named `L2TPv2` at 115, names `L2TPv3` instead ([#548](https://github.com/JarryShaw/PyPCAPKit/issues/548)). +- building any `MP_JOIN` option raised `AttributeError: 'TCP' object has no attribute '_flags'`. `TCP.make` constructed the options *before* assigning the `self._flags` that the option makers read, and `_make_mptcp_join` branches on it to choose among the three layouts of [RFC 8684](https://datatracker.ietf.org/doc/html/rfc8684) section 3.2 (figure 5, SYN, 12 octets; figure 6, SYN/ACK, 16; figure 7, ACK, 24). The parse path was never affected, since `read` assigns the flags first. Fixed by hoisting the flag resolution above the `_make_tcp_options` call, leaving only the data-offset computation, which needs the options' total length, after it. **Not** fixed by a zero default for `_flags`, which would have been worse: `Protocol.pack` is public and calls `make`, so an instance that had parsed a segment had the attribute set and silently built the option for the segment it had *read*. A parsed MP_JOIN-SYN instance asked to pack an MP_JOIN-ACK segment emitted an ACK header carrying figure 5's 12-octet SYN option, with the caller's 20-octet HMAC replaced by an all-zero token and nonce, and nothing raised. Hoisting also made one branch reachable for the first time, an MP_JOIN on a segment with neither SYN nor ACK: the accumulator was seeded with `cast('Enum_Flags', 0)`, and `typing.cast` being a runtime no-op, `self._flags` stayed a plain `int` and the first membership test raised `TypeError` rather than the documented `ProtocolError`. It is now seeded with `Enum_Flags(0)`, a real flagless member that equals `0` and ORs identically. The read path's seed is unchanged: a flagless MP_JOIN is rejected by `mptcp_data_selector` before `_read_mptcp_join` runs ([#587](https://github.com/JarryShaw/PyPCAPKit/issues/587)). +- the ILNP Nonce option builders in `HOPOPT` and `IPv6_Opts` sized the option with `math.ceil(nonce.bit_length() // 8)`, floor division dressed as a ceiling (`//` floors and `math.ceil` of an `int` is a no-op). The nonce is packed by a `NumberField` whose width *is* the declared `len`, so an under-declared length silently truncated the nonce on the wire. Every nonce whose bit length was not a multiple of eight was affected, and **small values were the worst case rather than boundary values**: any nonce below 256 was declared as *zero* octets and dropped (`nonce=9` packed to `b'\x8b\x00'` and parsed back as `0`). Fixed to `max(1, math.ceil(nonce.bit_length() / 8))`, the form used at five other sizing sites in `hip.py` and `mh.py`. The one-octet floor is the `mh.py` convention and is load-bearing because `nonce` defaults to `0`, whose bit length is `0`: without it the default builds an option with no Nonce Value field at all, collapsing "the nonce is 0" into "there is no nonce" when [RFC 6744](https://datatracker.ietf.org/doc/html/rfc6744) gives the option that field. The read path takes its width from the `len` octet on the wire and was never affected ([#601](https://github.com/JarryShaw/PyPCAPKit/issues/601)). +- `httpv1`'s `_RE_METHOD` was unanchored and `re.match` anchors only at the start, so it prefix-matched, and the request-line reader passed the whole `para1` to `Method.get` rather than the captured `method` group. So `b'Get'` matched on the single character `G`, satisfied the guard that decides a start-line is a request, and handed the mixed-case token to a lookup that raised on it. Fixing either half alone still gives a wrong answer (normalising the lookup would parse `b'Get'` as `GET` off a one-character match; passing the group would parse it as a method named `G`). The pattern is anchored at both ends and the captured group is looked up, so a non-method token is a malformed request line rather than a mis-parsed one. Method tokens are case-sensitive per [RFC 9110 Section 9.1](https://datatracker.ietf.org/doc/html/rfc9110#section-9.1), so no `re.I` was added: `GET` parses, `Get` and `get` are rejected ([#583](https://github.com/JarryShaw/PyPCAPKit/issues/583)). +- `_RE_STATUS` in the same reader had the same unanchored prefix defect, found by auditing `_RE_METHOD`'s siblings, and escaped as the wrong exception type. The pattern is only a guard (the value comes from `int(para2)` on the raw token), so a prefix match let a malformed status past the guard and out of `int()` uncaught, where `_read_http_header` documents `ProtocolError`: `200x` raised `ValueError: invalid literal for int()` and `2000` raised `ValueError: 2000 is not a valid StatusCode`; both are now `ProtocolError`. [RFC 9112 Section 4](https://datatracker.ietf.org/doc/html/rfc9112#section-4) gives `status-code = 3DIGIT`, exactly three, so the anchor is what the grammar said (the production lives in HTTP/1.1 because `status-code` is part of its `status-line`; [RFC 9110 Section 15](https://datatracker.ietf.org/doc/html/rfc9110#section-15) covers code semantics and the registry). `_RE_VERSION` was audited and is safe, since both call sites read the captured group ([#583](https://github.com/JarryShaw/PyPCAPKit/issues/583)). +- as a consequence of the above, the unhandled `MemoryError` at `pcapkit/protocols/protocol.py:1411` on a truncated PCAP-NG capture. `dhcp_little_endian.pcapng` cut to 161, 641 or 1389 octets left a one-octet read of a little-endian 32-bit block length, which `rjust()` turned into `0x78000000` or `0x84000000` (1.88 to 2.06 GiB) and passed straight to `self._file.read()` as an allocation size, from a 1772-octet file. Under a 1 GiB address-space cap all three raise `MemoryError`; with more space the parse fails anyway. With `ljust()` the same reads report 120 and 132, no large allocation is attempted, and all three end in the ordinary handled parse failure ([#604](https://github.com/JarryShaw/PyPCAPKit/issues/604)). +- reading a big-endian classic PCAP byte-swapped every record header field, and then crashed. `Frame.unpack` seeded the file's byte order under the key `bytesorder` where the schema's `byteorder_callback` reads `byteorder`, so the lookup never found it and fell back to `sys.byteorder`, the host's order rather than the file's. On a little-endian host reading a little-endian capture that is right by coincidence, and every capture in this repository was little-endian. Frame 1 of `big_endian.pcap` read `incl_len=1241513984` for a record whose real value is `74`. `incl_len` is the payload length, so that record consumed the whole file and the second was read with a negative length, raising `ValueError: read length must be non-negative or -1` -- the reported crash, and the *second* symptom. The sibling `Frame.pack` eleven lines earlier spelled the key correctly, marking this a slip rather than a second key, and the fallback made a misspelled key indistinguishable from an absent one; `byteorder_callback` now records that it defines the key and why the fallback hides a typo ([#605](https://github.com/JarryShaw/PyPCAPKit/issues/605)). +- documentation. `mptcp_dss_ack_selector`'s note said a corrected field-width lambda "would not have worked" and that fixing it belonged to `pcapkit.corekit.fields.numbers`, which is where [#598](https://github.com/JarryShaw/PyPCAPKit/pull/598) then fixed it; the same paragraph sat in `test_tcp_mptcp_length_arithmetic_unit.py`'s docstring, whose other stale claim was that MP_JOIN "cannot be built through the public `TCP()` constructor at all", true only until [#587](https://github.com/JarryShaw/PyPCAPKit/issues/587). A callable-length `NumberField` packs and unpacks both DSS widths now, and wire *absence* was never the obstacle: `MPTCPDSS.ssn`, `dl_len` and `checksum` have always been `ConditionalField` on the sibling `M` flag. The `SwitchField` form is kept for the narrower reason the note now gives: `ConditionalField`'s `length` forwards to the wrapped field without consulting the condition, so it is safe here only because `Schema.pack` and `Schema.unpack` special-case that wrapper by name, whereas a `SwitchField` always resolves to a concrete field. Replacing it would be a behaviour change and is not made ([#603](https://github.com/JarryShaw/PyPCAPKit/issues/603)). +- the one assertion [#604](https://github.com/JarryShaw/PyPCAPKit/issues/604) left pinning the old padding side, red on `mainline` since [#621](https://github.com/JarryShaw/PyPCAPKit/pull/621) merged. `TCPUDPUnitTests.test_a_truncated_option_still_parses_its_declared_length` expected a truncated TCP option's `data` as the synthesised zero octets *followed by* the real ones, as `rjust()` produced; [#621](https://github.com/JarryShaw/PyPCAPKit/pull/621) made the padding `ljust()` everywhere but could not retarget this file, since [#612](https://github.com/JarryShaw/PyPCAPKit/pull/612) owned it and editing concurrently risked discarding work that has since landed. The real octets now come first for both parametrised widths, and the docstring, which said the short read was *left*-padded, is corrected. Test-only; the sibling case in `tests/protocols/internet/test_ipv4_unit.py` was retargeted in [#621](https://github.com/JarryShaw/PyPCAPKit/pull/621) (two subtest failures before, none after; [#604](https://github.com/JarryShaw/PyPCAPKit/issues/604)). +- `main` went red the moment [#604](https://github.com/JarryShaw/PyPCAPKit/issues/604)'s `ljust()` landed, because that test still pinned the head-padded short read: `Reserved_79` declaring `length=12` over 6 real octets now reports `aabbccddeeff00000000` where the test expected `00000000aabbccddeeff`, failing both subtests (`declared_length=12` and `=32`) while the 17 other cases stayed green. The inputs discriminate: `trailing` is non-zero and the pad width is 4 and 24. [#621](https://github.com/JarryShaw/PyPCAPKit/pull/621) left the file alone because [#612](https://github.com/JarryShaw/PyPCAPKit/pull/612) owned it at the time, and merged two minutes ahead of the cross-review verdict that named it ([#604](https://github.com/JarryShaw/PyPCAPKit/issues/604), [#621](https://github.com/JarryShaw/PyPCAPKit/pull/621)). +- the HIP `SOLUTION` builder sized the parameter with `4 + math.ceil(max(random.bit_length(), solution.bit_length()) / 4)`, an invalid shorthand for two fields. [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5) gives `Length` as `4 + RHASH_len / 4` over a `Random #I` and a `Puzzle solution #J` of `RHASH_len / 8` octets **each**, and `/ 4` equals twice `/ 8` only because `RHASH_len` is a whole number of octets. For an arbitrary `bit_length()` the builder emitted an *odd* contents width, which its own reader refused: `_read_param_solution`'s `(schema.len - 4) % 2` guard exists because `SolutionParameter` splits that width into two equal halves. `random=0x1` with `solution=0xfff` declared `len=7` and raised `ProtocolError: HIPv2: [ParamNo 321] invalid format` on input the library had produced. The undersized length was worse: both fields take their width from that `len`, so `solution=0xfff` was packed into one octet and came back as `0xff`, silently. Fixed to `4 + 2 * math.ceil(max(...) / 8)`, the form the sibling `PUZZLE` builder uses, which is even by construction. Loosening the reader was rejected: an odd contents width has no meaning in the wire format (the RFC makes the fields equal-width and does not say which takes an extra octet), and a laxer guard would still have truncated the value ([#608](https://github.com/JarryShaw/PyPCAPKit/issues/608)). +- `TCP.read` seeded its connection-flag accumulator with `cast('Enum_Flags', 0)`. `typing.cast` is a runtime no-op, so the accumulator began as the plain `int` `0`, and only the `|=` promoting it to a `pcapkit.const.tcp.flags.Flags` member ever changed that. A segment with an all-zero flags octet (an nmap NULL scan) left `self._flags` an `int`, so a membership test raised `TypeError: argument of type 'int' is not a container or iterable`, and `TCP.connection` returned an `int` where it annotates `Flags`. Any flag masked it, which is how it survived 100% statement and branch coverage. The seed is `Flags(0)`, as [#597](https://github.com/JarryShaw/PyPCAPKit/pull/597) already did for the sibling accumulator in `make`; `Flags` declares no `_missing_`, so `Flags(0)` is an ordinary `aenum.IntFlag` pseudo-member and does not meet the `RecursionError` the same construction hits on the `pcapkit.const.mh` flag enumerations ([#623](https://github.com/JarryShaw/PyPCAPKit/issues/623)). Nothing a caller can reach changed: the three MP_JOIN layouts of [RFC 8684](https://datatracker.ietf.org/doc/html/rfc8684) section 3.2 are chosen from these membership tests, but `mptcp_data_selector` rejects a flagless MP_JOIN with a `FieldError` first, so the `TypeError` was latent, and only by virtue of a guard in another file. Called directly on a flagless parsed segment the dispatcher now raises `ProtocolError` naming an invalid flags combination. One difference *is* observable, a dump-format change rather than a value change: a flagless segment's `connection` renders as the string `Flags::None [0]` where it rendered as the number `0`, in the JSON, tree and PLIST outputs. The value and wire bytes are unchanged; `connection` no longer switches JSON type with the flags (a number for a flagless segment, a string such as `Flags::ACK [2048]` for every other). The literal `None` is a separate rendering defect in the hook `pcapkit.dumpkit.common.make_dumper` installs, which interpolates `o.name` without accounting for a nameless composite member; it is left to its own change and pinned by a test. The committed example dumps are unaffected (`examples/captures/in.pcap` has no flagless segment). Coverage cannot see this fix, since the changed line already executed; the added subtests are the evidence ([#616](https://github.com/JarryShaw/PyPCAPKit/issues/616)). +- **a breaking change to a public attribute.** `Frame.len` and `Frame.cap_len` were filled from *opposite* wire fields depending on the container format, so `frame.len` meant the captured length out of a `.pcap` and the on-wire length out of a `.pcapng`. The PCAP reader moved to match PCAP-NG: `len` is the on-wire length (the record header's `orig_len`) and `cap_len` the octets stored (`incl_len`). **Code reading `frame.len` or `frame.cap_len` from a `.pcap` gets the other field's value than before**, and for a truncated frame that is a different number rather than a relabelling. Which reader to move was a decision, not settled by seniority: the PCAP spelling is the *older* (`c43892af`, 2022-01-11, with the data model's docstrings agreeing a day later), the opposite spelling in `pcapkit.toolkit.pcapng` arrived 15 months later in `25f216f4` (2023-04-27), and both were internally consistent. What settles it is that the *names* are Wireshark's: `epan/dissectors/packet-frame.c` registers `frame.len` as "Frame length on the wire" and `frame.cap_len` as "Frame length stored into the capture file", and raises `frame.len_lt_caplen` expert info (`PI_MALFORMED`/`PI_ERROR`) on `frame_len < cap_len`, which could not be malformed if `len` were the smaller, captured one. Both formats define the underlying fields alike (`pcap-savefile(5)`'s `incl_len`/`orig_len`, `draft-ietf-opsawg-pcapng`'s Captured/Original Packet Length). The internal consumer moved with it: `Frame.read` hands `_decode_next_layer` the octets *present*, now `frame.cap_len`; handing over the on-wire length would be the declared-length-exceeds-available-octets fault of [#554](https://github.com/JarryShaw/PyPCAPKit/issues/554), [#573](https://github.com/JarryShaw/PyPCAPKit/issues/573) and [#594](https://github.com/JarryShaw/PyPCAPKit/issues/594) on every snapped frame. The dumpers are untouched (`PCAPIO` packs its record header from `frame_info.incl_len` and `orig_len`, already right in both readers). It went unnoticed because the two lengths differ only for a snapshot-cut frame and no such fixture existed until [#614](https://github.com/JarryShaw/PyPCAPKit/pull/614) added one (`big_endian.pcap`'s third frame, 1200 octets on the wire against 96 captured), so every earlier assertion compared a value against itself ([#618](https://github.com/JarryShaw/PyPCAPKit/issues/618)). +- the HIP `PUZZLE` and `SOLUTION` builders derived three wire-format values from the payload value instead of the data model, and all three were wrong. **This changes two public data models and the octets both parameters emit.** The width of `Random #I` and `Puzzle solution #J` came from `int.bit_length()` alone, so every leading zero octet was dropped on re-serialisation: a `SOLUTION` read with `Length = 20` rebuilt as `Length = 6`, a `PUZZLE` with `Length = 12` as `Length = 5`, silently. That width is `RHASH_len / 8` octets [[RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4), [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5)], a property of the Responder's HIT Suite, so both data models now carry it as `rhash_len` and both builders prefer it; a new `rhash_len=` keyword states it for a build from scratch, which under HIPv2 is the only place it can come from, since `RSA,DSA/SHA-256` is the REQUIRED HIT Suite [[RFC 7401 Section 5.2.10](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.10)] and makes the field 32 octets rather than 8. On the mandatory path about one parameter in 256 has a zero top octet and lost it ([#653](https://github.com/JarryShaw/PyPCAPKit/issues/653)). `SOLUTION`'s second contents octet is `Reserved`, "zero when sent, ignored when received" [[RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5), and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) identically], not the `Lifetime` that only [RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4) defines, which pcapkit encoded there as `2^(value - 32)` seconds. It wrote `0x20`, `0x21`, `0x25` or `0x2b` into a field the RFC requires to be zero, and could not write that zero: a conformant `SOLUTION` with `Reserved` `0x00` parsed to `timedelta(0)` and could not be re-serialised, escaping a bare `ValueError` from `math.log2(0.0)` that was not a `BaseError`. The field is renamed `reserved`, defaults to zero, and is carried verbatim across a round trip ([#654](https://github.com/JarryShaw/PyPCAPKit/issues/654)). And neither builder read its own `version` keyword, so `version=1` and `version=2` computed identical lengths, where [RFC 5201 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.4) and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) state both fields as literally 8 bytes and `Length` as literally 12 and 20. Under HIPv1 each builder accepted only a `bit_length()` of 57..64 and built, for anything narrower, a parameter this library's reader rejects -- the shape of [#608](https://github.com/JarryShaw/PyPCAPKit/issues/608), in the `PUZZLE` builder that [#629](https://github.com/JarryShaw/PyPCAPKit/pull/629) never opened ([#655](https://github.com/JarryShaw/PyPCAPKit/issues/655)). The two remaining `math.log2` sites, both in `PUZZLE` where a lifetime is real, raise `ProtocolError` (`(BaseError, ValueError)`, so a caller written around the bare exception still catches it). Migration: `SolutionParameter.lifetime` is now `reserved` and an `int` rather than a `timedelta`; `PuzzleParameter` and `SolutionParameter` gained a required `rhash_len`; and `_make_param_solution` no longer takes `lifetime=`. All three were reachable end to end only once [#608](https://github.com/JarryShaw/PyPCAPKit/issues/608) was fixed in [#629](https://github.com/JarryShaw/PyPCAPKit/pull/629), which removed the parity guard that had failed these rebuilds loudly; the round trip then stopped raising and started quietly emitting a different parameter, which is why they were fixed together ([#653](https://github.com/JarryShaw/PyPCAPKit/issues/653), [#654](https://github.com/JarryShaw/PyPCAPKit/issues/654), [#655](https://github.com/JarryShaw/PyPCAPKit/issues/655)). +- every HIP parameter pcapkit emitted was `4 (mod 8)` octets long, because the padding aligned the *contents* rather than the record. Corrected for **45 of the 46** HIP parameters; `LOCATOR_SET` is excluded, below. **This changes the octets those 45 parameters write, and the `length` their data models report.** [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1) requires every TLV parameter to have "a length (that includes the Type and Length fields), which is a multiple of 8 bytes" and gives `Total Length = 11 + Length - (Length + 3) % 8`; all 95 padding sites computed `(8 - (Length % 8)) % 8`, which aligns the contents alone. For every `Length` from 0 to 63 the result was never a multiple of eight and never the RFC's value, erring both ways: at `Length = 4` (a whole `SEQ`, which [RFC 7401 Section 5.3.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.3.5) puts on every `UPDATE`) pcapkit appended four octets that must not be there, and at `Length = 8` it appended none and went out four short. A conformant peer lands mid-parameter; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test could see it and why the fix is asserted against the RFC's arithmetic. 94 of the 95 sites (45 of 46 `PaddingField` callbacks in `pcapkit/protocols/schema/internet/hip.py` and 49 of 49 record lengths in `pcapkit/protocols/internet/hip.py`) are now one `parameter_total_len` and one `parameter_padding_len` ([#651](https://github.com/JarryShaw/PyPCAPKit/issues/651)). `LOCATOR_SET` kept the old expression because two defects there cancelled exactly: its padding callback never received the parameter's `len` (the nested `Locator` schemas share a packet context whose own `len` shadowed it, so it saw 4 for an IPv6 locator), and the parameter's `len` was written in 4-octet units where the RFC's `Length` is a byte count; the result, `4 + 24n + 4`, equalled the RFC total `24n + 8`. [#679](https://github.com/JarryShaw/PyPCAPKit/issues/679) later fixed both sites. `EncryptedParameter`'s `data` length callback is fixed in the same change: it subtracted the sixteen `iv` octets but not the four `reserved` ones, which cancelled the padding's four at `Length % 8` in `{0, 5, 6, 7}`, so correcting the padding alone would have made `ENCRYPTED` four octets too long at all eight residues. `HIP.make`'s `len = total_length // 8 + 4` is not part of the defect: [RFC 7401 Section 5.1.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.1.3) defines Header Length in 8-byte units excluding the first 8 bytes, so the floor division is exact once each parameter is a multiple of eight. Migration: a `SEQ` parameter's `length` is now 8 where it was 12, and the other 44 move likewise, so code comparing stored output byte for byte or asserting on `Data_*Parameter.length` sees the RFC's values. `LOCATOR_SET` is unchanged in both respects until [#679](https://github.com/JarryShaw/PyPCAPKit/issues/679) (below). `examples/generators/options.py`'s `HIP_COPIES` stayed at two for `R1_COUNTER`'s four-octet `counter` where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires eight, which this defect had masked, until [#672](https://github.com/JarryShaw/PyPCAPKit/issues/672) widened `counter` and [#679](https://github.com/JarryShaw/PyPCAPKit/issues/679) fixed `LOCATOR_SET`'s `Length` unit, after which [#689](https://github.com/JarryShaw/PyPCAPKit/issues/689) dropped `HIP_COPIES` to one ([#651](https://github.com/JarryShaw/PyPCAPKit/issues/651)). +- two HTTP/2 flag defects, one on each side of a round trip. **This changes the octets a reconstructed DATA frame emits, and the dumped value of a flagless frame's flags.** `_make_http_data` was the only one of the six `_make_http_*` methods that never read `frame.flags`, so a DATA frame parsed with `END_STREAM` set rebuilt with the bit clear (flags octet `0x01` to `0x00`): a well-formed frame saying the stream continues where the capture said it ended [[RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1)]. `PADDED` was re-derived from `pad_len`, and the five sibling builders already restored theirs, so this was a missing read-back ([#652](https://github.com/JarryShaw/PyPCAPKit/issues/652)). Separately, `FrameType.post_process` seeded its flag accumulator with a bare `0`, which `|=` promotes only as a side effect, so a frame with **no** bit set left `__flags__` a plain `int` where the schema and data model declare a `Flags`, and `END_STREAM in __value__` raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering `False`. That is the common case: a `0x00` flags octet is every opening `SETTINGS`, every non-final `DATA` and every non-ACK `PING`. It now seeds `self.Flags(0)`, as the construct path already did at all six of its sites. In the dump, `__value__` was a JSON *number* for a flagless frame and a *string* for every other frame in the same capture; it is consistently a string now, `"Flags::None [0]"` where it read `0`, the same change [#616](https://github.com/JarryShaw/PyPCAPKit/issues/616) made to TCP's `connection`. The seed is guarded because `FrameType.Flags` declares no members and, from Python 3.11, a memberless `enum.Flag` subclass refuses `Flags(0)`, so the five frame schemas that inherit it (`UnassignedFrame`, `PriorityFrame`, `RSTStreamFrame`, `GoawayFrame`, `WindowUpdateFrame`) keep the plain `int`, which nothing observes since all five pass `flags=None`. A DATA round trip is **still** lossy for an unrelated reason found while measuring: three payload length callbacks are mis-parenthesised, so an unpadded `DATA`, `HEADERS` or `PUSH_PROMISE` frame parses with its whole payload dropped ([#668](https://github.com/JarryShaw/PyPCAPKit/issues/668)). ([#652](https://github.com/JarryShaw/PyPCAPKit/issues/652), [#650](https://github.com/JarryShaw/PyPCAPKit/issues/650)). +- four MPTCP error messages carried a doubled separator, rendering as `TCP: : [OptNo 30] 1: invalid flags combination`. Cosmetic: the exception type, option number and subtype were right and nothing parses these strings, but an empty field reads as a value that failed to interpolate. Four sites (`_read_mptcp_join` and `_make_mptcp_join` for the invalid flag combination, both guards of `_make_mptcp_dss` for missing required fields), against 28 messages in the same file spelled correctly; now 32 correct against 0 doubled. Pre-existing since 2023-04-10. The new test asserts each message in full, since the existing `assertIn` on the body passes either side of the change ([#649](https://github.com/JarryShaw/PyPCAPKit/issues/649)). +- three names appeared in string annotations their own module never imported, so a type checker or documentation build could not resolve them while every test passed: `Any` in `pcapkit/utilities/logging.py`, and in `pcapkit/protocols/schema/internet/ipv6_route.py` both `Protocol` in a `payload:` stub and `Optional` inside a `typing.cast`. All three are now in their module's `TYPE_CHECKING` block, `Protocol` spelled `ProtocolBase as Protocol` as twenty-one sibling schema modules already do. The `cast` case is the one worth naming: `typing.cast` never evaluates its first argument, so no test that runs the line can see a bad name in it. Nothing resolves at runtime that did not before (`TYPE_CHECKING` is `False`); making these annotations resolve at runtime is a separate decision about the idiom. A new `tests/project/test_annotation_names.py` pins the invariant without a type checker: it resolves every string annotation in the package against the names its own module binds, following nested forward references such as `'list["Nested"]'` and treating `Literal` members and `Annotated` metadata as values, and reports exactly those three findings on the unfixed tree. The same change corrects the `MANIFEST.in` comment asserting that `include README.md` was the only thing putting the README in a source distribution and that an sdist without it could not be installed; both halves are false, and the [#619](https://github.com/JarryShaw/PyPCAPKit/pull/619) entry above carried the same claim and is corrected with it ([#642](https://github.com/JarryShaw/PyPCAPKit/issues/642)). +- every PCAP-NG packet block now carries the captured octets it declares. All four Enhanced Packet Blocks of the committed `examples/captures/dhcp.pcapng` reported `len(packet) == 0` against a `captured_len` of 314, 342, 314 and 342, as did the Simple Packet Block and obsolete Packet Block (synthesised; no committed fixture has either). Those three, exactly `PCAPNG.PACKET_TYPES`, are the whole blast radius. The octets were read correctly and thrown away: `PCAPNG.unpack` extracted them from the block schema, and `ProtocolBase.__init__` overwrote the result with `self.packet.payload`. The inherited `packet` splits a protocol at `self.length` octets of header and takes the rest as payload, which holds only for a header followed by its payload; `PCAPNG.length` is the *Block Total Length*, so the split consumed the whole block as header (`len(frame.packet.header) == 348` on a 348-octet block), and a block's trailer (option list, repeated total length) means no `length` could make it right. The fix is at the computation rather than the injection: `PCAPNG.packet` is overridden to take the payload from the block schema's `__payload__` field and the header from the octets ahead of it, summed out of the schema buffers so the three types' payload offsets (28, 12, 28) are not hard-coded. Teaching `ProtocolBase.__init__` to skip a protocol that had set `packet` was rejected: it heals `frame.info.packet` while `frame.packet` still reports the whole block as header. The override is a plain `property`, not the `cached_property` it replaces, which a cross-review on a second model settled: the inherited one caches because it *reads the stream*, where this one only walks buffers already filled, and caching it would let a second `unpack` on one instance return the *first* call's octets; the unit test swaps the block on a live instance and fails `b'payload' != b'cached'` under a cache. `ProtocolBase.packet` gains the contract in its docstring and no executable line. This moves the parse output of every PCAP-NG capture on the success path, so it ships labelled breaking: `frame.info.packet` goes from `b''` to hundreds of octets, `frame.packet.header` from the whole block to the pre-payload prefix. It was also a wire-format defect: `PCAPIO` writes `value.packet` after each 16-octet record header, so dumping those four blocks produced a **104-octet** file whose record headers declared hundreds of octets and delivered none, desynchronising any reader that walks by `incl_len` (pcapkit refused its own output with `ValueError: read length must be non-negative or -1`, and scapy returned 1 packet of 64 octets instead of 5). It round-trips now. `tests/protocols/test_pcapng_regression.py` grows 4 tests to 11 and 3 subtests to 24, with payload expectations derived twice (literals, and a `struct`-only re-parse independent of the code under test) and each synthesised payload a different length modulo 4, so an offset off by a field or a payload that picked up block padding cannot pass by coincidence; shapes no fixture exercised (an option area *after* the captured data, a big-endian section, snapped Enhanced and Simple Packet Blocks) are covered. `pcapkit/protocols/protocol.py` is identical in every coverage column, confirming its change is docstring-only. The uncaught `ValueError` on an EOF-truncated PCAP-NG ([#678](https://github.com/JarryShaw/PyPCAPKit/issues/678)) is untouched: over the 101 whole-percent prefixes `max(1, 1508 * i // 100)` both trees give 96 of that `ValueError`, one `FormatError`, one `ProtocolError` and three parses. `examples/captures/pcapng.txt`, a hand-regenerated legacy reference, still showed the old `packet -> NIL` here; [#685](https://github.com/JarryShaw/PyPCAPKit/issues/685) (below) later removed it from the index rather than regenerating it again ([#646](https://github.com/JarryShaw/PyPCAPKit/issues/646)). +- an unpadded HTTP/2 `DATA`, `HEADERS` or `PUSH_PROMISE` frame parsed with its whole payload silently discarded. Three payload length callbacks in `pcapkit/protocols/schema/application/httpv2.py` put the conditional in the wrong place: a conditional binds looser than `-`, so `pkt['__length__'] - pkt['pad_len'] if pkt['flags']['bit_3'] else 0` groups as `(pkt['__length__'] - pkt['pad_len']) if ... else 0`, and the `else` arm returned `0` ("read no octets") where it meant to subtract `0`. Padding is rare in HTTP/2, so the broken arm was the common one, and nothing raised or warned: `info.data` was `b''`. An unpadded `DATA` frame declaring `b'{"ok":true}\n'` now parses as those twelve octets, and unpadded `HEADERS` and `PUSH_PROMISE` keep their header block fragment. Padded frames are unaffected, since `(A - B) if T else 0` and `A - (B if T else 0)` are both `A - B` for truthy `T`; the padded arm has no off-by-one ([#668](https://github.com/JarryShaw/PyPCAPKit/issues/668) asked that be checked), because `Schema.unpack` decrements `packet['__length__']` by each field's width, so `pad_len`'s own octet is already out by the time the payload field runs. Three sibling sites that were never wrong (`ContinuationFrame.fragment`, `UnassignedFrame.data`, `GoawayFrame.debug`) localise the defect to the grouping and are pinned as controls. Nothing caught it: `test_option_roundtrip_unit` drives every frame type through `make` -> parse -> `make`, where both sides agreed on an empty payload (its generator passes no payload for these frames), and `test_httpv2_frame_readers_cover_successful_frames` uses hand-built schema stubs, so the callback never ran. The same lambda also governs **packing**, which neither the issue nor the first revision noticed and a cross-review on a second model did: `BytesField` consults `length` on both paths, so an unpadded `DATA` frame carrying twelve octets packed to a nine-octet header whose length field declares **21**, body absent (unpadded `HEADERS` and `PUSH_PROMISE` declared 29 against 9 and 33 against 13). pcapkit was *emitting* malformed HTTP/2 that desynchronises any reader walking a stream by that field; a `make` -> parse -> `make` cycle of a non-empty unpadded payload now closes byte-for-byte (`HTTPv2ConstructedFrameDeclaresWhatItWritesUnitTests`). The new `tests/protocols/application/test_httpv2_payload_length_unit.py` reads wire octets (23 tests, 12 subtests over padded, unpadded and padding-only shapes of all three frames plus both `PRIORITY` combinations) and asserts payload *octets*, never length, because a length assertion survives the right width read from the wrong offset; padded cases use non-zero padding `b'\xde\xad\xbe\xef'` rather than the zeros [RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1) tells a sender to use, so failing to subtract it yields a different byte string. Six wrong fixes written over the schema confirm the tests discriminate, each rejected; the cross-review's own, `max(computed, 1)` on the two `fragment` fields, **passed** the first revision's tests because only `DataFrame` had a padding-only case (a padding-only `HEADERS` frame then leaked one padding octet); the missing padding-only cases and `test_the_padded_arm_reaches_zero_and_is_not_clamped` close it, `0` being a legitimate answer. `EXPECTED_FAILURES` is unmoved (its one HTTP/2 entry `httpv2-frame/PRIORITY` keeps its `CONSTRUCT` status, `PriorityFrame` having no payload field or `PADDED` flag; its cited `httpv2.py:572` is stale, the `header.length != 9` guard now being at `:650`). The construct side needed no change: `_make_http_length` always computed `len(frame.data) + (pad_len + 1 if pad_len else 0)`, and the shared length callback disagreed on both paths. An AST sweep of all 496 package files found one other site of the same shape, `pcapkit/protocols/schema/internet/ipv4.py:336` (`TSOption.remainder`), and it is correct: its `else 0` is intentional because `ts_data` consumes the whole option data area in that arm. Ships labelled breaking: restoring a dropped payload moves the parse output of essentially every HTTP/2 capture, so a golden file or regression baseline that recorded the empty value sees it change, and constructed frames change byte-for-byte, from malformed to correct ([#668](https://github.com/JarryShaw/PyPCAPKit/issues/668)). +- the schema half of every option-like registrar overwrote silently while the parser half of the same call warned. `EnumSchema.register` was a bare `cls.__enum__[code] = schema`, and is the schema half of 14 public registrars, so one `register_ipv4_option` call replacing a built-in named the parser it displaced and said nothing about the schema. `EnumSchema.__init_subclass__` reaches the same registry without calling `register`, so a subclass declared with a `code=` keyword (the documented way to add a schema) stayed silent too; both are guarded, and the declaration hook's two branches are folded into one loop. Presence is a faithful test only because `_EnumRegistry.__missing__` returns a miss without recording it, so parsing an unknown code cannot make the next legitimate registration warn. The seven code-keyed parser registrars (on `ProtocolBase`, `Link`, `Internet`, `Frame`, `PCAPNG`, `Transport` and `SCTP`) now name the displaced entry and its replacement, where they reported only that something was overwritten. Their firing condition is harmonised onto the narrower guard [#681](https://github.com/JarryShaw/PyPCAPKit/pull/681) gave `register_protocol` ([#718](https://github.com/JarryShaw/PyPCAPKit/issues/718), [#726](https://github.com/JarryShaw/PyPCAPKit/pull/726)) rather than presence alone: that key is derived from the value it stores, whereas these take a caller-supplied `code`. Five of the seven (`Link` 7 entries, `Internet` 16, `Frame` 3, `PCAPNG` 3, `SCTP` 2) ship pre-seeded with unresolved `ModuleDescriptor` values, as do `TCP` (4) and `UDP` (3), each keeping its own dict rather than filling the one `ProtocolBase` and abstract `Transport` share and leave at 0 (`Transport.register` raises `UnsupportedCall`). Where a table is pre-seeded, the incumbent may be a two-string descriptor while the replacement is the class it names. Resolution happens earlier, on the incoming argument only (`if isinstance(protocol, ModuleDescriptor): protocol = protocol.klass`), so the guard's `is` comparison measures a resolved class against an unresolved descriptor and warns on re-registering the same class under its own pre-seeded code (`Internet.register(TransType.TCP, TCP)`), a false positive the harmonised guard does not close. `ContextRegistry.register` already raises on a duplicate, and the reassembly and ESP registrars append to lists with no key (duplicate security-association SPIs are designed, resolved by scoring), so none of the three takes a guard. `import pcapkit` holds at one warning, a third-party deprecation, and no `RegistryWarning`, over the 327 registry writes it performs (one being `R1CounterParameter`'s second code, from [#690](https://github.com/JarryShaw/PyPCAPKit/issues/690)), none landing on a key already present ([#692](https://github.com/JarryShaw/PyPCAPKit/issues/692)). +- every EOF-truncated PCAP-NG file raised a bare `ValueError` out of `Extractor`, losing the whole extraction rather than the one truncated block. Block Total Length is cross-checked against its trailing copy and never against the file, so a last block declaring more than the file holds left `PCAPNG.read` seeking *past* the end, and the next block read measured a **negative** remainder that `io.RawIOBase.read` refused. The seek is now clamped to the octets `_read_fileng` returned, with a `ProtocolWarning` naming the overrun, and a tail under the twelve octets a block needs is the quiet `StreamEOFError` that `Extractor.record_frames` already catches (deliberately not a clamp, which would fabricate a frame out of zero padding). A second route to the same shape is closed by a new `nonnegative()` helper, composed into `bounded_option` and `bounded_area` and applied to the eight unwrapped `length - N` spans and eight `__option_padding__` fields: `_TextField.__call__` builds `f'{length}s'` unconditionally, so a negative became `'-8s'` and `struct.calcsize` raised an exception caught nowhere a frame loop looks. Across all 1,509 octet boundaries of `examples/captures/dhcp.pcapng`, 6 levels parsed before and 1,495 do now; the 14 that still raise are twelve files too short to hold a block and two cuts into the Interface Description Block's `if_tsresol` option, so none costs a frame that was in the file. A mutation fuzz removes both of [#678](https://github.com/JarryShaw/PyPCAPKit/issues/678)'s families; two others remain, unchanged: `ValueError: N is not a valid BlockType` (filed as [#701](https://github.com/JarryShaw/PyPCAPKit/issues/701); an unassigned block type raises from `aenum` inside `EnumField.post_process`, so the registry's `UnknownBlock` default is unreachable) and `MemoryError` from [#593](https://github.com/JarryShaw/PyPCAPKit/pull/593)'s 32-bit band. Three bare exceptions in `SystemdJournalExportBlock.post_process`, reachable from valid input, are fixed with it: the body's NUL alignment padding was read as a field *name*, so every journal entry of unaligned length raised `struct.error` (a NUL-only line now ends the entry); a binary field's declared length was unbounded against its entry (`OverflowError` at `2**63` and above, silently wrong below; now clamped and reported); and a non-UTF-8 name, key or value raised `UnicodeDecodeError` (now `errors='replace'` and reported, as `StringField` does). The silent loss of every field after a binary one is filed as [#704](https://github.com/JarryShaw/PyPCAPKit/issues/704). The five unclamped non-packet option areas keep the per-block framing assumption [#676](https://github.com/JarryShaw/PyPCAPKit/pull/676)'s note describes. Also here, the fifth and last registrar with no overwrite guard: `Option.register` now reports a displaced option schema as a `RegistryWarning` naming every namespace the `ns='opt'` fan-out displaced something in, once per registration, tested by membership (subscripting the per-namespace `defaultdict` would insert `UnknownOption` for a code nobody registered). The condition is identity-based like the seven parser registrars and [#681](https://github.com/JarryShaw/PyPCAPKit/pull/681)'s guard for `register_protocol` ([#718](https://github.com/JarryShaw/PyPCAPKit/issues/718), [#726](https://github.com/JarryShaw/PyPCAPKit/pull/726)); a namespace the call itself creates is exempt. Flooring the three decryption-secrets payloads that read `__length__` *whole* was tried and reverted: `Schema.pack` leaves that key at `-1` for "unknown", so it packed nothing and turned `pcapng-secrets/TLS_Key_Log` and `.../WireGuard_Key_Log` from `MISMATCH` to `OK` (an empty payload equals an empty payload). Ships labelled breaking: well-formed captures are byte-identical, but any truncated PCAP-NG now yields the frames before the cut where it raised, so a caller reading "extraction raised" as "this file is unusable" gets a partial result whose last frame may carry zero-padded octets, and eight short-file depths move from `ProtocolError: unknown byteorder magic` to `StreamEOFError`. The [#683](https://github.com/JarryShaw/PyPCAPKit/pull/683) drift [#685](https://github.com/JarryShaw/PyPCAPKit/issues/685) tracks is unchanged ([#678](https://github.com/JarryShaw/PyPCAPKit/issues/678)). +- HIP's `R1_COUNTER`/`R1_Counter` parameter packed 12 octets where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires 16, and `LOCATOR_SET` declared its `Length` in 4-octet units where [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1)'s `Length` is a byte count -- two independent defects, fixed together because the evidence for either needed the other out of the way. `R1CounterParameter`'s `counter` was a `UInt32Field` where the RFC states the width twice, "8 bytes" in the diagram and "a 64-bit unsigned integer" in prose; both versions whose *packing* reaches the class, `R1_Counter` (128) and `R1_COUNTER` (129), packed a 12-octet record at `4 (mod 8)`. It is a `UInt64Field` now and the record is 16 octets ([#672](https://github.com/JarryShaw/PyPCAPKit/issues/672)). `LocatorSetParameter`'s `Length` was `sum(Locator.len)`, in the 4-octet units [RFC 8046 Section 4](https://datatracker.ietf.org/doc/html/rfc8046#section-4) gives `Locator Length`; a set of *n* plain IPv6 locators declared `4n` octets against `24n` present, and `Schema.unpack` handed the nested `ListField` only the declared octets, so `n = 2` and `n = 5` parsed one truncated locator and left the rest unconsumed. `HIP._make_param_locator_set` now sums `8 + Locator.len * 4` per locator ([#679](https://github.com/JarryShaw/PyPCAPKit/issues/679)). A second defect in the same parameter's padding, which [#651](https://github.com/JarryShaw/PyPCAPKit/issues/651) left alone because it cancelled this one, is fixed alongside: the nested `Locator` schemas share the parameter's packet context and pack their own `len` over it, so `padding`, evaluated after the list, saw 4 rather than the real length. A new `locator_set_len_callback` snapshots `Length` under a private key before any locator packs, and `locator_set_padding_len` reads that snapshot. The `counter` defaulted to `0` and the width defect could not be told apart from it, so the fixture now overrides both `R1_COUNTER` codes with `counter=0xaabbccdd`; new tests are `test_hip_r1_counter_width_unit.py` and `test_hip_locator_set_length_unit.py`. The `hip-parameter/R1_Counter` `EXPECTED_FAILURES` entry (code 128 registered no schema of its own, since `R1CounterParameter` declared only `code=129`) was deleted once that registry defect was fixed as [#690](https://github.com/JarryShaw/PyPCAPKit/issues/690), leaving 43 entries. `HIP_COPIES` dropped to one as [#689](https://github.com/JarryShaw/PyPCAPKit/issues/689), once [#672](https://github.com/JarryShaw/PyPCAPKit/issues/672) and [#679](https://github.com/JarryShaw/PyPCAPKit/issues/679) left it routing around nothing. +- `httpv1.HTTP` raised a bare exception for any payload that is not an HTTP/1 message, at four sites: the header/body split in `read`, the header's CRLF split and the start-line split in `_read_http_header`, and `item[1]` on a field line with no colon. `HTTP._guess_version` falls through on `ProtocolError` alone, so the HTTP/1 attempt aborted the guess and the HTTP/2 arm below was dead code for all four input classes. All four now raise `ProtocolError`, which each method's `Raises:` section documented, and a test pins the invariant (nothing escapes `httpv1.HTTP` that is not a `ProtocolError`), so a fifth bare-raising site would fail there. An obs-fold continuation line [[RFC 9112 Section 5.2](https://datatracker.ietf.org/doc/html/rfc9112#section-5.2)] is legal HTTP/1 and is now unfolded, the RFC's remedy, rather than read as a field of its own (with no colon it raised that `IndexError`; with one it parsed into a spurious extra field). Reaching the HTTP/2 arm exposed a payload under nine octets to `httpv2.HTTP`, where it usually fails inside the schema with a bare `struct.error` (not a `ValueError`, `ProtocolError` or pcapkit `StructError`, so uncatchable); `read` converts it and `_guess_version` suppresses it on the last arm only, since an arm that is not last hands the payload onward when it swallows. Separately, `PayloadField.protocol` looked a `str` up in `__proto__` as given while the registry is keyed on the upper-cased class name, so a name not already upper-case resolved to `None` and yielded `Raw`; it now folds case and warns with `RegistryWarning` for a genuinely unregistered name, and `__init__` assigns through that property instead of writing `_protocol` past it ([#787](https://github.com/JarryShaw/PyPCAPKit/issues/787)). +- a systemd journal entry whose last field lacks the format's mandatory trailing newline let the PCAP-NG block's 32-bit alignment padding be read as part of that field's value, with no exception or warning: `b'MESSAGE=hello'` plus three NULs returned `MESSAGE == 'hello\x00\x00\x00'`. The padding-only-line guard from [#678](https://github.com/JarryShaw/PyPCAPKit/issues/678) catches padding only on a line of its own, which needs the preceding field to have ended with a real newline; without one `readline()` runs through the value into the padding. Reachable on text fields only (a binary field's length-prefixed read is never in play, and a binary field *name* on such a line fails its own length-prefix read regardless). The fix is tolerant: block padding is 0-3 octets, so at most the trailing three octets of a terminator-less line are stripped, with a `SchemaWarning`, and anything past that is left as data. The gate reads the *raw* line, since a real trailing newline, or any other raw last octet including ASCII whitespace, proves the padding is zero ([#794](https://github.com/JarryShaw/PyPCAPKit/issues/794)). `examples/generators/pcapng.py`'s `_journal_entry` cited a bare line number for the defect its alignment workaround avoids and mis-attributed the fix; it now names [#678](https://github.com/JarryShaw/PyPCAPKit/issues/678)'s `not line.strip(b'\x00')` guard and the mechanism ([#791](https://github.com/JarryShaw/PyPCAPKit/issues/791)). +- `httpv2.HTTP.read` tested only the 24-bit declared length off the wire (`schema.length < 9`), never how many octets the buffer held, so a frame backed by far fewer real octets than declared parsed and reported the declared, attacker-controlled length. Sweeping the declared length against a four-octet buffer gave non-monotone accept/reject (9, 15, 65535 and 16777215 parsed, 10 and 100 did not), an artifact of `_read_http_settings`'s own `(declared - 9) % 6` check. `HTTP.unpack` now rejects a non-empty buffer under nine octets before the schema layer runs, preserving `StreamEOFError` for a genuinely exhausted implicit-length stream, and `read`'s guard also requires the declared length not to exceed the buffer. A buffer that clears nine octets can still carry a frame type whose fixed fields exceed what is left (GOAWAY at 9-16 octets, PUSH_PROMISE at 9-12, padded DATA, HEADERS and PUSH_PROMISE) and crash the schema layer with a bare `struct.error`; that root cause is in the generic `Schema.unpack`/ `FieldBase.length` machinery shared by ten schema modules, out of scope and filed as [#805](https://github.com/JarryShaw/PyPCAPKit/issues/805), which is why `_guess_version`'s last arm keeps suppressing `struct.error` alongside `ProtocolError` ([#802](https://github.com/JarryShaw/PyPCAPKit/pull/802)). +- `SystemdJournalExportBlock.post_process` skipped a binary field's one trailing newline with a bare `entry_data.read()`, which reads to **EOF**: the next `readline()` returned `b''` and ended the entry, discarding every field behind the first binary one. The skip is bound to `entry_data.read(1)`; a missing or non-`b'\n'` octet ends the entry with a `SchemaWarning` instead of resynchronising at the wrong offset. `test_journal_fields_following_a_binary_field_are_not_discarded` fails against the unfixed code with `MissingKeyError: 'SECOND'` ([#704](https://github.com/JarryShaw/PyPCAPKit/issues/704)). +- the same block's entries were split on a bare `b'\n\n'`, which shreds a binary field whose value contains that pair. Walking the entry instead of splitting it introduced a regression on first review: a `malformed` flag broke the *outer* per-block loop on a bad terminator, ending the whole block and dropping every well-formed entry behind it. Fixed by dropping `malformed`, leaving the per-entry `break` as the only effect. A second gap closed in the same change: a trailing separator on the block's last octet was indistinguishable from EOF and swallowed, so a rebuild wrote `length` one octet short ([#723](https://github.com/JarryShaw/PyPCAPKit/issues/723)). +- **a breaking change to** four `.get()`-backed enum fields: TCP/UDP/SCTP's `PortEnumField` and PCAP-NG's `OptionEnumField` minted a member through `aenum.extend_enum` for every value no row or documented span covers (111 calls on `http.pcap`, all ephemeral ports). All four now peek at what `.get()` would consult and mint only once the value is unresolvable; a genuine miss gets `EnumField._unregistered_member`, a real member of the same registry built through its storage base's `__new__`, so `isinstance` holds and the lookup table stops growing. `extend_enum` calls drop 111 to 0 on `http.pcap` and 15 to 2 on `many_interfaces.pcapng` (both from the untouched `_missing_`). Riding along: PCAP-NG option namespaces stop cross-contaminating: once a code was minted under `opt`, `OptionType.get` merged `opt` over the requested namespace and 6 of the other 7 namespaces answered `opt_unknown` for it; each now gets its own `_unknown`. **Breaking** because [#575](https://github.com/JarryShaw/PyPCAPKit/issues/575) asked for byte-identical output across the sample captures, as [#420](https://github.com/JarryShaw/PyPCAPKit/pull/420) and [#427](https://github.com/JarryShaw/PyPCAPKit/pull/427) did, and this deliberately misses that bar: 14 of 23 captures change, entirely from the new `` rendering plus the PLIST escaping above; normalising both leaves all 14 byte-identical. A `pickle` round-trip of a resolved unassigned member worked before (it was minted, so registered) and would have broken silently after; `_unregistered_member` now installs a `__reduce_ex__` rebuilding an equivalent unregistered member, checked on protocols 0-5 under both picklers and `copy`/`deepcopy` ([#575](https://github.com/JarryShaw/PyPCAPKit/issues/575)). +- `httpv2._guess_version` tried to parse a stream as HTTP/2 and read failure as "not HTTP/2", rather than identifying the version first, so a genuine connection preface followed by a real `SETTINGS` frame raised `ProtocolError: unknown HTTP version`. It now identifies before it parses: a preface is recognised as HTTP/2 outright, skipped rather than fed to `httpv2`, and counted as header so `info` stays byte-identical to reading the following frame alone (otherwise the injected `packet=self.packet.payload`, sliced from octet 9 of a buffer whose frame starts at 24, reported preface remnants as payload). A preface with no frame behind it raises `ProtocolError: HTTP/2: connection preface with no frame`; a bare mid-stream `SETTINGS` frame, an `Upgrade: h2c` request and non-HTTP input are unchanged, the first two left undecidable on purpose (`Upgrade: h2c` needs per-connection state a single payload cannot carry, and a mid-stream frame has no safe heuristic, since a `type <= 9` guess misfires on binary HTTP/1 bodies). Protochain over all 23 sample captures (1,604 frames, 231 HTTP-bearing) is byte-identical to the unfixed tree ([#800](https://github.com/JarryShaw/PyPCAPKit/issues/800)). +- `TCP.__proto__` bound `httpv1.HTTP` directly for ports 80 and 8080, so a segment on either port was HTTP/1 by assertion of the port alone; both versions share those ports, so the payload has to decide. Repointed to the generic `pcapkit.protocols.application.http.HTTP` proxy, whose identification became positive only once [#800](https://github.com/JarryShaw/PyPCAPKit/issues/800) and [#814](https://github.com/JarryShaw/PyPCAPKit/pull/814) landed, which is why this waited. `udp.py` already bound the proxy for both ports; this removes the asymmetry its docstring and `docs/source/pep.rst` documented as an open request (prose-only on that side). Protochain over all 23 captures (1,604 frames) is *not* byte-identical, and that is the fix: all 231 HTTP/1.1 frames keep their chain, and nine frames in `options-transport.pcap` change `Ethernet:IPv4:TCP:Raw` to `Ethernet:IPv4:TCP:HTTP/2` -- genuine HTTP/2 frames this library's own `httpv2.HTTP.make` built, previously refused by the HTTP/1 parser. `_guess_version`'s entry count over the corpus goes 0 to 252 (231 HTTP/1.1, 9 HTTP/2, 12 fall-throughs that stay `Raw`). Not labelled breaking, but not free: TCP:80/8080 traffic that is neither valid HTTP/1 nor preface-carrying is now exposed to the fall-through arm, where the direct `httpv1` binding left it `Raw` regardless; a caller depending on that is the one who would notice ([#682](https://github.com/JarryShaw/PyPCAPKit/issues/682)). +- **a breaking change to** 8 more bespoke registries: step 2 of [#860](https://github.com/JarryShaw/PyPCAPKit/issues/860), PR 1 of 2 (`AppType` is PR 2, [#874](https://github.com/JarryShaw/PyPCAPKit/pull/874) below), bringing `StatusCode`, `ReturnCode`, `ResponseKind`, `GroupingInformation`, `OptionType`, `FEATCode`, `Command` and `Method` onto `EnumRegistry` and applying [#775](https://github.com/JarryShaw/PyPCAPKit/issues/775)'s mint/unmint ruling to their `get()` as well as `_missing_`. The first five convert 12 unambiguous placeholder branches (`Unassigned`, `Unknown`, `opt_unknown`) to `_unregistered_member`; the three with a custom `__new__` (`StatusCode`, `ReturnCode`, `OptionType`) get an override reconstructing the attributes the base's generic helper would leave unset, and `StatusCode`/`ReturnCode`'s hand-written `get()`, still on the retired `default == -1` convention, is replaced by the base's (no caller relied on the old form). `OptionType` keeps its own `get()` for its multi-namespace dispatch, but a round-2 review found it still minted on both its int/namespace and `str` paths, and the live pcapng parse path (`PCAPNG._make_pcapng_options`) calls it with wire bytes, so parsing an undeclared option code still registered a permanent member; both paths now build an unregistered member. `FEATCode`, `Command` and `Method` mint the literal wire value as its own name rather than any placeholder; per the owner's ruling (*"get will not have sufficient information to create new ones"*: `Command` needs `feat`/`desc`/`type`/`conf` and `Method` needs `safe`/`idempotent`, which a bare wire string lacks), both `_missing_` and each class's own `get()` (a second, independent mint site) now build an unregistered member. `FEATCode`'s crawler now declares all 15 real FEAT-code values from the live IANA table instead of minting 10 of them as a side effect of evaluating `Command`'s rows at import time, the import-time-mutation shape [#861](https://github.com/JarryShaw/PyPCAPKit/pull/861) removed from `FilterType`; pinned count-agnostically so a future IANA update cannot fail a correct regeneration. Untouched, per rulings: `CommandType` stays `IntFlag` (real `A|P` composites in the generated data), and `TransportProtocol`'s `auto()` renumbering and `AppType` are PR 2's. `pcapkit/protocols/schema/misc/pcapng.py`'s `OptionEnumField.post_process` docstring is corrected: it still bypasses `OptionType.get()` directly, now for `pickle` round-trip safety ([#860](https://github.com/JarryShaw/PyPCAPKit/issues/860) already stops the minting that used to be the reason), since neither of `OptionType`'s two paths gets both pickling and correct rendering right at once ([#869](https://github.com/JarryShaw/PyPCAPKit/pull/869)). ### pcapkit.toolkit #### Added -- `util/pyshark_encap_map.py`, a generator that regenerates `ENCAP_TYPE_TO_LINKTYPE` (152 entries) and `FILTER_NAME_TO_LINKTYPE` (58) in place inside `pcapkit/toolkit/pyshark.py`, so the two tables [#850](https://github.com/JarryShaw/PyPCAPKit/pull/850) hand-built stop being hand-maintained. It lives under `util/` rather than `pcapkit/vendor/` because `Vendor`'s base class is hardwired to an HTTP fetch and a generated enum-class file, where this generator's source of truth is the local `tshark`/`editcap` binaries. Guards against its own worst failure mode: a naive `LinkType(dlt)` lookup cannot detect an unmapped DLT, because `_missing_` mints a placeholder rather than raising, so the generator snapshots every known `LinkType` value before any lookup and only calls `LinkType(dlt)` once a value has already been confirmed a member. The real-`tshark` sweep is gated on a newly-declared `HAS_WIRESHARK` flag, registered in `tests/_dependency_gates.py`'s `NON_DISTRIBUTION_FLAGS` next to `HAS_PROC_FD`, for things pip cannot install at all. The sweep arithmetic is version-pinned -- `editcap -T` accepts 226 encapsulations on Wireshark 4.6.9, 224 on 4.2.2 (CI's Ubuntu noble) -- so the count assertions run only under exactly the measured version, with a separate `VersionIndependentInvariantTests` class carrying what holds regardless. One line of `pcapkit/toolkit/pyshark.py` itself changes as a result: the `IPMB_LINUX` table entry becomes `I2C_LINUX`, the canonical name [#848](https://github.com/JarryShaw/PyPCAPKit/pull/848) gave value 209 -- the same member either way ([#853](https://github.com/JarryShaw/PyPCAPKit/pull/853)). +- `util/pyshark_encap_map.py`, a generator that regenerates `ENCAP_TYPE_TO_LINKTYPE` (152 entries) and `FILTER_NAME_TO_LINKTYPE` (58) in place inside `pcapkit/toolkit/pyshark.py`, so the two tables [#850](https://github.com/JarryShaw/PyPCAPKit/pull/850) hand-built stop being hand-maintained. It lives under `util/` rather than `pcapkit/vendor/` because `Vendor`'s base class is hardwired to an HTTP fetch and a generated enum-class file, where this generator's source of truth is the local `tshark`/`editcap` binaries. It guards against its worst failure mode: a naive `LinkType(dlt)` cannot detect an unmapped DLT, because `_missing_` mints a placeholder rather than raising, so the generator snapshots every known `LinkType` value first and calls `LinkType(dlt)` only once a value is confirmed a member. The real-`tshark` sweep is gated on a new `HAS_WIRESHARK` flag, registered in `tests/_dependency_gates.py`'s `NON_DISTRIBUTION_FLAGS` next to `HAS_PROC_FD`, for things pip cannot install. The sweep arithmetic is version-pinned (`editcap -T` accepts 226 encapsulations on Wireshark 4.6.9, 224 on 4.2.2, CI's Ubuntu noble), so count assertions run only under exactly the measured version, with a separate `VersionIndependentInvariantTests` class for what holds regardless. One line of `pcapkit/toolkit/pyshark.py` changes: the `IPMB_LINUX` entry becomes `I2C_LINUX`, the canonical name [#848](https://github.com/JarryShaw/PyPCAPKit/pull/848) gave value 209 -- the same member either way ([#853](https://github.com/JarryShaw/PyPCAPKit/pull/853)). #### Fixed -- the engine adapters, which were quietly wrong rather than loud. The `dpkt` toolkit split TCP and IPv4 headers at their fixed struct size instead of their real length, so option octets overwrote payload in the sequence-indexed reassembly buffer; it also read an `ipv6_frag.nh` that `dpkt` does not have, and passed fragment offsets unscaled ([#351](https://github.com/JarryShaw/PyPCAPKit/issues/351), [#370](https://github.com/JarryShaw/PyPCAPKit/issues/370), [#385](https://github.com/JarryShaw/PyPCAPKit/pull/385), [#395](https://github.com/JarryShaw/PyPCAPKit/pull/395)). `scapy` never loaded its layer registry, so that engine did not dissect at all ([#409](https://github.com/JarryShaw/PyPCAPKit/pull/409)), and its IPv4 fragment offset reached reassembly unscaled ([#483](https://github.com/JarryShaw/PyPCAPKit/issues/483), [#484](https://github.com/JarryShaw/PyPCAPKit/pull/484)). The four IPv6 adapters disagreed about whether the 8-octet Fragment header belongs to `ihl`, `header` and `tl`; per [RFC 8200](https://datatracker.ietf.org/doc/html/rfc8200) section 4.5 it belongs to none of them, and all four now agree ([#415](https://github.com/JarryShaw/PyPCAPKit/issues/415), [#424](https://github.com/JarryShaw/PyPCAPKit/pull/424)). -- `pypcapfile`'s `IP.src`/`.dst` are dotted-decimal ASCII text in a `ctypes.c_char_p` (confirmed against `pypcapfile` 0.12.0's own `ip.py`), not packed 4-byte values, so `ipaddress.IPv4Address(ipv4.src)` raised `AddressValueError` and `ipv4_header()`'s `struct.pack('...II', ipv4.src, ...)` raised `struct.error` on every call. A single `_parse_ipv4_address()` chokepoint replaces the four call sites, casting with `int(...)` where `struct.pack` needs the packed form and `str(...)` elsewhere, and also accepts a packed `bytes` or an `int` directly. The same engine's `IP.opt`/`.payload` carry hex-ASCII rather than raw bytes once decoding stops short of the transport layer; a new `_maybe_unhex()` covers the three sites that read them. Not `breaking`: nothing on this path worked on 3.10/3.11 before, raising on every call, so correct behaviour cannot regress a working caller ([#743](https://github.com/JarryShaw/PyPCAPKit/issues/743)). -- `pypcapfile`'s `_read_a_packet(layers=0)` hexlifies a whole frame into ASCII text and stops there, but `_decode()` fed that straight into `Ethernet(packet.packet, layers=1)`, whose `__init__` unpacks the header with `struct.unpack` -- every field came out garbage without raising. Fixed with `binascii.unhexlify(packet.packet)` ahead of the decoder call, in preference to decoding eagerly at `layers>0` inside the lazy generator, which would lose `_decode()`'s per-frame `AttributeWarning` fallback. Link-layer dispatch is unaffected, resolving from the header's `ll_type` rather than from frame bytes. Landed together with [#743](https://github.com/JarryShaw/PyPCAPKit/issues/743): standalone, this change turns a non-crashing extraction into a crashing one -- `Extractor.run()` catches only `EOFError`, `StopIteration` and `KeyboardInterrupt`, so the `AddressValueError` and `struct.error` [#743](https://github.com/JarryShaw/PyPCAPKit/issues/743) fixes on the same path escape to the caller until both land ([#746](https://github.com/JarryShaw/PyPCAPKit/issues/746)). -- **a breaking change to** pyshark link-type resolution in `tcp_traceflow`: an unrecognised encapsulation or filter name now raises `MissingKeyError` instead of substituting a plausible DLT. `pcapkit/toolkit/pyshark.py` resolved a frame's link type from `packet.layers[0].layer_name`, a Wireshark display-filter name, with a blanket `Enum_LinkType.get(name.upper())` fallback for anything not in its small lookup table. One filter name serves several encapsulations, so that fallback returned *wrong DLTs silently* -- `null` for a `DLT_LOOP` capture, `101` (`RAW`) for `rawip6`. Re-keyed onto `frame.encap_type`, which is distinct per encapsulation, and the fallback is gone. The two lookup tables were measured rather than transcribed -- Wireshark's source is not on the build host -- each entry round-tripped through `editcap -F pcap -T ` and read back with `tshark`: `ENCAP_TYPE_TO_LINKTYPE` grows to 152 entries with no DLT serving two keys, and `FILTER_NAME_TO_LINKTYPE` grows from the two entries [#838](https://github.com/JarryShaw/PyPCAPKit/pull/838) added to 58, every root filter name measured single-DLT across the 157 (of the 226 `editcap -T` accepts) encapsulations this host's Wireshark can write. `pcapkit/toolkit/pyshark.py` reaches 100% coverage under the new tests, which fail 26 ways against the unfixed tree; live pyshark parsing itself stays unexercised, since `pyshark.FileCapture` does not run at all on Python 3.14 ([#850](https://github.com/JarryShaw/PyPCAPKit/pull/850)). +- the engine adapters, which were quietly wrong rather than loud. The `dpkt` toolkit split TCP and IPv4 headers at their fixed struct size instead of their real length, so option octets overwrote payload in the sequence-indexed reassembly buffer; it also read an `ipv6_frag.nh` that `dpkt` does not have, and passed fragment offsets unscaled ([#351](https://github.com/JarryShaw/PyPCAPKit/issues/351), [#370](https://github.com/JarryShaw/PyPCAPKit/issues/370), [#385](https://github.com/JarryShaw/PyPCAPKit/pull/385), [#395](https://github.com/JarryShaw/PyPCAPKit/pull/395)). `scapy` never loaded its layer registry, so that engine did not dissect at all ([#409](https://github.com/JarryShaw/PyPCAPKit/pull/409)), and its IPv4 fragment offset reached reassembly unscaled ([#483](https://github.com/JarryShaw/PyPCAPKit/issues/483), [#484](https://github.com/JarryShaw/PyPCAPKit/pull/484)). The four IPv6 adapters disagreed about whether the 8-octet Fragment header belongs to `ihl`, `header` and `tl`; per [RFC 8200](https://datatracker.ietf.org/doc/html/rfc8200) section 4.5 it belongs to none, and all four now agree ([#415](https://github.com/JarryShaw/PyPCAPKit/issues/415), [#424](https://github.com/JarryShaw/PyPCAPKit/pull/424)). +- `pypcapfile`'s `IP.src`/`.dst` are dotted-decimal ASCII text in a `ctypes.c_char_p` (per `pypcapfile` 0.12.0's `ip.py`), not packed 4-byte values, so `ipaddress.IPv4Address(ipv4.src)` raised `AddressValueError` and `ipv4_header()`'s `struct.pack('...II', ipv4.src, ...)` raised `struct.error` on every call. A single `_parse_ipv4_address()` chokepoint replaces the four call sites, also accepting packed `bytes` or an `int`. The same engine's `IP.opt`/`.payload` carry hex-ASCII rather than raw bytes once decoding stops short of the transport layer; a new `_maybe_unhex()` covers the three sites that read them. Not `breaking`: nothing on this path worked on 3.10/3.11, so correct behaviour cannot regress a working caller ([#743](https://github.com/JarryShaw/PyPCAPKit/issues/743)). +- `pypcapfile`'s `_read_a_packet(layers=0)` hexlifies a whole frame into ASCII text and stops, but `_decode()` fed that into `Ethernet(packet.packet, layers=1)`, whose `__init__` unpacks the header with `struct.unpack`, so every field came out garbage without raising. Fixed with `binascii.unhexlify(packet.packet)` ahead of the decoder call, in preference to decoding eagerly at `layers>0` inside the lazy generator, which would lose `_decode()`'s per-frame `AttributeWarning` fallback. Link-layer dispatch resolves from the header's `ll_type` and is unaffected. Landed with [#743](https://github.com/JarryShaw/PyPCAPKit/issues/743): standalone, this turns a non-crashing extraction into a crashing one, since `Extractor.run()` catches only `EOFError`, `StopIteration` and `KeyboardInterrupt` and the `AddressValueError`/`struct.error` [#743](https://github.com/JarryShaw/PyPCAPKit/issues/743) fixes escape until both land ([#746](https://github.com/JarryShaw/PyPCAPKit/issues/746)). +- **a breaking change to** pyshark link-type resolution in `tcp_traceflow`: an unrecognised encapsulation or filter name now raises `MissingKeyError` instead of substituting a plausible DLT. `pcapkit/toolkit/pyshark.py` resolved a frame's link type from `packet.layers[0].layer_name`, a Wireshark display-filter name, with a blanket `Enum_LinkType.get(name.upper())` fallback. One filter name serves several encapsulations, so that fallback returned *wrong DLTs silently* (`null` for a `DLT_LOOP` capture, `101` (`RAW`) for `rawip6`). Re-keyed onto `frame.encap_type`, which is distinct per encapsulation, and the fallback is gone. The two tables were measured rather than transcribed (Wireshark's source is not on the build host): each entry round-tripped through `editcap -F pcap -T ` and read back with `tshark`. `ENCAP_TYPE_TO_LINKTYPE` grows to 152 entries with no DLT serving two keys, and `FILTER_NAME_TO_LINKTYPE` from [#838](https://github.com/JarryShaw/PyPCAPKit/pull/838)'s two entries to 58, every root filter name single-DLT across the 157 (of 226) encapsulations this host's Wireshark can write. `pcapkit/toolkit/pyshark.py` reaches 100% coverage under new tests that fail 26 ways against the unfixed tree; live pyshark parsing stays unexercised, since `pyshark.FileCapture` does not run on Python 3.14 ([#850](https://github.com/JarryShaw/PyPCAPKit/pull/850)). ### pcapkit.utilities @@ -190,57 +190,57 @@ This is the resolution of [#548](https://github.com/JarryShaw/PyPCAPKit/issues/5 #### Changed -- `pcapkit` no longer configures logging at import. It installs a `NullHandler` and sets no level, so verbosity is inherited from the application instead of being seized by whichever library was imported second; the old stderr handler stays as the `PCAPKIT_DEVMODE` opt-in. Three consequences worth knowing: the previous behaviour is `configure(logging.INFO, stream=sys.stderr)`; 38 registry and extractor `info` calls became `debug`, so those messages are invisible even at `INFO`; and the handler is no longer `logger.handlers[0]`. `verbose=` output stays on stdout and is not logging ([#384](https://github.com/JarryShaw/PyPCAPKit/pull/384)). -- each warning is reported once per channel, and `pcapkit` no longer inserts a `simplefilter('ignore', ...)` at the front of the process-global `warnings.filters` ([#362](https://github.com/JarryShaw/PyPCAPKit/issues/362)--[#364](https://github.com/JarryShaw/PyPCAPKit/issues/364), [#390](https://github.com/JarryShaw/PyPCAPKit/pull/390)). The application's own filter therefore wins now, which is the point of the change and also the sharp edge in it: under `-W error`, or pytest's `filterwarnings = error`, a pcapkit warning that used to be suppressed will raise. Suppress them deliberately with `warnings.filterwarnings('ignore', category=BaseWarning)`. `quiet=True` now means no record at any level and no longer sets `sys.tracebacklimit`, and the `pcapkit.utilities.warnings.DEVMODE` re-export is gone -- its canonical home is `pcapkit.utilities.logging`. +- `pcapkit` no longer configures logging at import. It installs a `NullHandler` and sets no level, so verbosity is inherited from the application instead of being seized by whichever library was imported second; the old stderr handler stays as the `PCAPKIT_DEVMODE` opt-in. Three consequences: the previous behaviour is `configure(logging.INFO, stream=sys.stderr)`; 38 registry and extractor `info` calls became `debug`, so those messages are invisible even at `INFO`; and the handler is no longer `logger.handlers[0]`. `verbose=` output stays on stdout and is not logging ([#384](https://github.com/JarryShaw/PyPCAPKit/pull/384)). +- each warning is reported once per channel, and `pcapkit` no longer inserts a `simplefilter('ignore', ...)` at the front of the process-global `warnings.filters` ([#362](https://github.com/JarryShaw/PyPCAPKit/issues/362)--[#364](https://github.com/JarryShaw/PyPCAPKit/issues/364), [#390](https://github.com/JarryShaw/PyPCAPKit/pull/390)). The application's own filter therefore wins, which is the point and also the sharp edge: under `-W error`, or pytest's `filterwarnings = error`, a pcapkit warning that used to be suppressed will raise. Suppress them deliberately with `warnings.filterwarnings('ignore', category=BaseWarning)`. `quiet=True` now means no record at any level and no longer sets `sys.tracebacklimit`, and the `pcapkit.utilities.warnings.DEVMODE` re-export is gone; its home is `pcapkit.utilities.logging`. #### Fixed -- two more gaps the [#514](https://github.com/JarryShaw/PyPCAPKit/issues/514) keyword audit turned up, neither previously covered by a test: `StreamEOFError`'s docstring did not say that `@prepare` always raises it with `quiet=True` -- the same end-of-stream convention `StructError` follows via its own `eof=True` -- so nothing pinned that silence against a future regression; and `register_extractor_engine`'s real keyword, `name`, was not itself under test, only its already-corrected docstring, so a future rename could put the two out of step again exactly as quietly as before ([#577](https://github.com/JarryShaw/PyPCAPKit/pull/577)). +- two more gaps the [#514](https://github.com/JarryShaw/PyPCAPKit/issues/514) keyword audit turned up, neither covered by a test: `StreamEOFError`'s docstring did not say that `@prepare` always raises it with `quiet=True` (the end-of-stream convention `StructError` follows via its own `eof=True`), so nothing pinned that silence against regression; and `register_extractor_engine`'s real keyword, `name`, was not itself under test, only its corrected docstring, so a rename could put the two out of step again ([#577](https://github.com/JarryShaw/PyPCAPKit/pull/577)). ### pcapkit.vendor #### Fixed -- thirteen `re.sub` sites under `pcapkit/vendor/` passed `re.MULTILINE` as the fourth *positional* argument, which is `count`, not `flags` -- capping substitution at 8 matches and raising `DeprecationWarning` on Python 3.13+, a future `TypeError`. Found by an AST sweep rather than by grep, since 8 of the 13 put the flag on a continuation line. Fixed by moving the flag to `flags=` at all 13 sites (`default.py` plus one site each in `hip/eddsa_curve.py`, `http/frame.py`, `http/method.py`, `http/status_code.py`, `ipv4/router_alert.py`, `ipv6/option.py`, `ipv6/router_alert.py`, `ipv6/tagger_id.py`, `reg/ethertype.py`, `tcp/flags.py`, `tcp/mp_tcp_option.py` and `tcp/option.py`). Behaviour preserving: none of the 13 sites' shared pattern (`r'\r*\n'`) carries a MULTILINE-sensitive anchor, so no `const/` regeneration was needed ([#796](https://github.com/JarryShaw/PyPCAPKit/issues/796)). -- `main` was red: 12 tests across three files, all `RecursionError` (`tests/const/test_const_enum_lookup.py`; two `test_const_enum_no_mint.py` methods, two registries each; 7 of the 12 in `tests/protocols/misc/test_pcapng_unit.py`). `pcapkit/vendor/pcapng/record_type.py` and `secrets_type.py` rendered a two-line `_missing_` body whose first line did not return -- `cls._unregistered_member(value, 'Unassigned')` followed unconditionally by `return cls(value)`. Harmless while that first line was `extend_enum(...)`, which registers the member so the following `cls(value)` found it; [#861](https://github.com/JarryShaw/PyPCAPKit/pull/861) (above) replaced it with `_unregistered_member`, which deliberately does not register, so `cls(value)` missed again and re-entered `_missing_`. [#861](https://github.com/JarryShaw/PyPCAPKit/pull/861) only half-corrected the two files: it fixed the *generated* modules and never touched the two *crawlers*, and even in the generated files left the old `return cls(value)` in place as unreachable dead code after the new return -- which is why [#861](https://github.com/JarryShaw/PyPCAPKit/pull/861)'s own CI stayed green until a live regeneration from the stale crawlers reintroduced the defect for real; [#861](https://github.com/JarryShaw/PyPCAPKit/pull/861)'s cross-review had verified byte-identical regeneration on a 5-of-82 sample that did not happen to include these two. Both crawlers now collapse to the single returning line the other 101 registries use, and the two const files are regenerated from them -- `record_type.py` fetches the network for its own `LINK` constant, so that regeneration is not reproducible without connectivity, though it came back byte-identical; `secrets_type.py` is offline. The new guard sits at the crawler-emission layer rather than the behavioural one `test_const_enum_lookup.py` already swept and that is what caught this: a hard-coded `_missing_` body must end in a return, and no emitted line may re-enter `_missing_` for the same value -- deliberately narrower than "every emitted line must return", since 53 crawlers legitimately emit 109 non-returning lines in the shape `Vendor.process` itself uses. The guard sees only 35 of 96 crawlers; the other 61 (53 `append`-built, 8 returning no `(enum, miss)` pair) stay invisible to it and are covered behaviourally instead. Verified against the four reverted source files: 4 of 6 new tests fail as methods; the other two pass either way, by design, since [#866](https://github.com/JarryShaw/PyPCAPKit/issues/866)'s own body still ended on a return. Closes [#866](https://github.com/JarryShaw/PyPCAPKit/issues/866) ([#867](https://github.com/JarryShaw/PyPCAPKit/pull/867)). -- `python -m pcapkit.vendor ` could not fail: `run()` swallowed every crawler exception as a filterable `VendorRuntimeWarning`, and `main()` always returned `0`, so a broken crawler regenerating nothing looked like success both to a human and to the cron job. Per the owner's ruling (*"only discard changes made by a non-zero sub-vendor... for cron jobs, at most warn about failed sub-vendors, but never fail the entire job"*), this lands in three pieces. **Exit code** (`pcapkit/vendor/__main__.py`): `run()` now returns whether its target succeeded; `main()` attempts every target regardless of earlier failures and returns `1` if any raised, with the failing target's qualified name and exception `repr` written to stderr unconditionally via `print()` rather than through the filterable warning. **Per-target snapshot and restore**: a new `_snapshot_and_restore` context manager backs up a target's const file with `tempfile.mkstemp` before it runs -- resolving the destination through `Vendor._dest_path`, now a `classmethod` (`pcapkit/vendor/default.py`) so it can be called before `Vendor.__init__` ever fetches, renders or writes -- and on any exception restores the backup via `os.replace` before re-raising, or discards it quietly on success. A target whose destination cannot be resolved (a crawler outside the real tree) runs unprotected rather than failing the snapshot step, since `Vendor.__init__` will hit the identical resolution failure on its own; a destination that does not exist yet is left alone, since there is nothing for a failure to discard. **Workflow** (`.github/workflows/cron-vendor.yml`): the "Update Vendor" step no longer aborts under `bash -e` when `pcapkit-vendor` exits non-zero -- the exit code is captured explicitly around that one command, so targets that did succeed still get `isort`ed, diffed, committed and pushed, while a "Warning" block naming the failed target(s) is appended to `$GITHUB_STEP_SUMMARY` and a `::warning::` annotation is emitted per failure; the step's own exit code is always `0` so later steps still run. A per-target `git checkout -- ` restore step was considered and dropped: with the snapshot/restore in place, a failed target's own const file is never touched at all, so there is nothing for the workflow itself to discard. `tests/vendor/test_vendor_exit_code_unit.py` (5 methods, through the real entrypoints with stub `Vendor` subclasses) and `tests/vendor/test_vendor_snapshot_restore_unit.py` (3: a successful run replaces the previous content, a failure leaves the previous file byte-for-byte intact, and a read-only destination is left exactly as it was) -- no network calls. Closes [#872](https://github.com/JarryShaw/PyPCAPKit/issues/872) ([#873](https://github.com/JarryShaw/PyPCAPKit/pull/873)). -- three follow-ups from [#873](https://github.com/JarryShaw/PyPCAPKit/pull/873)'s own review, none of which changed the behaviour that merged; one invariant was unpinned and two pieces of prose/lint were inaccurate. **1.** `except BaseException:` in `_snapshot_and_restore` is now pinned: a new `test_a_keyboard_interrupt_still_restores_and_propagates` asserts all three halves of the invariant -- content restored byte-for-byte, no `.bak` file surviving, and the `KeyboardInterrupt` itself still propagating rather than being converted into a `False` return. It discriminates: the real code passes clean, while mutating the clause to `except Exception:` fails on the restored content with the predicted `.bak` file left behind. **2.** A false docstring clause is corrected: it claimed a crawler whose `_dest_path` cannot be resolved "goes on to fail loudly anyway once `Vendor.__init__` calls the same `_dest_path` itself", which holds for one of the function's two triggers and not the other -- an instance-method-only override raises `TypeError` on the *class* call `_snapshot_and_restore` makes (no `self` to bind), but the *instance* call inside `__init__` binds `self` correctly and resolves fine, so that target runs to completion unprotected and **succeeds silently**: `run()` returns `True`, the file is replaced, and nothing warns that no snapshot was ever taken. Measured with a constructed instance-method-only stub. **3.** The `# pylint: disable=no-else-raise` pragma is **kept** -- removing it was going to be a third fix, on the mistaken belief that R1720 never applies to a `try`/`except`/`else`; that belief rested on a message-control artefact (`--disable=all --enable=no-else-raise` does not re-enable the check the way `--enable=R` does), and under the `Makefile`'s own flags -- what CI actually runs -- pylint 4.0.8 does flag it. The `else:` is deliberate: it is what makes the backup cleanup unreachable from the `except`, and deleting the `raise` that makes it so would be a mutation the suite must catch. Deliberately left out of scope: an untested `os.close(fd)` leak (harmless, since the path still exists and `copy2` reopens it) and the semantically-identical unindented-statement form of the same `else:`, which no test can distinguish from the current one. Closes [#875](https://github.com/JarryShaw/PyPCAPKit/issues/875) ([#876](https://github.com/JarryShaw/PyPCAPKit/pull/876)). -- `TLSKeyLabel` tracked the Mozilla NSS key-log wiki, but `draft-ietf-opsawg-pcapng-06` S4.7 now points at `[I-D.ietf-tls-keylogfile]` (draft-05), published as [RFC 9850](https://datatracker.ietf.org/doc/html/rfc9850), "The SSLKEYLOGFILE Format for TLS", whose S4.2 creates the IANA "TLS SSLKEYLOGFILE Labels" registry. Against that registry, `TLSKeyLabel` was missing `ECH_SECRET` and `ECH_CONFIG` ([RFC 9850](https://datatracker.ietf.org/doc/html/rfc9850) SS2.3/4.2), so `TLSKeyLabel('ECH_SECRET')` raised `ValueError` on an otherwise-legitimate Decryption Secrets Block; both are added, `ECH_SECRET` keeping the file's `# nosec B105` bandit convention for `*_SECRET` members. `RSA` has no row in [RFC 9850](https://datatracker.ietf.org/doc/html/rfc9850)'s registry -- it is an NSS-only label the NSS wiki records as removed in NSS 3.34 -- and is kept rather than deleted, since removing a public member is a breaking change, with a new docstring comment marking it the historical exception. `pcapkit/vendor/pcapng/secrets_type.py`'s stale `# NSS Key Log Format` comment is repointed at RFC 9850 to match, and `TLSKeyLabel`'s own docstring now cites it. A new test pins the member set against RFC 9850 plus the documented `RSA` exception, and asserts `ECH_SECRET`, `ECH_CONFIG` and `RSA` all resolve. Closes [#882](https://github.com/JarryShaw/PyPCAPKit/issues/882) ([#883](https://github.com/JarryShaw/PyPCAPKit/pull/883)). +- thirteen `re.sub` sites under `pcapkit/vendor/` passed `re.MULTILINE` as the fourth *positional* argument, which is `count`, not `flags`: substitution capped at 8 matches, and a `DeprecationWarning` on Python 3.13+ (a future `TypeError`). Found by an AST sweep rather than grep, since 8 of the 13 put the flag on a continuation line. The flag moves to `flags=` at all 13 sites (`default.py` plus one each in `hip/eddsa_curve.py`, `http/frame.py`, `http/method.py`, `http/status_code.py`, `ipv4/router_alert.py`, `ipv6/option.py`, `ipv6/router_alert.py`, `ipv6/tagger_id.py`, `reg/ethertype.py`, `tcp/flags.py`, `tcp/mp_tcp_option.py` and `tcp/option.py`). Behaviour preserving: the shared pattern (`r'\r*\n'`) has no MULTILINE-sensitive anchor, so no `const/` regeneration was needed ([#796](https://github.com/JarryShaw/PyPCAPKit/issues/796)). +- `main` was red ([#866](https://github.com/JarryShaw/PyPCAPKit/issues/866)): 12 tests across three files, all `RecursionError` (`tests/const/test_const_enum_lookup.py`; two `test_const_enum_no_mint.py` methods, two registries each; 7 of the 12 in `tests/protocols/misc/test_pcapng_unit.py`). `pcapkit/vendor/pcapng/record_type.py` and `secrets_type.py` rendered a two-line `_missing_` body whose first line did not return: `cls._unregistered_member(value, 'Unassigned')` followed by `return cls(value)`. That was harmless while the first line was `extend_enum(...)`, which registers the member so `cls(value)` found it; [#861](https://github.com/JarryShaw/PyPCAPKit/pull/861) (above) replaced it with `_unregistered_member`, which deliberately does not register, so `cls(value)` missed and re-entered `_missing_`. [#861](https://github.com/JarryShaw/PyPCAPKit/pull/861) half-corrected the two files: it fixed the *generated* modules (leaving the old `return cls(value)` as dead code) and never touched the two *crawlers*, which is why [#861](https://github.com/JarryShaw/PyPCAPKit/pull/861)'s own CI stayed green until a live regeneration from the stale crawlers reintroduced the defect; [#861](https://github.com/JarryShaw/PyPCAPKit/pull/861)'s cross-review had verified byte-identical regeneration on a 5-of-82 sample that missed these two. Both crawlers now collapse to the single returning line the other 101 registries use, and the two const files are regenerated (`record_type.py` fetches the network for its own `LINK` constant, so that regeneration needs connectivity, though it came back byte-identical; `secrets_type.py` is offline). The new guard sits at the crawler-emission layer rather than the behavioural one `test_const_enum_lookup.py` already swept: a hard-coded `_missing_` body must end in a return, and no emitted line may re-enter `_missing_` for the same value -- deliberately narrower than "every emitted line must return", since 53 crawlers legitimately emit 109 non-returning lines in the shape `Vendor.process` uses. The guard sees only 35 of 96 crawlers; the other 61 are covered behaviourally. Closes [#866](https://github.com/JarryShaw/PyPCAPKit/issues/866) ([#867](https://github.com/JarryShaw/PyPCAPKit/pull/867)). +- `python -m pcapkit.vendor ` could not fail: `run()` swallowed every crawler exception as a filterable `VendorRuntimeWarning` and `main()` always returned `0`, so a broken crawler regenerating nothing looked like success to a human and to the cron job. Per the owner's ruling (*"only discard changes made by a non-zero sub-vendor... for cron jobs, at most warn about failed sub-vendors, but never fail the entire job"*), three pieces land. **Exit code** (`pcapkit/vendor/__main__.py`): `run()` returns whether its target succeeded; `main()` attempts every target and returns `1` if any raised, writing the failing target's qualified name and exception `repr` to stderr unconditionally via `print()` rather than the filterable warning. **Per-target snapshot and restore**: a new `_snapshot_and_restore` context manager backs up a target's const file with `tempfile.mkstemp` before it runs (resolving the destination through `Vendor._dest_path`, now a `classmethod` in `pcapkit/vendor/default.py` so it can be called before `Vendor.__init__` fetches, renders or writes), restores the backup via `os.replace` on any exception before re-raising, and discards it on success. A target whose destination cannot be resolved (a crawler outside the real tree) runs unprotected rather than failing the snapshot step, since `Vendor.__init__` hits the same resolution failure itself; a destination that does not exist yet is left alone. **Workflow** (`.github/workflows/cron-vendor.yml`): the "Update Vendor" step no longer aborts under `bash -e` when `pcapkit-vendor` exits non-zero. The exit code is captured around that one command, so targets that succeeded are still `isort`ed, diffed, committed and pushed, while a "Warning" block naming the failed target(s) is appended to `$GITHUB_STEP_SUMMARY`, a `::warning::` annotation is emitted per failure, and the step's own exit code is always `0`. A per-target `git checkout -- ` restore step was dropped: with snapshot/restore, a failed target's const file is never touched. `tests/vendor/test_vendor_exit_code_unit.py` (5 methods, through the real entrypoints with stub `Vendor` subclasses) and `tests/vendor/test_vendor_snapshot_restore_unit.py` (3: success replaces the previous content, failure leaves it byte-for-byte intact, a read-only destination is left as it was) make no network calls. Closes [#872](https://github.com/JarryShaw/PyPCAPKit/issues/872) ([#873](https://github.com/JarryShaw/PyPCAPKit/pull/873)). +- three follow-ups from [#873](https://github.com/JarryShaw/PyPCAPKit/pull/873)'s review, none changing merged behaviour: one unpinned invariant and two inaccurate pieces of prose/lint. **1.** `except BaseException:` in `_snapshot_and_restore` is pinned: `test_a_keyboard_interrupt_still_restores_and_propagates` asserts content restored byte-for-byte, no `.bak` surviving, and the `KeyboardInterrupt` still propagating rather than becoming a `False` return; mutating the clause to `except Exception:` fails it. **2.** A false docstring clause is corrected: it said a crawler whose `_dest_path` cannot be resolved "goes on to fail loudly anyway once `Vendor.__init__` calls the same `_dest_path` itself", which holds for one of the function's two triggers and not the other. An instance-method-only override raises `TypeError` on the *class* call `_snapshot_and_restore` makes (no `self` to bind), but the *instance* call inside `__init__` resolves fine, so that target runs unprotected and **succeeds silently**: `run()` returns `True`, the file is replaced, and nothing warns that no snapshot was taken. **3.** The `# pylint: disable=no-else-raise` pragma is **kept**. Removing it rested on the belief that R1720 never applies to `try`/`except`/`else`, a message-control artefact (`--disable=all --enable=no-else-raise` does not re-enable the check the way `--enable=R` does); under the `Makefile`'s flags, which CI runs, pylint 4.0.8 flags it. The `else:` is deliberate: it makes the backup cleanup unreachable from the `except`. Left out of scope: an untested `os.close(fd)` leak (harmless, the path still exists and `copy2` reopens it) and the semantically identical unindented form of the same `else:`. Closes [#875](https://github.com/JarryShaw/PyPCAPKit/issues/875) ([#876](https://github.com/JarryShaw/PyPCAPKit/pull/876)). +- `TLSKeyLabel` tracked the Mozilla NSS key-log wiki, but `draft-ietf-opsawg-pcapng-06` S4.7 now points at `[I-D.ietf-tls-keylogfile]` (draft-05), published as [RFC 9850](https://datatracker.ietf.org/doc/html/rfc9850), "The SSLKEYLOGFILE Format for TLS", whose S4.2 creates the IANA "TLS SSLKEYLOGFILE Labels" registry. Against it, `TLSKeyLabel` was missing `ECH_SECRET` and `ECH_CONFIG` ([RFC 9850](https://datatracker.ietf.org/doc/html/rfc9850) SS2.3/4.2), so `TLSKeyLabel('ECH_SECRET')` raised `ValueError` on a legitimate Decryption Secrets Block; both are added, `ECH_SECRET` keeping the file's `# nosec B105` bandit convention for `*_SECRET` members. `RSA` has no row in [RFC 9850](https://datatracker.ietf.org/doc/html/rfc9850)'s registry (it is an NSS-only label the NSS wiki records as removed in NSS 3.34) and is kept, since removing a public member is breaking, with a docstring comment marking it the historical exception. `pcapkit/vendor/pcapng/secrets_type.py`'s stale `# NSS Key Log Format` comment is repointed at RFC 9850, and `TLSKeyLabel`'s docstring cites it. A new test pins the member set against RFC 9850 plus the `RSA` exception. Closes [#882](https://github.com/JarryShaw/PyPCAPKit/issues/882) ([#883](https://github.com/JarryShaw/PyPCAPKit/pull/883)). ### Project infrastructure #### Added - an end-to-end test tier ([#376](https://github.com/JarryShaw/PyPCAPKit/pull/376)), sample-capture generators so a fresh clone can rebuild every fixture ([#340](https://github.com/JarryShaw/PyPCAPKit/pull/340)), a Dockerised engine benchmark covering every supported Python version ([#410](https://github.com/JarryShaw/PyPCAPKit/pull/410)), and registry round-trip coverage that records the entries which cannot close the cycle rather than skipping them ([#440](https://github.com/JarryShaw/PyPCAPKit/pull/440), [#504](https://github.com/JarryShaw/PyPCAPKit/pull/504)). -- coverage for the untested half of the [#431](https://github.com/JarryShaw/PyPCAPKit/issues/431) accommodation: a TCP or IPv4 option whose declared length asks for more data than the capture actually holds, pinning that it still parses, with the short read zero-padded rather than rejected. The one test [#431](https://github.com/JarryShaw/PyPCAPKit/issues/431) left behind only covers an option area with no data behind it at all; a candidate fix for [#554](https://github.com/JarryShaw/PyPCAPKit/issues/554) turned the untested half into an unwrapped `FieldValueError` while the rest of the suite stayed green ([#571](https://github.com/JarryShaw/PyPCAPKit/pull/571), [#572](https://github.com/JarryShaw/PyPCAPKit/issues/572)). -- `CITATION.cff`, citation metadata in Citation File Format 1.2.0, which GitHub renders as the repository's "Cite this repository" button and which citation managers and dependency inventories read directly. It is the machine-readable half of the attribution BSD-3-Clause already asks for, so credit carries into a paper or a bill of materials rather than depending on a reader opening `LICENSE`. Validated with `cffconvert --validate` and against the published 1.2.0 schema; `doi` and `orcid` are omitted rather than invented, since neither exists for this project today and both are checked formats, so a wrong value would still validate. The licence itself is deliberately unchanged -- still BSD-3-Clause, no `NOTICE` file, no change to its terms. Alongside it the copyright line moves from `2018-2023` to `2018-2026` -- `LICENSE` was its only occurrence in the tree, since `docs/source/conf.py` already derives its own from the current year -- and a stray `s` after the closing `DAMAGE.` of the licence text, present since the Mozilla-to-BSD relicence, is removed, so the wording now matches canonical BSD-3-Clause exactly ([#615](https://github.com/JarryShaw/PyPCAPKit/pull/615)). -- `examples/generators/endian.py`, and the byte-order tests that read what it writes. There was no big-endian `.pcap` in the repository at all, which is why [#605](https://github.com/JarryShaw/PyPCAPKit/issues/605) survived its own code review: the one-character fix leaves the corrected path exactly as untested as the broken one. The generator writes three captures -- `big_endian.pcap` (magic `a1 b2 c3 d4`), `big_endian_nanosecond.pcap` (`a1 b2 3c 4d`, the first fixture to take that branch of the magic-number table) and `little_endian.pcap` (`d4 c3 b2 a1`) -- carrying the *same three records* in each container, so the tests can assert that the byte order makes no difference to what is read out rather than only that the big-endian file matches numbers written down in a test. Frame 3 is captured short, 1200 octets on the wire cut to a 96-octet `snaplen`, so `incl_len` and `orig_len` differ and cannot both be satisfied by one byte-swapped value. `test_frame_endian_runtime.py` drives all three through `extract()` and walks each file's record chain with `struct` to derive its own expectations; a unit-tier case in `test_header_frame_unit.py` builds a two-record big-endian capture in memory instead, so the regression is also caught by the fixture-free selection CI runs on every push. All four fail on the unfixed tree -- the three fixture-backed ones by that `ValueError`, the in-memory one by `AssertionError: 3106905 != 1500000000` -- while the little-endian twin passes on both trees, which is what shows the records themselves are not the variable ([#605](https://github.com/JarryShaw/PyPCAPKit/issues/605)). +- coverage for the untested half of the [#431](https://github.com/JarryShaw/PyPCAPKit/issues/431) accommodation: a TCP or IPv4 option whose declared length asks for more data than the capture holds, pinning that it still parses with the short read zero-padded rather than rejected. The one test [#431](https://github.com/JarryShaw/PyPCAPKit/issues/431) left covers only an option area with no data behind it; a candidate fix for [#554](https://github.com/JarryShaw/PyPCAPKit/issues/554) turned the untested half into an unwrapped `FieldValueError` while the rest of the suite stayed green ([#571](https://github.com/JarryShaw/PyPCAPKit/pull/571), [#572](https://github.com/JarryShaw/PyPCAPKit/issues/572)). +- `CITATION.cff`, citation metadata in Citation File Format 1.2.0, which GitHub renders as the "Cite this repository" button and which citation managers and dependency inventories read directly. It is the machine-readable half of the attribution BSD-3-Clause already asks for. Validated with `cffconvert --validate` and against the published 1.2.0 schema; `doi` and `orcid` are omitted rather than invented, since neither exists for this project and both are checked formats, so a wrong value would still validate. The licence is unchanged (BSD-3-Clause, no `NOTICE`). Alongside it the copyright line moves from `2018-2023` to `2018-2026` (`LICENSE` was its only occurrence; `docs/source/conf.py` derives its own from the current year), and a stray `s` after the closing `DAMAGE.` of the licence text, present since the Mozilla-to-BSD relicence, is removed so the wording matches canonical BSD-3-Clause ([#615](https://github.com/JarryShaw/PyPCAPKit/pull/615)). +- `examples/generators/endian.py`, and the byte-order tests that read what it writes. There was no big-endian `.pcap` in the repository, which is why [#605](https://github.com/JarryShaw/PyPCAPKit/issues/605) survived its own code review: the one-character fix left the corrected path as untested as the broken one. The generator writes `big_endian.pcap` (magic `a1 b2 c3 d4`), `big_endian_nanosecond.pcap` (`a1 b2 3c 4d`, the first fixture to take that branch of the magic-number table) and `little_endian.pcap` (`d4 c3 b2 a1`), carrying the *same three records* in each, so tests can assert that byte order makes no difference to what is read rather than only matching numbers written in a test. Frame 3 is captured short (1200 octets on the wire, 96-octet `snaplen`), so `incl_len` and `orig_len` differ and cannot both be satisfied by one byte-swapped value. `test_frame_endian_runtime.py` drives all three through `extract()` and derives expectations by walking each file's record chain with `struct`; a unit-tier case in `test_header_frame_unit.py` builds a two-record big-endian capture in memory, so the fixture-free selection CI runs on every push catches it too. All four fail on the unfixed tree (the three fixture-backed by that `ValueError`, the in-memory one by `AssertionError: 3106905 != 1500000000`) while the little-endian twin passes on both, showing the records themselves are not the variable ([#605](https://github.com/JarryShaw/PyPCAPKit/issues/605)). #### Changed - renames with no compatibility alias left behind: `examples/sample` and `examples/samples` -- one letter apart, holding different things -- are now `examples/captures` and `examples/generators`. -- the README is a landing page now, and Markdown rather than reStructuredText. `README.rst` (424 lines) became `README.md` (102), keeping what a reader arriving from PyPI or a search result actually needs -- what the library is, why it exists rather than Scapy or DPKT, how to install it, a worked example, and where the documentation lives -- and dropping the technical detail the documentation already carried. **Module Structure**, **Engine Comparison**, **Engine support by Python version**, **Test Environment**, **Test Results** and **Installation Notes** were each already duplicated in `docs/source/index.rst`, in a fuller form, so they are linked rather than restated. Two blocks existed nowhere else and moved rather than going: **Testing** is now `docs/source/testing.rst`, registered in the index toctree, and the `pipenv` and `make setup` local development block joined the Installation section of `docs/source/index.rst`. Requested by the project owner, and a deliberate exception to the convention that documentation here is reStructuredText -- for the README only, since it is the one documentation file whose renderers are GitHub and PyPI rather than Sphinx. Accordingly `setup.py` reads `README.md` and declares its content type as `text/markdown`, and `MANIFEST.in` gains an `include README.md` line because `global-include *.rst` no longer matches the file. That line is belt-and-braces rather than load-bearing, contrary to what this entry claimed when it was written: setuptools' own `sdist` command ships whichever of `README`, `README.rst`, `README.txt` and `README.md` exists, before `MANIFEST.in` is read at all, so deleting the line leaves the sdist's file listing byte-identical -- 861 entries either way, with an empty `diff` -- and the result still installs. `setup.py` does read the file unguarded, but it reads it from wherever `setup.py` is executing, which when `pip` installs an sdist is the unpacked sdist rather than a checkout, so that read cannot be made to fail by dropping the line either. Of the `include` lines in that file only `CHANGELOG.md` is load-bearing, which the [#631](https://github.com/JarryShaw/PyPCAPKit/pull/631) entry below measured independently. Verified with `twine check --strict` against a built sdist and wheel, both of which pass. The rename's own references moved with it, since a change that renames a file owns the references to it: `examples/benchmark/Dockerfile` copies `README.md` -- a literal `COPY` of the old name would have failed the layer outright and taken `make bench`, `make bench-quick` and `run.sh` with it -- and the benchmark harness prose that named the root README as the destination of its generated tables now names `docs/source/index.rst`, which is where those tables went. That covers the `Makefile` comment, `report.py`, `test_harness.py`, `run.sh` and the suite's own README, including the one place whose stated reason had inverted: the emitted markup is kept parseable by plain docutils, which is now a conservative choice rather than a hard requirement, because the page it lands on is rendered by Sphinx. `examples/benchmark/benchmark.py` still says `README.rst` and is left alone, because it means the benchmark suite's own README in the same directory, not the project's ([#619](https://github.com/JarryShaw/PyPCAPKit/pull/619)). -- `CODE_OF_CONDUCT.md` moves from Contributor Covenant 1.4 to Contributor Covenant 3.0, at the maintainer's request. The text is the canonical 3.0 Markdown fetched from https://www.contributor-covenant.org/version/3/0/code_of_conduct/code_of_conduct.md rather than a transcription, so the pledge, the encouraged and restricted behaviours and the scope are unaltered. Three things needed deciding rather than copying. 3.0 ships two `[NOTE` placeholders an adopter must fill: the reporting channel, which now names `jarryshaw@icloud.com` -- the same contact 1.4 carried and the one `SECURITY.md` already points at as its email fallback -- plus GitHub's report-abuse form for the case a single-maintainer project cannot otherwise cover, a report about the maintainer; and the enforcement section, whose placeholder is an instruction to the adopter and is removed. 3.0 then assigns enforcement throughout to plural "Community Moderators" (and once, inconsistently, to "Community Managers"), which this repository does not have, so all eight occurrences become the singular maintainer. The four-rung ladder -- Warning, Temporarily Limited Activities, Temporary Suspension, Permanent Ban -- is offered as a suggestion and is **kept**, because each rung maps onto a lever one person actually holds on GitHub: a private message, a locked thread, an interaction limit or block, a permanent block. Finally, 3.0 is licensed CC BY-SA 4.0 where 1.4's attribution paragraph carried no licence notice at all, so the attribution now names version 3.0, links the permanent `version/3/0/` URL, carries the CC BY-SA 4.0 notice and link, indicates that changes were made as BY requires, and says explicitly that the share-alike term covers this document only -- the code remains BSD-3-Clause and `LICENSE` is untouched. Rendering was checked against GitHub's own Markdown API rather than assumed: the ladder comes back as four list items each nesting three, which is what [#613](https://github.com/JarryShaw/PyPCAPKit/pull/613) had to repair in the 1.4 file when a stray list marker collapsed the whole document into one nested item ([#624](https://github.com/JarryShaw/PyPCAPKit/pull/624)). -- reconciled which private (`_xxx`) attributes and methods the Sphinx build documents, replacing an ad hoc mix with one stated rule: document the contract, hide the recipe. A member every subclass must implement, or one whose shape a caller genuinely depends on, stays documented even though its name starts with an underscore; a private helper that exists only to keep one method short does not. Seven module-level directives naming pure implementation detail were dropped -- `esp._resolve`, `esp._CRYPTO`, `ngap._convert`, `ngap._revert`, `ngap._PYCRATE`, `ngap._PDU_LOCK` and `pypcapfile._NamedStream` (whose nested `name`/`read` members go with it, being reachable only through a private class) -- while 39 `autoattribute` directives were added for class-private state that *is* contract: `Extractor._flag_f`, the `PyPCAP` and `PCAP_CT` engines' own `_backend` -- the only two of the six third-party engines that have one -- `TraceFlow`'s internal fields, and `FieldBase`/`Field` internals among them; the built-in `PCAP` and `PCAPNG` engines gained `_gbhdr`, `_vinfo` and `_nnsec` for the former and `_ctx`/`_ctx_list` for the latter, and no `_backend` at all. `_dlink` is not built-in-only: `PCAP` documents it alongside the three third-party engines that share it, `PyPCAP`, `PCAP_CT` and `PyPCAPFile`. The 128 runtime definitions of `_missing_` -- 121 under `pcapkit.const`, the other 7 inline in `pcapkit.protocols` -- gained one unified write-up in place of a directive per class: a new "Unrecognised Values" section in `docs/source/pcapkit/const/index.rst`, cross-referenced from `registry.rst`, since `conf.py` already excludes `_missing_` from every `autoclass` via `exclude-members` in `autodoc_default_options` -- there is no `automodule` directive anywhere under `docs/source/`. `CONTRIBUTING.md` gained the rule itself as a named section, so the next directive gets judged against a written test rather than against precedent. Verified with `sphinx-build -b html` under `PCAPKIT_SPHINX=1`: 53 warnings on `main` before this change, 54 after, the one addition being a pre-existing bare `Type` cross-reference ambiguity newly rendered by `TraceFlow._foutio`'s new directive rather than a defect this change introduced -- [#709](https://github.com/JarryShaw/PyPCAPKit/issues/709) tracked it. No line under `pcapkit/` changed ([#684](https://github.com/JarryShaw/PyPCAPKit/issues/684)). -- `examples/captures/out.json`, `out.plist`, `out.txt` and `pcapng.txt` are no longer tracked in git; they are build output, not fixtures, and a tracked rendering with no reader goes stale silently every time the code that produces it changes. `pcapng.txt` is exactly that: it still recorded `packet -> NIL` for the four Enhanced Packet Blocks of `dhcp.pcapng` long after [#683](https://github.com/JarryShaw/PyPCAPKit/pull/683) gave those blocks their captured octets back, and nothing had regenerated it. Rather than regenerate it once more and leave the same drift free to recur, the four files are removed from the index and folded into `examples/captures/`'s existing blanket `.gitignore` rule; `in.pcap` and `dhcp.pcapng`, the genuine inputs, stay tracked. `examples/legacy_smoke/Makefile` and its `README.rst` are reworded to describe regenerating these reports via `make fixtures` rather than implying they ship committed, and a new `tests/project/test_capture_tracking.py` (5 tests, 8 subtests) pins the invariant going forward; 2 of the 5 tests (4 of the 12 subtests) fail against the pre-change tree with the four reports restored to the index, the other 3 passing on both trees by construction ([#685](https://github.com/JarryShaw/PyPCAPKit/issues/685)). -- `tests/project/test_isort_clean.py` gated on `isort` with a `try`/`except ImportError` inside `setUpClass`, which `tests/_dependency_gates.py` cannot see -- `_gates_of()` walks a decorator list and never a function body -- and `isort` was in no `pyproject.toml` extra, so the file reported `OK (skipped=1)` on every CI leg and the dependency-gate guard could not tell. It now gates on a module-level `HAS_ISORT` and a class-level `@unittest.skipUnless` so `gated_scopes()` counts the gate, `MODULE_PROVIDERS` maps `isort` so the flag resolves instead of raising, and `isort` joins the `test` extra rather than being excluded -- `lint.yml` installs none and its own header says so, so unlike `mypy`'s gate there is no existing check to defer to. No workflow changed, every pytest job's install line already building on `.[test]`; `gated_scopes()` goes 250 to 251 and `dependency_gate_gaps()` holds at 18. The third instance of the hazard [#745](https://github.com/JarryShaw/PyPCAPKit/issues/745) exists to stop ([#766](https://github.com/JarryShaw/PyPCAPKit/issues/766)). -- `tests/vendor/test_vendor_reg_apptype_generator_unit.py` gated on `mypy` with no `MODULE_PROVIDERS` entry for it, so `_top_level_providers()` died on a bare `KeyError: 'mypy'` the moment the gate was made visible -- a further instance of the hazard [#745](https://github.com/JarryShaw/PyPCAPKit/issues/745) exists to stop, the same one [#766](https://github.com/JarryShaw/PyPCAPKit/issues/766) above also carries. `MODULE_PROVIDERS['mypy'] = ('mypy',)` fixes the lookup, and the three bare `MODULE_PROVIDERS[...]` subscripts route through that same function, which raises an `AssertionError` naming the module, the key looked up and the table to add it to on any future gap, rather than a bare `KeyError`. `mypy` is deliberately not added to a `pyproject.toml` extra: no extra carries it today, it is a `Pipfile` dev-dependency that `lint.yml` installs directly and already runs whole-package, and `lint.yml` runs no pytest job for `pytest_jobs()` to see -- so the fix is a new `DEPENDENCY_GATE_EXCLUSIONS['HAS_MYPY']` entry on `test`, `engine-tests` and `gate` rather than an install line. `dependency_gate_gaps()` goes 15 to 18 `(flag, job)` pairs, all three new and expected. `tests/test_tier_guard.py` goes 97 to 100 tests, 512 to 524 subtests -- measured under plain `unittest` with a `subTest` counter rather than trusting `pytest-subtests` ([#779](https://github.com/JarryShaw/PyPCAPKit/issues/779)). -- `engine='pyshark'` had no CI job driving it through a real `tshark`-parsed capture. `pypcap-parity` installs `pyshark` but its `apt-get install` step named no `tshark` package, so `PyShark.unsupported_reason()` declined the engine, the extractor fell back to the built-in parser, and the one real-extraction assertion, `assertGreater(extractor.length, 0)`, passed against the fallback's output -- a vacuous pass invisible in every skip count. `pypcap-parity` now installs `tshark` with the same debconf pre-seed `engine-tests` already carries, and the test proves which engine actually ran: no `EngineWarning` fires, `extractor._exnam == 'pyshark'` and the engine is a `PyShark` instance -- the same idiom the PyPCAP parity test already used. Only `tests/integration/test_engine_runtime.py` and `test_engine_parity.py` change among the test files; the length assertion itself is kept, not weakened ([#846](https://github.com/JarryShaw/PyPCAPKit/pull/846)). -- `engine-tests` ran one Python-version leg per interpreter regardless of which extraction engines that interpreter could actually install, so `pyproject.toml`'s own `PyPCAPFile = ["pypcapfile; python_version < '3.12'"]` marker resolved to *nothing* on 3.12--3.14, yet the `Engines Python 3.12/3.13/3.14` legs still passed and, once promoted to a required check, gated merges while exercising an engine that was never installed. Rebuilt as a genuine Python x engine matrix -- 6 interpreters x 6 engines, 36 cells after 10 `include` overrides -- with three honest outcomes: **26** `supported` cells that install through the real `pyproject.toml` extra and must pass, **6** `unsupported` cells (`PyPCAPFile` on 3.12--3.15, `PyShark` on 3.14/3.15) installed by a raw `pip install` to bypass the marker and assert the engine's own `PYTHON_CEILING` decline, and **4** `not-installable` cells (`PyPCAP` on 3.12--3.15) where a genuine C-extension build failure is logged and the test step skipped. The install step never passes silently in either direction. 3.15 is `continue-on-error` throughout; job names change shape (`Engines Python X` becomes `Engines Python X (engine)`), which needed the branch-protection ruleset's required-check list updated in the same window so a required check that stopped existing did not block every merge permanently ([#849](https://github.com/JarryShaw/PyPCAPKit/pull/849)). -- Ruleset `23497679`'s `required_status_checks` names 22 exact check contexts, and GitHub rulesets match literally with no wildcard, so the required list needs hand-editing on every matrix change. Five of the 22, the old single-cell `Engines Python ` names, are already dead since [#849](https://github.com/JarryShaw/PyPCAPKit/pull/849); the five `Compat Python 3.10`-`3.14` names stay live, emitted independently by `python-compatibility.yml`, and are not touched. Adds one job, `required-checks` (context `Required checks passed`), depending on `[test, integration, engine-tests, pypcap-parity]` with `if: always() && inputs.gate-only != true` and an explicit per-dependency check that fails loudly, naming the job, on anything but `success` -- a bare `needs:` list would leave this job silently *skipped* rather than failed the moment a dependency failed, and GitHub's own troubleshooting docs list a skipped required check as passing. That one context is meant to replace 17 of the ruleset's 22 hand-listed entries (`test`, `integration`, `Engines`, `pypcap-parity`) once the ruleset itself is updated separately -- a repository setting this PR does not touch, and the five `Compat` names are deliberately excluded from that replacement. One file, `.github/workflows/unit-tests.yml`, 167 lines added; no library or test code changes ([#856](https://github.com/JarryShaw/PyPCAPKit/pull/856)). +- the README is a landing page now, and Markdown rather than reStructuredText. `README.rst` (424 lines) became `README.md` (102), keeping what a reader arriving from PyPI or a search result needs (what the library is, why it exists rather than Scapy or DPKT, how to install it, a worked example, where the documentation lives). **Module Structure**, **Engine Comparison**, **Engine support by Python version**, **Test Environment**, **Test Results** and **Installation Notes** were already duplicated in fuller form in `docs/source/index.rst`, so they are linked rather than restated; the two blocks that existed nowhere else moved: **Testing** is now `docs/source/testing.rst`, in the index toctree, and the `pipenv` and `make setup` block joined the Installation section of `docs/source/index.rst`. Requested by the project owner, and a deliberate exception to the reStructuredText convention, for the README only, since its renderers are GitHub and PyPI rather than Sphinx. `setup.py` reads `README.md` and declares `text/markdown`, and `MANIFEST.in` gains `include README.md` because `global-include *.rst` no longer matches. That line is belt-and-braces rather than load-bearing, contrary to what this entry claimed when written: setuptools' `sdist` ships whichever of `README`, `README.rst`, `README.txt` and `README.md` exists before `MANIFEST.in` is read, so deleting the line leaves the sdist listing byte-identical, and `setup.py` reads the file from where it executes, which under `pip` is the unpacked sdist. Of the `include` lines only `CHANGELOG.md` is load-bearing, measured in the [#631](https://github.com/JarryShaw/PyPCAPKit/pull/631) entry below. References moved with the rename: `examples/benchmark/Dockerfile` copies `README.md` (a literal `COPY` of the old name would have failed the layer and taken `make bench` with it), and benchmark prose naming the root README as the destination of generated tables now names `docs/source/index.rst`. `examples/benchmark/benchmark.py` still says `README.rst` on purpose, meaning the benchmark suite's own README ([#619](https://github.com/JarryShaw/PyPCAPKit/pull/619)). +- `CODE_OF_CONDUCT.md` moves from Contributor Covenant 1.4 to 3.0, at the maintainer's request. The text is the canonical 3.0 Markdown fetched from https://www.contributor-covenant.org/version/3/0/code_of_conduct/code_of_conduct.md rather than transcribed. Three things needed deciding. 3.0's two `[NOTE` placeholders are filled or removed: the reporting channel names `jarryshaw@icloud.com` (the contact 1.4 carried and `SECURITY.md` uses as email fallback) plus GitHub's report-abuse form for a report about the maintainer, which a single-maintainer project cannot otherwise cover; the enforcement placeholder is an instruction to the adopter and is removed. 3.0's plural "Community Moderators" become the singular maintainer (eight occurrences), this repository having no moderators. The four-rung ladder (Warning, Temporarily Limited Activities, Temporary Suspension, Permanent Ban) is a suggestion and is **kept**, each rung mapping to a lever one person holds on GitHub. 3.0 is CC BY-SA 4.0 where 1.4 carried no licence notice, so the attribution names version 3.0, links the permanent `version/3/0/` URL, carries the CC BY-SA 4.0 notice, indicates changes were made as BY requires, and says the share-alike term covers this document only (the code remains BSD-3-Clause, `LICENSE` untouched). The ladder renders as four list items each nesting three, checked against GitHub's Markdown API; [#613](https://github.com/JarryShaw/PyPCAPKit/pull/613) had to repair the 1.4 file when a stray list marker collapsed it into one nested item ([#624](https://github.com/JarryShaw/PyPCAPKit/pull/624)). +- reconciled which private (`_xxx`) attributes and methods the Sphinx build documents, replacing an ad hoc mix with one rule: document the contract, hide the recipe. A member every subclass must implement, or whose shape a caller depends on, stays documented despite its underscore; a private helper that only keeps one method short does not. Seven module-level directives for pure implementation detail (`esp._resolve`, `esp._CRYPTO`, `ngap._convert`, `ngap._revert`, `ngap._PYCRATE`, `ngap._PDU_LOCK`, `pypcapfile._NamedStream`) were dropped, while 39 `autoattribute` directives were added for class-private state that *is* contract, among them `Extractor._flag_f`, the `PyPCAP` and `PCAP_CT` engines' `_backend` (the only two of the six third-party engines with one), `TraceFlow`'s internal fields, `FieldBase`/`Field` internals, the built-in `PCAP` engine's `_gbhdr`, `_vinfo`, `_nnsec` and `_dlink` (shared with `PyPCAP`, `PCAP_CT` and `PyPCAPFile`) and `PCAPNG`'s `_ctx`/`_ctx_list`. The 128 runtime definitions of `_missing_` (121 under `pcapkit.const`, 7 inline in `pcapkit.protocols`) gained one write-up in place of a directive per class, a new "Unrecognised Values" section in `docs/source/pcapkit/const/index.rst` cross-referenced from `registry.rst` (`conf.py` already excludes `_missing_` from every `autoclass`). `CONTRIBUTING.md` gained the rule as a named section. `sphinx-build -b html` gives 53 warnings on `main` and 54 after, the addition being the pre-existing bare `Type` ambiguity newly rendered by `TraceFlow._foutio`'s new directive, tracked as [#709](https://github.com/JarryShaw/PyPCAPKit/issues/709). No line under `pcapkit/` changed ([#684](https://github.com/JarryShaw/PyPCAPKit/issues/684)). +- `examples/captures/out.json`, `out.plist`, `out.txt` and `pcapng.txt` are no longer tracked in git; they are build output, not fixtures, and a tracked rendering with no reader goes stale silently. `pcapng.txt` is exactly that: it still recorded `packet -> NIL` for the four Enhanced Packet Blocks of `dhcp.pcapng` long after [#683](https://github.com/JarryShaw/PyPCAPKit/pull/683) gave those blocks their captured octets back. Rather than regenerate it again and leave the drift free to recur, the four files are removed from the index and folded into `examples/captures/`'s blanket `.gitignore` rule; the genuine inputs `in.pcap` and `dhcp.pcapng` stay tracked. `examples/legacy_smoke/Makefile` and its `README.rst` now describe regenerating these reports via `make fixtures`, and `tests/project/test_capture_tracking.py` (5 tests, 8 subtests) pins the invariant; 2 of the 5 fail against the pre-change tree with the four reports restored, the other 3 passing on both by construction ([#685](https://github.com/JarryShaw/PyPCAPKit/issues/685)). +- `tests/project/test_isort_clean.py` gated on `isort` with a `try`/`except ImportError` inside `setUpClass`, which `tests/_dependency_gates.py` cannot see (`_gates_of()` walks a decorator list, never a function body), and `isort` was in no `pyproject.toml` extra, so the file reported `OK (skipped=1)` on every CI leg and the dependency-gate guard could not tell. It now gates on a module-level `HAS_ISORT` and a class-level `@unittest.skipUnless` so `gated_scopes()` counts it, `MODULE_PROVIDERS` maps `isort`, and `isort` joins the `test` extra rather than being excluded (`lint.yml` installs none, so unlike `mypy` there is no existing check to defer to). No workflow changed, every pytest job already building on `.[test]`; `gated_scopes()` goes 250 to 251 and `dependency_gate_gaps()` holds at 18. The third instance of the hazard [#745](https://github.com/JarryShaw/PyPCAPKit/issues/745) exists to stop ([#766](https://github.com/JarryShaw/PyPCAPKit/issues/766)). +- `tests/vendor/test_vendor_reg_apptype_generator_unit.py` gated on `mypy` with no `MODULE_PROVIDERS` entry, so `_top_level_providers()` died on a bare `KeyError: 'mypy'` once the gate became visible -- a further instance of the hazard [#745](https://github.com/JarryShaw/PyPCAPKit/issues/745) exists to stop, like [#766](https://github.com/JarryShaw/PyPCAPKit/issues/766) above. `MODULE_PROVIDERS['mypy'] = ('mypy',)` fixes the lookup, and the three bare `MODULE_PROVIDERS[...]` subscripts route through that function, which raises an `AssertionError` naming the module, key and table on any future gap. `mypy` is deliberately not added to a `pyproject.toml` extra: no extra carries it, it is a `Pipfile` dev-dependency that `lint.yml` installs and already runs whole-package, and `lint.yml` runs no pytest job for `pytest_jobs()` to see, so the fix is a `DEPENDENCY_GATE_EXCLUSIONS['HAS_MYPY']` entry on `test`, `engine-tests` and `gate`. `dependency_gate_gaps()` goes 15 to 18 `(flag, job)` pairs, all three new and expected; ([#779](https://github.com/JarryShaw/PyPCAPKit/issues/779)). +- `engine='pyshark'` had no CI job driving it through a real `tshark`-parsed capture. `pypcap-parity` installs `pyshark` but its `apt-get install` named no `tshark` package, so `PyShark.unsupported_reason()` declined the engine, the extractor fell back to the built-in parser, and the one real-extraction assertion, `assertGreater(extractor.length, 0)`, passed against the fallback's output: a vacuous pass invisible in every skip count. `pypcap-parity` now installs `tshark` with the debconf pre-seed `engine-tests` carries, and the test proves which engine ran (no `EngineWarning`, `extractor._exnam == 'pyshark'`, a `PyShark` instance), as the PyPCAP parity test does. Only `tests/integration/test_engine_runtime.py` and `test_engine_parity.py` change; the length assertion is kept ([#846](https://github.com/JarryShaw/PyPCAPKit/pull/846)). +- `engine-tests` ran one Python-version leg per interpreter regardless of which engines that interpreter could install, so `pyproject.toml`'s `PyPCAPFile = ["pypcapfile; python_version < '3.12'"]` marker resolved to *nothing* on 3.12--3.14, yet the `Engines Python 3.12/3.13/3.14` legs passed and, once required, gated merges while exercising an engine that was never installed. Rebuilt as a Python x engine matrix (6 interpreters x 6 engines, 36 cells after 10 `include` overrides) with three outcomes: **26** `supported` cells that install through the real extra and must pass; **6** `unsupported` cells (`PyPCAPFile` on 3.12--3.15, `PyShark` on 3.14/3.15) installed by raw `pip install` to bypass the marker and assert the engine's own `PYTHON_CEILING` decline; and **4** `not-installable` cells (`PyPCAP` on 3.12--3.15) where a genuine C-extension build failure is logged and the test step skipped. The install step never passes silently in either direction. 3.15 is `continue-on-error` throughout. Job names change shape (`Engines Python X` becomes `Engines Python X (engine)`), so the branch-protection ruleset's required-check list was updated in the same window, lest a required check that stopped existing block every merge ([#849](https://github.com/JarryShaw/PyPCAPKit/pull/849)). +- Ruleset `23497679`'s `required_status_checks` names 22 exact check contexts, and GitHub rulesets match literally with no wildcard, so the list needs hand-editing on every matrix change. Five of the 22, the old `Engines Python ` names, are dead since [#849](https://github.com/JarryShaw/PyPCAPKit/pull/849); the five `Compat Python 3.10`-`3.14` names stay live, emitted by `python-compatibility.yml`, and are untouched. Adds one job, `required-checks` (context `Required checks passed`), depending on `[test, integration, engine-tests, pypcap-parity]` with `if: always() && inputs.gate-only != true` and an explicit per-dependency check that fails loudly, naming the job, on anything but `success`: a bare `needs:` list would leave it *skipped* rather than failed when a dependency failed, and GitHub's docs list a skipped required check as passing. That one context is meant to replace 17 of the ruleset's 22 entries (`test`, `integration`, `Engines`, `pypcap-parity`) once the ruleset is updated separately, a repository setting this PR does not touch; the five `Compat` names are excluded from the replacement. One file, `.github/workflows/unit-tests.yml`, 167 lines added; no library or test code ([#856](https://github.com/JarryShaw/PyPCAPKit/pull/856)). #### Fixed - 45 places where a documentation page contradicted the code ([#413](https://github.com/JarryShaw/PyPCAPKit/pull/413)), ambiguous cross-references and five autodoc signature failures ([#416](https://github.com/JarryShaw/PyPCAPKit/pull/416)), and `Extractor`'s documented exception plus 40 phantom or stale `Args:` labels ([#501](https://github.com/JarryShaw/PyPCAPKit/pull/501)). -- both halves of what the `Changelog drift` gate told an author, in `util/changelog_md.py`. `ResidualMarkupError` named the entry file but numbered its lines against the *converted* body, which rule 6 joins onto one line per block: measured on this entry, the body is 55 lines against the file's 580, so a reported number could not reach most of the file at all, and four roles written on two source lines were all reported as `line 46`. The conversion now carries a source map -- which line of the entry each stretch of output came from -- so every complaint cites a line of the file the message names, one complaint per construct rather than one per joined line, in the entry's own order ([#588](https://github.com/JarryShaw/PyPCAPKit/issues/588)). Rule 2 separately accepted only the bare-number spelling of Sphinx's `:rfc:` role, so a citation of [RFC 6554 Section 3](https://datatracker.ietf.org/doc/html/rfc6554#section-3) fell past the rule that exists for it and was reported as a role the rules do not cover; both spellings now convert, with the anchor carried into the link target and the link text taken from Sphinx's own so that the Markdown and the rendered history say the same thing about the same page ([#592](https://github.com/JarryShaw/PyPCAPKit/issues/592)). Regenerating `CHANGELOG.md` is byte-identical over all 37 committed entries, so the fix changes what the gate *says* and nothing about what it emits. -- `examples/generators/options.py`'s `TCP_BASE` spelled three header fields with names `TCP.make` does not declare, so every option fixture in `examples/captures/options-tcp.pcap` was built from values the generator had not asked for. `make` ends in `**kwargs` and reads nothing out of it, so an undeclared keyword is accepted and discarded with no warning and no `TypeError`, which is why this survived unnoticed. `'seq': 1` was `seq_no`, and all 25 captured frames therefore carried sequence number 0; `'urgent_pointer'` was `urgent`, whose requested value and unused default both happened to be 0; and `'ack_flag': False` was `ack`, while `'ack': 0` bound to `ack` itself -- the acknowledgement *flag* rather than the number, which is `ack_no` -- so that pair was the wrong way round in both directions. Each is renamed to the parameter `make` declares, keeping the values the table always read as. Measured: `TCP(**TCP_BASE).info.seq` was 0 against a declared 1 and is now 1, and `options-tcp.pcap` changes in exactly 25 bytes -- the low octet of each frame's sequence number -- with no case status and no `EXPECTED_FAILURES` entry moving. The module's six other base mappings were audited the same way: five pass nothing their `make` does not declare, and HIP's `extension` is declared by `HIP.read` instead, which is this library's documented way of forwarding a read-path keyword through `**kwargs` rather than a sixth instance of the defect ([#602](https://github.com/JarryShaw/PyPCAPKit/issues/602)). -- `util/bump_version.py` left `CITATION.cff` naming the previous release. Nothing else in the repository maintains that file -- no workflow, hook or packaging file mentions it -- so every bump since it landed in [#615](https://github.com/JarryShaw/PyPCAPKit/pull/615) would have stranded the `version` and `date-released` it renders as GitHub's "Cite this repository" button and that citation managers, Zenodo and dependency inventories read directly. Both fields now move with `__version__`. They have the same standing, since the file's own header says both describe the newest *published* release, and moving only one would assert that 1.5.0b5 was released on the day 1.5.0b4 was; the date is taken in UTC, because seven of the thirty most recent bumps were made late evening in US-Eastern where a local date is a day behind the publish it describes. That the two are the same day at all is measured rather than assumed: the bump is what triggers `create-release.yml`, the median gap to the PyPI upload is three minutes, and the UTC calendar dates agree 30 times out of 30. The rewrite is line-oriented, so the comment header, key ordering and each field's existing quoting survive -- `cff-version` and a `references` entry's own `version` are anchored out at column zero -- and the result is checked with `cffconvert --validate`. An absent file is reported on stderr and skipped rather than failing the vendor cron before its `git commit`, which would discard the whole registry crawl for the sake of a documentation file; a file present with no `version` field raises instead, before anything is written, because rewriting nothing while reporting success is the staleness this fixes. Two things came with it. The script gains a `main()` guard, having previously run the entire bump at import, which is why it had no testable surface; and the `import pcapkit` fallback in its version reader, which returned `"1.5.0b4'\n"` -- closing quote and newline included, which `packaging` rejects -- is fixed, a path that had never worked and went unnoticed because the only caller installs the package first. A new gate asserts the committed file still names the packaged version, covering the version changes made by hand, which never run this script at all -- 40 of the 159 commits that have moved `__version__` on `main`, a quarter over the project's life and 11 of the most recent 25 ([#625](https://github.com/JarryShaw/PyPCAPKit/pull/625)). -- `LICENSE`'s copyright notice began the term at 2018, a year after the work it covers. The repository's first commit is `c57f7d0b7` "Initial commit", dated 2017-11-07, so the notice understated the term and contradicted the only other copyright site in the tree: `docs/source/conf.py` computes its Sphinx footer as `f'2017-{datetime.date.today().year}, Jarry Shaw'` and has said 2017 for as long as it has existed. The two now agree on where the term starts. The wrong start year survived every maintenance pass the line has had, because each looked only at the other end of the range: `bc836cfa2` (2020-05-31) introduced `2018-2020` when the BSD-3-Clause text replaced MPL 2.0, and the range was then bumped by hand three times, to `2018-2022`, `2018-2023` and `2018-2026` ([#615](https://github.com/JarryShaw/PyPCAPKit/pull/615)), each bump correcting the end year and copying `2018` forward untouched. 2017 is being **restored** rather than newly asserted. The project's first `LICENSE` -- MIT, at `c57f7d0b7`, the initial commit -- already read `Copyright (c) 2017 Jarry Shaw` on line 3, and the GPL v3 era that followed carried `Copyright (C) 2017 Jarry Shaw` in its filled-in "how to apply" appendix. 2017 was lost on 2018-12-08, when `4f3e00d5f` relicensed to Apache 2.0 under `Copyright 2018 Jarry Shaw` -- a single year, correct for the year it was written -- and `1c69341dc` replaced that with MPL 2.0 text the same day, which carried no author notice at all. `bc836cfa2` then picked `2018` back up as a range start in 2020, by which point it was two years stale and no longer described anything. The end year is **dropped** rather than automated, which is the other half of the change: the notice ships inside the sdist and the wheel, so its text is fixed at build time in every copy already downloaded and cannot be computed the way the docs footer is; copyright subsists from creation whether or not a notice names the current year; and BSD-3-Clause's canonical form is `Copyright (c) `, singular. So a range here is six years of hand maintenance on one line buying nothing, and no workflow, script hook or other automation is added in its place -- that was considered and rejected, since automating a value that need not be current is worse than not carrying it. `conf.py` is deliberately untouched, a self-maintaining docs footer showing a range being both conventional and correct. The licence body is unaltered: the whole-file diff is one hunk at line 3, and the remaining 28 lines carry the canonical BSD-3-Clause wording verbatim. Three things distinguish the file from SPDX's bare `licenseText`, and all three predate this change -- the `BSD 3-Clause License` title line, which the choosealicense.com template carries and SPDX omits; the `*` bullets in place of `1.`/`2.`/`3.`; and the extra `All rights reserved.` line -- and the `DAMAGE.` that [#615](https://github.com/JarryShaw/PyPCAPKit/pull/615) repaired from a stray `DAMAGE.s` is still clean ([#630](https://github.com/JarryShaw/PyPCAPKit/pull/630)). -- `CITATION.cff` shipped in no source distribution. `MANIFEST.in` carries an `include` line each for `README.md`, `LICENSE` and `CHANGELOG.md` but had none for the citation file, and its two `global-include` patterns are `*.rst` and `*.py`, neither of which can match a `.cff` -- so the file sat in the repository and in nothing published. Nor was there a default to fall back on. Removing all three of those `include` lines and rebuilding shows `README.md` and `LICENSE` shipping anyway, setuptools adding the latter from `license_files` and recording it as `License-File: LICENSE` in `PKG-INFO`, while `CHANGELOG.md` vanishes -- so of the three only `CHANGELOG.md` is load-bearing, and a citation file, which no packaging default covers at all, is in the same position. The gap is invisible from the web UI, because GitHub renders the "Cite this repository" button from the repository itself; citation managers, Zenodo and dependency inventories read the published artifact, which is exactly the surface that was missing it, so the machine-readable half of the attribution stopped travelling with the code at the one point nobody can see the repository. [#615](https://github.com/JarryShaw/PyPCAPKit/pull/615) flagged the omission when it added the file and [#619](https://github.com/JarryShaw/PyPCAPKit/pull/619) did not address it. It matters now because [#625](https://github.com/JarryShaw/PyPCAPKit/pull/625) has just taught `util/bump_version.py` to keep the file's `version` and `date-released` in step with the bump, and a release exercising that path would otherwise publish an sdist omitting the very artefact under test. It also lets `RepositoryCitationTests` in `tests/project/test_bump_version.py` -- the gate [#625](https://github.com/JarryShaw/PyPCAPKit/pull/625) added for the hand-authored bumps that never run the script -- execute against an unpacked sdist, where today it skips itself with "CITATION.cff is not shipped in the source distribution". Measured both ways with `python -m build --sdist`: `tar tzf` found no `CITATION.cff` before and `pypcapkit-1.5.0b4/CITATION.cff` after, the two archive listings differ by that one added entry and nothing else -- 860 against 861 -- the shipped copy is byte-identical to the repository's, and `twine check --strict` reports `PASSED` on both archives ([#631](https://github.com/JarryShaw/PyPCAPKit/pull/631)). -- two inaccurate claims in the [#630](https://github.com/JarryShaw/PyPCAPKit/pull/630) entry above, caught by review after [#630](https://github.com/JarryShaw/PyPCAPKit/pull/630) had already merged. Both were about the history of `LICENSE` rather than about the change itself, and `LICENSE` is untouched here: 2017 was and remains the right answer, so this is a changelog-prose correction only. The entry asserted that the file "until then was stock MPL text carrying no author notice at all, so 2018 is the first start year the project ever asserted and it was already a year late when it was written". Every clause of it is wrong as a description of the era, though each was true of the single blob `bc836cfa2` happened to replace, which is where the mistake came from. The pre-BSD file was MIT, then GPL v3, then Apache 2.0, and only then MPL 2.0 -- four licences before BSD, not the one implied. The MIT and GPL texts both carried an author notice naming **2017**, the MIT one on line 3 of the initial commit. And `2018` first appeared in the Apache notice of 2018-12-08 as a single year, current for the year it was written rather than late. The claim also contradicted the entry's own opening sentence, which dates the first commit to 2017-11-07. What the history actually supports is a stronger argument for the change than the one that was written, which is why this is corrected rather than deleted: 2017 is the year the project asserted from its first commit, and [#630](https://github.com/JarryShaw/PyPCAPKit/pull/630) restores it rather than asserting it anew. The `BSD 3-Clause License` title line is reattributed too, from the opensource.org template -- whose licence text begins at the copyright line -- to choosealicense.com's, whose first line is literally that title, and which also carries the `Copyright (c) [year], [fullname]` comma this file uses. The error came from generalising the pre-BSD era off two blobs that turned out to be the same file, `bc836cfa2^:LICENSE` and `1c69341dc:LICENSE`; enumerating all 14 commits that ever touched `LICENSE` is what caught it ([#638](https://github.com/JarryShaw/PyPCAPKit/pull/638)). -- the release workflow published to PyPI and to Anaconda with nothing a human had to approve. `create-release.yml`'s `pypi` job carried its `environment: release` commented out while the `id-token: write` from the same upstream snippet had been re-added live below it, so the block read as disabled as a unit when only its gate still was; the `conda` job had no `environment:` at all, not even a commented one. Neither needed a tag either: the workflow also fires on the completion of `Vendor Update`, so a scheduled registry crawl that bumped the version was enough to publish to two public indexes, at a median three minutes from bump to upload. Four jobs reach outside the run and all four are now gated -- `github` on `github-release` for the GitHub Release and the `v*` tag it creates, `tag` on `conda-tag` for the commit it pushes to `main`, `pypi` on `pypi`, and `conda` on `anaconda`. One environment per credential rather than one shared `release`, because approval is granted to an environment and not to a job, so a single `release` approved once would release both indexes together. PyPI is the one that has to be answerable alone: a version number it has accepted cannot be reused, and `skip-existing: true` makes the re-upload *succeed* having published nothing, so an unintended publish permanently consumes the number the intended release wanted. The workflow half is only half the fix -- an `environment:` naming an environment that does not exist is created implicitly with no protection rules and the job proceeds unapproved, so this is inert until each of the four exists in repository settings with a required reviewer on it, and each one's "Deployment branches and tags" has to stay unrestricted or the `tags: v*` trigger fails outright instead of pausing. `github-pages` is the standing example of that failure in this repository, carrying no protection rule since 2021. The new test asserts the general rule rather than the current file -- no job running a publishing action or a `git push` may omit an `environment:` -- so a publishing job added later without a gate fails rather than ships ([#641](https://github.com/JarryShaw/PyPCAPKit/issues/641)). -- three test comments stated an exact count of committed captures under `examples/captures/`, a number that drifts every time that directory's tracked set changes and in one case was already wrong when written. `tests/_tiers.py` cited "six," correct at the time; `tests/test_tier_guard.py` said "the moment a seventh capture is committed" -- the same count, as an ordinal rather than the word "six"; `tests/integration/_helpers.py` said "four," which undercounted the tracked set the day it was written. All three now describe the invariant instead of a number that has to be kept in sync with it by hand: `_tiers.py` reads "the moment somebody commits another capture... or stops committing one," `_helpers.py` reads "...are fixtures -- some of them committed --...," and `test_tier_guard.py` drops its "seventh capture" phrasing the same way. Prose-only, verified with `coverage.parser.PythonParser` that no executable statement moved ([#700](https://github.com/JarryShaw/PyPCAPKit/issues/700)). -- `test_every_tracked_name_exists_and_matches_git` in `tests/test_tier_guard.py` checked neither the git index nor the filesystem despite its name: it asserted only that the tracked-capture list was non-empty and that no name contained a `/`, so a hardcoded list would pass it whether or not it matched reality. Sibling `test_capture_suggestions_are_captures` had the same gap. Found while cross-reviewing [#703](https://github.com/JarryShaw/PyPCAPKit/pull/703)'s first commit, itself prose-only and out of scope for this; the fix rides in as that PR's second and third commits, and [#703](https://github.com/JarryShaw/PyPCAPKit/pull/703)'s own description names both [#700](https://github.com/JarryShaw/PyPCAPKit/issues/700) and [#708](https://github.com/JarryShaw/PyPCAPKit/issues/708). `test_every_tracked_name_exists_and_matches_git` now re-derives the expected set independently -- shelling out to `git ls-files -z` under `examples/captures/` rather than calling back into `_tiers.py` -- and asserts equality against it rather than against itself, plus a per-name `Path.is_file()` check the old version never made. Its sibling `test_capture_suggestions_are_captures` gets a narrower fix: it asserts equality against `committed_capture_names()`'s own filter re-implemented inline over `_tiers.committed_captures()`, which catches a wrong filter but, unlike its sibling, still trusts `committed_captures()` for the tracked set itself rather than re-deriving it from git. A cross-review of the first version of this fix found it wrong twice over: a stale docstring in `_tiers.py` still claimed `test_tier_guard.py` "only stats `in.pcap`," falsified by the new per-name loop; and the new `assertEqual` on `test_capture_suggestions_are_captures` was vacuous on an empty tracked set (`() == ()` passes trivially), having dropped the non-emptiness guard the other test kept. Both are corrected in a follow-up commit. Even after the fix, a hardcoded literal that happens to match today's tracked names still passes -- the test detects a set that is wrong *at the moment it runs*, not hardcoding as a practice, which turns a permanent blind spot into a tripwire that fires on the next change to the tracked set ([#708](https://github.com/JarryShaw/PyPCAPKit/issues/708)). +- both halves of what the `Changelog drift` gate told an author, in `util/changelog_md.py`. `ResidualMarkupError` named the entry file but numbered its lines against the *converted* body, which rule 6 joins onto one line per block (55 lines against the file's 580 on this entry), so a reported number could not reach most of the file, and four roles on two source lines were all reported as `line 46`. The conversion now carries a source map, so every complaint cites a line of the file the message names, one per construct, in the entry's order ([#588](https://github.com/JarryShaw/PyPCAPKit/issues/588)). Rule 2 accepted only the bare-number spelling of Sphinx's `:rfc:` role, so [RFC 6554 Section 3](https://datatracker.ietf.org/doc/html/rfc6554#section-3) fell past it and was reported as a role the rules do not cover; both spellings now convert, the anchor carried into the link target and the link text taken from Sphinx's own ([#592](https://github.com/JarryShaw/PyPCAPKit/issues/592)). Regenerating `CHANGELOG.md` is byte-identical over all 37 committed entries: the fix changes what the gate *says*, not what it emits. +- `examples/generators/options.py`'s `TCP_BASE` spelled three header fields with names `TCP.make` does not declare, so every option fixture in `examples/captures/options-tcp.pcap` was built from values the generator had not asked for. `make` ends in `**kwargs` and reads nothing out of it, so an undeclared keyword is accepted and discarded with no warning. `'seq': 1` was `seq_no`, so all 25 captured frames carried sequence number 0; `'urgent_pointer'` was `urgent` (requested value and default both 0); and `'ack_flag': False` was `ack` while `'ack': 0` bound to `ack` itself, the acknowledgement *flag* rather than the number (`ack_no`), so that pair was wrong both ways. Each is renamed to the parameter `make` declares, keeping the intended values. `TCP(**TCP_BASE).info.seq` was 0 against a declared 1 and is now 1, and `options-tcp.pcap` changes in exactly 25 bytes (the low octet of each frame's sequence number), with no case status or `EXPECTED_FAILURES` entry moving. The module's six other base mappings were audited: five pass nothing their `make` does not declare, and HIP's `extension` is declared by `HIP.read`, this library's documented way of forwarding a read-path keyword, not a sixth instance ([#602](https://github.com/JarryShaw/PyPCAPKit/issues/602)). +- `util/bump_version.py` left `CITATION.cff` naming the previous release. Nothing else maintains that file, so every bump since [#615](https://github.com/JarryShaw/PyPCAPKit/pull/615) would have stranded the `version` and `date-released` that GitHub's "Cite this repository" button, citation managers, Zenodo and dependency inventories read. Both fields now move with `__version__`, together, since the file's header says both describe the newest *published* release and moving only one would assert that 1.5.0b5 was released on the day 1.5.0b4 was. The date is UTC, because seven of the thirty most recent bumps were made late evening US-Eastern, where a local date is a day behind the publish; the bump triggers `create-release.yml`, the median gap to the PyPI upload is three minutes, and the UTC dates agree 30 times of 30. The rewrite is line-oriented, so the comment header, key ordering and each field's quoting survive (`cff-version` and a `references` entry's own `version` are anchored out at column zero), and the result is checked with `cffconvert --validate`. An absent file is reported on stderr and skipped rather than failing the vendor cron before its `git commit`, which would discard a whole registry crawl for a documentation file; a file with no `version` field raises before anything is written, since reporting success while rewriting nothing is the staleness being fixed. The script gains a `main()` guard (it ran the whole bump at import, hence no testable surface), and the `import pcapkit` fallback in its version reader, which returned `"1.5.0b4'\n"` (closing quote and newline, which `packaging` rejects), is fixed; it never worked and went unnoticed because the only caller installs the package first. A new gate asserts the committed file still names the packaged version, covering hand-made version changes that never run this script (40 of the 159 commits that moved `__version__` on `main`, 11 of the latest 25) ([#625](https://github.com/JarryShaw/PyPCAPKit/pull/625)). +- `LICENSE`'s copyright notice began the term at 2018, a year after the work it covers. The repository's first commit is `c57f7d0b7`, dated 2017-11-07, so the notice understated the term and contradicted `docs/source/conf.py`, whose Sphinx footer is `f'2017-{datetime.date.today().year}, Jarry Shaw'`. The wrong start year survived every maintenance pass because each looked only at the other end of the range: `bc836cfa2` (2020-05-31) introduced `2018-2020` when the BSD-3-Clause text replaced MPL 2.0, and hand bumps to `2018-2022`, `2018-2023` and `2018-2026` ([#615](https://github.com/JarryShaw/PyPCAPKit/pull/615)) each copied `2018` forward. 2017 is **restored** rather than newly asserted: the initial MIT `LICENSE` read `Copyright (c) 2017 Jarry Shaw`, and the GPL v3 era carried `Copyright (C) 2017 Jarry Shaw` in its appendix; it was lost on 2018-12-08, when `4f3e00d5f` relicensed to Apache 2.0 under `Copyright 2018 Jarry Shaw` and `1c69341dc` replaced that with MPL 2.0 text with no author notice. The end year is **dropped** rather than automated: the notice ships inside the sdist and wheel, so its text is fixed at build time in every copy already downloaded and cannot be computed like the docs footer; copyright subsists from creation whether or not a notice names the current year; and BSD-3-Clause's canonical form is a singular `Copyright (c) `. A range was six years of hand maintenance buying nothing, and automation was considered and rejected, since automating a value that need not be current is worse than not carrying it. `conf.py` is untouched (a self-maintaining footer showing a range is conventional and correct). The licence body is unaltered: one hunk at line 3, and the `DAMAGE.` that [#615](https://github.com/JarryShaw/PyPCAPKit/pull/615) repaired from `DAMAGE.s` is still clean ([#630](https://github.com/JarryShaw/PyPCAPKit/pull/630)). +- `CITATION.cff` shipped in no source distribution. `MANIFEST.in` has an `include` each for `README.md`, `LICENSE` and `CHANGELOG.md` but none for the citation file, and its two `global-include` patterns (`*.rst`, `*.py`) cannot match a `.cff`. Removing all three `include` lines and rebuilding shows `README.md` and `LICENSE` shipping anyway (setuptools adds the latter from `license_files`, recorded as `License-File: LICENSE` in `PKG-INFO`) while `CHANGELOG.md` vanishes, so only `CHANGELOG.md` is load-bearing, and a citation file, which no packaging default covers, is in the same position. The gap is invisible from the web UI, because GitHub renders "Cite this repository" from the repository; citation managers, Zenodo and dependency inventories read the published artifact, the surface that was missing it. [#615](https://github.com/JarryShaw/PyPCAPKit/pull/615) flagged the omission and [#619](https://github.com/JarryShaw/PyPCAPKit/pull/619) did not address it. It matters now because [#625](https://github.com/JarryShaw/PyPCAPKit/pull/625) taught `util/bump_version.py` to keep the file's `version` and `date-released` in step, and a release exercising that path would publish an sdist omitting the artefact under test. It also lets `RepositoryCitationTests` in `tests/project/test_bump_version.py` (the gate [#625](https://github.com/JarryShaw/PyPCAPKit/pull/625) added for hand-authored bumps) run against an unpacked sdist, where it skipped itself with "CITATION.cff is not shipped in the source distribution". `python -m build --sdist` gives 860 archive entries before and 861 after (`pypcapkit-1.5.0b4/CITATION.cff` the only difference), the shipped copy is byte-identical, and `twine check --strict` passes on both ([#631](https://github.com/JarryShaw/PyPCAPKit/pull/631)). +- two inaccurate claims in the [#630](https://github.com/JarryShaw/PyPCAPKit/pull/630) entry above, caught in review after [#630](https://github.com/JarryShaw/PyPCAPKit/pull/630) had merged; both concern the history of `LICENSE`, which is untouched here. 2017 was and remains the right answer, so this is a changelog-prose correction. The entry said the file "until then was stock MPL text carrying no author notice at all, so 2018 is the first start year the project ever asserted and it was already a year late when it was written". That was true of the single blob `bc836cfa2` replaced and wrong as a description of the era: the pre-BSD file was MIT, then GPL v3, then Apache 2.0, then MPL 2.0; the MIT and GPL texts both carried an author notice naming **2017**; and `2018` first appeared in the Apache notice of 2018-12-08 as a single year, current when written. The claim also contradicted the entry's own dating of the first commit to 2017-11-07. The history supports a stronger argument than the one written, which is why this is corrected rather than deleted: 2017 is the year asserted from the first commit, and [#630](https://github.com/JarryShaw/PyPCAPKit/pull/630) restores it. The `BSD 3-Clause License` title line is reattributed from opensource.org's template to choosealicense.com's, whose first line is that title and which carries the `Copyright (c) [year], [fullname]` comma this file uses. The error came from generalising the pre-BSD era off two blobs that were the same file (`bc836cfa2^:LICENSE` and `1c69341dc:LICENSE`); enumerating all 14 commits that touched `LICENSE` caught it ([#638](https://github.com/JarryShaw/PyPCAPKit/pull/638)). +- the release workflow published to PyPI and to Anaconda with nothing a human had to approve. `create-release.yml`'s `pypi` job had `environment: release` commented out while the `id-token: write` from the same upstream snippet was re-added live below it, so the block read as disabled as a unit when only its gate still was; the `conda` job had no `environment:` at all. Neither needed a tag: the workflow also fires on completion of `Vendor Update`, so a scheduled registry crawl that bumped the version was enough to publish to two public indexes, a median three minutes from bump to upload. Four jobs reach outside the run and all four are now gated: `github` on `github-release` (the GitHub Release and the `v*` tag), `tag` on `conda-tag` (the commit pushed to `main`), `pypi` on `pypi`, `conda` on `anaconda`. One environment per credential rather than a shared `release`, because approval is granted to an environment and not a job, so one `release` approved once would release both indexes together. PyPI has to be answerable alone: an accepted version number cannot be reused, and `skip-existing: true` makes the re-upload *succeed* having published nothing, so an unintended publish permanently consumes the number the intended release wanted. The workflow half is only half the fix: an `environment:` naming one that does not exist is created implicitly with no protection rules and the job proceeds unapproved, so this is inert until each of the four exists in repository settings with a required reviewer, and each one's "Deployment branches and tags" must stay unrestricted or the `tags: v*` trigger fails outright instead of pausing. `github-pages` is the standing example, carrying no protection rule since 2021. The new test asserts the general rule, that no job running a publishing action or a `git push` may omit an `environment:`, so a publishing job added later without a gate fails rather than ships ([#641](https://github.com/JarryShaw/PyPCAPKit/issues/641)). +- three test comments stated an exact count of committed captures under `examples/captures/`, a number that drifts whenever the tracked set changes and in one case was wrong when written: `tests/_tiers.py` said "six," `tests/test_tier_guard.py` "the moment a seventh capture is committed," and `tests/integration/_helpers.py` "four," which undercounted on the day it was written. All three now describe the invariant (`_tiers.py`: "the moment somebody commits another capture... or stops committing one"; `_helpers.py`: "...are fixtures -- some of them committed --..."; `test_tier_guard.py` drops its ordinal). Prose-only, verified with `coverage.parser.PythonParser` that no executable statement moved ([#700](https://github.com/JarryShaw/PyPCAPKit/issues/700)). +- `test_every_tracked_name_exists_and_matches_git` in `tests/test_tier_guard.py` checked neither the git index nor the filesystem despite its name: it asserted only that the tracked-capture list was non-empty and no name contained a `/`, so a hardcoded list passed whether or not it matched reality. Sibling `test_capture_suggestions_are_captures` had the same gap. Found while cross-reviewing [#703](https://github.com/JarryShaw/PyPCAPKit/pull/703)'s first commit (prose-only, so out of scope there); the fix rides in as that PR's second and third commits, and [#703](https://github.com/JarryShaw/PyPCAPKit/pull/703)'s description names both [#700](https://github.com/JarryShaw/PyPCAPKit/issues/700) and [#708](https://github.com/JarryShaw/PyPCAPKit/issues/708). The first test now re-derives the expected set independently, shelling out to `git ls-files -z` under `examples/captures/` rather than calling back into `_tiers.py`, asserts equality against it, and adds a per-name `Path.is_file()` check. Its sibling gets a narrower fix: equality against `committed_capture_names()`'s filter re-implemented inline over `_tiers.committed_captures()`, which catches a wrong filter but still trusts `committed_captures()` for the tracked set. A cross-review of the first version found it wrong twice: a stale `_tiers.py` docstring still claimed `test_tier_guard.py` "only stats `in.pcap`," falsified by the new per-name loop; and the new `assertEqual` was vacuous on an empty tracked set (`() == ()`), having dropped the non-emptiness guard the other test kept. Both are corrected in a follow-up commit. Even now a hardcoded literal that happens to match today's names passes: the test detects a set that is wrong *at the moment it runs*, not hardcoding as a practice, turning a permanent blind spot into a tripwire that fires on the next change to the tracked set ([#708](https://github.com/JarryShaw/PyPCAPKit/issues/708)). --- diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index ef8bce567..27d13b53a 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -15,9 +15,9 @@ resolved only by ``pip install --pre``. ``1.5.0b1`` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because ``twine check`` found a Sphinx-only ``:mod:`` role in ``README.rst``, which ``pyproject.toml`` declares as the dynamic long description. ``1.5.0b2`` -is what reshipped it -- the release workflow is version-driven, so an existing -version cannot republish -- and ``1.5.0b3`` followed the CI change that stops a -TestPyPI outage from costing a release its wheels (:pr:`497`, :pr:`498`). +reshipped it -- an existing version cannot republish -- and ``1.5.0b3`` followed +the CI change that stops a TestPyPI outage from costing a release its wheels +(:pr:`497`, :pr:`498`). pcapkit.const ------------- @@ -25,476 +25,298 @@ pcapkit.const Changed ~~~~~~~ -* **a breaking change to** ``pcapkit.const.reg.apptype``. - ``AppType`` becomes a memberless base over one registry per transport - protocol -- ``TCP``, ``UDP``, ``SCTP`` and ``DCCP`` -- because ``aenum`` - refuses to subclass an enumeration that has members. - ``isinstance(TCP.http, AppType)`` still holds and +* **a breaking change to** ``pcapkit.const.reg.apptype``. ``AppType`` becomes a + memberless base over one registry per transport protocol -- ``TCP``, ``UDP``, + ``SCTP`` and ``DCCP`` -- because ``aenum`` refuses to subclass an enumeration that + has members. ``isinstance(TCP.http, AppType)`` still holds and ``AppType.get(port, proto=...)`` proxies to the registry, so nothing under - ``pcapkit.protocols`` changes. **The 1,004 rows - IANA assigns no port, and the 704 it assigns no transport protocol, stop - being members**; they become list-table docstrings on the registry that owns - them, which autodoc publishes. ``__members_proto__`` is now a per-registry - ``MultiDict`` keyed on port, so all three services IANA registers on port 80 - survive and ``extend_enum`` on an occupied port appends rather than - displacing what a lookup returned. ``get()`` answers with the canonical - service, curated from ``/etc/services`` because IANA names no precedence - among them, and ``get_all()`` reaches the aliases; ``get()``'s dead ``str`` - branch is removed rather than carried over (:pr:`754`). -* the six ``raise ValueError(...)`` calls still using ``%`` - formatting in the generated ``pcapkit.const.reg.apptype.apptype``, left that - way by :issue:`759` so its own diff over a 12,391-member file stayed reviewable, are - f-strings, matching the other 605 tree-wide (:issue:`792`). The remainder followed: - the three other ``_missing_`` guards still raising with ``%`` -- in - ``pcapkit.vendor.tcp.flags``, ``pcapkit.vendor.ftp.command`` and - ``pcapkit.vendor.http.method`` -- ``AppType``'s ``__new__``, ``__repr__`` - and ``__str__``, and the ``extend_enum`` mint fallbacks in ``get()`` and in - the span-handling tail. ``__new__``'s format sets every real member's - underlying ``StrEnum`` value, so that one is proven byte-identical member by - member across all 12,391 -- TCP 6,147, UDP 6,143, SCTP 91, DCCP 10 -- - rather than spot-checked. The module-level ``consider-using-f-string`` - disable drops from ``tcp.flags`` and ``reg.apptype.apptype``, which carry no - ``%``-formatted code left, and stays on ``ftp.command`` and ``http.method``, - whose ``__repr__`` still renders with one. Both rounds are edited in the - vendor templates, braces doubled since ``BASE`` and ``LINE`` are f-strings - emitting f-strings, and applied to the generated const files by hand rather - than by a crawl, which would fetch IANA's live registry and rewrite - unrelated member rows (:issue:`798`). -* the three ``__repr__`` methods :issue:`798` left on ``%`` formatting - -- ``pcapkit/const/ftp/command.py:40,119`` and - ``pcapkit/const/http/method.py:42`` -- are f-strings now, so the - ``consider-using-f-string`` disable drops from both files and from both - ``pcapkit/vendor/`` templates; each generator's own ``%``-formatted - ``wrap_comment`` argument went with them, dropping their two inline - disables. The other bespoke templates in these directories -- - ``{const,vendor}/ftp/return_code.py``, - ``{const,vendor}/http/status_code.py`` and ``vendor/http/frame.py`` -- and - the shared ``vendor/default.py`` one still carry the disable, so :issue:`804`'s - claim that nothing in ``pcapkit/{const,vendor}/{ftp,http}/`` needs it holds - for this pair only. Generated, so both sides changed: rendering each - ``LINE`` template with the real arguments reproduces the committed const - module byte-for-byte, and the render-against-committed diff is 6 changed - lines for ``ftp/command`` and 4 for ``http/method``, the intended pairs and - nothing else. ``test_the_disable_drops_only_where_nothing_else_needs_percent_formatting`` - asserted the opposite for this pair -- that the disable was retained -- so - it is flipped rather than added to; ``tests/const/test_const_enum_builtin_parity.py`` - goes 22 to 27 tests / 701 subtests, with 8 subtests across the 3 converted - methods failing against the unfixed pair. ``repr()`` is asserted member by - member against the old ``%`` expression's own output for all of - ``Command``, ``FEATCode`` and ``Method``, including a ``desc`` of ``None`` - (:issue:`804`). -* ``test_const_enum_no_mint.py``'s ``is_manufactured()`` - exemption for :pr:`878`'s (above) 53 preserved hex-suffixed names moves from a - per-*file* exemption to a per-*name-shape* one. Keying it on - ``HEX_SUFFIXED_NAME_EXEMPT_PATHS`` made the exemption shape-blind inside - those two files: rewriting a branch to ``'Xyplex_%d' % value`` -- exactly the - value-derived shape the predicate exists to flag -- still left the sweep - green, since ``exempt_hits`` stayed at 53 regardless of what the name - actually was. The check now does an AST structural comparison instead: a - ``%``-``BinOp`` whose left side is a string constant ending in ``'_0x%s'`` - and whose right side is structurally identical to a parsed - ``hex(value)[2:].upper().zfill(4)`` template, matched via ``ast.dump()`` - equality rather than a source-text or per-prefix pattern -- there are 53 - distinct prefixes, ``Xyplex_`` through ``Registered by Xerox_``, and the AST - match is immune to quote-style or whitespace drift in generated source. - ``HEX_SUFFIXED_NAME_EXEMPT_PATHS`` is dropped entirely. Verified - independently: mutating ``pcapkit/const/reg/ethertype.py``'s - ``Xyplex_0x%s``-style branch to the bare value-derived form now fails with - ``52 != 53: expected exactly 53 hex-suffixed-name exemptions to fire``, where - it used to pass; the unmutated file stays green, and a value-derived name in - an unrelated file (mutating ``pcapkit/const/arp/hardware.py`` to - ``'Unassigned_%d' % value``) is still flagged, naming that path. The exact - ``exempt_hits == 53`` bound is kept, so adding or removing a branch still - forces a human look. One file changed, - ``tests/const/test_const_enum_no_mint.py``; nothing under ``pcapkit/`` or - ``docs/``. Closes :issue:`879` (:pr:`881`). + ``pcapkit.protocols`` changes. **The 1,004 rows IANA assigns no port, and the 704 it + assigns no transport protocol, stop being members**; they become list-table + docstrings on the registry that owns them. ``__members_proto__`` is now a + per-registry ``MultiDict`` keyed on port, so all three services IANA registers on + port 80 survive. ``get()`` answers with the canonical service, curated from + ``/etc/services`` because IANA names no precedence, and ``get_all()`` reaches the + aliases (:pr:`754`). +* the six ``%``-formatted ``raise ValueError(...)`` calls in the generated + ``pcapkit.const.reg.apptype.apptype``, left by :issue:`759` so its diff over a + 12,391-member file stayed reviewable, are f-strings, matching the other 605 + tree-wide (:issue:`792`). The remainder followed: three ``_missing_`` guards + (``pcapkit.vendor.tcp.flags``, ``.ftp.command``, ``.http.method``), ``AppType``'s + ``__new__``/``__repr__``/``__str__``, and the ``extend_enum`` fallbacks in ``get()`` + and in the span-handling tail. The ``consider-using-f-string`` disable drops from + ``tcp.flags`` and ``reg.apptype.apptype`` and stays on ``ftp.command`` and + ``http.method``, whose ``__repr__`` still used ``%``. Edited in the vendor templates + and applied to the generated const files by hand, since a crawl would fetch IANA's + live registry and rewrite unrelated rows (:issue:`798`). +* the three ``__repr__`` methods :issue:`798` left on ``%`` + (``pcapkit/const/ftp/command.py:40,119``, ``pcapkit/const/http/method.py:42``) are + f-strings, so the disable drops from both files and both ``pcapkit/vendor/`` + templates. The other bespoke templates (``{const,vendor}/ftp/return_code.py``, + ``{const,vendor}/http/status_code.py``, ``vendor/http/frame.py``) and + ``vendor/default.py`` still carry it, so :issue:`804`'s claim that nothing in + ``pcapkit/{const,vendor}/{ftp,http}/`` needs it holds for this pair only. + ``test_the_disable_drops_only_where_nothing_else_needs_percent_formatting`` asserted + the opposite for this pair and is flipped; ``repr()`` is asserted member by member + against the old ``%`` output (:issue:`804`). +* ``test_const_enum_no_mint.py``'s ``is_manufactured()`` exemption for :pr:`878`'s + (above) 53 preserved hex-suffixed names moves from a per-*file* to a + per-*name-shape* one. The per-file ``HEX_SUFFIXED_NAME_EXEMPT_PATHS`` was + shape-blind: rewriting a branch to ``'Xyplex_%d' % value`` -- exactly the + value-derived shape the predicate exists to flag -- left the sweep green. The check + now matches an AST ``%``-``BinOp`` whose left side is a string ending in ``'_0x%s'`` + and whose right side equals a parsed ``hex(value)[2:].upper().zfill(4)`` (via + ``ast.dump()``, immune to quote-style or whitespace drift). The exact + ``exempt_hits == 53`` bound stays, so adding or removing a branch still forces a + human look. Only ``tests/const/test_const_enum_no_mint.py`` changed. Closes + :issue:`879` (:pr:`881`). Fixed ~~~~~ -* constant lookups that rejected a value the registry defines. - ``RouterAlert(0)`` is the only value :rfc:`2113` defines and the one IGMP, - RSVP and MLD actually send, and it was discarded because the vendor crawler - skipped a header row IANA's CSV does not have; IPX ``Socket(0)`` is that - protocol's own default, so ``bytes(IPX(...))`` crashed on its own defaults; - and two FTP ``_missing_`` overrides were plain methods rather than - classmethods, so every unregistered value raised ``TypeError`` instead of - extending the enumeration (:issue:`492`, :pr:`503`). +* constant lookups that rejected a value the registry defines. ``RouterAlert(0)`` is + the only value :rfc:`2113` defines and the one IGMP, RSVP and MLD actually send, and + it was discarded because the vendor crawler skipped a header row IANA's CSV does not + have; IPX ``Socket(0)`` is that protocol's own default, so ``bytes(IPX(...))`` + crashed on its own defaults; and two FTP ``_missing_`` overrides were plain methods + rather than classmethods, so every unregistered value raised ``TypeError`` instead + of extending the enumeration (:issue:`492`, :pr:`503`). * the string-keyed ``get()`` in ``pcapkit.const.ftp.command`` and ``pcapkit.const.http.method`` tested membership with the raw key but registered ``key.upper()``, so the *first* lowercase or mixed-case token - raised ``TypeError: 'RETR' already in use`` rather than resolving. Reachable - from wire data for FTP: ``pcapkit.protocols.application.ftp`` compiles its - request pattern with ``re.I`` and passes the match verbatim, and - :rfc:`959#section-5` makes FTP commands case-insensitive -- "Upper and lower - case alphabetic characters are to be treated identically", listing - ``RETR Retr retr ReTr rETr`` as the same command -- so ``retr file.txt`` was a - valid request this library could not parse. Both ``get()`` and ``_missing_`` - now look the key up under the same canonical upper-case name they register it - under, so every casing resolves to the one member that already exists instead - of colliding with it. Resolving rather than registering a second member - matters beyond not crashing -- a duplicate ``GET`` would carry neither the - ``safe`` nor the ``idempotent`` attribute of the real one (:issue:`582`, :issue:`583`). -* ``get()``'s documented ``default`` was ignored on the integer - path throughout the generated ``pcapkit.const`` tree, because ``get`` - delegated the lookup to the enum call and ``_missing_`` has no access to the - caller's ``default`` -- so ``Hardware.get(99999, 0)`` raised - ``ValueError: 99999 is not a valid Hardware`` instead of returning the - fallback it was handed. The integer path now consults ``default`` before - letting the lookup error escape. ``-1``, the placeholder the generated - signature already carried, was what separated "no default was supplied" - from "a default was supplied and should be used" at the time, so a caller - that asked for no fallback got the error rather than a silent substitution. - That ``-1`` convention is being replaced by an identity sentinel object in - :pr:`859`, not yet landed, once the shared base class :pr:`858` (below) puts ``get()`` - behind one call site for most of these registries instead of many. The sweep - :issue:`584` asked for puts the scope at 110 of the 118 integer registries, not the - three the issue named; the two carrying a bespoke integer fallback of their - own, ``pcapng`` ``OptionType`` and ``reg`` ``AppType``, are deliberately left - alone, since neither drops a default by raising. Not reachable from wire data - -- every value a wire field can carry already resolves -- so this is a - contract fix rather than a parse fix. Applied to the nine vendor templates as - well as the 113 generated modules, and a new test renders the shared template - and compares it against the module generated from it, so a regeneration - cannot quietly undo it (:issue:`584`). -* ``_missing_`` in the four Mobility Header flag enumerations ended in - ``return cls(value)``, the same constructor that had just failed to find the value, - so every in-range value that is not already a member re-entered ``_missing_`` - unbounded and raised ``RecursionError``. ``BindingACKFlag``, ``BindingUpdateFlag``, - ``HandoverACKFlag`` and ``HandoverInitiateFlag`` are ``IntFlag`` types whose - members are single bits, so the hole was not some exotic bit pattern: ``F(0)`` -- - no flags set, the most ordinary value a flag octet can carry -- recursed on all - four, and so did every composite of two defined bits, such as - ``BindingACKFlag(0x06)``. Defining ``_missing_`` at all is what caused it, because - it shadowed the ``aenum`` ``Flag`` machinery that resolves exactly those values; - ``pcapkit/const/tcp/flags.py`` defined no ``_missing_`` at the time and never had - the defect -- it has one now, added by :issue:`647` below, ending in the same - ``super()._missing_(value)`` tail for the same reason. All four now end in - ``return super()._missing_(value)``, which is what - ``pcapkit/vendor/default.py`` emits for every other generated enumeration and what - 75 of the 117 modules under ``pcapkit/const/`` already did -- 77 now, since :issue:`647` - below gives the same ending to three more classes but only two land in modules - the count did not already have, ``TransportProtocol`` sharing ``reg/apptype.py`` - with the already-counted ``AppType`` -- so ``F(0)`` is an empty flag and - ``BindingACKFlag(0x06)`` is ``S|D``. The ``extend_enum`` idiom that - ``pcapkit/const/pcapng/record_type.py`` and ``secrets_type.py`` use to mint a - member for an unassigned integer -- the only other two modules whose ``_missing_`` - ends in ``return cls(value)``, and which do not recurse precisely because that - ``extend_enum`` runs first -- was considered and rejected for a flag type: naming - ``0x06`` ``Unassigned_0x06`` would hide the composite and pollute ``_member_map_`` - with an entry per bit pattern, up to 65536 of them for ``BindingUpdateFlag``. The - range guard above it is untouched, so an out-of-range or non-integer value is - still the same ``ValueError``. Fixed in the four ``pcapkit/vendor/mh/`` templates - and regenerated, the ``pcapkit/const/mh/`` modules being generated output that the - next crawl would otherwise revert (:issue:`623`). -* six of the 123 constant registries under ``pcapkit/const/`` rejected an - invalid value in a way the built-in ``enum`` does not, and three of them did not - reject it at all. ``Flags``, ``ftp.command.CommandType`` and - ``reg.apptype.TransportProtocol`` are ``IntFlag`` types that defined no - ``_missing_``, so the ``aenum`` ``Flag`` machinery composed a pseudo-member for any - integer whatsoever: ``Flags(-1)`` returned ``65520``, the OR of every declared TCP - header flag, so a value no 16-bit wire field can hold read back as *every* flag set - at once, while ``Flags(-65536)`` read back as none set. ``ftp.command.Command``, - ``ftp.command.FEATCode`` and ``http.method.Method`` are ``StrEnum`` types whose - ``_missing_`` reached ``value.upper()`` before checking the type, so an integer - raised ``AttributeError: 'int' object has no attribute 'upper'``. All six now raise - a bare ``ValueError``, which is what ``enum.IntEnum`` raises for a value it does not - define and what 113 of the 117 modules already did. Issue :issue:`647`'s own proposal -- - replacing those 113 with ``pcapkit.utilities.exceptions.EnumError`` -- was - deliberately rejected: ``EnumError`` is ``(BaseError, TypeError)`` and not a - ``ValueError`` at all, so it would have diverged from the built-in it was meant to - improve on, walked past the ``except ValueError`` in all 113 generated ``get()`` - bodies and silently undone :issue:`584`, and logged CRITICAL from ``BaseError.__init__`` - once per discarded default. The one deliberate divergence from the built-in is kept - and now pinned: the mutable registries look a value up, miss, and ``extend_enum`` it - rather than raising, so ``Method('FROBNICATE')``, ``ProtectionAuthority(1 << 70)`` - and ``AppType.get(65000, proto=tcp)`` still register. The three flag guards end in - ``return super()._missing_(value)``, so in-range composites still decompose and :issue:`623` - stays fixed. Two of the four unguarded modules needed no change -- - ``ipv6/extension_header.py`` and ``ftp.command.ConformanceRequirement`` define no - ``_missing_`` either, and ``aenum`` already raises the bare ``ValueError`` for them. - ``pcapkit/vendor/tcp/flags.py`` turned out to carry the guard's value checker all - along, as ``FLAG = 'isinstance(value, int) and 4 <= value <= 15'``, while its - template never interpolated it; those are the registry's *bit offsets* and its - members are ``1 << offset``, so emitting it unchanged would have rejected every - composite, every member above bit 3, and ``Flags(0)``. It now reads - ``0 <= value <= 0xFFFF``, the 16-bit field those bits live in. - ``TransportProtocol`` reads its bound off ``cls.__members__`` instead of a literal, - because ``TransportProtocol.get`` extends the registry at runtime at ``max * 2`` and - a written-down bound would reject the member it had just grown. Fixed in the four - bespoke templates under ``pcapkit/vendor/`` rather than in the generated tree -- - ``pcapkit/vendor/default.py`` needed no change, since the 113 it emits were already - correct -- and regenerated from the live IANA registries: 42 insertions and zero - deletions across exactly 4 of the 134 files, so nothing else in the tree was stale, - including the 30k-line ``reg/apptype.py``. A new - ``tests/const/test_const_enum_builtin_parity.py`` asserts the property the fix is - actually about, that a constant registry and a stdlib ``enum.IntEnum`` raise the - same exception type for the same invalid value, over all 123 registries rather than - the six; it is keyed on ``aenum.Enum`` because an ``IntFlag`` is not a subclass of - ``IntEnum`` -- its MRO runs through ``Flag`` instead -- which is how three of these - six escaped the two existing sweeps. ``tests/const/test_const_enum_get.py`` drops ``Flags`` from - ``EXPECTED_TO_RESOLVE_ANYTHING`` and its sweep grows 110 to 111, a registry that - bounds its domain finally having a failure for ``default`` to fall back from (:issue:`647`). -* **a breaking change to** ``AppType.get``: an out-of-range port - is refused rather than minted. Its ``except ValueError`` caught the range - rejection ``_missing_`` raises and minted regardless, so - ``AppType.get(-1, proto='tcp')`` returned ``PORT_-1_tcp`` -- a - ``__members__`` key no attribute access can reach -- while ``TCP(-1)`` - raised. It now tests ``_missing_`` for ``None``, which is its "no row for - this port" answer, and lets the rejection through, so both entry points - raise the same ``ValueError``; a valid but unassigned port still mints, - which is what ``get`` is for. **``Transport._make_port`` passes an - unvalidated ``int``, so ``TCP.make(srcport=99999)`` now raises where it - minted junk.** Separately, the crawler rendered one ``_missing_`` branch per + raised ``TypeError: 'RETR' already in use``. Reachable from wire data: + ``pcapkit.protocols.application.ftp`` matches with ``re.I`` and passes the + match verbatim, and :rfc:`959#section-5` makes FTP commands case-insensitive, + so ``retr file.txt`` was a valid request the library could not parse. ``get()`` + and ``_missing_`` now look the key up under the canonical upper-case name, so + every casing resolves to the existing member; a duplicate ``GET`` would have + lacked the real one's ``safe`` and ``idempotent`` attributes (:issue:`582`, + :issue:`583`). +* ``get()``'s documented ``default`` was ignored on the integer path across the + generated ``pcapkit.const`` tree, because ``_missing_`` cannot see the caller's + ``default`` -- ``Hardware.get(99999, 0)`` raised ``ValueError`` instead of returning + ``0``. The integer path now consults ``default`` before letting the error escape. + ``-1``, the placeholder the generated signature already carried, still separates "no + default" from "a default"; it is to be replaced by an identity sentinel in :pr:`859` + (not yet landed) once the shared base :pr:`858` (below) puts ``get()`` behind one + call site. The sweep covers 110 of the 118 integer registries, not the three + :issue:`584` named; ``pcapng`` ``OptionType`` and ``reg`` ``AppType`` carry a + bespoke integer fallback and are deliberately left alone, since neither drops a + default by raising. Not reachable from wire data, so a contract fix rather than a + parse fix. Applied to the nine vendor templates and the 113 generated modules; a new + test renders the shared template and compares it with the generated module, so a + regeneration cannot undo it (:issue:`584`). +* ``_missing_`` in the four Mobility Header flag enumerations (``BindingACKFlag``, + ``BindingUpdateFlag``, ``HandoverACKFlag``, ``HandoverInitiateFlag``) ended in + ``return cls(value)``, the constructor that had just failed, so every in-range value + that is not already a member re-entered ``_missing_`` unbounded and raised + ``RecursionError`` -- ``F(0)``, no flags set, and every composite such as + ``BindingACKFlag(0x06)``. Defining ``_missing_`` at all caused it, by shadowing the + ``aenum`` ``Flag`` machinery that resolves exactly those values; + ``pcapkit/const/tcp/flags.py`` had none and so never had the defect (it has one now, + from :issue:`647` below, with the same tail). All four now end in + ``return super()._missing_(value)``, as ``pcapkit/vendor/default.py`` emits and 75 + of the 117 modules under ``pcapkit/const/`` already did (77 with :issue:`647`'s two + new modules). ``F(0)`` is an empty flag and ``BindingACKFlag(0x06)`` is ``S|D``. The + ``extend_enum`` mint used by ``pcapng/record_type.py`` and ``secrets_type.py`` was + rejected for a flag type: naming ``0x06`` ``Unassigned_0x06`` would hide the + composite and add a ``_member_map_`` entry per bit pattern, up to 65536 for + ``BindingUpdateFlag``. The range guard is untouched. Fixed in the four + ``pcapkit/vendor/mh/`` templates and regenerated (:issue:`623`). +* six of the 123 constant registries rejected an invalid value differently from the + built-in ``enum``, and three did not reject it at all. ``Flags``, + ``ftp.command.CommandType`` and ``reg.apptype.TransportProtocol`` are ``IntFlag`` + types with no ``_missing_``, so ``aenum`` composed a pseudo-member for any integer: + ``Flags(-1)`` returned ``65520`` (every TCP flag set, from a value no 16-bit field + can hold) and ``Flags(-65536)`` none. ``ftp.command.Command``, ``FEATCode`` and + ``http.method.Method`` are ``StrEnum`` types whose ``_missing_`` called + ``value.upper()`` before checking the type, so an integer raised ``AttributeError``. + All six now raise a bare ``ValueError``, as ``enum.IntEnum`` does and 113 of the 117 + modules already did. :issue:`647`'s proposal to replace those 113 with + ``pcapkit.utilities.exceptions.EnumError`` was deliberately rejected: ``EnumError`` + is ``(BaseError, TypeError)``, not a ``ValueError``, so it would diverge from the + built-in, walk past the ``except ValueError`` in all 113 generated ``get()`` bodies + (silently undoing :issue:`584`), and log CRITICAL from ``BaseError.__init__`` once + per discarded default. The one deliberate divergence is kept and pinned: mutable + registries look up, miss, and ``extend_enum`` rather than raising, so + ``Method('FROBNICATE')``, ``ProtectionAuthority(1 << 70)`` and + ``AppType.get(65000, proto=tcp)`` still register. The three flag guards end in + ``return super()._missing_(value)``, so in-range composites still decompose and + :issue:`623` stays fixed. ``pcapkit/vendor/tcp/flags.py`` carried the value checker + all along as ``FLAG = 'isinstance(value, int) and 4 <= value <= 15'`` without + interpolating it; those are *bit offsets*, so emitting it unchanged would have + rejected every composite and ``Flags(0)``. It now reads ``0 <= value <= 0xFFFF``. + ``TransportProtocol`` reads its bound from ``cls.__members__``, because ``get`` + grows the registry at runtime and a literal would reject the member it had just + added. Fixed in the four bespoke templates under ``pcapkit/vendor/`` and regenerated + from live IANA data (42 insertions, no deletions, in 4 of 134 files). A new + ``tests/const/test_const_enum_builtin_parity.py`` asserts that every one of the 123 + registries raises the same exception type as a stdlib ``enum.IntEnum``; it keys on + ``aenum.Enum`` because an ``IntFlag`` is not an ``IntEnum`` subclass, which is how + three of the six escaped the existing sweeps (:issue:`647`). +* **a breaking change to** ``AppType.get``: an out-of-range port is refused + rather than minted. Its ``except ValueError`` caught the range rejection from + ``_missing_`` and minted regardless, so ``AppType.get(-1, proto='tcp')`` + returned an unreachable ``PORT_-1_tcp`` while ``TCP(-1)`` raised. It now tests + ``_missing_`` for ``None`` ("no row for this port") and lets the rejection + through; a valid but unassigned port still mints. **``Transport._make_port`` + passes an unvalidated ``int``, so ``TCP.make(srcport=99999)`` now raises where + it minted junk.** Separately, the crawler rendered one ``_missing_`` branch per registry row, so the two spans IANA registers on more than one transport emitted the same condition twice and every registry answered with the first: - ``AppType.get(6010, proto='udp')`` gave a UDP member whose ``proto`` read - ``tcp``, ``6666/udp`` answered ``ircu`` rather than ``reserved``, and - ``proto='sctp'`` minted ``x11`` where IANA assigns nothing. A branch naming a - transport now tests ``cls.__transport__``; the 762 naming none answer every - registry as before (:pr:`764`). -* **a breaking change to** ``AppType._dispatch``: it resolved a - ``proto`` naming several transport protocols by taking its lowest set bit. - ``show_flag_values`` - iterates LSB-first and ``tcp`` is the lowest declared bit, so every - composite containing it dispatched into the TCP registry whatever else it - named -- silently, and undetectably to a caller checking equality, since - ``AppType.__eq__`` compares on ``port`` alone. ``TransportProtocol`` is an - ``aenum.IntFlag`` and a member carries the whole set IANA assigned the - service, so ``tcp | udp`` is an ordinary value to read off one and an - ordinary thing to pass back in. Sweeping all 10,625 multi-transport members - through ``get(m.port, proto=m.proto)`` gave 0 exceptions, 0 mints and 46 - answers naming another service, at 20 distinct ports; 44 of those differ - only in that the member is not its port's canonical, which a single-bit - lookup does too and which ``get`` documents, leaving two real defects -- - port 888, where ``accessbuilder`` resolved to TCP's ``cddbp``, and port 999, - where ``puprouter`` resolved to TCP's ``garcon`` against UDP's ``applix``. - It now raises ``ProtocolError``, a ``ValueError`` subclass, so the - documented contract of ``get`` and ``get_all`` still holds. Refusing costs no - caller: the ten library call sites into the lookup each name one transport - protocol, and ``register_apptype``, the one place that reads a member's - composite ``proto``, tests it with ``in`` and never reaches the lookup - (:issue:`759`). -* ``get()``'s string-key miss and every generated registry's - ``_missing_`` bounded-range branch both called ``aenum.extend_enum()`` on an - unrecognised value, so e.g. ``Hardware(40)`` minted a permanent member on - first call, forever. Fixed at the two shared sites in - ``pcapkit/vendor/default.py``, plus a new ``register(value, name)`` base - classmethod for the caller-named path that is still meant to mint. All 105 - generated ``pcapkit/const/`` registries are hand-edited to match and - independently proved by regenerating the 21 changed ones from live IANA data - -- byte-identical to the hand-edit; a further seven regenerate with no change - at all. ``get()``'s no-longer-minting miss applies to all 105 registries; the - ``_missing_`` fix only changes runtime behaviour for the **21** whose - ``_missing_`` carries a bounded-range ``extend_enum`` branch -- the other 84 - have no such branch to fix: 83 override ``process()`` with their own - ``extend_enum`` call, and one, ``hip.transport.Transport``, inherits the - unmodified template but has no unassigned gap for its FLAG-bound range to - mint against. Two toolkit call sites, ``pcapkit/toolkit/scapy.py`` and - ``pcapkit/toolkit/pyshark.py``, fed ``get()`` a string key with no default - and now raise ``MissingKeyError`` where they used to get back a minted - placeholder; four protocol reader sites in ``hopopt.py``/``ipv6_opts.py`` - convert the same ``KeyError`` into ``ProtocolError``. Riding along: + ``AppType.get(6010, proto='udp')`` gave a member whose ``proto`` read ``tcp``, + ``6666/udp`` answered ``ircu`` rather than ``reserved``, and ``proto='sctp'`` + minted ``x11`` where IANA assigns nothing. A branch naming a transport now + tests ``cls.__transport__``; the 762 naming none are unchanged (:pr:`764`). +* **a breaking change to** ``AppType._dispatch``: it resolved a ``proto`` naming + several transports by its lowest set bit, and ``tcp`` is lowest, so every composite + containing it dispatched into the TCP registry -- silently, and undetectably to a + caller checking equality, since ``AppType.__eq__`` compares on ``port`` alone. + ``TransportProtocol`` is an ``aenum.IntFlag`` and a member carries the whole set + IANA assigned the service, so ``tcp | udp`` is an ordinary value to pass back in. + Sweeping all 10,625 multi-transport members through ``get(m.port, proto=m.proto)`` + gave 46 answers naming another service at 20 ports; 44 differ only in not being the + port's canonical, which a single-bit lookup does too, leaving two real defects: port + 888 (``accessbuilder`` resolved to TCP's ``cddbp``) and port 999 (``puprouter`` + resolved to TCP's ``garcon``). It now raises ``ProtocolError``, a ``ValueError`` + subclass, so the documented contract holds. No caller is affected: the ten library + call sites each name one transport, and ``register_apptype``, the one reader of a + composite ``proto``, tests it with ``in`` (:issue:`759`). +* ``get()``'s string-key miss and every generated registry's ``_missing_`` + bounded-range branch called ``aenum.extend_enum()`` on an unrecognised value, so + e.g. ``Hardware(40)`` minted a permanent member. Fixed at the two shared sites in + ``pcapkit/vendor/default.py``, plus a new ``register(value, name)`` base classmethod + for the caller-named path that still mints. All 105 generated registries are + updated; the non-minting ``get()`` miss applies to all 105, but the ``_missing_`` + fix changes behaviour only for the **21** with a bounded-range ``extend_enum`` + branch. ``pcapkit/toolkit/scapy.py`` and ``pcapkit/toolkit/pyshark.py`` fed + ``get()`` a string key with no default and now raise ``MissingKeyError`` instead of + receiving a placeholder; four reader sites in ``hopopt.py``/``ipv6_opts.py`` convert + the ``KeyError`` into ``ProtocolError``. Riding along: ``pcapkit/toolkit/pyshark.py`` gains a two-entry ``FILTER_NAME_TO_LINKTYPE`` (``eth`` to ``ETHERNET``, ``tr`` to ``IEEE802_5``), consulted ahead of - ``LinkType.get`` since pyshark reports a filter name, not a DLT name. An AST - census puts what remains out of scope at 1026 minting call sites -- 1007 in - ``_missing_`` and 7 in ``get`` across ``pcapkit/const/``, plus 12 - hand-written ``extend_enum`` calls in ``pcapkit/protocols/internet/mh.py`` - and ``pcapkit/protocols/application/ngap.py`` -- all deferred to a later - tier; ``register``/``register_alias`` account for a further 106 sites that - are the by-design caller-named path and stay untouched (:pr:`838`). -* **a breaking change to** ``pcapkit.const.ipx.socket.Socket``'s - ``_missing_``: its five archived range branches were tested in table order, - and three were strict subsets of a range tested earlier -- - ``(0x0020, 0x003F)`` after ``(0x0001, 0x0BB8)``, and both - ``(0x4000, 0x4FFF)`` and ``(0x8000, 0xFFFF)`` after ``(0x0BB9, 0xFFFF)`` -- - so they never fired. Fixed in the generator, - ``pcapkit/vendor/ipx/socket.py``, which now lists each subset range ahead of - the range containing it; ``pcapkit/const/ipx/socket.py`` changes only as the - regenerated, byte-identical consequence. **36891 of the 65521 - ``_missing_``-resolved values (56.3%) change name** -- 32 from - ``Registered by Xerox`` to ``Experimental``, 4095 from - ``Dynamically Assigned`` to ``Dynamically Assigned Socket Numbers`` and 32764 - from ``Dynamically Assigned`` to ``Statically Assigned Socket Numbers``. No - defined member changes and every value still resolves; the shift is entirely - in which archived row's name a previously-shadowed value receives. The prior - tests had pinned the shadowing as intended behaviour, with comments reading - ``# masks 'Experimental'``; those assertions are corrected here rather than + ``LinkType.get`` since pyshark reports a filter name, not a DLT name. Out of scope + and deferred: 1026 minting call sites (1007 in ``_missing_``, 7 in ``get``, 12 + hand-written in ``pcapkit/protocols/internet/mh.py`` and + ``pcapkit/protocols/application/ngap.py``); the 106 ``register``/``register_alias`` + sites are the by-design caller-named path (:pr:`838`). +* **a breaking change to** ``pcapkit.const.ipx.socket.Socket``'s ``_missing_``: three + of its five archived range branches were strict subsets of a range tested earlier -- + ``(0x0020, 0x003F)`` after ``(0x0001, 0x0BB8)``, and ``(0x4000, 0x4FFF)`` and + ``(0x8000, 0xFFFF)`` after ``(0x0BB9, 0xFFFF)`` -- so they never fired. The + generator ``pcapkit/vendor/ipx/socket.py`` now lists each subset range ahead of the + range containing it. **36891 of the 65521 ``_missing_``-resolved values (56.3%) + change name**: 32 from ``Registered by Xerox`` to ``Experimental``, 4095 from + ``Dynamically Assigned`` to ``Dynamically Assigned Socket Numbers`` and 32764 from + ``Dynamically Assigned`` to ``Statically Assigned Socket Numbers``. No defined + member changes and every value still resolves. The prior tests had pinned the + shadowing as intended (``# masks 'Experimental'``); they are corrected rather than dropped, and a new test pins all three ranges as reachable (:pr:`847`). -* ``LinkType(209).name`` returned ``'IPMB_LINUX'``, the name - tcpdump's own table (and pcapkit's generated comment) marks "Legacy names (do - not use)", because that row was emitted above the current ``I2C_LINUX = 209`` - and ``aenum`` gives a value to whichever member is defined first -- - introduced by a 2024-05-04 regeneration that inserted the legacy row above - the current one. Fixed in the generator: ``pcapkit/vendor/reg/linktype.py`` - now sinks any row whose note mentions "legacy" into a bucket appended after - every current row, so the current name is always defined first for a value - the table double-assigns; today that is value 209 alone. - ``pcapkit/const/reg/linktype.py`` changes by a 3-line move, verified - byte-reproducible from the generator. ``IPMB_LINUX`` is not removed -- it - stays a reachable alias, so ``LinkType['IPMB_LINUX']`` still resolves to 209 - -- only the value-to-name direction changes. Not breaking: no name - disappears, no value changes and no membership changes; - ``LinkType.IPMB_LINUX is LinkType.I2C_LINUX`` holds both before and after, - and only the canonical name for the shared value moves (:pr:`848`). -* **a breaking change to** 82 more const registries' - ``_missing_``: applies the owner's mint/unmint ruling (settled on :pr:`847`, - confirmed on :issue:`775` -- *"a final concrete assigned name -> mint; a notation for - the readers -> unmint"*) across every one of the 89 ``_missing_`` files still - calling ``extend_enum`` on this tree. 80 registries convert wholly (164 - branches) -- each mints a bare status word (``Unassigned``, +* ``LinkType(209).name`` returned ``'IPMB_LINUX'``, which tcpdump's own table + marks "Legacy names (do not use)", because a 2024-05-04 regeneration emitted + that row above the current ``I2C_LINUX = 209`` and ``aenum`` gives a value to + the member defined first. ``pcapkit/vendor/reg/linktype.py`` now sinks any row + whose note mentions "legacy" after every current row, so the current name is + defined first; today that affects value 209 alone. ``IPMB_LINUX`` stays a + reachable alias (``LinkType['IPMB_LINUX']`` still resolves to 209). Not + breaking: no name disappears, no value or membership changes, and + ``LinkType.IPMB_LINUX is LinkType.I2C_LINUX`` holds before and after; only the + canonical name for the shared value moves (:pr:`848`). +* **a breaking change to** 82 more const registries' ``_missing_``: applies the + owner's mint/unmint ruling (settled on :pr:`847`, confirmed on :issue:`775` -- *"a + final concrete assigned name -> mint; a notation for the readers -> unmint"*) across + the 89 ``_missing_`` files still calling ``extend_enum``. 80 registries convert + wholly (164 branches), each minting a bare status word (``Unassigned``, ``Reserved [...]``, ``Reserved for Private/Experimental Use``, - ``Unspecified in the IANA registry``, ``Deprecated`` and similar), which the - ruling treats as a notation for the reader rather than an assignment. Two are - mixed: ``EtherType``'s ``DEC Unassigned`` (3 branches) and its "Old Xerox - Experimental values. Invalid as an Ethertype since 1983." branch (1) convert, - while 46 attributed vendor-company names across 50 range blocks (Xyplex, - Motorola, Walker Richer & Quinn and 43 more) keep minting, since a - proprietary protocol has no public name beyond the company's -- two more of - the surviving branches, ``IEEE802_3_Length_Field`` and - ``Berkeley_Trailer_encap_IP``, keep minting too but are not company names and - so are not part of that count; ``Socket`` (ipx)'s ``Experimental``, - ``Dynamically Assigned Socket Numbers``, - ``Statically Assigned Socket Numbers`` and ``Dynamically Assigned`` convert - -- each names an allocation policy, not a specific assignment -- while - ``Registered by Xerox`` keeps minting, a real ownership fact. 7 of the 89 are - left alone: ``CGAType``'s ``Tag_`` mint is not an IANA-style range at - all, and the other 6 files sit on 9 classes not yet on ``EnumRegistry`` - (``AppType``, ``StatusCode``, ``ftp.return_code``, ``ftp.command``, - ``Method``, ``OptionType``), blocked on :pr:`859`/:issue:`860`. Each conversion touches - both the vendor crawler and the generated ``const`` file, per :pr:`838`'s and - :pr:`858`'s precedent; verified byte-identical regeneration on a 5-file sample - spanning both crawler shapes and both mixed registries (``ipv4/tos_del.py``, - ``ipx/socket.py``, ``reg/ethertype.py``, ``mh/access_type.py``, - ``mh/status_code.py``). Fixed collateral breakage this caused in three - sibling test files that had pinned the pre-ruling mint behaviour for - ``ProtectionAuthority``, ``TransType`` and four of ``Socket``'s ranges. - ``docs/source/conventions.rst``'s worked examples already agreed with keeping - ``Registered by Xerox`` under Mint by the time this PR landed: two prior - commits corrected them, ``dbae27548`` and ``6b333d7e4``, the second of which - is this PR's own merge-commit parent (:pr:`861`). -* **a breaking change to** ``EtherType``'s ``_missing_`` ordering: - values ``0x0101``-``0x01FF`` now resolve to their Old Xerox Experimental name - instead of an ``IEEE802_3_Length_Field_0x0101``-style one. The generated - method tested ``0x0000 <= value <= 0x05DC`` (IEEE802.3 Length Field) before - ``0x0101 <= value <= 0x01FF`` (Old Xerox Experimental); the second range sits - wholly inside the first, and the method returns on its first matching ``if``, - so the Old Xerox branch was unreachable for every value it covers. Unlike - :issue:`841`'s IPX ``Socket`` fix, these bounds come from the live IANA CSV rather - than a literal table, so the fix is general rather than a manual swap: - ``pcapkit/vendor/reg/ethertype.py``'s ``process()`` now collects each range - row, and a new ``EtherType._insert_range`` places each range ahead of the - first already-placed range that fully contains it, leaving every - non-overlapping pair in the CSV's own row order. - ``pcapkit/const/reg/ethertype.py`` is regenerated; the diff is exactly the - two branches swapping places. Swept all 56 range tests in the generated - ``_missing_``: this pair is the only containment among them, so no other - subsumed range exists today. ``tests/const/test_const_ethertype_862_unit.py`` - pins both the symptom against the committed const file and the root cause - against ``process()`` fed a reproduction of IANA's own CSV rows, plus the - general ordering rule against synthetic nested ranges -- each verified to - fail against the pre-fix generator and const file. Closes :issue:`862` (:pr:`865`). -* **a breaking change to** ``Method``: ``__new__`` called - ``str.__new__(cls)`` with no argument, so every one of the 40 registered - members' ``str`` payload was permanently empty regardless of its declared - value -- ``str(Method.GET) == ''``, and ``Method.GET == 'GET'`` was - ``False``; ``.value`` was always correct. :pr:`869` (above) had fixed this class's - *unregistered* path only, exposing the asymmetry: - ``str(Method('frob')) == 'frob'`` but ``str(Method.GET) == ''``. Fixed to - ``str.__new__(cls, value)``, mirroring ``Command.__new__`` (never had this - defect), in both the generator (``pcapkit/vendor/http/method.py``) and the - generated module; a second regeneration is byte-identical. A survey of every - hand-rolled ``__new__`` under ``pcapkit/const/`` found no sibling with the - same defect: ``Command`` already passes its value, ``FEATCode`` has no custom - ``__new__``, and ``OptionType``/``AppType`` deliberately store a formatted - display string as their value. The consequences, all in the correcting - direction and all measured: ``Method.GET == 'GET'`` is ``True`` for all 40 - members where it was ``False``; ``bool(Method.GET)`` flips ``False`` to - ``True``, since an empty payload made every member falsy; - ``json.dumps(Method.GET)`` now emits ``"GET"`` where it emitted ``""``; - ``sorted()`` over members is now lexicographic rather than - input-order-stable; and a member-keyed ``dict``/``set`` no longer collapses - -- ``{m: ... for m in Method}`` gains 40 entries where it had 1, because - ``str.__hash__`` wins the MRO and all 40 members previously hashed as ``''`` - and compared equal to one another. Nothing under ``pcapkit/protocols`` relied - on the old behaviour; the one caller in ``httpv1.py``/``httpv2.py`` only - reads ``.value`` via ``Method.get()``. Closes :issue:`870` (:pr:`871`). -* **a breaking change to** ``TransportProtocol`` and ``AppType``: - step 2 of :issue:`860`, PR 2 of 2 (the 8 non-``AppType`` bespoke registries were - :pr:`869`, above). Brings ``AppType`` and its four transport subclasses onto - ``EnumRegistry`` and stops both of its mint sites -- ``_missing_``'s 766 - range branches, and ``get()``'s own second, independent mint - (``PORT_{port}_{transport}`` -> ``'unknown'``) -- both now building through a - new ``_unregistered_member`` override that reconstructs the - ``svc``/``port``/``proto`` attributes the base's generic helper does not know - about, per the owner's ruling that *"only IANA registered ones are legit - values... get will not have sufficient information to create new ones"*. All - 766+1 branches convert uniformly, including the 8 that name a real service - assigned to a whole port span rather than declared individually (7 distinct - names, ``x11`` appearing twice: ``x11``, ``active-net``, ``satvid-datalnk``, - ``vrml-multi-use``, ``ircu``, ``swx``, ``flex-lm``); unlike ``FEATCode``'s - fix (:pr:`869`, above) there is no import-time self-mutation defect to address by - declaring members statically, so these stay deliberately unregistered - lookups. ``AppType`` also gains a working ``register()``, previously absent - -- the base's generic one would have built a member with ``svc=''`` the - moment this class mixed in ``EnumRegistry`` unoverridden, since it calls - ``__new__`` with only one positional argument. **The breaking part**: + ``Unspecified in the IANA registry``, ``Deprecated``), which the ruling treats as a + notation rather than an assignment. Two are mixed: ``EtherType``'s + ``DEC Unassigned`` (3 branches) and "Old Xerox Experimental values. Invalid as an + Ethertype since 1983." (1) convert, while 46 vendor-company names across 50 range + blocks (Xyplex, Motorola, Walker Richer & Quinn and 43 more) keep minting, since a + proprietary protocol has no public name beyond the company's; + ``IEEE802_3_Length_Field`` and ``Berkeley_Trailer_encap_IP`` also keep minting but + are not company names. ``Socket`` (ipx)'s allocation-policy names (``Experimental``, + ``Dynamically Assigned Socket Numbers``, ``Statically Assigned Socket Numbers``, + ``Dynamically Assigned``) convert, while ``Registered by Xerox`` keeps minting, a + real ownership fact. 7 of the 89 are left alone: ``CGAType``'s ``Tag_`` mint is + not an IANA-style range, and the other 6 files sit on 9 classes not yet on + ``EnumRegistry`` (``AppType``, ``StatusCode``, ``ftp.return_code``, ``ftp.command``, + ``Method``, ``OptionType``), blocked on :pr:`859`/:issue:`860`. Each conversion + changes both the vendor crawler and the generated file, per :pr:`838`'s and + :pr:`858`'s precedent. ``docs/source/conventions.rst``'s worked examples already + agreed with keeping ``Registered by Xerox`` under Mint (:pr:`861`). +* **a breaking change to** ``EtherType``'s ``_missing_`` ordering: values + ``0x0101``-``0x01FF`` now resolve to their Old Xerox Experimental name instead of an + ``IEEE802_3_Length_Field_0x0101``-style one. The generated method tested + ``0x0000 <= value <= 0x05DC`` (IEEE802.3 Length Field) first, and the Old Xerox + range sits wholly inside it, so that branch was unreachable. Unlike :issue:`841`'s + IPX fix, these bounds come from the live IANA CSV, so the fix is general: + ``pcapkit/vendor/reg/ethertype.py``'s ``process()`` collects range rows and a new + ``EtherType._insert_range`` places each ahead of the first already-placed range that + fully contains it, leaving non-overlapping pairs in CSV order. Of all 56 range + tests, this pair is the only containment. + ``tests/const/test_const_ethertype_862_unit.py`` pins the symptom, the root cause + and the ordering rule. Closes :issue:`862` (:pr:`865`). +* **a breaking change to** ``Method``: ``__new__`` called ``str.__new__(cls)`` with no + argument, so all 40 registered members' ``str`` payload was empty regardless of + value -- ``str(Method.GET) == ''`` and ``Method.GET == 'GET'`` was ``False``; + ``.value`` was always correct. :pr:`869` (above) had fixed the *unregistered* path + only, leaving ``str(Method('frob')) == 'frob'`` beside ``str(Method.GET) == ''``. + Fixed to ``str.__new__(cls, value)``, mirroring ``Command.__new__``, in the + generator (``pcapkit/vendor/http/method.py``) and the generated module. No sibling + hand-rolled ``__new__`` has the defect (``OptionType``/``AppType`` deliberately + store a display string). Consequences, all corrections: ``Method.GET == 'GET'`` is + ``True``; ``bool(Method.GET)`` flips ``False`` to ``True``; + ``json.dumps(Method.GET)`` emits ``"GET"`` not ``""``; ``sorted()`` is lexicographic + rather than input-order-stable; and ``{m: ... for m in Method}`` gains 40 entries + where it had 1, because all 40 members previously hashed as ``''`` and compared + equal. Nothing under ``pcapkit/protocols`` relied on the old behaviour + (``httpv1.py``/``httpv2.py`` read only ``.value``). Closes :issue:`870` (:pr:`871`). +* **a breaking change to** ``TransportProtocol`` and ``AppType``: step 2 of + :issue:`860`, PR 2 of 2 (the 8 non-``AppType`` bespoke registries were :pr:`869`, + above). ``AppType`` and its four transport subclasses move onto ``EnumRegistry`` and + both mint sites -- ``_missing_``'s 766 range branches and ``get()``'s second mint + (``PORT_{port}_{transport}`` -> ``'unknown'``) -- now build through a new + ``_unregistered_member`` override that reconstructs ``svc``/``port``/``proto``, per + the owner's ruling that *"only IANA registered ones are legit values... get will not + have sufficient information to create new ones"*. All 766+1 branches convert, + including 8 naming a real service assigned to a whole port span (7 distinct names: + ``x11`` twice, ``active-net``, ``satvid-datalnk``, ``vrml-multi-use``, ``ircu``, + ``swx``, ``flex-lm``); unlike ``FEATCode``'s fix (:pr:`869`, above) there is no + import-time self-mutation to cure by declaring members statically, so these stay + deliberately unregistered lookups. ``AppType`` gains a working ``register()``; the + base's would have built a member with ``svc=''``. **The breaking part**: ``TransportProtocol`` moves from power-of-two values - (``undefined=0, tcp=1, udp=2, sctp=4, dccp=8``) to sequential ``auto()`` - (``undefined=0, tcp=1, udp=2, sctp=3, dccp=4``), per the owner's follow-up - ruling once ``|``-joined composite values stopped being parsed at all. - ``AppType.get``'s ``proto`` parameter treats whatever ``int`` it is given as - a whole rather than decoding it, on either numbering, so the renumbering - changes what specific integers *mean*, not only what hand-composed ones do -- - measured on ``AppType.get(80, proto=...)``: a bare ``4`` resolved to - ``SCTP.http`` on ``main`` and now **silently** resolves to ``DCCP.unknown``, - no exception, because a real member sits at ``4`` under the new numbering - too, just a different one; a bare ``8`` used to mint ``DCCP.PORT_80_dccp`` - and now raises ``ValueError``. The silent-reroute case is the one that - matters and is disclosed here rather than only in a code comment, though - deliberately left untested for the composed-value case specifically, per the - owner's own ruling that it is "an obsoleted path from a breaking change". All - five crawlers (``apptype``, ``tcp``, ``udp``, ``sctp``, ``dccp``) regenerate - byte-identically on a second run. Fixed 9 existing tests whose bodies pinned - the old minting or power-of-two behaviour and added 18 new tests to - ``test_const_enum_no_mint.py`` against 1 removed (net +17); 403 test methods - pass across ``tests/const/`` and ``tests/protocols/transport/``, plus a - further 142 across ``tests/vendor/`` and the other - ``TransportProtocol``-consuming suites (:pr:`874`). -* closes the remaining scope of :issue:`775`: ``EtherType`` (52 branches) - and ``Socket`` (1) were the last two registries whose ``_missing_`` still - minted a permanent member for an unrecognised value; both now return a - non-registering ``cls._unregistered_member(value, name)``. The same AST walk - the issue originally used, counting ``extend_enum`` inside - ``_missing_``/``get`` across every class in ``pcapkit/const/``, goes from 54 - sites (52 in ``reg/ethertype.py``, 1 each in ``ipx/socket.py`` and - ``mh/cga_type.py``) to the single ``CGAType`` carve-out that remains a - deliberate exception (its mint is not an IANA-style range at all) -- taking - the issue from its original 1,169 sites to that one. Measured both - directions: before, ``EtherType(0x0888)`` grew ``__members__`` from 160 to - 161 and a repeat lookup returned the identical object; after, it returns - ```` with ``__members__`` unchanged, absent - from ``_value2member_map_``, and equal-but-not-identical across lookups -- - the same shape for ``Socket(0x0010)``. Two asymmetries worth knowing: - ``pcapkit/vendor/ipx/socket.py``'s regeneration was verified byte-identical - offline, since ``Socket.LINK is None`` means that crawler never touches the - network, but ``pcapkit/vendor/reg/ethertype.py``'s live-IANA-CSV fetch is - off-limits here, so its correctness rests on a mechanical substitution - checked to match exactly 52 lines plus the generator run against the - already-committed CSV fixture -- weaker than byte-identity, and disclosed as - such. Preserving each branch's existing hex-suffixed name reintroduces the - value-derived name shape ``test_const_enum_no_mint.py``'s - ``is_manufactured()`` sweep forbids; rather than weakening that predicate, a - narrow ``HEX_SUFFIXED_NAME_EXEMPT_PATHS`` covers exactly these two files, on - the grounds that the naming collision the check guards against cannot occur - once ``_unregistered_member`` never registers. ``pcapkit/corekit/enum.py``'s - docstring is corrected from naming three still-minting registries - (``EtherType``, ``Socket``, ``CGAType``) to naming ``CGAType`` alone. - ``tests/vendor/test_ipx_socket_unit.py`` is not among the tests this - falsifies -- its code tokens are unchanged and its ``EXPECTED_MISSING_NAMES`` - still carries ``0x0010: 'Registered by Xerox_0x0010'`` and still passes, - since the name itself is preserved (:pr:`878`). + (``tcp=1, udp=2, sctp=4, dccp=8``) to sequential ``auto()`` (``sctp=3, dccp=4``), + per the owner's follow-up ruling once ``|``-joined composites stopped being parsed. + ``AppType.get`` treats whatever ``int`` it is given as a whole, so the renumbering + changes what specific integers *mean*: on ``AppType.get(80, proto=...)``, a bare + ``4`` resolved to ``SCTP.http`` on ``main`` and now **silently** resolves to + ``DCCP.unknown``, because a real member sits at ``4`` under both numberings; a bare + ``8`` used to mint ``DCCP.PORT_80_dccp`` and now raises ``ValueError``. The silent + reroute is disclosed here, and the composed-value case is deliberately untested, per + the owner's ruling that it is "an obsoleted path from a breaking change". + (:pr:`874`). +* closes the remaining scope of :issue:`775`: ``EtherType`` (52 branches) and + ``Socket`` (1), the last registries whose ``_missing_`` minted a permanent member, + now return a non-registering ``cls._unregistered_member(value, name)``. The issue's + AST census of ``extend_enum`` in ``_missing_``/``get`` goes from 54 sites to the + single ``CGAType`` carve-out (its mint is not an IANA-style range), from the + original 1,169. ``EtherType(0x0888)`` used to grow ``__members__`` from 160 to 161 + and return the identical object on a repeat; it now returns + ```` with ``__members__`` unchanged, absent from + ``_value2member_map_``, equal but not identical across lookups -- likewise + ``Socket(0x0010)``. Weaker verification, disclosed: ``vendor/ipx/socket.py`` + regenerates byte-identically offline, but ``vendor/reg/ethertype.py`` needs the live + IANA CSV, so it rests on a mechanical substitution matching exactly 52 lines plus a + run against the committed CSV fixture. Keeping each branch's hex-suffixed name + reintroduces the value-derived shape ``test_const_enum_no_mint.py``'s + ``is_manufactured()`` sweep forbids; rather than weaken the predicate, a narrow + ``HEX_SUFFIXED_NAME_EXEMPT_PATHS`` covers exactly these two files, since the + collision the check guards against cannot occur once ``_unregistered_member`` never + registers. ``pcapkit/corekit/enum.py``'s docstring now names ``CGAType`` alone as + still minting. ``tests/vendor/test_ipx_socket_unit.py`` is unaffected + (``EXPECTED_MISSING_NAMES`` still carries ``0x0010: 'Registered by Xerox_0x0010'``) + (:pr:`878`). pcapkit.corekit --------------- @@ -502,513 +324,360 @@ pcapkit.corekit Added ~~~~~ -* ``tests/corekit/test_fields_numbers_width_repair.py``, covering - each octet boundary in its own method rather than one parametrised sweep, since - the defect is a pattern and a single case would pass against a fix that - special-cased the reported width. Each boundary is asserted as a pair -- the - value below it, which always packed, and the value above it, which did not -- - so that a width shifted by one in the other direction fails too. Also swept - over all eight boundaries, pinned as the ``ceil(bit_length / 8)`` invariant, - checked for the smallest mis-sized value being ``1``, round-tripped through - pack and unpack, and given controls for the bit lengths that divide by eight - and for the reachability of the repair at all. The suite deliberately asserts - widths and octets rather than exception types, because the exception depends on - whether the mis-sized width happens to have a native ``struct`` code (:issue:`599`). -* ``FieldBaseShortReadPaddingSideTests``, ten cases over 133 - subtests covering both byte orders at 2, 4 and 8 octets and at every - truncation point: the two figures :issue:`604` reports as literals, the - value-preserving property stated over every width and shortfall rather than as - a table, the two failure directions as inequalities, a signed field, a - byte-string field, and an unpack-then-pack cycle. Eight of the ten fail on the - unfixed tree. The other two must pass on both and are the guard rails -- a full - read at every width and order, which may not move, and a read against an - entirely empty buffer, which pads to all zeros either way and is the :issue:`431` - behaviour the option and list loops depend on (:issue:`604`). +* ``tests/corekit/test_fields_numbers_width_repair.py``, covering each octet + boundary in its own method, since the defect is a pattern and a single case + would pass against a fix that special-cased the reported width. Each boundary + is asserted as a pair (the value below, which always packed, and the value + above, which did not), so a width shifted by one in the other direction fails + too. It asserts widths and octets rather than exception types, because the + exception depends on whether the mis-sized width has a native ``struct`` code + (:issue:`599`). +* ``FieldBaseShortReadPaddingSideTests``, ten cases over 133 subtests covering both + byte orders at 2, 4 and 8 octets and every truncation point, including the two + figures :issue:`604` reports as literals. Eight fail on the unfixed tree; the other + two are guard rails -- a full read at every width and order, which may not move, and + a read against an empty buffer, which pads to zeros either way and is the + :issue:`431` behaviour the option and list loops depend on (:issue:`604`). Changed ~~~~~~~ -* ``Probe``, ``CipherSuite`` and ``IntegritySuite`` are ``Info`` - subclasses rather than ``typing.NamedTuple``, and no ``NamedTuple`` remains in - the package. They are Mappings now, so ``len()`` and iteration yield field - names rather than values. -* ``ModuleDescriptor.klass`` reads an already-imported module out of - ``sys.modules`` rather than re-entering ``importlib.import_module``, which - matters because next layer dispatch resolves a descriptor there on a per-frame - path. A registry *hit* holding a ``ModuleDescriptor`` is resolved once and - written back, but a *miss* deliberately is not -- recording a miss in a - class-level ``collections.defaultdict`` is the defect :issue:`421`/:pr:`426` fixed at this - layer, :issue:`425`/:pr:`428` fixed for the option, chunk and block registries, and - :issue:`555`/:pr:`560` fixed at the schema layer -- so every unrecognised frame resolved - the same fallback descriptor again: 48 of the 52 ``ModuleDescriptor.klass`` - resolutions an extraction of ``many_interfaces.pcapng`` performs, and 4 of the 7 - on ``ipv4.pcap``. ``import_module`` keeps real per-call work for a module - ``sys.modules`` already holds, so that resolution now costs ~117 ns rather than - ~436 ns and the whole miss path ~526 ns rather than ~883 ns, on CPython 3.14.7. - **The scale is worth stating plainly: this is not measurable in** ``extract()`` - **wall clock.** 48 avoided calls is ~17 us against a ~37 ms extraction, two - orders of magnitude inside this host's run-to-run variance, and the "~40% of - cumulative time" reading that prompted the work was an artifact of - ``_import_next_layer`` being a recursive-descent dispatcher -- its *self* time is - 0.58%, while ``aenum.extend_enum`` is 16.7%. What the change is taken for is its - shape rather than its speed: nothing memoises the resolved class, anywhere, so - ``sys.modules`` stays the only module cache in play and its invalidation is the - interpreter's. A memo of the class would serve the pre-reload class after an - ``importlib.reload`` forever, and an instance of it fails ``isinstance`` against - the live one. Proposed by ``@Ts-Boom`` in :pr:`563`, whose profiling found the miss - path; the implementation differs because that one added a second, - never-invalidated cache of resolved classes (:issue:`574`). -* ``SeekableReader.truncate()`` raises instead of returning a size, and - the misspelled ``writeable()`` is now spelled ``writable()``. Both are breaks for an - external caller and neither breaks anything inside the package. ``io.IOBase`` - documents one gate over two methods -- "If False, write() and truncate() will raise - OSError" -- and this reader's ``writable()`` answers False, so a caller that checked - it first, which is exactly what the contract invites, got a surprise either way - round: ``write`` and ``writelines`` raised, and ``truncate`` returned its new size. - What settled the question is that this is not a pure-Python nicety of ``_pyio`` that - the accelerated path skips -- every ordinary read-only file object in CPython raises - here, ``open(path, 'rb').truncate()`` giving ``io.UnsupportedOperation: truncate`` - off the C ``BufferedReader``, and ``_pyio.BufferedReader`` the same type from - ``_BufferedIOMixin.truncate``'s ``_checkWritable()``. The refusal is - ``UnsupportedOperation('truncate')``, the same one ``write`` already raises, and no - trade-off was needed between the house exception and the ABC's: pcapkit's - ``UnsupportedOperation`` subclasses ``io.UnsupportedOperation``, which subclasses - ``OSError``, so the in-library exception *is* the one the contract names. The - resizing ``truncate`` used to perform is not deleted, only made private as - ``_truncate_buffer``. It never touched the underlying stream -- it resizes a private - lookback window, which is the counter-argument the issue itself raised -- and it is - the only route to the "position sits before the window" state that ``seek`` and the - four buffered read paths must refuse, which :issue:`643` and :issue:`644` landed tests for; deleting - it would have taken their mechanism with them. ``writeable()``, separately, was never - an override of anything: ``'writable' in SeekableReader.__dict__`` was False and - ``SeekableReader.writable is io.IOBase.writable`` was True, so ``io``, ``shutil`` and - any third-party caller read the inherited value and never saw the one defined in this - file. Both answered False, which is the coincidence that hid it -- there was no - symptom to notice, and editing the misspelled method would silently have had no - effect. The value reported is unchanged and was always honest, the reader genuinely - being unable to write; only where the method was defined was wrong. Nothing in the - package called either method, confirmed by grep, so the risk is entirely to external - callers, which is what made this a question asked before it was acted on. The new - test asserts the property rather than the behaviour -- that a ``SeekableReader`` - raises the same exception type a read-only ``io.BufferedReader`` raises for the same - call -- with that type captured by running the call rather than named in the test, so - it tracks CPython across the 3.10--3.15 matrix instead of restating a belief about - it. 31 to 34 tests and 35 to 41 subtests over ``tests/corekit/test_io.py``, the - module at 100% coverage before and after (:issue:`645`). -* **a breaking change to** ``@final``, now enforced at runtime - on ``Info`` and ``Schema``. ``info_final`` and ``schema_final`` both end - ``return final(cls)``, so every finalised class already carried - ``typing.final``'s ``__final__`` marker and nothing read it: the decorator - was a promise to the type checker that the interpreter was free to ignore. - **``@final`` alone on an ``Info`` or ``Schema`` subclass -- without - ``@info_final`` or ``@schema_final`` -- now raises ``InfoError`` or - ``SchemaError`` at first construction**, the class being marked final but - never finalised and so carrying no generated ``__init__`` and nothing - usable. ``__init_subclass__`` cannot catch that, ``final`` being applied - after class creation, so the check is nested inside the existing one-shot - ``FinalisedState.NONE`` branch and a finalised class pays nothing for it. - **Deriving from a finalised class raises too**, from new - ``Info.__init_subclass__`` and ``Schema.__init_subclass__`` hooks, where - before only ``EnumSchema`` had one. **``SchemaError`` is a ``ValueError``**, - so code catching ``TypeError`` around a subclass declaration will not see - it. ``@info_final @final`` and ``@final @info_final`` both finalise - silently, order being unable to matter, which is why the re-entry check keys - on ``__finalised__`` rather than ``__final__`` -- only the former records - *the decorator* having run, and decorators apply bottom-up, so a - ``__final__`` test reads a class marked by ``final`` an instant earlier as - already finalised and skips the generation. ``@info_final`` twice warns and - hands back the finalised class unchanged. Every check reads its marker out - of the class's own ``__dict__``, both markers being ordinary class - attributes and so inheriting, and a class that merely descends from a - finalised one has not been mismarked by anybody. - ``pcapkit.utilities.compat`` takes ``final`` from ``typing_extensions`` - below 3.11 rather than below 3.8, because ``typing.final`` only records - ``__final__`` from 3.11 on and the guards were otherwise a silent no-op on - the 3.10 leg. ``EnumSchema.__init_subclass__`` calls the base hook first so - a refused declaration cannot leave ``__enum__`` pointing at a discarded - class, and ``pcapkit.protocols.schema.misc.pcapng``'s - ``Option.__init_subclass__`` is reordered to match, having otherwise let a - refused ``Option`` subclass displace a built-in schema first. Additive in - tree: of ``Info``'s 488 descendants 455 carry ``__final__`` and none is - subclassed, of ``Schema``'s 445, 408 do and none is, and no class carries - ``__final__`` without ``FinalisedState.FINAL``, so the bare-``@final`` guard +* ``Probe``, ``CipherSuite`` and ``IntegritySuite`` are ``Info`` subclasses rather + than ``typing.NamedTuple``, and no ``NamedTuple`` remains in the package. They are + Mappings now, so ``len()`` and iteration yield field names rather than values. +* ``ModuleDescriptor.klass`` reads an already-imported module out of ``sys.modules`` + rather than re-entering ``importlib.import_module``, since next layer dispatch + resolves a descriptor there on a per-frame path. A registry *hit* holding a + ``ModuleDescriptor`` is resolved once and written back; a *miss* deliberately is not + -- recording a miss in a class-level ``collections.defaultdict`` is the defect + :issue:`421`/:pr:`426` fixed at this layer, :issue:`425`/:pr:`428` for the option, + chunk and block registries, and :issue:`555`/:pr:`560` at the schema layer. So every + unrecognised frame resolved the fallback descriptor again (48 of the 52 resolutions + in an extraction of ``many_interfaces.pcapng``); a resolution now costs ~117 ns + rather than ~436 ns. **This is not measurable in** ``extract()`` **wall clock**: 48 + avoided calls is ~17 us against a ~37 ms extraction, inside run-to-run variance, and + the "~40% of cumulative time" reading that prompted the work was an artifact of + ``_import_next_layer`` being a recursive dispatcher. The change is taken for its + shape: ``sys.modules`` stays the only module cache, so its invalidation is the + interpreter's, whereas a memo of the class would serve the pre-reload class after an + ``importlib.reload`` forever and fail ``isinstance`` against the live one. Proposed + by ``@Ts-Boom`` in :pr:`563`, whose profiling found the miss path; the + implementation differs because that one added a second, never-invalidated cache of + resolved classes (:issue:`574`). +* ``SeekableReader.truncate()`` raises instead of returning a size, and the + misspelled ``writeable()`` is now ``writable()``. Both break external + callers; nothing inside the package calls either. ``io.IOBase`` documents one + gate over two methods -- "If False, write() and truncate() will raise + OSError" -- and this reader's ``writable()`` answers False, yet ``truncate`` + returned its new size. Every ordinary read-only file object in CPython raises + here (``open(path, 'rb').truncate()`` gives ``io.UnsupportedOperation``), so + this is not a pure-Python nicety that the C path skips. The refusal is + ``UnsupportedOperation('truncate')``, the same one ``write`` raises; + pcapkit's subclasses ``io.UnsupportedOperation``, so the house exception *is* + the one the contract names. The old resizing is not deleted, only made + private as ``_truncate_buffer``: it resizes a private lookback window, not + the stream (the issue's own counter-argument), and it is the only route to + the "position sits before the window" state that ``seek`` and the four + buffered read paths must refuse, for which :issue:`643` and :issue:`644` + landed tests. ``writeable()`` was never an override of anything -- + ``SeekableReader.writable is io.IOBase.writable`` was True -- so ``io``, + ``shutil`` and third parties never saw it; both answered False, which hid it, + and the value reported is unchanged. The new test asserts the property, that + a ``SeekableReader`` raises the same exception type as a read-only + ``io.BufferedReader``, capturing that type by running the call so it tracks + CPython across 3.10--3.15. (:issue:`645`). +* **a breaking change to** ``@final``, now enforced at runtime on ``Info`` and + ``Schema``. ``info_final`` and ``schema_final`` both end ``return final(cls)``, so + every finalised class carried ``typing.final``'s ``__final__`` marker and nothing + read it. **``@final`` alone on an ``Info`` or ``Schema`` subclass -- without + ``@info_final`` or ``@schema_final`` -- now raises ``InfoError`` or ``SchemaError`` + at first construction**, the class being marked final but never finalised and so + carrying no generated ``__init__``. ``__init_subclass__`` cannot catch that, since + ``final`` applies after class creation, so the check sits in the existing one-shot + ``FinalisedState.NONE`` branch and a finalised class pays nothing. **Deriving from a + finalised class raises too**, from new ``Info.__init_subclass__`` and + ``Schema.__init_subclass__`` hooks (before, only ``EnumSchema`` had one). + **``SchemaError`` is a ``ValueError``**, so code catching ``TypeError`` around a + subclass declaration will not see it. ``@info_final @final`` and + ``@final @info_final`` both finalise silently; the re-entry check keys on + ``__finalised__`` rather than ``__final__``, because only the former records *the + decorator* having run and decorators apply bottom-up, so a ``__final__`` test would + read a class marked an instant earlier as already finalised. ``@info_final`` twice + warns and returns the class unchanged. Every check reads its marker from the class's + own ``__dict__``, since the markers are ordinary inheriting attributes. + ``pcapkit.utilities.compat`` takes ``final`` from ``typing_extensions`` below 3.11 + rather than 3.8, because ``typing.final`` only records ``__final__`` from 3.11 and + the guards were otherwise a silent no-op on the 3.10 leg. + ``EnumSchema.__init_subclass__`` calls the base hook first so a refused declaration + cannot leave ``__enum__`` pointing at a discarded class, and + ``pcapkit.protocols.schema.misc.pcapng``'s ``Option.__init_subclass__`` is reordered + to match. Additive in tree: no ``Info`` or ``Schema`` class is subclassed after + finalising or carries ``__final__`` without ``FinalisedState.FINAL``, so the guard cannot fire on the library itself (:issue:`778`). Fixed ~~~~~ -* stdlib exceptions leaking out where the library's own were - promised: a malformed IP field value raised a bare ``ValueError`` instead of - ``FieldValueError`` (:issue:`465`); a ``bool`` address was silently packed as - ``0.0.0.1`` or ``0.0.0.0``, ``bool`` being an ``int`` subclass (:issue:`491`, :pr:`500`); - and ``@prepare`` discarded extra arguments silently and treated a *declared* - zero length as end of stream, which is now distinguished from a genuinely - exhausted one and raises ``StreamEOFError`` (:issue:`454`, :issue:`458`). -* ``FieldBase.unpack`` zero-padded straight up to a field's - declared ``length`` with ``rjust()``, regardless of how little data - ``buffer`` actually held; a ~40-octet PCAP-NG Decryption Secrets Block with a - bogus inner length was enough to force a multi-gigabyte allocation, since - ``length`` is frequently wire-derived and so attacker-controlled. A declared - length past 262144 octets -- libpcap's own ``MAXIMUM_SNAPLEN``, and this - package's own default ``snaplen`` -- that the buffer cannot back now raises - ``FieldValueError`` instead of padding for it; the option and list loops' - own tolerance for a short read past a truncated area (:issue:`431`) is far under - that ceiling and is untouched (:issue:`554`). -* that ceiling bounds one field, and a packet holds many, so the - *sum* of a parse's zero padding was still unbounded: a declared length just - under 262144 octets is honoured however often it is declared. 200 minimal - PCAP-NG Decryption Secrets Blocks -- 4,800 wire octets, each declaring a - ``secrets_length`` of 262,142 against two supplied octets -- retained 50.0 MiB, - an amplification of 10,922x per block, with the per-field guard never firing - because every individual field was within it. ``FieldBase.unpack`` now keeps a - running per-context ledger of octets supplied against octets synthesised, and - raises ``FieldValueError`` once the total of shortfalls *past 65,536 octets* - passes ``262144 + 16 * supplied``. The same 200 blocks now retain 0.2 MiB, - 54.6x rather than 10,922x. +* stdlib exceptions leaking out where the library's own were promised: a malformed IP + field value raised a bare ``ValueError`` instead of ``FieldValueError`` + (:issue:`465`); a ``bool`` address was silently packed as ``0.0.0.1`` or + ``0.0.0.0``, ``bool`` being an ``int`` subclass (:issue:`491`, :pr:`500`); and + ``@prepare`` discarded extra arguments silently and treated a *declared* zero length + as end of stream, which is now distinguished from a genuinely exhausted one and + raises ``StreamEOFError`` (:issue:`454`, :issue:`458`). +* ``FieldBase.unpack`` zero-padded up to a field's declared ``length`` with + ``rjust()`` however little data ``buffer`` held; a ~40-octet PCAP-NG Decryption + Secrets Block with a bogus inner length forced a multi-gigabyte allocation, + since ``length`` is often wire-derived. A declared length past 262144 octets -- + libpcap's ``MAXIMUM_SNAPLEN`` and this package's default ``snaplen`` -- that + the buffer cannot back now raises ``FieldValueError``; the option and list + loops' tolerance for a short read past a truncated area (:issue:`431`) is far + under that ceiling and untouched (:issue:`554`). +* that ceiling bounds one field, and a packet holds many, so the *sum* of a parse's zero + padding was still unbounded: 200 minimal PCAP-NG Decryption Secrets Blocks (4,800 + wire octets, each declaring ``secrets_length`` 262,142 against two supplied octets) + retained 50.0 MiB, 10,922x per block, with the per-field guard never firing. + ``FieldBase.unpack`` now keeps a per-context ledger of octets supplied against octets + synthesised, and raises ``FieldValueError`` once the total of shortfalls *past 65,536 + octets* passes ``262144 + 16 * supplied``. The same 200 blocks retain 0.2 MiB (54.6x). A shortfall of 65,536 octets or fewer is padded unconditionally and charged to - nothing, and that band is the load-bearing part rather than a concession. 65,536 - is the whole span of a 16-bit wire length -- how an IP header, an IPv6 payload, - a TCP or IPv4 option and a PCAP-NG option all declare their size -- so no - shortfall a capture cut short by its snapshot length can produce is subject to - the budget at all, on the first frame or the ten-thousandth. Without that band, - a running budget alone made the *same* legitimate 54-octet frame declaring an - IPv4 total length of 65,535 parse to one result on 37 of 40 identical calls and - to another on calls 26, 33 and 39, since whether it fit depended on what had - been parsed before it; a guard whose answer moves with history is worse than the - amplification it bounds. The worst legitimate single shortfall measured anywhere - was 65,495 octets, from exactly that frame, and the worst from a truncated - PCAP-NG option was 64,750. - **What this does not close, measured rather than assumed**: the 16-bit band is - deliberately untouched, so a length declared by a 16-bit wire field can still be - repeated without limit. A crafted 80,048-octet PCAP-NG file of 2,000 Enhanced - Packet Blocks, each carrying one option declaring 65,535 octets against four - real ones, parses end to end through ``Extractor(store=True)`` and retains - 125.00 MiB of synthesised zeros for 216.88 MiB of RSS -- 1,637x its own size, - linear in the block count -- both before this change and after it. That is not - an oversight in the bound: parsing a bare 40-octet IPv4 header declaring a total - length of 65,535, which is what a legitimate capture of offload-sized segments - truncated to its snapshot length looks like, amplifies by **the same 1,637x**. - The two are not separable by any budget at this layer. Separating them needs the - frame's own ``incl_len``/``orig_len`` -- a crafted block claims nothing was - truncated while declaring more than it holds, and a snapshot-truncated frame - says so on the wire -- which is knowable at the protocol layer and not here. - Nor is the budget scoped per file: nothing in the package resets the ledger, so - as shipped the bound is over everything a context has parsed rather than over - one ``Extractor`` run. That is still proportionate to the octets that context - was genuinely given, and tightening it is a one-line change at whichever layer - owns a run. - Verified against every capture in ``examples/captures/``, every one of them - truncated at some 8,700 offsets, 190 snapshot-length rewrites, and the synthetic - offload shapes above. The comparison is of the instrumented padding and - supplied-octet tallies *and* of each parse's outcome -- frame count, or - exception type and message -- so a cut that changed from parsing to crashing - would show rather than be swallowed; every one is identical to before (:issue:`573`). -* which exception a malformed TCP SACK option raised depended on - unrelated process state: a clean interpreter raised ``ProtocolError`` as - documented, but a process that had already popped - ``pcapkit.corekit.fields.misc`` from ``sys.modules`` -- which the ``#439`` - ABC-cache regression tests do in every case's ``setUp``/``tearDown`` -- - raised ``FieldValueError`` instead, from a different layer entirely, before - the documented check was even reached (:issue:`525`). The cause was - ``ListField.unpack`` resolving ``SchemaField`` through a function-local - import re-run on every call; a module popped and reimported mid-process comes - back as a second, distinct class, so ``isinstance`` against it silently - misclassified the field and billed each item by its declared length instead - of by what it actually consumed. **Any caller relying on the - previously-observed** ``FieldValueError`` **for this case now gets** - ``ProtocolError`` **instead, deterministically**, matching the method's own - docstring. Fixed by importing at module level instead. -* a ``NumberField`` whose ``length`` was a callable could not pack - or parse at any width ``struct`` has a native integer code for. ``length`` is - a placeholder of ``-1`` until the callable is resolved, ``-1`` has no native - code, and the template builder raised ``_need_process`` for it and never put - it back -- so the flag was a latch. Resolving the real width rebuilt the - template and left the latch set, and ``pre_process`` then handed ``bytes`` to - a template that had become ``>Q``, raising - ``struct.error: required argument is not an integer``; parsing failed in the - mirror direction, calling ``int.from_bytes`` on the integer that - ``struct.unpack`` had already produced. The flag is now recomputed from the - width actually in force rather than only ever raised, which is also what - keeps a callable resolving to a width with no native code -- 3 octets, say -- - byte-packed as it must be. **All four native widths were affected, not only - the 8 that was reported**: the latch has nothing to do with the width it - latches into, so 1, 2 and 4 failed identically, on ``NumberField`` and on - ``EnumField``, both of which leave ``__template__`` unset. This is what made - every extended 8-octet MPTCP DSS form unbuildable, since those widths are - chosen at runtime from the DSS flags and so must come from a callable; :pr:`585` - worked around it in the TCP schema alone, leaving every other caller exposed - (:issue:`591`). -* ``NumberField.pre_process`` sized a value with floor division - dressed up as a ceiling. When a field is packed while its ``length`` is still - the ``-1`` placeholder, the width is derived from the value, and it was derived - with ``math.ceil(value.bit_length() // 8)``. ``math.ceil`` of an integer is - that integer, so the ``//`` had already floored the quotient and the outer call - did nothing at all; the expression was plain floor division and the width came - out one octet short. ``256`` was sized at one octet, ``65536`` at two, - ``16777216`` at three, and both ``int.to_bytes`` and ``struct.pack`` refuse a - value that does not fit the width they are given. **The reach is wider than - "just past a boundary"**: floor division is wrong for every bit length that is - not an exact multiple of eight, so ``1`` -- bit length 1, floored to *zero* - octets -- failed too, and every value from 1 to 127 with it. Now written as the - ceiling it was meant to be, matching the ``math.ceil(n / 8)`` idiom used - elsewhere in the package. The repair is reached only by packing a field the - caller never resolved, since a schema resolves every field before packing it - and ``__call__`` installs a real width; that narrowness is why the defect - survived the suite added for :issue:`591`, whose five repair-path values -- ``0xFF``, - ``0xFFFF``, ``0xFFFFFFFF``, ``0xFFFFFFFFFFFFFFFF`` and ``0x800001`` -- have bit - lengths of 8, 16, 32, 64 and 24 and so sat exactly where floor division and the - ceiling agree. :issue:`591`'s own fix neither caused nor masked this, but it did change - what the failure looks like: with ``_need_process`` now recomputed from the - width in force, a mis-sized 1, 2 or 4 octets surfaces from ``struct.pack`` as - ``'B' format requires 0 <= number <= 255`` where it used to surface from - ``int.to_bytes`` as ``OverflowError``, which is why the exception named in the - report is no longer the one a mis-sized octet boundary raises. Two things on - this path are deliberately left alone, both independent of the arithmetic: a - signed field is sized without room for its sign bit, so an unresolved signed - field still cannot pack ``128``; and an unresolved field's bit mask is ``-1``, - which makes the masking and the sign remap above no-ops. The identical - ``math.ceil(x.bit_length() // 8)`` expression also survives at the two ILNP - nonce option builders in ``hopopt.py`` and ``ipv6_opts.py``, which are a - separate change (:issue:`599`). -* ``FieldBase.unpack`` padded a short read on the wrong side. When a - field's buffer falls short of its declared length -- the deliberate - accommodation that lets a snapshot-truncated capture parse (:issue:`431`) -- the - shortfall was zero-filled with ``rjust()``, which places the zeros at the - *front*. That asserts the octets never read were the leading ones, and a short - read has lost the trailing ones: the buffer ran out. It is now ``ljust()``, - which is correct for **both** byte orders rather than only for big-endian - fields, so the correction is not byte-order-conditional. The two directions - fail differently and only one was loud. Measured: one octet of a four-octet - little-endian ``120`` read as ``2013265920``, inflated by ``2 ** 24``; three - octets of a four-octet big-endian ``0x01020304`` read as ``0x10203``, scaled - *down* by 256. The second is the dangerous one -- a value smaller than the - truth passes a sanity check, where an inflated one overruns -- which is why the - big-endian half went unnoticed. A full read is untouched at every width and - order, since the padding is only ever consulted when the buffer falls short - (:issue:`604`). -* ``SeekableReader.truncate`` put its padding where the reader's own - bookkeeping says the content is, and ``read`` then returned one octet fewer than - the buffer held. Same family as :issue:`604`, but two defects rather than one, and the - padding side is only half of it. The buffer keeps its content at - ``[0:_buffer_cur]`` with unwritten padding behind it, so a reduction has to keep - the octets it has and an extension has to append at the *tail* -- the latter is - what ``io.IOBase.truncate`` means by "the contents of the new file area", the area - past the old end. It did neither: ``temp[-size:]`` sliced the buffer rather than - the content, so it kept the trailing padding and discarded the octets actually - read, and ``temp.rjust(size)`` prefixed the new zeros, displacing the content past - where ``_buffer_set`` and ``_buffer_cur`` address it. Separately, ``read`` - capped a buffered read at ``min(size, self._buffer_cur - 1)``, a count less one - measured from the start of the buffer rather than the run remaining from the - position being read from -- one octet short at the start of the buffer, and - reaching past the content into the padding anywhere further in. The shortfall was - then made up from the stream, *past* the octet that had been skipped, which both - dropped that octet and left the return short. The reported symptom needed both: - ``read(4)``, ``truncate(8)``, ``seek(0)``, ``read(8)`` over ``b'abcde'`` returned - ``b'\x00\x00\x00e'``, four octets of an eight octet request with three of them - padding, and returns ``b'abcde'`` now -- five being the whole of what a five octet - stream can answer with. Three more of the method's contract were wrong and are - fixed with it: the position was reset to the start of the buffer rather than left - alone, so a read after a truncation resumed from the wrong octet; an omitted - ``size`` resized to ``0`` rather than to the current position; and - ``_buffer_cur`` was left addressing octets a reduced buffer no longer had, so the - next read raised + nothing, and that band is load-bearing. 65,536 is the span of a 16-bit wire length + (IP header, IPv6 payload, TCP/IPv4 option, PCAP-NG option), so no shortfall from a + snapshot-truncated capture is subject to the budget. Without the band, a running + budget alone made the *same* legitimate 54-octet frame declaring an IPv4 total length + of 65,535 parse one way on 37 of 40 identical calls and another on the rest, + depending on what had been parsed before; a guard whose answer moves with history is + worse than the amplification it bounds. **What this does not close**: a length + declared by a 16-bit field can still be repeated without limit. A crafted + 80,048-octet PCAP-NG file of 2,000 Enhanced Packet Blocks, each with one option + declaring 65,535 octets against four real ones, retains 125.00 MiB of zeros (1,637x + its size), before and after. That is not an oversight: a bare 40-octet IPv4 header + declaring total length 65,535, which is what a legitimate capture of offload-sized + segments truncated to its snapshot length looks like, amplifies by **the same + 1,637x**, and no budget at this layer separates them. Separating them needs the + frame's own ``incl_len``/``orig_len``, knowable at the protocol layer and not here. + Nor is the budget per file: nothing resets the ledger, so it bounds everything a + context has parsed, which is still proportionate to the octets that context was given; + tightening it is a one-line change at whichever layer owns a run. Padding tallies and + each parse's outcome are identical to before across every capture in + ``examples/captures/`` truncated at some 8,700 offsets (:issue:`573`). +* which exception a malformed TCP SACK option raised depended on unrelated + process state: a clean interpreter raised ``ProtocolError`` as documented, but + a process that had popped ``pcapkit.corekit.fields.misc`` from ``sys.modules`` + (as the ``#439`` ABC-cache regression tests do in every ``setUp``/``tearDown``) + raised ``FieldValueError`` from a different layer, before the documented check + (:issue:`525`). ``ListField.unpack`` resolved ``SchemaField`` through a + function-local import re-run on every call; a module reimported mid-process + is a second, distinct class, so ``isinstance`` misclassified the field and + billed each item by its declared length instead of what it consumed. **Any + caller relying on the previously-observed** ``FieldValueError`` **now gets** + ``ProtocolError``, deterministically, matching the docstring. Fixed by + importing at module level. +* a ``NumberField`` whose ``length`` was a callable could not pack or parse at any + width ``struct`` has a native integer code for. ``length`` is ``-1`` until the + callable resolves, ``-1`` has no native code, and the template builder raised + ``_need_process`` for it and never cleared it -- a latch. Resolving the real width + rebuilt the template but left the latch set, so ``pre_process`` handed ``bytes`` to + a template that had become ``>Q`` + (``struct.error: required argument is not an integer``), and parsing failed in + mirror by calling ``int.from_bytes`` on an integer ``struct.unpack`` had produced. + The flag is now recomputed from the width in force, which also keeps a callable + resolving to a width with no native code (3 octets) byte-packed. **All four native + widths were affected, not only the 8 reported**, on ``NumberField`` and + ``EnumField`` alike. This made every extended 8-octet MPTCP DSS form unbuildable, + those widths being chosen at runtime from the DSS flags; :pr:`585` worked around it + in the TCP schema alone (:issue:`591`). +* ``NumberField.pre_process`` sized a value with floor division dressed as a ceiling. + When a field is packed while ``length`` is still the ``-1`` placeholder, the width + came from ``math.ceil(value.bit_length() // 8)``; the ``//`` had already floored, so + ``math.ceil`` did nothing and the width was one octet short: ``256`` at one octet, + ``65536`` at two, ``16777216`` at three. **The reach is wider than "just past a + boundary"**: every bit length that is not a multiple of eight is wrong, so ``1`` + (bit length 1, floored to *zero* octets) and every value from 1 to 127 failed too. + It is now the ceiling it was meant to be, matching the ``math.ceil(n / 8)`` idiom + used elsewhere. The repair is reached only by packing a field the caller never + resolved (a schema resolves every field first), which is why it survived + :issue:`591`'s suite: its five repair-path values (``0xFF``, ``0xFFFF``, + ``0xFFFFFFFF``, ``0xFFFFFFFFFFFFFFFF``, ``0x800001``) have bit lengths of 8, 16, 32, + 64 and 24, where floor and ceiling agree. :issue:`591`'s fix neither caused nor + masked this, but it changed the failure: a mis-sized 1, 2 or 4 octets now surfaces + from ``struct.pack`` as ``'B' format requires 0 <= number <= 255`` rather than + ``OverflowError`` from ``int.to_bytes``. Deliberately left alone: a signed field is + sized without room for its sign bit, so an unresolved signed field still cannot pack + ``128``; and an unresolved field's bit mask is ``-1``, making the masking and sign + remap no-ops. The same ``math.ceil(x.bit_length() // 8)`` survives at the two ILNP + nonce option builders in ``hopopt.py`` and ``ipv6_opts.py``, a separate change + (:issue:`599`). +* ``FieldBase.unpack`` padded a short read on the wrong side. When a buffer falls + short of the declared length -- the accommodation that lets a snapshot-truncated + capture parse (:issue:`431`) -- the shortfall was zero-filled with ``rjust()``, + placing the zeros at the *front*, as if the octets never read were the leading ones; + a short read has lost the trailing ones. It is now ``ljust()``, correct for **both** + byte orders. The two directions failed differently and only one was loud: one octet + of a four-octet little-endian ``120`` read as ``2013265920`` (inflated by + ``2 ** 24``), while three octets of a four-octet big-endian ``0x01020304`` read as + ``0x10203`` (scaled *down* by 256). The second is the dangerous one, since a value + smaller than the truth passes a sanity check where an inflated one overruns, which + is why the big-endian half went unnoticed. A full read is untouched (:issue:`604`). +* ``SeekableReader.truncate`` put its padding where the reader's bookkeeping says the + content is, and ``read`` then returned one octet fewer than the buffer held. Same + family as :issue:`604`, but two defects. The buffer keeps content at + ``[0:_buffer_cur]`` with padding behind it, so a reduction must keep the octets it + has and an extension must append at the *tail*; ``temp[-size:]`` sliced the buffer + rather than the content, keeping trailing padding and discarding octets read, and + ``temp.rjust(size)`` prefixed the zeros. Separately, ``read`` capped at + ``min(size, self._buffer_cur - 1)``, one short at the start and reaching into + padding further in, and made up the shortfall from the stream *past* the skipped + octet. The reported symptom needed both: ``read(4)``, ``truncate(8)``, ``seek(0)``, + ``read(8)`` over ``b'abcde'`` returned ``b'\x00\x00\x00e'`` and now returns + ``b'abcde'``. Three more contract faults are fixed with it: the position was reset + to the buffer start rather than left alone; an omitted ``size`` resized to ``0`` + rather than the current position; and ``_buffer_cur`` was left addressing octets a + reduced buffer no longer had, so the next read raised ``ValueError: memoryview assignment: lvalue and rvalue have different structures`` - from ``_write_buffer`` rather than returning anything. ``truncate(0)`` raised the - same ``ValueError`` by a second route, and ``_write_buffer`` is fixed with it: - ``buf[-self._buffer_size:]`` is ``buf[-0:]`` for a buffer of no size at all -- - the whole of the octets just read rather than none of them -- so it is now counted - from the front. That state is reachable only through ``truncate``, since the - constructor refuses a non-positive ``buffer_size``. A reduction also advances - ``_buffer_set`` past the octets it drops, keeping ``_buffer_set + _buffer_cur`` - equal to how far the stream has been consumed, which ``seek`` reads as its licence - to fetch more. Clamping ``_buffer_cur`` alone made that sum *under*-report the - stream, and the next forward ``seek`` then spliced in octets from the wrong - absolute offset and said nothing: ``read(8)``, ``truncate(3)``, ``seek(6)``, - ``read(1)`` over ``b'abcdefghijklmnop'`` returned ``b'l'`` where ``b'g'`` is the - octet at offset 6 -- worse than the ``ValueError`` it replaced, being silent. All - three came out of fuzzing random operation sequences against the buffer's own - invariants, none from reading the code: of 3000 rounds, 2372 left the bookkeeping - inconsistent before and 683 raised an undocumented exception, and none of either - do now. Latent in this library rather - than live -- nothing here calls ``truncate``, confirmed by grep, though it is - public on a public class -- and invisible to the existing tests, which asserted - the return value and never the content (:issue:`622`). -* the 16-bit band the :issue:`573` entry above records as deliberately left - open. ``FieldBase.unpack`` pads a shortfall of 65,536 octets or fewer - unconditionally and charges it to nothing, which is load-bearing for a capture - cut short by its snapshot length, so a length declared by a 16-bit wire field - could still be repeated without limit. The crafted 80,048-octet PCAP-NG of - 2,000 Enhanced Packet Blocks that entry describes, each carrying one option - declaring 65,535 octets against none present, synthesised 131,070,000 octets of - zeros -- 1637.393x its own size, linear in the block count and so unbounded in - the input -- and now synthesises none, with all 2,000 frames and all 2,000 - options still parsed. The bound is taken one layer up, where the information - the field layer lacks already exists: a PCAP-NG block's Block Total Length is - authoritative and cross-checked against its own trailing copy, so the option - area is that length less the fixed fields, ``captured_len`` and - ``captured_len``'s padding, and an option declaring more payload than that area - has left is malformed however complete the file behind it is. A - snapshot-truncated capture says so through ``captured_len`` instead and leaves - its options whole, so it never trips this -- which is exactly what the :pr:`571` - ``len(buffer) < length`` rejection could not distinguish, and what it was - declined for. A new ``bounded_option`` clamps the payload to the octets the - area has left at that field and warns with ``SchemaWarning``; it is applied to - all fifteen variable-width option and record payloads in the PCAP-NG schema, - and deliberately not to the block-level payload fields, which are the 32-bit - band the running ledger already budgets. A second, ``bounded_area``, clamps a - packet block's option area to the octets the block itself holds, less the - trailing Block Total Length: the area is otherwise sized from a declared length - that nothing checks against the file -- ``BlockType.post_process`` compares it - only with its own trailing copy -- so a block declaring 1,000,000 octets while - holding 36 sized its area at 999,964, an option inside it declaring 65,535 was - under that and went unclamped, and 65,535 octets of zeros were synthesised from - 36 regardless, 1,820x with no warning at all. That came out of the change's - cross-review rather than from writing it, and it is a no-op on a well-formed - block, where the octets left of the block are exactly the area plus the - trailing length's four. The five non-packet blocks' option areas keep the - framing assumption, each computing its span with a different offset, and the - general fix for a declared length reaching a read at all is :issue:`678`. Clamping - rather than refusing is what keeps the :issue:`431` accommodation, since a PCAP-NG - block read has no catch point above ``FieldBase.unpack`` and one refusal would - abort a whole extraction rather than one block; and the clamp reads only the - block's own declared framing, never a running total, so byte-identical input - answers identically whatever preceded it. The skip for a remainder already past - zero is load-bearing on the unpacking path, where a ten-octet area leaves - ``__length__`` at -2 before a payload sizes itself and clamping to it would - hand the field a ``'-2s'`` struct template; it is *not* what protects the - packing path, since ``BytesField`` and ``StringField`` both repair a negative - width to ``len(value)`` in ``pre_process``. Zero clamps fired across all six - PCAP-NG sample captures, 338 options between them, and 501 truncation levels of - ``dhcp.pcapng`` swept an octet at a time gave byte-identical results including - the failures -- 497 of which are the uncaught ``ValueError`` that :issue:`678` records, - two a ``struct.error``, and two of which parse. The bound is pinned as a - property rather than as one input: every combination of one to three options - against nine declared lengths asserts that the octets a block's options report - holding never exceed the area, and the amplification ratio is asserted flat - across 1, 8, 64 and 512 blocks, which is what separates a bound from a smaller - constant. The worst ratio reachable over five adversarial shapes afterwards is - 0.862x, against 1,637x, 960x and 224x for the same three before (:issue:`594`). -* **a breaking change to** ``FieldBase.length``: it called - ``struct.calcsize`` on a template built from a negative resolved field - length -- e.g. ``'-5s'``, from a - ``length=`` callback whose running counter had gone negative once - overdrawn -- which raised a bare ``struct.error``, uncatchable by ordinary - caller code. The one chokepoint every affected schema module shares now - catches that and raises ``ProtocolError`` instead, covering - ``application/{ftp,httpv1,httpv2,ngap}.py``, - ``internet/{hip,ipv6_route,mh}.py``, ``link/ethernet.py``, - ``misc/pcapng.py`` and ``transport/sctp.py`` with no other file changed; a - non-negative length still returns the same value, so well-formed input is - unaffected. This closes the follow-on the ``httpv2`` length-guard entry - above filed as out of scope rather than fixing directly (:issue:`805`). -* **a breaking change to** ``EnumRegistry.register()``: it now - raises ``ValueError`` naming the existing member and pointing at - ``register_alias()``, where it used to silently alias an already-registered - value under the caller's new name -- ``aenum`` treats a taken value as an - alias request, so ``register(existing_value, 'TOTALLY_NEW_NAME')`` returned - the existing member unchanged, minted nothing and raised nothing, - contradicting the method's own docstring. + from ``_write_buffer``. ``truncate(0)`` raised the same by a second route + (``buf[-self._buffer_size:]`` is ``buf[-0:]``, the whole buffer), so + ``_write_buffer`` now counts from the front; that state is reachable only through + ``truncate``. A reduction also advances ``_buffer_set`` past the dropped octets, + keeping ``_buffer_set + _buffer_cur`` equal to how far the stream has been consumed, + which ``seek`` reads as its licence to fetch more; clamping ``_buffer_cur`` alone + made that sum under-report, and the next forward ``seek`` silently spliced in octets + from the wrong offset (``read(8)``, ``truncate(3)``, ``seek(6)``, ``read(1)`` over + ``b'abcdefghijklmnop'`` returned ``b'l'`` where ``b'g'`` sits at offset 6). All + three came from fuzzing random operation sequences against the buffer's invariants: + of 3000 rounds, 2372 left the bookkeeping inconsistent and 683 raised an + undocumented exception, and none do now. Latent rather than live -- nothing here + calls ``truncate``, though it is public -- and invisible to existing tests, which + asserted the return value and never the content (:issue:`622`). +* the 16-bit band the :issue:`573` entry above records as deliberately left open: a + length declared by a 16-bit wire field could still be repeated without limit. The + crafted 80,048-octet PCAP-NG of 2,000 Enhanced Packet Blocks that entry describes + synthesised 131,070,000 octets of zeros (1637.393x its size) and now synthesises + none, with all 2,000 frames and options still parsed. The bound is taken one layer + up, where the missing information exists: a PCAP-NG block's Block Total Length is + authoritative and cross-checked against its trailing copy, so the option area is that + length less the fixed fields, ``captured_len`` and its padding, and an option + declaring more payload than the area has left is malformed. A snapshot-truncated + capture says so through ``captured_len`` and leaves its options whole, so it never + trips this -- which the :pr:`571` ``len(buffer) < length`` rejection could not + distinguish, and why it was declined. A new ``bounded_option`` clamps the payload to + the octets the area has left and warns with ``SchemaWarning``; it is applied to all + fifteen variable-width option and record payloads in the PCAP-NG schema, and + deliberately not to the block-level payload fields, the 32-bit band the running + ledger already budgets. A second, ``bounded_area``, clamps a packet block's option + area to the octets the block itself holds less the trailing Block Total Length: the + area was otherwise sized from a declared length nothing checks against the file, so a + block declaring 1,000,000 octets while holding 36 had a 999,964-octet area and 65,535 + zeros were synthesised from 36 (1,820x) with no warning. The cross-review found that + one. Both are no-ops on a well-formed block. The five non-packet blocks' option areas + keep the framing assumption; the general fix for a declared length reaching a read is + :issue:`678`. Clamping rather than refusing keeps the :issue:`431` accommodation, + since a PCAP-NG block read has no catch point above ``FieldBase.unpack`` and one + refusal would abort a whole extraction rather than one block; and the clamp reads only + the block's own declared framing, never a running total, so identical input answers + identically whatever preceded it. The skip for a remainder already past zero is + load-bearing on the unpacking path (a ten-octet area leaves ``__length__`` at -2 + before a payload sizes itself, and clamping would give the field a ``'-2s'`` + template); it is *not* what protects packing, since ``BytesField`` and + ``StringField`` repair a negative width in ``pre_process``. No clamp fires across the + six PCAP-NG sample captures (338 options), and 501 truncation levels of + ``dhcp.pcapng`` give byte-identical outcomes, most being the uncaught ``ValueError`` + that :issue:`678` records. The bound is pinned as a property, not one + input: across every combination of one to three options and nine declared lengths the + options never report more octets than the area, and the ratio is flat across 1, 8, 64 + and 512 blocks, which separates a bound from a smaller constant. The worst ratio over + five adversarial shapes is now 0.862x, against 1,637x, 960x and 224x for the same + three before (:issue:`594`). +* **a breaking change to** ``FieldBase.length``: it called ``struct.calcsize`` on + a template built from a negative resolved length (e.g. ``'-5s'``, from a + ``length=`` callback whose running counter had been overdrawn), raising a bare + ``struct.error`` that ordinary caller code does not catch. The chokepoint every + affected schema shares now raises ``ProtocolError``, covering + ``application/{ftp,httpv1,httpv2,ngap}.py``, ``internet/{hip,ipv6_route,mh}.py``, + ``link/ethernet.py``, ``misc/pcapng.py`` and ``transport/sctp.py`` with no other + file changed; a non-negative length is unaffected. This closes the follow-on the + ``httpv2`` length-guard entry above filed as out of scope (:issue:`805`). +* **a breaking change to** ``EnumRegistry.register()``: it now raises ``ValueError`` + naming the existing member and pointing at ``register_alias()``, where it used to + silently alias an already-registered value under the caller's new name -- ``aenum`` + treats a taken value as an alias request, so + ``register(existing_value, 'TOTALLY_NEW_NAME')`` returned the existing member, + minted nothing and raised nothing, contradicting its docstring. ``pcapkit.corekit.enums.EnumRegistry`` is a new mixin implementing ``get``, ``get_all``, ``register``, ``register_alias``, ``register_aliases`` and - ``_unregistered_member`` in one place, per the maintainer's ruling on :issue:`842`; - it converts the first six registries with no bespoke ``__new__`` -- - ``ipv6.extension_header.ExtensionHeader``, ``tcp.flags.Flags`` and four - ``mh.*_flag`` modules -- and the one base serves ``IntEnum``, ``IntFlag`` and - ``StrEnum`` alike. The new guard reaches only those six: the other 105 const - registries still carry the old generator's bare - ``extend_enum(cls, name, value)``, so ``pcapkit.const`` ships two - contradictory ``register`` contracts until :pr:`858` (below) migrates the rest. - Riding along: the six also gain the generated template's - ``KeyError``-catching on a missing name, which the hand-copied ``get`` never - had, so e.g. ``Flags.get('NOPE', 0)`` returns ``0`` now rather than raising - -- a widening, not a regression, though undisclosed until this PR; and a key - that is neither ``int`` nor ``str`` used to raise ``TypeError`` on - the five ``IntFlag`` registries' hand-copied ``get`` -- e.g. - ``Flags.get(3.5)`` -- since ``cls[key]`` is a containment test on a ``Flag`` - subclass rather than a name lookup there; all five now raise ``ValueError`` - instead, tried as a value by the base. ``ExtensionHeader``, the sixth and the - only ``IntEnum`` among them, already raised ``KeyError`` on the same input - before this PR -- the shift :pr:`858`'s entry attributes to the other 105 -- so - for it this is not a new divergence at all, only the same one arriving six - registries early (:pr:`855`). -* **a breaking change to** ``EnumRegistry.register()``, extended: - the 105 const registries still on the generated template's bare - ``extend_enum(cls, name, value)`` now inherit the guard :pr:`855` (above) added, - so e.g. ``TransType.register(6, 'TOTALLY_NEW_NAME')`` now raises - ``ValueError``, naming the value and the member that already holds it and - pointing at ``register_alias()``, where it used to mint nothing and raise - nothing. ``pcapkit/vendor/default.py``'s generated template now emits + ``_unregistered_member`` once, per the maintainer's ruling on :issue:`842`; it + converts the first six registries with no bespoke ``__new__`` + (``ipv6.extension_header.ExtensionHeader``, ``tcp.flags.Flags`` and four + ``mh.*_flag`` modules), and serves ``IntEnum``, ``IntFlag`` and ``StrEnum`` alike. + The guard reaches only those six: the other 105 const registries still carry the old + generator's bare ``extend_enum(cls, name, value)``, so ``pcapkit.const`` ships two + contradictory ``register`` contracts until :pr:`858` (below). The six also gain the + generated template's ``KeyError`` catching on a missing name, so + ``Flags.get('NOPE', 0)`` returns ``0`` rather than raising (a widening, undisclosed + until this PR); and a key that is neither ``int`` nor ``str``, which raised + ``TypeError`` on the five ``IntFlag`` registries' hand-copied ``get`` + (``Flags.get(3.5)``, since ``cls[key]`` is a containment test on a ``Flag``), now + raises ``ValueError``, tried as a value. ``ExtensionHeader``, the only ``IntEnum`` + among them, already raised ``KeyError`` there, the shift :pr:`858`'s entry + attributes to the other 105 (:pr:`855`). +* **a breaking change to** ``EnumRegistry.register()``, extended: the 105 const + registries still on the generated template's bare ``extend_enum`` now inherit the + guard :pr:`855` (above) added, so ``TransType.register(6, 'TOTALLY_NEW_NAME')`` + raises ``ValueError`` naming the value and the member that holds it and pointing at + ``register_alias()``, where it used to mint and raise nothing. + ``pcapkit/vendor/default.py``'s template now emits ``class {NAME}(EnumRegistry, IntEnum)`` and drops its own - ``get``/``register``/``_unregistered_member`` entirely, so every const-enum - class it produces inherits them from the shared base instead. Census across - the 121 const modules holding 127 enum classes: 6 classes, one module each, - already used ``EnumRegistry`` (tier 1, :pr:`855`); of the other 115 modules, 105 - share the generated template byte-for-byte and are converted here, and the - remaining 10 -- three of which define more than one class -- keep their own - bespoke ``__new__`` and are left alone: ``ftp/command`` (4 classes), + ``get``/``register``/``_unregistered_member``. Across 121 const modules holding 127 + enum classes: 6 already used ``EnumRegistry`` (:pr:`855`); of the other 115 modules, + 105 share the template byte-for-byte and are converted, and the remaining 10 keep + their bespoke ``__new__`` and are left alone: ``ftp/command`` (4 classes), ``ftp/return_code`` (3), ``http/method``, ``http/status_code``, - ``pcapng/option_type``, ``reg/apptype/apptype.py`` (2, ``AppType`` and - ``TransportProtocol``) and its four transport subclasses - ``dccp.DCCP``/``sctp.SCTP``/``tcp.TCP``/``udp.UDP``. That brings the total - inheriting ``EnumRegistry`` to **111 of the 127 classes**, up from 6. - ``get()``'s dispatch matrix shifts only for a key that is neither ``int`` nor - ``str``: ``KeyError`` before (tried as a name), ``ValueError`` now (tried as - a value) -- a key type nothing in the library passes. Also renames - ``pcapkit/corekit/enums.py`` to **``enum.py``** (no ``-s``), the maintainer's - ruling. Five of the 105 hand-edited files were independently regenerated live - and came back byte-identical (:pr:`858`). -* ``EnumRegistry.get()``'s ``str``-key branch tried only a name - lookup (``cls._member_map_[key]``) and, on a miss, either raised or returned - ``cls(default)`` -- it never tried the *value* path at all, so on a - ``StrEnum`` registry a key that resolved fine through the constructor - (``_Str('known-value')``) raised ``KeyError`` through ``get()`` - (``_Str.get('known-value')``), and a supplied ``default`` compounded it by - overriding a good value match the code never attempted. Fixed by falling back - to a plain ``_value2member_map_`` lookup once the name lookup misses and - before consulting ``default``. Deliberately not ``cls(key)``: - ``FEATCode._missing_`` (``pcapkit/const/ftp/command.py``) mints directly via - ``extend_enum`` for any unrecognised string, so routing the fallback through - the constructor would let a failed name lookup mint a permanent member the - moment a registry like it converts onto this base -- a dict lookup never - reaches ``_missing_`` and so cannot mint, on any registry. A name still wins - when a string is both a name and a different member's value, matching the - existing behaviour for names that already resolve. Step 1 of :issue:`860` only; step - 2, converting the bespoke registries themselves, follows separately (:pr:`869`, - :pr:`874`, below). Not touched: the ``IntEnum``/``IntFlag`` non-``str`` path, - measured identical before and after on both ``TransType`` and - ``HandoverACKFlag``; nothing under ``pcapkit/const/`` or ``pcapkit/vendor/``. - Widens rather than breaks -- a call that raised ``KeyError`` before now - resolves, and no caller that got a correct result before gets a different one - now (:pr:`863`). -* **a breaking change to** ``EnumRegistry.get()``: its docstring - claimed "It never mints", but both of its ``cls(default)`` call sites reached - ``_missing_`` for a ``default`` that fell inside a still-minting registry's - own range, growing the registry as a side effect of resolving ``default`` - rather than ``key``. Per the owner's ruling (*"Only register can mint. get - should not mint unless it falls through the _missing_'s minted ranges"*, - endorsing resolving ``default`` through a plain ``_value2member_map_`` lookup - only), both sites are replaced; ``key`` resolution is unchanged, so - ``EtherType.get(0x0888)`` still mints ``Xyplex_0x0888`` through its own - ``_missing_`` exactly as :issue:`775`'s deliberate exception permits, landing in both - lookup tables the way ``register`` would. A ``default`` naming no registered - member now falls through to the same lookup error ``key`` itself would have - raised, rather than a fresh error about the default -- the accepted cost of - the ruling. The docstring's "It never mints" is now qualified to ``default`` - only, since ``key`` may still mint on the three still-minting registries. New - ``GetDefaultNoMintTests`` pin the issue's repro on the real ``EtherType`` - registry with member count asserted unchanged, the preserved key-path mint as - a no-change guard, and default resolution on both an ``int`` and a ``str`` - registry. Four pre-existing assertions in + ``pcapng/option_type``, ``reg/apptype/apptype.py`` (``AppType`` and + ``TransportProtocol``) and its four transport subclasses. That makes **111 of the + 127 classes** inherit ``EnumRegistry``, up from 6. ``get()``'s dispatch shifts only + for a key that is neither ``int`` nor ``str`` (``KeyError``, tried as a name; now + ``ValueError``, tried as a value), a key type nothing in the library passes. + ``pcapkit/corekit/enums.py`` is renamed **``enum.py``**, the maintainer's ruling + (:pr:`858`). +* ``EnumRegistry.get()``'s ``str``-key branch tried only a name lookup + (``cls._member_map_[key]``) and on a miss raised or returned ``cls(default)``, + never the *value* path, so on a ``StrEnum`` registry a key that resolved through + the constructor (``_Str('known-value')``) raised ``KeyError`` through ``get()``, + and a supplied ``default`` overrode a good value match the code never + attempted. It now falls back to a plain ``_value2member_map_`` lookup once the + name lookup misses and before consulting ``default``. Deliberately not + ``cls(key)``: ``FEATCode._missing_`` (``pcapkit/const/ftp/command.py``) mints via + ``extend_enum`` for any unrecognised string, so routing through the constructor + would let a failed name lookup mint a permanent member once a registry like it + converts; a dict lookup never reaches ``_missing_``. A name still wins when a + string is both a name and a different member's value. Step 1 of :issue:`860`; + step 2, converting the bespoke registries, follows (:pr:`869`, :pr:`874`, + below). The ``IntEnum``/``IntFlag`` non-``str`` path is unchanged, measured on + ``TransType`` and ``HandoverACKFlag``. Widens rather than breaks: a call that + raised ``KeyError`` now resolves, and no caller that got a correct result gets a + different one (:pr:`863`). +* **a breaking change to** ``EnumRegistry.get()``: its docstring claimed "It never + mints", but both ``cls(default)`` call sites reached ``_missing_`` for a ``default`` + inside a still-minting registry's range, growing the registry as a side effect of + resolving ``default`` rather than ``key``. Per the owner's ruling (*"Only register + can mint. get should not mint unless it falls through the _missing_'s minted + ranges"*), both sites now resolve ``default`` through a plain ``_value2member_map_`` + lookup; ``key`` resolution is unchanged, so ``EtherType.get(0x0888)`` still mints + ``Xyplex_0x0888`` through its own ``_missing_``, as :issue:`775`'s deliberate + exception permits. A ``default`` naming no registered member now falls through to + the error ``key`` itself would have raised, rather than a fresh error about the + default -- the accepted cost of the ruling. The docstring's "It never mints" is + qualified to ``default`` only. New ``GetDefaultNoMintTests`` pin the issue's repro + on the real ``EtherType`` registry, the preserved key-path mint, and default + resolution on an ``int`` and a ``str`` registry. Four assertions in ``tests/const/test_const_enum_get.py`` and - ``tests/const/test_const_enum_builtin_parity.py`` that encoded the old - contract (an unregistered default failing with its own error, or resolving - into a declared-but-unassigned range) are retargeted rather than weakened. A - fifth case -- ``Hardware.get('Definitely-Not-A-Member', 40)``, where ``40`` - sits in ``Hardware``'s declared-but-unassigned range and now raises - ``KeyError`` naming the original key rather than resolving -- was deferred a - round because its test file was contended with :pr:`865`, then applied once :pr:`865` - (above) merged, re-measured with ``Hardware``'s member count unchanged at 42 - before and after. Closes :issue:`864` (:pr:`868`). + ``tests/const/test_const_enum_builtin_parity.py`` that encoded the old contract are + retargeted rather than weakened; a fifth, + ``Hardware.get('Definitely-Not-A-Member', 40)`` (``40`` sits in ``Hardware``'s + declared-but-unassigned range and now raises ``KeyError`` naming the original key), + waited on :pr:`865` because its test file was contended, then landed once :pr:`865` + (above) merged, with ``Hardware``'s member count unchanged at 42. Closes + :issue:`864` (:pr:`868`). pcapkit.dumpkit --------------- @@ -1016,83 +685,61 @@ pcapkit.dumpkit Fixed ~~~~~ -* a flag value with no declared bits no longer dumps as - ``Type::None [0]``. **This changes user-visible output in every textual dump - format.** ``make_dumper``'s ``object_hook`` renders each enumeration member as - ``Type::name [value]`` and interpolated the name unguarded, but a ``Flag`` - value composed entirely of undeclared bits has ``name is None`` rather than a - string -- so the literal four characters ``None`` landed in the name half and - ``Flags(0)`` rendered as ``Flags::None [0]`` in ``json``, ``tree``, ``text``, +* a flag value with no declared bits no longer dumps as ``Type::None [0]``. **This + changes user-visible output in every textual dump format.** ``make_dumper``'s + ``object_hook`` renders each enumeration member as ``Type::name [value]`` and + interpolated the name unguarded, but a ``Flag`` value composed entirely of + undeclared bits has ``name is None``, so the literal ``None`` landed in the name + half: ``Flags(0)`` rendered as ``Flags::None [0]`` in ``json``, ``tree``, ``text``, ``txt``, ``plist`` and ``xml``, out of both ``Extractor`` and ``TraceFlow``. A - nameless member now renders with the value's own decimal spelling instead, so - ``Flags(0)`` gives ``Flags::0 [0]`` and ``Flags(8)`` gives ``Flags::8 [8]``. - That is the spelling the enumeration libraries already use for an undeclared - residue -- ``Flags(2057).name`` is ``ACK|9``, naming the declared bit and - giving the leftovers as one decimal number -- and it cannot be mistaken for a - member name, since a Python identifier may not begin with a digit, whereas - ``NONE`` is a real declared name elsewhere in the library. Three sites carried - the identical interpolation, not one: the ``OrderedMultiDict`` key path, the - ``addon`` branch and the scalar return, which now share one ``render_enum`` - helper so the guard cannot be applied to two of three. That guard is on - ``name is None`` and not on the value, because the defect never was about zero - -- ``Flags(1)``, ``Flags(8)``, ``Flags(9)`` and ``Flags(65536)`` are equally - nameless -- and it is not an ``aenum`` quirk either, since a stdlib - ``enum.IntFlag`` answers ``name is None`` at the same values. Five of the - library's seven flag registries are nameless at zero rather than the one - reported: ``Flags`` plus the four Mobility Header flag registries, the other - two declaring an explicit ``undefined = 0``. Pre-existing since 2023-04-28 and - surfaced rather than caused by :pr:`634`, which made a flagless TCP segment reach - this path. The committed example dumps do not move, because no member in them - lacks a name (:issue:`648`). -* ``tests/dumpkit/test_nameless_enum_rendering_unit.py`` (added - by :pr:`670`) carried two probes on the same wrong assumption that every - flag-enum registry is 16 bits wide: - ``test_scalar_return_renders_a_nameless_member_as_its_value`` checked one - fixed tuple ending in ``65536`` against ``tcp.flags.Flags`` and its own - ``StdFlags`` stand-in alone, and ``test_no_flag_registry_renders_the_literal_none`` - swept every flag-enum registry instead, but only against ``registry(0)``. - Seven registries are not *all* 16 bits, at four distinct widths: - ``ftp.command.CommandType`` (3 bits), ``reg.apptype.TransportProtocol`` (4, - computed at runtime), ``mh.binding_ack_flag.BindingACKFlag``, - ``mh.handover_ack_flag.HandoverACKFlag`` and - ``mh.handover_initiate_flag.HandoverInitiateFlag`` (8 each), and - ``mh.binding_update_flag.BindingUpdateFlag`` with ``tcp.flags.Flags`` (16, - the two the old probe actually fit). ``tcp.flags.Flags(65536)`` correctly - raises -- its own ``_missing_`` bounds itself to ``0 <= value <= 0xFFFF`` -- - so the sweep was failing on correct behaviour rather than reporting a - defect. A new ``_field_mask`` derives each registry's own all-ones bound - from its declared members, and ``_nameless_values`` returns the values that - bound admits but no member names -- zero, each undeclared bit alone, and - every undeclared bit combined -- in place of the one hard-coded tuple. A - new ``test_a_value_past_the_field_is_refused_rather_than_rendered`` pins - the boundary directly: the widest in-field value is accepted, one past it - is refused. The file goes from 1 failed / 5 passed / 16 subtests to 6 - passed / 64 subtests. A cross-review found that all seven guard - ``raise`` lines this reaches -- ``tcp/flags.py``'s own included -- were - already covered by a passing test predating this fix; none was - genuinely newly reached by it. No line under ``pcapkit/`` changed - (:issue:`702`). -* the plist writer emitted mapping keys unescaped, so a key - carrying ``&``, ``<`` or ``>`` produced a report that is not well-formed XML. - ``dictdumper``'s ``_append_dict`` interpolates the key into - ``'{item}'`` and calls ``_encode_value`` on the value only, so - both branches of ``pcapkit.dumpkit.common`` that build a mapping now escape - their own keys through one ``escape_key`` helper -- a ``MultiDict``, where - :issue:`575` escaped an enum-derived key inline and nothing else, and a plain dict, - where nothing was escaped at all. The json, tree and text writers are - untouched: ``escape_key`` is a no-op for them and a plain dict is not even - rebuilt, so each still receives the caller's own mapping with the caller's - own key objects in it. Measured across 6 captures and 5 formats, exactly 2 - of the 30 reports changed -- ``test.pcapng``'s plist and xml, one writer - under two names -- and by one line each, leaving the other 28 - byte-identical; that capture is why a non-``str`` key is handled at all, - its decryption secrets block keying the TLS key log entries by a raw bytes - client random whose repr carries all three characters. The upstream half is - ``JarryShaw/DictDumper#125``, where ``_append_dict`` should call - ``_encode_value`` on a key; this does not wait on it. - ``PcapngUnescapedKeyTests``' docstrings, which claimed both the json and the - plist report come out unparseable, are reworded to match -- only the json - writer's quoting defect still stands, upstream as + nameless member now renders with the value's decimal spelling, so ``Flags(0)`` gives + ``Flags::0 [0]`` and ``Flags(8)`` gives ``Flags::8 [8]``. That is the spelling the + enumeration libraries use for an undeclared residue (``Flags(2057).name`` is + ``ACK|9``), and it cannot be mistaken for a member name, since an identifier may not + begin with a digit, whereas ``NONE`` is a real declared name elsewhere. Three sites + carried the identical interpolation (the ``OrderedMultiDict`` key path, the + ``addon`` branch and the scalar return) and now share one ``render_enum`` helper. + The guard is on ``name is None`` and not on the value, because the defect was never + about zero (``Flags(1)``, ``Flags(8)``, ``Flags(9)`` and ``Flags(65536)`` are + equally nameless) and is not an ``aenum`` quirk: stdlib ``enum.IntFlag`` answers + ``name is None`` at the same values. Five of the seven flag registries are nameless + at zero (``Flags`` and the four Mobility Header flags; the other two declare + ``undefined = 0``). Pre-existing since 2023-04-28 and surfaced by :pr:`634`, which + made a flagless TCP segment reach this path. The committed example dumps do not move + (:issue:`648`). +* ``tests/dumpkit/test_nameless_enum_rendering_unit.py`` (added by :pr:`670`) carried + two probes assuming every flag-enum registry is 16 bits wide: + ``test_scalar_return_renders_a_nameless_member_as_its_value`` used one fixed tuple + ending in ``65536``, and ``test_no_flag_registry_renders_the_literal_none`` swept + every registry but only against ``registry(0)``. Widths differ: + ``ftp.command.CommandType`` 3 bits, ``reg.apptype.TransportProtocol`` 4 (computed at + runtime), the three ``mh`` ack/initiate flags 8, and + ``mh.binding_update_flag.BindingUpdateFlag`` and ``tcp.flags.Flags`` 16. + ``tcp.flags.Flags(65536)`` correctly raises (``0 <= value <= 0xFFFF``), so the sweep + was failing on correct behaviour. A new ``_field_mask`` derives each registry's own + bound from its declared members, ``_nameless_values`` returns the values that bound + admits but no member names (zero, each undeclared bit alone, all combined), and + ``test_a_value_past_the_field_is_refused_rather_than_rendered`` pins the boundary: + the widest in-field value is accepted, one past it is refused. A cross-review found + all seven guard ``raise`` lines it reaches (``tcp/flags.py``'s own included) were + already covered by a test predating the fix, so none was newly reached. Nothing + under ``pcapkit/`` changed (:issue:`702`). +* the plist writer emitted mapping keys unescaped, so a key carrying ``&``, ``<`` or + ``>`` produced a report that is not well-formed XML. ``dictdumper``'s + ``_append_dict`` interpolates the key into ``'{item}'`` and calls + ``_encode_value`` on the value only, so both branches of ``pcapkit.dumpkit.common`` + that build a mapping now escape their keys through one ``escape_key`` helper -- a + ``MultiDict``, where :issue:`575` escaped an enum-derived key inline and nothing + else, and a plain dict, where nothing was escaped. The json, tree and text writers + are untouched (``escape_key`` is a no-op for them and a plain dict is not rebuilt). + Across 6 captures and 5 formats, exactly 2 of 30 reports changed (``test.pcapng``'s + plist and xml, one writer under two names), by one line each; that capture is why a + non-``str`` key is handled at all, its decryption secrets block keying TLS key log + entries by a raw bytes client random whose repr carries all three characters. The + upstream half is ``JarryShaw/DictDumper#125`` (``_append_dict`` should call + ``_encode_value`` on a key); this does not wait on it. ``PcapngUnescapedKeyTests``' + docstrings, which claimed both the json and plist reports were unparseable, are + reworded: only the json writer's quoting defect stands, upstream as ``JarryShaw/DictDumper#121`` (:issue:`772`). pcapkit.foundation @@ -1101,286 +748,224 @@ pcapkit.foundation Added ~~~~~ -* three extraction engines: ``engine='pypcap'`` and - ``engine='pcap_ct'``, two independent distributions of the same ``libpcap`` - interface, and ``engine='pypcapfile'`` (:pr:`386`, :pr:`405`). They buy speed by doing - less -- neither ``pypcap`` nor ``pcap_ct`` dissects at all, so they offer - neither reassembly nor flow tracing, and ``pypcapfile`` has no IPv6 decoder. - Install only **one** of ``pypcap`` and ``pcap-ct``: both own the top-level - ``pcap`` module, and with both present ``pcap-ct`` wins the import and the - other becomes unselectable. The matching interface constants ``PyPCAP``, - ``PCAP_CT`` and ``PyPCAPFile`` were missing and are now exported alongside - ``DPKT``, ``Scapy``, ``PyShark`` and ``PCAPKit`` (:pr:`412`). That brings the - built-in set to seven engines; 3.11 is the last interpreter on which every one - of them can run, and even there two of them cannot coexist. -* ``EngineBase.unsupported_reason``, a preflight every engine - answers and ``Extractor.run`` consults before anything is imported. Asking for - an engine that cannot run in the current environment now gives one warning - naming the real cause -- a Python version, a missing ``tshark``, a missing - ``libpcap``, the wrong ``pcap`` distribution -- and a clean fall back to - ``pcapkit``'s own parser, rather than an error from inside the third-party - package (:pr:`396`, :pr:`405`). -* ``conflict`` on the reassembly data models: absolute, inclusive - ranges where two fragments claimed the same span with different bytes, which - was previously lost silently on both the IP (:pr:`482`) and TCP (:issue:`443`, :pr:`478`) paths. +* three extraction engines: ``engine='pypcap'`` and ``engine='pcap_ct'``, two + independent distributions of the same ``libpcap`` interface, and + ``engine='pypcapfile'`` (:pr:`386`, :pr:`405`). They buy speed by doing less -- + neither ``pypcap`` nor ``pcap_ct`` dissects at all, so they offer no reassembly + or flow tracing, and ``pypcapfile`` has no IPv6 decoder. Install only **one** of + ``pypcap`` and ``pcap-ct``: both own the top-level ``pcap`` module, and with + both present ``pcap-ct`` wins the import and the other becomes unselectable. The + missing interface constants ``PyPCAP``, ``PCAP_CT`` and ``PyPCAPFile`` are now + exported alongside ``DPKT``, ``Scapy``, ``PyShark`` and ``PCAPKit`` (:pr:`412`). + That makes seven built-in engines; 3.11 is the last interpreter on which every + one can run, and even there two cannot coexist. +* ``EngineBase.unsupported_reason``, a preflight every engine answers and + ``Extractor.run`` consults before anything is imported. An engine that cannot + run in the current environment now gives one warning naming the real cause (a + Python version, a missing ``tshark`` or ``libpcap``, the wrong ``pcap`` + distribution) and a clean fall back to ``pcapkit``'s own parser, rather than an + error from inside the third-party package (:pr:`396`, :pr:`405`). +* ``conflict`` on the reassembly data models: absolute, inclusive ranges where + two fragments claimed the same span with different bytes, previously lost + silently on both the IP (:pr:`482`) and TCP (:issue:`443`, :pr:`478`) paths. Changed ~~~~~~~ -* ``layer=`` and ``protocol=`` are honoured rather than inert. - Both were read under the wrong names, so every value a caller passed was - dropped into ``**kwargs`` and discarded; the CLI's ``-L`` also now validates - its argument instead of accepting anything. The packet context reaches the - schema layer for the first time as well, so a field the wire elides can be - resolved from its enclosing packet (:pr:`404`). ``follow_tcp_stream`` dispatches on - the engine type, where both branches of the old test were dead and the native - adapter ran against every engine's frames (:pr:`402`). -* two reassembly and flow-tracing defaults moved (:pr:`435`), and both - are visible to a caller. ``Datagram.completed`` widened from ``bool`` to a - ``Completion`` enumeration (``COMPLETE``, ``PARTIAL``, ``TIMEOUT``); only - ``COMPLETE`` is truthy, so ``if datagram.completed:`` is unaffected but - ``datagram.completed == True`` no longer holds. TCP flow tracing is - **bidirectional by default**, which merges each flow's two halves and closes - one only once both have FINed -- 331 flows become 111 on the sample HTTP - capture, the difference being single-frame stray tails; pass - ``trace_bidirectional=False`` for the old behaviour. IP reassembly also gained - the 60-second timeout [:rfc:`1122`, :rfc:`8200`], clocked off the capture's own - timestamps rather than the wall clock, tunable with ``reasm_timeout=``; TCP - reassembly gets no timeout by default. ``trace_analyse=`` is new, and - reassembles each traced flow's application layer. -* conflicting TCP overlaps resolve first-write-wins, per - :rfc:`9293` section 3.10, where they had silently resolved last-write-wins - (:issue:`443`, :pr:`478`). A deliberate behaviour break, and a narrow one: a conforming - retransmission carries identical bytes, so nothing changes for it. IP fragment - reassembly keeps last-write-wins, because :rfc:`791` specifies the opposite - resolution, and records the disagreement instead (:pr:`482`). -* renames with no compatibility alias left behind: ``HoleDiscriptor`` is - spelled ``HoleDescriptor`` and its package alias ``TCP_HoleDiscriptor`` is +* ``layer=`` and ``protocol=`` are honoured rather than inert. Both were read + under the wrong names, so every value a caller passed was dropped into + ``**kwargs``; the CLI's ``-L`` also now validates its argument. The packet + context reaches the schema layer for the first time, so a field the wire elides + can be resolved from its enclosing packet (:pr:`404`). ``follow_tcp_stream`` + dispatches on the engine type, where both branches of the old test were dead and + the native adapter ran against every engine's frames (:pr:`402`). +* two reassembly and flow-tracing defaults moved (:pr:`435`), both visible to a + caller. ``Datagram.completed`` widened from ``bool`` to a ``Completion`` enumeration + (``COMPLETE``, ``PARTIAL``, ``TIMEOUT``); only ``COMPLETE`` is truthy, so + ``if datagram.completed:`` is unaffected but ``datagram.completed == True`` no + longer holds. TCP flow tracing is **bidirectional by default**, which merges each + flow's two halves and closes one only once both have FINed -- 331 flows become 111 + on the sample HTTP capture, the difference being single-frame stray tails; pass + ``trace_bidirectional=False`` for the old behaviour. IP reassembly also gained the + 60-second timeout [:rfc:`1122`, :rfc:`8200`], clocked off the capture's own + timestamps, tunable with ``reasm_timeout=``; TCP reassembly gets no timeout by + default. ``trace_analyse=`` is new, and reassembles each traced flow's application + layer. +* conflicting TCP overlaps resolve first-write-wins, per :rfc:`9293` section 3.10, + where they had silently resolved last-write-wins (:issue:`443`, :pr:`478`). A + deliberate, narrow behaviour break: a conforming retransmission carries + identical bytes. IP fragment reassembly keeps last-write-wins, because + :rfc:`791` specifies the opposite resolution, and records the disagreement + instead (:pr:`482`). +* renames with no compatibility alias left behind: ``HoleDiscriptor`` is spelled + ``HoleDescriptor`` and its package alias ``TCP_HoleDiscriptor`` is ``TCP_HoleDescriptor`` (:pr:`350`). -* subclass registration is **opt-in** for ``Engine``, - ``Reassembly``, ``TraceFlow`` and ``dumpkit``'s ``Dumper`` (:issue:`514`). Each - registers if and only if its registry keyword is given -- ``engine=`` for - ``Engine``, ``protocol=`` for ``Reassembly`` and ``TraceFlow``, ``fmt=`` for - ``Dumper``. Previously an absent keyword fell back to the class' own name, so - *every* subclass of the public class was registered, and declining meant - subclassing the parallel ``*Base`` class under an alias -- which is what every - built-in does, and why the public classes had **0** subclasses between them - against the ``*Base`` classes' 9, 5, 2 and 3. **This breaks out-of-tree code - that subclasses one of the four and relies on the derived key**; pass the - keyword, or call the matching ``register_*`` function. Nothing the library - ships is affected, and the ``*Base`` classes remain importable. Two things - that were silent are now loud: an unrecognised class keyword raises - ``UnsupportedCall`` instead of being swallowed by ``**kwargs`` -- which used - to register the class under its own name, so passing ``name=`` to a - ``Reassembly`` subclass silently ignored the key it was given, ``protocol=`` - being the real one -- and ``Dumper``'s ``ext=`` without ``fmt=`` likewise. A class - attribute is not an opt-in: ``__engine_name__`` and ``__protocol_name__`` still - set the name a class reports, registered or not. Each metaclass also gained a - class-level ``registry`` property mirroring ``EnumSchema.registry``. As a side - effect a ``Dumper`` subclass no longer touches the filesystem while its - ``class`` statement runs: inferring ``fmt`` from the ``kind`` property meant - instantiating the class against a ``NamedTemporaryFile`` mid-definition. - ``Engine``'s keyword is ``engine=`` rather than the ``name=`` this first - shipped with, because ``name`` cannot be passed as a class keyword at all on - Python 3.10: ``mcls``, ``name``, ``bases`` and ``namespace`` collide with - ``abc.ABCMeta.__new__``'s own parameters, which are positional-or-keyword - before 3.11 and positional-only from 3.11, so a class statement naming any of - the four raises ``TypeError`` from the metaclass before the hook is reached. - Those four are the whole of the ``ABCMeta.__new__`` collision surface, measured - on 3.10.21, 3.11.15 and 3.14.7; ``engine=``, ``protocol=`` and ``fmt=`` are all outside it, - so the documented registration path works on every supported version. There is - no ``name=`` alias -- a keyword that worked on some interpreters and not others - is the trap being removed, not a compatibility measure. -* extraction is around 46% faster on a 1,117-frame HTTP capture, - with byte-identical output (:pr:`420`). A reassembled datagram's payload is now - analysed on first read rather than eagerly, which cuts IP reassembly's own - cost by 90.7% and TCP's by 23.7% -- IP reassembly submits a datagram for every - frame, fragmented or not (:pr:`424`). Flow tracing over the same capture went from - 1416.6 ms to 744.0 ms, because the flow dumper had been handing each record to - a ``Frame`` constructor that re-dissected the whole protocol stack to return - bytes it had just been given; options are no longer parsed twice either - (:pr:`427`). All output compared byte-for-byte across the sample captures in each - case. +* subclass registration is **opt-in** for ``Engine``, ``Reassembly``, ``TraceFlow`` + and ``dumpkit``'s ``Dumper`` (:issue:`514`). Each registers if and only if its + registry keyword is given -- ``engine=`` for ``Engine``, ``protocol=`` for + ``Reassembly`` and ``TraceFlow``, ``fmt=`` for ``Dumper``. Previously an absent + keyword fell back to the class' own name, so *every* subclass of the public class + was registered, and declining meant subclassing the parallel ``*Base`` class under + an alias, which every built-in does (hence the public classes had **0** subclasses + against the ``*Base`` classes' 9, 5, 2 and 3). **This breaks out-of-tree code that + subclasses one of the four and relies on the derived key**; pass the keyword, or + call the matching ``register_*`` function. Nothing the library ships is affected, + and the ``*Base`` classes remain importable. Two silent failures are now loud: an + unrecognised class keyword raises ``UnsupportedCall`` instead of being swallowed by + ``**kwargs`` (passing ``name=`` to a ``Reassembly`` subclass silently ignored the + key, since ``protocol=`` is the real one), and so does ``Dumper``'s ``ext=`` without + ``fmt=``. A class attribute is not an opt-in: ``__engine_name__`` and + ``__protocol_name__`` still set the name a class reports, registered or not. Each + metaclass gained a class-level ``registry`` property mirroring + ``EnumSchema.registry``, and a ``Dumper`` subclass no longer touches the filesystem + while its ``class`` statement runs (inferring ``fmt`` from ``kind`` meant + instantiating it against a ``NamedTemporaryFile``). ``Engine``'s keyword is + ``engine=`` rather than the ``name=`` first shipped, because ``name`` cannot be a + class keyword on Python 3.10: ``mcls``, ``name``, ``bases`` and ``namespace`` + collide with ``abc.ABCMeta.__new__``'s parameters (positional-or-keyword before + 3.11, positional-only from 3.11), raising ``TypeError`` before the hook runs. Those + four are the whole collision surface (measured on 3.10.21, 3.11.15 and 3.14.7), and + ``engine=``, ``protocol=`` and ``fmt=`` are outside it. There is no ``name=`` alias: + a keyword that works on some interpreters and not others is the trap being removed. +* extraction is around 46% faster on a 1,117-frame HTTP capture, with byte-identical + output (:pr:`420`). A reassembled datagram's payload is analysed on first read + rather than eagerly, cutting IP reassembly's own cost by 90.7% and TCP's by 23.7% + (IP reassembly submits a datagram for every frame, fragmented or not) (:pr:`424`). + Flow tracing over the same capture went from 1416.6 ms to 744.0 ms, because the flow + dumper handed each record to a ``Frame`` constructor that re-dissected the whole + stack to return the bytes it had just been given; options are no longer parsed twice + either (:pr:`427`). Fixed ~~~~~ -* TCP reassembly mixed absolute sequence numbers with - buffer-relative slicing, so on any capture carrying a SYN with a realistic - initial sequence number incomplete datagrams were dropped silently, and the - ``completed=False`` branch of the public API was unreachable (:issue:`349`, :pr:`376`). -* ``format='text'`` raised ``AttributeError`` before writing - anything, naming a ``dictdumper.Text`` that has never existed. It now points - at ``Tree``, as the ``'txt'`` alias beside it already did. -* ``Extractor`` had the ownership of its input stream inverted, so it - did both halves of the wrong thing at once: a handle it opened itself, from - ``fin`` given as a path, was **never closed**, while a stream the *caller* - supplied and still needed **was**. ``_cleanup`` closed under - ``not self._flag_s`` where ``self._flag_s`` is the flag that gated the - ``open()``, and the ``SeekableReader`` wrapping a non-seekable input asked for - ``stream_closing=not self._flag_s`` in the same wrong direction. The leak was - one descriptor per extraction for the life of the process, and the production - root cause of the ``ResourceWarning`` that reddened :pr:`577`, :pr:`596` and :pr:`600` from - three unrelated pull requests -- :issue:`606` made the assertion in - ``tests/utilities/test_stacklevel.py`` immune to foreign warnings, which - repaired the CI signal but could not stop the leak, because the leak was in - library code. Closing the caller's stream was the more dangerous half: silent - data loss for anyone passing an open file they meant to keep reading, with no - warning and no error to notice it by. Both conditions now follow ownership, via - a single ``Extractor._owns_input``, and a finaliser releases the handle of an - extraction *abandoned* before end of file -- ``auto=False``, iterated part way - and dropped -- which reaches ``_cleanup`` by no route at all and was the second - of the two warnings. Measured on ``6c3d1b0d9``: a path-given extraction left 1 - descriptor open and a stream-given one came back closed and unreadable; the two - files :issue:`610` names now emit 0 such warnings where they emitted 2. ``__exit__`` is - deliberately left closing unconditionally, since a caller who scopes an - ``Extractor`` with ``with`` has asked for exactly that (:issue:`610`). -* ``extract(..., no_eof=True)`` never returned. End of stream was - detected correctly and ``ExtractionWarning: EOF reached`` fired, but all three - loops that handle it -- ``record_frames``, ``__next__`` and ``__call__`` -- read - ``if self._flag_n: continue`` with nothing else to stop them, so the flag - suppressed the error that *ended* the loop without supplying any other ending - and an exhausted input was retried forever. The flag is not pointless, which is - why this is a termination condition rather than a deletion: ``__main__`` sets - ``no_eof`` exactly for ``fin='-'``, and the end of what has arrived on a live - capture is not the end of the capture. What was missing was a way to tell a - capture that has paused from one that is over, and the input's own position - answers it -- ``prepare`` raises end of stream when the bytes remaining measure - zero and restores the position before raising, so two consecutive ends of stream - at the same position mean nothing arrived between them. - ``Extractor._note_eof_progress`` is that check. A pipe is unaffected, because a - read there *blocks* while the writer is open but idle rather than reporting end - of stream -- measured at a full 1.5s pause on ``6c3d1b0d9``, after which the - frames arrived -- so a paused pipe never reaches the check at all. Also fixed - thereby, and not mentioned in the issue: ``pcapkit -`` hung on **any** finished - stdin, a pipe reporting end of stream once its writer closes; verified through - the real CLI, where ``cat in.pcap | python -m pcapkit -`` went from killed at - 25s to exiting 0 with a byte-identical dump. **One deliberate narrowing**: a - *seekable* input does not block, so its two probes fall microseconds apart and a - file still being appended to now ends at the data present when the extraction - reached it -- measured with the append landing on a record boundary 0.6s in, - six frames before and five after. The previous behaviour was unbounded by - construction, which is the defect itself, so some stopping rule had to be - chosen; a timed grace period would make the cut-off intermittent rather than - absent, so following a growing file is left to a policy of its own and the - decision is pinned by a test. Worth knowing alongside it, pre-existing and - untouched: an append landing *mid-record* raises - ``ValueError: read length must be non-negative or -1`` - instead, so ``no_eof`` over a growing file only ever worked for a writer - flushing whole records. Both docstrings for ``no_eof`` now say all of this. The end-to-end regression tests are bounded by a **child process** rather - than by ``tests._support.time_limit``, because the in-process deadline proved - *intermittent* on this loop: it usually expired on time and once escaped - entirely, running past ten minutes and 13.2 GB RSS before being killed by hand. - An intermittent guard against a hang is worse than none, since the run it misses - is a wedged suite rather than a red test. Two guesses at the cause are recorded - as ruled out, so they are not made again: the parse path's two - ``except Exception`` handlers are never entered during the spin, because - ``prepare`` raises before any next-layer decode, and the memory is the retry loop - emitting tens of thousands of ``EOF reached`` records a second which the runner - retains (:issue:`620`). -* ``register_protocol`` displaced one protocol class with another and - said nothing about it. The registry behind ``pcapkit.protocols.__proto__`` is keyed - on ``cls.__name__.upper()``, and three dispatchable classes are all named ``HTTP`` - -- the generic base ``pcapkit.protocols.application.http.HTTP`` and the two version - implementations in ``httpv1`` and ``httpv2`` -- so all three compete for the single - key ``'HTTP'`` and a bare assignment decided the winner. Measured before the fix: - ``__proto__['HTTP']`` went from ``pcapkit.protocols.application.http.HTTP`` to the - ``httpv2`` class on ``register_protocol(httpv2.HTTP)``, with no exception, no log - line and no ``RegistryWarning``, and ``ProtocolBase.expand_comp('HTTP')`` then - resolved to whichever class had registered last. The overwrite now raises a - ``RegistryWarning`` naming both the displaced and the replacing class, the defining - module being the only thing that tells the three apart. The sharpest path to it is - not a call to the registrar at all: ``Protocol.__init_subclass__`` registers - unconditionally, so merely *defining* a subclass of the public ``Protocol`` under a - name a built-in already holds displaced that built-in, with nothing anywhere in the - user's code to mark the moment the dispatch table changed meaning. A plain - ``import pcapkit`` is unaffected and emits zero new warnings, measured not assumed -- - the three built-in ``HTTP`` classes derive from ``ProtocolBase`` rather than from the - public ``Protocol``, so ``__init_subclass__`` never fires for them, and the - import-time seeding in ``pcapkit/protocols/__init__.py`` keys off the distinct names - ``HTTP``, ``HTTPv1`` and ``HTTPv2``. The issue's framing that this was the one - registrar in ``pcapkit/foundation/registry/`` with no ``RegistryWarning`` is right on - the count and wrong on the reason: no function in that package warns about anything, - the siblings warning only by delegating to a classmethod that carried the guarded - ``if code in cls.__xxx__: warn(...)`` at the time -- :pr:`726` later gave all seven the - identity guard this file describes elsewhere. The accurate statement is the - stronger one -- - ``register_protocol`` is the only keyed registrar in that package, and the only - ``__name__.upper()``-keyed registry anywhere in ``pcapkit/``, that mutates its target - with no guard *and* no delegate that could supply one, its target being a - module-level dict with no class behind it to hold the guard. The new guard - deliberately reads "key present and the incumbent is a different class" where every - sibling warned on mere presence at the time: the sibling keys are codes the caller - passes, whereas this key is derived from the class and this function is the funnel - all nine wrapper registrars end in, so registering one class under two codes -- - ``register_tcp`` then ``register_udp``, which is supported and documented -- reaches - it twice with nothing displaced, and warning there would put noise on a documented - path whose wholesale filter is precisely what would then hide the real collision. - The collision is reported, not resolved: re-keying is a registry-format change that - none of the bare-name readers can absorb, every one of them degrading *silently* to - a plain - string or to ``Raw`` on a miss rather than raising, so it belongs to the registry - redesign in :issue:`514`, which this change is sequenced ahead of rather than part of. 9 to - 13 tests and 80 to 87 subtests over ``tests/foundation/registry/``, the touched - module holding 88% coverage with its misses flat at 27 (:issue:`675`). -* two ``#:`` autodoc comments in - ``pcapkit/foundation/traceflow/traceflow.py`` named a bare ``Type``, which - Sphinx's cross-reference resolver resolves against every class named - ``Type`` in the project rather than against ``typing.Type`` -- there are - five -- and silently linked to ``pcapkit.const.l2tp.type.Type``, an L2TP - field-type enum with nothing to do with dumpers. Line 424 +* TCP reassembly mixed absolute sequence numbers with buffer-relative slicing, so + on any capture carrying a SYN with a realistic initial sequence number + incomplete datagrams were dropped silently, and the ``completed=False`` branch + of the public API was unreachable (:issue:`349`, :pr:`376`). +* ``format='text'`` raised ``AttributeError`` before writing anything, naming a + ``dictdumper.Text`` that has never existed. It now points at ``Tree``, as the + ``'txt'`` alias beside it already did. +* ``Extractor`` had the ownership of its input stream inverted: a handle it opened + itself, from ``fin`` given as a path, was **never closed**, while a stream the + *caller* supplied and still needed **was**. ``_cleanup`` closed under + ``not self._flag_s`` (the flag that gated the ``open()``), and the + ``SeekableReader`` wrapping a non-seekable input asked for + ``stream_closing=not self._flag_s`` in the same wrong direction. The leak was one + descriptor per extraction for the life of the process, and the production root cause + of the ``ResourceWarning`` that reddened :pr:`577`, :pr:`596` and :pr:`600`; + :issue:`606` made ``tests/utilities/test_stacklevel.py`` immune to foreign warnings, + which repaired the CI signal but not the leak. Closing the caller's stream was the + more dangerous half: silent data loss for anyone passing an open file they meant to + keep reading. Both conditions now follow ownership via a single + ``Extractor._owns_input``, and a finaliser releases the handle of an extraction + *abandoned* before end of file (``auto=False``, iterated part way and dropped), + which reaches ``_cleanup`` by no route and was the second warning. A path-given + extraction used to leave 1 descriptor open and a stream-given one came back closed + and unreadable; the two files :issue:`610` names emit 0 such warnings where they + emitted 2. ``__exit__`` deliberately still closes unconditionally, since a caller + who scopes an ``Extractor`` with ``with`` asked for that (:issue:`610`). +* ``extract(..., no_eof=True)`` never returned. End of stream was detected and + ``ExtractionWarning: EOF reached`` fired, but all three loops that handle it + (``record_frames``, ``__next__``, ``__call__``) read ``if self._flag_n: continue``, + so the flag suppressed the error that *ended* the loop without supplying another + ending and an exhausted input was retried forever. The flag is not pointless -- + ``__main__`` sets ``no_eof`` for ``fin='-'``, and the end of what has arrived on a + live capture is not the end of the capture -- so this is a termination condition + rather than a deletion. The input's own position tells a paused capture from a + finished one: ``prepare`` raises end of stream when zero bytes remain and restores + the position first, so two consecutive ends of stream at the same position mean + nothing arrived between them (``Extractor._note_eof_progress``). A pipe is + unaffected, since a read there *blocks* while the writer is open but idle. Also + fixed, not in the issue: ``pcapkit -`` hung on **any** finished stdin; + ``cat in.pcap | python -m pcapkit -`` went from killed at 25s to exiting 0 with a + byte-identical dump. **One deliberate narrowing**: a *seekable* input does not + block, so its two probes fall microseconds apart and a file still being appended to + ends at the data present when the extraction reached it. The previous behaviour was + unbounded, which is the defect; a timed grace period would make the cut-off + intermittent rather than absent, so following a growing file is left to a policy of + its own and the decision is pinned by a test. Pre-existing and untouched: an append + landing *mid-record* raises ``ValueError: read length must be non-negative or -1``, + so ``no_eof`` over a growing file only ever worked for a writer flushing whole + records. Both ``no_eof`` docstrings say all of this. The end-to-end regression tests + are bounded by a **child process** rather than ``tests._support.time_limit``, + because the in-process deadline proved *intermittent* on this loop (once it escaped + entirely, past ten minutes and 13.2 GB RSS), and an intermittent guard against a + hang leaves a wedged suite rather than a red test. Two guesses at the cause are + ruled out: the parse path's two ``except Exception`` handlers are never entered + during the spin (``prepare`` raises before any next-layer decode), and the memory is + the retry loop emitting tens of thousands of ``EOF reached`` records a second, which + the runner retains (:issue:`620`). +* ``register_protocol`` displaced one protocol class with another and said nothing. + The registry behind ``pcapkit.protocols.__proto__`` is keyed on + ``cls.__name__.upper()``, and three dispatchable classes are all named ``HTTP`` (the + base ``pcapkit.protocols.application.http.HTTP`` and the ``httpv1`` and ``httpv2`` + implementations), so a bare assignment decided the winner: + ``register_protocol(httpv2.HTTP)`` moved ``__proto__['HTTP']`` to the ``httpv2`` + class with no exception, log line or ``RegistryWarning``, and + ``ProtocolBase.expand_comp('HTTP')`` then resolved to whichever registered last. The + overwrite now raises a ``RegistryWarning`` naming the displaced and the replacing + class. The sharpest path is not a call to the registrar: + ``Protocol.__init_subclass__`` registers unconditionally, so merely *defining* a + subclass of the public ``Protocol`` under a name a built-in holds displaced that + built-in. A plain ``import pcapkit`` emits zero new warnings: the three built-in + ``HTTP`` classes derive from ``ProtocolBase``, so ``__init_subclass__`` never fires + for them, and import-time seeding keys off the distinct names ``HTTP``, ``HTTPv1`` + and ``HTTPv2``. The issue called this the one registrar in + ``pcapkit/foundation/registry/`` with no warning; more accurately, none warned by + itself, the siblings only by delegating to a classmethod with an + ``if code in cls.__xxx__: warn(...)`` guard (:pr:`726` later gave all seven the + identity guard described elsewhere), and this is the only ``__name__.upper()``-keyed + registry in ``pcapkit/`` that mutates its target with no guard and no delegate able + to supply one. The new guard deliberately reads "key present and the incumbent is a + different class" where siblings warned on mere presence then: this key is derived + from the class, and this function is the funnel all nine wrapper registrars end in, + so registering one class under two codes (``register_tcp`` then ``register_udp``, + supported and documented) reaches it twice with nothing displaced, and warning there + would put noise on a documented path whose wholesale filter would then hide the real + collision. The collision is reported, not resolved: re-keying is a registry-format + change that none of the bare-name readers can absorb (each degrades *silently* to a + string or ``Raw`` on a miss), so it belongs to the registry redesign in + :issue:`514`, which this is sequenced ahead of (:issue:`675`). +* two ``#:`` autodoc comments in ``pcapkit/foundation/traceflow/traceflow.py`` named a + bare ``Type``, which Sphinx resolves against every class named ``Type`` in the + project (five) rather than ``typing.Type``, and silently linked to + ``pcapkit.const.l2tp.type.Type``, an L2TP field-type enum. Line 424 (``#: ~typing.Type[Dumper]: Dumper class.``, spelled out since :issue:`709` fixed - it) is the live case: once :issue:`684` rendered - ``TraceFlow._foutio``, the built docs pointed a reader at the wrong class - with no warning that anything had gone sideways. Both sites now spell it - ``~typing.Type[Dumper]``, the same form eight other files in the tree - already use for the identical ambiguity. Line 146 (the first line of - ``__output__``'s ``#:`` block, which continues through line 149) is fixed - for the same reason but is currently inert: the ``# type:`` comment - sixteen lines below, at line 162, spells the identical bare - ``Type[Dumper]`` -- so whatever eventually renders this attribute's type - still has the same ambiguity to resolve. This half of the fix is - insurance against the day something reads it cleanly. This pair of sites - is part of :issue:`709`. Four more bare ``Type`` sites -- hand-written - ``:type:`` fields at - ``docs/source/pcapkit/foundation/engines/engine.rst:40``, + it) is the live case: once :issue:`684` rendered ``TraceFlow._foutio``, the built + docs pointed at the wrong class with no warning. Both sites now spell it + ``~typing.Type[Dumper]``, as eight other files already do. Line 146 (the first line + of ``__output__``'s ``#:`` block) is fixed for the same reason but is currently + inert, since the ``# type:`` comment at line 162 spells the same bare + ``Type[Dumper]``; it is insurance for whatever eventually renders that type. This + pair is part of :issue:`709`. Four more bare ``Type`` sites, the hand-written + ``:type:`` fields at ``docs/source/pcapkit/foundation/engines/engine.rst:40``, ``.../reassembly/reassembly.rst:33`` and ``:43``, and - ``.../traceflow/traceflow.rst:40`` -- carried the same ambiguity and are - fixed separately, by :pr:`714` (:issue:`709`). -* ``register_protocol``'s overwrite warning could claim a - protocol was replaced with itself. The guard that decides *whether* to - warn was already correct -- ``incumbent is not protocol``, an identity - check from :pr:`681` -- but the message built both operands with a bare - ``repr()``, and for an ordinary class that is just - ````. A factory that defines a same-named, - closure-local class on every call (as - ``tests/protocols/test_construction_keyword_check_unit.py``'s - ``_protocol_class`` does) produces two genuinely distinct classes sharing - one ``__module__`` and ``__qualname__``, so both reprs print identically - and a real, correct overwrite reads as "overwriting X with X." The message - now compares the two reprs first, and only when they coincide appends each - object's ``id()`` to tell them apart; the common case, where the two - classes are named differently, is untouched. ``__module__``/``__qualname__`` - was considered as the disambiguator instead of ``id()`` and rejected: for - the reported shape those are exactly what the coinciding repr already - renders, so they discriminate nothing an ``id()`` does not already have to. - A new ``test_register_protocol_disambiguates_classes_sharing_a_repr`` fails - against the unfixed message and passes against the fix; the targeted suite - goes 49 to 50 passed, and the file's coverage holds at 97% (:issue:`710`). -* five more registrars warned on mere key presence rather than - on an actual overwrite, outside the wording of :issue:`718`'s identity guard + ``.../traceflow/traceflow.rst:40``, are fixed separately by :pr:`714` + (:issue:`709`). +* ``register_protocol``'s overwrite warning could claim a protocol was replaced with + itself. The guard was correct (``incumbent is not protocol``, an identity check from + :pr:`681`), but the message built both operands with a bare ``repr()``. A factory + defining a same-named closure-local class on every call (as + ``tests/protocols/test_construction_keyword_check_unit.py``'s ``_protocol_class`` + does) produces two distinct classes sharing one ``__module__`` and ``__qualname__``, + so a real overwrite read "overwriting X with X." The message now compares the two + reprs and, only when they coincide, appends each object's ``id()``. + ``__module__``/``__qualname__`` was rejected as the disambiguator: for the reported + shape they are exactly what the coinciding repr already renders. The new + ``test_register_protocol_disambiguates_classes_sharing_a_repr`` fails against the + unfixed message (:issue:`710`). +* five more registrars warned on mere key presence rather than an actual overwrite, + outside the wording of :issue:`718`'s identity guard (``incumbent is not None and incumbent is not new``, landed by :pr:`726`), whose - issue named only code-keyed registrars: ``register_engine``, - ``register_reassembly`` and ``register_traceflow`` on ``Extractor``, and - ``register_dumper`` on both ``Extractor`` and ``TraceFlow``. All five now - compare the incumbent by identity before warning; both ``register_dumper`` - sites compare only the stored dumper, so a re-registration that changes just - the file extension stays silent too, judged defensible rather than - comparing the full ``(dumper, ext)`` pair. Non-breaking: a correct caller - sees strictly fewer warnings and no change to return value or exception. - Five new test methods pin the silent/warns-anyway split, each failing with - ``AssertionError: Expected 'warn' to not have been called`` against the - reverted guard alone (:issue:`739`). + issue named only code-keyed registrars: ``register_engine``, ``register_reassembly`` + and ``register_traceflow`` on ``Extractor``, and ``register_dumper`` on both + ``Extractor`` and ``TraceFlow``. All five now compare the incumbent by identity + before warning; both ``register_dumper`` sites compare only the stored dumper, so + re-registering with just a new file extension stays silent too, judged defensible + rather than comparing the full ``(dumper, ext)`` pair. Non-breaking: a correct + caller sees strictly fewer warnings and no change to return value or exception. Five + new tests pin the silent/warns-anyway split (:issue:`739`). pcapkit.protocols ----------------- @@ -1388,1406 +973,1043 @@ pcapkit.protocols Added ~~~~~ -* ESP parsing and construction [:rfc:`4303`], with optional payload - decryption and ICV verification through ``cryptography`` - (``pip install pypcapkit[crypto]``) (:pr:`378`). Keys reach the dissector through a - new caller-state channel, ``pcapkit.corekit.context``, surfaced as the - ``context=`` keyword on ``extract()`` and ``Extractor``, carrying an - ``esp.SecurityAssociation``. Nothing here raises: with no association the SPI - and sequence number are still reported and the ciphertext is left opaque, with - ``status=NO_SA``, and a failed decryption or ICV check is recorded on the - parsed result the same way. Extended Sequence Numbers, TFC padding and - anti-replay are not implemented, and an unsupported cipher or MAC is refused - with a clear error rather than half-processed. -* SCTP as a transport protocol [:rfc:`9260`]: all 13 chunk types, 8 - chunk parameters and 13 error causes, with the CRC32c both recorded and - verifiable -- it covers the SCTP packet alone, with no IP pseudo-header, so it - can be checked from the SCTP bytes. Upper layers register on the DATA chunk's - Payload Protocol Identifier through ``register_sctp``, not on a port (:pr:`379`). -* NGAP over SCTP (3GPP TS 38.413), decoding aligned PER through - ``pycrate`` (``pip install pypcapkit[NGAP]``) (:discussion:`251`, :pr:`417`). Decoding is generic - by ASN.1 shape rather than per-procedure, so all 81 elementary procedures and - 438 protocol IEs work and a new 3GPP release needs no code change. Registered - as a default on PPID 60 and 66, but only 60 decodes: PPID 66 is an NGAP PDU - inside a DTLS record, and there is no DTLS dissector. ``pycrate`` is - deliberately excluded from the ``all`` extra -- it is LGPL-2.1+ where this +* ESP parsing and construction [:rfc:`4303`], with optional payload decryption and ICV + verification through ``cryptography`` (``pip install pypcapkit[crypto]``) + (:pr:`378`). Keys reach the dissector through a new caller-state channel, + ``pcapkit.corekit.context``, surfaced as the ``context=`` keyword on ``extract()`` + and ``Extractor``, carrying an ``esp.SecurityAssociation``. Nothing here raises: + with no association the SPI and sequence number are still reported and the + ciphertext is left opaque with ``status=NO_SA``, and a failed decryption or ICV + check is recorded on the parsed result the same way. Extended Sequence Numbers, TFC + padding and anti-replay are not implemented, and an unsupported cipher or MAC is + refused with a clear error rather than half-processed. +* SCTP as a transport protocol [:rfc:`9260`]: all 13 chunk types, 8 chunk + parameters and 13 error causes, with the CRC32c both recorded and verifiable (it + covers the SCTP packet alone, with no IP pseudo-header). Upper layers register + on the DATA chunk's Payload Protocol Identifier through ``register_sctp``, not + on a port (:pr:`379`). +* NGAP over SCTP (3GPP TS 38.413), decoding aligned PER through ``pycrate`` + (``pip install pypcapkit[NGAP]``) (:discussion:`251`, :pr:`417`). Decoding is + generic by ASN.1 shape rather than per-procedure, so all 81 elementary + procedures and 438 protocol IEs work and a new 3GPP release needs no code + change. Registered as a default on PPID 60 and 66, but only 60 decodes: PPID 66 + is an NGAP PDU inside a DTLS record, and there is no DTLS dissector. ``pycrate`` + is deliberately excluded from the ``all`` extra -- it is LGPL-2.1+ where this package is BSD-3-Clause, and lands some 238 MB to obtain one module. -* the Mobility Header registry, completed (:pr:`383`, :pr:`437`). The - :rfc:`5568` fast-handover messages and options first, then all 24 registered - message data types, all 71 registered options -- with nested sub-option - registries for the flow identification, access network identifier, - quality-of-service and LMA-controlled MAG parameter families -- and all 4 CGA - extensions. The CGA Parameters option was the last to come off the generic - handler, which is what let the four CGA extensions round-trip end to end, - since it is the only option that can carry one on the wire. -* dispatch entries for dissectors that existed but were reachable - from no registry (:pr:`436`): FTP-DATA on TCP 20, HTTP/1 on TCP 8080, HTTP on UDP - 8080, ``L2TPv2`` on UDP 1701 and OSPF at ``TransType`` 89. ``VLAN`` became an - abstract base with ``C_Tag`` (802.1Q) and ``S_Tag`` (802.1ad) as concrete - subclasses, so a Q-in-Q frame no longer collapses into one opaque ``Raw``; - ``L2TP`` likewise became a base, with ``L2TPv2`` carrying the :rfc:`2661` - implementation. -* ``Protocol``/``ProtocolBase`` gain a ``code=`` class keyword for - the next-layer dispatch registries -- ``Link.__proto__``, - ``Internet.__proto__``, ``TCP.__proto__``, ``UDP.__proto__``, - ``SCTP.__proto__``, ``Frame.__proto__`` and ``PCAPNG.__proto__`` -- the same - opt-in treatment :issue:`514` gave ``Engine``, ``Reassembly``, ``TraceFlow`` and +* the Mobility Header registry, completed (:pr:`383`, :pr:`437`): the :rfc:`5568` + fast-handover messages and options, then all 24 registered message data types, all + 71 registered options (with nested sub-option registries for the flow + identification, access network identifier, quality-of-service and LMA-controlled MAG + parameter families) and all 4 CGA extensions. The CGA Parameters option, the only + option that can carry a CGA extension on the wire, was the last to come off the + generic handler, which is what let the four extensions round-trip end to end. +* dispatch entries for dissectors that existed but were reachable from no registry + (:pr:`436`): FTP-DATA on TCP 20, HTTP/1 on TCP 8080, HTTP on UDP 8080, ``L2TPv2`` + on UDP 1701 and OSPF at ``TransType`` 89. ``VLAN`` became an abstract base with + ``C_Tag`` (802.1Q) and ``S_Tag`` (802.1ad) as concrete subclasses, so a Q-in-Q + frame no longer collapses into one opaque ``Raw``; ``L2TP`` likewise became a + base, with ``L2TPv2`` carrying the :rfc:`2661` implementation. +* ``Protocol``/``ProtocolBase`` gain a ``code=`` class keyword for the next-layer + dispatch registries (``Link.__proto__``, ``Internet.__proto__``, ``TCP.__proto__``, + ``UDP.__proto__``, ``SCTP.__proto__``, ``Frame.__proto__``, ``PCAPNG.__proto__``), + the opt-in treatment :issue:`514` gave ``Engine``, ``Reassembly``, ``TraceFlow`` and ``Dumper`` above, extended to the one family that needed a registration key - invented rather than merely un-guarded. Omitting ``code`` leaves a subclass - unregistered, exactly as before this keyword existed: the built-in dispatch - tables are still populated by literal assignment in each layer module, not by - ``__init_subclass__``, so nothing the library ships moves. ``code`` accepts a - bare enum member, whose *type* infers the destination -- ``EtherType`` means - ``Link``, ``TransType`` means ``Internet``, ``PayloadProtocolIdentifier`` - means ``SCTP``, and ``LinkType`` means *both* ``Frame`` **and** ``PCAPNG``, - deterministically, mirroring what ``register_linktype`` already does by - hand -- or a ``{destination: key}`` mapping, required for a raw ``int`` such - as a TCP/UDP port number, which cannot say by itself which transport it - belongs to. Either form may appear in an iterable, so one declaration can - register a class into several registries at once, e.g. a ``L2TP`` subclass - reachable both by IP protocol number and by a UDP port. The explicit - mapping form is accepted even for a key whose type could be inferred -- - being more explicit than required is never an error. Inference refuses - rather than guesses: an enum member whose type names no known destination - raises ``RegistryError`` instead of silently doing nothing or picking an - arbitrary registry, and an unrecognised class keyword raises - ``UnsupportedCall``, matching the other four families. Backed by the new - ``pcapkit.foundation.registry.protocols.register_protocol_code``, which can - also be called directly to register a class that declined at class-definition - time. This was written up as the mechanism :issue:`548` (``TransType.L2TP`` - registered nowhere) needs, with fixing that issue described here as "now a - one-declaration change". Investigating :issue:`548` found otherwise -- 115 is an - :rfc:`3931` L2TPv3-over-IP header with no class to dispatch to, so the - declaration would have pointed the :rfc:`2661` parser at it. See the - corresponding **Fixed** entry below; the mechanism itself is unaffected, and - its worked example now names ``L2TPv3`` rather than ``L2TPv2`` (:pr:`570`). -* ``tests/protocols/test_dispatch_reachability_unit.py``, the - coverage :issue:`548` asked for: every ``ProtocolBase`` descendant whose ``__index__`` - returns an enum member is checked to be reachable under that code in the - registry its enum *type* designates, read from the same - ``_CODE_DESTINATIONS`` table backing ``code=`` so the two cannot drift. Where - ``test_dispatch_registry_unit.py`` walks the 38 entries that exist and checks - each parses, this walks the classes and catches one nothing registered at all - -- the shape in which ``OSPF`` once shipped reachable from no table. 23 claims - verified, no gaps; a companion case injects a gap and confirms the audit - reports it, so the guard cannot rot into a permanently green no-op (:issue:`548`). -* ``tests/protocols/transport/test_tcp_mptcp_join_flag_ordering_unit.py``, - covering all three MP_JOIN layouts through the *public* constructor, the - construct-pack-parse cycle for each, the stale-flags case that rules out a - zero-valued default, the statement order itself, and controls that the parse path - and the flag-independent options are unaffected. The gap it closes is why 100% - statement and branch coverage of the two changed modules coexisted with a - completely broken public path: the pre-existing cases reach ``_make_mptcp_join`` - by assigning a Python ``set`` to ``_flags`` on a bare ``TCP.__new__(TCP)``, which - executes every branch while bypassing both the ordering and the accumulator's - type. The now-stale ``tcp-mptcp/MP_JOIN`` entry is deleted from - ``EXPECTED_FAILURES``, and the MP_JOIN exclusion in + invented. Omitting ``code`` leaves a subclass unregistered, as before; the built-in + tables are still filled by literal assignment in each layer module, so nothing the + library ships moves. ``code`` accepts a bare enum member, whose *type* infers the + destination (``EtherType`` means ``Link``, ``TransType`` means ``Internet``, + ``PayloadProtocolIdentifier`` means ``SCTP``, and ``LinkType`` means *both* + ``Frame`` **and** ``PCAPNG``, mirroring ``register_linktype``), or a + ``{destination: key}`` mapping, required for a raw ``int`` such as a port number, + which cannot say which transport it belongs to. Either form may appear in an + iterable, so one declaration can register a class into several registries (e.g. a + ``L2TP`` subclass reachable by IP protocol number and by a UDP port), and the + mapping form is accepted even where inference would work. Inference refuses rather + than guesses: an enum member whose type names no known destination raises + ``RegistryError``, and an unrecognised class keyword raises ``UnsupportedCall``, + matching the other four families. Backed by + ``pcapkit.foundation.registry.protocols.register_protocol_code``, also callable + directly for a class that declined at definition time. It was written up as the + mechanism :issue:`548` (``TransType.L2TP`` registered nowhere) needs, "now a + one-declaration change"; investigating :issue:`548` found otherwise -- 115 is an + :rfc:`3931` L2TPv3-over-IP header with no class to dispatch to, so the declaration + would have pointed the :rfc:`2661` parser at it (see the corresponding **Fixed** + entry below). The mechanism is unaffected, and its worked example now names + ``L2TPv3`` rather than ``L2TPv2`` (:pr:`570`). +* ``tests/protocols/test_dispatch_reachability_unit.py``, the coverage :issue:`548` + asked for: every ``ProtocolBase`` descendant whose ``__index__`` returns an enum + member must be reachable under that code in the registry its enum *type* + designates, read from the same ``_CODE_DESTINATIONS`` table backing ``code=`` so + the two cannot drift. Where ``test_dispatch_registry_unit.py`` walks the 38 + entries that exist and checks each parses, this walks the classes and catches + one nothing registered, the shape in which ``OSPF`` once shipped reachable from + no table. 23 claims verified, no gaps; a companion case injects a gap and + confirms the audit reports it (:issue:`548`). +* ``tests/protocols/transport/test_tcp_mptcp_join_flag_ordering_unit.py``, covering + all three MP_JOIN layouts through the *public* constructor, the construct-pack-parse + cycle for each, the stale-flags case that rules out a zero-valued default, the + statement order, and controls for the parse path and the flag-independent options. + It closes the gap that let 100% statement and branch coverage of the two changed + modules coexist with a broken public path: the old cases reached + ``_make_mptcp_join`` by assigning a Python ``set`` to ``_flags`` on a bare + ``TCP.__new__(TCP)``, which executes every branch while bypassing both the ordering + and the accumulator's type. The stale ``tcp-mptcp/MP_JOIN`` entry is deleted from + ``EXPECTED_FAILURES`` and the MP_JOIN exclusion in ``test_tcp_mptcp_subtype_unit.py`` is lifted (:issue:`587`). -* ``tests/protocols/test_option_generator_tcp_base_unit.py``, asserting that - every ``TCP_BASE`` key is a parameter ``TCP.make`` declares -- derived from - ``inspect.signature``, so it catches a fourth misspelling nobody has made yet - -- and that the segment the mapping builds carries the stated header in both - the data model and the packed octets. Two of the three wrong names were - invisible to any assertion about a *value*, the value asked for being equal to - the default that was used instead, which is what the signature check is for +* ``tests/protocols/test_option_generator_tcp_base_unit.py``, asserting that every + ``TCP_BASE`` key is a parameter ``TCP.make`` declares (derived from + ``inspect.signature``, so it catches a fourth misspelling nobody has made yet) + and that the built segment carries the stated header in both data model and + packed octets. Two of the three wrong names were invisible to any assertion + about a *value*, the value asked for equalling the default used instead (:issue:`602`). * ILNP nonce sizing coverage in ``tests/protocols/internet/test_ipv6_extension_unit.py``, one test per protocol, - asserting the declared length, the exact packed octets and the - construct-pack-parse cycle over ten nonces. The reason the existing suite missed - this is that the only ILNP nonce it ever exercised was ``0xFFFFFF`` -- bit length - 24, an exact multiple of eight, precisely where floor division and the ceiling - agree -- the same blind spot that hid the identical typo in ``numbers.py`` behind - bit lengths 8, 16, 24, 32 and 64. Every new case bar two deliberate controls - therefore has a bit length that is *not* a multiple of eight, several of them - below 256. The table also guards itself: the test asserts that at least six of - its own values stay non-byte-aligned and that one stays below 256, so rounding - them off to convenient constants later cannot quietly disarm the regression - (:issue:`601`). + asserting the declared length, the exact packed octets and the construct-pack-parse + cycle over ten nonces. The existing suite missed the defect because its only ILNP + nonce was ``0xFFFFFF`` (bit length 24, a multiple of eight, where floor division and + the ceiling agree), the blind spot that hid the same typo in ``numbers.py`` behind + bit lengths 8, 16, 24, 32 and 64. Every new case bar two controls therefore has a + bit length that is *not* a multiple of eight, several below 256, and the test + asserts that at least six stay non-byte-aligned and one stays below 256, so rounding + them off later cannot disarm it (:issue:`601`). * ``SOLUTION`` parameter width coverage in ``tests/protocols/internet/test_hip_unit.py``: eight widths asserting the declared length, the reader's acceptance and the construct-pack-parse cycle, plus a 57-bit - case pinning the 20-octet length :rfc:`5201#section-5.2.5` requires of HIPv1. Five - of the eight widths -- 1, 9, 12, 17 and 25 bits -- are deliberately *not* multiples - of eight, because at a multiple of eight the defective ``ceil(bits / 4)`` coincides - with the correct width, which is why every fixture that reached this builder passed - through it unharmed; the round-trip case in ``examples/generators/options.py`` - supplies no ``random`` or ``solution`` at all, so it exercised the formula at zero - bits. 57 bits is what discriminates on the HIPv1 path for the same reason 64 does - not: both formulas give 20 at a full-width value (:issue:`608`). + case pinning the 20-octet length :rfc:`5201#section-5.2.5` requires of HIPv1. + Five widths (1, 9, 12, 17 and 25 bits) are deliberately *not* multiples of eight, + because at a multiple of eight the defective ``ceil(bits / 4)`` coincides with + the correct width, which is why every fixture reaching this builder passed; the + round-trip case in ``examples/generators/options.py`` supplies no ``random`` or + ``solution``, so it exercised the formula at zero bits. 57 bits discriminates on + the HIPv1 path where 64 does not: both formulas give 20 at a full-width value + (:issue:`608`). Changed ~~~~~~~ -* renames with no compatibility alias left behind: PCAP-NG ``Option`` - subclasses spell the namespace class keyword ``ns=`` instead of ``namespace=`` - (:issue:`439`). -* ``tests/protocols/transport/test_tcp_udp_unit.py`` now reaches the - MP_JOIN dispatchers through ``TCP()`` itself, instead of assigning a Python - ``set`` to ``_flags`` on a bare ``TCP.__new__(TCP)``. A ``set`` answers the - membership tests ``_make_mptcp_join`` and ``_read_mptcp_join`` use, so every flag - branch ran and both TCP modules read 100% statement and branch coverage -- while - the attribute had neither the ``aenum.IntFlag`` type production assigns nor the - ordering that governs when it exists at all, which is how :issue:`587` stayed invisible - behind that number and how the ``cast('Enum_Flags', 0)`` no-op behind it went - unnoticed too. Measured on the rewrite, against the 17 tests of that file: revert - :issue:`587`'s hoist and two of them fail with - ``AttributeError: 'TCP' object has no attribute '_flags'`` - where all 17 passed before; restore the ``cast`` and two fail with - ``TypeError: argument of type 'int' is not a container or iterable``, again - where all 17 passed. The library is unchanged and the file's tests still pass, so - the coverage numbers do not move -- the point is what the same numbers are now - worth (:issue:`603`). -* building a protocol through its constructor with a keyword that - names nothing now raises ``UnsupportedCall`` instead of discarding it. **This is - a behaviour - change to a public API**: every ``make`` in the tree ends its signature with - ``**kwargs`` and reads nothing out of it, so until now a misspelled keyword was - accepted, dropped, and the field it named kept its default -- wrong octets, with - nothing said. That is what :issue:`602` cost: ``examples/generators/options.py`` asked - for ``seq=1`` where ``TCP.make`` spells the parameter ``seq_no``, and 25 - generated fixture frames carried sequence number ``0`` against an empty - ``warnings`` list. :issue:`541` and :issue:`556` were the same silence. The schema layer has - never been so permissive -- ``Schema.__update__`` warns ``UnknownFieldWarning`` - for a field it does not know -- and the asymmetry between the two halves of the - same construction is what this closes. Checked in ``ProtocolBase.__init__`` - rather than in ``make``, because ``make`` is not the only consumer of the - keywords it is handed: ``__post_init__`` passes one ``**kwargs`` to the - construction *and* to the parse of what it has just constructed, so a keyword - declared only by ``read`` legitimately travels through ``make`` -- ``HIP.read`` - declares ``extension`` where ``HIP.make`` does not, and the option generator - depends on it. The accepted set is therefore the union of every keyword-taking - parameter of ``make``, ``read``, ``pack``, ``unpack``, ``__post_init__`` and - ``__init__`` across the whole MRO, computed once per class from - ``inspect.signature``. Parsing is deliberately untouched, since there the - keywords are whatever the engines and the four ``_import_next_layer`` - implementations forward and a protocol cannot know which of its ancestors' its - parent passed on -- and nothing was ever lost that way, a dropped parse keyword - changing how a packet is read rather than what its octets say. Two escapes exist - for the shapes a signature cannot express, both opt-in per class through a new - ``__keywords__``: a set, for a keyword read out of ``**kwargs`` by name as - ``ESP.read`` does with ``packet``; and ``None``, for a dispatcher whose real - signature belongs to a class chosen at call time, which is exactly ``HTTP.make`` - forwarding to ``HTTPv1``/``HTTPv2`` and the one place in the tree that uses it. - The message names the near neighbour it found, so ``seq`` reports *did you mean - 'seq_no'?*. ``from_data`` warns ``UnknownFieldWarning`` where a caller would be - raised at, because the keywords there are whatever ``_make_data`` returned - rather than anything anybody typed, so the defect is a key of that mapping - disagreeing with the signature it is spread into and the person who meets it is - not the person who can fix it. Expect this to surface latent bugs in code that - has been quietly losing a field, which is the point. It surfaced four in this - repository, all the residue of :issue:`602` and all fixed here: the ``_TCP_BASE`` of - ``examples/generators/dispatch.py`` and three stale copies of it under - ``tests/protocols/transport/``, each still passing - ``seq``/``ack_flag``/``urgent_pointer`` and so building segments with sequence - number ``0`` where they read as ``1``. It surfaced three more that are reported - rather than fixed, being a defect per protocol rather than one in this mechanism: - ``Frame._make_data`` returns ``ts_src`` where ``make`` declares ``ts_sec``, - ``L2TPv2._make_data`` returns ``prio`` where it declares ``priority``, and - ``Header._make_data`` returns a ``magic_number`` that ``Header.make`` does not - take at all -- so ``from_data`` has been dropping a frame's timestamp, an - L2TPv2 priority bit and a capture's byte order, and now says so. One limitation - worth knowing rather than discovering: a *direct* ``SomeProtocol.make(...)`` - call is not checked and still discards in silence, since the check sits where - every producer's keywords converge rather than inside each of the 30 ``make`` - implementations -- ``object.__new__(cls).make(**kwargs)`` is the idiom that - reaches it, and ``HTTP.make`` uses it to reach its versioned implementation +* renames with no compatibility alias left behind: PCAP-NG ``Option`` subclasses spell + the namespace class keyword ``ns=`` instead of ``namespace=`` (:issue:`439`). +* ``tests/protocols/transport/test_tcp_udp_unit.py`` now reaches the MP_JOIN + dispatchers through ``TCP()`` itself, instead of assigning a Python ``set`` to + ``_flags`` on a bare ``TCP.__new__(TCP)``. A ``set`` answers the membership tests + ``_make_mptcp_join`` and ``_read_mptcp_join`` use, so every flag branch ran and both + TCP modules read 100% coverage, while the attribute had neither the + ``aenum.IntFlag`` type production assigns nor the ordering that governs when it + exists -- how :issue:`587` stayed invisible behind that number, and the + ``cast('Enum_Flags', 0)`` no-op behind it. Reverting :issue:`587`'s hoist now fails + two of the file's 17 tests with + ``AttributeError: 'TCP' object has no attribute '_flags'``, and restoring the + ``cast`` fails two with + ``TypeError: argument of type 'int' is not a container or iterable``; all 17 passed + before. The library is unchanged and coverage does not move -- the point is what the + same numbers are now worth (:issue:`603`). +* building a protocol through its constructor with a keyword that names nothing now + raises ``UnsupportedCall`` instead of discarding it. **This is a behaviour change to + a public API**: every ``make`` in the tree ends its signature with ``**kwargs`` and + reads nothing out of it, so a misspelled keyword was accepted, dropped, and the field + it named kept its default -- wrong octets, nothing said. That is what :issue:`602` + cost: ``examples/generators/options.py`` asked for ``seq=1`` where ``TCP.make`` spells + it ``seq_no``, and 25 generated fixture frames carried sequence number ``0``. + :issue:`541` and :issue:`556` were the same silence. The schema layer was never so + permissive (``Schema.__update__`` warns ``UnknownFieldWarning``), and that asymmetry + is what this closes. The check sits in ``ProtocolBase.__init__`` rather than ``make``, + because ``__post_init__`` passes one ``**kwargs`` to the construction *and* to the + parse of what it has just constructed, so a keyword declared only by ``read`` + legitimately travels through ``make`` (``HIP.read`` declares ``extension``, which + ``HIP.make`` does not, and the option generator depends on it). The accepted set is + the union of every keyword-taking parameter of ``make``, ``read``, ``pack``, + ``unpack``, ``__post_init__`` and ``__init__`` across the MRO, computed once per class + from ``inspect.signature``. Parsing is deliberately untouched: there the keywords are + whatever the engines and the four ``_import_next_layer`` implementations forward, a + protocol cannot know which its parent passed on, and a dropped parse keyword changes + how a packet is read, not what its octets say. Two opt-in escapes exist per class + through a new ``__keywords__``: a set, for a keyword read out of ``**kwargs`` by name + (``ESP.read`` with ``packet``); and ``None``, for a dispatcher whose real signature + belongs to a class chosen at call time (only ``HTTP.make`` forwarding to + ``HTTPv1``/``HTTPv2``). The message names the near neighbour (``seq`` reports *did you + mean 'seq_no'?*). ``from_data`` warns ``UnknownFieldWarning`` rather than raising, + because its keywords are whatever ``_make_data`` returned, so the defect is a key of + that mapping disagreeing with the signature it is spread into, and the person who + meets it cannot fix it. This surfaced four latent bugs in this repository, all + residue of :issue:`602` and fixed here: the ``_TCP_BASE`` of + ``examples/generators/dispatch.py`` and three stale copies under + ``tests/protocols/transport/``, each building segments with sequence number ``0``. + Three more are reported, not fixed, being a defect per protocol: ``Frame._make_data`` + returns ``ts_src`` where ``make`` declares ``ts_sec``, ``L2TPv2._make_data`` returns + ``prio`` where it declares ``priority``, and ``Header._make_data`` returns a + ``magic_number`` that ``Header.make`` does not take, so ``from_data`` has been + dropping a frame's timestamp, an L2TPv2 priority bit and a capture's byte order, and + now says so. One limitation: a *direct* ``SomeProtocol.make(...)`` call is not checked + and still discards silently, since the check sits where every producer's keywords + converge rather than inside each of the 30 ``make`` implementations; + ``object.__new__(cls).make(**kwargs)``, the idiom ``HTTP.make`` uses, reaches it (:issue:`617`). Fixed ~~~~~ -* next-layer, option, chunk, block and parameter dispatch all read - ``defaultdict`` registries, so a lookup miss inserted the key into class-level - state shared by every later instance, after which a legitimate ``register_*`` - call warned that the code was already registered. Every read now goes through - a lookup that does not grow the table, and ``IPv4.__option__`` and - ``HIP.__parameter__`` became inspectable class attributes rather than names - assembled at call time (:pr:`426`, :pr:`428`, :issue:`429`, :pr:`434`). One break comes with it: a - tuple-registered handler pair written to the documented - ``OptionParser``/``OptionConstructor`` signature now works where it could - previously never be called at all, and a pair written with an explicit leading - ``self`` -- the only shape that used to work -- now does not. -* the identical defect one layer up, in the schema layer's own - ``EnumSchema.registry``: ``Option.registry[code]`` for an unregistered - ``code`` inserted the default schema under that code, so a single lookup - made an unassigned TCP option number, e.g. ``156``, read back as registered - for the rest of the process. ``EnumSchema.__enum__`` is now built (or, when a - subclass seeds it manually in its own class body -- ``PCAPNG.Option``'s - namespaced mapping, ``TCP.MPTCP``'s plain one) as a retention-safe mapping - that still returns the registered default on a miss, it just stops recording - it; ``.registry`` keeps returning the same object it always did, so nothing - that held a reference to it is affected (:issue:`555`). -* on Python 3.10 and older, no ``Schema`` subclass got its own - ``_abc_impl``: all of them fell through to ``collections.abc.Mapping``'s, so a - single ``isinstance`` or ``issubclass`` answer poisoned every later question - about that class for the rest of the process. A terminating PCAP-NG - ``EndRecord`` tested ``True`` as an ``IPv4Record`` (:issue:`439`). -* construction, which was broken in several places at once: the - generated typed ``__init__`` was never installed, so ``__post_init__`` did not - run and a schema built from a subset of its fields could not be packed at all - -- ``UDP(srcport=53, dstport=5353)`` now packs (:pr:`430`); IPv6 and Mobility Header - option padding was wrong, leaving construction wholly broken (:pr:`398`); - ``HTTP.make`` called the versioned ``make`` unbound, so every real call raised - ``TypeError`` (:issue:`452`, :pr:`462`); and ``IPv4._make_data`` returned the fragment - offset in octets where the wire wants 8-octet units, and read ``data.options`` - on a packet that has none (:issue:`494`, :pr:`499`). -* a truncated or under-declared area no longer parses - "successfully", and no longer wedges the process. The option and list loops - could spin forever with no exception on a truncated area, reachable from - untrusted input through HOPOPT, IPv6-Opts, MH, HIP and SCTP; each iteration - must now advance the stream by at least one octet, and the error names the - option, the offset and the octets remaining (:issue:`431`, :pr:`432`). Separately, - wire-derived lengths in ``ipv6_opts``, CALIPSO, MPL, REG_INFO and four HIP list - callbacks underflowed below zero, which ``ListField``'s own - ``while length > 0`` then turned into a silent empty list; they are floored and - raise instead (:pr:`449`, :pr:`456`, :pr:`460`, :issue:`463`). -* field widths and units, each measured against the specification - rather than inferred: HIP's ``TRANSPORT_FORMAT_LIST``, ``NAT_TRAVERSAL_MODE`` - and ``ESP_TRANSFORM`` list entries are two octets, not one [:rfc:`7401`, - :rfc:`5770`, :rfc:`7402`] (:issue:`463`, :issue:`472`); the MN-ID option sizes from its - subtype, not from ``identifier``'s Python type (:issue:`448`, :pr:`464`, :issue:`467`); - IPv6-Route's ``Hdr Ext Len`` is computed in 8-octet units on both sides - [:rfc:`8200`] (:issue:`487`, :pr:`489`); and the Fast Binding Update and Acknowledgment +* next-layer, option, chunk, block and parameter dispatch all read ``defaultdict`` + registries, so a lookup miss inserted the key into class-level state shared by + every later instance, after which a legitimate ``register_*`` call warned that + the code was already registered. Every read now goes through a lookup that does + not grow the table, and ``IPv4.__option__`` and ``HIP.__parameter__`` became + inspectable class attributes rather than names assembled at call time + (:pr:`426`, :pr:`428`, :issue:`429`, :pr:`434`). One break: a tuple-registered + handler pair written to the documented ``OptionParser``/``OptionConstructor`` + signature now works where it could never be called, and a pair written with an + explicit leading ``self`` -- the only shape that used to work -- now does not. +* the identical defect one layer up, in the schema layer's ``EnumSchema.registry``: + ``Option.registry[code]`` for an unregistered ``code`` inserted the default + schema under that code, so a single lookup made an unassigned TCP option number + such as ``156`` read back as registered for the rest of the process. + ``EnumSchema.__enum__`` is now built (or, when a subclass seeds it manually, as + ``PCAPNG.Option``'s namespaced mapping and ``TCP.MPTCP``'s plain one do) as a + retention-safe mapping that still returns the registered default on a miss but + stops recording it; ``.registry`` returns the same object as before, so nothing + holding a reference is affected (:issue:`555`). +* on Python 3.10 and older, no ``Schema`` subclass got its own ``_abc_impl``: all + fell through to ``collections.abc.Mapping``'s, so a single ``isinstance`` or + ``issubclass`` answer poisoned every later question about that class for the + rest of the process. A terminating PCAP-NG ``EndRecord`` tested ``True`` as an + ``IPv4Record`` (:issue:`439`). +* construction, broken in several places at once: the generated typed ``__init__`` + was never installed, so ``__post_init__`` did not run and a schema built from a + subset of its fields could not be packed -- ``UDP(srcport=53, dstport=5353)`` now + packs (:pr:`430`); IPv6 and Mobility Header option padding was wrong, leaving + construction wholly broken (:pr:`398`); ``HTTP.make`` called the versioned + ``make`` unbound, so every real call raised ``TypeError`` (:issue:`452`, + :pr:`462`); and ``IPv4._make_data`` returned the fragment offset in octets where + the wire wants 8-octet units, and read ``data.options`` on a packet that has none + (:issue:`494`, :pr:`499`). +* a truncated or under-declared area no longer parses "successfully", and no + longer wedges the process. The option and list loops could spin forever with no + exception on a truncated area, reachable from untrusted input through HOPOPT, + IPv6-Opts, MH, HIP and SCTP; each iteration must now advance the stream by at + least one octet, and the error names the option, the offset and the octets + remaining (:issue:`431`, :pr:`432`). Separately, wire-derived lengths in + ``ipv6_opts``, CALIPSO, MPL, REG_INFO and four HIP list callbacks underflowed + below zero, which ``ListField``'s own ``while length > 0`` turned into a silent + empty list; they are floored and raise instead (:pr:`449`, :pr:`456`, :pr:`460`, + :issue:`463`). +* field widths and units, each measured against the specification: HIP's + ``TRANSPORT_FORMAT_LIST``, ``NAT_TRAVERSAL_MODE`` and ``ESP_TRANSFORM`` list + entries are two octets, not one [:rfc:`7401`, :rfc:`5770`, :rfc:`7402`] + (:issue:`463`, :issue:`472`); the MN-ID option sizes from its subtype, not from + ``identifier``'s Python type (:issue:`448`, :pr:`464`, :issue:`467`); IPv6-Route's + ``Hdr Ext Len`` is computed in 8-octet units on both sides [:rfc:`8200`] + (:issue:`487`, :pr:`489`); and the Fast Binding Update and Acknowledgment Lifetimes are plain seconds [:rfc:`5568`], not :rfc:`6275`'s four-second units (:pr:`502`). -* the last four call sites where a ``_make_*`` method converts an - address itself, ahead of the schema, so the same ``bool``-as-``int`` mistake - reached them too, unguarded by the schema-layer fix above: - ``ARP._make_proto_resolve``, ``IPv6_Route._make_data_type_rpl`` (which also - derives its RPL compression lengths from the laundered value, corrupting the - address list alongside the exception it should have raised), the ``dst`` - parameter of ``IPv6_Route.make``, and the latent ``OSPF._make_id_numbers``, - which nothing calls yet but would have inherited the defect regardless. All - four now go through ``parse_ip_address``, the helper :pr:`539` and :issue:`552` route - their own sites through, completing the programme :pr:`481` and :issue:`508` started. - Pinning ``version=6`` on the two ``IPv6_Route`` sites is a second, smaller - behaviour change beyond bool rejection: both previously converted a plain - integer through the bare, family-inferring ``ipaddress.ip_address``, which - let ``ip=[258]`` pack a 4-octet IPv4 address (``0.0.1.2``) inside an - IPv6-only header; it now packs the 16-octet IPv6 form (``::102``) instead, - which is what an IPv6-only header should hold regardless of what an integer - argument happens to fit as an IPv4 address (:issue:`540`). -* PCAP and PCAP-NG output and parsing: ``bytes(frame)`` returned the - *next* frame's octets, ``files=True`` wrote names like ``Frame 1..json``, and a - PCAP-NG ``timestamp_epoch`` was shifted by the reading host's timezone (:pr:`403`); - seven further parser defects (:issue:`341`--:issue:`347`, :pr:`371`) and, on the write path, four - more block-parsing ones (:pr:`388`); ``BitField`` packed every named bit as set - (:issue:`359`, :pr:`374`); the extension-header walk failed to advance past the last IPv6 - extension header (:issue:`348`, :pr:`373`); and IPv6 fragment offsets went unscaled, with - reassembly keyed on the flow label -- optional, and routinely zero, so distinct - datagrams collapsed together -- rather than on the fragment identification - (:pr:`389`). -* ``TCP._make_mptcp_addaddr`` could not build an ``ADD_ADDR`` - option end to end: its ``kind=``/``length=`` arguments were rejected with - ``UnknownFieldWarning`` and silently dropped, and ``.pack()`` then raised - ``KeyError: 'length'`` from ``port``'s own condition, - ``pkt['length'] in (10, 22)``. The cause was one layer up -- ``MPTCP``, the - base class every Multipath TCP subtype schema inherits, declared ``kind`` and - ``length`` only under ``typing.TYPE_CHECKING`` rather than as real fields, - unlike ``Option``, which every non-Multipath TCP option schema inherits - instead. That silently dropped ``kind=``/``length=`` for every - ``_make_mptcp_*`` constructor, not only ``ADD_ADDR``'s, so ``MPTCP`` now - declares both for real, the same way ``Option`` already did (:issue:`541`). The same - missing fields broke parsing too: with no ``kind``/``length`` fields ahead of - it, a Multipath TCP subtype schema's own leading field read the ``kind`` - octet itself rather than the octet meant for it, an off-by-two in field - alignment rather than a wire-format change -- a correct sender's octets were - always right, only this library's reading of them was shifted. Spec-correct - ``ADD_ADDR`` and ``MP_PRIO`` options failed to parse with - ``FieldError: TCP: [OptNo 30] 3 invalid IP version`` and - ``KeyError: 'length'`` respectively; both parse correctly now. -* two dropped-keyword/wrong-cast defects flagged in review during - this release and never filed until now: HIP's ``_make_param_encrypted`` passed - ``cipher=`` to a schema with no such field, so the value was silently dropped - and an AES-cipher ``ENCRYPTED`` parameter built through ``make`` packed - without its IV; and IPv6-Route's ``RPL.post_process``, which runs on every - ``Schema.pack`` and not only after a parse, assumed ``self.addresses`` was - still the concatenated ``bytes`` a parse leaves it as, and raised slicing the - ``list[bytes]`` a ``make``-built multi-address header actually holds there - (:issue:`556`). -* two more defects :issue:`541` exposed rather than caused, both since it let - construction reach code that had never run before. ``MPTCP.subtype`` was still - ``typing.TYPE_CHECKING``-only, an annotation rather than a field, so +* the last four call sites where a ``_make_*`` method converts an address itself, + ahead of the schema, so the same ``bool``-as-``int`` mistake reached them, + unguarded by the schema-layer fix above: ``ARP._make_proto_resolve``, + ``IPv6_Route._make_data_type_rpl`` (which also derived its RPL compression + lengths from the laundered value, corrupting the address list), the ``dst`` + parameter of ``IPv6_Route.make``, and the latent ``OSPF._make_id_numbers``. All + four now go through ``parse_ip_address``, the helper :pr:`539` and :issue:`552` + route their own sites through, completing the programme :pr:`481` and + :issue:`508` started. Pinning ``version=6`` on the two ``IPv6_Route`` sites is a + second, smaller behaviour change: both converted a plain integer through the + family-inferring ``ipaddress.ip_address``, so ``ip=[258]`` packed a 4-octet IPv4 + address (``0.0.1.2``) inside an IPv6-only header; it now packs the 16-octet + ``::102`` (:issue:`540`). +* PCAP and PCAP-NG output and parsing: ``bytes(frame)`` returned the *next* + frame's octets, ``files=True`` wrote names like ``Frame 1..json``, and a PCAP-NG + ``timestamp_epoch`` was shifted by the reading host's timezone (:pr:`403`); seven + further parser defects (:issue:`341`--:issue:`347`, :pr:`371`) and, on the write + path, four more block-parsing ones (:pr:`388`); ``BitField`` packed every named + bit as set (:issue:`359`, :pr:`374`); the extension-header walk failed to advance + past the last IPv6 extension header (:issue:`348`, :pr:`373`); and IPv6 fragment + offsets went unscaled, with reassembly keyed on the flow label -- optional, and + routinely zero, so distinct datagrams collapsed together -- rather than on the + fragment identification (:pr:`389`). +* ``TCP._make_mptcp_addaddr`` could not build an ``ADD_ADDR`` option end to end: its + ``kind=``/``length=`` arguments were rejected with ``UnknownFieldWarning`` and + dropped, and ``.pack()`` then raised ``KeyError: 'length'`` from ``port``'s own + condition, ``pkt['length'] in (10, 22)``. The cause was one layer up: ``MPTCP``, the + base of every Multipath TCP subtype schema, declared ``kind`` and ``length`` only + under ``typing.TYPE_CHECKING`` rather than as real fields, unlike ``Option``. That + dropped ``kind=``/``length=`` for every ``_make_mptcp_*`` constructor, so ``MPTCP`` + now declares both for real (:issue:`541`). The missing fields broke parsing too: a + subtype schema's leading field read the ``kind`` octet itself, an off-by-two in + field alignment rather than a wire-format change (a correct sender's octets were + always right). Spec-correct ``ADD_ADDR`` and ``MP_PRIO`` options failed with + ``FieldError: TCP: [OptNo 30] 3 invalid IP version`` and ``KeyError: 'length'``; + both parse now. +* two dropped-keyword/wrong-cast defects flagged in review during this release and + never filed until now: HIP's ``_make_param_encrypted`` passed ``cipher=`` to a + schema with no such field, so an AES-cipher ``ENCRYPTED`` parameter built through + ``make`` packed without its IV; and IPv6-Route's ``RPL.post_process``, which runs + on every ``Schema.pack``, assumed ``self.addresses`` was still the concatenated + ``bytes`` a parse leaves, and raised slicing the ``list[bytes]`` a ``make``-built + multi-address header holds (:issue:`556`). +* two more defects :issue:`541` exposed by letting construction reach code that had + never run. ``MPTCP.subtype`` was still ``typing.TYPE_CHECKING``-only, so ``TCP(options=[(Enum_Option.Multipath_TCP, ...)])`` raised ``AttributeError: ... has no attribute 'subtype'`` for every subtype but - ``MP_JOIN``: the convenience constructor builds a schema in memory and reads it - straight back through ``_read_mptcp_*`` with no byte round trip, so - ``_MPTCP.post_process`` -- the only code that ever set ``subtype`` -- never ran. - Fixed on the construction path (``TCP._make_mode_mp``) rather than by adding a - third real field the way ``kind``/``length`` got in :issue:`541`: unlike those two, - ``subtype`` is already packed as 4 bits of each subtype's own ``test`` - bitfield, and a second, independent field for the same bits would either - double-encode them or need a "derive, don't pack" field kind this library does - not have (:issue:`566`). Separately, ``_make_mptcp_capable`` wrote - ``length=20 if rkey is None else 32`` where :rfc:`8684` section 3.1 gives 12 and - 20, and ``MPTCPCapable.rkey``'s own condition (``pkt['length'] != 32``) dropped - the receiver's key for exactly the length the maker used to mean "key present" - -- so a spec-correct, key-absent MP_CAPABLE could not be built at all, and a - key-present one silently lost its key on the wire. Both, and the matching guard - in ``_read_mptcp_capable``, now agree on 12/20. **This changes MP_CAPABLE's - packed output**: a 20-octet, key-present option built or parsed under the old - code becomes 12 octets with no key, or 20 octets with the key actually present, + ``MP_JOIN``: the convenience constructor reads the schema straight back through + ``_read_mptcp_*`` with no byte round trip, so ``_MPTCP.post_process``, the only code + that set ``subtype``, never ran. Fixed on the construction path + (``TCP._make_mode_mp``) rather than by adding a third real field the way + ``kind``/``length`` got in :issue:`541`: ``subtype`` is already packed as 4 bits of + each subtype's own ``test`` bitfield, and a second field for the same bits would + double-encode them or need a "derive, don't pack" field kind this library lacks + (:issue:`566`). Separately, ``_make_mptcp_capable`` wrote + ``length=20 if rkey is None else 32`` where :rfc:`8684` section 3.1 gives 12 and 20, + and ``MPTCPCapable.rkey``'s condition (``pkt['length'] != 32``) dropped the + receiver's key for exactly the length the maker used for "key present": a key-absent + MP_CAPABLE could not be built, and a key-present one lost its key on the wire. Both, + and the guard in ``_read_mptcp_capable``, now agree on 12/20. **This changes + MP_CAPABLE's packed output**: a 20-octet key-present option built or parsed under + the old code becomes 12 octets with no key, or 20 octets with the key present, depending on which the caller meant (:issue:`567`). -* IPv6-Route's RPL routing data was five octets wide at the front - where :rfc:`6554` section 3 gives four, and three further defects sat - stacked behind it. ``CmprI``, ``CmprE`` and ``Pad`` are each a 4-bit field, - sharing one 32-bit word with a 20-bit ``Reserved``, but the schema declared - ``cmpr_i`` and ``cmpr_e`` as whole octets -- so a constructed two-address - header packed to 41 octets while the ``Hdr Ext Len`` of 5 derived from that - inflated data area declared 48. Correcting the width is a **wire-format - change**, and it reshapes the schema: ``RPL(cmpr_i=..., cmpr_e=...)`` is now - ``RPL(cmpr={'cmpr_i': ..., 'cmpr_e': ...})``, beside the - ``pad={'pad_len': ...}`` that was already there. Behind it, the reader's - ``header.length % 16`` guard read ``Hdr Ext Len`` as an octet count and - assumed 16-octet addresses, which an SRH only carries when ``CmprI`` and - ``CmprE`` are both 0 -- the unit confusion :issue:`487` fixed for Source Route and - Type 2, flagged and deliberately left by :pr:`489` for want of a working RPL - round trip to validate a replacement against. It is replaced by section - 4.2's own address-count arithmetic, - ``n = (((Hdr Ext Len * 8) - Pad - (16 - CmprE)) / (16 - CmprI)) + 1``, which - the reader now requires to close -- non-negative and whole. - ``RPL.post_process`` subtracted ``pad_len`` a second time from a buffer - whose own length callback had already taken it off, losing one address per - ``16 - CmprI`` octets of padding, and set ``ip`` only when it had parsed - octets -- so once the guard stopped rejecting every constructed header, - ``IPv6_Route(type=..., data={'ip': [...]})`` raised a bare - ``AttributeError: 'RPL' object has no attribute 'ip'`` from the reader. And - ``_make_data_type_rpl`` computed ``Pad`` as ``8 - length % 8`` without the - outer ``% 8``, so an already-aligned address vector was handed a full 8 - octets of padding where section 3 requires that when ``CmprI`` and ``CmprE`` - are both 0, ``Pad`` MUST carry a value of 0. The narrower ``cmpr_i`` also - removes a latent divide-by-zero: a whole octet could hold 16, making - ``16 - CmprI`` zero, where a 4-bit field tops out at 15. - ``ipv6-route-type/RPL_Source_Route_Header`` round-trips now and its - ``EXPECTED_FAILURES`` entry is deleted; as with the guard it replaces, none - of this has been checked against a real RPL capture (:issue:`564`). -* ``L2TPv2`` parsed any version nibble, so an L2TPv3 datagram was - reported as v2 with a tunnel and session ID read out of v3's Control - Connection ID. :rfc:`2661` §3.1 fixes ``Ver`` at 2 and reserves 1 for L2F, and - ``L2TPv2.version`` already documented that "a datagram carrying any other - value is a different protocol reached through a different class" -- but nothing - enforced it, so the hard-coded ``Literal[2]`` property and ``info.version`` - disagreed on the same octets, answering 2 and 3. ``read`` now raises - ``ProtocolError``, which degrades the payload to ``Raw`` through the existing +* IPv6-Route's RPL routing data was five octets wide at the front where :rfc:`6554` + section 3 gives four, with three further defects behind it. ``CmprI``, ``CmprE`` and + ``Pad`` are each 4 bits sharing a 32-bit word with a 20-bit ``Reserved``, but the + schema declared ``cmpr_i`` and ``cmpr_e`` as whole octets, so a constructed + two-address header packed to 41 octets while its ``Hdr Ext Len`` of 5 declared 48. + Correcting the width is a **wire-format change** and reshapes the schema: + ``RPL(cmpr_i=..., cmpr_e=...)`` is now ``RPL(cmpr={'cmpr_i': ..., 'cmpr_e': ...})``, + beside the existing ``pad={'pad_len': ...}``. Behind it, the reader's + ``header.length % 16`` guard read ``Hdr Ext Len`` as an octet count and assumed + 16-octet addresses, which an SRH carries only when ``CmprI`` and ``CmprE`` are both + 0 -- the unit confusion :issue:`487` fixed for Source Route and Type 2, left by + :pr:`489` for want of a working RPL round trip to validate against. It is replaced + by section 4.2's address-count arithmetic, + ``n = (((Hdr Ext Len * 8) - Pad - (16 - CmprE)) / (16 - CmprI)) + 1``, which the + reader now requires to be non-negative and whole. ``RPL.post_process`` subtracted + ``pad_len`` a second time from a buffer whose length callback had already taken it + off, losing one address per ``16 - CmprI`` octets of padding, and set ``ip`` only + when it had parsed octets, so ``IPv6_Route(type=..., data={'ip': [...]})`` raised a + bare ``AttributeError: 'RPL' object has no attribute 'ip'``. And + ``_make_data_type_rpl`` computed ``Pad`` as ``8 - length % 8`` without the outer + ``% 8``, so an aligned address vector got a full 8 octets of padding where section 3 + requires ``Pad`` to be 0 when ``CmprI`` and ``CmprE`` are both 0. The narrower + ``cmpr_i`` also removes a latent divide-by-zero (a whole octet could hold 16, making + ``16 - CmprI`` zero). ``ipv6-route-type/RPL_Source_Route_Header`` round-trips and + its ``EXPECTED_FAILURES`` entry is deleted; none of this has been checked against a + real RPL capture (:issue:`564`). +* ``L2TPv2`` parsed any version nibble, so an L2TPv3 datagram was reported as v2 + with a tunnel and session ID read out of v3's Control Connection ID. :rfc:`2661` + §3.1 fixes ``Ver`` at 2 and reserves 1 for L2F, and ``L2TPv2.version`` documented + that any other value "is a different protocol reached through a different class", + but nothing enforced it, so the hard-coded ``Literal[2]`` property and + ``info.version`` answered 2 and 3 on the same octets. ``read`` now raises + ``ProtocolError``, degrading the payload to ``Raw`` through the existing ``beholder`` path with the reason recorded. **This affects real captures**: :rfc:`3931` §4.1.2 puts L2TPv3 on port 1701 too, the port ``UDP.__proto__`` - already binds, so ``Ethernet:IPv4:UDP:L2TPv2:Raw`` with invented field values - becomes ``Ethernet:IPv4:UDP:Raw`` with the octets preserved. + binds, so ``Ethernet:IPv4:UDP:L2TPv2:Raw`` with invented field values becomes + ``Ethernet:IPv4:UDP:Raw`` with the octets preserved. - This is the resolution of :issue:`548`, which reported ``TransType.L2TP`` (115) as - registered nowhere and proposed binding ``L2TPv2`` there. That binding is - wrong rather than merely awkward: :rfc:`3931` §4.1.1 gives 115 to *L2TPv3 over - IP*, whose session header is "free of any restrictions imposed by coexistence - with L2TPv2 and L2F" and carries **no version nibble at all**, so a v2 parser - cannot even detect that the datagram is not its own. Measured, it produced - ``version=4``, ``tunnelid=0x5678`` and ``sessionid=0xff03`` from the top half - of a Session ID and two octets of the PPP frame behind it. 115 is a missing - *class*, not a missing registration, and stays unbound until an ``L2TPv3`` - class exists; no dissector was invented here to fill it. The reasoning is now - recorded in ``pcapkit.protocols.link.l2tp`` rather than only in a test, and - ``register_protocol_code``'s worked example -- which named ``L2TPv2`` at 115 -- - names ``L2TPv3`` instead (:issue:`548`). + This resolves :issue:`548`, which reported ``TransType.L2TP`` (115) as registered + nowhere and proposed binding ``L2TPv2`` there. That binding is wrong rather than + awkward: :rfc:`3931` §4.1.1 gives 115 to *L2TPv3 over IP*, whose session header + carries **no version nibble at all**, so a v2 parser cannot detect that the + datagram is not its own. 115 is a missing *class*, not a missing registration, and stays + unbound until an ``L2TPv3`` class exists; no dissector was invented to fill it. + The reasoning is recorded in ``pcapkit.protocols.link.l2tp``, and + ``register_protocol_code``'s worked example, which named ``L2TPv2`` at 115, names + ``L2TPv3`` instead (:issue:`548`). * building any ``MP_JOIN`` option raised - ``AttributeError: 'TCP' object has no attribute '_flags'``. ``TCP.make`` - constructed the options *before* it assigned the ``self._flags`` that the option - makers read, and ``_make_mptcp_join`` branches on that attribute to choose - between the three layouts :rfc:`8684` section 3.2 gives for MP_JOIN -- figure 5 - for SYN at 12 octets, figure 6 for SYN/ACK at 16, figure 7 for ACK at 24. The - parse path was never affected: ``read`` assigns the flags before it parses the - options, so the identical branches in ``_read_mptcp_join`` always had them. Fixed - by hoisting the flag resolution above the ``_make_tcp_options`` call, leaving only - the header's data-offset computation -- which genuinely needs the options' total - length -- after it. **Not** fixed by giving ``_flags`` a zero default, which would - have been worse: ``Protocol.pack`` is public and calls ``make``, so an instance - that had already parsed a segment always had the attribute set, and it silently - built the option for the segment it had *read* rather than the one it was asked to - write. Measured pre-fix, a parsed MP_JOIN-SYN instance asked to pack an - MP_JOIN-ACK segment emitted an ACK header carrying figure 5's 12-octet SYN option, - with the caller's 20-octet HMAC -- the whole authentication payload of figure 7's - form -- replaced by an all-zero phantom token and nonce, and nothing raised. The - crash was the benign symptom; a default value fixes only that and leaves the - silent corruption. Hoisting also made one branch reachable for the first time, an - MP_JOIN asked for on a segment with neither SYN nor ACK set: the accumulator was - seeded with ``cast('Enum_Flags', 0)``, and ``typing.cast`` being a runtime no-op, - ``self._flags`` stayed a plain ``int`` on which the first membership test raised - ``TypeError`` rather than the ``ProtocolError`` the method documents. It is now - seeded with ``Enum_Flags(0)``, a real flagless member that compares equal to ``0`` - and ORs identically. The read path's own seed is deliberately unchanged -- a - flagless MP_JOIN is rejected by ``mptcp_data_selector`` before - ``_read_mptcp_join`` runs, so it cannot reach those branches (:issue:`587`). -* the ILNP Nonce option builders in ``HOPOPT`` and ``IPv6_Opts`` sized - the option with ``math.ceil(nonce.bit_length() // 8)``, which is floor division - dressed up as a ceiling: ``//`` floors, and ``math.ceil`` of an ``int`` is a - no-op, so the ceiling was never actually taken. The nonce is packed by a - ``NumberField`` whose width *is* that declared ``len``, so an under-declared - length did not merely mis-state the option -- it silently truncated the nonce on - the wire, with nothing raised. Every nonce whose bit length was not an exact - multiple of eight was affected, and **small values were the worst case rather - than boundary values**: any nonce below 256 was declared as *zero* octets and - dropped from the packet altogether, so ``nonce=9`` packed to ``b'\x8b\x00'`` and - parsed back as ``0``, while ``nonce=256`` and ``nonce=65536`` each truncated to - ``0`` as well. Fixed to ``max(1, math.ceil(nonce.bit_length() / 8))``, the form - already used at five other sizing sites across ``hip.py`` and ``mh.py``. The - one-octet floor is the ``mh.py`` convention and is load-bearing here because - ``nonce`` defaults to ``0``, whose bit length is ``0``: without it the default - argument builds an ILNP Nonce option carrying no Nonce Value field at all, - collapsing "the nonce is 0" into "there is no nonce" when :rfc:`6744` gives the - option that field. The read path was never affected, since it takes the width - from the ``len`` octet on the wire rather than recomputing it (:issue:`601`). -* ``httpv1``'s ``_RE_METHOD`` was unanchored and ``re.match`` - anchors only at the start, so it prefix-matched, and the request-line reader - then passed the whole ``para1`` to ``Method.get`` rather than the captured - ``method`` group. Together those meant ``b'Get'`` matched on the single - character ``G``, satisfied the guard that decides a start-line is a request, - and handed the entire mixed-case token to a lookup that raised on it. Fixing - either half alone still gives a wrong answer -- normalising the lookup would - parse ``b'Get'`` as ``GET`` off a one-character match, and passing the group - would parse it as a method named ``G``. The pattern is now anchored at both - ends and the captured group is what is looked up, so a token that is not a - method is a malformed request line rather than a mis-parsed one. Method tokens - are case-sensitive per :rfc:`9110#section-9.1`, so no ``re.I`` was added: - ``GET`` parses, ``Get`` and ``get`` are rejected (:issue:`583`). -* ``_RE_STATUS`` in the same reader carried the same unanchored - prefix defect, found by auditing ``_RE_METHOD``'s siblings, and it escaped as - the wrong exception type. That pattern is only a guard -- the value is taken - from ``int(para2)`` on the raw token -- so a prefix match let a malformed - status past the guard and then out of ``int()`` uncaught, where - ``_read_http_header`` documents ``ProtocolError``. Measured: a status of - ``200x`` raised - ``ValueError: invalid literal for int() with base 10: b'200x'``, and one of - ``2000`` raised ``ValueError: 2000 is not a valid StatusCode``; - both are now ``ProtocolError``. - :rfc:`9112#section-4` gives ``status-code = 3DIGIT``, exactly three, so the - anchor is what the grammar already said -- the production lives in HTTP/1.1 - because ``status-code`` is part of its ``status-line``, while - :rfc:`9110#section-15` covers the code semantics and the IANA registry rather - than the syntax. ``_RE_VERSION`` was audited at the same time and is safe as - it stands, because both of its call sites read the captured group rather than - the raw token (:issue:`583`). + ``AttributeError: 'TCP' object has no attribute '_flags'``. ``TCP.make`` constructed + the options *before* assigning the ``self._flags`` that the option makers read, and + ``_make_mptcp_join`` branches on it to choose among the three layouts of :rfc:`8684` + section 3.2 (figure 5, SYN, 12 octets; figure 6, SYN/ACK, 16; figure 7, ACK, 24). + The parse path was never affected, since ``read`` assigns the flags first. Fixed by + hoisting the flag resolution above the ``_make_tcp_options`` call, leaving only the + data-offset computation, which needs the options' total length, after it. **Not** + fixed by a zero default for ``_flags``, which would have been worse: + ``Protocol.pack`` is public and calls ``make``, so an instance that had parsed a + segment had the attribute set and silently built the option for the segment it had + *read*. A parsed MP_JOIN-SYN instance asked to pack an MP_JOIN-ACK segment emitted + an ACK header carrying figure 5's 12-octet SYN option, with the caller's 20-octet + HMAC replaced by an all-zero token and nonce, and nothing raised. Hoisting also made + one branch reachable for the first time, an MP_JOIN on a segment with neither SYN + nor ACK: the accumulator was seeded with ``cast('Enum_Flags', 0)``, and + ``typing.cast`` being a runtime no-op, ``self._flags`` stayed a plain ``int`` and + the first membership test raised ``TypeError`` rather than the documented + ``ProtocolError``. It is now seeded with ``Enum_Flags(0)``, a real flagless member + that equals ``0`` and ORs identically. The read path's seed is unchanged: a flagless + MP_JOIN is rejected by ``mptcp_data_selector`` before ``_read_mptcp_join`` runs + (:issue:`587`). +* the ILNP Nonce option builders in ``HOPOPT`` and ``IPv6_Opts`` sized the option + with ``math.ceil(nonce.bit_length() // 8)``, floor division dressed as a ceiling + (``//`` floors and ``math.ceil`` of an ``int`` is a no-op). The nonce is packed by + a ``NumberField`` whose width *is* the declared ``len``, so an under-declared + length silently truncated the nonce on the wire. Every nonce whose bit length was + not a multiple of eight was affected, and **small values were the worst case + rather than boundary values**: any nonce below 256 was declared as *zero* octets + and dropped (``nonce=9`` packed to ``b'\x8b\x00'`` and parsed back as ``0``). Fixed to + ``max(1, math.ceil(nonce.bit_length() / 8))``, the form used at five other sizing + sites in ``hip.py`` and ``mh.py``. The one-octet floor is the ``mh.py`` + convention and is load-bearing because ``nonce`` defaults to ``0``, whose bit + length is ``0``: without it the default builds an option with no Nonce Value + field at all, collapsing "the nonce is 0" into "there is no nonce" when + :rfc:`6744` gives the option that field. The read path takes its width from the + ``len`` octet on the wire and was never affected (:issue:`601`). +* ``httpv1``'s ``_RE_METHOD`` was unanchored and ``re.match`` anchors only at the + start, so it prefix-matched, and the request-line reader passed the whole + ``para1`` to ``Method.get`` rather than the captured ``method`` group. So + ``b'Get'`` matched on the single character ``G``, satisfied the guard that decides + a start-line is a request, and handed the mixed-case token to a lookup that raised + on it. Fixing either half alone still gives a wrong answer (normalising the lookup + would parse ``b'Get'`` as ``GET`` off a one-character match; passing the group + would parse it as a method named ``G``). The pattern is anchored at both ends and + the captured group is looked up, so a non-method token is a malformed request line + rather than a mis-parsed one. Method tokens are case-sensitive per + :rfc:`9110#section-9.1`, so no ``re.I`` was added: ``GET`` parses, ``Get`` and + ``get`` are rejected (:issue:`583`). +* ``_RE_STATUS`` in the same reader had the same unanchored prefix defect, found by + auditing ``_RE_METHOD``'s siblings, and escaped as the wrong exception type. The + pattern is only a guard (the value comes from ``int(para2)`` on the raw token), so a + prefix match let a malformed status past the guard and out of ``int()`` uncaught, + where ``_read_http_header`` documents ``ProtocolError``: ``200x`` raised + ``ValueError: invalid literal for int()`` and ``2000`` raised + ``ValueError: 2000 is not a valid StatusCode``; both are now ``ProtocolError``. + :rfc:`9112#section-4` gives ``status-code = 3DIGIT``, exactly three, so the anchor + is what the grammar said (the production lives in HTTP/1.1 because ``status-code`` + is part of its ``status-line``; :rfc:`9110#section-15` covers code semantics and the + registry). ``_RE_VERSION`` was audited and is safe, since both call sites read the + captured group (:issue:`583`). * as a consequence of the above, the unhandled ``MemoryError`` at ``pcapkit/protocols/protocol.py:1411`` on a truncated PCAP-NG capture. ``dhcp_little_endian.pcapng`` cut to 161, 641 or 1389 octets left a one-octet read of a little-endian 32-bit block length, which ``rjust()`` turned into - ``0x78000000`` or ``0x84000000`` -- 1.88 to 2.06 GiB -- and which was then - passed straight to ``self._file.read()`` as an allocation size, from a - 1772-octet file. Under a 1 GiB address-space cap all three raise - ``MemoryError`` there; with more address space the allocation succeeds and the - parse goes on to fail anyway, so the visible symptom depended on how much - memory the process could get. With ``ljust()`` the same reads report 120 and - 132, no large allocation is attempted, and all three end in the ordinary, - already-handled parse failure instead (:issue:`604`). -* reading a big-endian classic PCAP byte-swapped every record header - field, and then crashed. ``Frame.unpack`` seeded the file's declared byte order - under the key ``bytesorder`` where the frame schema's ``byteorder_callback`` - reads ``byteorder``, so the lookup never found it and always fell back to - ``sys.byteorder`` -- the reading host's order rather than the file's. On a - little-endian host reading a little-endian capture that fallback gives the - right answer by coincidence, and every capture in this repository was - little-endian, so the wrong code path has always produced correct results. - Against a big-endian capture, measured before the fix, frame 1 of - ``big_endian.pcap`` read ``ts_sec=3106905``, ``ts_usec=1088553216`` and - ``incl_len=1241513984`` for a record whose real values are ``1500000000``, - ``123456`` and ``74``, dating the frame to 1970-02-05 rather than to - 2017-07-14. ``incl_len`` is the payload length, so that first record - then consumed the whole file and the second was read with a negative payload - length, raising ``ValueError: read length must be non-negative or -1`` out of - the schema -- which is the reported crash, and it is the *second* symptom - rather than the first. The sibling ``Frame.pack`` eleven lines earlier spelled - the key correctly, which is what marks this as a slip rather than a second key, - and the fallback is what made a misspelled key indistinguishable from an absent - one; ``byteorder_callback`` now records that it is the definition of the key - and why the fallback hides a typo (:issue:`605`). -* documentation. ``mptcp_dss_ack_selector``'s note said a corrected - field-width lambda "would not have worked" and that fixing it belonged to - ``pcapkit.corekit.fields.numbers``, which is exactly where :pr:`598` then fixed it; the - same paragraph sat in ``test_tcp_mptcp_length_arithmetic_unit.py``'s module - docstring, whose other stale claim was that MP_JOIN "cannot be built through the - public ``TCP()`` constructor at all", true only until :issue:`587`. A callable-length - ``NumberField`` packs and unpacks both DSS widths now, and wire *absence* was - never the obstacle either: ``MPTCPDSS.ssn``, ``dl_len`` and ``checksum`` have - always been ``ConditionalField`` on the sibling ``M`` flag, so the class already - relied on that wrapper to keep a field off the wire. The ``SwitchField`` form is - kept for the narrower reason the note now gives -- ``ConditionalField``'s - ``length`` forwards to the wrapped field without consulting the condition, so it - is safe here only because ``Schema.pack`` and ``Schema.unpack`` special-case that - wrapper by name, whereas a ``SwitchField`` always resolves to a concrete field. - Replacing it would be a behaviour change and is not made (:issue:`603`). -* the one assertion :issue:`604` left pinning the old padding side, which had - been red on ``mainline`` since :pr:`621` merged. + ``0x78000000`` or ``0x84000000`` (1.88 to 2.06 GiB) and passed straight to + ``self._file.read()`` as an allocation size, from a 1772-octet file. Under a + 1 GiB address-space cap all three raise ``MemoryError``; with more space the + parse fails anyway. With ``ljust()`` the same reads report 120 and 132, no + large allocation is attempted, and all three end in the ordinary handled + parse failure (:issue:`604`). +* reading a big-endian classic PCAP byte-swapped every record header field, and then + crashed. ``Frame.unpack`` seeded the file's byte order under the key ``bytesorder`` + where the schema's ``byteorder_callback`` reads ``byteorder``, so the lookup never + found it and fell back to ``sys.byteorder``, the host's order rather than the + file's. On a little-endian host reading a little-endian capture that is right by + coincidence, and every capture in this repository was little-endian. Frame 1 of + ``big_endian.pcap`` read ``incl_len=1241513984`` for a record whose real value is + ``74``. ``incl_len`` is the payload length, so that record consumed the whole file + and the second was read with a negative length, raising + ``ValueError: read length must be non-negative or -1`` -- the reported crash, and + the *second* symptom. The sibling ``Frame.pack`` eleven lines earlier spelled the + key correctly, marking this a slip rather than a second key, and the fallback made a + misspelled key indistinguishable from an absent one; ``byteorder_callback`` now + records that it defines the key and why the fallback hides a typo (:issue:`605`). +* documentation. ``mptcp_dss_ack_selector``'s note said a corrected field-width + lambda "would not have worked" and that fixing it belonged to + ``pcapkit.corekit.fields.numbers``, which is where :pr:`598` then fixed it; the + same paragraph sat in ``test_tcp_mptcp_length_arithmetic_unit.py``'s docstring, + whose other stale claim was that MP_JOIN "cannot be built through the public + ``TCP()`` constructor at all", true only until :issue:`587`. A callable-length + ``NumberField`` packs and unpacks both DSS widths now, and wire *absence* was never + the obstacle: ``MPTCPDSS.ssn``, ``dl_len`` and ``checksum`` have always been + ``ConditionalField`` on the sibling ``M`` flag. The ``SwitchField`` form is kept + for the narrower reason the note now gives: ``ConditionalField``'s ``length`` + forwards to the wrapped field without consulting the condition, so it is safe here + only because ``Schema.pack`` and ``Schema.unpack`` special-case that wrapper by + name, whereas a ``SwitchField`` always resolves to a concrete field. Replacing it + would be a behaviour change and is not made (:issue:`603`). +* the one assertion :issue:`604` left pinning the old padding side, red on ``mainline`` + since :pr:`621` merged. ``TCPUDPUnitTests.test_a_truncated_option_still_parses_its_declared_length`` - expected a truncated TCP option's ``data`` as the synthesised zero octets - *followed by* the real ones, which is what ``rjust()`` produced; :pr:`621` made the - padding ``ljust()`` everywhere but could not retarget this file, since another - change (:pr:`612`) owned it at the time and editing it concurrently risked discarding - work that has since landed. The real octets now come first for both parametrised - widths, and the docstring above the assertion says tail-padding rather than - left-padding. Test-only: no library code changes, and - the sibling case in ``tests/protocols/internet/test_ipv4_unit.py`` was already - retargeted in :pr:`621`. Measured against ``main`` at ``2221c2d8f``: two subtest - failures before, none after (:issue:`604`). -* ``main`` went red the moment :issue:`604`'s ``ljust()`` landed, because - ``TCPUDPUnitTests.test_a_truncated_option_still_parses_its_declared_length`` - still pinned the head-padded short read that fix removed. The - ``Reserved_79`` option declaring ``length=12`` over 6 real octets now reports - ``aabbccddeeff00000000`` where the test expected ``00000000aabbccddeeff``, so - both subtests -- ``declared_length=12`` and ``=32`` -- failed on that one - assertion while the 17 other cases in the file stayed green: the parse itself - never changed, only which end the synthesised zeros sit at. The expectation - is inverted, and the docstring above it -- which said the short read was - *left*-padded and described the value as four zero octets followed by the six - real ones -- is corrected to match, since a docstring that contradicts its - own assertion is how the stale expectation survived in the first place. The - inputs do discriminate: ``trailing`` is non-zero and the pad width is 4 and - 24, so neither subtest would hold under the other order. :pr:`621` left this file - alone deliberately, because :pr:`612` owned it at the time, and merged two minutes - ahead of the cross-review verdict that named it (:issue:`604`, :pr:`621`). + expected a truncated TCP option's ``data`` as the synthesised zero octets *followed + by* the real ones, as ``rjust()`` produced; :pr:`621` made the padding ``ljust()`` + everywhere but could not retarget this file, since :pr:`612` owned it and editing + concurrently risked discarding work that has since landed. The real octets now come + first for both parametrised widths, and the docstring, which said the short read was + *left*-padded, is corrected. Test-only; the sibling case in + ``tests/protocols/internet/test_ipv4_unit.py`` was retargeted in :pr:`621` (two + subtest failures before, none after; :issue:`604`). +* ``main`` went red the moment :issue:`604`'s ``ljust()`` landed, because that test + still pinned the head-padded short read: ``Reserved_79`` declaring ``length=12`` over + 6 real octets now reports ``aabbccddeeff00000000`` where the test expected + ``00000000aabbccddeeff``, failing both subtests (``declared_length=12`` and ``=32``) + while the 17 other cases stayed green. The inputs discriminate: ``trailing`` is + non-zero and the pad width is 4 and 24. :pr:`621` left the file alone because + :pr:`612` owned it at the time, and merged two minutes ahead of the cross-review + verdict that named it (:issue:`604`, :pr:`621`). * the HIP ``SOLUTION`` builder sized the parameter with - ``4 + math.ceil(max(random.bit_length(), solution.bit_length()) / 4)``, which is - an invalid shorthand for the two fields it actually has to describe. - :rfc:`7401#section-5.2.5` gives the parameter's ``Length`` as + ``4 + math.ceil(max(random.bit_length(), solution.bit_length()) / 4)``, an invalid + shorthand for two fields. :rfc:`7401#section-5.2.5` gives ``Length`` as ``4 + RHASH_len / 4`` over a ``Random #I`` and a ``Puzzle solution #J`` of ``RHASH_len / 8`` octets **each**, and ``/ 4`` equals twice ``/ 8`` only because - ``RHASH_len`` -- the natural output length of a hash function, in bits -- is a - whole number of octets. Applied to an arbitrary ``bit_length()`` the identity - fails, and the builder emitted an *odd* contents width, which its own reader then - refused: ``_read_param_solution``'s ``(schema.len - 4) % 2`` guard exists because - ``SolutionParameter`` splits that width into two equal ``(len - 4) // 2`` halves, - so an odd width cannot be framed at all. ``random=0x1`` with ``solution=0xfff`` - declared ``len=7`` and raised + ``RHASH_len`` is a whole number of octets. For an arbitrary ``bit_length()`` the + builder emitted an *odd* contents width, which its own reader refused: + ``_read_param_solution``'s ``(schema.len - 4) % 2`` guard exists because + ``SolutionParameter`` splits that width into two equal halves. ``random=0x1`` with + ``solution=0xfff`` declared ``len=7`` and raised ``ProtocolError: HIPv2: [ParamNo 321] invalid format`` on input the library had - itself produced. The undersized length was the - worse half of it: because both fields take their width from that same ``len``, - ``solution=0xfff`` was packed into the one octet it allowed and came back as - ``0xff`` -- silent truncation, nothing raised. Fixed to - ``4 + 2 * math.ceil(max(...) / 8)``, the form the sibling ``PUZZLE`` builder - already uses, which is even by construction and so can never trip the guard. - Loosening the reader was rejected as the alternative: an odd contents width has - no meaning in the wire format, since the RFC makes the two fields equal-width and - says nothing about which would take an extra octet, and a laxer guard would still - have truncated the value (:issue:`608`). -* ``TCP.read`` seeded its connection-flag accumulator with - ``cast('Enum_Flags', 0)``. ``typing.cast`` is a runtime no-op -- it returns its second - argument unchanged -- so the accumulator began life as the plain ``int`` ``0``, and the - ``|=`` that promotes it to a ``pcapkit.const.tcp.flags.Flags`` member is the only thing - that ever did. A segment whose flags octet is all zero, an nmap NULL scan among them, - promoted nothing and left ``self._flags`` an ``int``, so a membership test against it - raised ``TypeError: argument of type 'int' is not a container or iterable`` rather than - answering, and ``TCP.connection`` returned an ``int`` where both that property and - ``Data_TCP.connection`` annotate ``Flags``. Any flag at all masked it, which is how it - survived a module at 100% statement and branch coverage. The seed is ``Flags(0)`` now, - which is what :pr:`597` had already done to the sibling accumulator in ``make`` for the same - reason; ``Flags`` declares no ``_missing_`` of its own, so ``Flags(0)`` is an ordinary - ``aenum.IntFlag`` pseudo-member and does not meet the ``RecursionError`` that the same - construction hits on the flag enumerations under ``pcapkit.const.mh`` (:issue:`623`). Nothing a - caller can reach changed: the three MP_JOIN layouts of :rfc:`8684` section 3.2 are - chosen by ``_read_mptcp_join`` from these very membership tests, but - ``mptcp_data_selector`` rejects a flagless MP_JOIN with a ``FieldError`` before the - dispatcher runs -- measured identical either side of the fix -- so the ``TypeError`` was - latent rather than live, and latent only by virtue of a guard in another file. Called - directly on a flagless parsed segment the dispatcher now raises the library's own - ``ProtocolError`` naming an invalid flags combination instead. One difference *is* - observable, and it is a dump-format change rather than a value change: a flagless - segment's ``connection`` renders as the string ``Flags::None [0]`` where it rendered as - the number ``0``, in all three of the JSON, tree and PLIST outputs. The value is - numerically the same and the wire bytes are untouched; what changed is that - ``connection`` no longer switches JSON type with the flags, having been a number for a - flagless segment and a string -- ``Flags::ACK [2048]`` -- for every other one. The - literal ``None`` inside it is a separate rendering defect: the hook - ``pcapkit.dumpkit.common.make_dumper`` installs interpolates ``o.name`` without - accounting for a nameless composite member, so it would spell any zero-valued flag - enumeration in the library the same way. That is left to its own change and pinned by a - test here so it cannot drift unnoticed. The committed example dumps are unaffected, - ``examples/captures/in.pcap`` carrying no flagless segment. Coverage cannot see the fix - itself, because the changed line already executed and ``tcp.py`` reads 100% statement - and branch either side; the added subtests are the evidence (:issue:`616`). -* **a breaking change to a public attribute.** ``Frame.len`` and - ``Frame.cap_len`` were filled from *opposite* wire fields depending on which - container format was read, so ``frame.len`` meant the captured length out of a - ``.pcap`` and the on-wire length out of a ``.pcapng``. The PCAP reader is the - one that moved, and now matches the PCAP-NG reader: ``len`` is the on-wire - length (the record header's ``orig_len``) and ``cap_len`` the octets actually - stored (``incl_len``). **Code reading ``frame.len`` or ``frame.cap_len`` from a - ``.pcap`` gets the other field's value than it did before**, and for a - truncated frame that is a different number rather than a relabelling. Which - reader to move was a real decision rather than the correction of a typo, and - not settled by seniority: the PCAP spelling is the *older* of the two, dating - to ``c43892af`` (2022-01-11) with the data model's docstrings agreeing with it - a day later, while the opposite spelling in ``pcapkit.toolkit.pcapng`` arrived - 15 months afterwards in ``25f216f4`` (2023-04-27). Both were internally - consistent. What settles it is that the *names* are Wireshark's, and its - ``epan/dissectors/packet-frame.c`` registers ``frame.len`` as "Frame length on - the wire" and ``frame.cap_len`` as "Frame length stored into the capture file", - and raises the ``frame.len_lt_caplen`` expert info, - ``PI_MALFORMED``/``PI_ERROR``, on ``frame_len < cap_len`` -- which could not be - malformed if ``len`` were the smaller, captured one. So the later spelling is - the one that matches the borrowed names. - Both formats define the underlying fields the same way: ``pcap-savefile(5)`` - gives ``incl_len`` as "the number of bytes of captured data that follow the - per-packet header" and ``orig_len`` as "the number of bytes that would have - been present had the packet not been truncated by the snapshot length", and - ``draft-ietf-opsawg-pcapng`` gives Captured Packet Length as "the number of - octets captured from the packet" against Original Packet Length's "number of - octets of packet data that would have been provided had the packet not been - truncated", which "SHOULD NOT be less than the Captured Packet Length". The - internal consumer moved with it: ``Frame.read`` hands ``_decode_next_layer`` - the octets that are *present*, which is now spelled ``frame.cap_len`` and was - ``frame.len`` when that was the captured length -- handing over the on-wire - length instead would be the declared-length-exceeds-available-octets fault of - :issue:`554`, :issue:`573` and :issue:`594` on every snapped frame. The dumpers are untouched, since - ``PCAPIO`` packs its record header from ``frame_info.incl_len`` and - ``frame_info.orig_len`` rather than from these two, and ``frame_info`` was - already right in both readers. This went unnoticed because the two lengths - differ only for a frame the snapshot length cut short, and no such fixture - existed until :pr:`614` added one -- ``big_endian.pcap``'s third frame, 1200 octets - on the wire against 96 captured -- so every earlier assertion compared a value - against itself (:issue:`618`). -* the HIP ``PUZZLE`` and ``SOLUTION`` builders derived three - wire-format values from the payload value instead of taking them from the data - model, and all three were wrong. **This changes two public data models and the - octets both parameters emit.** The width of ``Random #I``, and of - ``Puzzle solution #J``, came from ``int.bit_length()`` alone, and nothing else - was available to derive it from, so every leading zero octet was dropped on - re-serialisation: a ``SOLUTION`` read with ``Length = 20`` rebuilt as - ``Length = 6``, a ``PUZZLE`` read with ``Length = 12`` as ``Length = 5``, and - silently, because the integers survive and nothing raises. That width is - ``RHASH_len / 8`` octets [:rfc:`7401#section-5.2.4`, :rfc:`7401#section-5.2.5`], - a property of the Responder's HIT Suite rather than of the number that happens - to sit in the field, so both data models now carry it as ``rhash_len`` and both + produced. The undersized length was worse: both fields take their width from that + ``len``, so ``solution=0xfff`` was packed into one octet and came back as ``0xff``, + silently. Fixed to ``4 + 2 * math.ceil(max(...) / 8)``, the form the sibling + ``PUZZLE`` builder uses, which is even by construction. Loosening the reader was + rejected: an odd contents width has no meaning in the wire format (the RFC makes the + fields equal-width and does not say which takes an extra octet), and a laxer guard + would still have truncated the value (:issue:`608`). +* ``TCP.read`` seeded its connection-flag accumulator with ``cast('Enum_Flags', 0)``. + ``typing.cast`` is a runtime no-op, so the accumulator began as the plain ``int`` + ``0``, and only the ``|=`` promoting it to a ``pcapkit.const.tcp.flags.Flags`` + member ever changed that. A segment with an all-zero flags octet (an nmap NULL scan) + left ``self._flags`` an ``int``, so a membership test raised + ``TypeError: argument of type 'int' is not a container or iterable``, and + ``TCP.connection`` returned an ``int`` where it annotates ``Flags``. Any flag masked + it, which is how it survived 100% statement and branch coverage. The seed is + ``Flags(0)``, as :pr:`597` already did for the sibling accumulator in ``make``; + ``Flags`` declares no ``_missing_``, so ``Flags(0)`` is an ordinary + ``aenum.IntFlag`` pseudo-member and does not meet the ``RecursionError`` the same + construction hits on the ``pcapkit.const.mh`` flag enumerations (:issue:`623`). + Nothing a caller can reach changed: the three MP_JOIN layouts of :rfc:`8684` section + 3.2 are chosen from these membership tests, but ``mptcp_data_selector`` rejects a + flagless MP_JOIN with a ``FieldError`` first, so the ``TypeError`` was latent, and + only by virtue of a guard in another file. Called directly on a flagless parsed + segment the dispatcher now raises ``ProtocolError`` naming an invalid flags + combination. One difference *is* observable, a dump-format change rather than a + value change: a flagless segment's ``connection`` renders as the string + ``Flags::None [0]`` where it rendered as the number ``0``, in the JSON, tree and + PLIST outputs. The value and wire bytes are unchanged; ``connection`` no longer + switches JSON type with the flags (a number for a flagless segment, a string such as + ``Flags::ACK [2048]`` for every other). The literal ``None`` is a separate rendering + defect in the hook ``pcapkit.dumpkit.common.make_dumper`` installs, which + interpolates ``o.name`` without accounting for a nameless composite member; it is + left to its own change and pinned by a test. The committed example dumps are + unaffected (``examples/captures/in.pcap`` has no flagless segment). Coverage cannot + see this fix, since the changed line already executed; the added subtests are the + evidence (:issue:`616`). +* **a breaking change to a public attribute.** ``Frame.len`` and ``Frame.cap_len`` + were filled from *opposite* wire fields depending on the container format, so + ``frame.len`` meant the captured length out of a ``.pcap`` and the on-wire length + out of a ``.pcapng``. The PCAP reader moved to match PCAP-NG: ``len`` is the on-wire + length (the record header's ``orig_len``) and ``cap_len`` the octets stored + (``incl_len``). **Code reading ``frame.len`` or ``frame.cap_len`` from a ``.pcap`` + gets the other field's value than before**, and for a truncated frame that is a + different number rather than a relabelling. Which reader to move was a decision, not + settled by seniority: the PCAP spelling is the *older* (``c43892af``, 2022-01-11, + with the data model's docstrings agreeing a day later), the opposite spelling in + ``pcapkit.toolkit.pcapng`` arrived 15 months later in ``25f216f4`` (2023-04-27), and + both were internally consistent. What settles it is that the *names* are + Wireshark's: ``epan/dissectors/packet-frame.c`` registers ``frame.len`` as "Frame + length on the wire" and ``frame.cap_len`` as "Frame length stored into the capture + file", and raises ``frame.len_lt_caplen`` expert info + (``PI_MALFORMED``/``PI_ERROR``) on ``frame_len < cap_len``, which could not be + malformed if ``len`` were the smaller, captured one. Both formats define the + underlying fields alike (``pcap-savefile(5)``'s ``incl_len``/``orig_len``, + ``draft-ietf-opsawg-pcapng``'s Captured/Original Packet Length). The internal + consumer moved with it: ``Frame.read`` hands ``_decode_next_layer`` the octets + *present*, now ``frame.cap_len``; handing over the on-wire length would be the + declared-length-exceeds-available-octets fault of :issue:`554`, :issue:`573` and + :issue:`594` on every snapped frame. The dumpers are untouched (``PCAPIO`` packs its + record header from ``frame_info.incl_len`` and ``orig_len``, already right in both + readers). It went unnoticed because the two lengths differ only for a snapshot-cut + frame and no such fixture existed until :pr:`614` added one (``big_endian.pcap``'s + third frame, 1200 octets on the wire against 96 captured), so every earlier + assertion compared a value against itself (:issue:`618`). +* the HIP ``PUZZLE`` and ``SOLUTION`` builders derived three wire-format values from + the payload value instead of the data model, and all three were wrong. **This + changes two public data models and the octets both parameters emit.** The width of + ``Random #I`` and ``Puzzle solution #J`` came from ``int.bit_length()`` alone, so + every leading zero octet was dropped on re-serialisation: a ``SOLUTION`` read with + ``Length = 20`` rebuilt as ``Length = 6``, a ``PUZZLE`` with ``Length = 12`` as + ``Length = 5``, silently. That width is ``RHASH_len / 8`` octets + [:rfc:`7401#section-5.2.4`, :rfc:`7401#section-5.2.5`], a property of the + Responder's HIT Suite, so both data models now carry it as ``rhash_len`` and both builders prefer it; a new ``rhash_len=`` keyword states it for a build from scratch, which under HIPv2 is the only place it can come from, since ``RSA,DSA/SHA-256`` is the REQUIRED HIT Suite [:rfc:`7401#section-5.2.10`] and - makes the field 32 octets rather than 8. On the mandatory path roughly one - parameter in 256 has a zero top octet and lost it (:issue:`653`). ``SOLUTION``'s second + makes the field 32 octets rather than 8. On the mandatory path about one parameter + in 256 has a zero top octet and lost it (:issue:`653`). ``SOLUTION``'s second contents octet is ``Reserved``, "zero when sent, ignored when received" - [:rfc:`7401#section-5.2.5`, and :rfc:`5201#section-5.2.5` identically] -- not the - ``Lifetime`` that only :rfc:`7401#section-5.2.4` defines, and that pcapkit was - encoding there as ``2^(value - 32)`` seconds. It wrote ``0x20``, ``0x21``, - ``0x25`` or ``0x2b`` into a field the RFC requires to be zero, and could not - write that zero at all: an RFC-conformant ``SOLUTION`` whose ``Reserved`` is - ``0x00`` parsed to ``timedelta(0)`` and then could not be re-serialised, escaping - a bare ``ValueError`` from ``math.log2(0.0)`` that was not a ``BaseError`` and so - bypassed the library's own error handling entirely. The field is renamed - ``reserved``, defaults to the mandated zero, and is carried verbatim across a - round trip rather than re-derived (:issue:`654`). And neither builder read its own - ``version`` keyword, so ``version=1`` and ``version=2`` computed identical - lengths at every bit width, where :rfc:`5201#section-5.2.4` and - :rfc:`5201#section-5.2.5` state both fields as literally 8 bytes and ``Length`` - as literally 12 and 20. Under HIPv1 each builder therefore accepted only a - ``bit_length()`` of 57..64 and built, for everything narrower, a parameter this - library's own reader rejects -- the same shape as :issue:`608`, in the ``PUZZLE`` builder - that :pr:`629` never opened (:issue:`655`). The two remaining ``math.log2`` sites, both in - ``PUZZLE`` where a lifetime is real, now raise ``ProtocolError`` rather than - letting ``ValueError`` escape; ``ProtocolError`` is ``(BaseError, ValueError)``, - so a caller written around the old bare exception still catches it. Migration: - ``SolutionParameter.lifetime`` is now ``reserved`` and an ``int`` rather than a - ``timedelta``; both ``PuzzleParameter`` and ``SolutionParameter`` gained a - required ``rhash_len``; and ``_make_param_solution`` no longer takes - ``lifetime=``. All three were only reachable end to end once :issue:`608` was fixed in - :pr:`629`, which removed the parity guard that had been failing these rebuilds loudly - first -- so the round trip stopped raising and started quietly emitting a - different parameter, which is why they were worth fixing together (:issue:`653`, :issue:`654`, - :issue:`655`). -* every HIP parameter pcapkit emitted was ``4 (mod 8)`` octets long, - because the padding was computed to align the *contents* rather than the record. - Corrected for **45 of the 46** HIP parameters; ``LOCATOR_SET`` is deliberately - excluded, for the reason below. **This changes the octets those 45 parameters - write, and the ``length`` their data models report.** :rfc:`7401#section-5.2.1` requires that - "all of the encoded TLV parameters have a length (that includes the Type and - Length fields), which is a multiple of 8 bytes", and states the arithmetic - outright as ``Total Length = 11 + Length - (Length + 3) % 8``; all 95 padding - sites instead computed ``(8 - (Length % 8)) % 8``, which has no ``+ 4`` inside - the modulus and so aligns the contents alone. Across every ``Length`` from 0 to - 63 the result was never a multiple of eight and never the value the RFC gives, - and it erred in both directions: at ``Length = 4`` -- a whole ``SEQ``, and - :rfc:`7401#section-5.3.5` puts a ``SEQ`` or an ``ACK`` on every ``UPDATE`` -- - the record is complete in eight octets and pcapkit appended four that must not - be there, while at ``Length = 8`` the contents were already 8-aligned, nothing - was appended, and the record went out four octets short. A conformant peer - reading ``Length`` and consuming ``11 + Length - (Length + 3) % 8`` octets - therefore lands mid-parameter and reads the rest of the parameter area at a - wrong offset, in both directions; pcapkit did not, because its reader consumed - the same wrong count its writer wrote, which is why no round-trip test in the - suite could see this and why the fix is asserted against the RFC's arithmetic - rather than against a round trip. 94 of the 95 sites -- 45 of the 46 - ``PaddingField`` callbacks in ``pcapkit/protocols/schema/internet/hip.py`` and 49 - of the 49 record lengths in ``pcapkit/protocols/internet/hip.py`` -- are now one - ``parameter_total_len`` and one ``parameter_padding_len``, stating the RFC formula - once instead of 95 times (:issue:`651`). ``LOCATOR_SET`` kept the old expression at both - of its sites for now, on purpose, because two defects there cancelled each other - exactly and correcting only the padding would have broken a parameter that was, - at that point, right: its padding callback never received the parameter's - ``len`` (the nested ``Locator`` schemas shared a packet context whose own - ``len`` shadowed it, so the value seen was always 4 for an IPv6 locator), and - the parameter's ``len`` was written in 4-octet units where the RFC's ``Length`` - is a byte count. Always-4 padding gave ``4 + 24n + 4``, and because ``24n`` is a - multiple of 8 the RFC total for a byte-count ``Length`` of ``24n`` was the same - ``24n + 8`` -- measured at n = 1, 2 and 5 as 32, 56 and 128 octets both before - and after. :issue:`679` later fixed both sites. ``EncryptedParameter``'s ``data`` length callback is fixed in the same - change and could not have been left: it subtracted the sixteen ``iv`` octets but - not the four ``reserved`` ones, and those four cancelled the padding's four at - ``Length % 8`` in ``{0, 5, 6, 7}`` -- so correcting the padding alone would have - taken ``ENCRYPTED`` from right at four of the eight residues to four octets too - long at all eight. ``HIP.make``'s ``len = total_length // 8 + 4`` is *not* part - of the defect and is unchanged: :rfc:`7401#section-5.1.3` defines Header Length - as the header and parameters "in 8-byte units, excluding the first 8 bytes", so - the floor division is exact once each parameter is a multiple of eight, where - before it was exact only for an even number of them. Migration: a ``SEQ`` - parameter's ``length`` is now 8 where it was 12, and the other 44 corrected - parameters move likewise, so code comparing stored pcapkit output byte for byte, - or asserting on ``Data_*Parameter.length``, sees different values -- the RFC's - values. ``LOCATOR_SET`` stayed unchanged in both respects for the time being; - :issue:`679` (below) later fixed both. ``examples/generators/options.py``'s - ``HIP_COPIES`` stayed at two for now as well, no longer for this reason but for - ``R1_COUNTER``'s four-octet ``counter`` where :rfc:`7401#section-5.2.3` - requires eight, which this defect had been masking -- until :issue:`672` widened - ``counter`` and :issue:`679` fixed ``LOCATOR_SET``'s ``Length`` unit, after which - :issue:`689` dropped ``HIP_COPIES`` to one, once both had left it routing around - nothing (:issue:`651`). -* two HTTP/2 flag defects, one on each side of a round trip. - **This changes the octets a reconstructed DATA frame emits, and the dumped - value of a flagless frame's flags.** ``_make_http_data`` was the only one of - the six ``_make_http_*`` methods that never read ``frame.flags``, so a DATA - frame parsed with ``END_STREAM`` set rebuilt with the bit clear: the flags - octet went ``0x01`` to ``0x00``, silently, producing a well-formed frame + [:rfc:`7401#section-5.2.5`, and :rfc:`5201#section-5.2.5` identically], not the + ``Lifetime`` that only :rfc:`7401#section-5.2.4` defines, which pcapkit encoded + there as ``2^(value - 32)`` seconds. It wrote ``0x20``, ``0x21``, ``0x25`` or + ``0x2b`` into a field the RFC requires to be zero, and could not write that zero: + a conformant ``SOLUTION`` with ``Reserved`` ``0x00`` parsed to ``timedelta(0)`` and + could not be re-serialised, escaping a bare ``ValueError`` from ``math.log2(0.0)`` + that was not a ``BaseError``. The field is renamed ``reserved``, defaults to zero, + and is carried verbatim across a round trip (:issue:`654`). And neither builder + read its own ``version`` keyword, so ``version=1`` and ``version=2`` computed + identical lengths, where :rfc:`5201#section-5.2.4` and :rfc:`5201#section-5.2.5` + state both fields as literally 8 bytes and ``Length`` as literally 12 and 20. Under + HIPv1 each builder accepted only a ``bit_length()`` of 57..64 and built, for + anything narrower, a parameter this library's reader rejects -- the shape of + :issue:`608`, in the ``PUZZLE`` builder that :pr:`629` never opened (:issue:`655`). + The two remaining ``math.log2`` sites, both in ``PUZZLE`` where a lifetime is real, + raise ``ProtocolError`` (``(BaseError, ValueError)``, so a caller written around + the bare exception still catches it). Migration: ``SolutionParameter.lifetime`` is + now ``reserved`` and an ``int`` rather than a ``timedelta``; ``PuzzleParameter`` and + ``SolutionParameter`` gained a required ``rhash_len``; and ``_make_param_solution`` + no longer takes ``lifetime=``. All three were reachable end to end only once + :issue:`608` was fixed in :pr:`629`, which removed the parity guard that had failed + these rebuilds loudly; the round trip then stopped raising and started quietly + emitting a different parameter, which is why they were fixed together + (:issue:`653`, :issue:`654`, :issue:`655`). +* every HIP parameter pcapkit emitted was ``4 (mod 8)`` octets long, because the + padding aligned the *contents* rather than the record. Corrected for **45 of the + 46** HIP parameters; ``LOCATOR_SET`` is excluded, below. **This changes the octets + those 45 parameters write, and the ``length`` their data models report.** + :rfc:`7401#section-5.2.1` requires every TLV parameter to have "a length (that + includes the Type and Length fields), which is a multiple of 8 bytes" and gives + ``Total Length = 11 + Length - (Length + 3) % 8``; all 95 padding sites computed + ``(8 - (Length % 8)) % 8``, which aligns the contents alone. For every ``Length`` + from 0 to 63 the result was never a multiple of eight and never the RFC's value, + erring both ways: at ``Length = 4`` (a whole ``SEQ``, which + :rfc:`7401#section-5.3.5` puts on every ``UPDATE``) pcapkit appended four octets + that must not be there, and at ``Length = 8`` it appended none and went out four + short. A conformant peer lands mid-parameter; pcapkit did not, because its reader + consumed the same wrong count its writer wrote, which is why no round-trip test + could see it and why the fix is asserted against the RFC's arithmetic. 94 of the 95 + sites (45 of 46 ``PaddingField`` callbacks in + ``pcapkit/protocols/schema/internet/hip.py`` and 49 of 49 record lengths in + ``pcapkit/protocols/internet/hip.py``) are now one ``parameter_total_len`` and one + ``parameter_padding_len`` (:issue:`651`). ``LOCATOR_SET`` kept the old expression + because two defects there cancelled exactly: its padding callback never received the + parameter's ``len`` (the nested ``Locator`` schemas share a packet context whose own + ``len`` shadowed it, so it saw 4 for an IPv6 locator), and the parameter's ``len`` + was written in 4-octet units where the RFC's ``Length`` is a byte count; the result, + ``4 + 24n + 4``, equalled the RFC total ``24n + 8``. :issue:`679` later fixed both + sites. ``EncryptedParameter``'s ``data`` length callback is fixed in the same + change: it subtracted the sixteen ``iv`` octets but not the four ``reserved`` ones, + which cancelled the padding's four at ``Length % 8`` in ``{0, 5, 6, 7}``, so + correcting the padding alone would have made ``ENCRYPTED`` four octets too long at + all eight residues. ``HIP.make``'s ``len = total_length // 8 + 4`` is not part of + the defect: :rfc:`7401#section-5.1.3` defines Header Length in 8-byte units + excluding the first 8 bytes, so the floor division is exact once each parameter is a + multiple of eight. Migration: a ``SEQ`` parameter's ``length`` is now 8 where it was + 12, and the other 44 move likewise, so code comparing stored output byte for byte or + asserting on ``Data_*Parameter.length`` sees the RFC's values. ``LOCATOR_SET`` is + unchanged in both respects until :issue:`679` (below). + ``examples/generators/options.py``'s ``HIP_COPIES`` stayed at two for + ``R1_COUNTER``'s four-octet ``counter`` where :rfc:`7401#section-5.2.3` requires + eight, which this defect had masked, until :issue:`672` widened ``counter`` and + :issue:`679` fixed ``LOCATOR_SET``'s ``Length`` unit, after which :issue:`689` + dropped ``HIP_COPIES`` to one (:issue:`651`). +* two HTTP/2 flag defects, one on each side of a round trip. **This changes the octets + a reconstructed DATA frame emits, and the dumped value of a flagless frame's + flags.** ``_make_http_data`` was the only one of the six ``_make_http_*`` methods + that never read ``frame.flags``, so a DATA frame parsed with ``END_STREAM`` set + rebuilt with the bit clear (flags octet ``0x01`` to ``0x00``): a well-formed frame saying the stream continues where the capture said it ended - [:rfc:`9113#section-6.1`]. ``PADDED`` was never affected, being re-derived - from ``pad_len`` rather than read back, and all five sibling builders already - restored theirs -- so this was a missing read-back rather than a design, and - it now reads ``frame.flags.END_STREAM`` the way they do (:issue:`652`). Separately, - ``FrameType.post_process`` seeded its flag accumulator with a bare ``0``, and - ``|=`` promotes that only as a side effect, so a frame with **no** bit set - left ``__flags__`` a plain ``int`` where both the schema and the data model - declare a ``Flags`` -- and ``END_STREAM in __value__`` raised - ``TypeError: argument of type 'int' is not a container or iterable`` rather - than answering ``False``. That is the common case rather than an edge one: a - ``0x00`` flags - octet is every opening ``SETTINGS``, every non-final ``DATA`` and every - non-ACK ``PING``. It now seeds ``self.Flags(0)``, which the construct path had - been doing correctly at all six of its own sites. The visible consequence is - in the dump, where ``__value__`` was a JSON *number* for a flagless frame and - a JSON *string* for every other frame in the same capture; it is consistently - a string now, ``"Flags::None [0]"`` where it read ``0``, the same shape of - change :issue:`616` made to TCP's ``connection``. The seed is guarded rather than - unconditional because ``FrameType.Flags`` declares no members and, from - Python 3.11, a memberless ``enum.Flag`` subclass refuses ``Flags(0)`` - outright, so the five - frame schemas that inherit it -- ``UnassignedFrame``, ``PriorityFrame``, - ``RSTStreamFrame``, ``GoawayFrame`` and ``WindowUpdateFrame`` -- keep the - plain ``int``, which nothing observes since all five pass ``flags=None`` into - their data objects. Note that a DATA round trip is **still** lossy after - this, for an unrelated reason found while measuring it and left to its own - change: three payload length callbacks in the frame schemas are - mis-parenthesised, so an unpadded ``DATA``, ``HEADERS`` or ``PUSH_PROMISE`` - frame parses with its whole payload dropped (:issue:`668`). (:issue:`652`, :issue:`650`). -* four MPTCP error messages carried a doubled separator, rendering - as ``TCP: : [OptNo 30] 1: invalid flags combination`` with an empty field - between the protocol alias and the option number. Cosmetic: the exception - type, the option number and the subtype were always right, and nothing - downstream parses these strings, but it is the text a user sees when an - MP_JOIN or DSS option is rejected and an empty field reads as a value that - failed to interpolate. Four sites, not the one reported -- ``_read_mptcp_join`` - and ``_make_mptcp_join`` for the invalid flag combination, and both guards of - ``_make_mptcp_dss`` for the missing required fields -- against 28 messages in - the same file that already spelled the prefix correctly, so the four were - outliers against a house form in their own module and the counts are now 32 - correct against 0 doubled. Pre-existing since 2023-04-10. The new test asserts - each message in full rather than by substring, which is what the existing - ``assertIn`` on the message body could not do, since it passes either side of - the change (:issue:`649`). -* three names appeared in string annotations that their own module - never imported, so a type checker or a documentation build could not resolve - them while every test went on passing: ``Any`` in - ``pcapkit/utilities/logging.py``, and in + [:rfc:`9113#section-6.1`]. ``PADDED`` was re-derived from ``pad_len``, and the five + sibling builders already restored theirs, so this was a missing read-back + (:issue:`652`). Separately, ``FrameType.post_process`` seeded its flag accumulator + with a bare ``0``, which ``|=`` promotes only as a side effect, so a frame with + **no** bit set left ``__flags__`` a plain ``int`` where the schema and data model + declare a ``Flags``, and ``END_STREAM in __value__`` raised + ``TypeError: argument of type 'int' is not a container or iterable`` rather than + answering ``False``. That is the common case: a ``0x00`` flags octet is every + opening ``SETTINGS``, every non-final ``DATA`` and every non-ACK ``PING``. It now + seeds ``self.Flags(0)``, as the construct path already did at all six of its sites. + In the dump, ``__value__`` was a JSON *number* for a flagless frame and a *string* + for every other frame in the same capture; it is consistently a string now, + ``"Flags::None [0]"`` where it read ``0``, the same change :issue:`616` made to + TCP's ``connection``. The seed is guarded because ``FrameType.Flags`` declares no + members and, from Python 3.11, a memberless ``enum.Flag`` subclass refuses + ``Flags(0)``, so the five frame schemas that inherit it (``UnassignedFrame``, + ``PriorityFrame``, ``RSTStreamFrame``, ``GoawayFrame``, ``WindowUpdateFrame``) keep + the plain ``int``, which nothing observes since all five pass ``flags=None``. A DATA + round trip is **still** lossy for an unrelated reason found while measuring: three + payload length callbacks are mis-parenthesised, so an unpadded ``DATA``, ``HEADERS`` + or ``PUSH_PROMISE`` frame parses with its whole payload dropped (:issue:`668`). + (:issue:`652`, :issue:`650`). +* four MPTCP error messages carried a doubled separator, rendering as + ``TCP: : [OptNo 30] 1: invalid flags combination``. Cosmetic: the exception type, + option number and subtype were right and nothing parses these strings, but an empty + field reads as a value that failed to interpolate. Four sites (``_read_mptcp_join`` + and ``_make_mptcp_join`` for the invalid flag combination, both guards of + ``_make_mptcp_dss`` for missing required fields), against 28 messages in the same + file spelled correctly; now 32 correct against 0 doubled. Pre-existing since + 2023-04-10. The new test asserts each message in full, since the existing + ``assertIn`` on the body passes either side of the change (:issue:`649`). +* three names appeared in string annotations their own module never imported, so a + type checker or documentation build could not resolve them while every test passed: + ``Any`` in ``pcapkit/utilities/logging.py``, and in ``pcapkit/protocols/schema/internet/ipv6_route.py`` both ``Protocol`` in a - ``payload:`` stub and ``Optional`` inside a ``typing.cast``. All three are now - in their module's ``TYPE_CHECKING`` block, ``Protocol`` spelled - ``ProtocolBase as Protocol`` the way twenty-one sibling schema modules already - spell it in that same stub, which makes twenty-two. The ``cast`` case is the - one worth naming: ``typing.cast`` never evaluates its first argument, so no - test that runs the line can see a bad name in it. Nothing resolves at runtime - that did not resolve before -- ``TYPE_CHECKING`` is ``False`` when the - interpreter runs, so a name imported under it is absent from the module - namespace either way, and making these annotations resolve at runtime is a - separate decision about the idiom rather than a missing import. mypy 2.3.1 over - ``pcapkit`` reported three ``name-defined`` errors before and none after, its - total moving 115 to 112, so nothing else shifted. A new - ``tests/project/test_annotation_names.py`` pins the invariant without needing a - type checker installed: it resolves every string annotation in the package - against the names its own module binds, following a nested forward reference - such as ``'list["Nested"]'`` while treating ``Literal`` members and - ``Annotated`` metadata as the values they are, and it reports exactly those - three findings on the unfixed tree and none after. That module reaches the two - PEP 695 node classes it needs through ``getattr`` rather than naming them, - because ``ast.TypeAlias`` and ``ast.TypeVar`` arrived in Python 3.12 while the - supported range starts at 3.10. The same change corrects the ``MANIFEST.in`` - comment asserting that ``include README.md`` was the only thing putting the - README in a source distribution and that an sdist without it could not be - installed; both halves are false, and the :pr:`619` entry above carried the - identical claim and is corrected with it (:issue:`642`). -* every PCAP-NG packet block now carries the captured octets it declares. All - four Enhanced Packet Blocks of the committed ``examples/captures/dhcp.pcapng`` reported - ``len(packet) == 0`` against a ``captured_len`` of 314, 342, 314 and 342, and the Simple - Packet Block and the obsolete Packet Block did the same, measured on a capture - synthesised to carry one of each since no committed fixture has either. Those three -- - exactly ``PCAPNG.PACKET_TYPES``, and exactly the three schemas declaring - ``__payload__ = 'packet_data'`` -- are the whole of the blast radius; every other block - type carries no captured octets, reported ``b''`` before and still does. The octets were - read correctly and then thrown away: ``PCAPNG.unpack`` extracted them from the block - schema, and ``ProtocolBase.__init__`` overwrote the result with ``self.packet.payload``, - which was empty. The inherited ``packet`` splits a protocol at ``self.length`` octets of - header and takes everything after as the payload, a contract that holds for a protocol - laid out as a header followed by its payload and for nothing else. ``PCAPNG.length`` - returns the wire's *Block Total Length*, so the split consumed the entire per-block - buffer as header -- measured at ``frame.length == 348`` and - ``len(frame.packet.header) == 348`` on a 348-octet block -- and, separately, a PCAP-NG - block carries a *trailer* after its payload, the option list and a repeat of the total - length, so no value of ``length`` could have made that split right. ``captured_len`` - comes straight off the block header and survived, which is why the two disagreed rather - than both being empty. The fix is at the computation rather than at the injection: - ``PCAPNG.packet`` is overridden to take the payload from the block schema's - ``__payload__`` field and the header from the octets ahead of it, summed out of the - schema buffers so that the three block types' three different payload offsets -- 28, 12 - and 28 octets -- are not hard-coded. Teaching ``ProtocolBase.__init__`` to skip a - protocol that had already set ``packet`` was the smaller change and was rejected: it - heals ``frame.info.packet`` while leaving the public ``frame.packet`` still reporting - the whole block as header and ``b''`` as payload, which is the same defect from the - other side. ``PCAPNG.unpack`` now reads that property instead of extracting the payload - a second time of its own, leaving one source of truth where there were two attempts at - one. The property is a plain ``property`` rather than the ``cached_property`` it - overrides, which a cross-review on a second model is what settled: the inherited one - caches because it *reads the stream*, where a second read would consume octets that are - gone, while this one only walks buffers the schema layer has already filled and so has - nothing to amortise -- and caching it would have reintroduced the same staleness by - another route, a second ``unpack`` on one instance handing back the *first* call's - octets with ``get_payload`` never reached, an invariant the code held before this change - and has no reason to stop holding. Nothing calls ``unpack`` twice on one instance today, - ``__post_init__`` being its only caller, so the unit test asserts the property directly: - swap the block on a live instance, unpack again, and the payload must be the new - block's, which fails ``b'payload' != b'cached'`` under a cache. ``ProtocolBase.packet`` - gains the contract in its docstring and not one executable line, its 503 statements - unchanged. This moves the parse output of every PCAP-NG capture, for two properties, on - the success path, which is why it ships labelled breaking: ``frame.info.packet`` goes - from ``b''`` to hundreds of octets, ``frame.packet.header`` from the whole block to the - pre-payload prefix, and anything diffing a dumped file sees it change. That dump is what - made this a wire-format defect rather than an API wart. ``PCAPIO`` writes - ``value.packet`` after each 16-octet record header, so dumping those four blocks - produced a **104-octet** file -- 24 of global header plus four record headers and no - payload at all -- in which every record header declared hundreds of octets and delivered - none, desynchronising any reader that walks by ``incl_len``, which is all of them: - pcapkit refused its own output with - ``ValueError: read length must be non-negative or -1`` and scapy silently returned 1 - packet of 64 octets instead of 5. It round-trips now, the dumped file's total size and - each record's octets both asserted against the source capture's own hand-parsed bytes - rather than against a length. ``tests/protocols/test_pcapng_regression.py`` grows 4 - tests to 11 and 3 subtests to 24, with the payload expectations derived twice over -- - spelled-out head and tail literals, and a ``struct``-only re-parse of the fixture that - owes nothing to the code under test -- and each synthesised payload given a different - length modulo 4 so that an offset off by a field, or a payload that picked up the - block's 32-bit padding, cannot pass by coincidence. Two shapes no fixture exercised are - covered there too, both found while writing the tests rather than after: an option area - *after* the captured data, which every block of ``dhcp.pcapng`` lacks and which is - exactly what an offset walked from the wrong end would swallow, and a big-endian - section, whose fields differ even though the payload offsets do not. A snapped block is - covered in both the shapes that express it -- an Enhanced Packet Block declaring - ``captured_len`` below ``original_len``, and a Simple Packet Block whose captured length - is bounded by the interface's ``snaplen`` -- neither of which may come back padded out - to the on-wire length. The offsets hold at 28, 12 and 28 across all of it, which is the - property the walk has to have. With both trees running the same -- current -- tests, - which is the only comparison that isolates the code from the suite, - ``pcapkit/protocols/misc/pcapng.py`` holds 99.91% across the change, its 18 new - statements and 6 new branches all executed, its miss count flat at 1 and its - partial-branch count flat at 1, that single miss being the same pre-existing - ``_get_timezone`` statement renumbered 1248 to 1360; ``pcapkit/protocols/protocol.py`` - is identical in every column, 503 statements and 228 misses either side, which is the - check that its change really is docstring-only. The uncaught ``ValueError`` that an - EOF-truncated PCAP-NG raises (:issue:`678`) is untouched and was measured rather than assumed, - over a truncation set stated rather than described because "101 levels" admits several - readings that do not agree on the counts: the whole-percent prefixes - ``max(1, 1508 * i // 100) for i in range(101)``, which for this file are 101 distinct - lengths from 1 to 1508. Both trees give 96 of that ``ValueError``, one ``FormatError``, - one ``ProtocolError`` and three parses, and -- compared level by level rather than only - in aggregate, since a matching total can hide two levels that swapped -- the SHA-256 of - the sorted length-to-outcome mapping is - ``ca44d3ee658087cf2a667c454f839dd931c1c04790b2fe32cee8e1a2e861b182`` on each. - ``examples/captures/pcapng.txt``, a hand-regenerated legacy smoke reference, still showed - the old frame-level ``packet -> NIL`` at this point and wanted a separate refresh; - :issue:`685` (below) later removed it from the index entirely rather than regenerating - it again (:issue:`646`). -* an unpadded HTTP/2 ``DATA``, ``HEADERS`` or ``PUSH_PROMISE`` frame parsed - with its whole payload silently discarded. Three payload length callbacks in - ``pcapkit/protocols/schema/application/httpv2.py`` put the conditional expression in the - wrong place: a conditional binds looser than ``-``, so + ``payload:`` stub and ``Optional`` inside a ``typing.cast``. All three are now in + their module's ``TYPE_CHECKING`` block, ``Protocol`` spelled + ``ProtocolBase as Protocol`` as twenty-one sibling schema modules already do. The + ``cast`` case is the one worth naming: ``typing.cast`` never evaluates its first + argument, so no test that runs the line can see a bad name in it. Nothing resolves + at runtime that did not before (``TYPE_CHECKING`` is ``False``); making these + annotations resolve at runtime is a separate decision about the idiom. A new + ``tests/project/test_annotation_names.py`` pins the invariant without a type + checker: it resolves every string annotation in the package against the names its + own module binds, following nested forward references such as ``'list["Nested"]'`` + and treating ``Literal`` members and ``Annotated`` metadata as values, and reports + exactly those three findings on the unfixed tree. The same change corrects the + ``MANIFEST.in`` comment asserting that ``include README.md`` was the only thing + putting the README in a source distribution and that an sdist without it could not + be installed; both halves are false, and the :pr:`619` entry above carried the same + claim and is corrected with it (:issue:`642`). +* every PCAP-NG packet block now carries the captured octets it declares. All four + Enhanced Packet Blocks of the committed ``examples/captures/dhcp.pcapng`` reported + ``len(packet) == 0`` against a ``captured_len`` of 314, 342, 314 and 342, as did the + Simple Packet Block and obsolete Packet Block (synthesised; no committed fixture has + either). Those three, exactly ``PCAPNG.PACKET_TYPES``, are the whole blast radius. + The octets were read correctly and thrown away: ``PCAPNG.unpack`` extracted them + from the block schema, and ``ProtocolBase.__init__`` overwrote the result with + ``self.packet.payload``. The inherited ``packet`` splits a protocol at + ``self.length`` octets of header and takes the rest as payload, which holds only for + a header followed by its payload; ``PCAPNG.length`` is the *Block Total Length*, so + the split consumed the whole block as header (``len(frame.packet.header) == 348`` on + a 348-octet block), and a block's trailer (option list, repeated total length) means + no ``length`` could make it right. The fix is at the computation rather than the + injection: ``PCAPNG.packet`` is overridden to take the payload from the block + schema's ``__payload__`` field and the header from the octets ahead of it, summed + out of the schema buffers so the three types' payload offsets (28, 12, 28) are not + hard-coded. Teaching ``ProtocolBase.__init__`` to skip a protocol that had set + ``packet`` was rejected: it heals ``frame.info.packet`` while ``frame.packet`` still + reports the whole block as header. The override is a plain ``property``, not the + ``cached_property`` it replaces, which a cross-review on a second model settled: the + inherited one caches because it *reads the stream*, where this one only walks + buffers already filled, and caching it would let a second ``unpack`` on one instance + return the *first* call's octets; the unit test swaps the block on a live instance + and fails ``b'payload' != b'cached'`` under a cache. ``ProtocolBase.packet`` gains + the contract in its docstring and no executable line. This moves the parse output of + every PCAP-NG capture on the success path, so it ships labelled breaking: + ``frame.info.packet`` goes from ``b''`` to hundreds of octets, + ``frame.packet.header`` from the whole block to the pre-payload prefix. It was also + a wire-format defect: ``PCAPIO`` writes ``value.packet`` after each 16-octet record + header, so dumping those four blocks produced a **104-octet** file whose record + headers declared hundreds of octets and delivered none, desynchronising any reader + that walks by ``incl_len`` (pcapkit refused its own output with + ``ValueError: read length must be non-negative or -1``, and scapy returned 1 packet + of 64 octets instead of 5). It round-trips now. + ``tests/protocols/test_pcapng_regression.py`` grows 4 tests to 11 and 3 subtests to + 24, with payload expectations derived twice (literals, and a ``struct``-only + re-parse independent of the code under test) and each synthesised payload a + different length modulo 4, so an offset off by a field or a payload that picked up + block padding cannot pass by coincidence; shapes no fixture exercised (an option + area *after* the captured data, a big-endian section, snapped Enhanced and Simple + Packet Blocks) are covered. ``pcapkit/protocols/protocol.py`` is identical in every + coverage column, confirming its change is docstring-only. The uncaught + ``ValueError`` on an EOF-truncated PCAP-NG (:issue:`678`) is untouched: over the 101 + whole-percent prefixes ``max(1, 1508 * i // 100)`` both trees give 96 of that + ``ValueError``, one ``FormatError``, one ``ProtocolError`` and three parses. + ``examples/captures/pcapng.txt``, a hand-regenerated legacy reference, still showed + the old ``packet -> NIL`` here; :issue:`685` (below) later removed it from the index + rather than regenerating it again (:issue:`646`). +* an unpadded HTTP/2 ``DATA``, ``HEADERS`` or ``PUSH_PROMISE`` frame parsed with its + whole payload silently discarded. Three payload length callbacks in + ``pcapkit/protocols/schema/application/httpv2.py`` put the conditional in the wrong + place: a conditional binds looser than ``-``, so ``pkt['__length__'] - pkt['pad_len'] if pkt['flags']['bit_3'] else 0`` groups as - ``(pkt['__length__'] - pkt['pad_len']) if ... else 0`` and the ``else`` arm returned - ``0`` -- "read no octets at all" to ``BytesField`` -- where it was meant to subtract - ``0``. Since ``__length__`` is the *remaining* declared length at the field, the - unpadded arm wants ``__length__`` itself, which is what subtracting a zero padding - length gives. The grouping was read off the AST rather than by eye, the top-level node - being the ``IfExp`` whose ``orelse`` is a bare ``Constant(0)``, which is also what - showed that the outer parentheses on the ``HeadersFrame`` and ``PushPromiseFrame`` - forms were line-continuation only and changed nothing. Padding is rare in HTTP/2, so - the broken arm was the common one rather than the edge case, and nothing raised, warned - or logged: ``info.data`` was simply ``b''``. Measured on wire octets through the public - ``HTTP(io.BytesIO(raw), len(raw))`` path, an unpadded ``DATA`` frame declaring - ``b'{"ok":true}\n'`` parsed as ``b''`` and now parses as those twelve octets; unpadded - ``HEADERS`` and ``PUSH_PROMISE`` lost and now keep the RFC 7541 appendix C.4.1 header - block fragment ``b'\x82\x86\x84A\x0fwww.example.com'``, which is precisely what HPACK - decoding would have been handed. Padded frames are untouched, for a reason stronger - than the measurement: ``(A - B) if T else 0`` and ``A - (B if T else 0)`` are both - ``A - B`` for truthy ``T``, so only the ``else`` arm could ever have moved -- both arms - are asserted anyway, so that a repair to the unpadded one cannot break the other. The - padded arm also has no off-by-one, which :issue:`668` asked be checked rather than assumed: - ``Schema.unpack`` decrements ``packet['__length__']`` by each field's width as it goes, - so ``pad_len``'s own octet is already out of it by the time the payload field runs, and - a padded frame declaring ``1 + len(data) + pad_len`` arrives at the payload with - ``__length__ == len(data) + pad_len``. The three sibling sites that were never wrong - are what localise the defect to the grouping rather than to the field machinery: - ``ContinuationFrame.fragment``, ``UnassignedFrame.data`` and ``GoawayFrame.debug`` use - the plain ``length=lambda pkt: pkt['__length__']`` with no conditional, share the same - ``BytesField``, the same ``__length__`` bookkeeping and the same frame dispatch, and - carried their payload correctly throughout -- they are pinned as controls rather than - left as an argument. Why nothing caught it: ``test_option_roundtrip_unit`` does drive - every frame type, but through ``make`` -> parse -> ``make``, and ``make`` writes the - length field from ``_make_http_length``, so both sides agreed on an empty payload and - the octets matched; its generator passes no payload argument for these three frames at - all, measured as ``kwargs={}``, so what it round-tripped was ``b''`` and there was - nothing to lose, while ``test_httpv2_frame_readers_cover_successful_frames`` drives the - readers from hand-built schema stubs, so the callback never ran there either. The new - ``tests/protocols/application/test_httpv2_payload_length_unit.py`` reads wire octets, - which is the gap: 23 tests and 12 subtests over the padded, unpadded and padding-only - shapes of all three frames plus both ``PRIORITY`` combinations, asserting the payload - *octets* and never its length, because a length assertion survives a read of the right - width from the wrong offset. Its padded cases pad with a non-zero - ``b'\xde\xad\xbe\xef'`` rather than the zeros :rfc:`9113#section-6.1` tells a sender to - use -- pcapkit does not police padding content -- so that failing to subtract the - padding yields a different byte string instead of a coincidentally equal count. The - same lambda also governs **packing**, which neither the issue nor the first revision of - the fix noticed and a cross-review on a second model did: ``BytesField`` consults its - ``length`` callback on both paths, so the ``else`` arm did not merely discard a payload - on read, it declined to write one. Measured pre-fix, an unpadded ``DATA`` frame carrying - twelve octets of body packed to ``000015000000000001`` -- nine octets of header whose - length field declares **21**, with the body absent -- and unpadded ``HEADERS`` and - ``PUSH_PROMISE`` likewise declared 29 against 9 and 33 against 13. A frame overstating - its own length by its whole payload desynchronises any reader that walks a stream by - that field, so this was pcapkit *emitting* malformed HTTP/2 rather than only mis-reading - it, and ``HTTPv2ConstructedFrameDeclaresWhatItWritesUnitTests`` pins that half; a - ``make`` -> parse -> ``make`` cycle of a non-empty unpadded payload now closes - byte-for-byte where it could not before. Six wrong fixes were written over the schema to - check the tests discriminate, and each is rejected: dropping the conditional outright - (caught by ``packet length < 0: -4``), subtracting ``pad_len + 1`` (caught as - ``b'{"ok":true}'`` against ``b'{"ok":true}\n'``), subtracting the padding on the arm - where it is absent, fixing ``DataFrame`` while forgetting the other two, and - ``max(computed, 1)`` on the two ``fragment`` fields. That last one is the cross-review's - own construction and it **passed** the first revision's 15 tests and 9 subtests, because - every fixture then carried a non-empty payload and only ``DataFrame`` had a "frame of - only padding" case, so those two callbacks were never once asked for ``0``; under it a - padding-only ``HEADERS`` frame returned ``fragment=b'\xde'``, one padding octet leaked - in, alongside a swallowed ``SchemaWarning: packet length < 0: -1``. The three missing - padding-only cases and a callback-level ``test_the_padded_arm_reaches_zero_and_is_not_clamped`` - close it -- ``0`` is a legitimate answer from these callbacks and has to be asserted as - one. Seven tests and three subtests fail before the change and all pass after; 145 tests - and 550 subtests pass across ``tests/protocols/application/``, the round-trip module and - the four other HTTP/2-touching modules. ``EXPECTED_FAILURES`` is unmoved, its one HTTP/2 - entry ``httpv2-frame/PRIORITY`` still failing with the same ``CONSTRUCT`` status and - the same detail -- ``PriorityFrame`` has neither a payload field nor a ``PADDED`` flag, - so this change cannot reach it -- and none of the 43 entries was deleted; that entry's - own cited ``pcapkit/protocols/application/httpv2.py:572`` is stale, the - ``header.length != 9`` guard it describes now being at ``:650``. The schema module was - already at 100% coverage and stays there, 101 statements and 4 branches either side - with nothing missing or partial, because the changed lines did execute before -- just - with an empty payload -- so the number that moved is the suite: 73 to 96 tests and 399 - to 411 subtests over the same targets. The construct side needed no code change, - ``_make_http_length`` having always computed the DATA payload as - ``len(frame.data) + (pad_len + 1 if pad_len else 0)``, i.e. assuming ``data`` holds the - whole payload whether or not the frame is padded -- it was the shared length callback - that disagreed, on both paths at once. An AST sweep of all 496 files of the package - found one other site with the same syntactic shape, - ``pcapkit/protocols/schema/internet/ipv4.py:336`` (``TSOption.remainder``), and it is - correct rather than the same defect: its ``else 0`` is intentional because the sibling - ``ts_data`` field consumes the entire option data area in that arm, the two summing to - ``length - 4`` across all 2106 ``(length, pointer, flag)`` combinations, which is - exactly what was not true here. Ships labelled breaking on both counts: restoring a - dropped payload moves the parse output of essentially every HTTP/2 capture on the - success path, so anything holding a golden file or a regression baseline that recorded - the empty value sees it change -- and constructed frames change byte-for-byte too, from - malformed to correct (:issue:`668`). -* the schema half of every option-like registrar overwrote - silently while the parser half of the very same call warned. - ``EnumSchema.register`` was a bare ``cls.__enum__[code] = schema``, and it is - the schema half of 14 public registrars, so one ``register_ipv4_option`` call - replacing a built-in named the parser it displaced and said nothing about the - schema. ``EnumSchema.__init_subclass__`` reaches the same registry without - calling ``register``, so a subclass declared with a ``code=`` keyword -- the - documented way to add a schema -- stayed silent too; both are guarded now, - and the declaration hook's two branches are folded into one loop so the guard - is written once. Presence is a faithful test there only because - ``_EnumRegistry.__missing__`` returns a miss without recording it, so parsing - one packet carrying an unknown code cannot make the next legitimate - registration for that code warn about an entry no caller ever asked for. The - seven code-keyed parser registrars -- on ``ProtocolBase``, ``Link``, - ``Internet``, ``Frame``, ``PCAPNG``, ``Transport`` and ``SCTP`` -- now name - the displaced entry and its replacement, where before they reported only that - something had been overwritten and never which class it was. Their - firing condition is harmonised onto the narrower guard :pr:`681` gave - ``register_protocol`` (:issue:`718`, :pr:`726`), rather than firing on presence alone: - that one is keyed on a name derived from the value it stores, whereas these - seven take a ``code`` the caller supplies independently of the value. Five of - these seven -- ``Link`` (7 entries), ``Internet`` (16), ``Frame`` (3), ``PCAPNG`` - (3) and ``SCTP`` (2) -- ship pre-seeded with unresolved ``ModuleDescriptor`` - values, and so do ``TCP`` (4) and ``UDP`` (3), each keeping its own separate - dict rather than filling the one ``ProtocolBase`` and the abstract ``Transport`` - share and leave at 0 (``Transport.register`` itself raises - ``UnsupportedCall``). Where a table is pre-seeded, an incumbent may still be a - two-string descriptor while the replacement is the very class it names. - Resolution happens earlier, on the incoming argument only, three lines - above the guard -- - ``if isinstance(protocol, ModuleDescriptor): protocol = protocol.klass`` -- so - the guard's own ``is`` comparison still measures a resolved class against an - unresolved descriptor, warning on re-registering the very same class under its - own pre-seeded code -- - ``Internet.register(TransType.TCP, TCP)``, say -- a false positive the - harmonised guard does not close. - ``ContextRegistry.register`` already raises on a duplicate, and the - reassembly and ESP registrars append to lists with no key at all -- duplicate - security-association SPIs being a designed feature there, resolved by scoring - rather than by replacement -- so none of those three takes a guard. - ``import pcapkit`` holds at one warning, a third-party deprecation, and no - ``RegistryWarning``, measured over the 327 registry writes it performs -- one - of them being ``R1CounterParameter``'s second code, added by :issue:`690` -- none - of which lands on a key already present; ``tests/foundation/registry/`` goes - 13 to 14 tests and 87 to 95 subtests, the schema module holds 99% coverage - with its 5 new statements covered and its misses flat at 1, mypy and pylint - are unmoved at 112 errors and 364 messages, and ``EXPECTED_FAILURES`` is - unmoved at 43 entries (:issue:`692`). -* every EOF-truncated PCAP-NG file raised a bare ``ValueError`` out of - ``Extractor``, so the whole extraction was lost rather than the one truncated block. - The root is one layer above the reported site: Block Total Length is cross-checked - against its own trailing copy and never against the file, so a last block declaring - more than the file holds left ``PCAPNG.read`` seeking *past* the real end -- legal and - silent -- and the next block read then measured a **negative** remainder, since - ``prepare`` derives it as the end of the stream less the current position. That - negative reached ``SchemaField`` through ``pcapng_block_selector`` and - ``io.RawIOBase.read`` refused it, which is why nearly every truncation level failed - rather than only the one holding the cut. The seek is now clamped to the octets - ``_read_fileng`` actually returned, with a ``ProtocolWarning`` naming the overrun, and - a tail under the twelve octets a block needs at minimum is reported as the quiet - ``StreamEOFError`` that ``Extractor.record_frames`` already catches -- the one case - that is deliberately not a clamp, because at the end of the file no block is being - read and clamping would fabricate a frame out of zero padding. A second, independent - route to the same shape is closed with a new ``nonnegative()`` helper, composed into - ``bounded_option`` and ``bounded_area`` and applied to the eight unwrapped - ``length - N`` spans and the eight ``__option_padding__``-sized padding fields: - ``_TextField.__call__`` builds its ``struct`` template as ``f'{length}s'`` - unconditionally, so a negative became the format ``'-8s'`` and ``struct.calcsize`` - raised -- neither that nor the ``ValueError`` being one of - ``pcapkit.utilities.exceptions``, and neither being an ``EOFError``, so neither was - caught where the frame loop catches the end of a file. 28 of the module's 74 length - callbacks returned a negative before and none do now, from ``-1`` on an ``epb_hash`` - declaring no payload to ``-16777248`` on an Enhanced Packet Block's option area; the - 200-block, 8,048-octet ``captured_len = 0xFFFFFF`` vector that reached ``struct`` - through ``bounded_area``'s own ``nominal <= available`` test being true for a negative - nominal now parses to 200 frames. Measured over all 1,509 octet boundaries of - ``examples/captures/dhcp.pcapng`` with one harness either side: 6 levels parsed before - and 1,495 do now, against 1,479 ``ValueError`` and 10 ``struct.error`` before and none - from outside the library after, with the frame count degrading monotonically with the - cut; of the 14 levels that still raise, twelve leave a file too short to hold a block at - all and the other two cut into the Interface Description Block's ``if_tsresol`` option, - so none of them costs a frame that was in the file. That claim is about the truncation - sweep rather than about every input: a 4,000-round bounded mutation fuzz either side of - the change, same seed and cap, takes the parse rate from 2,118 to 3,438 and removes both - of :issue:`678`'s families, and the two foreign families left are measured unchanged -- - ``ValueError: N is not a valid BlockType``, filed as :issue:`701`, where an unassigned block type - raises from ``aenum`` inside ``EnumField.post_process`` so the ``UnknownBlock`` default - the registry declares is unreachable; and ``MemoryError`` at exactly 30 of 4,000 in both - trees, which is :pr:`593`'s 32-bit band. A third bare exception in this module *is* fixed here, - found by the cross-review and reachable from valid input rather than truncated: - ``SystemdJournalExportBlock.post_process`` unpacked a 64-bit binary-field length out of - whatever ``entry_data.read(8)`` returned, and since the block body is padded to a 32-bit - boundary with NULs that ``bytes.strip()`` does not strip, that padding was read as a field - *name* with no length prefix behind it -- so every journal entry of unaligned length raised - a bare ``struct.error``, measured on a 14-octet ``MESSAGE=hello`` entry -- a defect about - ordinary input rather than about truncation. A NUL-only line now ends the entry and a short - prefix ends it with a warning. The same function leaked two more of the family, found on the - cross-review's second pass and fixed with it: a binary field's declared length is the widest - in the format and nothing bounded it against the entry holding it, so at ``2**63`` and above - ``BytesIO.read`` refused it with a bare ``OverflowError`` while below that it silently - returned whatever was there -- the same malformed prefix fatal or invisible by magnitude - alone, now clamped to what the entry has left and reported; and a field name, key or value - that is not UTF-8 raised a bare ``UnicodeDecodeError``, fatal to a whole extraction over one - octet in one field, now decoded with ``errors='replace'`` and reported, which is the option - this module's own ``StringField`` already takes. The separate silent loss of every field - after a binary one, from a ``read()`` that reaches EOF where it means to skip one newline, is - filed as :issue:`704` rather than fixed. The five deliberately unclamped non-packet - option areas keep the per-block framing assumption :pr:`676`'s note describes, which this does - not close. Also - here because it is the fifth and last registrar in the package with no overwrite guard - at all: ``Option.register`` now reports a displaced option schema as a - ``RegistryWarning`` naming every namespace the ``ns='opt'`` fan-out displaced something - in, once per registration rather than once per namespace, tested by membership rather - than by subscripting because the per-namespace registries are plain ``defaultdict``\ s - and reading one to look would insert ``UnknownOption`` for a code nobody registered. - The firing condition is identity-based here too, the same as the seven code-keyed - parser registrars and the narrower guard :pr:`681` gave ``register_protocol`` (:issue:`718`, - :pr:`726`), even though this key is a caller-supplied ``code`` that ``__init_subclass__`` - passes once per subclass; a namespace the call itself creates is exempt, because it - starts as a copy of ``opt``'s defaults and nothing in it is a prior registration. - One draft did move ``EXPECTED_FAILURES``, and it is worth recording: flooring the - three decryption-secrets payloads that read ``__length__`` *whole* rather than - subtracting from it packs nothing, - because ``Schema.pack`` leaves that key at ``-1`` for "unknown" -- which emptied both - payloads and turned ``pcapng-secrets/TLS_Key_Log`` and ``.../WireGuard_Key_Log`` from - ``MISMATCH`` to ``OK``, an empty payload comparing equal to an empty payload. Reverted - and pinned; the entry count holds at 43 with its 35 PCAP-NG cases unmoved. Ships - labelled breaking: well-formed captures are byte-identical, verified by regenerating - ``examples/captures/pcapng.txt`` either side of the change, but any truncated PCAP-NG - now yields the frames before the cut where it previously raised -- so a caller reading - "extraction raised" as "this file is unusable" gets a partial result instead, whose last - frame may carry zero-padded octets -- and three exception classes change at the margins, - eight short-file depths moving from ``ProtocolError: unknown byteorder magic`` to - ``StreamEOFError``. The two modules hold 99.93% coverage with their 25 new statements - covered and their single miss flat, and ``examples/captures/pcapng.txt`` does not move - further, leaving the :pr:`683` drift :issue:`685` tracked exactly as it was (:issue:`678`). -* HIP's ``R1_COUNTER``/``R1_Counter`` parameter packed 12 octets - where :rfc:`7401#section-5.2.3` requires 16, and ``LOCATOR_SET`` declared its - ``Length`` in 4-octet units where :rfc:`7401#section-5.2.1`'s ``Length`` is a - byte count -- two independent defects, fixed together because the evidence - for either needed the other out of the way first. ``R1CounterParameter``'s - ``counter`` was a ``UInt32Field`` where the RFC states the R1 generation - counter's width twice, as "8 bytes" in the diagram and as "a 64-bit unsigned - integer" in prose; both HIP versions whose *packing* reaches this class -- - ``R1_Counter`` (128) and ``R1_COUNTER`` (129) -- packed a 12-octet record, landing at - ``4 (mod 8)`` instead of aligned. It is a ``UInt64Field`` now, and the record - is 16 octets (:issue:`672`). ``LocatorSetParameter``'s own ``Length`` was written as - ``sum(Locator.len)``, in the 4-octet units :rfc:`8046#section-4` gives - ``Locator Length``, where this parameter's own ``Length`` is a byte count; a - set of *n* plain IPv6 locators therefore declared ``4n`` octets against - ``24n`` actually present, and ``Schema.unpack`` handed the nested - ``ListField`` only the declared octets, so ``n = 2`` and ``n = 5`` both - parsed one truncated locator and left the rest of the record unconsumed. - ``HIP._make_param_locator_set`` now sums ``8 + Locator.len * 4`` per locator - -- the fixed header plus the RFC 8046 contents each one actually carries - (:issue:`679`). A second, independent defect in the same parameter's padding is - fixed alongside it, one :issue:`651` deliberately left alone because it happened to - cancel this one exactly: the nested ``Locator`` schemas share the - parameter's packet context and pack their own ``len`` over it, so - ``padding`` -- evaluated after the list -- always saw 4 rather than the - parameter's real length. A new ``locator_set_len_callback`` snapshots - ``Length`` under a private key before any locator packs into the context, - and ``locator_set_padding_len`` reads that snapshot rather than the shadowed - one. An RFC-only byte-stride walk over - ``examples/captures/options-internet.pcap``, independent of pcapkit's own - parser, went from 1 violation -- ``LOCATOR_SET``'s empty record, concealed - until now because ``R1_COUNTER``'s ``counter`` field defaulted to ``0`` with - no override anywhere in ``examples/generators/options.py``, a value the - width defect could not be told apart from -- to 2 once the counter was - patched non-zero, to 0 once both - parameters were fixed; the fixture now overrides both ``R1_COUNTER`` codes - with ``counter=0xaabbccdd`` so a future regression cannot hide behind a zero - again. 10 new test methods across two new files -- 4 in - ``test_hip_r1_counter_width_unit.py``, 6 in - ``test_hip_locator_set_length_unit.py`` -- fail against the unfixed code and - pass against the fix; the three HIP modules stay at 100% coverage, and - ``tests/`` subtests move 586 to 623. ``EXPECTED_FAILURES`` is 43 entries: - the ``hip-parameter/R1_Counter`` entry -- code 128 registered no schema of - its own, since ``R1CounterParameter`` declared only ``code=129`` -- was - deleted once that separate registry defect was fixed as :issue:`690`. ``HIP_COPIES`` - was dropped to one as :issue:`689`, once :issue:`672` and :issue:`679` left it routing around - nothing. -* ``httpv1.HTTP`` raised a bare exception for any payload that is - not an HTTP/1 message, at four sites: the header/body split in ``read``, the - header's CRLF split and the start-line split in ``_read_http_header``, and - ``item[1]`` on a field line with no colon. ``HTTP._guess_version`` falls - through on ``ProtocolError`` alone, so the HTTP/1 attempt aborted the guess - rather than failing it and the HTTP/2 arm below was dead code -- for every - one of those four input classes, not just the first. All four now raise - ``ProtocolError``, which each method's ``Raises:`` section already - documented, and a test pins the invariant rather than the list: nothing - escapes ``httpv1.HTTP`` that is not a ``ProtocolError``, so a fifth - bare-raising site would fail there. An obs-fold continuation line - [:rfc:`9112#section-5.2`] is legal HTTP/1 and is now unfolded, the RFC's own - remedy, rather than read as a field of its own: with no colon it raised that - ``IndexError``, and with one it parsed in silence into a spurious extra - field. Reaching the HTTP/2 arm exposed a payload under nine octets to - ``httpv2.HTTP`` for the first time, where it usually fails inside the schema - with a bare ``struct.error`` -- neither a ``ValueError`` nor a - ``ProtocolError`` nor pcapkit's ``StructError``, so no caller could catch it; - ``read`` converts it and ``_guess_version`` suppresses it on the last arm - only, since an arm that is not last hands the payload onward when it - swallows. Separately, ``PayloadField.protocol`` looked a ``str`` up in - ``__proto__`` as given while the registry is keyed on the upper-cased class - name, so a name not already upper-case resolved to ``None`` and silently - yielded ``Raw``; it now folds case and warns with ``RegistryWarning`` for a - genuinely unregistered name, and ``__init__`` assigns through that property - instead of writing ``_protocol`` past it (:issue:`787`). -* a systemd journal entry whose last field lacks the format's - mandatory trailing newline let the PCAP-NG block's own 32-bit alignment - padding be read as part of that field's value, with no exception and no - warning: ``b'MESSAGE=hello'`` plus three padding NULs returned - ``MESSAGE == 'hello\x00\x00\x00'``. - The padding-only-line guard from :issue:`678` only catches - padding that lands on a line of its own, which requires the preceding field - to have ended with a real newline; without one ``readline()`` runs straight - through the value and into the padding behind it. Reachable on text fields - only -- ``readline()`` returns a line without its own newline at end of - stream alone, so a binary field's length-prefixed value read is never in - play, and a binary field *name* landing on such a line fails its own - length-prefix read regardless. The fix is tolerant rather than refusing: - block padding is 0-3 octets, so at most the trailing three octets of a - terminator-less line are stripped as padding, with a ``SchemaWarning``, and - anything past that cannot be padding and is left as data. The gate reads the - *raw*, unstripped line, since a real trailing newline -- or any other raw - last octet, ordinary ASCII whitespace included -- proves the true padding is - zero (:issue:`794`). ``examples/generators/pcapng.py``'s ``_journal_entry`` cited a - bare line number for the defect its alignment workaround exists to avoid, - and mis-attributed the fix; it now names :issue:`678`'s own fix, the - ``not line.strip(b'\x00')`` guard, and the mechanism instead, the parser - rewrite having only moved that guard (:issue:`791`). -* ``httpv2.HTTP.read`` tested only the 24-bit declared length off - the wire (``schema.length < 9``), never how many octets the buffer actually - held, so a frame backed by far fewer real octets than it declared still - parsed and reported the declared, attacker-controlled length. Sweeping the - declared length against a four-octet buffer gave non-monotone - accept/reject -- 9, 15, 65535 and 16777215 parsed, 10 and 100 did not -- an - artifact of ``_read_http_settings``'s own unrelated ``(declared - 9) % 6`` - check rather than evidence the buffer held what was declared. - ``HTTP.unpack`` now rejects a non-empty buffer under nine octets before the - schema layer runs, preserving ``StreamEOFError`` for a genuinely exhausted - implicit-length stream, and ``read``'s guard also requires the declared - length not to exceed the available buffer. A buffer that clears nine octets - can still carry a frame type whose own fixed-width fields exceed what is - left after the header -- GOAWAY at 9-16 octets, PUSH_PROMISE at 9-12, - padded DATA, HEADERS and PUSH_PROMISE -- and still crash the schema layer - with a bare ``struct.error``; that root cause is in the generic - ``Schema.unpack``/``FieldBase.length`` machinery shared by ten schema - modules, out of scope there and filed as :issue:`805`, which is why - ``_guess_version``'s last arm keeps suppressing ``struct.error`` alongside - ``ProtocolError`` (:pr:`802`). -* ``SystemdJournalExportBlock.post_process`` skipped a binary - field's one trailing newline with a bare ``entry_data.read()``, which reads - to **EOF** rather than past that one octet: the entry loop's next - ``readline()`` then returned ``b''`` and ended the entry, discarding every - field behind the first binary one, however well-formed. The skip is now - bound to ``entry_data.read(1)``; a missing or non-``b'\n'`` octet ends the - entry with a ``SchemaWarning`` instead of resynchronising at the wrong - offset. ``test_journal_fields_following_a_binary_field_are_not_discarded`` - fails against the unfixed code with ``MissingKeyError: 'SECOND'`` and passes - after (:issue:`704`). -* the same block's entries were split on a bare ``b'\n\n'``, - which shreds a binary field whose value happens to contain that byte pair. - Walking the entry instead of splitting it introduced its own regression on - first review: a ``malformed`` flag broke the *outer* per-block loop on a bad - terminator, ending the whole block rather than just the one entry and - silently dropping every well-formed entry behind it. Fixed by dropping - ``malformed`` entirely, leaving the existing per-entry ``break`` as the only - effect. A second, independent gap closed in the same change: a trailing - separator landing on the block's last octet was indistinguishable from EOF - and swallowed, so a rebuild wrote ``length`` one octet short (:issue:`723`). -* **a breaking change to** four ``.get()``-backed enum fields: - TCP/UDP/SCTP's ``PortEnumField`` and PCAP-NG's ``OptionEnumField`` minted a - member through ``aenum.extend_enum`` for every value no row or documented - span covers -- 111 calls on ``http.pcap``, all ephemeral ports. All four now - peek at what ``.get()`` would consult and mint only once the value is - actually unresolvable; a genuine miss gets ``EnumField._unregistered_member`` - instead, a real member of the same registry built through its storage - base's ``__new__``, so ``isinstance`` still holds and the lookup table stops + ``(pkt['__length__'] - pkt['pad_len']) if ... else 0``, and the ``else`` arm + returned ``0`` ("read no octets") where it meant to subtract ``0``. Padding is rare + in HTTP/2, so the broken arm was the common one, and nothing raised or warned: + ``info.data`` was ``b''``. An unpadded ``DATA`` frame declaring ``b'{"ok":true}\n'`` + now parses as those twelve octets, and unpadded ``HEADERS`` and ``PUSH_PROMISE`` + keep their header block fragment. Padded frames are unaffected, since + ``(A - B) if T else 0`` and ``A - (B if T else 0)`` are both ``A - B`` for truthy + ``T``; the padded arm has no off-by-one (:issue:`668` asked that be checked), + because ``Schema.unpack`` decrements ``packet['__length__']`` by each field's width, + so ``pad_len``'s own octet is already out by the time the payload field runs. Three + sibling sites that were never wrong (``ContinuationFrame.fragment``, + ``UnassignedFrame.data``, ``GoawayFrame.debug``) localise the defect to the grouping + and are pinned as controls. Nothing caught it: ``test_option_roundtrip_unit`` drives + every frame type through ``make`` -> parse -> ``make``, where both sides agreed on + an empty payload (its generator passes no payload for these frames), and + ``test_httpv2_frame_readers_cover_successful_frames`` uses hand-built schema stubs, + so the callback never ran. The same lambda also governs **packing**, which neither + the issue nor the first revision noticed and a cross-review on a second model did: + ``BytesField`` consults ``length`` on both paths, so an unpadded ``DATA`` frame + carrying twelve octets packed to a nine-octet header whose length field declares + **21**, body absent (unpadded ``HEADERS`` and ``PUSH_PROMISE`` declared 29 against 9 + and 33 against 13). pcapkit was *emitting* malformed HTTP/2 that desynchronises any + reader walking a stream by that field; a ``make`` -> parse -> ``make`` cycle of a + non-empty unpadded payload now closes byte-for-byte + (``HTTPv2ConstructedFrameDeclaresWhatItWritesUnitTests``). The new + ``tests/protocols/application/test_httpv2_payload_length_unit.py`` reads wire octets + (23 tests, 12 subtests over padded, unpadded and padding-only shapes of all three + frames plus both ``PRIORITY`` combinations) and asserts payload *octets*, never + length, because a length assertion survives the right width read from the wrong + offset; padded cases use non-zero padding ``b'\xde\xad\xbe\xef'`` rather than the + zeros :rfc:`9113#section-6.1` tells a sender to use, so failing to subtract it + yields a different byte string. Six wrong fixes written over the schema confirm the + tests discriminate, each rejected; the cross-review's own, ``max(computed, 1)`` on + the two ``fragment`` fields, **passed** the first revision's tests because only + ``DataFrame`` had a padding-only case (a padding-only ``HEADERS`` frame then leaked + one padding octet); the missing padding-only cases and + ``test_the_padded_arm_reaches_zero_and_is_not_clamped`` close it, ``0`` being a + legitimate answer. ``EXPECTED_FAILURES`` is unmoved (its one HTTP/2 entry + ``httpv2-frame/PRIORITY`` keeps its ``CONSTRUCT`` status, ``PriorityFrame`` having + no payload field or ``PADDED`` flag; its cited ``httpv2.py:572`` is stale, the + ``header.length != 9`` guard now being at ``:650``). The construct side needed no + change: ``_make_http_length`` always computed + ``len(frame.data) + (pad_len + 1 if pad_len else 0)``, and the shared length + callback disagreed on both paths. An AST sweep of all 496 package files found one + other site of the same shape, ``pcapkit/protocols/schema/internet/ipv4.py:336`` + (``TSOption.remainder``), and it is correct: its ``else 0`` is intentional because + ``ts_data`` consumes the whole option data area in that arm. Ships labelled + breaking: restoring a dropped payload moves the parse output of essentially every + HTTP/2 capture, so a golden file or regression baseline that recorded the empty + value sees it change, and constructed frames change byte-for-byte, from malformed to + correct (:issue:`668`). +* the schema half of every option-like registrar overwrote silently while the parser + half of the same call warned. ``EnumSchema.register`` was a bare + ``cls.__enum__[code] = schema``, and is the schema half of 14 public registrars, so + one ``register_ipv4_option`` call replacing a built-in named the parser it displaced + and said nothing about the schema. ``EnumSchema.__init_subclass__`` reaches the same + registry without calling ``register``, so a subclass declared with a ``code=`` + keyword (the documented way to add a schema) stayed silent too; both are guarded, + and the declaration hook's two branches are folded into one loop. Presence is a + faithful test only because ``_EnumRegistry.__missing__`` returns a miss without + recording it, so parsing an unknown code cannot make the next legitimate + registration warn. The seven code-keyed parser registrars (on ``ProtocolBase``, + ``Link``, ``Internet``, ``Frame``, ``PCAPNG``, ``Transport`` and ``SCTP``) now name + the displaced entry and its replacement, where they reported only that something was + overwritten. Their firing condition is harmonised onto the narrower guard :pr:`681` + gave ``register_protocol`` (:issue:`718`, :pr:`726`) rather than presence alone: + that key is derived from the value it stores, whereas these take a caller-supplied + ``code``. Five of the seven (``Link`` 7 entries, ``Internet`` 16, ``Frame`` 3, + ``PCAPNG`` 3, ``SCTP`` 2) ship pre-seeded with unresolved ``ModuleDescriptor`` + values, as do ``TCP`` (4) and ``UDP`` (3), each keeping its own dict rather than + filling the one ``ProtocolBase`` and abstract ``Transport`` share and leave at 0 + (``Transport.register`` raises ``UnsupportedCall``). Where a table is pre-seeded, + the incumbent may be a two-string descriptor while the replacement is the class it + names. Resolution happens earlier, on the incoming argument only + (``if isinstance(protocol, ModuleDescriptor): protocol = protocol.klass``), so the + guard's ``is`` comparison measures a resolved class against an unresolved descriptor + and warns on re-registering the same class under its own pre-seeded code + (``Internet.register(TransType.TCP, TCP)``), a false positive the harmonised guard + does not close. ``ContextRegistry.register`` already raises on a duplicate, and the + reassembly and ESP registrars append to lists with no key (duplicate + security-association SPIs are designed, resolved by scoring), so none of the three + takes a guard. ``import pcapkit`` holds at one warning, a third-party deprecation, + and no ``RegistryWarning``, over the 327 registry writes it performs (one being + ``R1CounterParameter``'s second code, from :issue:`690`), none landing on a key + already present (:issue:`692`). +* every EOF-truncated PCAP-NG file raised a bare ``ValueError`` out of ``Extractor``, + losing the whole extraction rather than the one truncated block. Block Total Length + is cross-checked against its trailing copy and never against the file, so a last + block declaring more than the file holds left ``PCAPNG.read`` seeking *past* the + end, and the next block read measured a **negative** remainder that + ``io.RawIOBase.read`` refused. The seek is now clamped to the octets + ``_read_fileng`` returned, with a ``ProtocolWarning`` naming the overrun, and a tail + under the twelve octets a block needs is the quiet ``StreamEOFError`` that + ``Extractor.record_frames`` already catches (deliberately not a clamp, which would + fabricate a frame out of zero padding). A second route to the same shape is closed + by a new ``nonnegative()`` helper, composed into ``bounded_option`` and + ``bounded_area`` and applied to the eight unwrapped ``length - N`` spans and eight + ``__option_padding__`` fields: ``_TextField.__call__`` builds ``f'{length}s'`` + unconditionally, so a negative became ``'-8s'`` and ``struct.calcsize`` raised an + exception caught nowhere a frame loop looks. Across all 1,509 octet boundaries of + ``examples/captures/dhcp.pcapng``, 6 levels parsed before and 1,495 do now; the 14 + that still raise are twelve files too short to hold a block and two cuts into the + Interface Description Block's ``if_tsresol`` option, so none costs a frame that was + in the file. A mutation fuzz removes both of :issue:`678`'s families; two others + remain, unchanged: ``ValueError: N is not a valid BlockType`` (filed as + :issue:`701`; an unassigned block type raises from ``aenum`` inside + ``EnumField.post_process``, so the registry's ``UnknownBlock`` default is + unreachable) and ``MemoryError`` from :pr:`593`'s 32-bit band. Three bare exceptions + in ``SystemdJournalExportBlock.post_process``, reachable from valid input, are fixed + with it: the body's NUL alignment padding was read as a field *name*, so every + journal entry of unaligned length raised ``struct.error`` (a NUL-only line now ends + the entry); a binary field's declared length was unbounded against its entry + (``OverflowError`` at ``2**63`` and above, silently wrong below; now clamped and + reported); and a non-UTF-8 name, key or value raised ``UnicodeDecodeError`` (now + ``errors='replace'`` and reported, as ``StringField`` does). The silent loss of + every field after a binary one is filed as :issue:`704`. The five unclamped + non-packet option areas keep the per-block framing assumption :pr:`676`'s note + describes. Also here, the fifth and last registrar with no overwrite guard: + ``Option.register`` now reports a displaced option schema as a ``RegistryWarning`` + naming every namespace the ``ns='opt'`` fan-out displaced something in, once per + registration, tested by membership (subscripting the per-namespace ``defaultdict`` + would insert ``UnknownOption`` for a code nobody registered). The condition is + identity-based like the seven parser registrars and :pr:`681`'s guard for + ``register_protocol`` (:issue:`718`, :pr:`726`); a namespace the call itself creates + is exempt. Flooring the three decryption-secrets payloads that read ``__length__`` + *whole* was tried and reverted: ``Schema.pack`` leaves that key at ``-1`` for + "unknown", so it packed nothing and turned ``pcapng-secrets/TLS_Key_Log`` and + ``.../WireGuard_Key_Log`` from ``MISMATCH`` to ``OK`` (an empty payload equals an + empty payload). Ships labelled breaking: well-formed captures are byte-identical, + but any truncated PCAP-NG now yields the frames before the cut where it raised, so a + caller reading "extraction raised" as "this file is unusable" gets a partial result + whose last frame may carry zero-padded octets, and eight short-file depths move from + ``ProtocolError: unknown byteorder magic`` to ``StreamEOFError``. The :pr:`683` + drift :issue:`685` tracks is unchanged (:issue:`678`). +* HIP's ``R1_COUNTER``/``R1_Counter`` parameter packed 12 octets where + :rfc:`7401#section-5.2.3` requires 16, and ``LOCATOR_SET`` declared its ``Length`` + in 4-octet units where :rfc:`7401#section-5.2.1`'s ``Length`` is a byte count -- two + independent defects, fixed together because the evidence for either needed the other + out of the way. ``R1CounterParameter``'s ``counter`` was a ``UInt32Field`` where the + RFC states the width twice, "8 bytes" in the diagram and "a 64-bit unsigned integer" + in prose; both versions whose *packing* reaches the class, ``R1_Counter`` (128) and + ``R1_COUNTER`` (129), packed a 12-octet record at ``4 (mod 8)``. It is a + ``UInt64Field`` now and the record is 16 octets (:issue:`672`). + ``LocatorSetParameter``'s ``Length`` was ``sum(Locator.len)``, in the 4-octet units + :rfc:`8046#section-4` gives ``Locator Length``; a set of *n* plain IPv6 locators + declared ``4n`` octets against ``24n`` present, and ``Schema.unpack`` handed the + nested ``ListField`` only the declared octets, so ``n = 2`` and ``n = 5`` parsed one + truncated locator and left the rest unconsumed. ``HIP._make_param_locator_set`` now + sums ``8 + Locator.len * 4`` per locator (:issue:`679`). A second defect in the same + parameter's padding, which :issue:`651` left alone because it cancelled this one, is + fixed alongside: the nested ``Locator`` schemas share the parameter's packet context + and pack their own ``len`` over it, so ``padding``, evaluated after the list, saw 4 + rather than the real length. A new ``locator_set_len_callback`` snapshots ``Length`` + under a private key before any locator packs, and ``locator_set_padding_len`` reads + that snapshot. The ``counter`` defaulted to ``0`` and the width defect could not be + told apart from it, so the fixture now overrides both ``R1_COUNTER`` codes with + ``counter=0xaabbccdd``; new tests are ``test_hip_r1_counter_width_unit.py`` and + ``test_hip_locator_set_length_unit.py``. The ``hip-parameter/R1_Counter`` + ``EXPECTED_FAILURES`` entry (code 128 registered no schema of its own, since + ``R1CounterParameter`` declared only ``code=129``) was deleted once that registry + defect was fixed as :issue:`690`, leaving 43 entries. ``HIP_COPIES`` dropped to one + as :issue:`689`, once :issue:`672` and :issue:`679` left it routing around nothing. +* ``httpv1.HTTP`` raised a bare exception for any payload that is not an HTTP/1 + message, at four sites: the header/body split in ``read``, the header's CRLF split + and the start-line split in ``_read_http_header``, and ``item[1]`` on a field line + with no colon. ``HTTP._guess_version`` falls through on ``ProtocolError`` alone, so + the HTTP/1 attempt aborted the guess and the HTTP/2 arm below was dead code for all + four input classes. All four now raise ``ProtocolError``, which each method's + ``Raises:`` section documented, and a test pins the invariant (nothing escapes + ``httpv1.HTTP`` that is not a ``ProtocolError``), so a fifth bare-raising site would + fail there. An obs-fold continuation line [:rfc:`9112#section-5.2`] is legal HTTP/1 + and is now unfolded, the RFC's remedy, rather than read as a field of its own (with + no colon it raised that ``IndexError``; with one it parsed into a spurious extra + field). Reaching the HTTP/2 arm exposed a payload under nine octets to + ``httpv2.HTTP``, where it usually fails inside the schema with a bare + ``struct.error`` (not a ``ValueError``, ``ProtocolError`` or pcapkit ``StructError``, + so uncatchable); ``read`` converts it and ``_guess_version`` suppresses it on the + last arm only, since an arm that is not last hands the payload onward when it + swallows. Separately, ``PayloadField.protocol`` looked a ``str`` up in ``__proto__`` + as given while the registry is keyed on the upper-cased class name, so a name not + already upper-case resolved to ``None`` and yielded ``Raw``; it now folds case and + warns with ``RegistryWarning`` for a genuinely unregistered name, and ``__init__`` + assigns through that property instead of writing ``_protocol`` past it (:issue:`787`). +* a systemd journal entry whose last field lacks the format's mandatory trailing + newline let the PCAP-NG block's 32-bit alignment padding be read as part of that + field's value, with no exception or warning: ``b'MESSAGE=hello'`` plus three NULs + returned ``MESSAGE == 'hello\x00\x00\x00'``. The padding-only-line guard from + :issue:`678` catches padding only on a line of its own, which needs the preceding + field to have ended with a real newline; without one ``readline()`` runs through the + value into the padding. Reachable on text fields only (a binary field's + length-prefixed read is never in play, and a binary field *name* on such a line fails + its own length-prefix read regardless). The fix is tolerant: block padding is 0-3 + octets, so at most the trailing three octets of a terminator-less line are stripped, + with a ``SchemaWarning``, and anything past that is left as data. The gate reads the + *raw* line, since a real trailing newline, or any other raw last octet including + ASCII whitespace, proves the padding is zero (:issue:`794`). + ``examples/generators/pcapng.py``'s ``_journal_entry`` cited a bare line number for + the defect its alignment workaround avoids and mis-attributed the fix; it now names + :issue:`678`'s ``not line.strip(b'\x00')`` guard and the mechanism (:issue:`791`). +* ``httpv2.HTTP.read`` tested only the 24-bit declared length off the wire + (``schema.length < 9``), never how many octets the buffer held, so a frame backed by + far fewer real octets than declared parsed and reported the declared, + attacker-controlled length. Sweeping the declared length against a four-octet buffer + gave non-monotone accept/reject (9, 15, 65535 and 16777215 parsed, 10 and 100 did + not), an artifact of ``_read_http_settings``'s own ``(declared - 9) % 6`` check. + ``HTTP.unpack`` now rejects a non-empty buffer under nine octets before the schema + layer runs, preserving ``StreamEOFError`` for a genuinely exhausted implicit-length + stream, and ``read``'s guard also requires the declared length not to exceed the + buffer. A buffer that clears nine octets can still carry a frame type whose fixed + fields exceed what is left (GOAWAY at 9-16 octets, PUSH_PROMISE at 9-12, padded + DATA, HEADERS and PUSH_PROMISE) and crash the schema layer with a bare + ``struct.error``; that root cause is in the generic ``Schema.unpack``/ + ``FieldBase.length`` machinery shared by ten schema modules, out of scope and filed + as :issue:`805`, which is why ``_guess_version``'s last arm keeps suppressing + ``struct.error`` alongside ``ProtocolError`` (:pr:`802`). +* ``SystemdJournalExportBlock.post_process`` skipped a binary field's one trailing + newline with a bare ``entry_data.read()``, which reads to **EOF**: the next + ``readline()`` returned ``b''`` and ended the entry, discarding every field behind + the first binary one. The skip is bound to ``entry_data.read(1)``; a missing or + non-``b'\n'`` octet ends the entry with a ``SchemaWarning`` instead of + resynchronising at the wrong offset. + ``test_journal_fields_following_a_binary_field_are_not_discarded`` fails against the + unfixed code with ``MissingKeyError: 'SECOND'`` (:issue:`704`). +* the same block's entries were split on a bare ``b'\n\n'``, which shreds a binary + field whose value contains that pair. Walking the entry instead of splitting it + introduced a regression on first review: a ``malformed`` flag broke the *outer* + per-block loop on a bad terminator, ending the whole block and dropping every + well-formed entry behind it. Fixed by dropping ``malformed``, leaving the per-entry + ``break`` as the only effect. A second gap closed in the same change: a trailing + separator on the block's last octet was indistinguishable from EOF and swallowed, so + a rebuild wrote ``length`` one octet short (:issue:`723`). +* **a breaking change to** four ``.get()``-backed enum fields: TCP/UDP/SCTP's + ``PortEnumField`` and PCAP-NG's ``OptionEnumField`` minted a member through + ``aenum.extend_enum`` for every value no row or documented span covers (111 calls on + ``http.pcap``, all ephemeral ports). All four now peek at what ``.get()`` would + consult and mint only once the value is unresolvable; a genuine miss gets + ``EnumField._unregistered_member``, a real member of the same registry built through + its storage base's ``__new__``, so ``isinstance`` holds and the lookup table stops growing. ``extend_enum`` calls drop 111 to 0 on ``http.pcap`` and 15 to 2 on - ``many_interfaces.pcapng`` (both from the untouched ``_missing_``). Riding - along: PCAP-NG option namespaces stop cross-contaminating -- on the unfixed - tree, once a code was minted under ``opt``, ``OptionType.get`` merged - ``opt`` over the requested namespace and 6 of the other 7 namespaces all - answered ``opt_unknown`` for it, where each now gets its own + ``many_interfaces.pcapng`` (both from the untouched ``_missing_``). Riding along: + PCAP-NG option namespaces stop cross-contaminating: once a code was minted under + ``opt``, ``OptionType.get`` merged ``opt`` over the requested namespace and 6 of the + other 7 namespaces answered ``opt_unknown`` for it; each now gets its own ``_unknown``. **Breaking** because :issue:`575` asked for byte-identical output across the sample captures, as :pr:`420` and :pr:`427` did, and this deliberately - misses that bar: 14 of 23 captures change, entirely from the new - ```` rendering plus the PLIST escaping above; normalising both - leaves all 14 byte-identical. A ``pickle`` round-trip of a resolved - unassigned member worked before this change (it was minted, so registered) - and would have broken silently on read-back after; ``_unregistered_member`` - now installs a ``__reduce_ex__`` rebuilding an equivalent unregistered - member instead, verified on protocols 0-5 under both picklers and + misses that bar: 14 of 23 captures change, entirely from the new ```` + rendering plus the PLIST escaping above; normalising both leaves all 14 + byte-identical. A ``pickle`` round-trip of a resolved unassigned member worked before + (it was minted, so registered) and would have broken silently after; + ``_unregistered_member`` now installs a ``__reduce_ex__`` rebuilding an equivalent + unregistered member, checked on protocols 0-5 under both picklers and ``copy``/``deepcopy`` (:issue:`575`). -* ``httpv2._guess_version`` tried to parse a stream as HTTP/2 and - read failure as "not HTTP/2," rather than identifying the version first, so - a genuine connection preface followed by a real ``SETTINGS`` frame still - raised ``ProtocolError: unknown HTTP version`` instead of parsing. - ``_guess_version`` now identifies before it parses: a preface is recognised - as HTTP/2 outright, skipped rather than fed to ``httpv2``, and counted as - header so ``info`` stays byte-identical to reading the following frame - alone -- without which the injected ``packet=self.packet.payload``, sliced - from octet 9 of a buffer whose frame starts at 24, reported preface - remnants as payload. A preface with no frame behind it now raises - ``ProtocolError: HTTP/2: connection preface with no frame``, naming the real - cause; a bare mid-stream ``SETTINGS`` frame, an ``Upgrade: h2c`` request and - non-HTTP input are all unchanged, the first two left undecidable on - purpose -- ``Upgrade: h2c`` needs per-connection state a single payload - cannot carry, and a mid-stream frame has no safe heuristic, since a - ``type <= 9`` guess misfires on binary HTTP/1 bodies. Protochain over all 23 - sample captures (1,604 frames, 231 HTTP-bearing) is byte-identical to the - unfixed tree (:issue:`800`). -* ``TCP.__proto__`` bound ``httpv1.HTTP`` directly for ports 80 - and 8080, so a segment on either port was HTTP/1 by assertion of the port - number alone; both versions share those ports on the wire, so the port - cannot decide the version and the payload has to. Repointed to the generic - ``pcapkit.protocols.application.http.HTTP`` proxy, whose identification - became positive only once :issue:`800` and :pr:`814` landed, which is why this waited. - ``udp.py`` already bound the proxy for both ports; this removes the - asymmetry its docstring and ``docs/source/pep.rst`` documented as an open - request, prose-only on that side since its port rows already pointed at - the proxy. Protochain over all 23 sample captures (1,604 frames) is *not* - byte-identical, and that is the fix: all 231 HTTP/1.1 frames keep their - chain, and nine frames in ``options-transport.pcap`` change - ``Ethernet:IPv4:TCP:Raw`` to ``Ethernet:IPv4:TCP:HTTP/2`` -- genuine HTTP/2 - frames this library's own ``httpv2.HTTP.make`` built, previously refused by - the HTTP/1 parser. ``_guess_version``'s entry count over the corpus goes 0 - to 252: 231 HTTP/1.1, 9 HTTP/2 and 12 fall-throughs that stay ``Raw``. Not - labelled breaking, but not free: TCP:80/8080 traffic that is neither valid - HTTP/1 nor preface-carrying is now exposed to ``_guess_version``'s - fall-through arm, where the direct ``httpv1`` binding used to leave it - ``Raw`` outright regardless of shape -- the 12 fall-throughs above are that - arm firing on this corpus, and a caller depending on the old direct bind - for traffic that is neither is the one who would notice (:issue:`682`). -* **a breaking change to** 8 more bespoke registries: step 2 of - :issue:`860`, PR 1 of 2 (``AppType`` is PR 2, :pr:`874` below), bringing ``StatusCode``, - ``ReturnCode``, ``ResponseKind``, ``GroupingInformation``, ``OptionType``, - ``FEATCode``, ``Command`` and ``Method`` onto ``EnumRegistry`` and applying - :issue:`775`'s mint/unmint ruling to their ``get()`` as well as their ``_missing_``. - The first five convert 12 unambiguous placeholder branches (``Unassigned``, - ``Unknown``, ``opt_unknown``) to ``_unregistered_member``; each of the three - with a custom ``__new__`` (``StatusCode``, ``ReturnCode``, ``OptionType``) - gets its own override reconstructing the attributes the base's generic helper - would otherwise leave unset, and ``StatusCode``/``ReturnCode``'s hand-written - ``get()`` -- still on the retired ``default == -1`` convention -- is replaced - by the base's, with no caller in this tree found relying on the old form. - ``OptionType`` keeps its own ``get()`` for its genuine multi-namespace - dispatch, but a round-2 review found ``get()`` itself still minted on both - its int/namespace path and its ``str`` path -- the live pcapng parse path - (``PCAPNG._make_pcapng_options``) calls it with wire bytes, so parsing an - undeclared option code was still registering a permanent member; both paths - now build an unregistered member too. ``FEATCode``, ``Command`` and - ``Method`` mint the literal, unmodified wire value as its own name rather - than any manufactured placeholder; per the owner's ruling (*"get will not - have sufficient information to create new ones"* -- ``Command`` needs - ``feat``/``desc``/``type``/``conf`` and ``Method`` needs - ``safe``/``idempotent``, neither of which a bare wire string carries), both - ``_missing_`` and each class's own ``get()`` -- a second, independent mint - site bypassing ``_missing_`` -- now build an unregistered member instead. - ``FEATCode``'s own crawler now declares all 15 real FEAT-code values from the - live IANA table instead of minting 10 of them as a side effect of evaluating - ``Command``'s rows at import time, the same import-time-mutation shape :pr:`861` - removed from ``FilterType``; pinned count-agnostically so a future IANA - update cannot fail a correct regeneration. Untouched, per explicit rulings: - ``CommandType`` stays ``IntFlag`` (real ``A|P`` composites in the generated - data), and ``TransportProtocol``'s ``auto()`` renumbering plus ``AppType`` - itself are PR 2's territory. All five crawlers regenerate byte-identically on - a third run. 332 test methods pass across the seven touched test files plus - ``tests/protocols/misc/test_pcapng_unit.py``; touched const modules land at - 92-100% coverage. ``pcapkit/protocols/schema/misc/pcapng.py``'s - ``OptionEnumField.post_process`` docstring is corrected to explain that it - still bypasses ``OptionType.get()`` directly -- now for ``pickle`` round-trip - safety, since :issue:`860` already stops the minting that used to be the reason, and - neither of ``OptionType``'s own two paths gets both pickling and correct - rendering right at once (:pr:`869`). +* ``httpv2._guess_version`` tried to parse a stream as HTTP/2 and read failure as "not + HTTP/2", rather than identifying the version first, so a genuine connection preface + followed by a real ``SETTINGS`` frame raised ``ProtocolError: unknown HTTP version``. + It now identifies before it parses: a preface is recognised as HTTP/2 outright, + skipped rather than fed to ``httpv2``, and counted as header so ``info`` stays + byte-identical to reading the following frame alone (otherwise the injected + ``packet=self.packet.payload``, sliced from octet 9 of a buffer whose frame starts at + 24, reported preface remnants as payload). A preface with no frame behind it raises + ``ProtocolError: HTTP/2: connection preface with no frame``; a bare mid-stream + ``SETTINGS`` frame, an ``Upgrade: h2c`` request and non-HTTP input are unchanged, the + first two left undecidable on purpose (``Upgrade: h2c`` needs per-connection state a + single payload cannot carry, and a mid-stream frame has no safe heuristic, since a + ``type <= 9`` guess misfires on binary HTTP/1 bodies). Protochain over all 23 sample + captures (1,604 frames, 231 HTTP-bearing) is byte-identical to the unfixed tree + (:issue:`800`). +* ``TCP.__proto__`` bound ``httpv1.HTTP`` directly for ports 80 and 8080, so a segment + on either port was HTTP/1 by assertion of the port alone; both versions share those + ports, so the payload has to decide. Repointed to the generic + ``pcapkit.protocols.application.http.HTTP`` proxy, whose identification became + positive only once :issue:`800` and :pr:`814` landed, which is why this waited. + ``udp.py`` already bound the proxy for both ports; this removes the asymmetry its + docstring and ``docs/source/pep.rst`` documented as an open request (prose-only on + that side). Protochain over all 23 captures (1,604 frames) is *not* byte-identical, + and that is the fix: all 231 HTTP/1.1 frames keep their chain, and nine frames in + ``options-transport.pcap`` change ``Ethernet:IPv4:TCP:Raw`` to + ``Ethernet:IPv4:TCP:HTTP/2`` -- genuine HTTP/2 frames this library's own + ``httpv2.HTTP.make`` built, previously refused by the HTTP/1 parser. + ``_guess_version``'s entry count over the corpus goes 0 to 252 (231 HTTP/1.1, 9 + HTTP/2, 12 fall-throughs that stay ``Raw``). Not labelled breaking, but not free: + TCP:80/8080 traffic that is neither valid HTTP/1 nor preface-carrying is now exposed + to the fall-through arm, where the direct ``httpv1`` binding left it ``Raw`` + regardless; a caller depending on that is the one who would notice (:issue:`682`). +* **a breaking change to** 8 more bespoke registries: step 2 of :issue:`860`, PR 1 of 2 + (``AppType`` is PR 2, :pr:`874` below), bringing ``StatusCode``, ``ReturnCode``, + ``ResponseKind``, ``GroupingInformation``, ``OptionType``, ``FEATCode``, ``Command`` + and ``Method`` onto ``EnumRegistry`` and applying :issue:`775`'s mint/unmint ruling + to their ``get()`` as well as ``_missing_``. The first five convert 12 unambiguous + placeholder branches (``Unassigned``, ``Unknown``, ``opt_unknown``) to + ``_unregistered_member``; the three with a custom ``__new__`` (``StatusCode``, + ``ReturnCode``, ``OptionType``) get an override reconstructing the attributes the + base's generic helper would leave unset, and ``StatusCode``/``ReturnCode``'s + hand-written ``get()``, still on the retired ``default == -1`` convention, is + replaced by the base's (no caller relied on the old form). ``OptionType`` keeps its + own ``get()`` for its multi-namespace dispatch, but a round-2 review found it still + minted on both its int/namespace and ``str`` paths, and the live pcapng parse path + (``PCAPNG._make_pcapng_options``) calls it with wire bytes, so parsing an undeclared + option code still registered a permanent member; both paths now build an unregistered + member. ``FEATCode``, ``Command`` and ``Method`` mint the literal wire value as its + own name rather than any placeholder; per the owner's ruling (*"get will not have + sufficient information to create new ones"*: ``Command`` needs + ``feat``/``desc``/``type``/``conf`` and ``Method`` needs ``safe``/``idempotent``, + which a bare wire string lacks), both ``_missing_`` and each class's own ``get()`` + (a second, independent mint site) now build an unregistered member. ``FEATCode``'s + crawler now declares all 15 real FEAT-code values from the live IANA table instead of + minting 10 of them as a side effect of evaluating ``Command``'s rows at import time, + the import-time-mutation shape :pr:`861` removed from ``FilterType``; pinned + count-agnostically so a future IANA update cannot fail a correct regeneration. + Untouched, per rulings: ``CommandType`` stays ``IntFlag`` (real ``A|P`` composites in + the generated data), and ``TransportProtocol``'s ``auto()`` renumbering and + ``AppType`` are PR 2's. ``pcapkit/protocols/schema/misc/pcapng.py``'s + ``OptionEnumField.post_process`` docstring is corrected: it still bypasses + ``OptionType.get()`` directly, now for ``pickle`` round-trip safety (:issue:`860` + already stops the minting that used to be the reason), since neither of + ``OptionType``'s two paths gets both pickling and correct rendering right at once + (:pr:`869`). pcapkit.toolkit --------------- @@ -2796,89 +2018,75 @@ Added ~~~~~ * ``util/pyshark_encap_map.py``, a generator that regenerates - ``ENCAP_TYPE_TO_LINKTYPE`` (152 entries) and ``FILTER_NAME_TO_LINKTYPE`` (58) - in place inside ``pcapkit/toolkit/pyshark.py``, so the two tables :pr:`850` - hand-built stop being hand-maintained. It lives under ``util/`` rather than - ``pcapkit/vendor/`` because ``Vendor``'s base class is hardwired to an HTTP - fetch and a generated enum-class file, where this generator's source of truth - is the local ``tshark``/``editcap`` binaries. Guards against its own worst - failure mode: a naive ``LinkType(dlt)`` lookup cannot detect an unmapped DLT, - because ``_missing_`` mints a placeholder rather than raising, so the - generator snapshots every known ``LinkType`` value before any lookup and only - calls ``LinkType(dlt)`` once a value has already been confirmed a member. The - real-``tshark`` sweep is gated on a newly-declared ``HAS_WIRESHARK`` flag, - registered in ``tests/_dependency_gates.py``'s ``NON_DISTRIBUTION_FLAGS`` - next to ``HAS_PROC_FD``, for things pip cannot install at all. The sweep - arithmetic is version-pinned -- ``editcap -T`` accepts 226 encapsulations on - Wireshark 4.6.9, 224 on 4.2.2 (CI's Ubuntu noble) -- so the count assertions - run only under exactly the measured version, with a separate - ``VersionIndependentInvariantTests`` class carrying what holds regardless. - One line of ``pcapkit/toolkit/pyshark.py`` itself changes as a result: the - ``IPMB_LINUX`` table entry becomes ``I2C_LINUX``, the canonical name :pr:`848` - gave value 209 -- the same member either way (:pr:`853`). + ``ENCAP_TYPE_TO_LINKTYPE`` (152 entries) and ``FILTER_NAME_TO_LINKTYPE`` (58) in + place inside ``pcapkit/toolkit/pyshark.py``, so the two tables :pr:`850` hand-built + stop being hand-maintained. It lives under ``util/`` rather than ``pcapkit/vendor/`` + because ``Vendor``'s base class is hardwired to an HTTP fetch and a generated + enum-class file, where this generator's source of truth is the local + ``tshark``/``editcap`` binaries. It guards against its worst failure mode: a naive + ``LinkType(dlt)`` cannot detect an unmapped DLT, because ``_missing_`` mints a + placeholder rather than raising, so the generator snapshots every known ``LinkType`` + value first and calls ``LinkType(dlt)`` only once a value is confirmed a member. The + real-``tshark`` sweep is gated on a new ``HAS_WIRESHARK`` flag, registered in + ``tests/_dependency_gates.py``'s ``NON_DISTRIBUTION_FLAGS`` next to ``HAS_PROC_FD``, + for things pip cannot install. The sweep arithmetic is version-pinned + (``editcap -T`` accepts 226 encapsulations on Wireshark 4.6.9, 224 on 4.2.2, CI's + Ubuntu noble), so count assertions run only under exactly the measured version, with + a separate ``VersionIndependentInvariantTests`` class for what holds regardless. One + line of ``pcapkit/toolkit/pyshark.py`` changes: the ``IPMB_LINUX`` entry becomes + ``I2C_LINUX``, the canonical name :pr:`848` gave value 209 -- the same member either + way (:pr:`853`). Fixed ~~~~~ -* the engine adapters, which were quietly wrong rather than loud. - The ``dpkt`` toolkit split TCP and IPv4 headers at their fixed struct size - instead of their real length, so option octets overwrote payload in the - sequence-indexed reassembly buffer; it also read an ``ipv6_frag.nh`` that - ``dpkt`` does not have, and passed fragment offsets unscaled (:issue:`351`, :issue:`370`, - :pr:`385`, :pr:`395`). ``scapy`` never loaded its layer registry, so that engine did not - dissect at all (:pr:`409`), and its IPv4 fragment offset reached reassembly - unscaled (:issue:`483`, :pr:`484`). The four IPv6 adapters disagreed about whether the - 8-octet Fragment header belongs to ``ihl``, ``header`` and ``tl``; per - :rfc:`8200` section 4.5 it belongs to none of them, and all four now agree - (:issue:`415`, :pr:`424`). -* ``pypcapfile``'s ``IP.src``/``.dst`` are dotted-decimal ASCII - text in a ``ctypes.c_char_p`` (confirmed against ``pypcapfile`` 0.12.0's own - ``ip.py``), not packed 4-byte values, so ``ipaddress.IPv4Address(ipv4.src)`` - raised ``AddressValueError`` and ``ipv4_header()``'s - ``struct.pack('...II', ipv4.src, ...)`` raised ``struct.error`` on every - call. A single ``_parse_ipv4_address()`` chokepoint replaces the four call - sites, casting with ``int(...)`` where ``struct.pack`` needs the packed form - and ``str(...)`` elsewhere, and also accepts a packed ``bytes`` or an - ``int`` directly. The same engine's ``IP.opt``/``.payload`` carry hex-ASCII - rather than raw bytes once decoding stops short of the transport layer; a - new ``_maybe_unhex()`` covers the three sites that read them. Not - ``breaking``: nothing on this path worked on 3.10/3.11 before, raising on - every call, so correct behaviour cannot regress a working caller (:issue:`743`). -* ``pypcapfile``'s ``_read_a_packet(layers=0)`` hexlifies a whole - frame into ASCII text and stops there, but ``_decode()`` fed that straight - into ``Ethernet(packet.packet, layers=1)``, whose ``__init__`` unpacks the - header with ``struct.unpack`` -- every field came out garbage without - raising. Fixed with ``binascii.unhexlify(packet.packet)`` ahead of the - decoder call, in preference to decoding eagerly at ``layers>0`` inside the - lazy generator, which would lose ``_decode()``'s per-frame - ``AttributeWarning`` fallback. Link-layer dispatch is unaffected, resolving - from the header's ``ll_type`` rather than from frame bytes. Landed together - with :issue:`743`: standalone, this change turns a non-crashing extraction into a - crashing one -- ``Extractor.run()`` catches only ``EOFError``, - ``StopIteration`` and ``KeyboardInterrupt``, so the ``AddressValueError`` - and ``struct.error`` :issue:`743` fixes on the same path escape to the caller until - both land (:issue:`746`). -* **a breaking change to** pyshark link-type resolution in - ``tcp_traceflow``: an unrecognised encapsulation or filter name now raises - ``MissingKeyError`` instead of substituting a plausible DLT. - ``pcapkit/toolkit/pyshark.py`` resolved a frame's link type from - ``packet.layers[0].layer_name``, a Wireshark display-filter name, with a - blanket ``Enum_LinkType.get(name.upper())`` fallback for anything not in its - small lookup table. One filter name serves several encapsulations, so that - fallback returned *wrong DLTs silently* -- ``null`` for a ``DLT_LOOP`` - capture, ``101`` (``RAW``) for ``rawip6``. Re-keyed onto - ``frame.encap_type``, which is distinct per encapsulation, and the fallback - is gone. The two lookup tables were measured rather than transcribed -- - Wireshark's source is not on the build host -- each entry round-tripped - through ``editcap -F pcap -T `` and read back with ``tshark``: - ``ENCAP_TYPE_TO_LINKTYPE`` grows to 152 entries with no DLT serving two keys, - and ``FILTER_NAME_TO_LINKTYPE`` grows from the two entries :pr:`838` added to 58, - every root filter name measured single-DLT across the 157 (of the 226 - ``editcap -T`` accepts) encapsulations this host's Wireshark can write. - ``pcapkit/toolkit/pyshark.py`` reaches 100% coverage under the new tests, - which fail 26 ways against the unfixed tree; live pyshark parsing itself - stays unexercised, since ``pyshark.FileCapture`` does not run at all on - Python 3.14 (:pr:`850`). +* the engine adapters, which were quietly wrong rather than loud. The ``dpkt`` + toolkit split TCP and IPv4 headers at their fixed struct size instead of their real + length, so option octets overwrote payload in the sequence-indexed reassembly buffer; + it also read an ``ipv6_frag.nh`` that ``dpkt`` does not have, and passed fragment + offsets unscaled (:issue:`351`, :issue:`370`, :pr:`385`, :pr:`395`). ``scapy`` never + loaded its layer registry, so that engine did not dissect at all (:pr:`409`), and its + IPv4 fragment offset reached reassembly unscaled (:issue:`483`, :pr:`484`). The four + IPv6 adapters disagreed about whether the 8-octet Fragment header belongs to ``ihl``, + ``header`` and ``tl``; per :rfc:`8200` section 4.5 it belongs to none, and all four + now agree (:issue:`415`, :pr:`424`). +* ``pypcapfile``'s ``IP.src``/``.dst`` are dotted-decimal ASCII text in a + ``ctypes.c_char_p`` (per ``pypcapfile`` 0.12.0's ``ip.py``), not packed 4-byte values, + so ``ipaddress.IPv4Address(ipv4.src)`` raised ``AddressValueError`` and + ``ipv4_header()``'s ``struct.pack('...II', ipv4.src, ...)`` raised ``struct.error`` on + every call. A single ``_parse_ipv4_address()`` chokepoint replaces the four call + sites, also accepting packed ``bytes`` or an ``int``. The same engine's + ``IP.opt``/``.payload`` carry hex-ASCII rather than raw bytes once decoding stops + short of the transport layer; a new ``_maybe_unhex()`` covers the three sites that + read them. Not ``breaking``: nothing on this path worked on 3.10/3.11, so correct + behaviour cannot regress a working caller (:issue:`743`). +* ``pypcapfile``'s ``_read_a_packet(layers=0)`` hexlifies a whole frame into ASCII + text and stops, but ``_decode()`` fed that into ``Ethernet(packet.packet, layers=1)``, + whose ``__init__`` unpacks the header with ``struct.unpack``, so every field came out + garbage without raising. Fixed with ``binascii.unhexlify(packet.packet)`` ahead of + the decoder call, in preference to decoding eagerly at ``layers>0`` inside the lazy + generator, which would lose ``_decode()``'s per-frame ``AttributeWarning`` fallback. + Link-layer dispatch resolves from the header's ``ll_type`` and is unaffected. Landed + with :issue:`743`: standalone, this turns a non-crashing extraction into a crashing + one, since ``Extractor.run()`` catches only ``EOFError``, ``StopIteration`` and + ``KeyboardInterrupt`` and the ``AddressValueError``/``struct.error`` :issue:`743` + fixes escape until both land (:issue:`746`). +* **a breaking change to** pyshark link-type resolution in ``tcp_traceflow``: an + unrecognised encapsulation or filter name now raises ``MissingKeyError`` instead of + substituting a plausible DLT. ``pcapkit/toolkit/pyshark.py`` resolved a frame's link + type from ``packet.layers[0].layer_name``, a Wireshark display-filter name, with a + blanket ``Enum_LinkType.get(name.upper())`` fallback. One filter name serves several + encapsulations, so that fallback returned *wrong DLTs silently* (``null`` for a + ``DLT_LOOP`` capture, ``101`` (``RAW``) for ``rawip6``). Re-keyed onto + ``frame.encap_type``, which is distinct per encapsulation, and the fallback is gone. + The two tables were measured rather than transcribed (Wireshark's source is not on the + build host): each entry round-tripped through ``editcap -F pcap -T `` and read + back with ``tshark``. ``ENCAP_TYPE_TO_LINKTYPE`` grows to 152 entries with no DLT + serving two keys, and ``FILTER_NAME_TO_LINKTYPE`` from :pr:`838`'s two entries to 58, + every root filter name single-DLT across the 157 (of 226) encapsulations this host's + Wireshark can write. ``pcapkit/toolkit/pyshark.py`` reaches 100% coverage under new + tests that fail 26 ways against the unfixed tree; live pyshark parsing stays + unexercised, since ``pyshark.FileCapture`` does not run on Python 3.14 (:pr:`850`). pcapkit.utilities ----------------- @@ -2886,48 +2094,45 @@ pcapkit.utilities Added ~~~~~ -* ``pcapkit.utilities.logging`` as a real interface: - ``get_logger()`` for per-module children, ``configure()`` to set level, - handler, stream, format or propagation at runtime, ``reset()`` to return to - library-neutral, and ``ensure_output()``. Seventeen modules now log under their - own ``__name__``, so a consumer can silence ``pcapkit.foundation.registry`` - while keeping ``pcapkit.foundation.extraction`` (:pr:`384`). +* ``pcapkit.utilities.logging`` as a real interface: ``get_logger()`` for per-module + children, ``configure()`` to set level, handler, stream, format or propagation at + runtime, ``reset()`` to return to library-neutral, and ``ensure_output()``. Seventeen + modules now log under their own ``__name__``, so a consumer can silence + ``pcapkit.foundation.registry`` while keeping ``pcapkit.foundation.extraction`` + (:pr:`384`). Changed ~~~~~~~ -* ``pcapkit`` no longer configures logging at import. It - installs a ``NullHandler`` and sets no level, so verbosity is inherited from - the application instead of being seized by whichever library was imported - second; the old stderr handler stays as the ``PCAPKIT_DEVMODE`` opt-in. Three - consequences worth knowing: the previous behaviour is - ``configure(logging.INFO, stream=sys.stderr)``; 38 registry and extractor - ``info`` calls became ``debug``, so those messages are invisible even at - ``INFO``; and the handler is no longer ``logger.handlers[0]``. ``verbose=`` - output stays on stdout and is not logging (:pr:`384`). -* each warning is reported once per channel, and ``pcapkit`` no - longer inserts a ``simplefilter('ignore', ...)`` at the front of the - process-global ``warnings.filters`` (:issue:`362`--:issue:`364`, :pr:`390`). The application's own - filter therefore wins now, which is the point of the change and also the sharp - edge in it: under ``-W error``, or pytest's ``filterwarnings = error``, a - pcapkit warning that used to be suppressed will raise. Suppress them - deliberately with - ``warnings.filterwarnings('ignore', category=BaseWarning)``. ``quiet=True`` - now means no record at any level and no longer sets ``sys.tracebacklimit``, - and the ``pcapkit.utilities.warnings.DEVMODE`` re-export is gone -- its - canonical home is ``pcapkit.utilities.logging``. +* ``pcapkit`` no longer configures logging at import. It installs a ``NullHandler`` + and sets no level, so verbosity is inherited from the application instead of being + seized by whichever library was imported second; the old stderr handler stays as the + ``PCAPKIT_DEVMODE`` opt-in. Three consequences: the previous behaviour is + ``configure(logging.INFO, stream=sys.stderr)``; 38 registry and extractor ``info`` + calls became ``debug``, so those messages are invisible even at ``INFO``; and the + handler is no longer ``logger.handlers[0]``. ``verbose=`` output stays on stdout and + is not logging (:pr:`384`). +* each warning is reported once per channel, and ``pcapkit`` no longer inserts a + ``simplefilter('ignore', ...)`` at the front of the process-global + ``warnings.filters`` (:issue:`362`--:issue:`364`, :pr:`390`). The application's own + filter therefore wins, which is the point and also the sharp edge: under + ``-W error``, or pytest's ``filterwarnings = error``, a pcapkit warning that used to + be suppressed will raise. Suppress them deliberately with + ``warnings.filterwarnings('ignore', category=BaseWarning)``. ``quiet=True`` now + means no record at any level and no longer sets ``sys.tracebacklimit``, and the + ``pcapkit.utilities.warnings.DEVMODE`` re-export is gone; its home is + ``pcapkit.utilities.logging``. Fixed ~~~~~ -* two more gaps the :issue:`514` keyword audit turned up, neither - previously covered by a test: ``StreamEOFError``'s docstring did not say - that ``@prepare`` always raises it with ``quiet=True`` -- the same - end-of-stream convention ``StructError`` follows via its own ``eof=True`` -- - so nothing pinned that silence against a future regression; and - ``register_extractor_engine``'s real keyword, ``name``, was not itself - under test, only its already-corrected docstring, so a future rename could - put the two out of step again exactly as quietly as before (:pr:`577`). +* two more gaps the :issue:`514` keyword audit turned up, neither covered by a test: + ``StreamEOFError``'s docstring did not say that ``@prepare`` always raises it with + ``quiet=True`` (the end-of-stream convention ``StructError`` follows via its own + ``eof=True``), so nothing pinned that silence against regression; and + ``register_extractor_engine``'s real keyword, ``name``, was not itself under test, + only its corrected docstring, so a rename could put the two out of step again + (:pr:`577`). pcapkit.vendor -------------- @@ -2935,141 +2140,113 @@ pcapkit.vendor Fixed ~~~~~ -* thirteen ``re.sub`` sites under ``pcapkit/vendor/`` passed - ``re.MULTILINE`` as the fourth *positional* argument, which is ``count``, - not ``flags`` -- capping substitution at 8 matches and raising - ``DeprecationWarning`` on Python 3.13+, a future ``TypeError``. Found by an - AST sweep rather than by grep, since 8 of the 13 put the flag on a - continuation line. Fixed by moving the flag to ``flags=`` at all 13 sites - (``default.py`` plus one site each in ``hip/eddsa_curve.py``, - ``http/frame.py``, ``http/method.py``, ``http/status_code.py``, - ``ipv4/router_alert.py``, ``ipv6/option.py``, ``ipv6/router_alert.py``, - ``ipv6/tagger_id.py``, ``reg/ethertype.py``, ``tcp/flags.py``, - ``tcp/mp_tcp_option.py`` and ``tcp/option.py``). Behaviour preserving: none - of the 13 sites' shared pattern (``r'\r*\n'``) carries a - MULTILINE-sensitive anchor, so no ``const/`` regeneration was needed (:issue:`796`). -* ``main`` was red: 12 tests across three files, all - ``RecursionError`` (``tests/const/test_const_enum_lookup.py``; two - ``test_const_enum_no_mint.py`` methods, two registries each; 7 of the 12 in +* thirteen ``re.sub`` sites under ``pcapkit/vendor/`` passed ``re.MULTILINE`` as the + fourth *positional* argument, which is ``count``, not ``flags``: substitution capped + at 8 matches, and a ``DeprecationWarning`` on Python 3.13+ (a future ``TypeError``). + Found by an AST sweep rather than grep, since 8 of the 13 put the flag on a + continuation line. The flag moves to ``flags=`` at all 13 sites (``default.py`` plus + one each in ``hip/eddsa_curve.py``, ``http/frame.py``, ``http/method.py``, + ``http/status_code.py``, ``ipv4/router_alert.py``, ``ipv6/option.py``, + ``ipv6/router_alert.py``, ``ipv6/tagger_id.py``, ``reg/ethertype.py``, + ``tcp/flags.py``, ``tcp/mp_tcp_option.py`` and ``tcp/option.py``). Behaviour + preserving: the shared pattern (``r'\r*\n'``) has no MULTILINE-sensitive anchor, so + no ``const/`` regeneration was needed (:issue:`796`). +* ``main`` was red (:issue:`866`): 12 tests across three files, all ``RecursionError`` + (``tests/const/test_const_enum_lookup.py``; two ``test_const_enum_no_mint.py`` + methods, two registries each; 7 of the 12 in ``tests/protocols/misc/test_pcapng_unit.py``). - ``pcapkit/vendor/pcapng/record_type.py`` and ``secrets_type.py`` rendered a - two-line ``_missing_`` body whose first line did not return -- - ``cls._unregistered_member(value, 'Unassigned')`` followed unconditionally by - ``return cls(value)``. Harmless while that first line was - ``extend_enum(...)``, which registers the member so the following - ``cls(value)`` found it; :pr:`861` (above) replaced it with - ``_unregistered_member``, which deliberately does not register, so - ``cls(value)`` missed again and re-entered ``_missing_``. :pr:`861` only - half-corrected the two files: it fixed the *generated* modules and never - touched the two *crawlers*, and even in the generated files left the old - ``return cls(value)`` in place as unreachable dead code after the new return - -- which is why :pr:`861`'s own CI stayed green until a live regeneration from the - stale crawlers reintroduced the defect for real; :pr:`861`'s cross-review had - verified byte-identical regeneration on a 5-of-82 sample that did not happen - to include these two. Both crawlers now collapse to the single returning line - the other 101 registries use, and the two const files are regenerated from - them -- ``record_type.py`` fetches the network for its own ``LINK`` constant, - so that regeneration is not reproducible without connectivity, though it came - back byte-identical; ``secrets_type.py`` is offline. The new guard sits at - the crawler-emission layer rather than the behavioural one - ``test_const_enum_lookup.py`` already swept and that is what caught this: a - hard-coded ``_missing_`` body must end in a return, and no emitted line may - re-enter ``_missing_`` for the same value -- deliberately narrower than - "every emitted line must return", since 53 crawlers legitimately emit 109 - non-returning lines in the shape ``Vendor.process`` itself uses. The guard - sees only 35 of 96 crawlers; the other 61 (53 ``append``-built, 8 returning - no ``(enum, miss)`` pair) stay invisible to it and are covered behaviourally - instead. Verified against the four reverted source files: 4 of 6 new tests - fail as methods; the other two pass either way, by design, since :issue:`866`'s own - body still ended on a return. Closes :issue:`866` (:pr:`867`). -* ``python -m pcapkit.vendor `` could not fail: ``run()`` - swallowed every crawler exception as a filterable ``VendorRuntimeWarning``, - and ``main()`` always returned ``0``, so a broken crawler regenerating - nothing looked like success both to a human and to the cron job. Per the - owner's ruling (*"only discard changes made by a non-zero sub-vendor... for - cron jobs, at most warn about failed sub-vendors, but never fail the entire - job"*), this lands in three pieces. **Exit code** - (``pcapkit/vendor/__main__.py``): ``run()`` now returns whether its target - succeeded; ``main()`` attempts every target regardless of earlier failures - and returns ``1`` if any raised, with the failing target's qualified name and - exception ``repr`` written to stderr unconditionally via ``print()`` rather - than through the filterable warning. **Per-target snapshot and restore**: a - new ``_snapshot_and_restore`` context manager backs up a target's const file - with ``tempfile.mkstemp`` before it runs -- resolving the destination through - ``Vendor._dest_path``, now a ``classmethod`` (``pcapkit/vendor/default.py``) - so it can be called before ``Vendor.__init__`` ever fetches, renders or - writes -- and on any exception restores the backup via ``os.replace`` before - re-raising, or discards it quietly on success. A target whose destination - cannot be resolved (a crawler outside the real tree) runs unprotected rather - than failing the snapshot step, since ``Vendor.__init__`` will hit the - identical resolution failure on its own; a destination that does not exist - yet is left alone, since there is nothing for a failure to discard. - **Workflow** (``.github/workflows/cron-vendor.yml``): the "Update Vendor" - step no longer aborts under ``bash -e`` when ``pcapkit-vendor`` exits - non-zero -- the exit code is captured explicitly around that one command, so - targets that did succeed still get ``isort``ed, diffed, committed and pushed, - while a "Warning" block naming the failed target(s) is appended to - ``$GITHUB_STEP_SUMMARY`` and a ``::warning::`` annotation is emitted per - failure; the step's own exit code is always ``0`` so later steps still run. A - per-target ``git checkout -- `` restore step was considered and - dropped: with the snapshot/restore in place, a failed target's own const file - is never touched at all, so there is nothing for the workflow itself to - discard. ``tests/vendor/test_vendor_exit_code_unit.py`` (5 methods, through - the real entrypoints with stub ``Vendor`` subclasses) and - ``tests/vendor/test_vendor_snapshot_restore_unit.py`` (3: a successful run - replaces the previous content, a failure leaves the previous file - byte-for-byte intact, and a read-only destination is left exactly as it was) - -- no network calls. Closes :issue:`872` (:pr:`873`). -* three follow-ups from :pr:`873`'s own review, none of which changed - the behaviour that merged; one invariant was unpinned and two pieces of - prose/lint were inaccurate. **1.** ``except BaseException:`` in - ``_snapshot_and_restore`` is now pinned: a new - ``test_a_keyboard_interrupt_still_restores_and_propagates`` asserts all three - halves of the invariant -- content restored byte-for-byte, no ``.bak`` file - surviving, and the ``KeyboardInterrupt`` itself still propagating rather than - being converted into a ``False`` return. It discriminates: the real code - passes clean, while mutating the clause to ``except Exception:`` fails on the - restored content with the predicted ``.bak`` file left behind. **2.** A false - docstring clause is corrected: it claimed a crawler whose ``_dest_path`` - cannot be resolved "goes on to fail loudly anyway once ``Vendor.__init__`` - calls the same ``_dest_path`` itself", which holds for one of the function's - two triggers and not the other -- an instance-method-only override raises - ``TypeError`` on the *class* call ``_snapshot_and_restore`` makes (no - ``self`` to bind), but the *instance* call inside ``__init__`` binds ``self`` - correctly and resolves fine, so that target runs to completion unprotected - and **succeeds silently**: ``run()`` returns ``True``, the file is replaced, - and nothing warns that no snapshot was ever taken. Measured with a - constructed instance-method-only stub. **3.** The - ``# pylint: disable=no-else-raise`` pragma is **kept** -- removing it was - going to be a third fix, on the mistaken belief that R1720 never applies to a - ``try``/``except``/``else``; that belief rested on a message-control artefact - (``--disable=all --enable=no-else-raise`` does not re-enable the check the - way ``--enable=R`` does), and under the ``Makefile``'s own flags -- what CI - actually runs -- pylint 4.0.8 does flag it. The ``else:`` is deliberate: it - is what makes the backup cleanup unreachable from the ``except``, and - deleting the ``raise`` that makes it so would be a mutation the suite must - catch. Deliberately left out of scope: an untested ``os.close(fd)`` leak - (harmless, since the path still exists and ``copy2`` reopens it) and the - semantically-identical unindented-statement form of the same ``else:``, which - no test can distinguish from the current one. Closes :issue:`875` (:pr:`876`). + ``pcapkit/vendor/pcapng/record_type.py`` and ``secrets_type.py`` rendered a two-line + ``_missing_`` body whose first line did not return: + ``cls._unregistered_member(value, 'Unassigned')`` followed by ``return cls(value)``. + That was harmless while the first line was ``extend_enum(...)``, which registers the + member so ``cls(value)`` found it; :pr:`861` (above) replaced it with + ``_unregistered_member``, which deliberately does not register, so ``cls(value)`` + missed and re-entered ``_missing_``. :pr:`861` half-corrected the two files: it + fixed the *generated* modules (leaving the old ``return cls(value)`` as dead code) + and never touched the two *crawlers*, which is why :pr:`861`'s own CI stayed green + until a live regeneration from the stale crawlers reintroduced the defect; + :pr:`861`'s cross-review had verified byte-identical regeneration on a 5-of-82 + sample that missed these two. Both crawlers now collapse to the single returning + line the other 101 registries use, and the two const files are regenerated + (``record_type.py`` fetches the network for its own ``LINK`` constant, so that + regeneration needs connectivity, though it came back byte-identical; + ``secrets_type.py`` is offline). The new guard sits at the crawler-emission layer + rather than the behavioural one ``test_const_enum_lookup.py`` already swept: a + hard-coded ``_missing_`` body must end in a return, and no emitted line may re-enter + ``_missing_`` for the same value -- deliberately narrower than "every emitted line + must return", since 53 crawlers legitimately emit 109 non-returning lines in the + shape ``Vendor.process`` uses. The guard sees only 35 of 96 crawlers; the other 61 + are covered behaviourally. Closes :issue:`866` (:pr:`867`). +* ``python -m pcapkit.vendor `` could not fail: ``run()`` swallowed + every crawler exception as a filterable ``VendorRuntimeWarning`` and + ``main()`` always returned ``0``, so a broken crawler regenerating nothing + looked like success to a human and to the cron job. Per the owner's ruling + (*"only discard changes made by a non-zero sub-vendor... for cron jobs, at + most warn about failed sub-vendors, but never fail the entire job"*), three + pieces land. **Exit code** (``pcapkit/vendor/__main__.py``): ``run()`` + returns whether its target succeeded; ``main()`` attempts every target and + returns ``1`` if any raised, writing the failing target's qualified name and + exception ``repr`` to stderr unconditionally via ``print()`` rather than the + filterable warning. **Per-target snapshot and restore**: a new + ``_snapshot_and_restore`` context manager backs up a target's const file with + ``tempfile.mkstemp`` before it runs (resolving the destination through + ``Vendor._dest_path``, now a ``classmethod`` in ``pcapkit/vendor/default.py`` + so it can be called before ``Vendor.__init__`` fetches, renders or writes), + restores the backup via ``os.replace`` on any exception before re-raising, + and discards it on success. A target whose destination cannot be resolved (a + crawler outside the real tree) runs unprotected rather than failing the + snapshot step, since ``Vendor.__init__`` hits the same resolution failure + itself; a destination that does not exist yet is left alone. **Workflow** + (``.github/workflows/cron-vendor.yml``): the "Update Vendor" step no longer + aborts under ``bash -e`` when ``pcapkit-vendor`` exits non-zero. The exit + code is captured around that one command, so targets that succeeded are still + ``isort``ed, diffed, committed and pushed, while a "Warning" block naming the + failed target(s) is appended to ``$GITHUB_STEP_SUMMARY``, a ``::warning::`` + annotation is emitted per failure, and the step's own exit code is always + ``0``. A per-target ``git checkout -- `` restore step was dropped: with + snapshot/restore, a failed target's const file is never touched. + ``tests/vendor/test_vendor_exit_code_unit.py`` (5 methods, through the real + entrypoints with stub ``Vendor`` subclasses) and + ``tests/vendor/test_vendor_snapshot_restore_unit.py`` (3: success replaces + the previous content, failure leaves it byte-for-byte intact, a read-only + destination is left as it was) make no network calls. Closes :issue:`872` + (:pr:`873`). +* three follow-ups from :pr:`873`'s review, none changing merged behaviour: one + unpinned invariant and two inaccurate pieces of prose/lint. **1.** + ``except BaseException:`` in ``_snapshot_and_restore`` is pinned: + ``test_a_keyboard_interrupt_still_restores_and_propagates`` asserts content restored + byte-for-byte, no ``.bak`` surviving, and the ``KeyboardInterrupt`` still + propagating rather than becoming a ``False`` return; mutating the clause to + ``except Exception:`` fails it. **2.** A false docstring clause is corrected: it + said a crawler whose ``_dest_path`` cannot be resolved "goes on to fail loudly + anyway once ``Vendor.__init__`` calls the same ``_dest_path`` itself", which holds + for one of the function's two triggers and not the other. An instance-method-only + override raises ``TypeError`` on the *class* call ``_snapshot_and_restore`` makes + (no ``self`` to bind), but the *instance* call inside ``__init__`` resolves fine, so + that target runs unprotected and **succeeds silently**: ``run()`` returns ``True``, + the file is replaced, and nothing warns that no snapshot was taken. **3.** The + ``# pylint: disable=no-else-raise`` pragma is **kept**. Removing it rested on the + belief that R1720 never applies to ``try``/``except``/``else``, a message-control + artefact (``--disable=all --enable=no-else-raise`` does not re-enable the check the + way ``--enable=R`` does); under the ``Makefile``'s flags, which CI runs, pylint + 4.0.8 flags it. The ``else:`` is deliberate: it makes the backup cleanup unreachable + from the ``except``. Left out of scope: an untested ``os.close(fd)`` leak (harmless, + the path still exists and ``copy2`` reopens it) and the semantically identical + unindented form of the same ``else:``. Closes :issue:`875` (:pr:`876`). * ``TLSKeyLabel`` tracked the Mozilla NSS key-log wiki, but - ``draft-ietf-opsawg-pcapng-06`` S4.7 now points at - ``[I-D.ietf-tls-keylogfile]`` (draft-05), published as :rfc:`9850`, "The - SSLKEYLOGFILE Format for TLS", whose S4.2 creates the IANA "TLS SSLKEYLOGFILE - Labels" registry. Against that registry, ``TLSKeyLabel`` was missing - ``ECH_SECRET`` and ``ECH_CONFIG`` (:rfc:`9850` SS2.3/4.2), so - ``TLSKeyLabel('ECH_SECRET')`` raised ``ValueError`` on an - otherwise-legitimate Decryption Secrets Block; both are added, ``ECH_SECRET`` - keeping the file's ``# nosec B105`` bandit convention for ``*_SECRET`` - members. ``RSA`` has no row in :rfc:`9850`'s registry -- it is an NSS-only - label the NSS wiki records as removed in NSS 3.34 -- and is kept rather than - deleted, since removing a public member is a breaking change, with a new - docstring comment marking it the historical exception. - ``pcapkit/vendor/pcapng/secrets_type.py``'s stale ``# NSS Key Log Format`` - comment is repointed at RFC 9850 to match, and ``TLSKeyLabel``'s own - docstring now cites it. A new test pins the member set against RFC 9850 plus - the documented ``RSA`` exception, and asserts ``ECH_SECRET``, ``ECH_CONFIG`` - and ``RSA`` all resolve. Closes :issue:`882` (:pr:`883`). + ``draft-ietf-opsawg-pcapng-06`` S4.7 now points at ``[I-D.ietf-tls-keylogfile]`` + (draft-05), published as :rfc:`9850`, "The SSLKEYLOGFILE Format for TLS", whose S4.2 + creates the IANA "TLS SSLKEYLOGFILE Labels" registry. Against it, ``TLSKeyLabel`` + was missing ``ECH_SECRET`` and ``ECH_CONFIG`` (:rfc:`9850` SS2.3/4.2), so + ``TLSKeyLabel('ECH_SECRET')`` raised ``ValueError`` on a legitimate Decryption + Secrets Block; both are added, ``ECH_SECRET`` keeping the file's ``# nosec B105`` + bandit convention for ``*_SECRET`` members. ``RSA`` has no row in :rfc:`9850`'s + registry (it is an NSS-only label the NSS wiki records as removed in NSS 3.34) and + is kept, since removing a public member is breaking, with a docstring comment + marking it the historical exception. ``pcapkit/vendor/pcapng/secrets_type.py``'s + stale ``# NSS Key Log Format`` comment is repointed at RFC 9850, and + ``TLSKeyLabel``'s docstring cites it. A new test pins the member set against RFC + 9850 plus the ``RSA`` exception. Closes :issue:`882` (:pr:`883`). Project infrastructure ---------------------- @@ -3077,55 +2254,47 @@ Project infrastructure Added ~~~~~ -* an end-to-end test tier (:pr:`376`), sample-capture generators so a - fresh clone can rebuild every fixture (:pr:`340`), a Dockerised engine benchmark - covering every supported Python version (:pr:`410`), and registry round-trip - coverage that records the entries which cannot close the cycle rather than - skipping them (:pr:`440`, :pr:`504`). -* coverage for the untested half of the :issue:`431` accommodation: a TCP - or IPv4 option whose declared length asks for more data than the capture - actually holds, pinning that it still parses, with the short read zero-padded - rather than rejected. The one test :issue:`431` left behind only covers an option - area with no data behind it at all; a candidate fix for :issue:`554` turned the - untested half into an unwrapped ``FieldValueError`` while the rest of the - suite stayed green (:pr:`571`, :issue:`572`). -* ``CITATION.cff``, citation metadata in Citation File Format - 1.2.0, which GitHub renders as the repository's "Cite this repository" button - and which citation managers and dependency inventories read directly. It is - the machine-readable half of the attribution BSD-3-Clause already asks for, so - credit carries into a paper or a bill of materials rather than depending on a - reader opening ``LICENSE``. Validated with ``cffconvert --validate`` and - against the published 1.2.0 schema; ``doi`` and ``orcid`` are omitted rather - than invented, since neither exists for this project today and both are - checked formats, so a wrong value would still validate. The licence itself is - deliberately unchanged -- still BSD-3-Clause, no ``NOTICE`` file, no change to - its terms. Alongside it the copyright line moves from ``2018-2023`` to - ``2018-2026`` -- ``LICENSE`` was its only occurrence in the tree, since - ``docs/source/conf.py`` already derives its own from the current year -- and a - stray ``s`` after the closing ``DAMAGE.`` of the licence text, present since - the Mozilla-to-BSD relicence, is removed, so the wording now matches canonical - BSD-3-Clause exactly (:pr:`615`). -* ``examples/generators/endian.py``, and the byte-order tests that - read what it writes. There was no big-endian ``.pcap`` in the repository at all, - which is why :issue:`605` survived its own code review: the one-character fix leaves the - corrected path exactly as untested as the broken one. The generator writes three - captures -- ``big_endian.pcap`` (magic ``a1 b2 c3 d4``), - ``big_endian_nanosecond.pcap`` (``a1 b2 3c 4d``, the first fixture to take that - branch of the magic-number table) and ``little_endian.pcap`` (``d4 c3 b2 a1``) -- - carrying the *same three records* in each container, so the tests can assert - that the byte order makes no difference to what is read out rather than only - that the big-endian file matches numbers written down in a test. Frame 3 is - captured short, 1200 octets on the wire cut to a 96-octet ``snaplen``, so - ``incl_len`` and ``orig_len`` differ and cannot both be satisfied by one - byte-swapped value. ``test_frame_endian_runtime.py`` drives all three through - ``extract()`` and walks each file's record chain with ``struct`` to derive its - own expectations; a unit-tier case in ``test_header_frame_unit.py`` builds a - two-record big-endian capture in memory instead, so the regression is also - caught by the fixture-free selection CI runs on every push. All four fail on the - unfixed tree -- the three fixture-backed ones by that ``ValueError``, the - in-memory one by ``AssertionError: 3106905 != 1500000000`` -- while the - little-endian twin passes on both trees, which is what shows the records - themselves are not the variable (:issue:`605`). +* an end-to-end test tier (:pr:`376`), sample-capture generators so a fresh clone can + rebuild every fixture (:pr:`340`), a Dockerised engine benchmark covering every + supported Python version (:pr:`410`), and registry round-trip coverage that records + the entries which cannot close the cycle rather than skipping them (:pr:`440`, + :pr:`504`). +* coverage for the untested half of the :issue:`431` accommodation: a TCP or IPv4 + option whose declared length asks for more data than the capture holds, pinning that + it still parses with the short read zero-padded rather than rejected. The one test + :issue:`431` left covers only an option area with no data behind it; a candidate fix + for :issue:`554` turned the untested half into an unwrapped ``FieldValueError`` while + the rest of the suite stayed green (:pr:`571`, :issue:`572`). +* ``CITATION.cff``, citation metadata in Citation File Format 1.2.0, which GitHub + renders as the "Cite this repository" button and which citation managers and + dependency inventories read directly. It is the machine-readable half of the + attribution BSD-3-Clause already asks for. Validated with ``cffconvert --validate`` + and against the published 1.2.0 schema; ``doi`` and ``orcid`` are omitted rather than + invented, since neither exists for this project and both are checked formats, so a + wrong value would still validate. The licence is unchanged (BSD-3-Clause, no + ``NOTICE``). Alongside it the copyright line moves from ``2018-2023`` to ``2018-2026`` + (``LICENSE`` was its only occurrence; ``docs/source/conf.py`` derives its own from the + current year), and a stray ``s`` after the closing ``DAMAGE.`` of the licence text, + present since the Mozilla-to-BSD relicence, is removed so the wording matches + canonical BSD-3-Clause (:pr:`615`). +* ``examples/generators/endian.py``, and the byte-order tests that read what it + writes. There was no big-endian ``.pcap`` in the repository, which is why + :issue:`605` survived its own code review: the one-character fix left the corrected + path as untested as the broken one. The generator writes ``big_endian.pcap`` (magic + ``a1 b2 c3 d4``), ``big_endian_nanosecond.pcap`` (``a1 b2 3c 4d``, the first fixture + to take that branch of the magic-number table) and ``little_endian.pcap`` + (``d4 c3 b2 a1``), carrying the *same three records* in each, so tests can assert + that byte order makes no difference to what is read rather than only matching + numbers written in a test. Frame 3 is captured short (1200 octets on the wire, + 96-octet ``snaplen``), so ``incl_len`` and ``orig_len`` differ and cannot both be + satisfied by one byte-swapped value. ``test_frame_endian_runtime.py`` drives all + three through ``extract()`` and derives expectations by walking each file's record + chain with ``struct``; a unit-tier case in ``test_header_frame_unit.py`` builds a + two-record big-endian capture in memory, so the fixture-free selection CI runs on + every push catches it too. All four fail on the unfixed tree (the three + fixture-backed by that ``ValueError``, the in-memory one by + ``AssertionError: 3106905 != 1500000000``) while the little-endian twin passes on + both, showing the records themselves are not the variable (:issue:`605`). Changed ~~~~~~~ @@ -3133,453 +2302,322 @@ Changed * renames with no compatibility alias left behind: ``examples/sample`` and ``examples/samples`` -- one letter apart, holding different things -- are now ``examples/captures`` and ``examples/generators``. -* the README is a landing page now, and Markdown rather than - reStructuredText. ``README.rst`` (424 lines) became ``README.md`` (102), - keeping what a reader arriving from PyPI or a search result actually needs -- - what the library is, why it exists rather than Scapy or DPKT, how to install - it, a worked example, and where the documentation lives -- and dropping the - technical detail the documentation already carried. **Module Structure**, - **Engine Comparison**, **Engine support by Python version**, **Test - Environment**, **Test Results** and **Installation Notes** were each already - duplicated in ``docs/source/index.rst``, in a fuller form, so they are linked - rather than restated. Two blocks existed nowhere else and moved rather than - going: **Testing** is now ``docs/source/testing.rst``, registered in the index - toctree, and the ``pipenv`` and ``make setup`` local development block joined - the Installation section of ``docs/source/index.rst``. Requested by the project - owner, and a deliberate exception to the convention that documentation here is - reStructuredText -- for the README only, since it is the one documentation file - whose renderers are GitHub and PyPI rather than Sphinx. Accordingly - ``setup.py`` reads ``README.md`` and declares its content type as - ``text/markdown``, and ``MANIFEST.in`` gains an ``include README.md`` line - because ``global-include *.rst`` no longer matches the file. That line is - belt-and-braces rather than load-bearing, contrary to what this entry claimed - when it was written: setuptools' own ``sdist`` command ships whichever of - ``README``, ``README.rst``, ``README.txt`` and ``README.md`` exists, before - ``MANIFEST.in`` is read at all, so deleting the line leaves the sdist's file - listing byte-identical -- 861 entries either way, with an empty ``diff`` -- and - the result still installs. ``setup.py`` does read the file unguarded, but it - reads it from wherever ``setup.py`` is executing, which when ``pip`` installs an - sdist is the unpacked sdist rather than a checkout, so that read cannot be made - to fail by dropping the line either. Of the ``include`` lines in that file only - ``CHANGELOG.md`` is load-bearing, which the :pr:`631` entry below measured - independently. Verified with ``twine check --strict`` against a built sdist and - wheel, both of which pass. The rename's own references moved with it, since a - change that renames a file owns the references to it: - ``examples/benchmark/Dockerfile`` copies ``README.md`` -- a literal ``COPY`` of - the old name would have failed the layer outright and taken ``make bench``, - ``make bench-quick`` and ``run.sh`` with it -- and the benchmark harness prose - that named the root README as the destination of its generated tables now names - ``docs/source/index.rst``, which is where those tables went. That covers the - ``Makefile`` comment, ``report.py``, ``test_harness.py``, ``run.sh`` and the - suite's own README, including the one place whose stated reason had inverted: the - emitted markup is kept parseable by plain docutils, which is now a conservative - choice rather than a hard requirement, because the page it lands on is rendered - by Sphinx. ``examples/benchmark/benchmark.py`` still says ``README.rst`` and is - left alone, because it means the benchmark suite's own README in the same - directory, not the project's (:pr:`619`). -* ``CODE_OF_CONDUCT.md`` moves from Contributor Covenant 1.4 to - Contributor Covenant 3.0, at the maintainer's request. The text is the canonical - 3.0 Markdown fetched from +* the README is a landing page now, and Markdown rather than reStructuredText. + ``README.rst`` (424 lines) became ``README.md`` (102), keeping what a reader arriving + from PyPI or a search result needs (what the library is, why it exists rather than + Scapy or DPKT, how to install it, a worked example, where the documentation lives). + **Module Structure**, **Engine Comparison**, **Engine support by Python version**, + **Test Environment**, **Test Results** and **Installation Notes** were already + duplicated in fuller form in ``docs/source/index.rst``, so they are linked rather than + restated; the two blocks that existed nowhere else moved: **Testing** is now + ``docs/source/testing.rst``, in the index toctree, and the ``pipenv`` and + ``make setup`` block joined the Installation section of ``docs/source/index.rst``. Requested + by the project owner, and a deliberate exception to the reStructuredText convention, + for the README only, since its renderers are GitHub and PyPI rather than Sphinx. + ``setup.py`` reads ``README.md`` and declares ``text/markdown``, and ``MANIFEST.in`` + gains ``include README.md`` because ``global-include *.rst`` no longer matches. That + line is belt-and-braces rather than load-bearing, contrary to what this entry claimed + when written: setuptools' ``sdist`` ships whichever of ``README``, ``README.rst``, + ``README.txt`` and ``README.md`` exists before ``MANIFEST.in`` is read, so deleting the + line leaves the sdist listing byte-identical, and ``setup.py`` reads the file from + where it executes, which under ``pip`` is the unpacked sdist. Of the ``include`` lines + only ``CHANGELOG.md`` is load-bearing, measured in the :pr:`631` entry below. References + moved with the rename: ``examples/benchmark/Dockerfile`` copies ``README.md`` (a + literal ``COPY`` of the old name would have failed the layer and taken ``make bench`` + with it), and benchmark prose naming the root README as the destination of generated + tables now names ``docs/source/index.rst``. ``examples/benchmark/benchmark.py`` still + says ``README.rst`` on purpose, meaning the benchmark suite's own README (:pr:`619`). +* ``CODE_OF_CONDUCT.md`` moves from Contributor Covenant 1.4 to 3.0, at the + maintainer's request. The text is the canonical 3.0 Markdown fetched from https://www.contributor-covenant.org/version/3/0/code_of_conduct/code_of_conduct.md - rather than a transcription, so the pledge, the encouraged and restricted - behaviours and the scope are unaltered. Three things needed deciding rather than - copying. 3.0 ships two ``[NOTE`` placeholders an adopter must fill: the reporting - channel, which now names ``jarryshaw@icloud.com`` -- the same contact 1.4 carried - and the one ``SECURITY.md`` already points at as its email fallback -- plus - GitHub's report-abuse form for the case a single-maintainer project cannot - otherwise cover, a report about the maintainer; and the enforcement section, whose - placeholder is an instruction to the adopter and is removed. 3.0 then assigns - enforcement throughout to plural "Community Moderators" (and once, inconsistently, - to "Community Managers"), which this repository does not have, so all eight - occurrences become the singular maintainer. The four-rung ladder -- Warning, - Temporarily Limited Activities, Temporary Suspension, Permanent Ban -- is offered - as a suggestion and is **kept**, because each rung maps onto a lever one person - actually holds on GitHub: a private message, a locked thread, an interaction limit - or block, a permanent block. Finally, 3.0 is licensed CC BY-SA 4.0 where 1.4's - attribution paragraph carried no licence notice at all, so the attribution now - names version 3.0, links the permanent ``version/3/0/`` URL, carries the CC BY-SA - 4.0 notice and link, indicates that changes were made as BY requires, and says - explicitly that the share-alike term covers this document only -- the code remains - BSD-3-Clause and ``LICENSE`` is untouched. Rendering was checked against GitHub's - own Markdown API rather than assumed: the ladder comes back as four list items each - nesting three, which is what :pr:`613` had to repair in the 1.4 file when a stray list - marker collapsed the whole document into one nested item (:pr:`624`). -* reconciled which private (``_xxx``) attributes and methods - the Sphinx build documents, replacing an ad hoc mix with one stated rule: - document the contract, hide the recipe. A member every subclass must - implement, or one whose shape a caller genuinely depends on, stays - documented even though its name starts with an underscore; a private - helper that exists only to keep one method short does not. Seven - module-level directives naming pure implementation detail were dropped -- - ``esp._resolve``, ``esp._CRYPTO``, ``ngap._convert``, ``ngap._revert``, - ``ngap._PYCRATE``, ``ngap._PDU_LOCK`` and ``pypcapfile._NamedStream`` - (whose nested ``name``/``read`` members go with it, being reachable only - through a private class) -- while 39 ``autoattribute`` directives were - added for class-private state that *is* contract: ``Extractor._flag_f``, - the ``PyPCAP`` and ``PCAP_CT`` engines' own ``_backend`` -- the only two of - the six third-party engines that have one -- ``TraceFlow``'s internal - fields, and ``FieldBase``/``Field`` internals among them; the built-in - ``PCAP`` and ``PCAPNG`` engines gained ``_gbhdr``, ``_vinfo`` and - ``_nnsec`` for the former and ``_ctx``/``_ctx_list`` for the latter, and - no ``_backend`` at all. ``_dlink`` is not built-in-only: ``PCAP`` documents - it alongside the three third-party engines that share it, ``PyPCAP``, - ``PCAP_CT`` and ``PyPCAPFile``. The 128 - runtime definitions of ``_missing_`` -- 121 under ``pcapkit.const``, the - other 7 inline in ``pcapkit.protocols`` -- gained one unified write-up in - place of a directive per class: a new "Unrecognised Values" section in - ``docs/source/pcapkit/const/index.rst``, cross-referenced from - ``registry.rst``, since ``conf.py`` already excludes ``_missing_`` from - every ``autoclass`` via ``exclude-members`` in ``autodoc_default_options`` - -- there is no ``automodule`` directive anywhere under ``docs/source/``. - ``CONTRIBUTING.md`` gained the - rule itself as a named section, so the next directive gets judged against a - written test rather than against precedent. Verified with - ``sphinx-build -b html`` under ``PCAPKIT_SPHINX=1``: 53 warnings on ``main`` - before this change, 54 after, the one addition being a pre-existing bare - ``Type`` cross-reference ambiguity newly rendered by ``TraceFlow._foutio``'s - new directive rather than a defect this change introduced -- :issue:`709` tracked - it. No line under ``pcapkit/`` changed (:issue:`684`). -* ``examples/captures/out.json``, ``out.plist``, ``out.txt`` - and ``pcapng.txt`` are no longer tracked in git; they are build output, not - fixtures, and a tracked rendering with no reader goes stale silently every - time the code that produces it changes. ``pcapng.txt`` is exactly that: it - still recorded ``packet -> NIL`` for the four Enhanced Packet Blocks of - ``dhcp.pcapng`` long after :pr:`683` gave those blocks their captured octets - back, and nothing had regenerated it. Rather than regenerate it once more - and leave the same drift free to recur, the four files are removed from - the index and folded into ``examples/captures/``'s existing blanket - ``.gitignore`` rule; ``in.pcap`` and ``dhcp.pcapng``, the genuine inputs, - stay tracked. ``examples/legacy_smoke/Makefile`` and its ``README.rst`` are - reworded to describe regenerating these reports via ``make fixtures`` - rather than implying they ship committed, and a new - ``tests/project/test_capture_tracking.py`` (5 tests, 8 subtests) pins the - invariant going forward; 2 of the 5 tests (4 of the 12 subtests) fail - against the pre-change tree with the four reports restored to the index, - the other 3 passing on both trees by construction (:issue:`685`). -* ``tests/project/test_isort_clean.py`` gated on ``isort`` with - a ``try``/``except ImportError`` inside ``setUpClass``, which - ``tests/_dependency_gates.py`` cannot see -- ``_gates_of()`` walks a - decorator list and never a function body -- and ``isort`` was in no - ``pyproject.toml`` extra, so the file reported ``OK (skipped=1)`` on every CI - leg and the dependency-gate guard could not tell. It now gates on a - module-level ``HAS_ISORT`` and a class-level ``@unittest.skipUnless`` so - ``gated_scopes()`` counts the gate, ``MODULE_PROVIDERS`` maps ``isort`` so - the flag resolves instead of raising, and ``isort`` joins the ``test`` extra - rather than being excluded -- ``lint.yml`` installs none and its own header - says so, so unlike ``mypy``'s gate there is no existing check to defer to. - No workflow changed, every pytest job's install line already building on - ``.[test]``; ``gated_scopes()`` goes 250 to 251 and - ``dependency_gate_gaps()`` holds at 18. The third instance of the hazard - :issue:`745` exists to stop (:issue:`766`). -* ``tests/vendor/test_vendor_reg_apptype_generator_unit.py`` - gated on ``mypy`` with no ``MODULE_PROVIDERS`` entry for it, so - ``_top_level_providers()`` died on a bare ``KeyError: 'mypy'`` the moment - the gate was made visible -- a further instance of the hazard :issue:`745` exists - to stop, the same one :issue:`766` above also carries. - ``MODULE_PROVIDERS['mypy'] = ('mypy',)`` fixes the lookup, and the three - bare ``MODULE_PROVIDERS[...]`` subscripts route through that same function, - which raises an ``AssertionError`` naming the module, the key looked up and - the table to add it to on any future gap, rather than a bare ``KeyError``. - ``mypy`` is deliberately not added to a ``pyproject.toml`` extra: no extra - carries it today, it is a ``Pipfile`` dev-dependency that ``lint.yml`` - installs directly and already runs whole-package, and ``lint.yml`` runs no - pytest job for ``pytest_jobs()`` to see -- so the fix is a new - ``DEPENDENCY_GATE_EXCLUSIONS['HAS_MYPY']`` entry on ``test``, - ``engine-tests`` and ``gate`` rather than an install line. - ``dependency_gate_gaps()`` goes 15 to 18 ``(flag, job)`` pairs, all three - new and expected. ``tests/test_tier_guard.py`` goes 97 to 100 tests, 512 to - 524 subtests -- measured under plain ``unittest`` with a ``subTest`` - counter rather than trusting ``pytest-subtests`` (:issue:`779`). -* ``engine='pyshark'`` had no CI job driving it through a real - ``tshark``-parsed capture. ``pypcap-parity`` installs ``pyshark`` but its - ``apt-get install`` step named no ``tshark`` package, so - ``PyShark.unsupported_reason()`` declined the engine, the extractor fell back - to the built-in parser, and the one real-extraction assertion, - ``assertGreater(extractor.length, 0)``, passed against the fallback's output - -- a vacuous pass invisible in every skip count. ``pypcap-parity`` now - installs ``tshark`` with the same debconf pre-seed ``engine-tests`` already - carries, and the test proves which engine actually ran: no ``EngineWarning`` - fires, ``extractor._exnam == 'pyshark'`` and the engine is a ``PyShark`` - instance -- the same idiom the PyPCAP parity test already used. Only - ``tests/integration/test_engine_runtime.py`` and ``test_engine_parity.py`` - change among the test files; the length assertion itself is kept, not - weakened (:pr:`846`). -* ``engine-tests`` ran one Python-version leg per interpreter - regardless of which extraction engines that interpreter could actually - install, so ``pyproject.toml``'s own + rather than transcribed. Three things needed deciding. 3.0's two ``[NOTE`` + placeholders are filled or removed: the reporting channel names + ``jarryshaw@icloud.com`` (the contact 1.4 carried and ``SECURITY.md`` uses as email + fallback) plus GitHub's report-abuse form for a report about the maintainer, which a + single-maintainer project cannot otherwise cover; the enforcement placeholder is an + instruction to the adopter and is removed. 3.0's plural "Community Moderators" become + the singular maintainer (eight occurrences), this repository having no moderators. + The four-rung ladder (Warning, Temporarily Limited Activities, Temporary Suspension, + Permanent Ban) is a suggestion and is **kept**, each rung mapping to a lever one + person holds on GitHub. 3.0 is CC BY-SA 4.0 where 1.4 carried no licence notice, so + the attribution names version 3.0, links the permanent ``version/3/0/`` URL, carries + the CC BY-SA 4.0 notice, indicates changes were made as BY requires, and says the + share-alike term covers this document only (the code remains BSD-3-Clause, ``LICENSE`` + untouched). The ladder renders as four list items each nesting three, checked against + GitHub's Markdown API; :pr:`613` had to repair the 1.4 file when a stray list marker + collapsed it into one nested item (:pr:`624`). +* reconciled which private (``_xxx``) attributes and methods the Sphinx build + documents, replacing an ad hoc mix with one rule: document the contract, hide the + recipe. A member every subclass must implement, or whose shape a caller depends on, + stays documented despite its underscore; a private helper that only keeps one method + short does not. Seven module-level directives for pure implementation detail + (``esp._resolve``, ``esp._CRYPTO``, ``ngap._convert``, ``ngap._revert``, + ``ngap._PYCRATE``, ``ngap._PDU_LOCK``, ``pypcapfile._NamedStream``) were dropped, + while 39 ``autoattribute`` directives were added for class-private state that *is* + contract, among them ``Extractor._flag_f``, the ``PyPCAP`` and ``PCAP_CT`` engines' + ``_backend`` (the only two of the six third-party engines with one), ``TraceFlow``'s + internal fields, ``FieldBase``/``Field`` internals, the built-in ``PCAP`` engine's + ``_gbhdr``, ``_vinfo``, ``_nnsec`` and ``_dlink`` (shared with ``PyPCAP``, + ``PCAP_CT`` and ``PyPCAPFile``) and ``PCAPNG``'s ``_ctx``/``_ctx_list``. The 128 + runtime definitions of ``_missing_`` (121 under ``pcapkit.const``, 7 inline in + ``pcapkit.protocols``) gained one write-up in place of a directive per class, a new + "Unrecognised Values" section in ``docs/source/pcapkit/const/index.rst`` + cross-referenced from ``registry.rst`` (``conf.py`` already excludes ``_missing_`` + from every ``autoclass``). ``CONTRIBUTING.md`` gained the rule as a named section. + ``sphinx-build -b html`` gives 53 warnings on ``main`` and 54 after, the addition being + the pre-existing bare ``Type`` ambiguity newly rendered by ``TraceFlow._foutio``'s new + directive, tracked as :issue:`709`. No line under ``pcapkit/`` changed (:issue:`684`). +* ``examples/captures/out.json``, ``out.plist``, ``out.txt`` and ``pcapng.txt`` are no + longer tracked in git; they are build output, not fixtures, and a tracked rendering + with no reader goes stale silently. ``pcapng.txt`` is exactly that: it still recorded + ``packet -> NIL`` for the four Enhanced Packet Blocks of ``dhcp.pcapng`` long after + :pr:`683` gave those blocks their captured octets back. Rather than regenerate it again + and leave the drift free to recur, the four files are removed from the index and folded + into ``examples/captures/``'s blanket ``.gitignore`` rule; the genuine inputs + ``in.pcap`` and ``dhcp.pcapng`` stay tracked. ``examples/legacy_smoke/Makefile`` and + its ``README.rst`` now describe regenerating these reports via ``make fixtures``, and + ``tests/project/test_capture_tracking.py`` (5 tests, 8 subtests) pins the invariant; 2 + of the 5 fail against the pre-change tree with the four reports restored, the other 3 + passing on both by construction (:issue:`685`). +* ``tests/project/test_isort_clean.py`` gated on ``isort`` with a + ``try``/``except ImportError`` inside ``setUpClass``, which + ``tests/_dependency_gates.py`` cannot see (``_gates_of()`` walks a decorator list, + never a function body), and ``isort`` was in no ``pyproject.toml`` extra, so the + file reported ``OK (skipped=1)`` on every CI leg and the dependency-gate guard could + not tell. It now gates on a module-level ``HAS_ISORT`` and a class-level + ``@unittest.skipUnless`` so ``gated_scopes()`` counts it, ``MODULE_PROVIDERS`` maps + ``isort``, and ``isort`` joins the ``test`` extra rather than being excluded + (``lint.yml`` installs none, so unlike ``mypy`` there is no existing check to defer + to). No workflow changed, every pytest job already building on ``.[test]``; + ``gated_scopes()`` goes 250 to 251 and ``dependency_gate_gaps()`` holds at 18. The + third instance of the hazard :issue:`745` exists to stop (:issue:`766`). +* ``tests/vendor/test_vendor_reg_apptype_generator_unit.py`` gated on ``mypy`` with no + ``MODULE_PROVIDERS`` entry, so ``_top_level_providers()`` died on a bare + ``KeyError: 'mypy'`` once the gate became visible -- a further instance of the + hazard :issue:`745` exists to stop, like :issue:`766` above. + ``MODULE_PROVIDERS['mypy'] = ('mypy',)`` fixes the lookup, and the three bare + ``MODULE_PROVIDERS[...]`` subscripts route through that function, which raises an + ``AssertionError`` naming the module, key and table on any future gap. ``mypy`` is + deliberately not added to a ``pyproject.toml`` extra: no extra carries it, it is a + ``Pipfile`` dev-dependency that ``lint.yml`` installs and already runs + whole-package, and ``lint.yml`` runs no pytest job for ``pytest_jobs()`` to see, so + the fix is a ``DEPENDENCY_GATE_EXCLUSIONS['HAS_MYPY']`` entry on ``test``, + ``engine-tests`` and ``gate``. ``dependency_gate_gaps()`` goes 15 to 18 + ``(flag, job)`` pairs, all three new and expected; (:issue:`779`). +* ``engine='pyshark'`` had no CI job driving it through a real ``tshark``-parsed + capture. ``pypcap-parity`` installs ``pyshark`` but its ``apt-get install`` named no + ``tshark`` package, so ``PyShark.unsupported_reason()`` declined the engine, the + extractor fell back to the built-in parser, and the one real-extraction assertion, + ``assertGreater(extractor.length, 0)``, passed against the fallback's output: a vacuous + pass invisible in every skip count. ``pypcap-parity`` now installs ``tshark`` with the + debconf pre-seed ``engine-tests`` carries, and the test proves which engine ran (no + ``EngineWarning``, ``extractor._exnam == 'pyshark'``, a ``PyShark`` instance), as the + PyPCAP parity test does. Only ``tests/integration/test_engine_runtime.py`` and + ``test_engine_parity.py`` change; the length assertion is kept (:pr:`846`). +* ``engine-tests`` ran one Python-version leg per interpreter regardless of which + engines that interpreter could install, so ``pyproject.toml``'s ``PyPCAPFile = ["pypcapfile; python_version < '3.12'"]`` marker resolved to - *nothing* on 3.12--3.14, yet the ``Engines Python 3.12/3.13/3.14`` legs still - passed and, once promoted to a required check, gated merges while exercising - an engine that was never installed. Rebuilt as a genuine Python x engine - matrix -- 6 interpreters x 6 engines, 36 cells after 10 ``include`` overrides - -- with three honest outcomes: **26** ``supported`` cells that install - through the real ``pyproject.toml`` extra and must pass, **6** - ``unsupported`` cells (``PyPCAPFile`` on 3.12--3.15, ``PyShark`` on - 3.14/3.15) installed by a raw ``pip install`` to bypass the marker and assert - the engine's own ``PYTHON_CEILING`` decline, and **4** ``not-installable`` - cells (``PyPCAP`` on 3.12--3.15) where a genuine C-extension build failure is - logged and the test step skipped. The install step never passes silently in - either direction. 3.15 is ``continue-on-error`` throughout; job names change - shape (``Engines Python X`` becomes ``Engines Python X (engine)``), which - needed the branch-protection ruleset's required-check list updated in the - same window so a required check that stopped existing did not block every - merge permanently (:pr:`849`). -* Ruleset ``23497679``'s ``required_status_checks`` names 22 - exact check contexts, and GitHub rulesets match literally with no wildcard, - so the required list needs hand-editing on every matrix change. Five of the - 22, the old single-cell ``Engines Python `` names, are already dead - since :pr:`849`; the five ``Compat Python 3.10``-``3.14`` names stay live, emitted - independently by ``python-compatibility.yml``, and are not touched. Adds one - job, ``required-checks`` (context ``Required checks passed``), depending on + *nothing* on 3.12--3.14, yet the ``Engines Python 3.12/3.13/3.14`` legs passed and, + once required, gated merges while exercising an engine that was never installed. + Rebuilt as a Python x engine matrix (6 interpreters x 6 engines, 36 cells after 10 + ``include`` overrides) with three outcomes: **26** ``supported`` cells that install + through the real extra and must pass; **6** ``unsupported`` cells (``PyPCAPFile`` on + 3.12--3.15, ``PyShark`` on 3.14/3.15) installed by raw ``pip install`` to bypass the + marker and assert the engine's own ``PYTHON_CEILING`` decline; and **4** + ``not-installable`` cells (``PyPCAP`` on 3.12--3.15) where a genuine C-extension + build failure is logged and the test step skipped. The install step never passes + silently in either direction. 3.15 is ``continue-on-error`` throughout. Job names + change shape (``Engines Python X`` becomes ``Engines Python X (engine)``), so the + branch-protection ruleset's required-check list was updated in the same window, lest + a required check that stopped existing block every merge (:pr:`849`). +* Ruleset ``23497679``'s ``required_status_checks`` names 22 exact check contexts, and + GitHub rulesets match literally with no wildcard, so the list needs hand-editing on + every matrix change. Five of the 22, the old ``Engines Python `` names, are + dead since :pr:`849`; the five ``Compat Python 3.10``-``3.14`` names stay live, + emitted by ``python-compatibility.yml``, and are untouched. Adds one job, + ``required-checks`` (context ``Required checks passed``), depending on ``[test, integration, engine-tests, pypcap-parity]`` with - ``if: always() && inputs.gate-only != true`` and an explicit per-dependency - check that fails loudly, naming the job, on anything but ``success`` -- a - bare ``needs:`` list would leave this job silently *skipped* rather than - failed the moment a dependency failed, and GitHub's own troubleshooting docs - list a skipped required check as passing. That one context is meant to - replace 17 of the ruleset's 22 hand-listed entries (``test``, - ``integration``, ``Engines``, ``pypcap-parity``) once the ruleset itself is - updated separately -- a repository setting this PR does not touch, and the - five ``Compat`` names are deliberately excluded from that replacement. One - file, ``.github/workflows/unit-tests.yml``, 167 lines added; no library or - test code changes (:pr:`856`). + ``if: always() && inputs.gate-only != true`` and an explicit per-dependency check + that fails loudly, naming the job, on anything but ``success``: a bare ``needs:`` + list would leave it *skipped* rather than failed when a dependency failed, and + GitHub's docs list a skipped required check as passing. That one context is meant to + replace 17 of the ruleset's 22 entries (``test``, ``integration``, ``Engines``, + ``pypcap-parity``) once the ruleset is updated separately, a repository setting this + PR does not touch; the five ``Compat`` names are excluded from the replacement. One + file, ``.github/workflows/unit-tests.yml``, 167 lines added; no library or test code + (:pr:`856`). Fixed ~~~~~ -* 45 places where a documentation page contradicted the code - (:pr:`413`), ambiguous cross-references and five autodoc signature failures (:pr:`416`), - and ``Extractor``'s documented exception plus 40 phantom or stale ``Args:`` - labels (:pr:`501`). -* both halves of what the ``Changelog drift`` gate told an author, - in ``util/changelog_md.py``. ``ResidualMarkupError`` named the entry file but - numbered its lines against the *converted* body, which rule 6 joins onto one - line per block: measured on this entry, the body is 55 lines against the file's - 580, so a reported number could not reach most of the file at all, and four - roles written on two source lines were all reported as ``line 46``. The - conversion now carries a source map -- which line of the entry each stretch of - output came from -- so every complaint cites a line of the file the message - names, one complaint per construct rather than one per joined line, in the - entry's own order (:issue:`588`). Rule 2 separately accepted only the bare-number - spelling of Sphinx's ``:rfc:`` role, so a citation of :rfc:`6554#section-3` - fell past the rule that exists for it and was reported as a role the rules do - not cover; both spellings now convert, with the anchor carried into the link - target and the link text taken from Sphinx's own so that the Markdown and the - rendered history say the same thing about the same page (:issue:`592`). Regenerating - ``CHANGELOG.md`` is byte-identical over all 37 committed entries, so the fix - changes what the gate *says* and nothing about what it emits. -* ``examples/generators/options.py``'s ``TCP_BASE`` spelled three - header fields with names ``TCP.make`` does not declare, so every option - fixture in ``examples/captures/options-tcp.pcap`` was built from values the - generator had not asked for. ``make`` ends in ``**kwargs`` and reads nothing - out of it, so an undeclared keyword is accepted and discarded with no warning - and no ``TypeError``, which is why this survived unnoticed. ``'seq': 1`` was - ``seq_no``, and all 25 captured frames therefore carried sequence number 0; - ``'urgent_pointer'`` was ``urgent``, whose requested value and unused default - both happened to be 0; and ``'ack_flag': False`` was ``ack``, while - ``'ack': 0`` bound to ``ack`` itself -- the acknowledgement *flag* rather than - the number, which is ``ack_no`` -- so that pair was the wrong way round in - both directions. Each is renamed to the parameter ``make`` declares, keeping - the values the table always read as. Measured: +* 45 places where a documentation page contradicted the code (:pr:`413`), ambiguous + cross-references and five autodoc signature failures (:pr:`416`), and + ``Extractor``'s documented exception plus 40 phantom or stale ``Args:`` labels + (:pr:`501`). +* both halves of what the ``Changelog drift`` gate told an author, in + ``util/changelog_md.py``. ``ResidualMarkupError`` named the entry file but numbered + its lines against the *converted* body, which rule 6 joins onto one line per block (55 + lines against the file's 580 on this entry), so a reported number could not reach most + of the file, and four roles on two source lines were all reported as ``line 46``. The + conversion now carries a source map, so every complaint cites a line of the file the + message names, one per construct, in the entry's order (:issue:`588`). Rule 2 accepted + only the bare-number spelling of Sphinx's ``:rfc:`` role, so :rfc:`6554#section-3` + fell past it and was reported as a role the rules do not cover; both spellings now + convert, the anchor carried into the link target and the link text taken from + Sphinx's own (:issue:`592`). Regenerating ``CHANGELOG.md`` is byte-identical over all + 37 committed entries: the fix changes what the gate *says*, not what it emits. +* ``examples/generators/options.py``'s ``TCP_BASE`` spelled three header fields with + names ``TCP.make`` does not declare, so every option fixture in + ``examples/captures/options-tcp.pcap`` was built from values the generator had not + asked for. ``make`` ends in ``**kwargs`` and reads nothing out of it, so an + undeclared keyword is accepted and discarded with no warning. ``'seq': 1`` was + ``seq_no``, so all 25 captured frames carried sequence number 0; ``'urgent_pointer'`` + was ``urgent`` (requested value and default both 0); and ``'ack_flag': False`` was + ``ack`` while ``'ack': 0`` bound to ``ack`` itself, the acknowledgement *flag* + rather than the number (``ack_no``), so that pair was wrong both ways. Each is renamed + to the parameter ``make`` declares, keeping the intended values. ``TCP(**TCP_BASE).info.seq`` was 0 against a declared 1 and is now 1, and - ``options-tcp.pcap`` changes in exactly 25 bytes -- the low octet of each - frame's sequence number -- with no case status and no ``EXPECTED_FAILURES`` - entry moving. The module's six other base mappings were audited the same way: - five pass nothing their ``make`` does not declare, and HIP's ``extension`` is - declared by ``HIP.read`` instead, which is this library's documented way of - forwarding a read-path keyword through ``**kwargs`` rather than a sixth - instance of the defect (:issue:`602`). -* ``util/bump_version.py`` left ``CITATION.cff`` naming the previous - release. Nothing else in the repository maintains that file -- no workflow, hook - or packaging file mentions it -- so every bump since it landed in :pr:`615` would have - stranded the ``version`` and ``date-released`` it renders as GitHub's "Cite this - repository" button and that citation managers, Zenodo and dependency inventories - read directly. Both fields now move with ``__version__``. They have the same - standing, since the file's own header says both describe the newest *published* - release, and moving only one would assert that 1.5.0b5 was released on the day - 1.5.0b4 was; the date is taken in UTC, because seven of the thirty most recent - bumps were made late evening in US-Eastern where a local date is a day behind the - publish it describes. That the two are the same day at all is measured rather - than assumed: the bump is what triggers ``create-release.yml``, the median gap to - the PyPI upload is three minutes, and the UTC calendar dates agree 30 times out - of 30. The rewrite is line-oriented, so the comment header, key ordering and each - field's existing quoting survive -- ``cff-version`` and a ``references`` entry's - own ``version`` are anchored out at column zero -- and the result is checked with - ``cffconvert --validate``. An absent file is reported on stderr and skipped - rather than failing the vendor cron before its ``git commit``, which would - discard the whole registry crawl for the sake of a documentation file; a file - present with no ``version`` field raises instead, before anything is written, - because rewriting nothing while reporting success is the staleness this fixes. - Two things came with it. The script gains a ``main()`` guard, having previously - run the entire bump at import, which is why it had no testable surface; and the - ``import pcapkit`` fallback in its version reader, which returned ``"1.5.0b4'\n"`` - -- closing quote and newline included, which ``packaging`` rejects -- is fixed, - a path that had never worked and went unnoticed because the only caller installs - the package first. A new gate asserts the committed file still names the packaged - version, covering the version changes made by hand, which never run this script at - all -- 40 of the 159 commits that have moved ``__version__`` on ``main``, a quarter - over the project's life and 11 of the most recent 25 (:pr:`625`). -* ``LICENSE``'s copyright notice began the term at 2018, a year after - the work it covers. The repository's first commit is ``c57f7d0b7`` "Initial - commit", dated 2017-11-07, so the notice understated the term and contradicted the - only other copyright site in the tree: ``docs/source/conf.py`` computes its Sphinx - footer as ``f'2017-{datetime.date.today().year}, Jarry Shaw'`` and has said 2017 - for as long as it has existed. The two now agree on where the term starts. The - wrong start year survived every maintenance pass the line has had, because each - looked only at the other end of the range: ``bc836cfa2`` (2020-05-31) introduced - ``2018-2020`` when the BSD-3-Clause text replaced MPL 2.0, and the range was then - bumped by hand three times, to ``2018-2022``, ``2018-2023`` and ``2018-2026`` - (:pr:`615`), each bump correcting the end year and copying ``2018`` forward untouched. - 2017 is being **restored** rather than newly asserted. The project's first - ``LICENSE`` -- MIT, at ``c57f7d0b7``, the initial commit -- already read - ``Copyright (c) 2017 Jarry Shaw`` on line 3, and the GPL v3 era that followed - carried ``Copyright (C) 2017 Jarry Shaw`` in its filled-in "how to apply" - appendix. 2017 was lost on 2018-12-08, when ``4f3e00d5f`` relicensed to Apache 2.0 - under ``Copyright 2018 Jarry Shaw`` -- a single year, correct for the year it was - written -- and ``1c69341dc`` replaced that with MPL 2.0 text the same day, which - carried no author notice at all. ``bc836cfa2`` then picked ``2018`` back up as a - range start in 2020, by which point it was two years stale and no longer described - anything. The end year is - **dropped** rather than automated, which is the other half of the change: the - notice ships inside the sdist and the wheel, so its text is fixed at build time in - every copy already downloaded and cannot be computed the way the docs footer is; - copyright subsists from creation whether or not a notice names the current year; - and BSD-3-Clause's canonical form is ``Copyright (c) ``, singular. So - a range here is six years of hand maintenance on one line buying nothing, and no - workflow, script hook or other automation is added in its place -- that was - considered and rejected, since automating a value that need not be current is worse - than not carrying it. ``conf.py`` is deliberately untouched, a self-maintaining - docs footer showing a range being both conventional and correct. The licence body - is unaltered: the whole-file diff is one hunk at line 3, and the remaining 28 lines - carry the canonical BSD-3-Clause wording verbatim. Three things distinguish the - file from SPDX's bare ``licenseText``, and all three predate this change -- the - ``BSD 3-Clause License`` title line, which the choosealicense.com template carries - and SPDX omits; the ``*`` bullets in place of ``1.``/``2.``/``3.``; and the extra - ``All rights reserved.`` line -- and the ``DAMAGE.`` that :pr:`615` repaired from a - stray ``DAMAGE.s`` is still clean (:pr:`630`). -* ``CITATION.cff`` shipped in no source distribution. ``MANIFEST.in`` - carries an ``include`` line each for ``README.md``, ``LICENSE`` and - ``CHANGELOG.md`` but had none for the citation file, and its two - ``global-include`` patterns are ``*.rst`` and ``*.py``, neither of which can match - a ``.cff`` -- so the file sat in the repository and in nothing published. Nor was - there a default to fall back on. Removing all three of those ``include`` lines and - rebuilding shows ``README.md`` and ``LICENSE`` shipping anyway, setuptools adding - the latter from ``license_files`` and recording it as ``License-File: LICENSE`` in - ``PKG-INFO``, while ``CHANGELOG.md`` vanishes -- so of the three only + ``options-tcp.pcap`` changes in exactly 25 bytes (the low octet of each frame's + sequence number), with no case status or ``EXPECTED_FAILURES`` entry moving. The + module's six other base mappings were audited: five pass nothing their ``make`` does + not declare, and HIP's ``extension`` is declared by ``HIP.read``, this library's + documented way of forwarding a read-path keyword, not a sixth instance (:issue:`602`). +* ``util/bump_version.py`` left ``CITATION.cff`` naming the previous release. Nothing + else maintains that file, so every bump since :pr:`615` would have stranded the + ``version`` and ``date-released`` that GitHub's "Cite this repository" button, + citation managers, Zenodo and dependency inventories read. Both fields now move with + ``__version__``, together, since the file's header says both describe the newest + *published* release and moving only one would assert that 1.5.0b5 was released on the + day 1.5.0b4 was. The date is UTC, because seven of the thirty most recent bumps were + made late evening US-Eastern, where a local date is a day behind the publish; the bump + triggers ``create-release.yml``, the median gap to the PyPI upload is three minutes, and + the UTC dates agree 30 times of 30. The rewrite is line-oriented, so the comment + header, key ordering and each field's quoting survive (``cff-version`` and a + ``references`` entry's own ``version`` are anchored out at column zero), and the result + is checked with ``cffconvert --validate``. An absent file is reported on stderr and + skipped rather than failing the vendor cron before its ``git commit``, which would + discard a whole registry crawl for a documentation file; a file with no ``version`` + field raises before anything is written, since reporting success while rewriting + nothing is the staleness being fixed. The script gains a ``main()`` guard (it ran the + whole bump at import, hence no testable surface), and the ``import pcapkit`` fallback in + its version reader, which returned ``"1.5.0b4'\n"`` (closing quote and newline, which + ``packaging`` rejects), is fixed; it never worked and went unnoticed because the only + caller installs the package first. A new gate asserts the committed file still names + the packaged version, covering hand-made version changes that never run this script + (40 of the 159 commits that moved ``__version__`` on ``main``, 11 of the latest 25) + (:pr:`625`). +* ``LICENSE``'s copyright notice began the term at 2018, a year after the work it + covers. The repository's first commit is ``c57f7d0b7``, dated 2017-11-07, so the + notice understated the term and contradicted ``docs/source/conf.py``, whose Sphinx + footer is ``f'2017-{datetime.date.today().year}, Jarry Shaw'``. The wrong start year + survived every maintenance pass because each looked only at the other end of the + range: ``bc836cfa2`` (2020-05-31) introduced ``2018-2020`` when the BSD-3-Clause + text replaced MPL 2.0, and hand bumps to ``2018-2022``, ``2018-2023`` and + ``2018-2026`` (:pr:`615`) each copied ``2018`` forward. 2017 is **restored** rather + than newly asserted: the initial MIT ``LICENSE`` read + ``Copyright (c) 2017 Jarry Shaw``, and the GPL v3 era carried + ``Copyright (C) 2017 Jarry Shaw`` in its appendix; it was lost on 2018-12-08, when + ``4f3e00d5f`` relicensed to Apache 2.0 under ``Copyright 2018 Jarry Shaw`` and + ``1c69341dc`` replaced that with MPL 2.0 text with no author notice. The end year is + **dropped** rather than automated: the notice ships inside the sdist and wheel, so + its text is fixed at build time in every copy already downloaded and cannot be + computed like the docs footer; copyright subsists from creation whether or not a + notice names the current year; and BSD-3-Clause's canonical form is a singular + ``Copyright (c) ``. A range was six years of hand maintenance buying + nothing, and automation was considered and rejected, since automating a value that + need not be current is worse than not carrying it. ``conf.py`` is untouched (a + self-maintaining footer showing a range is conventional and correct). The licence + body is unaltered: one hunk at line 3, and the ``DAMAGE.`` that :pr:`615` repaired + from ``DAMAGE.s`` is still clean (:pr:`630`). +* ``CITATION.cff`` shipped in no source distribution. ``MANIFEST.in`` has an ``include`` + each for ``README.md``, ``LICENSE`` and ``CHANGELOG.md`` but none for the citation file, + and its two ``global-include`` patterns (``*.rst``, ``*.py``) cannot match a ``.cff``. + Removing all three ``include`` lines and rebuilding shows ``README.md`` and ``LICENSE`` + shipping anyway (setuptools adds the latter from ``license_files``, recorded as + ``License-File: LICENSE`` in ``PKG-INFO``) while ``CHANGELOG.md`` vanishes, so only ``CHANGELOG.md`` is load-bearing, and a citation file, which no packaging default - covers at all, is in the same position. The gap is invisible from the web UI, - because GitHub renders the "Cite this repository" - button from the repository itself; citation managers, Zenodo and dependency - inventories read the published artifact, which is exactly the surface that was - missing it, so the machine-readable half of the attribution stopped travelling - with the code at the one point nobody can see the repository. :pr:`615` flagged the - omission when it added the file and :pr:`619` did not address it. It matters now - because :pr:`625` has just taught ``util/bump_version.py`` to keep the file's - ``version`` and ``date-released`` in step with the bump, and a release exercising - that path would otherwise publish an sdist omitting the very artefact under test. - It also lets ``RepositoryCitationTests`` in - ``tests/project/test_bump_version.py`` -- the gate :pr:`625` added for the hand-authored - bumps that never run the script -- execute against an unpacked sdist, where today - it skips itself with "CITATION.cff is not shipped in the source distribution". - Measured both ways with ``python -m build --sdist``: ``tar tzf`` found no - ``CITATION.cff`` before and ``pypcapkit-1.5.0b4/CITATION.cff`` after, the two - archive listings differ by that one added entry and nothing else -- 860 against - 861 -- the shipped copy is byte-identical to the repository's, and - ``twine check --strict`` reports ``PASSED`` on both archives (:pr:`631`). -* two inaccurate claims in the :pr:`630` entry above, caught by review after - :pr:`630` had already merged. Both were about the history of ``LICENSE`` rather than - about the change itself, and ``LICENSE`` is untouched here: 2017 was and remains - the right answer, so this is a changelog-prose correction only. The entry asserted - that the file "until then was stock MPL text carrying no author notice at all, so - 2018 is the first start year the project ever asserted and it was already a year - late when it was written". Every clause of it is wrong as a description of the era, - though each was true of the single blob ``bc836cfa2`` happened to replace, which is - where the mistake came from. The pre-BSD file was MIT, then GPL v3, then Apache 2.0, - and only then MPL 2.0 -- four licences before BSD, not the one implied. The - MIT and GPL texts both carried an author notice naming **2017**, the MIT one on - line 3 of the initial commit. And ``2018`` first appeared in the Apache notice of - 2018-12-08 as a single year, current for the year it was written rather than late. - The claim also contradicted the entry's own opening sentence, which dates the first - commit to 2017-11-07. What the history actually supports is a stronger argument for - the change than the one that was written, which is why this is corrected rather than - deleted: 2017 is the year the project asserted from its first commit, and :pr:`630` - restores it rather than asserting it anew. The ``BSD 3-Clause License`` title line - is reattributed too, from the opensource.org template -- whose licence text begins - at the copyright line -- to choosealicense.com's, whose first line is literally that - title, and which also carries the ``Copyright (c) [year], [fullname]`` comma this - file uses. The error came from generalising the pre-BSD era off two blobs that - turned out to be the same file, ``bc836cfa2^:LICENSE`` and ``1c69341dc:LICENSE``; - enumerating all 14 commits that ever touched ``LICENSE`` is what caught it (:pr:`638`). -* the release workflow published to PyPI and to Anaconda with - nothing a human had to approve. ``create-release.yml``'s ``pypi`` job carried - its ``environment: release`` commented out while the ``id-token: write`` from - the same upstream snippet had been re-added live below it, so the block read as - disabled as a unit when only its gate still was; the ``conda`` job had no - ``environment:`` at all, not even a commented one. Neither needed a tag either: - the workflow also fires on the completion of ``Vendor Update``, so a scheduled - registry crawl that bumped the version was enough to publish to two public - indexes, at a median three minutes from bump to upload. Four jobs reach outside - the run and all four are now gated -- ``github`` on ``github-release`` for the - GitHub Release and the ``v*`` tag it creates, ``tag`` on ``conda-tag`` for the - commit it pushes to ``main``, ``pypi`` on ``pypi``, and ``conda`` on - ``anaconda``. One environment per credential rather than one shared - ``release``, because approval is granted to an environment and not to a job, so - a single ``release`` approved once would release both indexes together. PyPI is - the one that has to be answerable alone: a version number it has accepted - cannot be reused, and ``skip-existing: true`` makes the re-upload *succeed* - having published nothing, so an unintended publish permanently consumes the - number the intended release wanted. The workflow half is only half the fix -- - an ``environment:`` naming an environment that does not exist is created - implicitly with no protection rules and the job proceeds unapproved, so this is - inert until each of the four exists in repository settings with a required - reviewer on it, and each one's "Deployment branches and tags" has to stay - unrestricted or the ``tags: v*`` trigger fails outright instead of pausing. - ``github-pages`` is the standing example of that failure in this repository, - carrying no protection rule since 2021. The new test asserts the general rule - rather than the current file -- no job running a publishing action or a - ``git push`` may omit an ``environment:`` -- so a publishing job added later - without a gate fails rather than ships (:issue:`641`). -* three test comments stated an exact count of committed - captures under ``examples/captures/``, a number that drifts every time that - directory's tracked set changes and in one case was already wrong when - written. ``tests/_tiers.py`` cited "six," correct at the time; - ``tests/test_tier_guard.py`` said "the moment a seventh capture is - committed" -- the same count, as an ordinal rather than the word "six"; - ``tests/integration/_helpers.py`` said "four," which undercounted the - tracked set the day it was written. All three now - describe the invariant instead of a number that has to be kept in sync with - it by hand: ``_tiers.py`` reads "the moment somebody commits another - capture... or stops committing one," ``_helpers.py`` reads "...are - fixtures -- some of them committed --...," and ``test_tier_guard.py`` drops - its "seventh capture" phrasing the same way. Prose-only, verified with + covers, is in the same position. The gap is invisible from the web UI, because GitHub + renders "Cite this repository" from the repository; citation managers, Zenodo and + dependency inventories read the published artifact, the surface that was missing it. + :pr:`615` flagged the omission and :pr:`619` did not address it. It matters now because + :pr:`625` taught ``util/bump_version.py`` to keep the file's ``version`` and + ``date-released`` in step, and a release exercising that path would publish an sdist + omitting the artefact under test. It also lets ``RepositoryCitationTests`` in + ``tests/project/test_bump_version.py`` (the gate :pr:`625` added for hand-authored + bumps) run against an unpacked sdist, where it skipped itself with "CITATION.cff is not + shipped in the source distribution". ``python -m build --sdist`` gives 860 archive + entries before and 861 after (``pypcapkit-1.5.0b4/CITATION.cff`` the only difference), the + shipped copy is byte-identical, and ``twine check --strict`` passes on both (:pr:`631`). +* two inaccurate claims in the :pr:`630` entry above, caught in review after :pr:`630` + had merged; both concern the history of ``LICENSE``, which is untouched here. 2017 was + and remains the right answer, so this is a changelog-prose correction. The entry said + the file "until then was stock MPL text carrying no author notice at all, so 2018 is + the first start year the project ever asserted and it was already a year late when it + was written". That was true of the single blob ``bc836cfa2`` replaced and wrong as a + description of the era: the pre-BSD file was MIT, then GPL v3, then Apache 2.0, then + MPL 2.0; the MIT and GPL texts both carried an author notice naming **2017**; and + ``2018`` first appeared in the Apache notice of 2018-12-08 as a single year, current + when written. The claim also contradicted the entry's own dating of the first commit to + 2017-11-07. The history supports a stronger argument than the one written, which is why + this is corrected rather than deleted: 2017 is the year asserted from the first commit, + and :pr:`630` restores it. The ``BSD 3-Clause License`` title line is reattributed from + opensource.org's template to choosealicense.com's, whose first line is that title and + which carries the ``Copyright (c) [year], [fullname]`` comma this file uses. The error + came from generalising the pre-BSD era off two blobs that were the same file + (``bc836cfa2^:LICENSE`` and ``1c69341dc:LICENSE``); enumerating all 14 commits that + touched ``LICENSE`` caught it (:pr:`638`). +* the release workflow published to PyPI and to Anaconda with nothing a human had to + approve. ``create-release.yml``'s ``pypi`` job had ``environment: release`` commented + out while the ``id-token: write`` from the same upstream snippet was re-added live + below it, so the block read as disabled as a unit when only its gate still was; the + ``conda`` job had no ``environment:`` at all. Neither needed a tag: the workflow also + fires on completion of ``Vendor Update``, so a scheduled registry crawl that bumped the + version was enough to publish to two public indexes, a median three minutes from bump to + upload. Four jobs reach outside the run and all four are now gated: ``github`` on + ``github-release`` (the GitHub Release and the ``v*`` tag), ``tag`` on ``conda-tag`` + (the commit pushed to ``main``), ``pypi`` on ``pypi``, ``conda`` on ``anaconda``. One + environment per credential rather than a shared ``release``, because approval is + granted to an environment and not a job, so one ``release`` approved once would release + both indexes together. PyPI has to be answerable alone: an accepted version number + cannot be reused, and ``skip-existing: true`` makes the re-upload *succeed* having + published nothing, so an unintended publish permanently consumes the number the + intended release wanted. The workflow half is only half the fix: an ``environment:`` + naming one that does not exist is created implicitly with no protection rules and the + job proceeds unapproved, so this is inert until each of the four exists in repository + settings with a required reviewer, and each one's "Deployment branches and tags" must + stay unrestricted or the ``tags: v*`` trigger fails outright instead of pausing. + ``github-pages`` is the standing example, carrying no protection rule since 2021. The + new test asserts the general rule, that no job running a publishing action or a + ``git push`` may omit an ``environment:``, so a publishing job added later without a gate + fails rather than ships (:issue:`641`). +* three test comments stated an exact count of committed captures under + ``examples/captures/``, a number that drifts whenever the tracked set changes and in + one case was wrong when written: ``tests/_tiers.py`` said "six," ``tests/test_tier_guard.py`` + "the moment a seventh capture is committed," and ``tests/integration/_helpers.py`` + "four," which undercounted on the day it was written. All three now describe the + invariant (``_tiers.py``: "the moment somebody commits another capture... or stops + committing one"; ``_helpers.py``: "...are fixtures -- some of them committed --..."; + ``test_tier_guard.py`` drops its ordinal). Prose-only, verified with ``coverage.parser.PythonParser`` that no executable statement moved (:issue:`700`). -* ``test_every_tracked_name_exists_and_matches_git`` in - ``tests/test_tier_guard.py`` checked neither the git index nor the - filesystem despite its name: it asserted only that the tracked-capture list - was non-empty and that no name contained a ``/``, so a hardcoded list would - pass it whether or not it matched reality. Sibling - ``test_capture_suggestions_are_captures`` had the same gap. Found while - cross-reviewing :pr:`703`'s first commit, itself prose-only and out of scope - for this; the fix rides in as that PR's second and third commits, and - :pr:`703`'s own description names both :issue:`700` and :issue:`708`. - ``test_every_tracked_name_exists_and_matches_git`` now re-derives the - expected set independently -- shelling out to ``git ls-files -z`` under - ``examples/captures/`` rather than calling back into ``_tiers.py`` -- and - asserts equality against it rather than against itself, plus a per-name - ``Path.is_file()`` check the old version never made. Its sibling - ``test_capture_suggestions_are_captures`` gets a narrower fix: it asserts - equality against ``committed_capture_names()``'s own filter re-implemented - inline over ``_tiers.committed_captures()``, which catches a wrong filter - but, unlike its sibling, still trusts ``committed_captures()`` for the - tracked set itself rather than re-deriving it from git. A cross-review of - the first version of this fix found - it wrong twice over: a stale docstring in ``_tiers.py`` still claimed - ``test_tier_guard.py`` "only stats ``in.pcap``," falsified by the new - per-name loop; and the new ``assertEqual`` on - ``test_capture_suggestions_are_captures`` was vacuous on an empty tracked - set (``() == ()`` passes trivially), having dropped the non-emptiness guard - the other test kept. Both are corrected in a follow-up commit. Even after - the fix, a hardcoded literal that happens to match today's tracked - names still passes -- the test detects a set that is wrong *at the moment it - runs*, not hardcoding as a practice, which turns a permanent blind spot into - a tripwire that fires on the next change to the tracked set (:issue:`708`). +* ``test_every_tracked_name_exists_and_matches_git`` in ``tests/test_tier_guard.py`` + checked neither the git index nor the filesystem despite its name: it asserted only + that the tracked-capture list was non-empty and no name contained a ``/``, so a + hardcoded list passed whether or not it matched reality. Sibling + ``test_capture_suggestions_are_captures`` had the same gap. Found while cross-reviewing + :pr:`703`'s first commit (prose-only, so out of scope there); the fix rides in as that + PR's second and third commits, and :pr:`703`'s description names both :issue:`700` and + :issue:`708`. The first test now re-derives the expected set independently, shelling + out to ``git ls-files -z`` under ``examples/captures/`` rather than calling back into + ``_tiers.py``, asserts equality against it, and adds a per-name ``Path.is_file()`` + check. Its sibling gets a narrower fix: equality against + ``committed_capture_names()``'s filter re-implemented inline over + ``_tiers.committed_captures()``, which catches a wrong filter but still trusts + ``committed_captures()`` for the tracked set. A cross-review of the first version found + it wrong twice: a stale ``_tiers.py`` docstring still claimed ``test_tier_guard.py`` + "only stats ``in.pcap``," falsified by the new per-name loop; and the new + ``assertEqual`` was vacuous on an empty tracked set (``() == ()``), having dropped the + non-emptiness guard the other test kept. Both are corrected in a follow-up commit. Even + now a hardcoded literal that happens to match today's names passes: the test detects a + set that is wrong *at the moment it runs*, not hardcoding as a practice, turning a + permanent blind spot into a tripwire that fires on the next change to the tracked set + (:issue:`708`).