Skip to content

fix(dumpkit): a nameless flag member no longer renders as Type::None [0] (#648) - #670

Merged
JarryShaw merged 1 commit into
mainfrom
fix/dumpkit-nameless-enum-648
Sep 22, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
fix/dumpkit-nameless-enum-648

Conversation

@JarryShaw

@JarryShaw JarryShaw commented Sep 22, 2026

Copy link
Copy Markdown
Owner

Fixes #648.

The defect

make_dumper's object_hook renders every enumeration member as Type::name [value], interpolating o.name with no guard. A Flag value composed entirely of undeclared bits has name is None rather than a string, so the literal four characters None landed in the name half:

JSON:  "connection": "Flags::None [0]",
Tree:  |-- connection -> Flags::None [0]
PLIST: <string>Flags::None [0]</string>

The three sites

All three carried character-for-character the same interpolation, and all three are in pcapkit/dumpkit/common.py:

pre-change line branch before after
:204 MultiDict/OrderedMultiDict key path {'Flags::None [0]': [...]} {'Flags::0 [0]': [...]}
:213 the addon branch's 'enum' key {'enum': 'Sub::None [0]', ...} {'enum': 'Sub::0 [0]', ...}
:216 the scalar return 'Flags::None [0]' 'Flags::0 [0]'

They now share one render_enum helper, so the guard cannot be applied to two of three, and a fourth site cannot be added without it. render_enum is documented under Internal Definitions in docs/source/pcapkit/dumpkit/common.rst, alongside DumperBase — it is deliberately not added to __all__.

What a nameless member renders as, and why

The value's own decimal spelling. Flags(0)'Flags::0 [0]', Flags(8)'Flags::8 [8]'.

Three reasons, each measured on this tree under CPython 3.14.7:

  1. It is what the enumeration libraries already do for an undeclared residue. Flags(2049).name is 'ACK|1' and Flags(2057).name is 'ACK|9' — the declared bit is named and the leftovers are given as one decimal number. A wholly-undeclared value is that same rendering with no declared bit in front, so the fallback continues an existing convention instead of inventing one.
  2. It cannot be mistaken for a member name. A Python identifier may not begin with a digit, so a bare decimal is structurally impossible as a declared name — 0 of the 1867 identifiers declared under pcapkit/const is a pure decimal. 'None' had no such protection, and NONE is a real declared name elsewhere in the library (Integrity(0).name == 'NONE'), so a consumer splitting the rendering on :: genuinely could not tell "no flags set" from a member so named.
  3. It needs no special case for zero, which matters because the defect never was about zero.

Alternatives considered and rejected: repr(o) gives 'Flags::<Flags: 0> [0]', which is redundant and nests brackets inside the bracketed half; an explicit token (NONE, UNNAMED, <unnamed>) reintroduces exactly the collision problem in (2); and dropping the ::name half to give Flags [0] changes the shape every consumer parses, which is a larger break than changing one field's content.

Two corrections the issue established, both reproduced

Not limited to value 0. The guard is on name is None, not on the value, because any all-undeclared value is nameless:

Flags(0)      name=None -> 'Flags::0 [0]'
Flags(1)      name=None -> 'Flags::1 [1]'
Flags(8)      name=None -> 'Flags::8 [8]'
Flags(9)      name=None -> 'Flags::9 [9]'
Flags(65536)  name=None -> 'Flags::65536 [65536]'

A fix keyed on zero would have left four of those five still emitting None. The test covers all five, so it cannot pass a zero-only fix.

Not an aenum quirk. A stdlib enum.IntFlag built from the same members answers name is None identically on 3.14.7, so the guard belongs at the interpolation rather than in a choice of enumeration library. The test asserts both libraries side by side.

One thing the issue undercounted

Five of the seven flag registries are nameless at zero, not onepcapkit.const.tcp.flags.Flags plus the four Mobility Header flag registries (BindingACKFlag, BindingUpdateFlag, HandoverACKFlag, HandoverInitiateFlag). The remaining two (TransportProtocol, CommandType) declare an explicit undefined = 0 and were always fine. The new test discovers the registries rather than naming them, so an eighth cannot slip past, and it asserts the nameless count is at least five so it cannot silently stop exercising the fix.

Interaction with #634

#634 has merged, so a flagless TCP segment now reaches this path and connection became a string where it had been the JSON number 0. #634's reviewer judged that an improvement because the field is now consistently a string instead of switching type with the flag bits — that judgement is unaffected here. What changes is only the name half of that string: "Flags::None [0]" becomes "Flags::0 [0]". The field stays a string, so the consistency #634 bought is kept; what it gains is that the string no longer claims a member name of None. #634's own test comment said the None was "left to its own change" — this is that change, and its assertion and prose are updated here.

Blame puts all three lines in dfac5d1767 (2023-04-28), three years before #634: pre-existing, not introduced by it.

The committed fixtures do not move

examples/captures/out.json, out.plist and out.txt are unchanged, and this was measured rather than assumed. All three were regenerated from in.pcap with and without the change and the two sets are byte-identicalrender_enum is the identity on every member that has a name, and no committed fixture contains a nameless one (grep '::None \[' returns nothing in all three).

A side-finding, reported and deliberately not fixed here: the committed fixtures have already drifted from what the current tree produces, independently of this change. That drift is substantial, not cosmetic — an earlier draft of this description called it "a +00:00 timezone suffix and different packet-byte offsets", which undersold it, and the cross-review was right to push back. It also includes whole frames whose hex payloads differ in content rather than offset, protocol -> NIL becoming protocol -> 5001, and packet -> NIL becoming real WireGuard-secrets bytes.

None of it is caused by this PR — the A/B regeneration above isolates that question and answers it byte-identically — but it means examples/captures/out.* no longer reflect what the library emits, which is worth its own change and is not one I have made here.

Failing, then passing

Both runs used the same harness: __editable__* stripped from sys.meta_path, the worktree at sys.path[0], pcapkit.__file__ asserted and printed before any other import, and pytest's exit code read from a file rather than a pipeline — the wrapper reported rc=0 while pytest exited 1, which is exactly the trap.

MEASURED pcapkit.__file__ = …/.claude/worktrees/agent-a3c59f390d4ed0e74/pcapkit/__init__.py

On main (fix reverted, tests present):

tests/dumpkit/test_nameless_enum_rendering_unit.py
  AssertionError: 'StdFlags::None [65536]' != 'StdFlags::65536 [65536]'
  18 failed, 2 passed, 2 subtests passed
  exit code (from file): 1

test_tcp_udp_unit.py::…::test_a_flagless_segment_seeds_its_connection_flags_as_an_enum
  AssertionError: 'Flags::None [0]' != 'Flags::0 [0]'
  1 failed, 4 subtests passed
  exit code (from file): 1

With the fix:

tests/dumpkit/ + test_tcp_udp_unit.py::…::test_a_flagless_segment_…
  15 passed, 25 subtests passed
  exit code (from file): 0

That 15 is the whole of tests/dumpkit/ — the pre-existing test_common_unit.py (9 tests) plus the new module (5) — and the one amended tcp test. Spelled out because it is not the new module's own figure: alone the new module is 5 passed / 17 subtests, and with the amended tcp test 6 passed / 21 subtests. The cross-review could not reconcile 15 against a module-only run, which is the ambiguity this paragraph removes.

Wider scoped run — tests/dumpkit/ tests/foundation/ tests/protocols/transport/test_tcp_udp_unit.py:

1 failed, 256 passed, 11 skipped, 397 subtests passed

That one failure is test_tcp_runtime.py::TCPReassemblyRuntimeTests::test_sample_capture_reassembles_every_stream_byte_exactly, which raises FileNotFoundError for a generated fixture in a tree where make samples has not been run. It fails identically with the fix reverted, so it is pre-existing and unrelated — the fixture-dependent tier, exactly as tests/_tiers.py documents.

The suite was never run whole; every run was scoped.

Coverage

pcapkit/dumpkit/common.py, statement and branch, measured with coverage directly:

statements branches cover tests subtests
before 71 32 100% 9 4
after 76 34 100% 14 21

Coverage does not go backwards: it holds at 100% while the file grows by 5 statements and 2 branches, and the new name is None guard is covered on both arms.

Labels — fix + breaking

fix is uncontested. On breaking, arguing it both ways as asked:

For. This changes user-visible output in all six format names (json, tree, text, txt, plist, xml) out of both Extractor and TraceFlow. Since #634 merged it is reachable from a real capture — any flagless segment, an nmap NULL scan being the canonical example — so it is not hypothetical. A consumer that string-matched Flags::None [0] breaks. .github/release.yml reads PR labels only, so omitting the label is the only way this becomes a silent change to the library's principal output, and the repo's own precedent (#635, fix(pcap) + a BREAKING CHANGE note for a public-attribute meaning change) points the same way.

Against. The label reads "Alters public API or wire output", and the wire bytes are untouched — o.value and the [value] half are unchanged, only the name half moves. Read narrowly, a dump-rendering change is neither the public API nor the wire. And the set of affected inputs is small: no registry other than the five nameless-at-zero flag enumerations can produce it.

Applied, because breaking is additive and the asymmetry is decisive: a false positive costs one honest line in the release notes, while a false negative ships a changed output format with nothing telling anyone. Happy to drop it if the label is meant strictly as "wire".

One more thing found and deliberately not fixed

pcapkit.const.pcapng.option_type.OptionType(0) has a string-typed .value ('opt_endofopt [0]'), so it already renders with nested brackets — 'OptionType::opt_endofopt [opt_endofopt [0]]'. That member is named, so render_enum treats it identically to the old code and this PR neither causes nor worsens it. Surfaced by the cross-review; a latent oddity in the rendering format, left to its own change.

Cross-review

Dispatched on a different model, briefed to falsify rather than bless; verdict posted as a comment below.

… [0]` (#648)

BREAKING CHANGE to dumped output. `make_dumper`'s `object_hook` interpolated
`o.name` unguarded, and a `Flag` value composed entirely of undeclared bits has
`name is None` rather than a string -- so the literal four characters `None`
landed in the name half and `pcapkit.const.tcp.flags.Flags(0)` rendered as
`Flags::None [0]` in `json`, `tree`, `text`, `txt`, `plist` and `xml`, out of
both `Extractor` and `TraceFlow`.

* Three sites, not one, all with character-for-character the same
  interpolation: the `MultiDict`/`OrderedMultiDict` *key* path, the `addon`
  branch's `'enum'` key, and the scalar return. They now share one
  `render_enum` helper, so the guard cannot be applied to two of three.
* The fallback is the value's own decimal spelling -- `Flags::0 [0]`,
  `Flags::8 [8]`. That is what the enumeration libraries already use for an
  undeclared residue: `Flags(2057).name` is `'ACK|9'`, naming the declared bit
  and giving the leftovers as one number, so a wholly-undeclared value is the
  same rendering with no declared bit in front. It also cannot be mistaken for
  a member name, since a Python identifier may not begin with a digit -- none
  of the 1867 identifiers under `pcapkit/const` is a bare decimal. `'None'`
  could be mistaken for one, and `NONE` is a real declared name elsewhere.
* Not limited to zero, so the guard is on `name is None` and not on the value.
  `Flags(1)`, `Flags(8)`, `Flags(9)` and `Flags(65536)` are equally nameless; a
  fix keyed on zero would have left four of five cases emitting `None`.
* Not an `aenum` quirk either -- a stdlib `enum.IntFlag` answers `name is None`
  identically on CPython 3.14.7, so the guard belongs at the interpolation
  rather than in a choice of enumeration library.
* Five of the seven flag registries in the library are nameless at zero, not
  one: `Flags` plus the four Mobility Header flag registries. The new test
  discovers them rather than naming them.
* #634 pinned the old string in `test_tcp_udp_unit.py` and said it was "left to
  its own change"; that assertion and its prose now read `Flags::0 [0]`. A
  flagless TCP segment is the library's only wire-reachable producer of a
  nameless member, which makes it the test that notices if the guard goes away.

`examples/captures/out.json`, `out.plist` and `out.txt` do not move: regenerating
all three from `in.pcap` with and without the change gives byte-identical output,
because `render_enum` is the identity on every member that has a name and no
committed fixture contains a nameless one.

`tests/dumpkit/ tests/foundation/ tests/protocols/transport/test_tcp_udp_unit.py`
passes 256 tests and 397 subtests. The new module is 5 tests and 21 subtests and
fails on `main` with `'Flags::None [65536]' != 'Flags::65536 [65536]'` among 18
failures; the amended #634 assertion fails there with `'Flags::None [0]' !=
'Flags::0 [0]'`. `pcapkit/dumpkit/common.py` holds 100% statement and branch
coverage across the change, 71 statements and 32 branches before, 76 and 34
after. One pre-existing failure is unrelated and unchanged by this commit:
`test_tcp_runtime.py::…::test_sample_capture_reassembles_every_stream_byte_exactly`
raises `FileNotFoundError` for a generated fixture in a tree where `make samples`
has not been run, identically with and without the change.

Fixes #648
@JarryShaw JarryShaw added fix Pull requests that fix a defect (fix: subject prefix) breaking Alters public API or wire output (apply alongside the type label) labels Sep 22, 2026
JarryShaw added a commit that referenced this pull request Sep 22, 2026
…s in #670 and #671

Two bullets, not one, because the two defects are unrelated: one changes what the
dumpers emit, the other only the text of four exception messages. They share a
file only by accident of being found in the same pass.

The #648 bullet leads with the output change and says so in bold, because that is
what a reader upgrading needs to see: a flag value with no declared bits dumped
as `Type::None [0]` in all six textual format names, out of both `Extractor` and
`TraceFlow`. It then justifies the *replacement* rather than just stating it,
since "render it as its decimal value" looks arbitrary until you know the
enumeration libraries already spell an undeclared residue that way -- and that a
decimal cannot collide with a member name where `None` can, `NONE` being a real
declared name elsewhere. Three things a reader would otherwise get wrong are
recorded: three sites carried the interpolation and not one, the guard is on
`name is None` rather than on zero because the defect never was about zero, and
it is not an `aenum` quirk since stdlib `enum.IntFlag` behaves identically.

It also corrects the issue on a point of fact. #648 said `Flags` was the only
registry nameless at zero; a sweep of all seven finds five, the four Mobility
Header flag registries included. And it states that the committed example dumps
do not move, which was measured by regenerating all three with and without the
change rather than assumed -- a reader of a bullet this emphatic will otherwise
wonder whether `examples/captures/` drifted.

The #649 bullet says "cosmetic" in its second sentence so nobody reads it as a
behavioural change, then gives the one reason it was worth doing at all: it is
the text a user sees when an option is rejected. It names all four sites, and the
28-against-4 count in the same file, because that count is what makes the correct
form a fact about the module rather than a preference.

Neither bullet claims a guard it does not have: #648's third site, the `addon`
branch, is not reachable from any registry in the library today, and the bullet
does not imply otherwise.

`CHANGELOG.md` regenerated with `util/changelog_md.py`, not edited by hand.
`--check` exits 0 and `tests/project/` is green at 96 passed, 469 subtests. Each
`` literal is kept on one line, since the generator rejects one spanning a line
break with `ResidualMarkupError`.

Committed from a detached HEAD on d14577d and pushed to the branch ref, because
`docs/changelog-1.5.0` is checked out in another agent's worktree and could not be
taken here. Note d14577d, not the 69a6e13 I was given: the branch had already
moved on with #665's, #651's, #661's and #652/#650's entries.

Refs #648
Refs #649
@JarryShaw

Copy link
Copy Markdown
Owner Author

Cross-review verdict: GOOD TO GO

Independent cross-review, run on a different model (Sonnet) from the one that authored the change, in its own worktree, briefed to falsify rather than bless and to treat the issue's own measurements as claims rather than ground truth. It was read-only with respect to this PR: no amend, no push, no commit to the branch.

Tree it measured: …/.claude/worktrees/agent-a61980e1fa8991520/pcapkit/__init__.py, at commit 1db6232d, with every __editable__-named finder stripped from sys.meta_path, cached pcapkit* modules purged, and the worktree pinned at sys.path[0] on every run. Python 3.14.7.

Per-claim verdicts

claim verdict evidence it obtained
A three sites, all fixed, none missed TRUE render_enum at common.py:168 and its three call sites :258, :267, :270 are the only Type::name [value] rendering in pcapkit/ or tests/. Other .name interpolations (esp.py, extraction.py, pcapng.py key-log writers, fields/*.py) are unrelated messages or a different format.
B nameless renders as its decimal value TRUE Ran the real hook: Flags(0)'Flags::0 [0]', (1)'Flags::1 [1]', (8), (9), (65536) likewise.
C Flags(2049).name == 'ACK|1', Flags(2057).name == 'ACK|9' TRUE Executed directly. The PR's central justification holds.
D no declared identifier is a pure decimal TRUE Reproduced 1867 distinct names from 2068 declaration lines, and swept 10,367 canonical members at runtime. Zero pure decimals by both countings.
E not an aenum quirk TRUE Stdlib enum.IntFlag gives name is None at 0, 8, 65536 identically.
F five registries nameless at zero, not one TRUE — "the PR is right where the issue was wrong" Flags, BindingACKFlag, BindingUpdateFlag, HandoverACKFlag, HandoverInitiateFlag; TransportProtocol and CommandType name their zero.
G fixtures do not move Core claim TRUE; my drift description was an understatement Controlled A/B with only common.py toggled: diff empty for all three files. But the committed-vs-current drift is larger than I said — see below.
H fails on main, passes with the fix TRUE, exact match Pre-fix: 18 failed / 2 passed, exit file 1, including the quoted 'StdFlags::None [65536]' != 'StdFlags::65536 [65536]'. Post-fix: exit file 0.
I the test discriminates a wrong fix TRUE — the important one A zero-only guard ('0' if o.value == 0 else None) → 10 failed, exit 1. An unconditional 'NONE'18 failed / 3 passed, exit 1. Both wrong fixes are caught.
J the edit to test_tcp_udp_unit.py is honest and minimal TRUE Only adds an assertion (assertNotIn('None', rendered)) beside the changed pinned value; nothing weakened or deleted. It independently traced the four MH registries through mh.py and confirmed they are never constructed as IntFlag instances on the read path (MH flags are decomposed into named bits via BitField), so the rewritten claim that a flagless TCP segment is the library's only wire-reachable producer of a nameless member is true.
K the breaking label "reasonable, not dishonest" Wire bytes are untouched, but #634 — the closely analogous predecessor that changed the same connection field's dumped type — also carries fix+breaking, so applying it here is consistent with the repo's own precedent.
L correctness risks not mentioned No new bug str(o.value) is safe: all five currently-reachable nameless members have int values. Exhaustively checked all 10,391 named memberszero differ between the old rendering and render_enum. render_enum outside __all__ under "Internal Definitions" matches 15 other doc pages.

Structural checks also passed: one commit, author Jarry Shaw <jarryshaw@icloud.com>, Fixes #648, only a .rst doc touched (no .md), no GH-nnn.

What it disputed, and what I changed

Two findings, both acted on rather than argued away:

  1. My side-finding on fixture drift undersold it. I had written "a +00:00 timezone suffix and different packet-byte offsets". The reviewer found whole frames whose hex payloads differ in content, protocol -> NIL becoming protocol -> 5001, and packet -> NIL becoming real WireGuard-secrets bytes. I verified this myself before accepting it. The description is corrected in the PR body. The core claim is unaffected — the A/B regeneration isolates this PR's contribution and it is byte-identical — but the drift is substantial and examples/captures/out.* no longer reflect what the library emits. Still deliberately not fixed here; it wants its own change.

  2. A number it could not reconcile. It could not locate or reproduce "15 passed / 25 subtests" against a module-only run, getting 5/17 (module alone) and 6/21 (module plus the amended tcp test). Both figures are right: my 15 was the whole of tests/dumpkit/ — the pre-existing test_common_unit.py plus the new module — and the tcp test. The PR body now spells out the selection so the number is checkable.

  3. It also surfaced a latent, pre-existing oddity: OptionType(0) has a string-typed .value, so it already renders with nested brackets as 'OptionType::opt_endofopt [opt_endofopt [0]]'. That member is named, so render_enum is identical to the old code there. Recorded in the PR body, deliberately not fixed.

Nothing in A–L was left unverified.

@JarryShaw
JarryShaw merged commit 6a33a3c into main Sep 22, 2026
26 checks passed
@JarryShaw
JarryShaw deleted the fix/dumpkit-nameless-enum-648 branch September 22, 2026 22:25
JarryShaw added a commit that referenced this pull request Sep 23, 2026
…idth (#702)

`tests/dumpkit/test_nameless_enum_rendering_unit.py` swept the literal
`NAMELESS_VALUES = (0, 1, 8, 9, 65536)` against `pcapkit.const.tcp.flags.Flags`,
and `65536` is `0x10000` -- one bit past the sixteen-bit field that registry's
`_missing_` bounds itself to. The guard is right, so the probe was what had to
move: a value a registry is correct to refuse cannot also be a value the dumper
is expected to render. `main` has been failing on the `library='aenum'` subtest
since #670 introduced the file, and every PR whose Unit Tests job reached it
failed for reasons that were not its own.

Swept all seven flag registries under `pcapkit/const/` before choosing a
direction, since aborting at the first failure hid the rest:

* All seven bound `_missing_` to their own field, at four distinct widths --
  three bits for `CommandType`, four for `TransportProtocol`, eight for
  `BindingACKFlag`, `HandoverACKFlag` and `HandoverInitiateFlag`, sixteen for
  `BindingUpdateFlag` and `Flags`. Exempting the width-guarded registries would
  therefore have exempted every one of them and left this half of the sweep with
  nothing in it, which is why the probe is derived per registry instead.
* `65536` is out of range for all seven, not only for `Flags`. And `8` and `9`
  are *named* in `BindingACKFlag` and `TransportProtocol` rather than nameless,
  so one fixed literal was never right for more than one registry.
* For all seven the bound is exactly the smallest all-ones mask covering every
  declared bit, which `TransportProtocol` already spells for itself as
  `max(cls.__members__.values()) * 2 - 1`. `_field_mask` derives that, so it
  needs no table to maintain and cannot drift from the guard.

* `_nameless_values` returns the values a registry admits that no member names:
  no bits at all, each undeclared bit alone, and every undeclared bit at once.
  The last takes over `9`'s multi-bit role and is the largest nameless value the
  field holds, which is what the out-of-range literal was reaching for.
* `test_no_flag_registry_renders_the_literal_none` sweeps those values rather
  than `registry(0)` alone. It was broad across registries and one value deep in
  each, so the four Mobility Header registries only ever saw a single value.
* `test_a_value_past_the_field_is_refused_rather_than_rendered` asserts that
  rejection instead of tripping over it, and pins the derivation itself: the
  widest in-field value is accepted and the next one up is not, which holds only
  if the derived mask is the bound each registry declared.

No `pcapkit` line changed. The file goes from `1 failed, 5 passed, 16 subtests
passed` to `6 passed, 64 subtests passed`, and `tests/dumpkit/` is 15 passed.
Coverage rises rather than falls, which deleting the literal on its own would not
have managed: `tcp/flags.py:91` was reached only by this file failing on it, and
the same `raise` in the other six registries was reached by nothing at all. All
seven are now covered -- `ftp/command.py:76`, `mh/binding_ack_flag.py:72`,
`mh/binding_update_flag.py:87`, `mh/handover_ack_flag.py:60`,
`mh/handover_initiate_flag.py:63`, `reg/apptype.py:65` and `tcp/flags.py:91` --
for six fewer missed statements and six fewer partially-covered branches across
those modules. Against the #648 guard reverted the sweep fails 27 subtests and
the scalar test 21; against `Flags`' width widened to `0xFFFFFFFF` the new test
fails on `value=65536`, which is the trade #702 warns against.

Fixes #702
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking Alters public API or wire output (apply alongside the type label) fix Pull requests that fix a defect (fix: subject prefix)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dumpkit's object_hook interpolates o.name unguarded, so a nameless flag member renders as Type::None [0]

1 participant