Re-serialising a parsed HIP PUZZLE or SOLUTION loses leading zero octets: a len=20 parameter rebuilds as len=6

[JarryShaw]

_make_param_puzzle and _make_param_solution recompute the width of Random #I and Puzzle solution #J from int.bit_length(), and the data model carries no width to recompute it from. So a parameter that arrived on the wire with an 8-octet Random #I of 0x0000000000000001 re-serialises into a single octet: the value survives, the field width does not. A SOLUTION parsed with Length = 20 rebuilds as Length = 6.

This matters because the field width is not decoration in this parameter. RHASH_len is the output length of the puzzle hash, and both fields are defined as exactly that wide — RFC 7401 §5.2.5 gives Random #I as a "random number of size RHASH_len bits" and Length as 4 + RHASH_len / 4. Narrowing the field changes the implied RHASH_len, so the rebuilt parameter describes a different puzzle from the one that was received.

The sites

pcapkit/protocols/internet/hip.py:3195-3196, in _make_param_solution's param is not None branch — the path taken when re-serialising something that was parsed:

random = param.random
solution = param.solution

pcapkit/protocols/internet/hip.py:3212, sizing from those ints and nothing else:

len=4 + 2 * math.ceil(max(random.bit_length(), solution.bit_length()) / 8),

The sibling _make_param_puzzle has the same shape at :3152 and :3160:

len=4 + math.ceil(random.bit_length() / 8),

There is nothing wrong with bit_length() as such — the problem is that it is the only source of width available. pcapkit/protocols/data/internet/hip.py:196 and :213 define the data models, and neither carries a width field:

Data_SolutionParameter fields = ['type', 'critical', 'length', 'index', 'lifetime', 'opaque', 'random', 'solution']

length is the record length, not the field width, and it is not consulted by either builder. The reader discards the width on the way in: pcapkit/protocols/schema/internet/hip.py:454 and :456 size the two fields at (pkt['len'] - 4) // 2 each and hand back plain ints, so by the time _read_param_solution builds the data object the octet count is gone.

Measured

On origin/main (a62aed134), CPython 3.14.7, importing from an immutable git archive snapshot with pcapkit.__file__ asserted against it:

MEASURED pcapkit.__file__ = /tmp/hipissues/tA62/pcapkit/__init__.py
CPython 3.14.7

A HIPv2 SOLUTION was built with full-width 64-bit fields so that Length is 20, then the two 8-octet payload fields were overwritten with leading-zero values of the same width. Length stays 20; only the contents change. Parsed, then rebuilt through HIP.make. The parameter octets, before:

01 41  00 14  00  20  00 00  00 00 00 00 00 00 00 01  00 00 00 00 00 00 00 01  00 00 00 00
type   len=20 #K  Res opaque Random #I (8 octets)     solution #J (8 octets)   padding

and after:

01 41  00 06  00  20  00 00  01  01  00 00
type   len=6  #K  Res opaque R#I sol padding

parsed  : random = 1, solution = 1
rebuilt : wire Length field  before: 20   after: 6
          parameter octets   before: 56   after: 24    (two copies per packet)
          bytes identical across the round trip? False
re-parsed: random = 1, solution = 1

The integers survive; the eight-octet fields become one octet each. A conformant peer reading the rebuilt parameter sees RHASH_len = 8 bits where the original said 64.

It reproduces on 375e9d411 too, but masked

On 375e9d411 — a62aed134's parent, before #629 landed — the packet-level rebuild does not get far enough to show the width loss:

rebuild raised pcapkit.utilities.exceptions.ProtocolError: HIPv2: [ParamNo 321] invalid format
  builder called with param=<parsed> -> schema.len = 5 (wire Length was 20)

That ProtocolError is #608: with the old ceil(bits / 4) formula the rebuild produced len = 5, and (5 - 4) % 2 != 0 tripped the reader's parity guard first. Calling the builder directly shows the width loss is there regardless — 20 down to 5 rather than 20 down to 6. So this defect predates #629 and was hidden behind #608's louder failure; fixing #608 is what made it observable end to end. Worth stating plainly because it inverts the usual reading: the round trip now succeeds and silently produces a different parameter, where before it failed noisily.

Why nothing caught it

The round-trip suite builds every parameter from keyword arguments and never from octets with leading zeros. examples/generators/options.py:975-977 passes only {'lifetime': 1} for both PUZZLE and SOLUTION, so random and solution default to 0 — the one value for which no width can be lost, because there is no width. The parse-then-rebuild path is exercised, and the comparison is pcapkit against pcapkit, so a width that both sides narrow identically compares equal.

Catching it needs a parameter whose octets came from somewhere other than this builder, carrying a value narrower than its field — which no HIP fixture has, because every HIP fixture was generated by the code under test. #608's own write-up records the same gap in two neighbouring cases it did verify, #591's repair-path bit lengths and #601's nonce: 0xFFFFFF; I have not re-checked those two here, so that part is its claim rather than mine.

Reachability

Reachable from ordinary conformant input, and not only from a crafted one. RFC 7401 §2.3 defines RHASH_len as "the natural output length of RHASH in bits", where RHASH is the hash of the Responder's HIT Suite; §5.2.10 makes RSA,DSA/SHA-256 the REQUIRED suite, so on the mandatory path RHASH_len is 256 bits and Random #I is a 32-octet field. Any value with a zero top octet — one in 256, for a random number — loses at least one octet on re-serialisation, and a value with two zero top octets loses two. A pcapkit-based tool that parses a HIP base exchange and re-emits it will corrupt roughly that fraction of SOLUTION and PUZZLE parameters, silently.

What is not established is whether anything in this repository re-serialises HIP parameters outside the test suite. The defect is on a public construction path (HIP.make with a parsed param=), so a consumer can reach it, but I have not traced an internal caller.

Notes

• Found by the worker fixing HIP SOLUTION builder sizes with ceil(bits/4), emitting a parameter its own reader rejects #608 while measuring that defect, reported in PR fix(hip): size the SOLUTION parameter as two whole-octet fields (#608) #629's body, and deliberately not folded in — correctly, since the fix is a data-model change rather than an arithmetic one.
• The fix means carrying the on-wire field width through Data_PuzzleParameter and Data_SolutionParameter and having both builders prefer it over bit_length(), falling back to bit_length() only when constructing from scratch. That is a change to the public data model, which is why it is not a one-line repair.
• Related: HIP SOLUTION builder sizes with ceil(bits/4), emitting a parameter its own reader rejects #608 and PR fix(hip): size the SOLUTION parameter as two whole-octet fields (#608) #629, where this was found and where the masking described above was removed. The sibling side-findings from the same work are HIP parameter padding aligns the contents, not the record, so every parameter pcapkit emits is 4 (mod 8) octets #651 (padding aligns the contents, not the record), SOLUTION's Reserved octet is written as a PUZZLE Lifetime, and lifetime=0 escapes a bare ValueError from math.log2 #654 (SOLUTION's Reserved octet written as a Lifetime) and Neither the HIP PUZZLE nor SOLUTION builder consults version when sizing, so HIPv1 rejects what it builds #655 (neither builder consults version).
• Closest to Neither the HIP PUZZLE nor SOLUTION builder consults version when sizing, so HIPv1 rejects what it builds #655, and deliberately filed apart from it: both come from width being derived rather than declared, but sizing to a fixed 8 octets under HIPv1 — Neither the HIP PUZZLE nor SOLUTION builder consults version when sizing, so HIPv1 rejects what it builds #655's fix — would not help HIPv2, where RHASH_len genuinely varies with the Responder's HIT Suite. Conversely, carrying the width through the data model fixes this one and does nothing for a caller who constructs from scratch with version=1.