Neither the HIP PUZZLE nor SOLUTION builder consults version when sizing, so HIPv1 rejects what it builds

[JarryShaw]

_make_param_puzzle and _make_param_solution both take a version keyword and neither reads it when sizing the parameter. Both derive Length from int.bit_length() alone. Under HIPv1 that is not allowed to vary: RFC 5201 fixes Random #I and Puzzle solution #J at 8 octets each and therefore fixes Length at 12 for PUZZLE and 20 for SOLUTION. So any value narrower than the full field builds a parameter that this library's own reader then rejects — the same shape as #608, in a second guise, and present in the PUZZLE builder that #629 deliberately did not touch.

The RFC text, fetched

Fetched from https://www.rfc-editor.org/rfc/rfc5201.txt (240,492 octets, sha256 8b42d181a8e239713eb8d608c11e8e75829561d994adbd3caa96c6db6e69cef6). Unlike RFC 7401, HIPv1 states both widths as literal constants. RFC 5201 §5.2.4, PUZZLE, verbatim:

      |                      Random #I, 8 bytes                       |
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      Type           257
      Length         12

and RFC 5201 §5.2.5, SOLUTION:

      |                      Random #I, 8 bytes                       |
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                 Puzzle solution #J, 8 bytes                   |
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      Type               321
      Length             20

"Random #I and Random #J are represented as 64-bit integers" (§5.2.5) and "Random #I is represented as a 64-bit integer" (§5.2.4) leave no room for a narrower field: Length is the literal 12 and 20, not a formula. RFC 7401 §5.2.4/§5.2.5 is where the widths become variable, at RHASH_len / 8 octets each — so bit_length() is wrong under HIPv1 specifically, and merely unreliable under HIPv2. (Note SOLUTION is §5.2.5 and PUZZLE is §5.2.4 in both RFCs.)

The sites

pcapkit/protocols/internet/hip.py:3160, in _make_param_puzzle:

len=4 + math.ceil(random.bit_length() / 8),

pcapkit/protocols/internet/hip.py:3212, in _make_param_solution as #629 left it:

len=4 + 2 * math.ceil(max(random.bit_length(), solution.bit_length()) / 8),

Both functions are declared with version: 'int' in their signature — :3126 and :3168 — and in both cases the parameter is consumed only by the # pylint: disable=unused-argument on the def line. The readers, by contrast, enforce the version's constraint. pcapkit/protocols/internet/hip.py:1028-1029 for PUZZLE:

if version == 1 and schema.len != 12:
    raise ProtocolError(f'HIPv{version}: [ParamNo {schema.type}] invalid format')

and :1081-1082 for SOLUTION:

if version == 1 and schema.len != 20:
    raise ProtocolError(f'HIPv{version}: [ParamNo {schema.type}] invalid format')

The asymmetry is the defect: one side of the library knows HIPv1 has a fixed width and the other does not.

Measured

On origin/main (a62aed134), CPython 3.14.7, importing from an immutable git archive snapshot with pcapkit.__file__ asserted against it. The builders were called directly, so the Length they compute is visible rather than hidden behind the reader's rejection:

MEASURED pcapkit.__file__ = /tmp/hipissues/tA62/pcapkit/__init__.py
CPython 3.14.7

  bits | SOLUTION len (v1) | SOLUTION len (v2) | PUZZLE len (v1) | PUZZLE len (v2)
     1 |          6        |          6        |         5       |         5
     8 |          6        |          6        |         5       |         5
    32 |         12        |         12        |         8       |         8
    56 |         18        |         18        |        11       |        11
    57 |         20        |         20        |        12       |        12
    64 |         20        |         20        |        12       |        12
   128 |         36        |         36        |        20       |        20

The v1 and v2 columns are identical at every width. That is the finding stated as directly as it can be: passing version=1 changes nothing about the size. SOLUTION needs Length = 20 under HIPv1 and only reaches it from 57 bits up; PUZZLE needs 12 and only reaches it from 57 bits up. Every narrower value is wrong, and 128 bits overshoots in both — a perfectly ordinary 16-octet value builds Length = 36 for SOLUTION where HIPv1 permits only 20.

Through the public construction path, the rejection is immediate, because HIP's construction packs and then re-reads on the same instance:

  SOLUTION under version=1, want len=20:
    bits=1   -> build raised ProtocolError: HIPv1: [ParamNo 321] invalid format
    bits=8   -> build raised ProtocolError: HIPv1: [ParamNo 321] invalid format
    bits=32  -> build raised ProtocolError: HIPv1: [ParamNo 321] invalid format
    bits=56  -> build raised ProtocolError: HIPv1: [ParamNo 321] invalid format
    bits=57  -> len=20  accepted
    bits=64  -> len=20  accepted

  PUZZLE under version=1, want len=12:
    bits=1   -> build raised ProtocolError: HIPv1: [ParamNo 257] invalid format
    bits=8   -> build raised ProtocolError: HIPv1: [ParamNo 257] invalid format
    bits=32  -> build raised ProtocolError: HIPv1: [ParamNo 257] invalid format
    bits=56  -> build raised ProtocolError: HIPv1: [ParamNo 257] invalid format
    bits=57  -> len=12  accepted
    bits=64  -> len=12  accepted

So under HIPv1 each builder accepts only a narrow window of inputs: a Random #I whose bit_length() falls in 57..64, and nothing else. Below the window the parameter is undersized, above it — a 16-octet value, say — it overshoots to Length = 36 where HIPv1 permits 20.

How often the window is missed depends on where the value comes from. For a uniformly random 8-octet Random #I, bit_length() >= 57 holds unless the top octet is zero, so about one in 256 conformant HIPv1 puzzles cannot be constructed — infrequent, but silent until it happens and indistinguishable from a malformed packet when it does. For anything that is not uniformly random the rate is far worse: a small literal, a counter, a value from a fixture or a test all fail outright, which is exactly the population a library's own callers draw from.

Also on 375e9d411, and unchanged in PUZZLE

Same probe against 375e9d411, a62aed134's parent, before #629:

  bits | SOLUTION len (v1) | SOLUTION len (v2) | PUZZLE len (v1) | PUZZLE len (v2)
     1 |          5        |          5        |         5       |         5
     8 |          6        |          6        |         5       |         5
    32 |         12        |         12        |         8       |         8
    56 |         18        |         18        |        11       |        11
    57 |         19        |         19        |        12       |        12
    64 |         20        |         20        |        12       |        12
   128 |         36        |         36        |        20       |        20

The v1/v2 columns are identical here too. #629 moved exactly two SOLUTION rows — bits=1 from 5 to 6 and bits=57 from 19 to 20 — and left every other row, and the whole PUZZLE builder, as it was. The PUZZLE column is byte-for-byte identical between the two commits, which is the clean demonstration that this is not a SOLUTION defect or a #629 regression: it is the same omission in two builders, one of which #629 never opened.

Why this is not just #608 again

#608 was an arithmetic error — ceil(b/4) used as shorthand for two fields of ceil(b/8) — and #629 fixed it so that the width is now correct for a value of that bit length. This is a different question: whether bit_length() is the right input at all when the protocol version fixes the field width. It is not, and no correction to the formula helps, because the formula's input is the wrong quantity. bits=32 builds Length = 12 both before and after #629, and HIPv1 wants 20 either way.

It is also distinct from the sibling finding that re-serialising a parsed parameter loses leading zero octets, though the two share a cause in width being derived rather than declared. Sizing to a fixed 8 octets under HIPv1 would fix both for HIPv1, and neither would be fixed for HIPv2, where RHASH_len genuinely varies with the Responder's HIT Suite (RFC 7401 §2.3 and §5.2.10) and so the width really does have to come from somewhere other than the version.

Why nothing caught it

There is no HIPv1 PUZZLE or SOLUTION fixture with a narrow value. examples/generators/options.py:975-977 overrides both parameters with {'lifetime': 1} and nothing else, so random and solution stay at their 0 defaults and the HIPv1 codes are the ones HIP_VERSION = {129: 2, 128: 1} at :947 routes elsewhere. #629 added a test at 57 bits that pins the single HIPv1 row that change moved, and its body says explicitly that it "does not claim general HIPv1 correctness" — so the gap is known and recorded, not newly discovered here.

Notes

• Raised by PR fix(hip): size the SOLUTION parameter as two whole-octet fields (#608) #629's cross-reviewer, reproduced by that PR's author on both trees, and recorded in the PR body as the fourth of four findings left deliberately unfixed. Filing it so it is not rediscovered at full cost.
• Re-verified here against a62aed134 and 375e9d411 before filing, by calling both builders directly with version=1 and version=2 and comparing the computed Length — which is what makes the "identical columns" claim a measurement rather than an inference from reading the source.
• The fix is for both builders to size to 8 octets per field when version == 1, and to keep the bit_length() path for HIPv2. That is a behaviour change in two builders, one of them untouched by fix(hip): size the SOLUTION parameter as two whole-octet fields (#608) #629, which is why it was not ridden along with a one-line length repair.
• Related: HIP SOLUTION builder sizes with ceil(bits/4), emitting a parameter its own reader rejects #608 and PR fix(hip): size the SOLUTION parameter as two whole-octet fields (#608) #629 (the SOLUTION len arithmetic, and where this was found). The sibling side-findings from the same work are HIP parameter padding aligns the contents, not the record, so every parameter pcapkit emits is 4 (mod 8) octets #651 (padding aligns the contents, not the record), Re-serialising a parsed HIP PUZZLE or SOLUTION loses leading zero octets: a len=20 parameter rebuilds as len=6 #653 (re-serialising loses leading zero octets) and SOLUTION's Reserved octet is written as a PUZZLE Lifetime, and lifetime=0 escapes a bare ValueError from math.log2 #654 (SOLUTION's Reserved octet written as a Lifetime).
• Closest to Re-serialising a parsed HIP PUZZLE or SOLUTION loses leading zero octets: a len=20 parameter rebuilds as len=6 #653, and deliberately filed apart from it: both come from width being derived from the value rather than declared. Sizing to a fixed 8 octets under HIPv1 fixes this one and leaves Re-serialising a parsed HIP PUZZLE or SOLUTION loses leading zero octets: a len=20 parameter rebuilds as len=6 #653 open for HIPv2; carrying the on-wire width through the data model fixes Re-serialising a parsed HIP PUZZLE or SOLUTION loses leading zero octets: a len=20 parameter rebuilds as len=6 #653 and leaves this one open for a caller constructing from scratch.