NumberField with a callable length cannot pack at 8 octets: _need_process latches True from the placeholder and is never cleared

[JarryShaw]

A NumberField whose length is a callable is unpackable once that callable resolves to 8 octets: _need_process is latched True while the field is still a placeholder and is never cleared when the real length arrives, so pre_process returns bytes to a struct template that now expects an integer.

Mechanism

numbers.py:109 sets the flag false in __init__, and the template builder sets it true for any length that is not 8, 4, 2 or 1:

else:                 # do not unpack
    struct_fmt = f'{length}s'
    self._need_process = True

A callable length is a placeholder of -1 at construction, which takes that branch. When __call__ later resolves the real width the template is rebuilt correctly — but nothing resets _need_process. So pre_process at numbers.py:194 takes the process path, self._length is now 8 rather than negative so the _length < 0 repair at :196-203 is skipped, and :205 returns

return value.to_bytes(self._length, self._byteorder, signed=self._signed)

bytes, into a template that is by then >Q.

Measured

Executed on origin/main (9c240a60e), CPython 3.14.7. The comparison is the proof — identical resolved length and identical template, differing only in how the length was supplied:

--- callable length (resolves to 8) ---
length= 8  _need_process= True   template= >Q
  error: required argument is not an integer

--- static length 8, for comparison ---
length= 8  _need_process= False  template= >Q
pack -> 0000010000000000

So NumberField(length=lambda pkt: 8) cannot pack, while NumberField(length=8) packs the same value through the same template.

Impact on the wire

This makes every extended (8-octet) MPTCP DSS form unpackable, since those widths are selected at runtime from the DSS flags and therefore must come from a callable. Reported as:

_make_mptcp_dss(DSS, ack=1 << 40)  ->  struct.error: required argument is not an integer

Any other schema that resolves a numeric width to 8 octets through a callable has the same failure.

Notes

• Found by the _make_mptcp_*/_read_mptcp_* length arithmetic is wrong at six more sites beyond MP_CAPABLE #576 worker while fixing MPTCP length arithmetic. pcapkit/corekit/fields/numbers.py was outside that change's scope, so fix(tcp): correct MPTCP option length arithmetic at all six #576 sites #585 works around it with a SwitchField over UInt32Field/UInt64Field rather than touching this file. The workaround leaves this defect live for every other caller, which is why it needs its own fix.
• The obvious fix is to recompute _need_process when the resolved length is applied, rather than only at construction. Whether 1/2/4 are equally affected is not something I checked — the reproduction above is the 8-octet case only, and build_template takes the same else branch for any width outside {1, 2, 4, 8}, so a callable resolving to, say, 3 would legitimately still need processing. A fix must distinguish "placeholder" from "genuinely needs byte packing".
• Related but distinct: A per-field padding ceiling does not bound their sum: ~10,900x amplification survives the #554 fix #573 concerns a padding ceiling in pcapkit/corekit/fields/field.py. This is a separate defect in a sibling module.

[JarryShaw]

Scope correction to this issue, established by the fix in #598 and verified here.

This issue understated the defect in two ways.

1. All four native widths were affected, not only 8. The body said whether 1/2/4 are equally affected "is NOT something I checked". They are: a callable length resolving to 1, 2, 4 or 8 each latched _need_process = True against templates >B, >H, >I and >Q respectively, and each raised required argument is not an integer. Width 3 correctly packed throughout, because a 3-octet field genuinely does need byte processing. The latch has nothing to do with the width it latches into — which is obvious in hindsight and is why the 8-octet DSS case was simply the one that got noticed.

EnumField was affected identically, since it also leaves __template__ unset. The eight subclasses that fix __template__ never latched anything, because __init__ skips build_template for them.

2. Parsing was broken in the mirror direction, not just packing. The title says "cannot pack". The same flag made post_process (numbers.py:220-224) call int.from_bytes on the integer struct.unpack had already produced. So the field could not read what it could not write.

On the fix's shape, since this issue speculated about needing to distinguish "placeholder" from "genuinely needs processing": #598 does not add such a bit. It makes build_template assign _need_process rather than only ever raise it, so the flag becomes a function of the width currently in force — True for the -1 placeholder, False for 8, True again for 3. The placeholder's True was never wrong; it was only sticky.

One separate defect was found on the same path and is filed independently, because #598 neither fixes nor masks it: pre_process's repair at numbers.py:198 uses math.ceil(value.bit_length() // 8), which is floor division, so every octet boundary is sized one octet short and raises OverflowError.