fix(corekit): bound FieldBase.unpack's rjust() padding to a sane ceiling (#554) - #569
Conversation
|
✅ GOOD TO MERGE Cross-model review (Opus 5; the PR was authored on Sonnet), head The bound is genuinely derived, and I verified both halves of the derivation I could reach. The two-condition rule and its boundary, measured myself (nothing above 256 KiB materialised):
#431 is preserved structurally, not incidentally, and I confirmed it against The tests do fail a nearly-right bound — I applied the mutation myself rather than taking the claim: So the standard is met. But it is worth saying plainly where the protection lives: that one test is load-bearing. The other ten read
Three non-blocking observations, the first worth a look before merge:
Green: Detail and provenance of each measurement below. |
Detailed cross-review — #569 @
|
| declared length | available | outcome |
|---|---|---|
| 1 | 0 | ACCEPTED |
| 2 | 1 | ACCEPTED |
| 262143 | 0 | ACCEPTED |
| 262144 | 0 | ACCEPTED — a 256 KiB zero-pad from an empty buffer |
| 262144 | 262144 | ACCEPTED |
| 262145 | 0 | REJECTED |
| 262145 | 262144 | REJECTED |
| 262145 | 262145 | ACCEPTED |
Two readings worth stating. First, the guard is not merely a magnitude check — row 7 shows a one-octet shortfall is still refused once the declared length is past the ceiling, so it is genuinely the conjunction it claims to be. Second, the largest single-field zero-pad still permitted is 256 KiB from an empty buffer (row 4).
My view on whether row 4 is a gap or a limit: a deliberate, documented limit. The docstring states the trade-off and names #431 as the cause, and #431's reads are 1–2 octet fixed-width fields — five orders of magnitude clear of the ceiling. A tighter number would not help; what would is a different shape of rule (per-field-kind, or the packet-context discrimination), and choosing that shape is a design question rather than a defect in this one. Whether many such fields can aggregate is a property of the option/list loop bounds, not of this guard, and is outside its remit.
4. #431 preserved — verified on both trees, identical output
I ran the same probe on main 8cfd6ab01 and on fef728bc1. Byte-identical results:
UInt8Field().unpack(b'') -> 0
UInt16Field().unpack(b'') -> 0
UInt16Field().unpack(b'\x01') -> 1
IPv4(bytes.fromhex('4a00001800010000400600000a0000010a000002'))
-> PARSED OK; options = [(0, 'EOOLOption')]
Preserved structurally: each of those is a one- or two-octet fixed-width read, so length > 262144 is false and the guard is unreachable on that path. That is a stronger guarantee than "the tests still pass", which is what I was asked to establish.
The parallel run additionally reproduced the TCP reproduction named in OptionField.unpack's own NOTE (data offset 10, four option octets → [(1, 'NoOperation'), (0, 'EndOfOptionList')]) identically on both trees. I did not re-run that one.
5. Mutation testing — the standard is met, with one caveat
Measured by me:
| mutation | rc (from file) | caught by |
|---|---|---|
0x40_000 → 0x40_001 |
1 | test_the_ceiling_is_262144_octets only (10 others passed) |
From the parallel run, not measured by me:
| mutation | rc | caught by |
|---|---|---|
> → >= |
1 | test_field_at_the_ceiling_still_zero_pads |
0x40_000 → 0x3F_FFF |
1 | test_the_ceiling_is_262144_octets only |
| guard removed entirely | 1 | test_field_one_past_the_ceiling_with_insufficient_buffer_is_rejected, test_rejection_message_names_the_declared_and_available_counts |
So a nearly-right bound is failed, not merely an absent one. The caveat: the wrong-constant protection is a single point of failure. Ten of the eleven tests read self.ceiling back from the module in setUp, so they move with the mutation and cannot detect it; only the one test that hard-codes 262144 can. The author saw this coming — that test's docstring says "A test that only ever reads the constant back from the module would still pass if that figure were quietly changed" — which is why I am flagging it as something to protect rather than as a defect. If that test is ever "tidied up" to use the constant, the suite silently stops guarding the number.
I restored field.py from a byte-identical copy after the mutation and proved it: md5 f2e28a7d489ee08ba80b2902022f47b9, git status empty, and the file back to 11 passed / exit 0.
Also worth crediting: the body's fails-without proof used a monkeypatched surrogate rather than a plain revert, and that was justified rather than lazy — a plain revert of field.py removes _MAX_ZERO_PAD_LENGTH, which setUp reads, so all eleven tests would error in setUp and prove nothing about the guard.
6. Non-blocking observations
6.1 The check is after the read. buffer.read(length) at :271, guard at :285. On the file-backed path an unvalidated length reaches read() before anything can reject it; the guard then only constrains rjust(). read(n) returns at most what is available, so no data is fabricated — and I want to be clear I did not measure whether CPython's buffered read(n) pre-sizes an n-octet destination, so I am not claiming a live problem. Only that hoisting the check above the read is strictly stronger, costs nothing and changes no behaviour.
6.2 Two wordings promise more than they deliver. The body says both mutants "were caught by the boundary tests" — measurably not so for the constant mutants (§5). And test_..._rejected_without_allocating asserts rejection inside a 5 s signal.alarm deadline, not non-allocation; signals land between bytecodes, so it could not interrupt one large C-level rjust() mid-flight even if the guard were gone. Its docstring is honest; the name is what over-promises. The real protection there is the assertRaises(FieldValueError).
6.3 test_rejection_message_names_the_declared_and_available_counts asserts assertIn('1', message) for the available count. '1' occurs inside 262145, so that assertion holds even if the available count were dropped from the message entirely. Asserting the full rendered substring would pin it.
6.4 Cosmetic. The message uses {self.name!r} where collections.py's sibling length errors use {self.name}. Mine rendered as Field '<bytes>' declares ….
7. Tests and changelog
pytest tests/corekit/test_fields_field.py→ exit 0, 11 passed. Measured by me, after restoration.- Full tier
pytest tests→ exit 0,1271 passed, 17 skipped, 2850 subtests passed. From the parallel run; it matches the body exactly and is consistent with baselines I measured myself in this batch (mainat 1260 passed / 2850 subtests — I measured 1261/2850 on fix(corekit): stop a malformed TCP SACK's exception type depending on sys.modules state (#525) #562's branch, which ismainplus one new test, and 1279/2850 on feat(protocols): opt-in code= registration with enum-type inference for Protocol/ProtocolBase #570's, which ismainplus 19). 1260 + 11 new tests = 1271, and the subtest count is unchanged because the new tests add none. Both line up. python util/changelog_md.py --check→ exit 0, in step. The entry names the ceiling, its dual provenance, and explicitly records that OptionField.unpack never returns for a well-formed HOPOPT header with an SMF_DPD option #431's tolerance is untouched — which, having verified that tolerance on both trees, I can say is accurate.
Could not verify
- libpcap's
MAXIMUM_SNAPLEN == 262144from any local source. No libpcap headers on this host (/usr/include/pcap*absent, no bindings in the venv). It matches my knowledge ofpcap-int.h, but that is knowledge, not measurement — so of the two halves of the derivation, the internal one (header.py:190) is measured and the external one is not. Worth a maintainer's eye if that matters to you. - The resource magnitude behind the issue — deliberately not measured by either run. Arithmetically the expressible ceiling is clear enough (
DecryptionSecretsBlock.secrets_lengthis aUInt32Field, so up to ~4 GiB is declarable), but I am not putting a figure on peak cost. - Whether CPython's buffered
read(n)pre-allocates (§6.1). - Any interpreter other than CPython 3.14.7.
- CI. Reached on local evidence only, per my brief; no claim about its tally.
- The
>=and0x3F_FFFmutations, the guard-removal run, the TCP#431reproduction, and the full-tier numbers were measured by a delegated Opus 5 run rather than by me. I re-derived the0x40_001mutation and the#431field-level behaviour independently and they agreed, which is why I am reporting the rest.
One cross-PR note, out of scope: this test file's setUp adds eleven more purge_modules(['pcapkit']) calls — the same sys.modules eviction #562's SchemaField hoist defends against, and which is still latent in ListField.pack at collections.py:105 (where it cannot be hoisted, since schema.py imports collections.py). Harmless here; worth knowing while both are in flight.
…ing (#554) length is frequently wire-derived -- resolved by a Field's _length_callback against the packet under parse, or built by a schema selector from a value it just read off the wire (e.g. DecryptionSecretsBlock's secrets_data field) -- and so attacker- or corruption-controlled. buffer[:length].rjust(length, b'\x00') zero-padded straight up to that length regardless of how little data buffer actually held, so a ~40-octet PCAP-NG Decryption Secrets Block with a bogus inner length could force a multi-gigabyte allocation. - Raise FieldValueError when a declared length exceeds both the buffer's actual size and a 262144-octet (0x40_000) ceiling -- libpcap's own MAXIMUM_SNAPLEN, and this package's own default snaplen -- before the rjust() that would otherwise allocate and zero-fill on the packet's own say-so. FieldValueError, not BoolError ("must be a bool"), matches the precedent for an invalid/insufficient length in ListField.unpack. - The ceiling is deliberate rather than "any shortfall is bogus": ListField.unpack and OptionField.unpack (collections.py) depend on a short, sometimes empty, tail read past a truncated option area decoding as zero, so an over-long ihl or a capture cut short by its snapshot length reads as end-of-option-list/Pad1 instead of raising (#431). An unconditional reject broke 15 existing tests exercising that mechanism; every one of those fields is a handful of octets, far under the ceiling, and is untouched. - Add tests/corekit/test_fields_field.py: small-field short reads still zero-pad, the ceiling boundary in both directions, a full buffer past the ceiling is accepted, and a 16 GiB declared length against a 2-octet buffer is rejected under a 5s deadline without allocating. Verified against three mutants (loosened comparison, shifted ceiling operator, wrong constant) to confirm the bound is exact. - Update docs/source/changelog/1.5.0.rst and regenerate CHANGELOG.md. Build: pytest tests -- 1271 passed, 17 skipped, 2850 subtests passed, 0 failed. Spell the error message's field name plain rather than quoted, matching the 8/8 precedent across `pcapkit/corekit/` including `field.py:207` five lines away. Adopted from @lux-liang's #571, which proposed the same correction.
fef728b to
f206ba4
Compare
|
Amended to |
|
✅ GOOD TO MERGE — re-pointed to Cross-model review (Opus 5; PR authored on Sonnet). My earlier verdict was posted against Re-verified at this head rather than carried over:
Two non-blocking notes in the detailed comment below: one stale line citation in the amend note, and one nearly-vacuous assertion in the message test. |
Detailed cross-review — #569 @
|
| claim | measured |
|---|---|
_MAX_ZERO_PAD_LENGTH == 0x40_000 |
262144, True |
Header.make's snaplen default equals the ceiling |
262144, True |
libpcap's MAXIMUM_SNAPLEN is 262144 |
could not verify — external to this repo, no libpcap headers on this host |
That last one I am explicitly not vouching for. It is the load-bearing justification for the constant's value, and it is the one claim in the PR I could not independently obtain. It is consistent with my recollection of pcap-int.h, but recollection is not evidence and I am not presenting it as such.
Guard boundary, derived by direct BytesField.unpack calls rather than read off the diff:
length == ceiling, 1-octet buffer -> pads, returns 262144 octets
length == ceiling + 1, 1-octet buffer -> FieldValueError: Field <bytes> declares a
length of 262145 octet(s), but only 1 octet(s)
are available.
length == ceiling + 10, full buffer -> ok, 262154 octets (guard fires on the padding,
not on size alone — as the PR claims)
length 10, 6-octet buffer -> pads to 10 (#431 mechanism intact)
length 4, empty buffer -> pads to 4 (#431 mechanism intact)
The #431 truncated-data path, the thing #572 exists to protect and the thing #571 broke. Constructed independently from the issue text, run on main (8cfd6ab01) and on this head:
declared=12 -> parses, options=[('79', 12)]
data=b'\x00\x00\x00\x00\x01\x02\x03\x04\x05\x06'
declared=32 -> parses, options=[('79', 32)]
data=b'\x00'*24 + b'\x01\x02\x03\x04\x05\x06'
Byte-identical on both trees. This is the strongest thing I can say for the PR and it is the clean discriminator against #571: the guard's length > 262144 conjunct means a 10- or 30-octet field never reaches it.
Worth recording precisely because it is easy to misread: the data comes back left-padded with zeros to the declared width, not as the six octets actually present. Anyone writing the #572 test should assert the padded value, not b'\x01\x02\x03\x04\x05\x06'.
Test runs (exit codes from files):
| run | exit | result |
|---|---|---|
pytest tests/corekit/test_fields_field.py |
0 |
11 passed |
pytest tests/corekit tests/protocols/internet/test_ipv4_unit.py -k "option or truncat or length or unpack or field" |
0 |
93 passed, 144 subtests, 38 deselected |
pytest tests (full, plain — no coverage instrumentation) |
0 |
1271 passed, 17 skipped, 2850 subtests passed, 1189 s |
util/changelog_md.py --check |
0 |
CHANGELOG.md in step with docs/source/changelog/1.5.0.rst |
The full tier reproduces the body's claimed tally exactly. The targeted run includes IPv4UnitTests::test_an_option_area_longer_than_the_datagram_still_parses, the existing #431 guard test.
One design observation, not a defect
The bound caps amplification rather than removing it: length == 262144 against a 1-octet buffer still allocates and zero-fills 256 KiB, a ~262,000× amplification, and it does so per field. That is three-plus orders of magnitude better than the multi-gigabyte case #554 reports and is the deliberate consequence of scoping the guard to preserve #431, so I am not asking for a change — the alternative is the unconditional guard that broke 15 tests. Recording it so nobody later reads this PR as having closed the amplification question entirely.
CI
Filtered to __typename == "CheckRun" (the pyup.io/safety-ci StatusContext excluded, as it is not a check run):
SUCCESS 9 | SKIPPED 4 | CANCELLED 21 | pending 13 | FAILURE 0
The 21 CANCELLED are the superseded fef728bc1 run, not failures. This is not the 22 ok / 0 fail / 2 skip clean state the rest of the board sits at — 13 checks were still in flight when I looked, and per standing practice I did not wait on Actions, since the ruleset grants an admin bypass and waiting costs ten minutes per round. Reporting the tally as observed rather than as clean.
Verdict
GOOD TO MERGE at f206ba428. The delta is one line in a message string, it moves the line onto the convention every other site in pcapkit/corekit/ already follows, nothing asserts the old form, the full tier is green with the exit code read from a file, and the #431 path I care most about is byte-identical to main.
Claims I could not verify: libpcap's MAXIMUM_SNAPLEN value (external). Claims I disputed: the field.py:207 citation in the amend note is wrong — the precedent is field.py:231.
|
Correction to my amend note above, and to the commit message which is now merged as I wrote that the plain- The substantive claim still holds: plain is the house convention, 9 Provenance, since it matters more than the error: I took that line number from an earlier analysis rather than checking it, and repeated it in a commit message and a public comment. Caught by the cross-review, which enumerated all nine sites rather than accepting the count. |
Summary
FieldBase.unpackzero-padded straight up to a field's declaredlengthwithrjust(), regardless of how little data the buffer actually held.lengthis frequently wire-derived -- resolved by aField's_length_callbackagainst the packet under parse, or built by a schema selector from a value it just read off the wire (e.g.DecryptionSecretsBlock'ssecrets_data: BytesField(length=lambda pkt: pkt['__length__'])) -- so a corrupt or hostile capture could declare an arbitrarily large one. A ~40-octet PCAP-NG Decryption Secrets Block with a bogus inner length was enough to force a multi-gigabyte allocation.The bound, and where it comes from.
pcapkit/corekit/fields/field.pynow raisesFieldValueErrorwhen a declared length exceeds both (a) the buffer's actual size and (b) a 262144-octet (0x40_000) ceiling. That ceiling is not picked "by feel": it is libpcap's ownMAXIMUM_SNAPLEN-- the point past which libpcap itself treats a capture's declared snapshot length as corrupt or byte-order-swapped -- and it is the exact figure this package's own PCAP writer already defaultssnaplento (Header.make'ssnaplen: int = 0x40_000).FieldValueError(notBoolError, which means "must be a bool") is used because it already is this codebase's precedent for an invalid/insufficient field length -- seeListField.unpack'sraise FieldValueError(f'Field {self.name} has invalid length.')incollections.py.Why not "any declared length beyond what the buffer holds", full stop. That was my first attempt, and it is closer to the issue's literal wording, but the full test suite caught it:
ListField.unpack/OptionField.unpack(collections.py) deliberately read a short, sometimes entirely empty, tail past a truncated option area and depend on it decoding as zero -- that is how an over-longihl, or a capture cut short by its snapshot length, reads as end-of-option-list orPad1instead of wedging or raising (#431). An unconditional "any shortfall is bogus" guard broke 15 existing tests exercising exactly that mechanism (IPv4UnitTests::test_an_option_area_longer_than_the_datagram_still_parses, the HOPOPT/IPv6-Opts truncation tests, the TCP SACK overrun tests, severalSchemaUnitTests). Every one of those fields is a handful of octets -- far under the 256 KiB ceiling -- so scoping the guard to fire only past it leaves that mechanism completely untouched while still stopping the actual reported danger, which is always multiple orders of magnitude past 256 KiB.Test plan
tests/corekit/test_fields_field.py: exact-length and trailing-extra buffers still work; a small field short by one octet, and one over a completely empty buffer, still zero-pad (proving OptionField.unpack never returns for a well-formed HOPOPT header with an SMF_DPD option #431's mechanism survives); the ceiling boundary in both directions (length == ceilingstill pads,length == ceiling + 1with insufficient data raises); a field past the ceiling with a full buffer is still accepted (the guard fires on the padding, not on size alone); a 16 GiB declared length against a 2-octet buffer is rejected under a 5s deadline without ever allocating; the exception message names both counts.FieldBase.unpack, using a safe 5 MiB/50 MiB surrogate for the magnitude case rather than the real 16 GiB, per the resource constraint below).length >= _MAX_ZERO_PAD_LENGTH(off-by-one at the ceiling) and to a wrong constant (0x40_001) -- both mutants were caught by the boundary tests.pytest tests/corekit/test_fields_field.py-- 11 passed.pytest tests(full tier, no coverage instrumentation) -- 1271 passed, 17 skipped, 2850 subtests passed, 0 failed. This is the run that caught the first, broken attempt at this fix (15 failures) and confirms the corrected version regresses nothing.coverage run -m pytest tests/corekit-- 101 passed, 120 subtests passed.No test in this PR allocates anything close to the declared "huge" lengths it exercises -- the largest actual allocation any test performs is ~256 KiB (the ceiling boundary case itself).