fix(schema): a header cut inside a fixed-width field rebuilds as captured (#1458, #1451)

[JarryShaw]

• Searched for similar pull requests

• Followed the coding style: mypy reports nothing on the new lines; pylint and isort not run

• make test passes, and a test case covers the change: every run_unittest_leg.py leg (foundation 528, dumpkit 232, corekit 633, protocols/internet 617, protocols rest 1160, project 606, all OK) and the four edge modules

• Changelog entry — N/A — centralised in docs(changelog): shared 1.5.0 changelog — long-lived, merges last (#610, #616, #617, #618, #620) #657

• fix — corrects a defect

---

Closes #1458
Closes #1451

When a capture ends inside a field, Schema.unpack now records that field and how many octets it read, as __short_read__. Schema.pack writes back only those octets. Schema.to_dict and Schema.from_dict carry the record, as they already do __remainder__ (#1380). The Link, Internet, Transport and Application bases copy the record into info after a parse, and from_data cuts the rebuild back to it.

The record is an ordinary info key, present only on a truncated header. Info.to_dict therefore carries it, and from_data(info) and from_data(info.to_dict()) both rebuild the capture exactly. Decoded values are unchanged, and no exceptions change, so there is no !.

Dumps: all 69 dumps of the 23 captures (JSON, plist and tree) are byte-identical. A dump of a truncated header gains one entry under that layer, __short_read__: tuple ('data', 19).

Edge tables: the #1454 tables drop their #1451 and #1458 rows. Five SCTP rows for padding of the last chunk remain, now filed under #1474. The two harness defect strings they used are removed.

Sweep: the #1454 edge cases cut at every offset take 45,756 runs, and failures fall from 14,012 to 22. The 22 left are the SCTP last-chunk padding (#1455) and the HOPOPT extension-mode trim (#1446).

tests/protocols/test_short_read_roundtrip_unit.py has 8 tests and 1340 subtests. On main it fails 821. On this PR's previous head, its to_dict assertions fail 812.

[github-actions]

Coverage: 89.59% (unit tier, Python 3.14, 540c8887f, Unit Tests run success)

Package	Statements	Missed	Branches	Partial	Cover
pcapkit (top level)	104	4	20	4	93.55%
pcapkit/const	18795	955	2356	868	91.02%
pcapkit/corekit	2043	89	636	25	94.77%
pcapkit/dumpkit	242	2	88	2	98.79%
pcapkit/foundation	2648	98	930	48	94.80%
pcapkit/interface	112	7	40	5	92.11%
pcapkit/protocols	16965	221	4584	186	98.07%
pcapkit/toolkit	539	74	168	3	84.58%
pcapkit/utilities	429	4	122	4	98.55%
pcapkit/vendor	4409	2342	1006	157	43.25%

Per-file detail: the coverage-html artifact of this run.

[JarryShaw]

Verdict: NEEDS CHANGES at e48586e11 (Sonnet cross-review; the author ran on Opus).

Blocking: the dict-form rebuild of a cut header still writes the field at full width. Data.__excluded__ (pcapkit/protocols/data/data.py:19) hides __short_read__ from Info.to_dict(), so replay_short_read never sees it. I reproduced this on this head:

e = Ethernet(bytes(12) + b'\x08', 13)
len(Ethernet.from_data(e.info).data)            # 13
len(Ethernet.from_data(e.info.to_dict()).data)  # 14

The reviewer saw this in 615 of 3894 cut runs. #1447/#1452 made the dict form a first-class round trip, so this needs fixing, with a test. The PR body's claim that "to_dict/from_dict carry the record" holds for Schema.to_dict, not Info.to_dict.

Holds

• Object form: every one of 3894 cuts from the capture chains is exact, and so are 1407 hand-built cuts.
• Nested SchemaField: handled.
• ConditionalField after the cut: handled.
• ForwardMatchField: the next field records the short read.
• Untruncated packets: info for all 1616 frames is byte-identical to main.
• Merged with fix(ipv6-route): rebuild RPL source routes byte for byte (#1456, #1457) #1463: all 99,840 RPL cuts are exact.
• Merged with test(protocols): edge-case round trips for every layer family (#1202) #1454's tables: only the stale fix(ipv6): a truncated IPv6 extension header rebuilds padded to its declared length #1451/fix(corekit): a header cut inside a fixed-width field is zero-filled on parse and rebuilt at full width #1458 rows fail.
• Speed: extract over http.pcap is about 2.7% slower.
• Tests and legs: without the fix, 819 tests fail. All six legs are OK.

Found outside this PR's scope, and filed:

• fix(pcap,pcapng): a cut PCAP header or PCAP-NG block rebuilds at full width #1470: cut PCAP and PCAP-NG headers, which don't inherit the layer bases.
• fix(ipv6): a cut upper layer after an extension header falls back to Raw that duplicates the header #1471: an IPv6 extension header followed by a cut upper layer, where the Raw fallback duplicates the header.

[JarryShaw]

Verdict: GOOD TO GO at 540c8887f. Round-2 cross-review on Sonnet; the author ran on Opus.

• Round-1 blocker fixed. A cut Ethernet header (13 octets) now rebuilds to 13 from both info and info.to_dict(), and a whole header carries no __short_read__ key. I re-probed this myself on an archive of the head.
• Independent cut sweep. 1207 parseable cuts across 11 protocols:
  • object form: 1010 of 1010 exact;
  • dict form: 903 exact, plus 107 crashes, all of which are the IPv6_Ext stand-in already filed as fix(ipv6): from_data(info.to_dict()) crashes when an IPv6_Ext stands in for a dedicated extension header #1473 (the same counts appear on main);
  • the five SCTP rows remain fix(sctp): a last chunk sent without its padding rebuilds with the padding zero-filled #1474.
• Untruncated output unchanged. Per-frame info over 23 captures matches main, and the 92 dumps are byte-identical (diff -rq empty).
• Tests bite. With pcapkit/ reverted, 821 fail. Re-adding __short_read__ to Data.__excluded__ gives 812 fail, so the dict assertions catch the round-1 bug.
• Merges cleanly with fix(corekit)!: to_dict() returns an InfoDict, so a repeated IPv6 extension header survives the dict rebuild (#1453) #1466, and all six affected modules pass on the merge. The foundation, dumpkit, corekit, internet, rest and project legs are all OK.

One point for the changelog: truncated headers now expose __short_read__ in info, iteration and dumps (e.g. ["icv", 8]). It is the first dunder bookkeeping key in a protocol info.