[JarryShaw]

Please follow the guide below

• You will be asked some questions, please read them carefully and answer honestly

• Put an x into all the boxes [ ] relevant to your pull request (like that [x])

• Use Preview tab to see how your pull request will actually look like

• Searched for similar pull requests

• Followed the coding style (make pylint, make mypy, make isort)

• make test passes, and a test case covers the change

• Added a changelog entry under docs/source/changelog/ and regenerated CHANGELOG.md, if the change is user-visible — N/A — centralised in docs(changelog): shared 1.5.0 changelog — long-lived, merges last (#610, #616, #617, #618, #620) #657

What is the purpose of your pull request?

Tick the commit type your subject line carries.

• fix — corrects a defect
• feat — adds a feature
• perf — changes performance, not behaviour
• refactor — changes neither behaviour nor performance
• test — tests only
• docs — documentation only
• ci — workflows or build tooling
• release — bumps the version or rolls up a distribution
• chore — anything else

---

Description of your pull request and other information

Root cause (lines on e9f3340f7), all in SeekableReader, which wraps every non-seekable fin:

• pcapkit/corekit/io.py:351 — seek(0, SEEK_END) was the buffer's far edge. Once the 131,072-octet buffer fills, that edge is exactly how far the stream has been read, so Frame._check_record_floor (frame.py:487) measures 0 left after the first record ending past it: frame 870 of http.pcap. Now _read_ahead reads past the position, or to the real end.
• io.py:522, io.py:538 — read(n) made one stream read, short on a pipe or raw stream (7-octet chunks gave 0 frames). io.py:386-387 — a forward seek did the same, and could overshoot. Both now go through _fill.
• extraction.py:1391, engines/pcapng.py:372 — a short peek(4) failed the magic number, and skipped the packet-block-before-IDB check. Both fall back to read-and-seek-back.

Read-ahead covers exactly the next record, so a live pcapkit - is not held back: a PCAP record header and its incl_len; a PCAP-NG block's Total Length (BOM byte order for an SHB). The buffer grows to hold a record longer than itself, so an oversize EPB, SHB or PCAP record is read whole; a bogus length reads in 1 MiB pieces to the real end, as a file read does. A block shorter than its fixed fields, or not word-aligned, keeps the buffer's worth, and a DSB extends to its Secrets Length: both are parsed past their end from a file too. Live pipe, 100 frames 50 ms apart: PCAP 100 before close (median 0.4 ms), PCAP-NG 100 (0.6 ms); main 50 and 48.

Also: flush() skips an already-closed stream (the dpkt NeedData "Exception ignored" noise); positions before a read-ahead SEEK_END are documented as possibly gone; test_io.py's seek(-1, SEEK_END) pin moves 3→4.

Tests: 24, all failing on e9f3340f7: pipe/raw/buffered at 1/7/4096/65536 octets × no_eof 0/1, PCAP and two-section PCAP-NG, default and dpkt (scapy refuses streams); truncation; malformed and oversize records, including dpkt's seek back over an oversize SHB; live PCAP and PCAP-NG; CLI on stdin, PCAP→PCAP and PCAP-NG→JSON. Frames and warnings match the path read.

Selections run (not the whole suite): the two new test files 11 and 13 passed; tests/corekit/test_io.py 34; other test_extraction*.py and test_io*.py 98; tests/foundation/engines 142 passed, 840 skipped; tests/protocols/misc, tests/cli, test_cli_subprocess.py 250. Coverage: io.py 100%, engines/pcapng.py 99%, as before.

Closes #1586

[github-actions]

Coverage: 89.66% (unit tier, Python 3.14, 4bb654921, Unit Tests run success)

Package	Statements	Missed	Branches	Partial	Cover
pcapkit (top level)	104	3	20	3	95.16%
pcapkit/const	18858	957	2366	871	91.03%
pcapkit/corekit	2366	67	760	35	96.48%
pcapkit/dumpkit	275	3	100	3	98.40%
pcapkit/foundation	2841	154	1002	42	93.55%
pcapkit/interface	138	7	52	5	93.68%
pcapkit/protocols	16988	227	4540	192	98.04%
pcapkit/toolkit	708	96	234	6	85.35%
pcapkit/utilities	431	4	124	4	98.56%
pcapkit/vendor	4409	2342	1006	157	43.25%

Per-file detail: the coverage-html artifact of this run.

[JarryShaw]

Verdict: NEEDS CHANGES at 912118bb2. Cross-reviewed on Sonnet; the author ran Opus.

Blocking: a live PCAP-NG pipe now stalls. seek(0, SEEK_END) reads ahead a full buffer (131,072 octets), so a slow PCAP-NG writer's frames are held until that much has arrived or the pipe closes.

• In process: 100 frames sent 50 ms apart came out as 0 frames before the pipe closed. On main it was 48.
• Via pcapkit -: 20 frames, pipe held open 6 s: no output file at all until close. On main, 7 frames.

Fix: limit the PCAP-NG read-ahead to the block's declared length, capped at the buffer. _check_floor/_captured (pcapng.py:1543, :1562) only need to know whether the whole block is present. Then add a live PCAP-NG pipe test, and a PCAP-NG→JSON case to the stdin CLI test.

Confirmed:

• Pipes now read everything. Every frame comes through byte-identical, with the same repr(info) and warnings as a path run. That holds for 14 fixtures (PCAP, multi-section and multi-interface PCAP-NG) × 12 stream types (os.pipe, unbuffered pipe, short reads of 1/7/4096/65536, buffered) × no_eof, on both default and dpkt. cat http.pcap | gives 1,117 of 1,117, which I checked myself.
• Live PCAP streams again. 20 of 20 frames arrive while the pipe is open, median 1 ms (main median 1.28 s).
• Truncation unchanged. Twelve cut captures still end with "EOF reached" and no new SeekWarning.
• Test change justified. The test_io.py move from 3 to 4 is correct, because the old value was a buffer artifact.
• Fail-before and merges. All 16 new tests fail on main. The PR merges cleanly with fix(protocols): drop the warnings of a next layer replaced by Raw (#1580) #1588, fix(toolkit): the scapy adapters stop at the IP datagram's declared length (#1584) #1589 and fix(protocols): dissect LINKTYPE_RAW by version nibble; add the BSD loopback layer for LINKTYPE_NULL/LOOP (#1502, #1574) #1594.

[JarryShaw]

Verdict: NEEDS CHANGES at 35117e105. Round 2 was a Sonnet review of 12192f5fc. The only change in 35117e105 is the 2-line secrets→dsb_block rename, which does not affect the verdict.

Confirmed:

• Byte-identical reads. The harness covered 26 examples/captures files plus multi-section, mixed-byte-order and DSB PCAP-NG fixtures, over 12 stream types with no_eof on and off. Frames, repr(info), warnings and exceptions match the path read on default and dpkt.
• Live pipe. 100 frames written 50 ms apart all arrive before close for PCAP and PCAP-NG, median 0.4–0.6 ms. That holds even with an SHB written 1 octet at a time.

Required: a block or record larger than the 131,072-octet buffer still differs from a path read. I measured this on 35117e105 with three frames of 1,000, 140,000 and 500 octets on the default engine:

• PCAP-NG EPB: the path read gives 140,000 octets. The pipe gives 131,044 octets, plus block length mismatch: 140032 != 0 and captured length 140000 runs past the block.
• PCAP (snaplen 262144): the path read gives all three frames. The pipe raises SeekError('cannot seek before the beginning of the buffer: 1040 < 9984').

main also fails here (it drops the third PCAP-NG frame), so this is not a regression. It is still #1586's defect, and the description's claim that frames match the path read does not hold for it. The read-ahead already extends a DSB to its Secrets Length; extending it to the block's Total Length (PCAP-NG) and to incl_len (PCAP) should close the gap. Please add a test for each.

Round 3 will also cover what round 2 did not reach: malformed blocks other than oversize ones, fail-before on e9f3340f7, and the pcapkit - CLI on a live pipe.

[JarryShaw]

Verdict: GOOD TO GO at 4bb654921. Round 3 was a Sonnet review; the author ran on Opus. I re-ran the checks it did not reach.

• Oversize records now match a path read (my measurement). These are identical through a pipe and by path, on default and dpkt: frames, lengths and a payload hash. Cases were PCAP-NG EPBs of 140,000 and 300,000 octets, a ~150 KB SHB, and a 140,000-octet PCAP record.
• Malformed blocks (Sonnet), 23 cases. Cases: PCAP-NG Total Length 0/8/11, not ×4, or below the block minimum; mismatched trailing length; 0xFFFFFFF0; past the stream; unknown BOM; huge DSB Secrets Length; truncations; PCAP incl_len above snaplen, above the stream, 0xFFFFFFF0 or 0. Frames, warnings and exceptions match the path read with 65,536- and 7-octet pipe writes, on default and dpkt.
• Memory (Sonnet). 20,000 records with one 300,000-octet record grow the buffer to 300,016 and hold it there, with RSS +3 MB. A bogus 0xFFFFFFF0 length in a 16 MB stream reads to the real end at +48–62 MB peak.
• Fail-before (my run, PR tests on e9f3340f7 sources). All 11 functions in test_extraction_non_seekable_stream_1586_unit.py fail, as do all 13 in test_io_seekable_reader_short_reads_1586_unit.py. That includes the live-pipe and CLI-on-stdin tests.
• Narrow runs on the head (mine).
  • non-seekable: 11 passed, 67 subtests
  • short reads: 13 passed
  • test_io.py: 34 passed, 41 subtests
• Conflicts. merge-tree is clean against main and against each of the other 7 open PRs.

Non-blocking. The grown buffer never shrinks, so it is bounded by the largest record seen. A bogus 32-bit length on a never-ending live pipe can buffer up to ~4 GiB before it errors, where a file read stops at the file's end.