examples: add a Dockerised engine benchmark covering every supported Python version

[JarryShaw]

examples/benchmark/run.sh          # or: make bench -- every supported Python, one command
make bench-quick                   # 50 rounds x 2 passes, smoke check
make bench-test                    # 116 self-tests, no Docker needed

One invocation builds an image per interpreter (3.10 through 3.14 by default, 3.15 opt-in), measures seven environments, and writes two paste-ready reStructuredText snippets. A full default matrix is on the order of two hours, most of it pyshark spawning a tshark per extraction; --engines default,dpkt,scapy,pypcap,pcap_ct,pypcapfile drops it to minutes, and --pythons 3.11 to one column.

Why this exists

The engine speed table in the README can't be reproduced, and can't even be completed on one machine:

• pypcap and pcap_ct are mutually exclusive — both distributions own the import name pcap, and with both installed pcap-ct wins while upstream is shadowed.
• pypcap needs a compiler plus libpcap headers, and won't build past 3.11.
• pyshark needs the tshark binary and can't run on 3.14.
• pypcapfile can't be imported on 3.12+.

Python 3.11 is the last interpreter where all seven run, and even there two can't coexist. That used to be a constraint on the run; it is now a fact the table reports, as -- cells carrying each engine's own reason.

The matrix

One image per Python version, each pinned by digest in python-images.txt — a table rather than logic, so a reader can check it, and adding a version is one line. Inside each image, one or two virtualenvs, so an environment is a (Python version, pcap provider) pair:

Python	environments	why
3.10, 3.11	-pypcap and -pcap_ct	both providers install, and they're mutually exclusive
3.12, 3.13, 3.14	-pcap_ct only	pypcap can't be installed, so a second venv would compile for minutes to produce nothing
3.15	opt-in	no released image; the pin names an rc

Environment labels are derived by asking each interpreter its own version, so a mislabelled build cannot put one column's figures under another's heading. pypcap on 3.12+ reports unsupported on this interpreter rather than failed — it didn't fail, it was never attempted, and the reader is owed the cause rather than the consequence.

Two tables, because there are two questions

File	Question	Arithmetic
table-versions.rst	how does each engine do on each interpreter?	absolute ms/packet, one column per version — README's Test Results, verbatim
table.rst	which engine is faster, on any machine?	ratios against default, pooled across the matrix

The per-version table has to be absolute: its columns came from one host differing only in the interpreter, which is exactly when absolute times are comparable. A ratio across two interpreters is neither engine speed nor interpreter speed. The ratio table stays because absolute times can't be reproduced across machines — which is the state the current README table is in.

A missing column is held to the same standard as a missing row

Five images mean five chances for something outside these pins to break. None of them ends the run: run.sh records the reason per version, and the reporting pass turns it into a column of -- with the reason attached rather than a narrower table. The exit status says which happened — 0 all measured, 3 tables written but a version lost, 1 no table at all, 2 bad arguments. A make bench that got one column of five is not a failure and not a success either.

A real failure mode the first full run exposed

tshark crashed on pass 3 after ~2,000 spawns (TSharkCrashException, retcode 255). The exception propagated and the run produced no output at all — six working engines lost because the seventh hiccupped once.

Now contained at two levels, both counted: a failed extraction is discarded (up to 5 per engine per pass) and the pass continues; a failed pass costs that engine that pass, is marked with what was lost, and an engine becomes *not measured* only if every pass failed. Same principle in the Dockerfile — a pypcap build failure becomes that row's reason instead of aborting the image.

Two things are never tolerated, since tolerating them means publishing a number that isn't true: the escalated EngineWarning meaning the engine fell back to pcapkit's own parser, and a capture that yielded no packets.

Every measurement asserts the engine that actually ran

The harness's most important property. A missing or unusable engine only warns and falls back to pcapkit's parser, so a run that looks successful can be timing the wrong thing three times under three names. type(extractor.engine).__engine_name__ is checked on every extraction, against drivers derived from Extractor.__engine__ rather than hand-written.

Apple Silicon

No --platform override needed. Every pinned digest is a multi-arch index including linux/arm64/v8; pcap-ct, libpcap, dpkt, scapy, pyshark, pypcapfile are arch-independent, lxml has aarch64 wheels for cp310–cp314, and pypcap — the only compile — has no arch-specific code (its setup.py branches on sys.maxsize > 2**32, not the machine).

If --platform linux/amd64 is forced, run.sh warns on stderr and threads the caveat into the emitted table, because whoever reads the table later is not whoever saw the stderr.

Sample output

From a real 1,000 × 3 run — but on 3.11 only, before the matrix existed. Kept because the numbers are real; it is not what the matrix now emits, which is the per-version table alongside this one:

============== ============= ============== ======
Engine         Relative time Observed range Passes
============== ============= ============== ======
``pypcapfile`` 0.0548        0.0540-0.0555  6
``dpkt``       0.0616        0.0612-0.0620  6
``scapy``      0.151         0.149-0.152    6
``pypcap``     0.183         0.1833-0.1834  3
``pcap_ct``    0.210         0.209-0.211    3
``pcapkit``    1             *baseline*     6
``pyshark``    89.1          88.2-89.2      6
============== ============= ============== ======

Both snippets are plain-docutils-safe — no Sphinx-only roles — so they go straight into README.rst. Nothing host-identifying: the point of containerising is that the host stops mattering.

Two results that contradict the current README

Worth a look before anything is pasted:

• pypcapfile measures faster than dpkt (0.0548 vs 0.0616), reversing the published ordering. Plausible on a 6-packet capture where it does the least work, but it contradicts the existing figures.
• pyshark is 89×, not the ~123× implied today (24.68/0.2004). Same order of magnitude, different hardware.

Verification

116 self-tests (was 55) · shellcheck clean · both emitted snippets parse under plain docutils with zero messages · a six-version --quick matrix built and measured every column including the 3.15 rc · a forced-missing column produced a marked column, a version-level note, unblamed engines and exit 3.

Honest caveat: no full-fidelity matrix run has been made. Every run was --quick (50 × 2), which proves the harness and both tables but is not publishable. The full 1,000 × 3 matrix should also run on the machine whose figures the README quotes — its Test Environment block names an M2 Pro, and numbers from another host aren't comparable with it.

Three bugs found while verifying, all fixed

1. awk compared versions numerically, so --pythons 3.1 matched the 3.10 row and started building py3.1 — a typo silently becoming a run.
2. Reason strings reached RST unescaped. Reachable by design, since the Dockerfile records a failed pypcap build's pip output as that engine's reason: a line containing **kwargs, GNU's cannot find `pcap.h, a trailing imp_ or a trailing :: each turn the pasted table into a visible error block on GitHub.
3. render_text called a partly-measured version "not measured" while printing its figures three lines above.

Notes

• examples/benchmark/README.md is now README.rst, for consistency with the project README — git mv, so history follows. RST doesn't nest inline markup, so the 17 places where bold wrapped a code span are now one or the other; verified through the parsed doctree, since docutils keeps the stray backticks as text without warning.
• No .gitignore change needed — out/ already covers examples/benchmark/out/.

[Copilot]

🟡 Changes recommended

The report stitching logic in examples/benchmark/report.py uses truthiness checks for baseline divisors and records environments even when no ratios were contributed, which can lead to incorrect cross-environment reporting.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds a containerised benchmarking harness under examples/benchmark/ to reproducibly generate the engine speed table (as paste-ready RST) from a pinned Docker image, including self-tests and Makefile entry points to run the suite with one command.

Changes:

• Add a Docker-based benchmark runner (run.sh + Dockerfile + entrypoint.sh) that measures engines across two isolated virtualenvs and emits a README-ready RST snippet plus raw JSON/locks.
• Add the benchmark measurement and reporting implementation (benchmark.py, report.py) and a dedicated self-test suite (test_harness.py).
• Add pinned requirements files for the two environments and documentation (examples/benchmark/README.md) plus Make targets (bench, bench-quick, bench-test).

File summaries

File	Description
Makefile	Adds bench, bench-quick, and bench-test targets to run the harness and its self-tests.
examples/benchmark/benchmark.py	Implements per-environment engine timing, driver assertion, and JSON output contract.
examples/benchmark/report.py	Stitches per-env JSON documents and renders operator text + README-ready RST tables.
examples/benchmark/test_harness.py	Adds self-tests for ratio math, stitching, markup validity, and failure handling.
examples/benchmark/run.sh	Host-side script to build/run the image and copy out/render results.
examples/benchmark/entrypoint.sh	Image entrypoint to run measurements in both venvs then generate reports.
examples/benchmark/Dockerfile	Builds a pinned-image benchmark environment with two venvs and required system deps.
examples/benchmark/README.md	Documents the benchmark’s rationale, methodology, and how to run it.
examples/benchmark/requirements-common.txt	Pins shared engine/runtime dependencies for reproducible runs.
examples/benchmark/requirements-pypcap.txt	Pins the pypcap-specific environment requirements.
examples/benchmark/requirements-pcap_ct.txt	Pins the pcap-ct-specific environment requirements.

Review details

Suppressed comments (1)

examples/benchmark/report.py:708

• In _ratios_for(), if by_repeat.get(sample['repeat']) is a truthiness check that will also drop repeats where the baseline divisor is 0, and it duplicates the baseline-handling logic from collect() in a less explicit way. Switching to an explicit None/zero check makes the behavior clearer and avoids surprises from falsy numeric values.

        by_repeat = {sample['repeat']: sample['ms_per_packet'] for sample in base['repeats']}
        return [sample['ms_per_packet'] / by_repeat[sample['repeat']]
                for sample in entry['repeats'] if by_repeat.get(sample['repeat'])]

• Files reviewed: 11/11 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.