Implement ESP, with optional payload decryption

[JarryShaw]

Closes the ESP item on the Help Wanted page. Scope chosen by @JarryShaw: full ESP including decryption, accepting the crypto dependency and key-management surface.

ESP was already advertised, and absent

pcapkit/protocols/internet/ipsec.py returned id() == ('AH', 'ESP') — asserted by tests/protocols/internet/test_ah_unit.py — the docs carried a broken xref to pcapkit.protocols.internet.esp.ESP, and IP protocol 50 fell through to Raw. The only source was a 98-line sketch under NotImplemented/ whose read() body was pass, in a directory with no __init__.py. It was read for intent, then the real thing written fresh; the sketch is deleted.

The structural problem, and the general fix

RFC 4303 does not put the payload length, the Pad Length/Next Header trailer or the ICV length on the wire — they need SA context. And nothing user-supplied reached a protocol during extraction: Protocol._import_next_layer passed a fixed kwarg set, and Extractor.__init__ a fixed keyword list.

So this adds a channel rather than an ESP special case: pcapkit/corekit/context.py, keying caller-supplied state on Protocol.id(). Extractor and extract() take a context= argument, and _import_next_layer threads it down as __context__, mirroring the existing __packet__ convention. Any protocol needing caller state can use it; a protocol finds its own entry without knowing how the caller spelled the registry. All three signatures gained keyword-only arguments with defaults, so nothing existing changes shape.

sa = SecurityAssociation(spi=0x8765, encryption=Cipher.AES_CBC,
                         encryption_key=key, integrity=Integrity.HMAC_SHA2_256_128,
                         integrity_key=mac_key, destination='192.168.123.200')
extraction = pcapkit.extract('esp.pcap', context=ESPContext(sa))
extraction.frame[0].protochain     # Ethernet:IPv4:ESP:IPv4:ICMP

Behaviour with and without keys

Without an SA — reports SPI and sequence, hands the remainder over as opaque ciphertext with status=NO_SA, leaves next/pad_len/padding/plaintext as None rather than guessing a trailer, and surfaces the ciphertext as Raw so payload and protochain behave normally. It never raises. Verified on this branch: 0x8765 seq=1, payload class Raw.

With an SA — derives the ICV split, decrypts, strips padding by Pad Length, and dispatches the plaintext through Next Header.

Algorithms

Supported: ENCR_NULL, ENCR_AES_CBC (128/192/256), ENCR_AES_GCM_8/12/16; integrity AUTH_NONE, AUTH_HMAC_SHA1_96, AUTH_HMAC_SHA2_256_128/384_192/512_256 — RFC 8221's mandatory-to-implement anchor. Not implemented and rejected with a clear error rather than half-working: AES-CCM, ChaCha20-Poly1305, 3DES, DES, AES-XCBC-96, AES-GMAC, HMAC-MD5-96. Documented in two tables rendered on the ESP docs page.

cryptography is an optional extra; pcapkit imports and works without it, and ESP degrades to the no-keys path with a warning if a key is supplied while it is missing.

Deliberately not implemented

• Extended Sequence Numbers — the high-order 32 bits are not transmitted and a stateless parser cannot recover them, though the ICV and GCM AAD both need them. An ESN packet fails its integrity check cleanly rather than being decoded.
• TFC padding — indistinguishable from payload without the inner protocol's length field, so it reaches the next layer as plaintext.
• Anti-replay — the sequence number is reported, never checked.

ICV failures are reported, not raised, so one bad packet cannot abort a capture.

Keys are kept out of anything that gets written

Checked rather than asserted: the only thing serialised is frame.info.to_dict(), and the ESP data model holds no key field by construction; ProtocolBase.__repr__ renders _info only, so an SA held on the context cannot appear; SecurityAssociation.__repr__ prints SPI, algorithm names, ICV length and destination. A test extracts a real capture to a tree dump and asserts the key hex is absent from it.

Tests

28 tests / 36 subtests, on published vectors: RFC 3602 §4 cases 5 and 7 (AES-CBC-128, transport and tunnel mode — case 7 asserts the tunnelled datagram decodes to IPv4:ESP:IPv4:ICMP with the inner addresses), and draft-mcgrew-gcm-test-01 §4 for AES-GCM at 128 and 256 bits. HMAC-SHA-256-128 has no published ESP packet vector, so its expected ICV is both recomputed with stdlib hmac over the RFC 4303 §2.8 coverage and pinned as hex.

Also covered: wrong cipher key → DECRYPT_FAILED with plaintext=None, wrong AEAD key → AUTH_FAILED, wrong integrity key → AUTH_FAILED before decryption, corrupted and truncated ICV, non-block-multiple ciphertext, make round trips including encrypt=True across 20 payload sizes and all three GCM ICV lengths, degradation with cryptography mocked absent, destination-keyed SA matching, and ESP as an IPv6 extension header.

Full suite on this branch: 467 passed, 4 skipped, 227 subtests passed — exactly +28/+36 over main, nothing else moved.

[Copilot]

🟡 Changes recommended

A few concrete issues remain around context/packet forwarding semantics and avoiding potentially sensitive context repr output in engine warnings.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR adds first-class support for IPsec ESP (protocol 50) including optional payload decryption, and introduces a generic “protocol-keyed parsing context” mechanism so protocols can consume caller-supplied out-of-band state (e.g., ESP Security Associations) without protocol-specific Extractor kwargs.

Changes:

• Implement pcapkit.protocols.internet.esp.ESP with SA-driven trailer/ICV handling and optional decryption via cryptography (extra pypcapkit[crypto]).
• Introduce pcapkit.corekit.context.ContextRegistry and propagate it through extraction and nested protocol parsing via __context__ / _get_context().
• Add comprehensive ESP unit tests and update docs/registries so ESP is importable, documented, and reachable from IP protocol 50.

File summaries

File	Description
tests/protocols/internet/test_esp_unit.py	New unit tests covering ESP registry, parsing, decryption, failure modes, and extraction end-to-end.
tests/foundation/engines/test_runtime_engines.py	Updates runtime-engine test helper to include an extractor context registry.
pyproject.toml	Adds optional dependency extra crypto = ["cryptography>=3.4"] and includes it in all.
Pipfile	Adds cryptography to the dev environment dependencies.
pcapkit/protocols/schema/internet/esp.py	Adds ESP schema (SPI/SEQ + opaque payload field).
pcapkit/protocols/schema/internet/init.py	Exports ESP schema in the internet schema package.
pcapkit/protocols/protocol.py	Adds context plumbing: stores __context__, exposes _get_context(), forwards context to next layers.
pcapkit/protocols/internet/NotImplemented/esp.py	Removes abandoned ESP sketch implementation.
pcapkit/protocols/internet/ipv6.py	Forwards __context__ when importing next layers.
pcapkit/protocols/internet/ipv4.py	Threads __packet__ (src/dst) for downstream protocols (used by ESP destination matching).
pcapkit/protocols/internet/ipsec.py	Updates docs/comments now that ESP is implemented.
pcapkit/protocols/internet/internet.py	Registers ESP in protocol registry and forwards __context__ to next layers.
pcapkit/protocols/internet/esp.py	New ESP implementation (SA, algorithms, decryption/encryption, and opaque fallback).
pcapkit/protocols/internet/init.py	Exports ESP from the internet protocols package and updates TODO list.
pcapkit/protocols/data/internet/esp.py	Adds ESP data model capturing ciphertext/plaintext split and status.
pcapkit/protocols/data/internet/init.py	Exports ESP data model in the internet data package.
pcapkit/protocols/init.py	Exports ESP at the top-level protocols namespace.
pcapkit/interface/core.py	Adds context= argument to extract() public API and documents its use.
pcapkit/foundation/extraction.py	Stores normalized caller context on the Extractor (_exctx).
pcapkit/foundation/engines/scapy.py	Warns when context is supplied but ignored by Scapy engine.
pcapkit/foundation/engines/pyshark.py	Warns when context is supplied but ignored by PyShark engine.
pcapkit/foundation/engines/pcapng.py	Passes __context__ into PCAPNG parsing path.
pcapkit/foundation/engines/pcap.py	Passes __context__ into PCAP frame parsing path.
pcapkit/foundation/engines/dpkt.py	Warns when context is supplied but ignored by DPKT engine.
pcapkit/corekit/context.py	New ContextRegistry / ProtocolContext implementation and documentation.
pcapkit/init.py	Exports ESP at package top-level.
docs/source/pep.rst	Updates Help Wanted / ESP status notes to reflect implementation.
docs/source/pcapkit/protocols/internet/ipsec.rst	Removes “ESP not implemented” footnote references.
docs/source/pcapkit/protocols/internet/index.rst	Adds ESP to the internet protocol docs toctree and updates TODO.
docs/source/pcapkit/protocols/internet/esp.rst	New ESP documentation page.
docs/source/pcapkit/corekit/index.rst	Adds context channel to corekit docs index.
docs/source/pcapkit/corekit/context.rst	New documentation page for parsing context mechanism.

Review details

Suppressed comments (1)

pcapkit/protocols/internet/esp.py:1038

• Passing packet=packet or None drops an empty-but-intentionally-shared packet-info dict (and is inconsistent with how other protocols forward packet). Forward the dict as-is so downstream layers can rely on object identity.

        return self._decode_next_layer(esp, next_type, len(inner), packet=packet or None,
                                       version=version, payload=inner)

• Files reviewed: 32/32 changed files
• Comments generated: 5
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

It introduces substantial new crypto-adjacent parsing and a new context-propagation mechanism across core extraction/protocol plumbing, warranting final human review despite limited nits.

Review details

• Files reviewed: 32/32 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

There are correctness issues in ESP’s UNSUPPORTED handling and packet-context propagation that can lead to wrong status/ICV reporting and missing schema context in nested parsing.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 38/44 changed files
• Comments generated: 4
• Review effort level: Lite

[JarryShaw]

@copilot resolve the merge conflicts in this pull request

[Copilot]

@copilot resolve the merge conflicts in this pull request

Merged origin/main into feat/esp-decryption (commit 904140b). The only conflict was in docs/source/pep.rst — the "not yet implemented" protocol lists: ESP is now implemented on this branch so it stays out of that list, and SCTP was completed on main so it's removed from the transport-layer entry there too. Everything else merged cleanly.

[JarryShaw]

All ten inline comments triaged. Six were already fixed in earlier pushes (the three ext._exctx!r warnings, kwargs.get('packet') or {}, the per-layer ContextRegistry.make() copy, the pyshark engine label). Of the four newer ones:

Fixed here (2d3c7db7a) — the ICV/UNSUPPORTED ordering at esp.py:1056. Correct catch. read() checked association.unavailable() before the ICV split and the truncation check, but icv_length comes from the security association and does not depend on cryptography being installed. So with the backend missing, the ICV was never reported — including a combined-mode authentication tag a caller may well want — and a packet too short for the declared ICV came back UNSUPPORTED instead of TRUNCATED. Both checks now run first, and payload_data excludes the ICV for UNSUPPORTED as it does for every other status. New test asserts both; verified failing before the change (AssertionError: b'' != b'\xa0…\xaf') and passing after.

Filed as #382, not fixed here — the three __packet__ comments (protocol.py:1222, internet.py:255, ipv6.py:415). The underlying observation is right: packet= reaches the protocol but schemas read their context from the separate __packet__ keyword (protocol.py:284,304), and nothing bridges them, so schema/internet/ipv6_route.py:157's packet.get('dst') is always None. But it is pre-existing on main — git show origin/main:pcapkit/protocols/protocol.py has the identical packet=packet call with no __packet__, from before ESP existed; this PR only added __context__ threading alongside it. Fixing it means changing next-layer dispatch for every protocol, which wants its own change and its own tests, and #382 also records the caveat that the user-visible effect is currently masked by unrelated type-dispatch bugs in ipv6_route.py.

Verification on this head: 3.14 full suite 502 passed / 4 skipped / 285 subtests; 3.10 CI selection 362 passed / 28 skipped / 0 failed — the configuration that was previously red, now green on the rebase onto #381.