diff --git a/README.md b/README.md
index 3c554a41..f7eda9cb 100644
--- a/README.md
+++ b/README.md
@@ -272,7 +272,7 @@ What a call costs inside your process, from the wall-clock guards in `tests/perf
| Redis L2 hit with L1 disabled, same dict, Redis in a container on the same machine | 539 ± 3 µs |
- With no backend, L1 keeps the returned object itself, whatever the payload's size, and a repeat call with a few short scalar arguments is answered by one call into the Rust extension that takes no lock ([which calls](https://github.com/cachekit-io/cachekit-py/blob/main/docs/performance.md#decorator--l1-hit-hot-path)). Any other call builds its key every time, as does every call to a function with a backend.
-- With an L2 backend, L1 holds the serialized bytes and every hit deserializes them: for this dict, about 910 times the L1-only hit. This run predates hits skipping the structure walk over bytes they already checked, which took about 20% off this hit. Encryption adds a decrypt to each such hit and keeps the walk a plaintext hit now skips; before that skip, this run measured the difference at 9.3 ± 1.0 µs.
+- With an L2 backend, L1 holds the serialized bytes and every hit deserializes them: for this dict, about 910 times the L1-only hit. This run predates the Rust decode every such hit now runs, which bounds and decodes the MessagePack in one pass at about half the instructions. Encryption adds a decrypt to each such hit; this run measured the difference at 9.3 ± 1.0 µs.
- An L2 hit adds the backend's round trip. A Redis on another machine adds its network latency to the last row.
Hosted runners come with other CPU models too: in a run of an earlier commit, an EPYC 9V45 runner took about half as long on every row, and other machines give other figures. The [Performance Guide](https://github.com/cachekit-io/cachekit-py/blob/main/docs/performance.md#github-hosted-runner) names the guard behind each figure and how to run it, and the repository's `Bench` workflow repeats the run.
diff --git a/docs/data-flow-architecture.md b/docs/data-flow-architecture.md
index 287d069a..302bf0b8 100644
--- a/docs/data-flow-architecture.md
+++ b/docs/data-flow-architecture.md
@@ -31,7 +31,7 @@ cachekit uses a hybrid Python-Rust architecture to provide production caching wi
> **Architecture Principles:**
> - **L1 + L2 Caching**: In-memory bytes cache (L1) + pluggable backend storage (L2) for optimal performance
> - **Backend Abstraction**: Protocol-based L2 backend layer (Redis, CachekitIO, File, Memcached, custom)
-> - **Bytes-Only Storage**: with a backend configured, L1 stores serialized bytes (encrypted or plaintext), not Python objects; L1-only mode (`backend=None`) stores the object itself. After its first hit, a plaintext entry of the default serializer with integrity checking on is held marked as walked, as its verified MessagePack where that is no longer than the envelope, so later hits skip re-checking it (see [Storage Architecture](#storage-architecture)); encrypted caches keep the ciphertext
+> - **Bytes-Only Storage**: with a backend configured, L1 stores serialized bytes (encrypted or plaintext), not Python objects; L1-only mode (`backend=None`) stores the object itself. After its first hit, a plaintext entry of the default serializer with integrity checking on is held as its verified MessagePack where that is no longer than the envelope, so later hits skip the decompress and checksum; every hit still runs the bounded decode (see [Storage Architecture](#storage-architecture)); encrypted caches keep the ciphertext
> - **Security First**: Decompression bomb protection, checksum validation, size limits, encrypted-at-rest
> - **Performance Optimized**: Thread affinity, connection pooling, pre-cached serializers
> - **Flexible Configuration**: L1+L2, L1-only, L2-only, or observability-only modes
@@ -134,7 +134,7 @@ cachekit uses a hybrid Python-Rust architecture to provide production caching wi
│ STEP 6: DESERIALIZATION (Bytes → Python Objects) │
│ File: src/cachekit/cache_handler.py (_deserialize_bytes) │
│ File: src/cachekit/serializers/standard_serializer.py │
-│ File: rust/src/byte_storage.rs │
+│ File: rust/src/python_bindings.rs (cachekit-core ByteStorage) │
├─────────────────────────────────────────────────────────────────────────────┤
│ _deserialize_bytes(data: bytes) → Python object │
│ │
@@ -153,7 +153,7 @@ cachekit uses a hybrid Python-Rust architecture to provide production caching wi
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ Path 2: Rust ByteStorage (Compressed MessagePack) │ │
│ │ ByteStorage.retrieve(envelope_bytes) │ │
-│ │ ↓ Rust layer (byte_storage.rs) │ │
+│ │ ↓ Rust layer (cachekit-core) │ │
│ │ 1. Deserialize envelope (MessagePack) │ │
│ │ 2. Security checks (size limits, compression ratio) │ │
│ │ 3. LZ4 decompress │ │
@@ -239,7 +239,7 @@ cachekit uses a hybrid Python-Rust architecture to provide production caching wi
┌─────────────────────────────────────────────────────────────────────────────┐
│ STEP 14: SERIALIZATION (Python Objects → Redis) │
│ File: src/cachekit/serializers/standard_serializer.py │
-│ File: rust/src/byte_storage.rs │
+│ File: rust/src/python_bindings.rs (cachekit-core ByteStorage) │
├─────────────────────────────────────────────────────────────────────────────┤
│ StandardSerializer.serialize(result) → (bytes, metadata) │
│ │
@@ -315,8 +315,8 @@ cachekit uses a hybrid Python-Rust architecture to provide production caching wi
│ objects. L1-only mode (backend=None) stores the object itself. │
│ │
│ L1 admits and evicts envelopes. A later L1 hit on a plaintext entry of the │
-│ default serializer (integrity on) marks it in place as walked, shrinking │
-│ it to its verified MessagePack where that is no longer than the envelope: │
+│ default serializer (integrity on) shrinks it in place to its verified │
+│ MessagePack where that is no longer than the envelope: │
│ see Storage Architecture. Encrypted caches keep ciphertext. │
└──────────────────────────────────┬──────────────────────────────────────────┘
│
@@ -491,10 +491,10 @@ L1-only mode (`backend=None`) skips this layer: it stores the Python object itse
- **After refactor**: Stores bytes (encrypted or plaintext) → `len(bytes)` for size estimation
- **Performance improvement**: O(1) size calculation vs recursive traversal
- **Security improvement**: Encrypted bytes at rest (decrypt-at-read-time only)
-- **Kept entries**: L1 stores, admits and evicts envelopes. The first L1 hit on a plaintext entry of the default serializer with integrity checking on decodes the envelope as every read does: `ByteStorage.retrieve()` verifies the MessagePack, the structure walk (`check_msgpack_structure`) bounds it, and the decoder's object hook restores datetimes. L1 then holds the entry marked as walked from then on (same expiry and LRU place): the verified MessagePack (`VerifiedMsgpack`) when it is no longer than the envelope, else the envelope itself (`WalkedEnvelope`). Each records whether any of the strings `__datetime__`, `__date__` or `__time__` (or text starting `__date`) appears anywhere in the MessagePack. Later hits skip the walk, since the bytes cannot change, and skip the object hook only when none appears, since the hook would then return every map unchanged; held MessagePack also skips the decompress and checksum. Compressible values are usually longer as MessagePack, so they keep the envelope; small and incompressible ones switch. The swap never grows an entry, so L1 never holds fewer entries than it would hold envelopes, and it never touches an entry stored or removed since the hit read it. Bytes from L2, and bytes still held as bytes, never take this path: L1 marks a kept entry by its type, which no backend returns, so they are walked on every decode
+- **Kept entries**: L1 stores, admits and evicts envelopes. The first L1 hit on a plaintext entry of the default serializer with integrity checking on decodes the envelope as every read does: `ByteStorage.retrieve()` checks the envelope and returns the MessagePack its checksum covers, and the Rust decode (`unpackb_standard`) bounds and decodes it in one pass. When the verified MessagePack is no longer than the envelope, L1 then holds it in the envelope's place (`VerifiedMsgpack`, same expiry and LRU place), and later hits skip the decompress and checksum; otherwise L1 keeps the envelope. Every hit still runs the bounded decode. Compressible values are usually longer as MessagePack, so they keep the envelope; small and incompressible ones switch. The swap never grows an entry, so L1 never holds fewer entries than it would hold envelopes, and it never touches an entry stored or removed since the hit read it. Bytes from L2, and bytes still held as bytes, never skip `retrieve()`: L1 marks a kept entry by its type, which no backend returns
**Hierarchical Read Path:**
-1. Check L1 cache (bytes) → if hit, deserialize and return (the first hit on a plaintext entry of the default serializer with integrity checking on marks it as walked, so later hits skip the structure walk, and the object hook when no datetime marker string appears in it)
+1. Check L1 cache (bytes) → if hit, deserialize and return (the first hit on a plaintext entry of the default serializer with integrity checking on marks it as verified, so later hits on held MessagePack skip the decompress and checksum)
2. If L1 miss, check L2 backend (bytes) → if hit, deserialize, populate L1 with bytes, return
3. If L2 miss, execute function → serialize to bytes → store in L2 and L1 → return
@@ -595,7 +595,7 @@ def observability_function():
## Serialization Path
-**Files:** `src/cachekit/serializers/standard_serializer.py`, `rust/src/byte_storage.rs`
+**Files:** `src/cachekit/serializers/standard_serializer.py`, `rust/src/python_bindings.rs` (cachekit-core's ByteStorage)
cachekit v0.1.0 uses only:
@@ -642,7 +642,7 @@ import msgpack
# Python layer
msgpack_data = msgpack.packb(obj)
-# Rust layer (byte_storage.rs) - conceptual flow:
+# Rust layer (cachekit-core ByteStorage) - conceptual flow:
# 1. LZ4 compress
# 2. xxHash3-64 checksum
# 3. Create StorageEnvelope {
@@ -716,13 +716,12 @@ except BackendError as e:
## Deserialization Path
-**Files:** `src/cachekit/serializers/standard_serializer.py`, `rust/src/byte_storage.rs`
+**Files:** `src/cachekit/serializers/standard_serializer.py`, `rust/src/python_bindings.rs` (cachekit-core's ByteStorage), `rust/src/msgpack_decode.rs`
### Deserialization Flow
```python notest
from typing import Any
-import msgpack
def deserialize(data: bytes, metadata) -> Any:
# ULTRA-FAST: NumPy arrays
@@ -739,8 +738,11 @@ def deserialize(data: bytes, metadata) -> Any:
# 4. xxHash3-64 validate
# 5. Size validate
- # MessagePack unpack
- return msgpack.unpackb(original_data)
+ # MessagePack decode (StandardSerializer): one Rust pass that enforces the untrusted-decode
+ # bounds (nesting <= 1,024; no header claiming more than the input can back), builds the
+ # Python objects, and runs the datetime hook only on maps holding a marker key. A decode that
+ # fails re-runs the structure walk over the whole input, so the walk's error wins.
+ return unpackb_standard(original_data, MSGPACK_MAX_NESTING, _standard_object_hook) # illustrative
```
### Security Protections
@@ -824,8 +826,8 @@ For measured latencies, see the Performance Guide's [GitHub-Hosted Runner](perfo
**Serialization Layer:**
- `src/cachekit/serializers/standard_serializer.py` - StandardSerializer (MessagePack + ByteStorage)
- `src/cachekit/serializers/encryption_wrapper.py` - EncryptionWrapper (AES-256-GCM)
-- `rust/src/byte_storage.rs` - Rust ByteStorage (LZ4 + xxHash3-64)
-- `rust/src/encryption/` - Rust encryption module
+- `rust/src/python_bindings.rs` - bindings to cachekit-core's ByteStorage (LZ4 + xxHash3-64) and encryption
+- `rust/src/msgpack_decode.rs` - StandardSerializer's bounded one-pass MessagePack decode
**Connection Layer:**
- `src/cachekit/connection.py` - Redis connection pooling (used by RedisBackend)
diff --git a/docs/performance.md b/docs/performance.md
index b21d96bb..e1acc2ed 100644
--- a/docs/performance.md
+++ b/docs/performance.md
@@ -12,7 +12,7 @@
> **Key numbers** (mean of five run medians; GitHub-hosted `ubuntu-latest` runner, AMD EPYC 7763, CPython 3.14.7, 2026-10-31; see [GitHub-Hosted Runner](#github-hosted-runner)):
> - **Decorator + L1 hit, `@cache(backend=None)`, 100-user nested dict (23.5KB as plain MessagePack)**: 0.311 ± 0.000μs (about 2,500 instructions per call on CPython 3.14, see [Instruction Budgets](#instruction-budgets))
> - **Same hit, 10K-row DataFrame**: 0.31 ± 0.00μs. An L1-only hit never serializes, so payload size barely matters
-> - **Decorator + L1 hit with an L2 backend configured, same dict**: 284 ± 2μs. With a backend, L1 holds bytes and every hit deserializes them. This dict compresses, so L1 holds its envelope. Measured before hits stopped re-checking bytes they had already checked, which took about 20% off this hit; see [Why an L1-Only Hit Costs the Same for Any Payload](#why-an-l1-only-hit-costs-the-same-for-any-payload)
+> - **Decorator + L1 hit with an L2 backend configured, same dict**: 284 ± 2μs. With a backend, L1 holds bytes and every hit deserializes them. This dict compresses, so L1 holds its envelope. Measured before the Rust decode every such hit now runs, which bounds and decodes the MessagePack in one pass at about half the instructions; see [Why an L1-Only Hit Costs the Same for Any Payload](#why-an-l1-only-hit-costs-the-same-for-any-payload)
> - **Redis L2 hit**, L1 disabled, same dict, Redis in a container on the same runner: 539 ± 3μs
> - **Raw L1 byte-cache lookup** (`L1Cache.get`, no decorator): 354–362ns on a shared x86_64 Linux host (see [Measured Figures](#measured-figures))
@@ -50,13 +50,13 @@ The `Bench` workflow (`.github/workflows/bench.yml`) runs five guards from `test
|------|------------------------------|-----------:|
| Decorator + L1 hit, `@cache(backend=None)`, 100-user nested dict (23.5KB as MessagePack) | `test_production_realism.py::test_decorator_overhead_complex_dict` | 0.311 ± 0.000μs |
| Decorator + L1 hit, `@cache(backend=None, serializer="auto")`, 10K-row DataFrame | `test_production_realism.py::test_decorator_overhead_dataframe` | 0.31 ± 0.00μs |
-| Decorator + L1 hit with an L2 backend configured (in-process, never reached), same 100-user dict | `test_production_realism.py::test_decorator_overhead_l1_hit_with_backend` | 284 ± 2μs (before the walk skip, see [Key Numbers](#key-numbers)) |
-| Same hit, plaintext and encrypted arms, interleaved | `test_production_realism.py::test_encryption_overhead` | 286 ± 1μs and 295 ± 1μs (before the walk skip) |
+| Decorator + L1 hit with an L2 backend configured (in-process, never reached), same 100-user dict | `test_production_realism.py::test_decorator_overhead_l1_hit_with_backend` | 284 ± 2μs (before the decode changes, see [Key Numbers](#key-numbers)) |
+| Same hit, plaintext and encrypted arms, interleaved | `test_production_realism.py::test_encryption_overhead` | 286 ± 1μs and 295 ± 1μs (before the decode changes) |
| Redis L2 hit, L1 disabled, same 100-user dict | `test_production_realism.py::test_redis_l2_roundtrip` | 539 ± 3μs |
The job summary prints each figure in the unit its guard reports: the dict's L1-only hit, the hit with a backend configured and the encryption arms in ns, the DataFrame hit and the Redis L2 hit in μs. This table is in μs, so divide the ns figures by 1,000 when you copy them in.
-The dict's L1-only hit timed 311ns in every one of its five runs. The encryption guard's arms differ by 9.3 ± 1.0μs (Welch 95%): an encrypted hit adds a decrypt. Both arms were measured before the walk skip, which plaintext hits take and encrypted hits do not (see [Why an L1-Only Hit Costs the Same for Any Payload](#why-an-l1-only-hit-costs-the-same-for-any-payload)). That is the first hosted run whose band excludes zero, and [the 2026-10-23 run](https://github.com/cachekit-io/cachekit-py/actions/runs/39987019881) put the difference at 10 ± 21μs, so one run does not pin the cost down (see [Not Measured Here](#not-measured-here)). Under the object cache's lock, as it ran until the lock-free hit, that hit took 0.906 ± 0.007μs in the 2026-10-23 run and 0.874 ± 0.005μs in [a run on an EPYC 7763 the same day as the table's](https://github.com/cachekit-io/cachekit-py/actions/runs/41177503539). Hosted runners do not always get the same CPU model: [a run of an earlier commit on an AMD EPYC 9V45 runner](https://github.com/cachekit-io/cachekit-py/actions/runs/39964388582/attempts/2) took 44-59% as long on every row (0.529μs for the dict's L1-only hit, 242 ± 12μs for the Redis L2 hit).
+The dict's L1-only hit timed 311ns in every one of its five runs. The encryption guard's arms differ by 9.3 ± 1.0μs (Welch 95%): an encrypted hit adds a decrypt. Both arms were measured before the decode changes; since then, a plaintext hit on a kept entry skips envelope decompression and checksum validation, which an encrypted hit does not (see [Why an L1-Only Hit Costs the Same for Any Payload](#why-an-l1-only-hit-costs-the-same-for-any-payload)). That is the first hosted run whose band excludes zero, and [the 2026-10-23 run](https://github.com/cachekit-io/cachekit-py/actions/runs/39987019881) put the difference at 10 ± 21μs, so one run does not pin the cost down (see [Not Measured Here](#not-measured-here)). Under the object cache's lock, as it ran until the lock-free hit, that hit took 0.906 ± 0.007μs in the 2026-10-23 run and 0.874 ± 0.005μs in [a run on an EPYC 7763 the same day as the table's](https://github.com/cachekit-io/cachekit-py/actions/runs/41177503539). Hosted runners do not always get the same CPU model: [a run of an earlier commit on an AMD EPYC 9V45 runner](https://github.com/cachekit-io/cachekit-py/actions/runs/39964388582/attempts/2) took 44-59% as long on every row (0.529μs for the dict's L1-only hit, 242 ± 12μs for the Redis L2 hit).
The workflow runs these guards with this command, against a Redis it starts on the runner:
@@ -71,9 +71,11 @@ Without `REDIS_URL`, the Redis guard uses the pytest-redis server the suite star
With `backend=None`, cachekit keeps the returned object itself in memory and hands that same object back on a hit (see [backend=None](backends/none.md)). Nothing is serialized or deserialized, so the 100-user dict and a 10K-row DataFrame cost about the same: 0.311μs and 0.31μs on the hosted EPYC 7763, for the wrapper's call and one lookup in the extension (see [Decorator + L1 Hit](#decorator--l1-hit-hot-path)).
-With an L2 backend configured, L1 holds the serialized bytes instead, and a hit deserializes them, so that hit costs more as the payload grows. For the 100-user dict it took 284 ± 2μs on the same runner, measured before the walk skip described below, about 910 times an L1-only hit of the same dict. Nearly all of the difference is deserializing 23.5KB of MessagePack back into about 400 nested dicts. It is still about half of a Redis L2 hit there (539 ± 3μs), which deserializes the same bytes after the round trip.
+With an L2 backend configured, L1 holds the serialized bytes instead, and a hit deserializes them, so that hit costs more as the payload grows. For the 100-user dict it took 284 ± 2μs on the same runner, measured before the decode changes described below, about 910 times an L1-only hit of the same dict. Nearly all of the difference is deserializing 23.5KB of MessagePack back into about 400 nested dicts. It is still about half of a Redis L2 hit there (539 ± 3μs), which deserializes the same bytes after the round trip.
-For a plaintext cache with the default serializer and integrity checking on, an entry's first L1 hit decodes it as every read does, under the structure walk that bounds untrusted MessagePack, and then L1 holds the entry marked as walked: its verified MessagePack when that is no longer than the envelope, which is true of small values and incompressible ones, and otherwise the envelope itself. Later hits skip the walk, because the bytes it passed cannot change. They also skip the decoder's object hook when none of the strings `__datetime__`, `__date__` or `__time__` (or any text starting `__date`) appears anywhere in the MessagePack, because the hook would then hand back every map unchanged, and a hit on held MessagePack skips the envelope's LZ4 decompress and xxHash3-64 checksum as well. Bytes read from L2 are walked on every read. The 100-user dict compresses to a 3,171-byte envelope from 23.5KB of MessagePack, so L1 keeps its envelope, and its hits skip the walk and the hook: on CPython 3.14.3 the hit fell from 2,731,277 to 2,192,392 instructions (−19.7%, callgrind with the [Instruction Budgets](#instruction-budgets) method; no budget gates this large-value hit, and `l1_hit_backed` gates the small-value hit on held MessagePack), and an interleaved wall-clock run on a shared x86_64 Linux host put it 20.9% faster (median of 12 paired processes; the base against itself, −1.2%). The same dict with datetimes in it keeps the hook and fell 5.6%. The hosted figures above were measured before this change. Encrypted caches always keep the ciphertext envelope, and every hit on one still decrypts, walks and runs the hook.
+Every read with the default serializer decodes its MessagePack in the Rust extension, in one pass that also enforces the bounds on untrusted MessagePack: nesting at most 1,024 levels deep, and no header declaring more elements or bytes than the rest of the input can back. That covers L2 reads, L1 hits and the decrypted plaintext of encrypted reads. The decode builds the Python objects directly. It shares most repeated short strings, map keys up to 64 bytes and values up to 32, as one object within a value (a small table per decode; two strings that land in one slot replace each other). It runs the Python hook that restores datetimes only on maps holding a `__datetime__`, `__date__` or `__time__` key, because the hook hands back any other map unchanged. It returns the same values and refuses the same inputs with the same exception classes as the decode it replaced: cachekit-core's structure walk, then msgpack-python's C unpacker calling the hook once per map. On the 100-user dict (23,547 bytes of MessagePack, about 3,400 objects), the decode fell from 2,580,104 to 1,238,204 instructions on CPython 3.14 (−52.0%, callgrind main-thread instructions per call, the [Instruction Budgets](#instruction-budgets) method), level with `pickle.loads` of the same value (1,203,437). The 1,000-user dict (239 KB) fell 51.0% and a 1,090-byte one 45.3%. A list of 200 records each holding a datetime fell 33.2%, since every datetime still runs the hook. On the abi3t extension that CPython 3.15 and later load, the 100-user dict fell 50.5%.
+
+For a plaintext cache with the default serializer and integrity checking on, an entry's first L1 hit decodes it as every read does, and then, when its verified MessagePack is no longer than the envelope, which is true of small values and incompressible ones, L1 holds that MessagePack in the envelope's place. Later hits on it skip the envelope's LZ4 decompress and xxHash3-64 checksum. Every hit still runs the bounded decode, which costs no more than the unbounded one did. The 100-user dict compresses to a 3,171-byte envelope from 23.5KB of MessagePack, so L1 keeps its envelope; no budget gates this large-value hit, and `l1_hit_backed` in [Instruction Budgets](#instruction-budgets) gates the small-value hit on held MessagePack. Encrypted caches always keep the ciphertext envelope, and every hit on one decrypts and decodes.
### Decorator + L1 Hit (Hot Path)
@@ -154,7 +156,7 @@ def expensive_function(user_id: int):
**L1 gives you** (hosted EPYC 7763 figures, [GitHub-Hosted Runner](#github-hosted-runner)):
- No network round trip on a hit
- An L1-only hit in 0.311μs, against 539μs for a Redis L2 hit of the same 100-user dict
-- With an L2 backend configured, an L1 hit of the 100-user dict in 284μs, measured before hits stopped re-checking bytes they had already checked (about 20% less since): it skips the round trip but still deserializes
+- With an L2 backend configured, an L1 hit of the 100-user dict in 284μs, measured before the Rust decode (about half the decode's instructions since): it skips the round trip but still deserializes
### 2. Choose the Right Serializer
diff --git a/rust/src/lib.rs b/rust/src/lib.rs
index ddedef62..ae210ba7 100644
--- a/rust/src/lib.rs
+++ b/rust/src/lib.rs
@@ -18,6 +18,10 @@ pub use cachekit_core::{
#[cfg(feature = "python")]
pub mod python_bindings;
+// StandardSerializer's one-pass bounded MessagePack decode
+#[cfg(feature = "python")]
+pub mod msgpack_decode;
+
// The decorator's lock-free L1-hit counter. Not gated: its Python class is, so the counter's
// own tests run in the pure-Rust test build too.
pub mod hit_counter;
@@ -84,6 +88,8 @@ fn _rust_serializer(_py: Python<'_>, m: &Bound<'_, PyModule>) -> PyResult<()> {
python_bindings::check_msgpack_structure_py,
m
)?)?;
+ // StandardSerializer's read decode: the walk above and msgpack.unpackb in one pass
+ m.add_function(wrap_pyfunction!(python_bindings::unpackb_standard_py, m)?)?;
// Test-only probe of the zero-copy buffer borrow (tests/critical/test_decrypt_buffer_input.py)
m.add_function(wrap_pyfunction!(python_bindings::bytes_view_borrows_py, m)?)?;
diff --git a/rust/src/msgpack_decode.rs b/rust/src/msgpack_decode.rs
new file mode 100644
index 00000000..79d5a776
--- /dev/null
+++ b/rust/src/msgpack_decode.rs
@@ -0,0 +1,485 @@
+//! StandardSerializer's MessagePack decode: cachekit-core's structure walk and the decode in one pass.
+//!
+//! It returns what `msgpack.unpackb(data, raw=False, use_list=True, strict_map_key=True,
+//! object_hook=hook)` returns after `check_msgpack_structure(data, max_depth)` passed, and raises
+//! the same exception class where either refuses (`serializers/base.py::unpackb_bounded`).
+//!
+//! The bounds are the walk's, applied at the same points: before a header's container is
+//! allocated, its depth and the elements every open container still owes are checked against the
+//! input left, so total pre-allocation stays bounded by the input length. Any failure re-runs the
+//! walk over the whole input and raises its error if it has one, which keeps today's precedence:
+//! a bound the walk would refuse anywhere in the document wins over a decode error before it.
+//!
+//! The hook runs only on a map holding a `str` key equal to a marker key. Any other map it would
+//! return unchanged, because each marker test reads its key with `dict.get`.
+
+use std::os::raw::{c_char, c_long};
+use std::ptr;
+
+use cachekit_core::check_msgpack_structure;
+use pyo3::exceptions::PyValueError;
+use pyo3::ffi;
+use pyo3::prelude::*;
+use pyo3::sync::PyOnceLock;
+use pyo3::types::PyBytes;
+
+/// The keys `_standard_object_hook` tests (`standard_serializer.py`).
+const MARKER_KEYS: [&[u8]; 3] = [b"__datetime__", b"__date__", b"__time__"];
+
+/// Strings at most this long go through a per-decode cache, so a repeated one is one object: map
+/// keys in one table, values in another.
+const KEY_CACHE_MAX_LEN: usize = 64;
+const VALUE_CACHE_MAX_LEN: usize = 32;
+const CACHE_BITS: u32 = 8;
+const CACHE_SLOTS: usize = 1 << CACHE_BITS;
+
+/// A Python error is already set (`PyErr::fetch` takes it), or the input broke a bound.
+enum Fail {
+ Py,
+ Bound,
+}
+
+/// An open array or map, holding a strong reference to its container.
+struct Frame {
+ obj: *mut ffi::PyObject,
+ /// Elements still to read: items for an array, keys plus values for a map.
+ owed: u64,
+ /// Array: next index. Map: unused.
+ index: ffi::Py_ssize_t,
+ /// Map: the key read, awaiting its value (strong reference), or null.
+ key: *mut ffi::PyObject,
+ is_map: bool,
+ /// Map: a `str` key equal to a marker key was read.
+ marker: bool,
+}
+
+/// One cached string: its bytes' position in the input, and the `str` (a strong reference).
+#[derive(Clone, Copy)]
+struct StrSlot {
+ start: usize,
+ len: usize,
+ obj: *mut ffi::PyObject,
+}
+
+struct Decoder<'a> {
+ data: &'a [u8],
+ pos: usize,
+ max_depth: usize,
+ /// Elements owed across every open container, the root included (the walk's `pending`).
+ pending: u64,
+ stack: Vec,
+ /// Cached strings; a table slot holds an index into it plus one, or 0 when empty. A slot holds
+ /// one string at a time, a colliding one replacing it, so this never exceeds `2 * CACHE_SLOTS`.
+ cached: Vec,
+ keys: [u16; CACHE_SLOTS],
+ values: [u16; CACHE_SLOTS],
+}
+
+impl Drop for Decoder<'_> {
+ fn drop(&mut self) {
+ // SAFETY: every pointer here is a strong reference this decoder owns, or null; the
+ // thread is attached (the decoder never outlives the pyfunction call).
+ unsafe {
+ for f in self.stack.drain(..) {
+ ffi::Py_XDECREF(f.key);
+ ffi::Py_DECREF(f.obj);
+ }
+ for slot in &self.cached {
+ ffi::Py_DECREF(slot.obj);
+ }
+ }
+ }
+}
+
+/// The cache slot for `bytes`: its first and last eight bytes, multiplied and shifted.
+#[inline]
+fn slot_of(bytes: &[u8]) -> usize {
+ fn word(b: &[u8]) -> u64 {
+ let mut w = [0u8; 8];
+ w[..b.len()].copy_from_slice(b);
+ u64::from_le_bytes(w)
+ }
+ let n = bytes.len();
+ let head = word(&bytes[..n.min(8)]);
+ let tail = if n > 8 { word(&bytes[n - 8..]) } else { 0 };
+ let h = (head ^ tail.rotate_left(29) ^ (n as u64)).wrapping_mul(0x9e37_79b9_7f4a_7c15);
+ (h >> (64 - CACHE_BITS)) as usize
+}
+
+#[inline]
+unsafe fn decode_utf8(bytes: &[u8]) -> *mut ffi::PyObject {
+ ffi::PyUnicode_DecodeUTF8(
+ bytes.as_ptr().cast::(),
+ bytes.len() as ffi::Py_ssize_t,
+ ptr::null(),
+ )
+}
+
+impl<'a> Decoder<'a> {
+ fn new(data: &'a [u8], max_depth: usize) -> Self {
+ Self {
+ data,
+ pos: 0,
+ max_depth,
+ pending: 1,
+ stack: Vec::new(),
+ cached: Vec::new(),
+ keys: [0; CACHE_SLOTS],
+ values: [0; CACHE_SLOTS],
+ }
+ }
+
+ /// The walk's per-element accounting, for an element whose prefix is consumed: `payload`
+ /// bytes follow and it opens a container of `children` elements.
+ #[inline]
+ fn account(&mut self, payload: u64, children: u64) -> Result<(), Fail> {
+ let remaining = (self.data.len() - self.pos) as u64;
+ if payload > remaining {
+ return Err(Fail::Bound);
+ }
+ self.pending += children;
+ if self.pending > remaining - payload {
+ return Err(Fail::Bound);
+ }
+ Ok(())
+ }
+
+ /// Consume `len` payload bytes `account` admitted.
+ #[inline]
+ fn take(&mut self, len: u64) -> &'a [u8] {
+ let start = self.pos;
+ // `account` checked len <= data.len() - pos, so this cannot truncate or overrun.
+ self.pos += len as usize;
+ &self.data[start..self.pos]
+ }
+
+ /// Whether the next element read is a map key.
+ #[inline]
+ fn at_key(&self) -> bool {
+ matches!(self.stack.last(), Some(f) if f.is_map && f.key.is_null())
+ }
+
+ fn decode(
+ &mut self,
+ py: Python<'_>,
+ hook: &Bound<'_, PyAny>,
+ ) -> Result<*mut ffi::PyObject, Fail> {
+ loop {
+ let marker = *self.data.get(self.pos).ok_or(Fail::Bound)?;
+ self.pos += 1;
+ self.pending -= 1;
+ if let Some(f) = self.stack.last_mut() {
+ f.owed -= 1;
+ }
+ // SAFETY: the thread is attached; every constructor returns a new reference or null
+ // with an exception set, which `check` turns into Fail::Py.
+ let mut value = unsafe {
+ match marker {
+ 0x00..=0x7f => self.scalar(ffi::PyLong_FromLong(c_long::from(marker)))?,
+ 0xe0..=0xff => self.scalar(ffi::PyLong_FromLong(c_long::from(marker as i8)))?,
+ 0xc0 => self.scalar(ffi::Py_NewRef(ffi::Py_None()))?,
+ 0xc2 => self.scalar(ffi::Py_NewRef(ffi::Py_False()))?,
+ 0xc3 => self.scalar(ffi::Py_NewRef(ffi::Py_True()))?,
+ 0xa0..=0xbf => self.string(u64::from(marker & 0x1f))?,
+ 0xd9 => self.prefixed(1).and_then(|n| self.string(n))?,
+ 0xda => self.prefixed(2).and_then(|n| self.string(n))?,
+ 0xdb => self.prefixed(4).and_then(|n| self.string(n))?,
+ 0xc4 => self.prefixed(1).and_then(|n| self.binary(n))?,
+ 0xc5 => self.prefixed(2).and_then(|n| self.binary(n))?,
+ 0xc6 => self.prefixed(4).and_then(|n| self.binary(n))?,
+ 0xc7 => self.prefixed(1).and_then(|n| self.ext(py, n))?,
+ 0xc8 => self.prefixed(2).and_then(|n| self.ext(py, n))?,
+ 0xc9 => self.prefixed(4).and_then(|n| self.ext(py, n))?,
+ 0xd4..=0xd8 => self.ext(py, 1u64 << (marker - 0xd4))?,
+ 0xca..=0xd3 => self.number(marker)?,
+ 0x90..=0x9f => self.open(false, u64::from(marker & 0x0f))?,
+ 0xdc => self.prefixed(2).and_then(|n| self.open(false, n))?,
+ 0xdd => self.prefixed(4).and_then(|n| self.open(false, n))?,
+ 0x80..=0x8f => self.open(true, u64::from(marker & 0x0f))?,
+ 0xde => self.prefixed(2).and_then(|n| self.open(true, n))?,
+ 0xdf => self.prefixed(4).and_then(|n| self.open(true, n))?,
+ 0xc1 => return Err(Fail::Bound),
+ }
+ };
+ // A non-empty container was pushed: read its first element.
+ if value.is_null() {
+ continue;
+ }
+ // Hand the value to its container, closing every container it completes.
+ loop {
+ let Some(f) = self.stack.last_mut() else {
+ return Ok(value);
+ };
+ // SAFETY: `f.obj` is a strong reference to a list or dict this decoder built;
+ // `value` and `f.key` are strong references the set calls consume.
+ unsafe {
+ if !f.is_map {
+ if ffi::PyList_SetItem(f.obj, f.index, value) != 0 {
+ return Err(Fail::Py);
+ }
+ f.index += 1;
+ } else if f.key.is_null() {
+ f.key = value;
+ break;
+ } else {
+ let key = std::mem::replace(&mut f.key, ptr::null_mut());
+ let ok = key_admitted(key);
+ let rc = if ok {
+ ffi::PyDict_SetItem(f.obj, key, value)
+ } else {
+ 0
+ };
+ if !ok {
+ set_key_error(py, key);
+ }
+ ffi::Py_DECREF(key);
+ ffi::Py_DECREF(value);
+ if !ok || rc != 0 {
+ return Err(Fail::Py);
+ }
+ }
+ if f.owed > 0 {
+ break;
+ }
+ let done = self.stack.pop().expect("a frame was just read");
+ value = done.obj;
+ if done.marker {
+ let obj = Bound::from_owned_ptr(py, value);
+ value = hook.call1((obj,)).map_err(|e| restore(py, e))?.into_ptr();
+ }
+ }
+ }
+ }
+ }
+
+ /// A `width`-byte big-endian length prefix.
+ #[inline]
+ fn prefixed(&mut self, width: usize) -> Result {
+ let end = self
+ .pos
+ .checked_add(width)
+ .filter(|e| *e <= self.data.len())
+ .ok_or(Fail::Bound)?;
+ let n = big_endian(&self.data[self.pos..end]);
+ self.pos = end;
+ Ok(n)
+ }
+
+ /// A one-byte element, already built as `obj`.
+ #[inline]
+ fn scalar(&mut self, obj: *mut ffi::PyObject) -> Result<*mut ffi::PyObject, Fail> {
+ // Accounting first: on a bound failure the object is dropped, not leaked.
+ if let Err(e) = self.account(0, 0) {
+ // SAFETY: obj is a new reference or null.
+ unsafe { ffi::Py_XDECREF(obj) };
+ return Err(e);
+ }
+ check(obj)
+ }
+
+ unsafe fn string(&mut self, len: u64) -> Result<*mut ffi::PyObject, Fail> {
+ self.account(len, 0)?;
+ let at_key = self.at_key();
+ let start = self.pos;
+ let bytes = self.take(len);
+ if at_key && MARKER_KEYS.contains(&bytes) {
+ if let Some(f) = self.stack.last_mut() {
+ f.marker = true;
+ }
+ }
+ let max = if at_key {
+ KEY_CACHE_MAX_LEN
+ } else {
+ VALUE_CACHE_MAX_LEN
+ };
+ if bytes.len() > max {
+ return check(decode_utf8(bytes));
+ }
+ let i = slot_of(bytes);
+ let held = if at_key { self.keys[i] } else { self.values[i] };
+ if let Some(s) = (held as usize).checked_sub(1).map(|j| self.cached[j]) {
+ if s.len == bytes.len() && &self.data[s.start..s.start + s.len] == bytes {
+ return Ok(ffi::Py_NewRef(s.obj));
+ }
+ }
+ // Not interned, unlike msgpack-python's keys: equal either way, and interning a distinct key
+ // costs more than every later lookup of it saves.
+ let obj = check(decode_utf8(bytes))?;
+ let slot = StrSlot {
+ start,
+ len: bytes.len(),
+ obj: ffi::Py_NewRef(obj),
+ };
+ if held == 0 {
+ self.cached.push(slot);
+ // At most 2 * CACHE_SLOTS entries (see `cached`), so the index fits.
+ *(if at_key {
+ &mut self.keys[i]
+ } else {
+ &mut self.values[i]
+ }) = self.cached.len() as u16;
+ } else {
+ let old = std::mem::replace(&mut self.cached[held as usize - 1], slot);
+ ffi::Py_DECREF(old.obj);
+ }
+ Ok(obj)
+ }
+
+ unsafe fn binary(&mut self, len: u64) -> Result<*mut ffi::PyObject, Fail> {
+ self.account(len, 0)?;
+ let bytes = self.take(len);
+ check(ffi::PyBytes_FromStringAndSize(
+ bytes.as_ptr().cast::(),
+ bytes.len() as ffi::Py_ssize_t,
+ ))
+ }
+
+ unsafe fn number(&mut self, marker: u8) -> Result<*mut ffi::PyObject, Fail> {
+ let width = 1u64 << (marker & 0x03);
+ self.account(width, 0)?;
+ let raw = big_endian(self.take(width));
+ check(match marker {
+ 0xca => ffi::PyFloat_FromDouble(f64::from(f32::from_bits(raw as u32))),
+ 0xcb => ffi::PyFloat_FromDouble(f64::from_bits(raw)),
+ 0xcc..=0xce => ffi::PyLong_FromLongLong(raw as i64),
+ 0xcf => ffi::PyLong_FromUnsignedLongLong(raw),
+ 0xd0 => ffi::PyLong_FromLong(c_long::from(raw as u8 as i8)),
+ 0xd1 => ffi::PyLong_FromLong(c_long::from(raw as u16 as i16)),
+ 0xd2 => ffi::PyLong_FromLong(c_long::from(raw as u32 as i32)),
+ _ => ffi::PyLong_FromLongLong(raw as i64),
+ })
+ }
+
+ /// An ext of `len` data bytes after its type byte, decoded as msgpack-python's defaults do:
+ /// type -1 to `msgpack.Timestamp` (through its own `from_bytes`), any other to
+ /// `msgpack.ExtType(code, data)`. cachekit never writes an ext, so this path is not hot.
+ unsafe fn ext(&mut self, py: Python<'_>, len: u64) -> Result<*mut ffi::PyObject, Fail> {
+ self.account(len + 1, 0)?;
+ let code = self.take(1)[0] as i8;
+ let data = self.take(len);
+ let made = if code == -1 {
+ msgpack_attr(py, "Timestamp")
+ .and_then(|t| t.call_method1("from_bytes", (PyBytes::new(py, data),)))
+ } else {
+ msgpack_attr(py, "ExtType").and_then(|t| t.call1((code, PyBytes::new(py, data))))
+ };
+ made.map(Bound::into_ptr).map_err(|e| restore(py, e))
+ }
+
+ /// Open a container of `n` elements (`2n` slots for a map). Returns null after pushing a
+ /// non-empty one, or the empty container.
+ unsafe fn open(&mut self, is_map: bool, n: u64) -> Result<*mut ffi::PyObject, Fail> {
+ // An empty container is still a level, so depth is checked before the n > 0 push.
+ if self.stack.len() >= self.max_depth {
+ return Err(Fail::Bound);
+ }
+ let owed = if is_map { 2 * n } else { n };
+ self.account(0, owed)?;
+ // n <= the input length now, so it fits Py_ssize_t.
+ let obj = check(if is_map {
+ ffi::PyDict_New()
+ } else {
+ ffi::PyList_New(n as ffi::Py_ssize_t)
+ })?;
+ if n == 0 {
+ return Ok(obj);
+ }
+ self.stack.push(Frame {
+ obj,
+ owed,
+ index: 0,
+ key: ptr::null_mut(),
+ is_map,
+ marker: false,
+ });
+ Ok(ptr::null_mut())
+ }
+}
+
+#[inline]
+fn big_endian(bytes: &[u8]) -> u64 {
+ bytes.iter().fold(0u64, |acc, b| (acc << 8) | u64::from(*b))
+}
+
+#[inline]
+fn check(obj: *mut ffi::PyObject) -> Result<*mut ffi::PyObject, Fail> {
+ if obj.is_null() {
+ Err(Fail::Py)
+ } else {
+ Ok(obj)
+ }
+}
+
+/// Put a fetched error back so the `Fail::Py` path finds it.
+fn restore(py: Python<'_>, e: PyErr) -> Fail {
+ e.restore(py);
+ Fail::Py
+}
+
+/// `strict_map_key=True`: a map key must be exactly `str` or `bytes`.
+unsafe fn key_admitted(key: *mut ffi::PyObject) -> bool {
+ ffi::PyUnicode_CheckExact(key) != 0 || ffi::PyBytes_CheckExact(key) != 0
+}
+
+unsafe fn set_key_error(py: Python<'_>, key: *mut ffi::PyObject) {
+ let name = Bound::from_borrowed_ptr(py, key)
+ .get_type()
+ .name()
+ .map(|n| n.to_string())
+ .unwrap_or_else(|_| "object".to_owned());
+ PyValueError::new_err(format!(
+ "{name:.100} is not allowed for map key when strict_map_key=True"
+ ))
+ .restore(py);
+}
+
+fn msgpack_attr<'py>(py: Python<'py>, name: &str) -> PyResult> {
+ static MSGPACK: PyOnceLock> = PyOnceLock::new();
+ MSGPACK
+ .get_or_try_init(py, || py.import("msgpack").map(Bound::unbind))?
+ .bind(py)
+ .getattr(name)
+}
+
+/// Decode `data` as `unpackb_bounded(data, raw=False, use_list=True, object_hook=hook)` does
+/// (see the module docs), calling `hook` only on maps holding a marker key.
+pub fn unpackb_standard(
+ py: Python<'_>,
+ data: &[u8],
+ max_depth: usize,
+ hook: &Bound<'_, PyAny>,
+) -> PyResult> {
+ let mut decoder = Decoder::new(data, max_depth);
+ let outcome = decoder.decode(py, hook);
+ let consumed = decoder.pos;
+ drop(decoder);
+ let failure = match outcome {
+ Ok(value) => {
+ // SAFETY: a strong reference the decode returned.
+ let value = unsafe { Bound::from_owned_ptr(py, value) };
+ if consumed == data.len() {
+ return Ok(value.unbind());
+ }
+ let extra = msgpack_attr(py, "ExtraData")?;
+ return Err(PyErr::from_value(
+ extra.call1((value, PyBytes::new(py, &data[consumed..])))?,
+ ));
+ }
+ Err(Fail::Py) => PyErr::fetch(py),
+ // Only reachable if this decoder's inline accounting drifts from the walk: the walk below
+ // then accepts what the decode refused. Fails closed, without the guard's own prefix, so a
+ // drift cannot pass for a refused hostile document.
+ Err(Fail::Bound) => PyValueError::new_err(
+ "MessagePack decode bound disagrees with the structure walk (internal error)",
+ ),
+ };
+ // The walk's verdict wins over anything the decode raised, as it does when it runs first.
+ check_msgpack_structure(data, max_depth).map_err(structure_error)?;
+ Err(failure)
+}
+
+/// The ValueError for a document `check_msgpack_structure` refused: `Unpack failed: MessagePack
+/// document `, the text the bounds tests pin as the guard's own.
+pub(crate) fn structure_error(what: impl std::fmt::Display) -> PyErr {
+ PyValueError::new_err(format!("Unpack failed: MessagePack document {what}"))
+}
diff --git a/rust/src/python_bindings.rs b/rust/src/python_bindings.rs
index eda090d7..cc60a133 100644
--- a/rust/src/python_bindings.rs
+++ b/rust/src/python_bindings.rs
@@ -150,14 +150,30 @@ pub fn check_msgpack_structure_py(
max_depth: usize,
) -> PyResult<()> {
let view = bytes_view(py, data)?;
- check_msgpack_structure(view.as_slice(), max_depth).map_err(|what| {
- PyValueError::new_err(format!("Unpack failed: MessagePack document {what}"))
- })
+ check_msgpack_structure(view.as_slice(), max_depth)
+ .map_err(crate::msgpack_decode::structure_error)
+}
+
+/// Decode one untrusted MessagePack document as StandardSerializer reads it, under the bounds
+/// `check_msgpack_structure` enforces, in one pass (`crate::msgpack_decode`). `object_hook` runs
+/// only on maps holding a `__datetime__`, `__date__` or `__time__` str key. `data` goes through
+/// the strict `bytes_view`, so a `memoryview` of any format but `B`/`c` raises BufferError where
+/// `unpackb_bounded` accepts it: callers pass `bytes`, or normalise with `immutable_buffer` first.
+#[pyfunction]
+#[pyo3(name = "unpackb_standard")]
+pub fn unpackb_standard_py(
+ py: Python<'_>,
+ data: &Bound<'_, PyAny>,
+ max_depth: usize,
+ object_hook: &Bound<'_, PyAny>,
+) -> PyResult> {
+ let view = bytes_view(py, data)?;
+ crate::msgpack_decode::unpackb_standard(py, view.as_slice(), max_depth, object_hook)
}
/// Whether `bytes_view` borrows `obj` (True) or copies it (False). Test-only: it pins what
/// `bytes_view` decides for an input, which tracemalloc cannot see (a Rust-side copy never
-/// touches the Python heap). It covers `retrieve` and `check_msgpack_structure`, and the exact
+/// touches the Python heap). It covers `retrieve`, `check_msgpack_structure` and `unpackb_standard`, and the exact
/// `bytes` and `B`/`c` `memoryview` inputs the decrypt bindings read through `ciphertext_view`;
/// that function's `bytes(obj)` route for every other input is outside it. It does not pin how a
/// caller uses the view: that the callers pass `as_slice()` on without copying it is held by
@@ -265,8 +281,8 @@ use zeroize::Zeroizing;
/// hands the wrapper, go through `bytes_view` (borrowed when the storage is immutable `bytes`).
/// Every other input (a `bytes` subclass, a `bytearray`, a custom exporter) is read by
/// `bytes(obj)` itself, so it gets exactly the semantics of the `bytes(data)` coercion these
-/// bindings replaced: `__bytes__` first, then a single export. `retrieve` and
-/// `check_msgpack_structure` keep the strict `bytes_view` on purpose.
+/// bindings replaced: `__bytes__` first, then a single export. `retrieve`,
+/// `check_msgpack_structure` and `unpackb_standard` keep the strict `bytes_view` on purpose.
#[cfg(feature = "encryption")]
fn ciphertext_view<'py>(py: Python<'py>, obj: &Bound<'py, PyAny>) -> PyResult> {
if !(obj.is_exact_instance_of::() || obj.is_exact_instance_of::()) {
diff --git a/src/cachekit/cache_handler.py b/src/cachekit/cache_handler.py
index 3ea8d7c5..fb3a791a 100644
--- a/src/cachekit/cache_handler.py
+++ b/src/cachekit/cache_handler.py
@@ -64,7 +64,7 @@
KeyringConfigurationError,
require_cross_sdk_serializer,
)
-from cachekit.serializers.standard_serializer import KeptEntry, StandardSerializer, VerifiedMsgpack, WalkedEnvelope
+from cachekit.serializers.standard_serializer import StandardSerializer, VerifiedMsgpack
from cachekit.serializers.wrapper import SerializationWrapper
if TYPE_CHECKING:
@@ -1351,47 +1351,40 @@ def write_serialized_to(self, sink: BinaryIO, data: Any) -> None:
def deserialize_l1(
self,
- data: bytes | VerifiedMsgpack | WalkedEnvelope,
+ data: bytes | VerifiedMsgpack,
cache_key: str = "",
args: tuple[Any, ...] = (),
kwargs: dict[str, Any] | None = None,
- ) -> tuple[Any, VerifiedMsgpack | WalkedEnvelope | None]:
+ ) -> tuple[Any, VerifiedMsgpack | None]:
""":meth:`deserialize_data` for an entry read from L1, plus what L1 should hold in its place, if anything.
- When this handler keeps MessagePack (``keeps_msgpack_in_l1``) and the entry is its own plaintext
- envelope, that is what this decode verified and walked: the MessagePack (:class:`VerifiedMsgpack`) when it
- is no longer than the entry, else the entry marked as walked (:class:`WalkedEnvelope`). Either records
- whether the MessagePack may hold a marker map. The caller's swap (``L1Cache.shrink_if_held``) costs no
- second decode, and later hits skip the structure walk, the object hook when no marker map can be present,
- and, for VerifiedMsgpack, retrieve(). Everything else gets None: a kept entry decodes that way, and any
- other entry goes to deserialize_data, which refuses another handler's kept entry. Raises as
+ When this handler keeps MessagePack (``keeps_msgpack_in_l1``), the entry is its own plaintext envelope, and
+ the MessagePack this decode verified is no longer than the entry, that MessagePack (:class:`VerifiedMsgpack`),
+ for the caller to swap in (``L1Cache.shrink_if_held``) at no second decode: later hits skip the envelope's
+ retrieve(), its decompress and checksum. Everything else gets None: a kept entry decodes that way, a longer
+ MessagePack would grow the entry so L1 keeps the envelope, and any other entry goes to deserialize_data,
+ which refuses another handler's kept entry. Every decode, kept or not, runs the bounded decode. Raises as
deserialize_data.
Examples:
>>> handler = CacheSerializationHandler()
>>> envelope = handler.serialize_data("hello", cache_key="greeting")
>>> value, entry = handler.deserialize_l1(envelope, cache_key="greeting")
- >>> value, type(entry).__name__, len(entry) <= len(envelope), entry.hook
- ('hello', 'VerifiedMsgpack', True, False)
- >>> handler.deserialize_l1(entry, cache_key="greeting") # already walked MessagePack
+ >>> value, type(entry).__name__, len(entry) <= len(envelope)
+ ('hello', 'VerifiedMsgpack', True)
+ >>> handler.deserialize_l1(entry, cache_key="greeting") # already verified MessagePack
('hello', None)
>>> compressible = handler.serialize_data("ab" * 600, cache_key="k")
- >>> value, entry = handler.deserialize_l1(compressible, cache_key="k") # its MessagePack is longer
- >>> type(entry).__name__, entry.frame is compressible
- ('WalkedEnvelope', True)
- >>> handler.deserialize_l1(entry, cache_key="k") == (value, None)
+ >>> handler.deserialize_l1(compressible, cache_key="k")[1] is None # its MessagePack is longer: L1 keeps the envelope
True
"""
# Inline, not helpers: this is every L1 hit's decode, and an envelope it keeps must cost what
# deserialize_data costs.
if not self.keeps_msgpack_in_l1:
return self.deserialize_data(data, cache_key, args, kwargs), None
- # Only L1 holds these types, and only such a handler swapped them in, after a decode that walked them.
+ # Only L1 holds this type, and only such a handler swapped it in, after a decode that accepted it.
if type(data) is VerifiedMsgpack:
- return self._base_serializer._deserialize_verified(data.data, data.hook), None # type: ignore[union-attr]
- if type(data) is WalkedEnvelope:
- payload = SerializationWrapper.unwrap_metadata(data.frame)[0] # parsed when it was walked: cannot fail
- return self._base_serializer._deserialize_walked(payload, data.hook), None # type: ignore[union-attr]
+ return self._base_serializer._deserialize_verified(data.data), None # type: ignore[union-attr]
try:
payload, metadata, serializer_name = SerializationWrapper.unwrap_metadata(data)
except (AttributeError, KeyError, TypeError, ValueError):
@@ -1402,11 +1395,11 @@ def deserialize_l1(
if serializer_name != self._serializer_string_name or metadata.encrypted or not metadata.compressed:
return self.deserialize_data(data, cache_key, args, kwargs), None
value, kept = self._base_serializer._deserialize_keeping_msgpack(payload) # type: ignore[union-attr]
- return value, kept if len(kept) <= len(data) else WalkedEnvelope(data, kept.hook)
+ return value, kept if len(kept) <= len(data) else None
def deserialize_data(
self,
- data: str | bytes | memoryview | KeptEntry,
+ data: str | bytes | memoryview | VerifiedMsgpack,
cache_key: str = "",
args: tuple[Any, ...] = (),
kwargs: dict[str, Any] | None = None,
@@ -1483,7 +1476,7 @@ def deserialize_data(
True
"""
if self.interop_mode:
- # A KeptEntry here is another handler's swapped L1 entry: the bytes-only decode refuses it.
+ # A VerifiedMsgpack here is another handler's swapped L1 entry: the bytes-only decode refuses it.
return self._deserialize_interop(data, cache_key) # pyright: ignore[reportArgumentType]
try:
@@ -1495,11 +1488,11 @@ def deserialize_data(
# A custom SerializerProtocol may write to the metadata it is handed, so it gets its own
# copy rather than the header memo's shared, read-only instance.
serialized_data, metadata, serializer_name = SerializationWrapper.unwrap_metadata(
- data, # pyright: ignore[reportArgumentType] # a KeptEntry raises here and is refused below
+ data, # pyright: ignore[reportArgumentType] # a VerifiedMsgpack raises here and is refused below
shared=not self._custom_serializer,
)
except (AttributeError, KeyError, TypeError, ValueError) as e:
- if self.encryption and type(data) in (VerifiedMsgpack, WalkedEnvelope):
+ if self.encryption and type(data) is VerifiedMsgpack:
# A plaintext handler's swapped L1 entry under this key: refused as the plaintext
# envelope it replaced would be, and counted the same way (suspicious_envelope).
raise SuspiciousCacheEntryError(
diff --git a/src/cachekit/decorators/wrapper.py b/src/cachekit/decorators/wrapper.py
index 3be9479f..4448504f 100644
--- a/src/cachekit/decorators/wrapper.py
+++ b/src/cachekit/decorators/wrapper.py
@@ -78,7 +78,7 @@
from ..backends.base import BaseBackend, LockableBackend
from ..serializers.base import SerializerProtocol
- from ..serializers.standard_serializer import KeptEntry
+ from ..serializers.standard_serializer import VerifiedMsgpack
def _resolve_lazy_backend() -> BaseBackend:
@@ -1086,7 +1086,7 @@ def _on_l2_deserialize_error(error: Exception, key: str) -> None:
# applies only when this namespace's cache is first created — namespaces share one
# L1Cache, so give functions with distinct budgets distinct namespaces (issue #163).
_l1_cache = get_l1_cache(namespace or "default", max_size_mb=l1_max_size_mb) if l1_enabled else None
- # Whether an L1 hit may swap its entry for one marked as walked (deserialize_l1). Every other handler's
+ # Whether an L1 hit may swap its entry for its verified MessagePack (deserialize_l1). Every other handler's
# hit decodes exactly as before, at no extra cost.
_l1_keeps_msgpack = serialization_handler.keeps_msgpack_in_l1
@@ -1837,13 +1837,13 @@ def _flight_tenant(args: tuple[Any, ...], kwargs: dict[str, Any], cache_key: str
return None
def _decode_shared(
- entry: bytes | KeptEntry | None, cache_key: str, args: tuple[Any, ...], kwargs: dict[str, Any]
+ entry: bytes | VerifiedMsgpack | None, cache_key: str, args: tuple[Any, ...], kwargs: dict[str, Any]
) -> tuple[bool, Any]:
"""A joined backed-mode caller's own copy of the trip's envelope, or of the L1 entry the re-check
- found (which may be a KeptEntry), decoded as an L1 hit would be: every caller gets its own
+ found (which may be a VerifiedMsgpack), decoded as an L1 hit would be: every caller gets its own
object, and the decrypt checks bind to this caller's tenant. It never swaps the L1 entry, so
bytes go to deserialize_data, which decodes them as deserialize_l1 does without building a
- KeptEntry for it to discard.
+ VerifiedMsgpack for it to discard.
(False, None) when there is no entry or it does not decode for this caller, who then takes
its own trip, unshared: its probe slot went back on joining, so it is admitted afresh.
@@ -2182,7 +2182,7 @@ def _sync_call(args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any: # noqa: P
l1_value, l1_kept = operation_handler.serialization_handler.deserialize_l1(
l1_bytes, cache_key, args, kwargs
)
- if l1_kept is not None: # walked and marked: later hits skip the walk (deserialize_l1)
+ if l1_kept is not None: # verified and kept: later hits skip retrieve() (deserialize_l1)
_l1_cache.shrink_if_held(cache_key, l1_bytes, l1_kept)
else: # as before; a swapped entry fails: SuspiciousCacheEntryError if encrypted, else a corrupt frame
l1_value = operation_handler.serialization_handler.deserialize_data(
@@ -2344,9 +2344,9 @@ def _start_trip() -> tuple[Any, bytes | None]:
decoded, value = _decode_shared(envelope, cache_key, args, kwargs)
return value if decoded else _own_trip_sync(cache_key, args, kwargs, twin_key)[0]
- def _peek_l1(cache_key: str) -> tuple[bool, tuple[None, bytes | KeptEntry | None]]:
+ def _peek_l1(cache_key: str) -> tuple[bool, tuple[None, bytes | VerifiedMsgpack | None]]:
"""The sync backed flight's re-check (ThreadFlights.run): the L1 entry a trip that settled since
- the caller's own L1 lookup left (its envelope, or the KeptEntry a hit swapped in), for a joined
+ the caller's own L1 lookup left (its envelope, or the VerifiedMsgpack a hit swapped in), for a joined
caller to decode. Never an L2 read: it runs under the flight map's lock."""
assert _l1_cache is not None # noqa: S101 - passed only with L1 on
entry = _l1_cache.peek(cache_key)
diff --git a/src/cachekit/l1_cache.py b/src/cachekit/l1_cache.py
index cb5177f4..ea653d7d 100644
--- a/src/cachekit/l1_cache.py
+++ b/src/cachekit/l1_cache.py
@@ -21,7 +21,7 @@
from cachekit.hash_utils import redact_error_for_log, redact_key_for_log
if TYPE_CHECKING:
- from cachekit.serializers.standard_serializer import KeptEntry
+ from cachekit.serializers.standard_serializer import VerifiedMsgpack
# Default L1 entry lifetime when the caller supplies no TTL. Shared with the
# decorator's freshness backfill bound: the server's Fresh-For may only ever
@@ -55,12 +55,12 @@ class CacheEntry:
"""L1 cache entry with value, TTL, and size tracking.
Storage: Stores bytes (encrypted or plaintext msgpack) for unified
- encrypted-at-rest architecture, or the KeptEntry a hit swapped in for a
+ encrypted-at-rest architecture, or the VerifiedMsgpack a hit swapped in for a
plaintext entry. Decryption/deserialization happens at read time in
CacheHandler, not storage time.
"""
- value: "bytes | KeptEntry" # a KeptEntry once a hit swapped it in (shrink_if_held)
+ value: "bytes | VerifiedMsgpack" # a VerifiedMsgpack once a hit swapped it in (shrink_if_held)
expires_at: float
size_bytes: int
# This entry's tuple in its state's expiry heap. A tuple is live only while it is this very object,
@@ -161,7 +161,7 @@ class L1Cache:
- LRU eviction when memory limit reached
- In-process lookups, with no network round trip
- Background TTL synchronization
- - Stores bytes (encrypted or plaintext msgpack), or the KeptEntry a hit swapped in, not Python objects
+ - Stores bytes (encrypted or plaintext msgpack), or the VerifiedMsgpack a hit swapped in, not Python objects
Storage Architecture:
- Stores serialized bytes for unified encrypted-at-rest
@@ -211,21 +211,21 @@ def __init__(
ttl_buffer_seconds,
)
- def _estimate_size(self, value: "bytes | KeptEntry") -> int:
+ def _estimate_size(self, value: "bytes | VerifiedMsgpack") -> int:
"""Estimate memory size of an entry's value.
Simplified from recursive sys.getsizeof() - faster and more accurate
- for bytes storage. A KeptEntry's length is that of the bytes it holds.
+ for bytes storage. A VerifiedMsgpack's length is that of the bytes it holds.
Args:
- value: Bytes, or a KeptEntry, to estimate size for
+ value: Bytes, or a VerifiedMsgpack, to estimate size for
Returns:
Size in bytes
"""
return len(value)
- def get(self, key: str) -> "tuple[bool, Optional[bytes | KeptEntry]]":
+ def get(self, key: str) -> "tuple[bool, Optional[bytes | VerifiedMsgpack]]":
"""Get value from L1 cache if present and not expired.
Args:
@@ -233,7 +233,7 @@ def get(self, key: str) -> "tuple[bool, Optional[bytes | KeptEntry]]":
Returns:
Tuple of (found, value) where found is True if hit.
- Value is bytes (encrypted or plaintext msgpack), or the KeptEntry a hit swapped in
+ Value is bytes (encrypted or plaintext msgpack), or the VerifiedMsgpack a hit swapped in
(shrink_if_held), not deserialized object.
Caller (CacheHandler) is responsible for decryption/deserialization.
"""
@@ -259,8 +259,8 @@ def get(self, key: str) -> "tuple[bool, Optional[bytes | KeptEntry]]":
return True, entry.value
- def peek(self, key: str) -> "Optional[bytes | KeptEntry]":
- """The value ``get`` would return (bytes, or the KeptEntry a hit swapped in), without counting a hit or miss or moving the entry in LRU order.
+ def peek(self, key: str) -> "Optional[bytes | VerifiedMsgpack]":
+ """The value ``get`` would return (bytes, or the VerifiedMsgpack a hit swapped in), without counting a hit or miss or moving the entry in LRU order.
For a second lookup within one call, such as the single-flight re-check after a miss, which
the call's first lookup already counted. An expired entry reads as absent and is left for
@@ -276,7 +276,7 @@ def peek(self, key: str) -> "Optional[bytes | KeptEntry]":
entry = s.cache.get(key)
return None if entry is None or entry.is_expired() else entry.value
- def shrink_if_held(self, key: str, held: "bytes | KeptEntry", smaller: "KeptEntry") -> bool:
+ def shrink_if_held(self, key: str, held: "bytes | VerifiedMsgpack", smaller: "VerifiedMsgpack") -> bool:
"""Hold ``smaller`` in place of ``held``, the very object a read of ``key`` returned, if the entry still
holds it and ``smaller`` is no longer. The entry keeps its expiry and LRU place. Returns whether it did.
diff --git a/src/cachekit/serializers/base.py b/src/cachekit/serializers/base.py
index 467eae66..47b07d2a 100644
--- a/src/cachekit/serializers/base.py
+++ b/src/cachekit/serializers/base.py
@@ -365,7 +365,10 @@ class EnvelopeShapeError(SerializationError):
# ---------------------------------------------------------------------------
#: cachekit's own nesting ceiling, enforced by cachekit-core's ``check_msgpack_structure``
-#: walk (through the Rust extension) before msgpack-python ever sees the document. Every
+#: walk (through the Rust extension) before msgpack-python ever sees the document, and by
+#: StandardSerializer's one-pass Rust decode (``unpackb_standard``), which repeats the walk's accounting
+#: inline and runs the walk itself only after a failure (tests/unit/test_standard_decode.py holds the
+#: two to the same verdicts). Every
#: array or map header on the path counts as a level, an empty one included. Two
#: constraints pin it: the protocol requires every SDK's bound to sit in 32..=1024, and
#: it must not exceed msgpack-python's C unpacker stack (a document at the ceiling has to
diff --git a/src/cachekit/serializers/standard_serializer.py b/src/cachekit/serializers/standard_serializer.py
index b4dac7ac..8b3c0791 100644
--- a/src/cachekit/serializers/standard_serializer.py
+++ b/src/cachekit/serializers/standard_serializer.py
@@ -25,15 +25,15 @@
import msgpack
-from cachekit._rust_serializer import ByteStorage
+from cachekit._rust_serializer import ByteStorage, unpackb_standard
from .base import (
+ MSGPACK_MAX_NESTING,
PAYLOAD_DECODE_ERRORS,
SerializationError,
SerializationFormat,
SerializationMetadata,
immutable_buffer,
- unpackb_bounded,
)
# Every envelope ByteStorage has written opens with a fixarray-4 (0x94) and then its payload slot's
@@ -50,52 +50,24 @@
@final # the read paths test `type(data) is VerifiedMsgpack`
class VerifiedMsgpack:
- """MessagePack a successful ``ByteStorage.retrieve()`` returned and the structure walk
- (``check_msgpack_structure``) passed, which an L1 entry holds in place of its envelope
- (``CacheSerializationHandler.deserialize_l1``). It decodes without another retrieve() or walk, and
- without the object hook when ``hook`` is False (:func:`_may_hold_marker`).
+ """MessagePack a successful ``ByteStorage.retrieve()`` returned and a decode in this process accepted,
+ which an L1 entry holds in place of its envelope (``CacheSerializationHandler.deserialize_l1``). It decodes
+ without another retrieve().
- The type is the mark, not the bytes: no backend returns this type, so bytes from L2 never take
- the unverified decode, whatever they hold. It holds retrieve()'s own bytes object rather than
+ The type is the mark, not the bytes: no backend returns this type, so bytes from L2 never skip
+ retrieve(), whatever they hold. It holds retrieve()'s own bytes object rather than
copying it into a bytes subclass, and its length is theirs, which L1 accounts.
"""
- __slots__ = ("data", "hook")
+ __slots__ = ("data",)
- def __init__(self, data: bytes, hook: bool) -> None:
+ def __init__(self, data: bytes) -> None:
self.data = data
- self.hook = hook
def __len__(self) -> int:
return len(self.data)
-@final # the read paths test `type(data) is WalkedEnvelope`
-class WalkedEnvelope:
- """A plaintext envelope frame whose MessagePack a decode in this process verified and walked, as
- :class:`VerifiedMsgpack` is, which an L1 entry holds in place of its frame bytes when the MessagePack is
- longer than the frame. It decodes with retrieve() but without the walk, and without the object hook when
- ``hook`` is False.
-
- The walk's answer cannot change: the frame is an immutable bytes object and retrieve() is deterministic.
- Marked by its type for the reason VerifiedMsgpack is. Its length is the frame's, so swapping it in for the
- frame leaves L1's accounting unchanged.
- """
-
- __slots__ = ("frame", "hook")
-
- def __init__(self, frame: bytes, hook: bool) -> None:
- self.frame = frame
- self.hook = hook
-
- def __len__(self) -> int:
- return len(self.frame)
-
-
-#: An L1 entry a hit swapped in for the envelope it decoded (``CacheSerializationHandler.deserialize_l1``).
-KeptEntry = VerifiedMsgpack | WalkedEnvelope
-
-
# Error message constants for unsupported types (Task 2)
NUMPY_ERROR_MESSAGE = (
"StandardSerializer does not support NumPy arrays (Python-specific type). "
@@ -207,10 +179,10 @@ def _standard_object_hook(obj: Any) -> Any:
Returns:
Restored Python object or original obj if not a datetime marker
"""
- # msgpack calls this once per decoded map. Every marker has exactly two keys (sentinel and
- # value, as _standard_default writes them), so the length check returns any other map without
- # the .get() calls. Keep it inline: a wrapper call costs more than it saves on text-heavy values.
- # AutoSerializer's hook must not copy it: its set and ndarray markers have 3 and 4 keys.
+ # The Rust decode (unpackb_standard) calls this only for a map holding a marker key: any other map
+ # it would return unchanged, as each marker test reads its key with .get(). Every marker has exactly
+ # two keys (sentinel and value, as _standard_default writes them), so a look-alike map with more
+ # comes back unchanged. AutoSerializer's hook must not copy it: its set and ndarray markers have 3 and 4 keys.
if isinstance(obj, dict) and len(obj) == 2:
if obj.get("__datetime__"):
value = obj.get("value")
@@ -231,24 +203,6 @@ def _standard_object_hook(obj: Any) -> Any:
return obj
-def _may_hold_marker(msgpack_data: bytes) -> bool:
- """Whether a MessagePack document may hold a map :func:`_standard_object_hook` acts on.
-
- False only when no marker key's UTF-8 (``__datetime__``, ``__date__``, ``__time__``) occurs anywhere
- in it. A str decodes from its bytes verbatim, so such a document has no marker key at any depth, the
- hook returns every map it is given unchanged, and the decode without the hook is the same value. Any
- other document, a value that merely mentions a marker key included, keeps the hook.
-
- Examples:
- >>> from datetime import date
- >>> _may_hold_marker(msgpack.packb({"user": "ada"})), _may_hold_marker(msgpack.packb({"__time__": 1}))
- (False, True)
- >>> _may_hold_marker(msgpack.packb(date(2024, 1, 15), default=_standard_default))
- True
- """
- return b"__date" in msgpack_data or b"__time__" in msgpack_data # "__date" covers "__datetime__" too
-
-
class StandardSerializer:
"""Language-agnostic MessagePack serializer for multi-language cache interoperability.
@@ -344,13 +298,6 @@ def __init__(self, enable_integrity_checking: bool = True):
"strict_types": False, # Allow mixed types (more flexible)
"default": _standard_default, # Handle datetime/date/time
}
- self._msgpack_unpack_opts = {
- "use_list": True, # Decode arrays as lists (not tuples)
- "raw": False, # Decode strings properly (not bytes)
- "object_hook": _standard_object_hook, # Restore datetime/date/time
- }
- # For MessagePack _may_hold_marker cleared: the hook would return every map unchanged.
- self._msgpack_unpack_opts_hookless = {k: v for k, v in self._msgpack_unpack_opts.items() if k != "object_hook"}
def serialize(self, obj: Any) -> tuple[bytes, SerializationMetadata]:
"""Serialize object to pure MessagePack bytes with optional ByteStorage wrapper.
@@ -442,12 +389,12 @@ def deserialize(self, data: bytes | memoryview, metadata: SerializationMetadata
if self.enable_integrity_checking:
# Unwrap ByteStorage envelope (decompress + validate integrity)
msgpack_data, _ = self._byte_storage.retrieve(data)
- return unpackb_bounded(msgpack_data, **self._msgpack_unpack_opts)
+ return unpackb_standard(msgpack_data, MSGPACK_MAX_NESTING, _standard_object_hook)
# No unwrap here: an enveloped entry is refused, not decoded (see Raises).
if metadata is not None and metadata.compressed:
raise SerializationError(_CROSS_CONFIG_ERROR)
data = immutable_buffer(data) # the decode and the envelope probe must judge one snapshot
- value = unpackb_bounded(data, **self._msgpack_unpack_opts)
+ value = unpackb_standard(data, MSGPACK_MAX_NESTING, _standard_object_hook)
# A decrypted read probes only the legacy layout. The AAD binds compressed, and the one writer
# that sealed an envelope under compressed=False (AutoSerializer before 0.12.0) predates bin.
decrypted = metadata is not None and metadata.encrypted
@@ -463,47 +410,31 @@ def deserialize(self, data: bytes | memoryview, metadata: SerializationMetadata
def _deserialize_keeping_msgpack(self, data: bytes | memoryview) -> tuple[Any, VerifiedMsgpack]:
"""What :meth:`deserialize` returns for a ByteStorage envelope when integrity checking is on, plus the
- MessagePack ``retrieve()`` verified and this decode walked, with whether it may hold a marker map, for the
- caller to keep and decode later without the walk (:meth:`_deserialize_verified`,
- :meth:`_deserialize_walked`). It always unwraps the envelope, so call it only for an integrity-on
- serializer. Raises SerializationError as deserialize does then.
+ MessagePack ``retrieve()`` verified and this decode accepted, for the caller to keep and decode later
+ without retrieve() (:meth:`_deserialize_verified`). It always unwraps the envelope, so call it only for
+ an integrity-on serializer. Raises SerializationError as deserialize does then.
Examples:
>>> serializer = StandardSerializer()
>>> envelope, _ = serializer.serialize({"test": 123})
>>> value, kept = serializer._deserialize_keeping_msgpack(envelope)
- >>> value, kept.hook, serializer._deserialize_verified(kept.data, kept.hook)
- ({'test': 123}, False, {'test': 123})
+ >>> value, serializer._deserialize_verified(kept.data)
+ ({'test': 123}, {'test': 123})
"""
try:
msgpack_data, _ = self._byte_storage.retrieve(data)
- value = unpackb_bounded(msgpack_data, **self._msgpack_unpack_opts)
- except PAYLOAD_DECODE_ERRORS as e:
- raise SerializationError(f"Failed to deserialize MessagePack data: {e}") from e
- return value, VerifiedMsgpack(msgpack_data, _may_hold_marker(msgpack_data))
-
- def _deserialize_verified(self, msgpack_data: bytes, hook: bool) -> Any:
- """Decode MessagePack a decode in this process already verified and walked (:meth:`_deserialize_keeping_msgpack`):
- no retrieve(), no walk, and no object hook unless ``hook``. Raises SerializationError as deserialize does."""
- # unpackb_bounded's caps, inline rather than through a shared helper: the extra call cost every walked
- # decode about 2,000 instructions. test_kept_decodes_use_unpackb_bounded_caps pins the two together.
- n = len(msgpack_data)
- opts = self._msgpack_unpack_opts if hook else self._msgpack_unpack_opts_hookless
- try:
- return msgpack.unpackb(
- msgpack_data, max_str_len=n, max_bin_len=n, max_array_len=n, max_map_len=n, max_ext_len=n, **opts
- )
+ value = unpackb_standard(msgpack_data, MSGPACK_MAX_NESTING, _standard_object_hook)
except PAYLOAD_DECODE_ERRORS as e:
raise SerializationError(f"Failed to deserialize MessagePack data: {e}") from e
+ return value, VerifiedMsgpack(msgpack_data)
- def _deserialize_walked(self, data: bytes | memoryview, hook: bool) -> Any:
- """:meth:`_deserialize_verified` for the envelope whose MessagePack :meth:`_deserialize_keeping_msgpack`
- walked, with retrieve() run again. Raises SerializationError as deserialize does."""
+ def _deserialize_verified(self, msgpack_data: bytes) -> Any:
+ """Decode MessagePack a decode in this process already verified (:meth:`_deserialize_keeping_msgpack`),
+ without retrieve(). Raises SerializationError as deserialize does."""
try:
- msgpack_data, _ = self._byte_storage.retrieve(data)
+ return unpackb_standard(msgpack_data, MSGPACK_MAX_NESTING, _standard_object_hook)
except PAYLOAD_DECODE_ERRORS as e:
raise SerializationError(f"Failed to deserialize MessagePack data: {e}") from e
- return self._deserialize_verified(msgpack_data, hook)
def _is_verified_envelope(self, data: bytes | memoryview, value: Any, markers: frozenset[int]) -> bool:
"""True only when ``data``, already decoded to ``value``, is a ByteStorage envelope whose payload marker
diff --git a/tests/unit/conftest.py b/tests/unit/conftest.py
index cf62aa23..388efad2 100644
--- a/tests/unit/conftest.py
+++ b/tests/unit/conftest.py
@@ -11,7 +11,10 @@
from cachekit.decorators import orchestrator as orchestrator_module
from cachekit.reliability.circuit_breaker import CircuitBreaker
-from cachekit.serializers import base as serializers_base
+from cachekit.serializers import standard_serializer
+
+# check_msgpack_structure's error for a document outside its bounds, which unpackb_standard raises as its own.
+_GUARD_ERROR_PREFIX = "Unpack failed: MessagePack document "
@pytest.fixture(autouse=True)
@@ -52,18 +55,20 @@ def backend_configured(monkeypatch: pytest.MonkeyPatch) -> None:
@pytest.fixture
-def walks(monkeypatch: pytest.MonkeyPatch) -> list[str]:
- """Each structure walk (check_msgpack_structure) a decode runs: "ok", or "refused" when the walk raised."""
+def decodes(monkeypatch: pytest.MonkeyPatch) -> list[str]:
+ """Each StandardSerializer decode (unpackb_standard, which enforces the structure bounds in the same pass): "ok",
+ "refused" when the bounds refused the document (the guard's own error), or the type of anything else it raised."""
seen: list[str] = []
- real = serializers_base.check_msgpack_structure
+ real = standard_serializer.unpackb_standard
- def recording(data: Any, limit: int) -> None:
+ def recording(data: Any, limit: int, object_hook: Any) -> Any:
try:
- real(data, limit)
- except ValueError:
- seen.append("refused")
+ value = real(data, limit, object_hook)
+ except Exception as e:
+ seen.append("refused" if isinstance(e, ValueError) and str(e).startswith(_GUARD_ERROR_PREFIX) else type(e).__name__)
raise
seen.append("ok")
+ return value
- monkeypatch.setattr(serializers_base, "check_msgpack_structure", recording)
+ monkeypatch.setattr(standard_serializer, "unpackb_standard", recording)
return seen
diff --git a/tests/unit/protocol/test_decode_bounds.py b/tests/unit/protocol/test_decode_bounds.py
index 1f236abe..305d6f1c 100644
--- a/tests/unit/protocol/test_decode_bounds.py
+++ b/tests/unit/protocol/test_decode_bounds.py
@@ -36,13 +36,13 @@
import pytest
import cachekit.serializers.base as serializers_base
-from cachekit._rust_serializer import ByteStorage, check_msgpack_structure
+from cachekit._rust_serializer import ByteStorage, check_msgpack_structure, unpackb_standard
from cachekit.cache_handler import CacheSerializationHandler
from cachekit.interop import decode_interop_value
from cachekit.serializers.auto_serializer import AutoSerializer
from cachekit.serializers.base import MSGPACK_MAX_NESTING, SerializationError, unpackb_bounded
from cachekit.serializers.interop_serializer import InteropSerializer
-from cachekit.serializers.standard_serializer import StandardSerializer
+from cachekit.serializers.standard_serializer import StandardSerializer, _standard_object_hook
from cachekit.serializers.wrapper import SerializationWrapper
from tests.utils.tracemalloc_isolation import measure_in_subprocess
@@ -84,9 +84,16 @@ def _forged_entry(payload: bytes, serializer: str = "default") -> bytes:
return SerializationWrapper.wrap(_envelope(payload), metadata, serializer_name)
-# Every path that decodes backend-supplied MessagePack. Each must reach unpackb_bounded.
+def _standard_decode(data: bytes | bytearray | memoryview) -> Any:
+ """StandardSerializer's decode: the walk's bounds and the decode in one Rust pass."""
+ return unpackb_standard(data, MSGPACK_MAX_NESTING, _standard_object_hook)
+
+
+# Every path that decodes backend-supplied MessagePack. Each must reach unpackb_bounded, or for
+# StandardSerializer the Rust decode that enforces the same bounds in its one pass.
DECODE_PATHS: dict[str, Callable[[bytes], Any]] = {
"unpackb_bounded": lambda b: unpackb_bounded(b, raw=False),
+ "unpackb_standard": _standard_decode,
"interop": decode_interop_value,
"interop_serializer": InteropSerializer().deserialize,
"standard/plain": StandardSerializer(enable_integrity_checking=False).deserialize,
@@ -237,16 +244,23 @@ def test_value_deeper_than_the_envelope_bound_still_falls_through(self) -> None:
assert AutoSerializer().deserialize(doc) == msgpack.unpackb(doc)
+# The two bounded decoders: unpackb_bounded (walk, then msgpack-python) and StandardSerializer's one-pass decode.
+BOUNDED_DECODERS = pytest.mark.parametrize(
+ "decode", [unpackb_bounded, _standard_decode], ids=["unpackb_bounded", "unpackb_standard"]
+)
+
+
class TestOwnedBounds:
"""SDK-local guards that go beyond the shared vectors."""
- def test_nesting_ceiling_is_exactly_the_pinned_constant(self) -> None:
+ @BOUNDED_DECODERS
+ def test_nesting_ceiling_is_exactly_the_pinned_constant(self, decode: Callable[[bytes], Any]) -> None:
# The walk rejects one level past MSGPACK_MAX_NESTING; a document AT the ceiling
# must still decode, so the constant may not exceed msgpack-python's C stack.
at_bound = b"\x91" * MSGPACK_MAX_NESTING + b"\xc0"
# Walked iteratively, not compared with `==`: nested-list equality recurses in CPython
# too, and would blow the same recursion limit this test is bounding.
- value: object = unpackb_bounded(at_bound)
+ value: object = decode(at_bound)
depth = 0
while isinstance(value, list):
assert len(value) == 1
@@ -254,9 +268,10 @@ def test_nesting_ceiling_is_exactly_the_pinned_constant(self) -> None:
assert depth == MSGPACK_MAX_NESTING
assert value is None
with pytest.raises(ValueError, match=f"nests deeper than {MSGPACK_MAX_NESTING} levels"):
- unpackb_bounded(b"\x91" * (MSGPACK_MAX_NESTING + 1) + b"\xc0")
+ decode(b"\x91" * (MSGPACK_MAX_NESTING + 1) + b"\xc0")
- def test_an_empty_innermost_collection_counts_toward_the_ceiling(self) -> None:
+ @BOUNDED_DECODERS
+ def test_an_empty_innermost_collection_counts_toward_the_ceiling(self, decode: Callable[[bytes], Any]) -> None:
# Every array header is a level, an empty one included, so one level past the ceiling is the
# guard's rejection, not msgpack-python's StackError (also a ValueError) after the walk let it by.
check_msgpack_structure(b"\x91" * (MSGPACK_MAX_NESTING - 1) + b"\x90", MSGPACK_MAX_NESTING)
@@ -265,25 +280,33 @@ def test_an_empty_innermost_collection_counts_toward_the_ceiling(self) -> None:
check_msgpack_structure(too_deep, MSGPACK_MAX_NESTING)
assert str(excinfo.value).startswith(GUARD_ERROR_PREFIX)
with pytest.raises(ValueError) as excinfo:
- unpackb_bounded(too_deep)
+ decode(too_deep)
_assert_guard_rejected(excinfo.value)
- def test_trailing_bytes_still_rejected(self) -> None:
+ @BOUNDED_DECODERS
+ def test_trailing_bytes_still_rejected(self, decode: Callable[[bytes], Any]) -> None:
with pytest.raises(msgpack.exceptions.ExtraData):
- unpackb_bounded(b"\xc0\xc0")
+ decode(b"\xc0\xc0")
- def test_mutable_exporters_are_accepted(self) -> None:
+ @BOUNDED_DECODERS
+ def test_mutable_exporters_are_accepted(self, decode: Callable[..., Any]) -> None:
# A bytearray (or a memoryview over one) is snapshotted so the walk and the decode see one
# immutable document; a memoryview of bytes stays zero-copy. All three must decode.
doc = msgpack.packb({"t": 1})
- assert unpackb_bounded(bytearray(doc), raw=False) == {"t": 1}
- assert unpackb_bounded(memoryview(bytearray(doc)), raw=False) == {"t": 1}
- assert unpackb_bounded(memoryview(doc)[0:], raw=False) == {"t": 1}
-
- def test_memoryview_shapes_and_formats_decode_like_bytes(self) -> None:
+ assert decode(bytearray(doc)) == {"t": 1}
+ assert decode(memoryview(bytearray(doc))) == {"t": 1}
+ assert decode(memoryview(doc)[0:]) == {"t": 1}
+
+ @pytest.mark.parametrize(
+ "decode",
+ [lambda b: unpackb_bounded(b, raw=False), StandardSerializer(enable_integrity_checking=False).deserialize],
+ ids=["unpackb_bounded", "standard/plain"],
+ )
+ def test_memoryview_shapes_and_formats_decode_like_bytes(self, decode: Callable[..., Any]) -> None:
# The Rust walk takes PyBuffer; msgpack takes any itemsize-1 buffer. Views are normalised
# to a flat "B" view first so the two agree, len(data) is the byte count the caps need, and a
# legitimate document is never rejected for the shape or format of the view it arrived in.
+ # StandardSerializer's Rust decode takes PyBuffer too, so its plain read normalises the same way.
doc = next(
d
for d in (msgpack.packb({"k": b"x" * m, "l": [1, 2, 3]}, use_bin_type=True) for m in range(1, 9))
@@ -297,10 +320,10 @@ def test_memoryview_shapes_and_formats_decode_like_bytes(self) -> None:
"uint16": memoryview(doc).cast("H"),
}
for name, view in views.items():
- assert unpackb_bounded(view, raw=False) == expected, name
+ assert decode(view) == expected, name
# A non-contiguous view has no flat form: it is copied, then decoded like the bytes it selects.
interleaved = bytes(b for pair in zip(doc, doc, strict=True) for b in pair)
- assert unpackb_bounded(memoryview(interleaved)[::2], raw=False) == expected
+ assert decode(memoryview(interleaved)[::2]) == expected
# One exact-width document per fixed-width marker family: float32/64, uint8..64, int8..64,
# fixext 1/2/4/8/16, ext8/16/32 (2-byte payload), str8/16/32 + fixstr, bin8/16/32.
@@ -332,15 +355,19 @@ def test_memoryview_shapes_and_formats_decode_like_bytes(self) -> None:
b"\xc6\x00\x00\x00\x01x",
]
+ @BOUNDED_DECODERS
@pytest.mark.parametrize("doc", FIXED_WIDTH_DOCS, ids=lambda d: f"0x{d[0]:02x}")
- def test_every_marker_is_walked_to_its_exact_width(self, doc: bytes) -> None:
+ def test_every_marker_is_walked_to_its_exact_width(self, doc: bytes, decode: Callable[[bytes], Any]) -> None:
# Exact length passes the walk; one byte short is a truncation; a trailing byte reaches the
# decoder as ExtraData — together they pin that the walk consumed exactly the marker's width.
check_msgpack_structure(doc, MSGPACK_MAX_NESTING)
with pytest.raises(ValueError, match="Unpack failed"):
check_msgpack_structure(doc[:-1], MSGPACK_MAX_NESTING)
+ decode(doc)
+ with pytest.raises(ValueError, match=GUARD_ERROR_PREFIX):
+ decode(doc[:-1])
with pytest.raises(msgpack.exceptions.ExtraData):
- unpackb_bounded(doc + b"\xc0")
+ decode(doc + b"\xc0")
def test_reserved_marker_is_rejected(self) -> None:
with pytest.raises(ValueError, match="reserved marker 0xc1"):
diff --git a/tests/unit/test_l1_verified_msgpack.py b/tests/unit/test_l1_verified_msgpack.py
index a145c0fc..49c4842b 100644
--- a/tests/unit/test_l1_verified_msgpack.py
+++ b/tests/unit/test_l1_verified_msgpack.py
@@ -1,19 +1,22 @@
-"""A non-encrypted backed cache's L1 hit swaps its entry for one marked as walked: the verified MessagePack, where
-that is no longer, else the envelope itself (``WalkedEnvelope``).
+"""A non-encrypted backed cache's L1 hit swaps its envelope for the verified MessagePack (``VerifiedMsgpack``) when
+that is no longer; otherwise L1 keeps the envelope.
-The first L1 hit on an entry decodes the envelope as before, and its retrieve() yields the MessagePack;
-L1 then holds the kept entry in place of the envelope (``CacheSerializationHandler.deserialize_l1``,
-``L1Cache.shrink_if_held``). Later hits skip the structure walk, skip the object hook when the MessagePack holds no
-marker key, and, from VerifiedMsgpack, skip retrieve(). Each rule that keeps this from costing L1 hits or security
-is pinned here:
+The first L1 hit on an entry decodes the envelope as before, and its retrieve() yields the MessagePack; when that is
+no longer than the envelope, L1 then holds it in place of the envelope (``CacheSerializationHandler.deserialize_l1``,
+``L1Cache.shrink_if_held``), and later hits skip retrieve(). An envelope L1 keeps is decoded in full on every hit.
+No hit skips the structure bounds: every StandardSerializer decode is one Rust pass (``unpackb_standard``) that
+enforces them as it decodes, kept entry or not. Each rule that keeps this from costing L1 hits or security is
+pinned here:
- L1 admits and stores envelopes exactly as before; a hit only ever shrinks an entry in place. So L1 never
holds fewer entries than it would hold envelopes (the replay property below).
- MessagePack enters L1 only from a retrieve() that verified it; a corrupt envelope is never swapped.
-- Only a kept entry skips the walk. Bytes, whatever L1 or L2 path they come from, are walked on every decode.
-- A decode without the object hook equals the decode with it (the equality fuzz below).
-- Encrypted handlers keep their ciphertext envelope in L1, and every hit on it decrypts, walks and runs the
- hook (protocol PRE-10, PRE-11: the secure preset holds only ciphertext in L1, and no bit derived from the
+- Every decode, of bytes from L1 or L2 and of a kept entry alike, goes through the bounded decode, so a nesting
+ bomb is refused wherever it comes from, and nothing refused is ever kept.
+- Every decode equals the decode every read ran before the Rust one: the structure walk, then msgpack with
+ the object hook (the equality fuzz below), datetimes and look-alike marker maps at any depth included.
+- Encrypted handlers keep their ciphertext envelope in L1, and every hit on it decrypts and decodes it in full
+ (protocol PRE-10, PRE-11: the secure preset holds only ciphertext in L1, and nothing derived from the
plaintext sits beside it).
"""
@@ -24,7 +27,7 @@
import random
import uuid
from collections.abc import Callable, Iterator
-from datetime import UTC, date, datetime, timedelta, timezone
+from datetime import UTC, timedelta, timezone
from typing import Any
import msgpack
@@ -39,15 +42,16 @@
from cachekit.config.nested import L1CacheConfig
from cachekit.l1_cache import L1Cache, get_l1_cache
from cachekit.serializers import base
-from cachekit.serializers.base import SerializationError, SuspiciousCacheEntryError
+from cachekit.serializers.base import SerializationError, SuspiciousCacheEntryError, unpackb_bounded
from cachekit.serializers.encryption_wrapper import EncryptionWrapper
-from cachekit.serializers.standard_serializer import VerifiedMsgpack, WalkedEnvelope
+from cachekit.serializers.standard_serializer import VerifiedMsgpack, _standard_object_hook
from cachekit.serializers.wrapper import SerializationWrapper
from tests.fixtures.zero_config import clear_cachekit_env
pytestmark = pytest.mark.unit
MASTER_KEY = "a" * 64
+GUARD_ERROR_PREFIX = "Unpack failed: MessagePack document " # only the structure bounds raise this
@pytest.fixture(autouse=True)
@@ -128,7 +132,8 @@ def compute(x: int) -> Any:
assert compute.cache_info().l1_hits == before + 2 # type: ignore[attr-defined]
-def test_a_value_longer_as_msgpack_keeps_its_envelope_marked_as_walked() -> None:
+def test_a_value_longer_as_msgpack_keeps_its_envelope() -> None:
+ """Swapping in the longer MessagePack would grow the entry, so L1 keeps the envelope bytes it admitted."""
backend, namespace = _Backend(), _namespace()
@cache(backend=backend, ttl=60, namespace=namespace)
@@ -142,8 +147,8 @@ def compute(x: int) -> str:
assert compute(1) == "ab" * 600
_, entry = _stored(backend, namespace)
assert len(msgpack.packb("ab" * 600)) > len(envelope)
- assert type(entry) is WalkedEnvelope and entry.frame == envelope
- assert len(entry) == len(envelope) and get_l1_cache(namespace)._state.memory_bytes == held
+ assert type(entry) is bytes and entry == envelope
+ assert get_l1_cache(namespace)._state.memory_bytes == held
assert backend.gets == 1
@@ -227,17 +232,18 @@ def test_deserialize_l1_keeps_msgpack_only_when_no_longer_and_it_decodes_the_sam
expected = handler.deserialize_data(envelope, cache_key="k")
assert decoded == expected
if len(msgpack.packb(value, use_bin_type=True)) > len(envelope):
- assert type(entry) is WalkedEnvelope and entry.frame is envelope
+ assert entry is None
+ assert handler.deserialize_l1(envelope, cache_key="k") == (expected, None) # every hit on it: the same again
else:
assert type(entry) is VerifiedMsgpack and len(entry) <= len(envelope)
- assert handler.deserialize_l1(entry, cache_key="k") == (expected, None)
+ assert handler.deserialize_l1(entry, cache_key="k") == (expected, None)
def test_msgpack_exactly_as_long_as_the_envelope_is_kept_and_one_byte_longer_is_not() -> None:
"""Random bytes followed by k zero bytes: each zero lengthens the MessagePack by one byte, and LZ4 folds the run,
so the MessagePack overtakes the envelope, passing through both lengths."""
handler, head = CacheSerializationHandler(), random.Random(0).randbytes(300)
- for extra, kept in ((0, VerifiedMsgpack), (1, WalkedEnvelope)):
+ for extra, kept in ((0, VerifiedMsgpack), (1, type(None))):
for k in range(2000):
value = head + bytes(k)
envelope = handler.serialize_data(value, cache_key="k")
@@ -259,14 +265,13 @@ def test_msgpack_exactly_as_long_as_the_envelope_is_kept_and_one_byte_longer_is_
}
-def _planted() -> list[VerifiedMsgpack | WalkedEnvelope]:
- """A kept entry of each type a plaintext default-serializer handler leaves in L1."""
- own = CacheSerializationHandler(encryption=False).serialize_data({"x": 2}, cache_key="k")
- return [VerifiedMsgpack(msgpack.packb({"x": 2}), hook=False), WalkedEnvelope(own, hook=False)]
+def _planted() -> VerifiedMsgpack:
+ """The kept entry a plaintext default-serializer handler leaves in L1."""
+ return VerifiedMsgpack(msgpack.packb({"x": 2}))
@pytest.mark.parametrize(("make", "refusal"), list(_OTHER_HANDLERS.values()), ids=list(_OTHER_HANDLERS))
-def test_other_handlers_keep_their_envelope_and_refuse_kept_entries(
+def test_other_handlers_keep_their_envelope_and_refuse_a_kept_entry(
make: Callable[[], CacheSerializationHandler], refusal: tuple[type[Exception], str]
) -> None:
"""The L1 hit sites send these handlers' entries to deserialize_data, which refuses a swapped entry another
@@ -276,10 +281,9 @@ def test_other_handlers_keep_their_envelope_and_refuse_kept_entries(
assert not handler.keeps_msgpack_in_l1
assert handler.deserialize_l1(envelope, cache_key="k") == ({"x": 1}, None)
error, message = refusal
- for planted in _planted():
- for decode in (handler.deserialize_l1, handler.deserialize_data):
- with pytest.raises(error, match=message):
- decode(planted, cache_key="k") # type: ignore[arg-type]
+ for decode in (handler.deserialize_l1, handler.deserialize_data):
+ with pytest.raises(error, match=message):
+ decode(_planted(), cache_key="k") # type: ignore[arg-type]
def _frames() -> dict[str, bytes]:
@@ -344,26 +348,25 @@ def outcome(decode: Callable[[], Any]) -> Any:
assert (isinstance(got, tuple) and got[1] is not None) == (name == "own-envelope")
-# ── what a hit walks and hooks ───────────────────────────────────────────────────────────────
+# ── what a hit decodes and retrieves ─────────────────────────────────────────────────────────
@pytest.fixture
-def hooks(monkeypatch: pytest.MonkeyPatch) -> list[bool]:
- """For each msgpack.unpackb call, whether it ran an object hook."""
- seen: list[bool] = []
- real = msgpack.unpackb
+def retrieves(monkeypatch: pytest.MonkeyPatch) -> list[int]:
+ """One entry per ByteStorage.retrieve() call, by any instance."""
+ seen: list[int] = []
+ real = ByteStorage.retrieve
- def recording(data: Any, **kwargs: Any) -> Any:
- seen.append("object_hook" in kwargs)
- return real(data, **kwargs)
+ def recording(self: ByteStorage, data: Any) -> Any:
+ seen.append(1)
+ return real(self, data)
- monkeypatch.setattr(msgpack, "unpackb", recording)
+ monkeypatch.setattr(ByteStorage, "retrieve", recording)
return seen
_PLAIN = {"id": 7, "tags": ["a"]} # its MessagePack is no longer than its envelope: kept as VerifiedMsgpack
-_COMPRESSIBLE = {"users": [{"name": f"user {i}", "theme": "dark"} for i in range(50)]} # kept as WalkedEnvelope
-_DATED = {"users": [{"seen": datetime(2024, 1, 15, 12, 30, tzinfo=UTC), "on": date(2024, 1, 15)}] * 50}
+_COMPRESSIBLE = {"users": [{"name": f"user {i}", "theme": "dark"} for i in range(50)]} # L1 keeps its envelope
def _call(fn: Callable[[int], Any], mode: str) -> Any:
@@ -388,46 +391,33 @@ def compute(x: int) -> Any:
@pytest.mark.parametrize("mode", ["sync", "async"])
@pytest.mark.parametrize(
- ("value", "kept", "hook"),
- [(_PLAIN, VerifiedMsgpack, False), (_COMPRESSIBLE, WalkedEnvelope, False), (_DATED, WalkedEnvelope, True)],
- ids=["msgpack", "envelope", "dated-envelope"],
+ ("value", "kept"),
+ [(_PLAIN, VerifiedMsgpack), (_COMPRESSIBLE, bytes)],
+ ids=["msgpack", "envelope"],
)
-def test_hits_on_a_kept_entry_run_no_walk_and_no_hook_unless_a_marker_may_be_present(
- value: Any, kept: type, hook: bool, mode: str, walks: list[str], hooks: list[bool]
+def test_every_hit_runs_the_bounded_decode_and_only_kept_msgpack_skips_retrieve(
+ value: Any, kept: type, mode: str, decodes: list[str], retrieves: list[int]
) -> None:
+ """Kept MessagePack saves a hit retrieve(), never the structure bounds: each hit's decode is one bounded pass,
+ the first hit's and the kept entry's alike. An envelope L1 keeps is retrieved and decoded on every hit."""
backend, namespace = _Backend(), _namespace()
fn = _decorated(backend, namespace, value, mode)
assert _call(fn, mode) == value # miss: L1 holds the envelope
- assert _call(fn, mode) == value # first hit: walks the envelope's MessagePack, keeps it marked
- assert walks == ["ok"]
+ assert _call(fn, mode) == value # first hit: retrieves and decodes the envelope's MessagePack, keeps it if no longer
+ assert (decodes, len(retrieves)) == (["ok"], 1)
_, entry = _stored(backend, namespace)
- assert type(entry) is kept and entry.hook is hook # type: ignore[union-attr]
+ assert type(entry) is kept
- walks.clear()
- hooks.clear()
+ decodes.clear()
+ retrieves.clear()
for _ in range(2):
assert _call(fn, mode) == value
- assert (walks, hooks, backend.gets) == ([], [hook, hook], 1)
+ assert (decodes, len(retrieves), backend.gets) == (["ok", "ok"], 0 if kept is VerifiedMsgpack else 2, 1)
-def test_kept_decodes_use_unpackb_bounded_caps(monkeypatch: pytest.MonkeyPatch) -> None:
- """A kept entry's decode passes msgpack.unpackb exactly the max_*_len caps a walked decode passes."""
- calls: list[dict[str, Any]] = []
- real = msgpack.unpackb
- monkeypatch.setattr(msgpack, "unpackb", lambda data, **kw: calls.append(kw) or real(data, **kw))
- handler = CacheSerializationHandler()
- for value in (_PLAIN, _COMPRESSIBLE):
- envelope = handler.serialize_data(value, cache_key="k")
- _, kept = handler.deserialize_l1(envelope, cache_key="k")
- handler.deserialize_l1(kept, cache_key="k") # type: ignore[arg-type]
- assert len(calls) == 4
- caps = [{k: v for k, v in kw.items() if k.startswith("max_")} for kw in calls]
- assert caps[0] == caps[1] and caps[2] == caps[3] and len(caps[0]) == 5
-
-
-def test_bytes_in_l1_are_walked_on_every_hit_until_kept(walks: list[str]) -> None:
- """The type is the mark: an entry still held as bytes is walked, whoever put it there. A nesting bomb put in L1
- directly is refused by the walk, and the call reads L2 instead."""
+def test_a_nesting_bomb_put_in_l1_as_bytes_is_refused_by_the_bounded_decode(decodes: list[str]) -> None:
+ """The type is the mark: an entry still held as bytes takes the full decode, whoever put it there. A nesting bomb
+ put in L1 directly is refused by the bounds, and the call reads L2 instead."""
backend, namespace = _Backend(), _namespace()
@cache(backend=backend, ttl=60, namespace=namespace)
@@ -438,9 +428,9 @@ def compute(x: int) -> dict[str, int]:
((key, envelope),) = backend.store.items()
l1 = get_l1_cache(namespace)
l1.put(key, _bomb_frame(envelope), redis_ttl=60)
- walks.clear()
- assert compute(1) == {"x": 1} # the L1 hit is refused by its walk, then L2 is read and walked
- assert walks == ["refused", "ok"]
+ decodes.clear()
+ assert compute(1) == {"x": 1} # the L1 hit is refused by the bounds, then L2 is read and decoded
+ assert decodes == ["refused", "ok"]
assert backend.gets == 2
@@ -452,7 +442,7 @@ def _bomb_frame(own: bytes) -> bytes:
@pytest.mark.parametrize("mode", ["sync", "async"])
-def test_an_l2_read_of_a_nesting_bomb_is_still_walked_and_refused(mode: str, walks: list[str]) -> None:
+def test_an_l2_read_of_a_nesting_bomb_is_refused_by_the_bounded_decode(mode: str, decodes: list[str]) -> None:
backend, namespace, runs = _Backend(), _namespace(), []
def compute(x: int) -> dict[str, int]:
@@ -467,25 +457,24 @@ async def acompute(x: int) -> dict[str, int]:
((key, envelope),) = backend.store.items()
backend.store[key] = _bomb_frame(envelope)
get_l1_cache(namespace).clear()
- walks.clear()
+ decodes.clear()
- assert _call(fn, mode) == {"x": 1} # the L2 entry is refused by its walk and recomputed
- assert walks == ["refused"]
+ assert _call(fn, mode) == {"x": 1} # the L2 entry is refused by the bounds and recomputed
+ assert decodes == ["refused"]
assert runs == [1, 1]
assert backend.store[key] == envelope
-def test_a_kept_entry_is_never_built_from_an_entry_that_fails_its_walk(walks: list[str]) -> None:
+def test_a_kept_entry_is_never_built_from_an_entry_that_fails_its_bounds(decodes: list[str]) -> None:
handler = CacheSerializationHandler()
bomb = _bomb_frame(handler.serialize_data({"x": 1}, cache_key="k"))
- with pytest.raises(SerializationError):
- handler.deserialize_l1(bomb, cache_key="k")
- with pytest.raises(SerializationError):
- handler.deserialize_data(bomb, cache_key="k")
- assert walks == ["refused", "refused"]
+ for decode in (handler.deserialize_l1, handler.deserialize_data):
+ with pytest.raises(SerializationError, match=GUARD_ERROR_PREFIX): # raises, so returns nothing to keep
+ decode(bomb, cache_key="k")
+ assert decodes == ["refused", "refused"]
-# ── a decode without the hook equals the decode with it ───────────────────────────────────────
+# ── every decode equals the walked, hooked decode ─────────────────────────────────────────────
_MARKER_KEYS = ("__datetime__", "__date__", "__time__")
_ZONES = st.sampled_from([None, UTC, timezone(timedelta(hours=5, minutes=30)), timezone(-timedelta(hours=8))])
@@ -529,25 +518,38 @@ def _outcome(decode: Callable[[], Any]) -> Any:
return type(e)
+def _verified(envelope: bytes) -> VerifiedMsgpack:
+ """``envelope``'s MessagePack as retrieve() verifies it, held as L1 holds a kept entry, whatever its length."""
+ return VerifiedMsgpack(ByteStorage("msgpack").retrieve(SerializationWrapper.unwrap(envelope)[0])[0])
+
+
+def _walked_hooked(envelope: bytes) -> Any:
+ """The outcome of the decode every StandardSerializer read ran before the Rust one: retrieve(), the structure
+ walk, then msgpack with the object hook on every map. Any error is SerializationError, as the read paths raise it."""
+ raw = _verified(envelope).data
+ outcome = _outcome(lambda: unpackb_bounded(raw, raw=False, use_list=True, object_hook=_standard_object_hook))
+ return outcome if isinstance(outcome, str) else SerializationError
+
+
@settings(max_examples=500, deadline=None)
@given(value=_DATED_VALUES)
def test_every_decode_of_a_kept_entry_equals_the_walked_hooked_decode(value: Any) -> None:
- """deserialize_data is the decode every read ran before kept entries existed: walked, and hooked. Both kept forms
- decode to exactly its outcome, datetimes and look-alike maps at any depth included."""
+ """The L2 read, the first L1 hit and the kept MessagePack decode to exactly the walked, hooked decode's outcome,
+ datetimes and look-alike maps at any depth included, though the Rust decode calls the hook only on a map with
+ a marker key."""
handler = CacheSerializationHandler()
envelope = handler.serialize_data(value, cache_key="k")
- expected = _outcome(lambda: handler.deserialize_data(envelope, cache_key="k"))
+ expected = _walked_hooked(envelope)
+ assert _outcome(lambda: handler.deserialize_data(envelope, cache_key="k")) == expected
try:
first, kept = handler.deserialize_l1(envelope, cache_key="k")
except SerializationError:
- assert expected is SerializationError # refused as deserialize_data refuses it, and nothing is kept
+ assert expected is SerializationError # refused as the reference refuses it, and nothing is kept
return
assert repr(first) == expected
- assert kept is not None
- raw, _ = ByteStorage("msgpack").retrieve(SerializationWrapper.unwrap(envelope)[0])
- assert kept.hook or not any(m.encode() in raw for m in _MARKER_KEYS) # a marker key present keeps the hook
- for entry in (VerifiedMsgpack(raw, kept.hook), WalkedEnvelope(envelope, kept.hook)):
- assert _outcome(lambda entry=entry: handler.deserialize_l1(entry, cache_key="k")[0]) == expected
+ verified = _verified(envelope)
+ assert kept is None or kept.data == verified.data # what L1 would hold is what retrieve() verified
+ assert _outcome(lambda: handler.deserialize_l1(verified, cache_key="k")[0]) == expected
@pytest.mark.parametrize(
@@ -561,13 +563,15 @@ def test_every_decode_of_a_kept_entry_equals_the_walked_hooked_decode(value: Any
],
ids=["datetime-look-alike-deep", "date-look-alike", "time-look-alike", "falsy-sentinel", "text-only"],
)
-def test_look_alike_maps_decode_as_before_and_keep_the_hook(value: Any) -> None:
+def test_look_alike_maps_decode_as_the_walked_hooked_decode(value: Any) -> None:
+ """Pinned cases of the fuzz above: a caller's map that looks like a marker, at any depth, comes back from every
+ decode as the walked, hooked decode returns it, and text that merely mentions a marker key is text."""
handler = CacheSerializationHandler()
envelope = handler.serialize_data(value, cache_key="k")
- expected = repr(handler.deserialize_data(envelope, cache_key="k"))
- first, kept = handler.deserialize_l1(envelope, cache_key="k")
- assert kept is not None and kept.hook is True
- assert repr(first) == repr(handler.deserialize_l1(kept, cache_key="k")[0]) == expected
+ expected = _walked_hooked(envelope)
+ first, _ = handler.deserialize_l1(envelope, cache_key="k")
+ assert repr(handler.deserialize_data(envelope, cache_key="k")) == expected
+ assert repr(first) == repr(handler.deserialize_l1(_verified(envelope), cache_key="k")[0]) == expected
@pytest.mark.parametrize("sentinel", _MARKER_KEYS)
@@ -590,7 +594,7 @@ def test_shrink_if_held_swaps_the_entry_in_place(size: int) -> None:
l1.put("b", b"b" * 10, redis_ttl=60)
expires_at = l1._state.cache["a"].expires_at
- new = VerifiedMsgpack(b"m" * size, hook=True)
+ new = VerifiedMsgpack(b"m" * size)
assert l1.shrink_if_held("a", old, new)
assert l1.peek("a") is new
@@ -606,13 +610,13 @@ def test_shrink_if_held_leaves_an_entry_it_did_not_read_or_would_grow(change: st
l1 = L1Cache(max_memory_mb=1, namespace=_namespace())
old = b"e" * 100
l1.put("a", old, redis_ttl=60)
- new = VerifiedMsgpack(b"m" * 40, hook=True)
+ new = VerifiedMsgpack(b"m" * 40)
if change == "removed":
l1.invalidate("a")
elif change == "stored-again":
l1.put("a", bytes(bytearray(old)), redis_ttl=60) # equal bytes, another object: a newer write
else:
- new = VerifiedMsgpack(b"m" * 101, hook=True)
+ new = VerifiedMsgpack(b"m" * 101)
held = l1.peek("a")
assert not l1.shrink_if_held("a", old, new)
@@ -664,11 +668,12 @@ def compute(x: int) -> dict[str, int]:
@pytest.mark.parametrize("decorate", list(_SECURE.values()), ids=list(_SECURE))
-def test_every_encrypted_l1_hit_still_walks_and_hooks_its_plaintext(
- decorate: Callable[[str, _Backend], Any], walks: list[str], hooks: list[bool]
+def test_every_encrypted_l1_hit_decodes_its_plaintext_through_the_bounded_decode(
+ decorate: Callable[[str, _Backend], Any], decodes: list[str]
) -> None:
- """Decided for encrypted handlers: no skip. A flag saying whether the plaintext may hold a marker map would be a
- plaintext-derived bit kept beside the ciphertext in L1, which PRE-10 and PRE-11 keep to ciphertext alone."""
+ """Decided for encrypted handlers: nothing is kept. Every hit decrypts the ciphertext L1 holds and decodes the
+ plaintext in full, and no entry or flag derived from the plaintext sits beside the ciphertext in L1, which
+ PRE-10 and PRE-11 keep to ciphertext alone."""
backend, namespace = _Backend(), _namespace()
@decorate(namespace, backend)
@@ -676,18 +681,15 @@ def compute(x: int) -> dict[str, int]:
return {"x": x}
compute(1)
- walks.clear()
- hooks.clear()
+ decodes.clear()
for _ in range(3):
assert compute(1) == {"x": 1}
- assert walks == ["ok"] * 3
- assert hooks == [True] * 3
+ assert decodes == ["ok"] * 3
envelope, entry = _stored(backend, namespace)
assert type(entry) is bytes and entry == envelope
-@pytest.mark.parametrize("kind", [VerifiedMsgpack, WalkedEnvelope], ids=lambda t: t.__name__)
-def test_an_encrypted_reader_refuses_a_kept_entry_in_its_l1(kind: type) -> None:
+def test_an_encrypted_reader_refuses_a_kept_entry_in_its_l1() -> None:
"""A plaintext wrapper sharing the namespace and key cannot make an encrypted one serve plaintext."""
backend, namespace = _Backend(), _namespace()
@@ -698,11 +700,11 @@ def compute(x: int) -> dict[str, int]:
compute(1)
((key, _),) = backend.store.items()
l1 = get_l1_cache(namespace)
- (planted,) = [p for p in _planted() if type(p) is kind]
+ planted = _planted()
held = l1.peek(key)
assert held is not None and len(planted) <= len(held)
assert l1.shrink_if_held(key, held, planted)
- assert type(l1.peek(key)) is kind
+ assert type(l1.peek(key)) is VerifiedMsgpack
assert compute(1) == {"x": 1} # read from L2 instead
assert type(l1.peek(key)) is bytes
diff --git a/tests/unit/test_single_flight.py b/tests/unit/test_single_flight.py
index 8d98128e..07001d95 100644
--- a/tests/unit/test_single_flight.py
+++ b/tests/unit/test_single_flight.py
@@ -1005,11 +1005,11 @@ def run_after_another_trip_and_hit(self: ThreadFlights, *args: Any) -> Any:
def test_sync_backed_joiners_each_decode_their_own_copy_of_the_trips_envelope(
- waiting: list[int], decoded: list[type], decoded_data: list[type], walks: list[str]
+ waiting: list[int], decoded: list[type], decoded_data: list[type], decodes: list[str]
) -> None:
"""Each joined caller decodes the trip's envelope once, for its own copy, as before. It goes to
- deserialize_data: a joiner keeps nothing in L1, so it builds no kept entry (deserialize_l1). The envelope
- came from the trip, not from a decode in this process, so each joiner's decode walks it."""
+ deserialize_data: a joiner keeps nothing in L1, so it builds no kept entry (deserialize_l1). Each joiner's
+ decode is a full bounded decode (unpackb_standard) of the envelope's MessagePack."""
def compute(x: int) -> dict[str, int]:
_until(lambda: waiting[0] == HERD - 1)
@@ -1023,7 +1023,7 @@ def compute(x: int) -> dict[str, int]:
assert backend.requests == ["get", "set"]
assert decoded == []
assert decoded_data == [bytes] * (HERD - 1)
- assert walks == ["ok"] * (HERD - 1)
+ assert decodes == ["ok"] * (HERD - 1)
def test_sync_backed_recheck_hit_hands_its_half_open_probe_slot_back(
diff --git a/tests/unit/test_standard_decode.py b/tests/unit/test_standard_decode.py
new file mode 100644
index 00000000..bdeacaaa
--- /dev/null
+++ b/tests/unit/test_standard_decode.py
@@ -0,0 +1,358 @@
+"""StandardSerializer's Rust decode (``unpackb_standard``) against the decode it replaced.
+
+Every StandardSerializer read used to run ``unpackb_bounded(data, raw=False, use_list=True,
+object_hook=_standard_object_hook)``: cachekit-core's structure walk, then msgpack-python. The Rust
+extension now does both in one pass. These tests hold it to that reference, input for input:
+
+- every input the reference accepts decodes to an equal value of the same types (``repr`` tells
+ 1 from 1.0 from True, a datetime from the marker map it came from, bytes from str);
+- every input it refuses raises the same exception class, and a bound it refuses with is refused
+ with the same ``Unpack failed: MessagePack document ...`` message, so the bounds tests and the
+ read paths' error handling see what they saw before;
+- the hook runs only on maps holding a marker key (the rule the speed rests on), and a heap
+ amplification payload costs no more than it did;
+- no path leaks a reference, and the decode is safe to run from many threads.
+
+The inputs are values packed as the serializer packs them, the same with ``bytes`` packed as str
+(invalid UTF-8), and both after random byte edits, so the bounds, truncation, reserved-marker and
+trailing-byte paths are reached as well as the happy path.
+"""
+
+from __future__ import annotations
+
+import gc
+import threading
+from collections.abc import Callable
+from datetime import UTC, datetime, timedelta, timezone
+from typing import Any
+
+import msgpack
+import pytest
+from hypothesis import given, settings
+from hypothesis import strategies as st
+
+from cachekit._rust_serializer import unpackb_standard
+from cachekit.serializers.base import MSGPACK_MAX_NESTING, SerializationError, unpackb_bounded
+from cachekit.serializers.standard_serializer import _standard_default, _standard_object_hook
+from tests.utils.tracemalloc_isolation import measure_in_subprocess
+
+pytestmark = pytest.mark.unit
+
+GUARD_ERROR_PREFIX = "Unpack failed: MessagePack document "
+
+
+def reference(data: bytes) -> Any:
+ """The decode every StandardSerializer read ran before this one."""
+ return unpackb_bounded(data, raw=False, use_list=True, object_hook=_standard_object_hook)
+
+
+def rust(data: bytes) -> Any:
+ return unpackb_standard(data, MSGPACK_MAX_NESTING, _standard_object_hook)
+
+
+def outcome(decode: Callable[[bytes], Any], data: bytes) -> tuple[Any, ...]:
+ """What ``decode`` returned, as a repr, or the class it raised. A bound's message is kept: it names the bound."""
+ try:
+ return ("value", repr(decode(data)))
+ except msgpack.ExtraData as e:
+ return ("extra", repr(e.unpacked), e.extra)
+ except Exception as e: # noqa: BLE001 - the outcome under comparison
+ message = str(e)
+ return ("raised", type(e), message if message.startswith(GUARD_ERROR_PREFIX) else None)
+
+
+def assert_same(data: bytes) -> None:
+ assert outcome(rust, data) == outcome(reference, data), data.hex()
+
+
+# ── generated values ──────────────────────────────────────────────────────────────────────────
+
+_MARKER_KEYS = ("__datetime__", "__date__", "__time__")
+_ZONES = st.sampled_from([None, UTC, timezone(timedelta(hours=5, minutes=30)), timezone(-timedelta(hours=8))])
+_TEMPORAL = st.datetimes(timezones=_ZONES) | st.dates() | st.times(timezones=_ZONES)
+_ISO = _TEMPORAL.map(lambda t: t.isoformat())
+_FLAGS = st.sampled_from([True, False, 1, 0, 0.0, float("nan"), "yes", "", b"", [], [0], {}, None])
+# Maps a caller wrote that look like markers: a sentinel, truthy or not, with a value that parses, does not
+# parse, is not a str, or is missing; and maps of another size that carry a sentinel.
+_MIMICS = (
+ st.builds(
+ lambda k, f, v: {k: f, "value": v},
+ st.sampled_from(_MARKER_KEYS),
+ _FLAGS,
+ _ISO | st.text(max_size=12) | st.none() | st.integers(-(2**63), 2**64 - 1) | st.binary(max_size=4),
+ )
+ | st.builds(lambda k, f, o: {k: f, o: 1}, st.sampled_from(_MARKER_KEYS), _FLAGS, st.text(max_size=6))
+ | st.builds(lambda k, v: {k: True, "value": v, "extra": 1}, st.sampled_from(_MARKER_KEYS), _ISO)
+ | st.builds(lambda k: {k: True}, st.sampled_from(_MARKER_KEYS))
+ | st.builds(lambda k, v: {k.encode(): True, "value": v}, st.sampled_from(_MARKER_KEYS), _ISO) # bin key
+)
+_EXT = st.builds(msgpack.ExtType, st.integers(0, 127), st.binary(max_size=20)) | st.builds(
+ msgpack.Timestamp, st.integers(-(2**40), 2**40), st.integers(0, 999_999_999)
+)
+_KEYS = st.text(max_size=8) | st.binary(max_size=4) | st.sampled_from((*_MARKER_KEYS, "value")) | st.integers(-3, 3)
+_LEAVES = (
+ st.none()
+ | st.booleans()
+ | st.integers(-(2**63), 2**64 - 1)
+ | st.floats() # NaN, infinities and -0.0 included
+ | st.text()
+ | st.binary()
+ | _TEMPORAL
+ | _MIMICS
+ | _EXT
+ | st.sampled_from(_MARKER_KEYS)
+)
+VALUES = st.recursive(
+ _LEAVES,
+ lambda inner: st.lists(inner, max_size=6) | st.dictionaries(_KEYS, inner, max_size=6) | st.tuples(inner, inner),
+ max_leaves=40,
+)
+
+
+def _pack(value: Any, use_bin_type: bool, single_float: bool = False) -> bytes:
+ """``single_float`` reaches the float32 marker, for values whose floats all fit one."""
+ try:
+ return msgpack.packb(value, use_bin_type=use_bin_type, use_single_float=single_float, default=_standard_default)
+ except OverflowError:
+ return msgpack.packb(value, use_bin_type=use_bin_type, default=_standard_default)
+
+
+@settings(max_examples=1500, deadline=None)
+@given(value=VALUES, use_bin_type=st.booleans(), single_float=st.booleans())
+def test_every_packed_value_decodes_as_before(value: Any, use_bin_type: bool, single_float: bool) -> None:
+ """use_bin_type=False packs bytes as str, so invalid UTF-8 reaches the str path; int keys reach strict_map_key."""
+ assert_same(_pack(value, use_bin_type, single_float))
+
+
+_EDITS = st.lists(
+ st.tuples(st.sampled_from(["set", "insert", "delete"]), st.integers(0, 2**16), st.integers(0, 255)), max_size=4
+)
+
+
+@settings(max_examples=1500, deadline=None)
+@given(value=VALUES, use_bin_type=st.booleans(), edits=_EDITS, cut=st.integers(0, 3), tail=st.binary(max_size=3))
+def test_every_edited_document_decodes_or_refuses_as_before(
+ value: Any, use_bin_type: bool, edits: list[tuple[str, int, int]], cut: int, tail: bytes
+) -> None:
+ """Byte edits, a truncation and trailing bytes reach the bounds, the reserved marker, ExtraData and broken markers."""
+ data = bytearray(_pack(value, use_bin_type))
+ for op, at, byte in edits:
+ at %= len(data) + 1
+ if op == "insert":
+ data.insert(at, byte)
+ elif at < len(data):
+ if op == "set":
+ data[at] = byte
+ else:
+ del data[at]
+ assert_same(bytes(data[: max(len(data) - cut, 0)]) + tail)
+
+
+@settings(max_examples=3000, deadline=None)
+@given(data=st.binary(max_size=48))
+def test_arbitrary_bytes_decode_or_refuse_as_before(data: bytes) -> None:
+ assert_same(data)
+
+
+# ── explicit cases ────────────────────────────────────────────────────────────────────────────
+
+CASES = {
+ "empty input": b"",
+ "nil": b"\xc0",
+ "reserved 0xc1": b"\x91\xc1",
+ "trailing nil": b"\xc0\xc0",
+ "trailing garbage after a map": msgpack.packb({"a": 1}) + b"\xc1\xff",
+ "int key": msgpack.packb({1: "a"}),
+ "float key": msgpack.packb({1.5: "a"}),
+ "nil key": msgpack.packb({None: 1}),
+ "list key": b"\x81\x91\x01\x02",
+ "map key holding a datetime marker": b"\x81" + msgpack.packb({"__date__": True, "value": "2024-01-15"}) + b"\x01",
+ "bytes key": msgpack.packb({b"k": 1}, use_bin_type=True),
+ "duplicate keys, last wins": b"\x82\xa1a\x01\xa1a\x02",
+ "duplicate keys collapse to a marker": b"\x83\xa8__date__\xc3\xa8__date__\xc3\xa5value\xaa2024-01-15",
+ "invalid UTF-8 str": b"\xa2\xff\xfe",
+ "invalid UTF-8 key": b"\x81\xa2\xff\xfe\x01",
+ "invalid UTF-8 after a bound violation": b"\x92\xa2\xff\xfe\xdd\xff\xff\xff\xff",
+ "lone surrogate": b"\xa3\xed\xa0\x80",
+ "str8 marker key": b"\x82\xd9\x08__date__\xc3\xa5value\xaa2024-01-15",
+ "marker missing value": msgpack.packb({"__datetime__": True, "x": 1}),
+ "marker with nil value": msgpack.packb({"__time__": True, "value": None}),
+ "marker with int value": msgpack.packb({"__date__": True, "value": 7}),
+ "marker with unparseable value": msgpack.packb({"__datetime__": True, "value": "yesterday"}),
+ "marker with falsy flag": msgpack.packb({"__datetime__": 0, "value": "2024-01-15T00:00:00"}),
+ "marker with nan flag": msgpack.packb({"__date__": float("nan"), "value": "2024-01-15"}),
+ "two sentinels": msgpack.packb({"__datetime__": False, "__date__": True}),
+ "marker error before a trailing byte": msgpack.packb({"__date__": True, "value": "x"}) + b"\xc0",
+ "nested marker": msgpack.packb([{"a": {"__time__": True, "value": "12:30:00+05:30"}}]),
+ "uint64 max": msgpack.packb(2**64 - 1),
+ "int64 min": msgpack.packb(-(2**63)),
+ "float32": msgpack.packb(1.1, use_single_float=True),
+ "nan": msgpack.packb(float("nan")),
+ "negative zero": msgpack.packb(-0.0),
+ "ext": msgpack.packb(msgpack.ExtType(5, b"abc")),
+ "fixext 16": b"\xd8\x05" + b"\x00" * 16,
+ "negative ext code": b"\xd4\xfe\x00",
+ "timestamp 32": msgpack.packb(msgpack.Timestamp(1, 0)),
+ "timestamp 64": msgpack.packb(msgpack.Timestamp(2**33, 5)),
+ "timestamp 96": msgpack.packb(msgpack.Timestamp(-(2**40), 999_999_999)),
+ "timestamp bad length": b"\xc7\x05\xff" + b"\x00" * 5,
+ "timestamp bad nanoseconds": b"\xd7\xff" + (0x3FFFFFFF << 34 | 1).to_bytes(8, "big"),
+ "array32 over-claim": b"\xdd\xff\xff\xff\xff",
+ "map32 over-claim": b"\xdf\xff\xff\xff\xff",
+ "str32 over-claim": b"\xdb\xff\xff\xff\xff",
+ "ext32 over-claim": b"\xc9\xff\xff\xff\xff\x05",
+ "truncated length prefix": b"\xda\x00",
+ "ends mid-array": b"\x93\x01\x02",
+ "nested-header bomb": b"\xdc\x07\xd0" * 5000,
+}
+
+
+@pytest.mark.parametrize("data", CASES.values(), ids=CASES.keys())
+def test_explicit_case_decodes_or_refuses_as_before(data: bytes) -> None:
+ assert_same(data)
+
+
+def test_the_hook_errors_are_the_serializers_own() -> None:
+ # Not only the same class as before: the class the read paths already turn into a miss.
+ with pytest.raises(SerializationError, match="missing 'value' field"):
+ rust(CASES["marker missing value"])
+
+
+@pytest.mark.parametrize("header", [b"\x91", b"\x81\xa0"], ids=["array", "map"])
+def test_the_nesting_ceiling_is_the_walks(header: bytes) -> None:
+ """At the ceiling it decodes; one level past it, or an empty container one level past it, the walk's error."""
+ at = header * MSGPACK_MAX_NESTING + b"\xc0"
+ value, depth = rust(at), 0
+ while isinstance(value, (list, dict)): # iteratively: nested == recurses past the recursion limit
+ depth, value = depth + 1, next(iter(value.values()) if isinstance(value, dict) else iter(value))
+ assert (depth, value) == (MSGPACK_MAX_NESTING, None)
+ empty = b"\x90" if header == b"\x91" else b"\x80"
+ for past in (header * (MSGPACK_MAX_NESTING + 1) + b"\xc0", header * MSGPACK_MAX_NESTING + empty):
+ with pytest.raises(ValueError, match=f"^{GUARD_ERROR_PREFIX}nests deeper than {MSGPACK_MAX_NESTING} levels$"):
+ rust(past)
+
+
+def test_buffer_inputs_decode_like_bytes() -> None:
+ doc = msgpack.packb({"k": [1, 2, b"x"]}, use_bin_type=True)
+ for data in (bytearray(doc), memoryview(doc), memoryview(bytearray(doc)), memoryview(doc).cast("c")):
+ assert rust(data) == {"k": [1, 2, b"x"]}
+
+
+# ── the marker rule ───────────────────────────────────────────────────────────────────────────
+
+
+def test_the_hook_runs_only_on_maps_holding_a_marker_key() -> None:
+ """The reference ran the hook on every map; it returns any map without a marker str key unchanged, because
+ each marker test reads its key with dict.get. So skipping those maps changes no value."""
+ seen: list[dict[Any, Any]] = []
+
+ def hook(obj: dict[Any, Any]) -> Any:
+ seen.append(dict(obj))
+ return _standard_object_hook(obj)
+
+ doc = _pack(
+ {
+ "plain": {"a": 1},
+ "dated": datetime(2024, 1, 15),
+ "look-alike": {"__date__": False, "value": 1, "x": 2},
+ "bin key": {b"__date__": True, "value": "2024-01-15"},
+ "as a value": {"k": "__date__"},
+ },
+ use_bin_type=True,
+ )
+ unpackb_standard(doc, MSGPACK_MAX_NESTING, hook)
+ assert seen == [{"__datetime__": True, "value": "2024-01-15T00:00:00"}, {"__date__": False, "value": 1, "x": 2}]
+
+
+# ── heap, references, threads ─────────────────────────────────────────────────────────────────
+
+# Legal payloads that materialise many objects per byte: N empty maps (the shape the decode bounds leave
+# open), N empty arrays, and N repeats of a short str.
+_N = 200_000
+AMPLIFIERS = {
+ "empty maps": b"\xdd" + _N.to_bytes(4, "big") + b"\x80" * _N,
+ "empty arrays": b"\xdd" + _N.to_bytes(4, "big") + b"\x90" * _N,
+ "short strs": b"\xdd" + _N.to_bytes(4, "big") + b"\xa2ab" * _N,
+}
+
+
+def _measure_amplifier_peaks() -> dict[str, list[int]]:
+ import tracemalloc
+
+ peaks = {}
+ for name, data in AMPLIFIERS.items():
+ pair = []
+ for decode in (reference, rust):
+ # Collection paused: on a free-threaded build a cycle collection that lands mid-decode allocates a
+ # worklist (one pointer per tracked object) inside the traced window, and where it lands differs by decoder.
+ gc.collect()
+ gc.disable()
+ tracemalloc.start()
+ value = decode(data)
+ pair.append(tracemalloc.get_traced_memory()[1])
+ tracemalloc.stop()
+ gc.enable()
+ del value
+ peaks[name] = pair
+ return peaks
+
+
+def test_an_amplifying_payload_costs_no_more_heap_than_before() -> None:
+ for name, (before, now) in measure_in_subprocess(_measure_amplifier_peaks).items():
+ assert now <= before, f"{name}: {now} bytes at peak, {before} before"
+
+
+def _measure_leaks() -> list[int]:
+ import tracemalloc
+
+ def every_path() -> None:
+ for data in (*CASES.values(), _pack({"k": ["v" * 40, datetime(2024, 1, 15), b"b"] * 3}, True)):
+ try:
+ rust(data)
+ except (ValueError, SerializationError, TypeError): # noqa: S110 - every refusal CASES pins; any other error fails
+ pass
+
+ for _ in range(50): # warm caches and lazily imported msgpack classes
+ every_path()
+ gc.collect()
+ tracemalloc.start()
+ sizes = []
+ for _ in range(LEAK_ROUNDS + 1):
+ for _ in range(LEAK_CALLS_PER_ROUND):
+ every_path()
+ gc.collect()
+ sizes.append(tracemalloc.get_traced_memory()[0])
+ tracemalloc.stop()
+ return sizes
+
+
+# A leak on any path keeps at least one object per call, and the smallest object (an int) is 28 bytes, so it adds at
+# least 500 * 28 = 13.7 KiB to EVERY round. The interpreter's own growth (caches, free lists, a dict resizing) comes
+# in a few one-off steps, and one such step can exceed any fixed budget on the total. So the gate bounds the quietest
+# of five rounds, which grows by under 1 KiB when nothing leaks.
+LEAK_ROUNDS = 5
+LEAK_CALLS_PER_ROUND = 500
+LEAK_ROUND_BUDGET = 8 * 1024
+
+
+def test_no_path_leaks_a_reference() -> None:
+ """A leaked reference keeps its object alive, so traced memory would climb in every round."""
+ sizes = measure_in_subprocess(_measure_leaks)
+ rounds = [after - before for before, after in zip(sizes, sizes[1:], strict=False)]
+ assert min(rounds) < LEAK_ROUND_BUDGET, sizes
+
+
+def test_threads_decode_one_document_concurrently() -> None:
+ doc = _pack({"users": [{"id": i, "name": f"user {i}", "at": datetime(2024, 1, 15)} for i in range(50)]}, True)
+ expected, failures, barrier = reference(doc), [], threading.Barrier(8)
+
+ def work() -> None:
+ barrier.wait()
+ failures.extend(1 for _ in range(200) if rust(doc) != expected)
+
+ threads = [threading.Thread(target=work) for _ in range(8)]
+ for t in threads:
+ t.start()
+ for t in threads:
+ t.join()
+ assert not failures
diff --git a/tests/unit/test_standard_serializer.py b/tests/unit/test_standard_serializer.py
index 5c6ac0d2..6c8aa828 100644
--- a/tests/unit/test_standard_serializer.py
+++ b/tests/unit/test_standard_serializer.py
@@ -701,14 +701,14 @@ def test_decode_and_probe_judge_one_snapshot_of_a_mutable_buffer(self, monkeypat
if not healthy_at_decode:
buf[pos] ^= 1
decoded_fields = msgpack.unpackb(bytes(buf))
- real_unpackb_bounded = standard_serializer.unpackb_bounded
+ real_unpackb_standard = standard_serializer.unpackb_standard
- def decode_then_flip(*args, **kwargs):
- value = real_unpackb_bounded(*args, **kwargs)
+ def decode_then_flip(*args):
+ value = real_unpackb_standard(*args)
buf[pos] ^= 1 # the caller's buffer changes after the decode has read it
return value
- monkeypatch.setattr(standard_serializer, "unpackb_bounded", decode_then_flip)
+ monkeypatch.setattr(standard_serializer, "unpackb_standard", decode_then_flip)
off = StandardSerializer(enable_integrity_checking=False)
if healthy_at_decode:
with pytest.raises(SerializationError, match=f"^{re.escape(CROSS_CONFIG_MESSAGE)}$"):