Skip to content

performance.md

Latest commit

 

History

History
412 lines (286 loc) · 54.2 KB

performance.md

File metadata and controls

412 lines (286 loc) · 54.2 KB
Raw

Home › Architecture › Performance

Performance Guide

Sub-millisecond cache operations, measured and benchmarked

Key Numbers

Tip

Key numbers (mean of five run medians; GitHub-hosted ubuntu-latest runner, AMD EPYC 7763, CPython 3.14.7, 2026-10-31; see GitHub-Hosted Runner):

  • Decorator + L1 hit, @cache(backend=None), 100-user nested dict (23.5KB as plain MessagePack): 0.311 ± 0.000μs (about 2,500 instructions per call on CPython 3.14, see Instruction Budgets)
  • Same hit, 10K-row DataFrame: 0.31 ± 0.00μs. An L1-only hit never serializes, so payload size barely matters
  • Decorator + L1 hit with an L2 backend configured, same dict: 284 ± 2μs. With a backend, L1 holds bytes and every hit deserializes them. This dict compresses, so L1 holds its envelope. Measured before the Rust decode every such hit now runs, which bounds and decodes the MessagePack in one pass at about half the instructions; see Why an L1-Only Hit Costs the Same for Any Payload
  • Redis L2 hit, L1 disabled, same dict, Redis in a container on the same runner: 539 ± 3μs
  • Raw L1 byte-cache lookup (L1Cache.get, no decorator): 354–362ns on a shared x86_64 Linux host (see Measured Figures)

Measurement Methodology

Every figure on this page comes from a guard in tests/performance/:

  • Timer: time.perf_counter_ns() around each call
  • Runs, not samples: each guard takes 5 independent runs, with a forced garbage collection before each. The figure is the mean of the per-run medians with its 95% t band at df = 4 (stats_utils.summarize)
  • Every raw sample kept: percentiles cover all samples; outliers are counted, never filtered
  • Warm-up: each run warms up for at least 5,000 calls (2,000 for the raw L1 guard), stopping once the last 1,000 calls vary by less than 10%, and at most twice that minimum. The 10-thread guard warms up 5,000 calls once, then 100 per thread before each run
  • Pre-flight: each session prints whether the host is throttled or loaded (measurement_env.py)
  • No tail claims at 5 runs: the P95 line of a guard's summary reads "inconclusive" unless it has at least 10 runs and 400 samples (the P99 line needs 10 runs and 2,000 samples). Every guard below runs 5, so both lines read "inconclusive" and this page quotes no tail percentile. The guards' own lines after the summary ("Total measured", "Decorator + complex payload") still print the raw p95 of every sample, and record it with record_property; no guard asserts on it, and at 5 runs it does not support a tail claim
  • Behaviour, not budgets: a guard asserts that its timed calls took the path it names (the cached function ran once, an L1 hit never reached L2, the value came back intact), never a time

Measured Figures

Each cell is the mean of five run medians ± its 95% t band (the summary's Run median: line), from two back-to-back passes of the same guard, CPython 3.14.3, x86_64 Linux, 2026-10-03. The pre-flight reported no throttling or load, but the host is shared, so the figures are indicative: compare them with each other rather than with your machine. Sessions on 2026-10-03 and 2026-10-04 also timed the decorated hits and a Redis L2 hit; later changes made those paths faster, and the GitHub-Hosted Runner table has their current figures.

Path Guard (tests/performance/) Pass 1 Pass 2
Raw L1 byte-cache lookup, L1Cache.get of 1KB, no decorator test_statistical_rigor.py::test_l1_cache_hit_statistically_rigorous 354 ± 68ns 362 ± 41ns

In this table the band is ±19% of its figure in pass 1 and ±11% in pass 2. That spread is why these suites inform rather than gate (see Performance Regression Testing).

Run it yourself:

uv run pytest tests/performance/test_statistical_rigor.py::test_l1_cache_hit_statistically_rigorous -v -s

GitHub-Hosted Runner

The Bench workflow (.github/workflows/bench.yml) runs five guards from tests/performance/ on a GitHub-hosted ubuntu-latest runner and writes their run medians into its job summary; run it from the Actions tab to refresh this table. The README quotes it. Its run on 2026-10-31 used an AMD EPYC 7763 (4 vCPUs), CPython 3.14.7 and Redis 7 in a container on the same runner. That is a different machine from the table above, so compare figures within one table.

Path Guard (tests/performance/) Run median
Decorator + L1 hit, @cache(backend=None), 100-user nested dict (23.5KB as MessagePack) test_production_realism.py::test_decorator_overhead_complex_dict 0.311 ± 0.000μs
Decorator + L1 hit, @cache(backend=None, serializer="auto"), 10K-row DataFrame test_production_realism.py::test_decorator_overhead_dataframe 0.31 ± 0.00μs
Decorator + L1 hit with an L2 backend configured (in-process, never reached), same 100-user dict test_production_realism.py::test_decorator_overhead_l1_hit_with_backend 284 ± 2μs (before the decode changes, see Key Numbers)
Same hit, plaintext and encrypted arms, interleaved test_production_realism.py::test_encryption_overhead 286 ± 1μs and 295 ± 1μs (before the decode changes)
Redis L2 hit, L1 disabled, same 100-user dict test_production_realism.py::test_redis_l2_roundtrip 539 ± 3μs

The job summary prints each figure in the unit its guard reports: the dict's L1-only hit, the hit with a backend configured and the encryption arms in ns, the DataFrame hit and the Redis L2 hit in μs. This table is in μs, so divide the ns figures by 1,000 when you copy them in.

The dict's L1-only hit timed 311ns in every one of its five runs. The encryption guard's arms differ by 9.3 ± 1.0μs (Welch 95%): an encrypted hit adds a decrypt. Both arms were measured before the decode changes; since then, a plaintext hit on a kept entry skips envelope decompression and checksum validation, which an encrypted hit does not (see Why an L1-Only Hit Costs the Same for Any Payload). That is the first hosted run whose band excludes zero, and the 2026-10-23 run put the difference at 10 ± 21μs, so one run does not pin the cost down (see Not Measured Here). Under the object cache's lock, as it ran until the lock-free hit, that hit took 0.906 ± 0.007μs in the 2026-10-23 run and 0.874 ± 0.005μs in a run on an EPYC 7763 the same day as the table's. Hosted runners do not always get the same CPU model: a run of an earlier commit on an AMD EPYC 9V45 runner took 44-59% as long on every row (0.529μs for the dict's L1-only hit, 242 ± 12μs for the Redis L2 hit).

The workflow runs these guards with this command, against a Redis it starts on the runner:

REDIS_URL=redis://localhost:6379 uv run pytest tests/performance/test_production_realism.py -s -q -p no:cacheprovider \
  -k "complex_dict or dataframe or with_backend or encryption_overhead or redis_l2"

Without REDIS_URL, the Redis guard uses the pytest-redis server the suite starts. The encryption guard generates a throwaway key, so it needs no CACHEKIT_MASTER_KEY.

Why an L1-Only Hit Costs the Same for Any Payload

With backend=None, cachekit keeps the returned object itself in memory and hands that same object back on a hit (see backend=None). Nothing is serialized or deserialized, so the 100-user dict and a 10K-row DataFrame cost about the same: 0.311μs and 0.31μs on the hosted EPYC 7763, for the wrapper's call and one lookup in the extension (see Decorator + L1 Hit).

With an L2 backend configured, L1 holds the serialized bytes instead, and a hit deserializes them, so that hit costs more as the payload grows. For the 100-user dict it took 284 ± 2μs on the same runner, measured before the decode changes described below, about 910 times an L1-only hit of the same dict. Nearly all of the difference is deserializing 23.5KB of MessagePack back into about 400 nested dicts. It is still about half of a Redis L2 hit there (539 ± 3μs), which deserializes the same bytes after the round trip.

Every read with the default serializer decodes its MessagePack in the Rust extension, in one pass that also enforces the bounds on untrusted MessagePack: nesting at most 1,024 levels deep, and no header declaring more elements or bytes than the rest of the input can back. That covers L2 reads, L1 hits and the decrypted plaintext of encrypted reads. The decode builds the Python objects directly. It shares most repeated short strings, map keys up to 64 bytes and values up to 32, as one object within a value (a small table per decode; two strings that land in one slot replace each other). It runs the Python hook that restores datetimes only on maps holding a __datetime__, __date__ or __time__ key, because the hook hands back any other map unchanged. It returns the same values and refuses the same inputs with the same exception classes as the decode it replaced: cachekit-core's structure walk, then msgpack-python's C unpacker calling the hook once per map. Running out of memory is the one difference. A large string, bin, ext or list still raises MemoryError. If CPython cannot allocate a dict, a list of up to 64 items, a float or an int outside -5 to 256, the process aborts. With Linux's default overcommit, the OOM killer usually ends the process before either outcome. On the 100-user dict (23,547 bytes of MessagePack, about 3,400 objects), the decode fell from 2,577,967 to 1,220,541 instructions on CPython 3.14 (−52.7%, callgrind main-thread instructions per call, the Instruction Budgets method), level with pickle.loads of the same value (1,199,580). The 1,000-user dict (239 KB) fell 51.5% and a 1,090-byte one 45.8%. A list of 200 records each holding a datetime fell 33.9%, since every datetime still runs the hook. On the abi3t extension that CPython 3.15 and later load, the 100-user dict fell 51.0%.

For a plaintext cache with the default or auto serializer and integrity checking on, an entry's first L1 hit decodes it as every read does, and then, when its verified MessagePack is no longer than the envelope, which is true of small values and incompressible ones, L1 holds that MessagePack in the envelope's place. Later hits on it skip the envelope's LZ4 decompress and xxHash3-64 checksum. The auto serializer keeps only plain MessagePack envelopes; its NumPy, Arrow, DataFrame and Series entries are verified on every hit. The orjson serializer keeps the JSON under its checksum, which is always shorter than the envelope, so its later hits skip the checksum. Every hit still runs the bounded decode, which costs no more than the unbounded one did. The 100-user dict compresses to a 3,171-byte envelope from 23.5KB of MessagePack, so L1 keeps its envelope; no budget gates this large-value hit, and l1_hit_backed and l1_hit_backed_auto in Instruction Budgets gate the small-value hit on held MessagePack. Encrypted caches always keep the ciphertext envelope, and every hit on one decrypts and decodes.

Decorator + L1 Hit (Hot Path)

An L1-only hit takes 0.311μs for the 100-user dict and 0.31μs for the DataFrame on the hosted EPYC 7763 (GitHub-Hosted Runner). The wrapper makes one call into the extension (HitMemo.get), which takes no lock and runs:

  • The memo key. The call's arguments become a key: one int or str argument is its own key, and any other call's key holds its argument tuple. The memo takes a call only when it has at most 8 arguments, every one an int, float, bool, None, str or bytes matched by its exact type, and its strings, bytes and keyword names hold at most 256 characters between them. A float, bool or bytes puts the argument types in the key, so 1, 1.0 and True, which compare equal, keep their own slots. Any other call (a list, a dict, a datetime, an IntEnum, longer strings) builds its cache key every time and looks it up under the object cache's lock: a hit with one list argument costs about 22 times the instructions of a memoised one. The cache keys themselves are unchanged.
  • One dict lookup in the function's memo, for the entry the object cache stored for that key.
  • Its liveness and deadline. The entry must still be stored, and its deadline not yet reached: its hard expiry, or with SWR the earliest point a refresh could be flagged (ttl * swr_threshold_ratio * 0.9 after it was stored), past which the hit goes to the lock, which decides on the refresh.
  • The function's names. The memo answers only while the function's __module__ and __qualname__ are the objects it was built with, as its cache keys embed them: renaming the function re-keys its calls.
  • The use time for the cache's LRU order, and the hit count for cache_info(), both atomic.

A memoised L1-only hit records no Prometheus metric. Only a CachekitIO request reads the per-function stats context, so a call sets it only once it is past its L1 lookup and may reach the backend: no L1-only call and no backed L1 hit sets it. A miss takes more Python frames than it did before the memo, so a recursive @cache(backend=None) function reaches the recursion limit sooner: on CPython 3.14 at the default limit of 1,000, it recurses 197 levels deep instead of 247.

The object cache keeps the memo in step under its lock. A store, or a read under the lock, points the call's slot at its entry, and every removal (expiry, eviction, invalidate_cache(), cache_clear(), the reset in a forked child) ends the entry's hits and drops its slots before it returns. So a hit never serves a value once its removal has returned, and the memo keeps no removed value. It keeps a call's arguments only while their slot lasts, and at most 1,024 slots per function: a full memo empties before it takes a new key. Full, it holds about 110 to 140KiB when each call passes one int, and under 4.5MiB when every call fills its 256 characters with 4-byte ones, the most a call can make it keep (3.5 to 3.8MiB measured on CPython 3.12, 3.14 and free-threaded 3.14 and 3.15).

Only an L1-only function keeps a memo. A function with a backend builds its key on every call, as it did before the memo: other processes can invalidate its entries, and the memo would keep the arguments they erase; with L1 off, the process keeps nothing else from a call; and an encrypted function must keep no plaintext argument. Its key is also a small part of what its hit costs.

With ten threads calling one function on the same key at once, test_concurrent_cache_access times each call once its thread is running: execution time after scheduling, not request latency. Each sample starts inside its thread's loop, so on a GIL build, where only one thread runs Python at a time, the time a thread waits for its turn falls outside every sample. Its last figures predate the memo, and the hosted workflow does not run it. No guard measures how long a request waits under contention.

The deterministic figure is the l1_hit row of the instruction budgets: about 2,500 instructions per call on CPython 3.14 and 2,600 on 3.12, against 10,900 and 11,600 when the hit took the object cache's lock, and about 60,000 before the memo and the fast path.

The raw L1 byte-cache lookup, without the decorator, takes 354–362ns. An L1-only hit does not use that cache.

L1-Only Miss at a Full Budget

An L1-only miss stores its result in the object cache. Once the function's byte budget (l1_max_size_mb, 100MB by default) is full, each store first removes the entries that have expired, then evicts least-recently-used ones until the new value fits. Expired entries go first wherever they sit in LRU order, so an entry read just before it expired never pushes out a live one. A min-heap of deadlines finds them: when nothing has expired, the store looks at the earliest deadline and evicts, without scanning the cache for expired entries. The one exception is the heap rebuild described below. Before the heap, every such store scanned every entry while holding the lock the function's misses and its hits under the lock wait on, so with ttl=None (stored for a year, so nothing expires) each one cost time in proportion to the number of entries cached.

The least recently used entry is found without a scan too. A lock-free hit records the time of its use on the entry but cannot reorder the cache, so the cache of an L1-only function with a memo (see Decorator + L1 Hit) keeps one LRU record per entry: each store files its entry's record in clock order, and an eviction that reaches an entry used since its record was filed re-files the record at that use, in a min-heap, and moves on. Evicting an entry unused since its store costs O(1), and each re-filed record O(log n). A cache whose hits all take the lock, as @cache.local()'s do, keeps its LRU order in the cache itself, as before.

The object_cache_put_full row of the instruction budgets measures a store into a full budget of 10,000 entries: about 160,000 instructions on CPython 3.14, most of it the size estimate of the stored value, against 5.64 million when every store scanned. A whole L1-only miss into a full 1MB budget, decorator and function included, costs about 238,800 instructions on CPython 3.14, 1.6% more than before the lock-free hit: each store also creates the record its hits read and points the memo at it, and each eviction ends them.

The heap is memory the budget does not count: about 64 bytes per entry on CPython 3.12 and 72 on 3.14, plus one stale record for each entry removed or refreshed since the heap was last rebuilt. Once the stale records outnumber the entries, the store that finds it rebuilds the heap from the cache: one pass over the entries, about once per as many stores as there are entries. With a memo, each entry also carries the record its hits read and its LRU record, about 170 to 180 bytes per entry the budget does not count (CPython 3.12 and 3.14), and the LRU records are rebuilt the same way, sorted back into clock order.

Backed L1 Expiry Sweep

With a backend, L1 runs a background sweep every l1_cleanup_interval_seconds (30 by default) that removes expired entries from each namespace's byte cache. Every get and put in the namespace waits on that cache's lock, so the sweep never holds it for a scan: it pops a min-heap of deadlines, at most 128 due entries per hold, and gives up the GIL between holds so a waiting get goes first. When nothing has expired, the sweep reads the earliest deadline and stops. Removing 100,000 expired entries holds the lock for at most about 0.4ms at a time (test_l1_expiry_sweep_lock_holds, thread CPU time, CPython 3.12 and 3.14). A get on another thread waits at most 0.05ms. When the sweep was a single scan, it held the lock 23–35ms and the get waited 20–33ms.

The heap costs about 64 bytes per entry on CPython 3.12 and 72 on 3.14, which the budget does not count. A removed, evicted or overwritten entry leaves its old heap record behind, including an overwrite with the same deadline: each entry points at its own record, and only that record is live. Once stale records outnumber the entries, the sweep drops them in batches of 1,000 per lock hold, freeing each batch before it yields, so a reader thread is never kept from the GIL while a large list is freed. It never rebuilds the heap in one hold, which would take about 33ms at 100,000 entries. With 100,000 live entries and 600,000 stale records, a reader waits at most a few milliseconds between gets during the compaction, against 53–80ms when the stale records were freed in one go. Where no sweep runs, in a child forked without at-fork hooks, a store rebuilds the heap in one pass once stale records outnumber the entries eight to one. That happens at most once per seven stores per entry. With the sweep running, it happens only when stores outrun the sweep by more than seven per entry per interval: over 23,000 stores a second into one namespace of 100,000 entries at the default 30 seconds. That store then holds the lock for the pass. Not rebuilding would leave those stale records, about 70 bytes each, outside the budget until the next sweep.

L2 Backend (Redis) Performance

An L2 hit with L1 disabled, the 100-user dict (23.5KB as MessagePack) and Redis in a container on the same runner takes 539 ± 3μs on the hosted EPYC 7763: the decorator, the round trip and deserialization. That is about 1,730 times an L1-only hit of the same dict on the same runner. A Redis on another machine adds its network latency on top; no guard measures that.

Latency by Serving Layer

tests/performance/terrain_canary.py records what each call of a cached read workload costs, split by the layer that answered it. It drives a Zipf workload (by default exponent 0.99 over 10,000 keys) through one unencrypted @cache function with L1 and a Redis L2. One caller makes the calls one at a time, at Poisson arrival times (200/s by default); when a slow call makes it late, it starts the next call as soon as it is free and drops the rest of the backlog, so it never runs above --rate to catch up. Each miss outlasts several arrival gaps, so the caller runs below schedule all run, most during warm-up; the recording states the rate it achieved, overall and in its busiest slice. It also states the values' size as MessagePack, for the median call and the 95th percentile (0.9 KB and 4.4 KB with the default 10,000 keys). It times each call with perf_counter_ns and attributes it to an L1 hit, an L2 hit or a miss by the change in cache_info(), cross-checked by a flag the function sets; a call whose counters and flag disagree is counted as unattributed, never guessed. The function sleeps 20 ms in place of a database call, so a miss's figure is that sleep plus the cache's own work.

It writes per-layer latency histograms (16 log-spaced bins per decade from 100 ns to 1 s) for each slice of time (3 s by default) as JSON, and prints each layer's share with its exact p50 and p99. The run starts from an empty cache under a fresh namespace and deletes nothing: entries expire after their ttl. The recording names where the Redis runs in the words --l2-where gives, which every run must pass: a URL cannot tell, since a tunnel or port-forward on localhost can reach a remote Redis. Its figures are one run on one host, not a guard:

uv run python tests/performance/terrain_canary.py --redis redis://localhost:6379/0 \
  --l2-where "same host, loopback TCP" --minutes 10 --out terrain.json \
  --harness https://github.com/cachekit-io/cachekit-py/blob/<sha>/tests/performance/terrain_canary.py

The recording names the cachekit it ran by version. A checkout reports the last release's version until the next release bumps it, so an editable install from a git checkout also gets its commit: cachekit 0.25.1+1ef1d05e, with .dirty added when src/, rust/, Cargo.toml or Cargo.lock has uncommitted or untracked changes. A VCS URL install gets its commit too. Each label holds only when the cachekit imported is the install its metadata describes. The harness refuses an install whose commit it cannot tell: a wheel file, a non-editable install of a directory, which copies the tree when it is installed, or a cachekit that something earlier on sys.path, such as a checkout on PYTHONPATH, imports in its place. A free-threaded build adds whether the GIL was on when the run ended, and on macOS the host names the chip from sysctl.

A checkout run cannot vouch that its compiled extension was rebuilt for its commit, because uv sync does not rebuild it when only rust/ changed. A recording you publish should therefore measure a release. Install it from PyPI into a fresh environment and run the harness from the checkout with that environment's Python:

uv venv --python 3.14 /tmp/terrain-env
uv pip install --python /tmp/terrain-env 'cachekit==<version>'
/tmp/terrain-env/bin/python tests/performance/terrain_canary.py --redis redis://localhost:6379/0 \
  --l2-where "same host, loopback TCP" --minutes 10 --out terrain.json \
  --harness https://github.com/cachekit-io/cachekit-py/blob/<sha>/tests/performance/terrain_canary.py

Not Measured Here

These have no figure from the run-level harness that supports a claim:

  • Encryption overhead through the decorator: test_encryption_overhead interleaves its plaintext and encrypted runs and prints both results. On the 100-user dict, the difference between the arms' run medians stayed inside its own band in every pass on a shared host and on the hosted runner until the 2026-10-31 hosted run, which put it at 9.3 ± 1.0μs (GitHub-Hosted Runner). One run that resolves it, after runs that did not, does not pin the cost down. The deterministic costs are the secure_l1_hit and serializer_encrypted rows of Instruction Budgets, beside l1_hit_backed, the same value's L1 hit through plaintext @cache with a backend (the two presets differ in more than encryption); see Zero-Knowledge Encryption for how encryption works
  • Request latency under contention: the 10-thread guard times each call after its thread is scheduled (see Decorator + L1 Hit)
  • The async decorator, serializer comparisons and a Redis on another machine

For choosing a serializer, see the Serializer Guide.

Performance Optimization Tips

1. Use L1 Cache Aggressively

Default configuration already enables L1:

from cachekit import cache

@cache  # L1 enabled by default
def expensive_function(user_id: int):
    return fetch_user_data(user_id)

L1 gives you (hosted EPYC 7763 figures, GitHub-Hosted Runner):

  • No network round trip on a hit
  • An L1-only hit in 0.311μs, against 539μs for a Redis L2 hit of the same 100-user dict
  • With an L2 backend configured, an L1 hit of the 100-user dict in 284μs, measured before the Rust decode (about half the decode's instructions since): it skips the round trip but still deserializes

2. Choose the Right Serializer

For DataFrames:

from cachekit.serializers import ArrowSerializer

@cache(serializer=ArrowSerializer())
def get_large_dataset(date: str):
    return load_dataframe(date)  # columnar Arrow format

For everything else:

@cache  # StandardSerializer (msgpack) is fine
def get_user_config(user_id: int):
    return {"settings": {...}, "preferences": {...}}

3. Batch Similar Queries

Bad (many small cache hits):

for user_id in user_ids:
    data = get_user_data(user_id)  # one decorator call per user

Good (one large cache hit):

@cache
def get_users_batch(user_ids: list[int]):
    return [fetch_user_data(uid) for uid in user_ids]

data = get_users_batch(user_ids)  # one decorator call for the batch

4. Tune TTL for Hit Rate

Short TTL (high freshness, lower hit rate):

@cache(ttl=60)  # 1 minute
def get_real_time_price(symbol: str):
    return fetch_current_price(symbol)

Long TTL (lower freshness, higher hit rate):

@cache(ttl=86400)  # 24 hours
def get_historical_data(symbol: str, date: str):
    return fetch_historical(symbol, date)  # Immutable data

5. Monitor Cache Performance

Hit rate per function: every decorated function has cache_info(), with hits, misses, the L1/L2 split and the average L2 latency, on any backend and with no scrape (API Reference).

Prometheus: a decorator with a backend records three series on the default prometheus_client registry, with no configuration:

  • cache_operations_total: operation count. Labels: operation, namespace, success, serializer
  • cache_operation_duration_ms: operation latency histogram, in milliseconds (buckets 0.01 to 1000)
  • cache_operation_size_bytes: payload size histogram, in bytes

circuit_breaker_state counts live circuit breakers per namespace and state. No series carries a hit/miss label, and the only tier label is serializer (l1_memory for an L1 hit, rust for the L2 path), so read the hit rate from cache_info(). An L1-only decorator records no get or set operations in the three series: one with backend=None, or one that resolves no backend (Backend Resolution Priority). MonitoringConfig(collect_stats=False) turns the three series off, as @cache.minimal and @cache.test do. cachekit starts no HTTP server; your app exposes the registry. See Prometheus Metrics for exposition, queries and alerts.

Performance Bottlenecks and Mitigations

Bottleneck 1: The L2 Round Trip

Problem: a Redis L2 hit takes 539μs on the hosted EPYC 7763, about 1,730 times an L1-only hit of the same 100-user dict (23.5KB as MessagePack). A Redis on another machine adds its network latency.

Mitigations:

  • Keep L1 on (the default), so repeat reads skip the round trip
  • Tune L1 size: raise L1CacheConfig(max_size_mb=...) or CACHEKIT_L1_MAX_SIZE_MB (default: 100MB per namespace) if entries are evicted early
  • Optimize L1 TTL: Match L1 TTL to data freshness requirements
  • Reduce payload size: cache only what you need; an L2 hit, and an L1 hit with a backend configured, deserialize the whole value

Bottleneck 2: Decorator Overhead

Problem: every decorated call pays for its cache key and the decorator's bookkeeping: 0.311μs per L1-only hit on the hosted EPYC 7763 when the memo answers it, and more when the memo does not take the call, so its key is built every time.

Mitigations:

  • This is acceptable for any function slow enough to be worth caching
  • Pass scalar arguments (int, short str, ...) to hot backend=None functions, so the memo answers them (see Decorator + L1 Hit)
  • Batch queries: one call that returns many items pays the overhead once

Bottleneck 3: Page Faults on Large Values

Problem: on Linux with glibc, an encrypted get or set of a 1MB value can spend a third of its CPU or more on page faults, and an incompressible set over three quarters.

Mitigation: set MALLOC_MMAP_THRESHOLD_ and MALLOC_TRIM_THRESHOLD_ together, if most cached values are around 1MB (see Large Payloads and glibc malloc).

Large Payloads and glibc malloc

A set holds several payload-sized buffers at once: the MessagePack output, the LZ4 output, the envelope, the ciphertext and the stored frame; a get holds as many. When they are freed, glibc hands the top of its heap back to the kernel once more than its trim threshold is free, and any buffer at or above its mmap threshold that neither a freed chunk nor the heap's free top can hold gets a fresh mmap. The next call takes those pages back, one page fault per 4KB. glibc raises both thresholds at run time to fit the largest mmap-ed buffer freed so far, so the fault count depends on every large allocation in the process and on the payload's shape: removing one buffer can raise the count for another shape.

Two glibc settings, set together, removed every fault from both 1MB sets measured and cut the JSON-like get's from 632 to 252 (the random-bytes get had none):

export MALLOC_MMAP_THRESHOLD_=4194304   # 4 MiB: buffers below this come from the heap, not a fresh mmap
export MALLOC_TRIM_THRESHOLD_=16777216  # 16 MiB: return the heap's free top to the kernel only once it exceeds this

Set them in the environment the process starts with, such as a container's ENV or a systemd unit's Environment=. glibc reads them once at startup, so setting os.environ from Python is too late. They apply to the whole process, not only to cachekit.

tests/performance/large_payload_bench.py ab measured each setting against the default allocator, in 30 interleaved process pairs per cell, in the Bench workflow's run on 2026-10-25 on a GitHub-hosted ubuntu-latest runner: AMD EPYC 7763, glibc 2.39, CPython 3.14.7, cachekit-core 0.7.0. The workload is @cache.io with encryption, L1 off and an in-process backend, so the HTTP response body a real backend allocates on every get is not in these figures. That is one more payload-sized buffer per get, and since every such buffer moves the fault count, the get columns can differ against a real backend in either direction; a set sends its value without one. The value is about 1MB of JSON-like text in a dict, or 1,000,000 random bytes. Each cell is the change in CPU per call, with the 95% CI of the paired change, and the faults per call under that setting:

Setting 1MB JSON-like get 1MB JSON-like set 1MB random-bytes get 1MB random-bytes set
Default glibc malloc 2.03ms, 632 faults 2.91ms, 556 faults 0.51ms, 0 faults 3.06ms, 1,442 faults
Both: MALLOC_MMAP_THRESHOLD_ 4 MiB and MALLOC_TRIM_THRESHOLD_ 16 MiB −28.5% [−32.1, −24.8], 252 faults −33.1% [−33.4, −32.8], 0 −0.1% [−1.3, +1.1], 0 −77.9% [−78.3, −77.6], 0
MALLOC_TRIM_THRESHOLD_ 16 MiB alone +34.8% [+29.5, +40.2], 981 +26.9% [+26.6, +27.3], 921 +494% [+492, +496], 1,719 +38.5% [+37.9, +39.2], 2,212
MALLOC_TOP_PAD_ 16 MiB alone −27.9% [−32.1, −23.5], 252 −33.2% [−33.7, −32.7], 0 +310% [+308, +313], 982 −23.2% [−23.7, −22.6], 983
MALLOC_TOP_PAD_ 64 MiB alone −29.9% [−34.0, −25.7], 252 −34.0% [−34.3, −33.6], 0 −1.1% [−2.8, +0.8], 0 −78.7% [−79.4, −77.9], 0

The default row is the base arm of the "both" job. A second run of that job agreed within about 1 point on every 1MB cell. Do not set one threshold alone: glibc stops adjusting either threshold once one is set, so the other stays at its 128KB default. MALLOC_TRIM_THRESHOLD_ alone was slower on all four 1MB paths. MALLOC_MMAP_THRESHOLD_ 4 MiB alone, on an Intel Xeon Platinum 8370C runner in the same run, slowed the JSON-like get by 26.6%, the random-bytes get by 80.0% and the JSON-like set by 2.3%, and sped the random-bytes set by 17.9%.

The cost of setting both, on the same runner:

  • Memory: peak RSS per process rose by 0.2 to 1.1MB, in a single-threaded process. The trim threshold is not a cap on retained memory: it only decides when the free space at the top of a heap goes back to the kernel. free() never trims free chunks below a live allocation, and buffers under 4 MiB no longer get their own mmap, so a heap fragmented by long-lived objects can hold far more than 16 MiB of free memory, in each of the several arenas a threaded process has. Size memory limits from your own process's RSS under load, not from these settings; that was not measured here.
  • A 64KB incompressible get took about a third longer: +32.7% [+31.3, +34.3], 67.0μs to 88.9μs, and +32.6% in the second run, with no page faults in either arm. Both MALLOC_TOP_PAD_ settings did the same, and also slowed 64KB sets by 4% to 6%. Every other 64B, 1KB and 64KB cell changed by 1.2% or less.

So set both for a process whose cached values are mostly around 1MB, and measure your own machine first: uv run python tests/performance/large_payload_bench.py ab --candidate-env MALLOC_MMAP_THRESHOLD_=4194304 --candidate-env MALLOC_TRIM_THRESHOLD_=16777216 runs the same comparison, and --shapes json-1m bin-1m narrows it to the 1MB cells. Nothing between 64KB and 1MB was measured, so where the settings start to pay off is not known. The MessagePack encoder grows its buffer to up to twice the value's size, so from about 2MB some buffers reach the 4 MiB mmap threshold and can be mmap-ed again once no freed chunk or free top in the heap can hold them; larger values were not measured. Free-threaded CPython (3.13t, 3.14t) allocates Python objects with its own allocator, and musl (Alpine) and macOS allocators work differently; none of them was measured.

Performance Regression Testing

Wall-clock benchmarks do not gate CI, and no test in tests/performance/ asserts a time budget: on a shared machine the run-level bands are too wide to gate on (see Measured Figures). The wall-clock suites below run in the default pytest tests/ invocation, report their figures, and fail only on behaviour:

uv run pytest tests/performance/ -v -m performance

The regression gate is the instruction budget: make perf-ir on your machine, and the Instruction budget check on every pull request (see In CI).

Instruction Budgets

make perf-ir counts the instructions each hot path executes per call and fails when a path costs 1% or more above its committed budget. It warns from 0.2%. Instruction counts are deterministic where wall clock is not: in repeated runs on one machine, every path but four held its figure within 0.04%; the Arrow round trips and file_set keep up to 0.25% of noise in one layout's figure (see Method).

Method (tests/performance/ir_budget.py; the measured process is ir_workload.py):

  • Each path runs under valgrind --tool=callgrind --separate-threads=yes, and only the main thread is counted. cachekit's background threads (L1 cleanup) vary by tens of percent between identical runs.
  • Per-call cost is (Ir[3000 calls] - Ir[1000 calls]) / 2000, so interpreter startup (~2 billion instructions) and warmup cancel. Interpreter teardown is skipped. The two real-data paths cost about 60 million instructions per call, so they run 10 and 30 calls after 5 warmup calls; startup still cancels.
  • Startup cancels only if it costs the same in every run. Three things made it vary, and the measured process takes each out. pyarrow's bundled jemalloc starts a background thread at import, and how it was scheduled against the main thread put a run in one of two startup states: JE_ARROW_MALLOC_CONF=background_thread:false,dirty_decay_ms:-1,muzzy_decay_ms:-1 turns it off without moving its real-time purging onto the main thread. pyarrow's default allocator, mimalloc, purges freed memory after a real-time delay that the freeing thread checks: MIMALLOC_PURGE_DELAY=-1. numpy.random, which pandas imports, seeds its global generator from OS entropy, and the value drawn moves the heap layout: ir_workload.pin_os_entropy seeds Python's OS entropy first. With all three, 16 runs at one layout agree within 4,000 instructions at each loop size on 16 of the 20 paths, on CPython 3.12 and 3.14.
  • Four paths keep noise this setup does not reach. The three Arrow round trips (serializer_arrow and the two real-data paths) run part of each call on pyarrow's thread pool, and how that work interleaves with the main thread's allocations moves the main thread's count by 0.5 to 2.8 million instructions per run, growing with the number of calls, with or without these settings: up to 0.1% of one layout's figure for serializer_arrow and 0.25% for the real-data paths. file_set's startup still varies by up to 126,000 instructions, from a source not identified: up to 0.15% of one layout's figure. All four sit well inside the 1% threshold.
  • The measured process has a fixed environment (PYTHONHASHSEED=0, one BLAS/OpenMP thread, a one-day L1 cleanup interval, pyarrow's allocators set as above, nothing inherited), seeded log sampling and OS entropy, and main-thread clocks that advance 1μs per read. Code that records its own duration otherwise executes more instructions when it runs slower. The cleanup sweep reads the real clock, so inside a run it would evict entries stamped with the pinned one.
  • Nothing that runs on real time lands in the measured loop: no background thread wakes, cyclic GC is off, and the GIL switch interval is long enough that the main thread gives up the GIL only where the code releases it.
  • Each path runs at five heap layouts (0 to 880 extra objects held before the workload) and its figure is the median. Any code change moves the layout, and the allocators take shorter or longer paths in some heap states: single layouts of the orjson round trip and the minimal L1 hit sat up to 2.4% from the rest. The median ignores one or two such layouts. The cheapest layout is not used, because a lucky one can sit 1% below the rest, and a budget recorded there makes every later run look like a regression.
  • ir_workload.py's text is compiled into every measured process, so editing it moves the layout the budgets were recorded at: re-record after any change to it. The gate itself (ir_budget.py) is kept out of the measured process, so editing it does not move a budget. The measured script's directory, tests/performance/, is first on the measured process's import path, so anything added there moves the layout too. A __pycache__ there, which importing ir_budget or ir_workload without -B creates (pytest does), moved serializer_orjson on CPython 3.12 by 0.16% to 2.0%, depending on the tree. Record and gate with that directory as a fresh checkout has it, and change nothing in it while a run is measuring.
  • The L2 paths and the backed L1 hit use an in-process dict backend, so they cost instructions only: no sockets, retries or timeouts.
  • The metrics collector starts synchronous and switches to batched mode (generic metrics queue for a worker thread) when its 5 s check sees more than 100 records/s. Cache operations record into per-thread cells in either mode. Pinned clocks never reach that check, so the plain paths measure a synchronous collector, and l2_hit_async_metrics the batched collector a busy long-lived process runs; it checks that no cache operation queues. Every metrics path records once per call through the same method, so one batched path covers that mode.

Budgets (instructions per call, tests/performance/ir_baselines.json):

Path What one call does CPython 3.12 CPython 3.14
l1_hit @cache(backend=None) L1 hit 2,597 2,487
l1_hit_backed @cache with a backend configured, L1 hit (never reaches the in-process dict backend; decodes the verified MessagePack L1 holds after the first hit). The backend keeps an L1 hit-latency report, as a CachekitIO backend with telemetry on does, so each call pays the 1-in-256 timing draw and the timed hits' clock reads (Telemetry) 81,354 81,592
l1_hit_backed_auto @cache with serializer="auto" and a backend that keeps no hit-latency report, L1 hit (decodes the verified MessagePack the auto serializer keeps) 92,337 93,325
minimal_l1_hit @cache.minimal(backend=None) L1 hit 2,598 2,487
l2_hit @cache, L1 disabled, L2 hit, as one in-process single-flight trip 163,783 158,141
miss @cache, L1 disabled, L2 miss, compute, L2 write, all as one in-process single-flight trip 212,909 204,218
secure_l1_hit @cache.secure L1 hit (builds its key, decrypts the ciphertext L1 holds) 125,617 126,238
l2_hit_async_metrics l2_hit with the metrics collector in batched mode 163,570 158,590
serializer_default StandardSerializer round trip, small dict 48,425 48,754
serializer_default_records StandardSerializer round trip, list of 100 six-field records (dict-heavy decode) 1,026,419 1,006,373
serializer_auto AutoSerializer round trip 64,500 65,557
serializer_orjson OrjsonSerializer round trip 21,779 22,531
serializer_arrow ArrowSerializer round trip, 100-row DataFrame 1,970,387 1,979,303
serializer_arrow_usgs ArrowSerializer round trip, real 12,535-row DataFrame (see Real data below) 61,764,630 61,311,031
serializer_auto_usgs AutoSerializer round trip, same frame (it hands the frame to Arrow) 61,759,824 61,345,651
serializer_encrypted EncryptionWrapper encrypt + decrypt round trip 89,696 91,051
byte_storage_store_40b ByteStorage.store() alone, 40-byte compressible input (under cachekit-core's 64-byte literals-only LZ4 threshold) 2,685 2,559
byte_storage_store_64k ByteStorage.store() alone, 64 KiB of address-like text 720,567 720,683
file_set FileBackend.set() overwriting one key in a 1,000-entry cache (no eviction) 75,751 73,691
object_cache_put_full ObjectCache.put() of a new key into a full 10,000-entry byte budget (evicts the LRU entry; nothing expired) 164,793 160,601

Budgets are per interpreter (minor version, build flavour, machine); an interpreter without budgets fails with no budget. Every committed budget comes from one reference machine: x86_64, Ubuntu 24.04, glibc 2.39, valgrind 3.22.0, CPython 3.12.12 and 3.14.3 (uv's builds), the release extension that uv sync builds, and the dependencies in uv.lock. Counts depend on that whole build, so on any other machine, record a baseline on main first (--update --allow-increase) and compare your branch against it. l2_hit and l2_hit_async_metrics cost the same within 0.4%: a cache operation records into per-thread cells in either mode.

Sensitivity: before the key memo, one extra BLAKE2b hash of the cache key per call raised every key-generating path by 4,700 to 4,900 instructions (l1_hit +6.1%, l2_hit_async_metrics +1.5%, miss +1.4%) and failed the gate, while the serializer paths, which generate no key, stayed within 0.11%. An interleaved wall-clock run agreed in sign (+286ns per L1 hit, median of 12 paired processes).

Limits: instruction counts do not weight cache misses or branch mispredictions. A claimed speed-up still needs an interleaved wall-clock comparison; the instruction count only guarantees the work did not grow. Paths that wait on a network backend are not covered. Cyclic-GC cost is outside the budgets; allocation and reference counting are inside. A Prometheus scrape, which sums each thread's cache-operation cells, runs outside the measured calls and is not budgeted. Part of pyarrow's work on the real-data paths is not budgeted either: about half of their instructions run on pyarrow's threads. Unrelated changes can still move a path's figure by up to 0.6% (a WARN), and a LOWER verdict on a path the change did not touch is a layout shift, not a saving: ratchet only the paths the change touched (--update --path <path>). A change to the Rust extension moves the paths that run through it more, with no change in their work: main's own source, built with and without symbol stripping, moved the ByteStorage round trip of serializer_default by 1.9%, while every function in it cost the same over 5,000 round trips, so the move is a one-time allocator event that the binary's layout puts inside one loop size. After an extension change, re-record the serializer paths that fail (--update --allow-increase --path <path>) and say why in the PR.

make perf-ir         # gate: fail on a >=1% per-call regression (needs valgrind; 200 runs, several minutes)
make perf-ir-update  # ratchet: write lower measured figures back as budgets, never higher

The gate runs at most 8 callgrind processes at a time (fewer on a smaller machine), and each path peaks at 0.5 to 1.0 GiB per run, so it can share a machine with other work. --jobs N changes that. Callgrind runs still need a bound: a workload that grows under callgrind can use many GiB, and a FileBackend set() workload passed 9 GiB in under four minutes. The gate kills a run that takes longer than --child-timeout minutes (default 15) and fails, and it kills every live run when it exits, fails, or gets SIGINT or SIGTERM. On Linux with systemd, also cap the whole gate's memory and run time:

systemd-run --user --scope -p MemoryMax=10G -p MemorySwapMax=0 -p RuntimeMaxSec=90min -- \
    uv run python tests/performance/ir_budget.py

The gate does not call systemd-run itself, because CI runners and macOS lack it. A deliberate cost increase (a new feature on the hot path) is recorded with uv run python tests/performance/ir_budget.py --update --allow-increase, and the PR states why.

Real data. tests/data/usgs_earthquakes_2024-01.parquet is a pinned slice of a public dataset: one month of the USGS earthquake catalogue, 12,535 rows by 13 columns, with timezone-aware timestamps, a nullable integer, categoricals and high-cardinality strings. tests/data/README.md records its source, licence, sha256 and the commands that made it. tests/unit/test_real_dataset.py runs on every pull request. It round-trips the frame and three results derived from it (a groupby, a pivot_table and a re-indexed filter) through AutoSerializer and ArrowSerializer, and a float64 array of its numeric columns through AutoSerializer only (ArrowSerializer takes tables, not bare arrays). Equality is exact, each encoding must carry its integrity checksum and stay at most 10% above a committed size, and the test serves a cached aggregation from a FileBackend through @cache and @cache.secure. The serializer_*_usgs paths budget the full-frame round trip:

uv run pytest tests/unit/test_real_dataset.py
uv run python tests/performance/ir_budget.py --path serializer_arrow_usgs --path serializer_auto_usgs

In CI

Every pull request runs the gate in the Instruction budget (shard K/4) jobs of .github/workflows/ci.yml, on GitHub-hosted ubuntu-latest runners with CPython 3.12. They do not compare against the committed budgets: those belong to the machine and interpreter build that recorded them, and a CPython 3.12 patch release alone read 2.0% to 3.6% over them with no code change. Instead each job builds two checkouts side by side, the pull request (its merge commit) and its base (the base branch tip), measures both on the same runner, and fails when a path costs 1% or more above the base. That is ir_budget.py --base DIR, which never writes the committed budgets; CI adds --report FILE, which writes the verdict and every path's figures as JSON.

  • A release-please pull request (head branch release-please--*) is measured against the last release instead: the tag v<version>, where <version> is the "." entry of the base's .release-please-manifest.json. That pull request is rebased onto main on every push, so against main it would measure nothing but a version bump. Against the tag, the comment shows each path's cumulative drift since the previous release, including the increases accepted one pull request at a time, and paths added since then show as new. A tag that cannot be fetched fails the job; it never falls back to main. The Instruction budget base job picks the base once per workflow run with .github/scripts/perf-ir-base.sh, and the shards and the comment both use its result, so a re-run shard or a tag moved mid-run cannot make the comment name a base the shards did not measure.
  • Both checkouts must sit at absolute paths of the same length, and the gate refuses a base that does not. The measured process's argv and import paths move its heap layout: a relative workload path against an absolute one read l1_hit 0.3% apart on identical source; at equal-length absolute paths the same pair agreed within 0.01%.
  • The two serializer_*_usgs paths are skipped (--skip): pyarrow's parquet reader can execute an instruction valgrind does not support on some CPUs, and the run then dies with SIGILL.
  • Four runners split the remaining paths (--shard 1/4 to --shard 4/4), so the check takes minutes rather than half an hour.
  • A path the base does not have yet is reported as new and not compared.
  • Each pull request carries one comment from github-actions[bot] with the result of its latest completed run: the overall verdict, the head and base commits measured, the interpreter, and every path's verdict, Ir/op on both sides and delta. Each run updates the same comment in place. A shard that ends without a verdict (a crash, an environment error, a timeout) shows as "no result", linked to its job, and the verdict reads INCOMPLETE, never PASS. The Instruction budget report job renders it from the shards' --report files with tests/performance/ir_report.py, and Instruction budget comment posts it: only that job can write to the pull request, and it never checks the code out. Pull requests from forks get no comment, because their token cannot write to the pull request; their shard logs still carry the result.
  • CI keeps no budgets of its own, so there is nothing to re-record for it, and it ignores the committed ones entirely. They are still ratcheted on your machine with make perf-ir-update. A deliberate cost increase (--update --allow-increase, with the reason in the PR) therefore shows as a red Instruction budget check and a FAIL in the pull request's instruction-budget comment: the check is not part of CI Success, and the reviewer accepts the increase at merge. A raised committed budget cannot waive it, because a budget diff cannot tell a deliberate increase from a re-record on another interpreter or extension build, which moved one path 8.8% with no code change.

To reproduce a CI verdict, check the base out beside your branch at a path of the same length, run uv sync --group dev in both, and run the gate from your branch:

git worktree add --detach ../base origin/main   # ../base must be as long as this checkout's path
(cd ../base && uv sync --python 3.12 --group dev)
uv run python tests/performance/ir_budget.py --base ../base --skip serializer_arrow_usgs --skip serializer_auto_usgs

Next Steps

Previous: Data Flow Architecture - Understand the system design Next: Comparison Guide - How cachekit compares to alternatives

See Also