perf(object-cache): find expired entries with a deadline heap, not a scan (LAB-7106)

docs/performance.md

@@ -93,6 +93,12 @@ The deterministic figure is the `l1_hit` row of the [instruction budgets](#instr
 
 The raw L1 byte-cache lookup, without the decorator, takes 354–362ns. An L1-only hit uses a separate object cache, whose lookup no guard times on its own.
 
+### L1-Only Miss at a Full Budget
+
+An L1-only miss stores its result in the object cache. Once the function's byte budget (`l1_max_size_mb`, 100MB by default) is full, each store first removes the entries that have expired, then evicts least-recently-used ones until the new value fits. Expired entries go first wherever they sit in LRU order, so an entry read just before it expired never pushes out a live one. A min-heap of deadlines finds them: when nothing has expired, the store looks at the earliest deadline and evicts, without scanning the cache for expired entries. The one exception is the heap rebuild described below. Before the heap, every such store scanned every entry while holding the lock the function's hits wait on, so with `ttl=None` (stored for a year, so nothing expires) each one cost time in proportion to the number of entries cached.
+
+The `object_cache_put_full` row of the [instruction budgets](#instruction-budgets) measures a store into a full budget of 10,000 entries: about 160,000 instructions on CPython 3.14, most of it the size estimate of the stored value, against 5.64 million when every store scanned. The heap is memory the budget does not count: about 64 bytes per entry on CPython 3.12 and 72 on 3.14, plus one stale record for each entry removed or refreshed since the heap was last rebuilt. Once the stale records outnumber the entries, the store that finds it rebuilds the heap from the cache: one pass over the entries, about once per as many stores as there are entries.
+
 ## L2 Backend (Redis) Performance
 
 An L2 hit with L1 disabled, the 100-user dict (23.5KB as MessagePack) and Redis in a container on the same runner takes 551 ± 18μs on the hosted EPYC 7763: the decorator, the round trip and deserialization. That is about 610 times an L1-only hit of the same dict on the same runner. A Redis on another machine adds its network latency on top; no guard measures that.
@@ -290,23 +296,24 @@ The regression gate is the instruction budget: `make perf-ir` on your machine, a
 
 | Path | What one call does | CPython 3.12 | CPython 3.14 |
 |------|--------------------|-------------:|-------------:|
-| `l1_hit` | `@cache(backend=None)` L1 hit | 11,619 | 10,911 |
-| `minimal_l1_hit` | `@cache.minimal(backend=None)` L1 hit | 10,510 | 9,796 |
-| `l2_hit` | `@cache`, L1 disabled, L2 hit, as one in-process single-flight trip | 189,164 | 181,850 |
-| `miss` | `@cache`, L1 disabled, L2 miss, compute, L2 write, all as one in-process single-flight trip | 222,591 | 212,467 |
-| `secure_l1_hit` | `@cache.secure` L1 hit (builds its key, decrypts the ciphertext L1 holds) | 158,029 | 156,881 |
-| `l2_hit_async_metrics` | `l2_hit` with the metrics collector in batched mode | 178,747 | 172,693 |
-| `serializer_default` | `StandardSerializer` round trip, small dict | 58,618 | 59,638 |
-| `serializer_default_records` | `StandardSerializer` round trip, list of 100 six-field records (dict-heavy decode) | 1,423,191 | 1,424,545 |
-| `serializer_auto` | `AutoSerializer` round trip | 65,821 | 66,683 |
-| `serializer_orjson` | `OrjsonSerializer` round trip | 23,679 | 23,775 |
-| `serializer_arrow` | `ArrowSerializer` round trip, 100-row DataFrame | 1,975,014 | 1,970,889 |
-| `serializer_arrow_usgs` | `ArrowSerializer` round trip, real 12,535-row DataFrame (see Real data below) | 61,849,178 | 61,414,623 |
-| `serializer_auto_usgs` | `AutoSerializer` round trip, same frame (it hands the frame to Arrow) | 61,813,019 | 61,434,085 |
-| `serializer_encrypted` | `EncryptionWrapper` encrypt + decrypt round trip | 99,816 | 101,866 |
-| `byte_storage_store_40b` | `ByteStorage.store()` alone, 40-byte compressible input (under cachekit-core's 64-byte literals-only LZ4 threshold) | 2,656 | 2,527 |
-| `byte_storage_store_64k` | `ByteStorage.store()` alone, 64 KiB of address-like text | 720,633 | 720,645 |
-| `file_set` | `FileBackend.set()` overwriting one key in a 1,000-entry cache (no eviction) | 75,556 | 73,669 |
+| `l1_hit` | `@cache(backend=None)` L1 hit | 11,632 | 10,896 |
+| `minimal_l1_hit` | `@cache.minimal(backend=None)` L1 hit | 10,505 | 9,790 |
+| `l2_hit` | `@cache`, L1 disabled, L2 hit, as one in-process single-flight trip | 188,634 | 182,083 |
+| `miss` | `@cache`, L1 disabled, L2 miss, compute, L2 write, all as one in-process single-flight trip | 222,372 | 212,493 |
+| `secure_l1_hit` | `@cache.secure` L1 hit (builds its key, decrypts the ciphertext L1 holds) | 157,389 | 155,969 |
+| `l2_hit_async_metrics` | `l2_hit` with the metrics collector in batched mode | 178,846 | 172,851 |
+| `serializer_default` | `StandardSerializer` round trip, small dict | 58,743 | 59,601 |
+| `serializer_default_records` | `StandardSerializer` round trip, list of 100 six-field records (dict-heavy decode) | 1,421,931 | 1,422,896 |
+| `serializer_auto` | `AutoSerializer` round trip | 65,943 | 66,651 |
+| `serializer_orjson` | `OrjsonSerializer` round trip | 22,068 | 22,688 |
+| `serializer_arrow` | `ArrowSerializer` round trip, 100-row DataFrame | 1,980,225 | 1,982,998 |
+| `serializer_arrow_usgs` | `ArrowSerializer` round trip, real 12,535-row DataFrame (see Real data below) | 61,768,020 | 61,261,299 |
+| `serializer_auto_usgs` | `AutoSerializer` round trip, same frame (it hands the frame to Arrow) | 61,770,277 | 61,264,653 |
+| `serializer_encrypted` | `EncryptionWrapper` encrypt + decrypt round trip | 100,048 | 101,948 |
+| `byte_storage_store_40b` | `ByteStorage.store()` alone, 40-byte compressible input (under cachekit-core's 64-byte literals-only LZ4 threshold) | 2,656 | 2,540 |
+| `byte_storage_store_64k` | `ByteStorage.store()` alone, 64 KiB of address-like text | 720,634 | 720,648 |
+| `file_set` | `FileBackend.set()` overwriting one key in a 1,000-entry cache (no eviction) | 75,881 | 73,987 |
+| `object_cache_put_full` | `ObjectCache.put()` of a new key into a full 10,000-entry byte budget (evicts the LRU entry; nothing expired) | 164,507 | 160,678 |
 
 Budgets are per interpreter (minor version, build flavour, machine); an interpreter without budgets fails with `no budget`. They were recorded on CPython 3.12.12 and 3.14.3, x86_64, glibc 2.39, with the release extension that `uv sync` builds. Counts depend on that whole build, so on a different interpreter, extension or C library, record a baseline on `main` first (`--update --allow-increase`) and compare your branch against it. Batched mode costs the caller about 9,000 to 10,000 fewer instructions per L2 hit than synchronous recording, because the Prometheus update moves to the worker thread. The gap was about 50,000 until each label tuple's Prometheus series was bound once instead of looked up per call.
 

src/cachekit/object_cache.py

@@ -7,6 +7,7 @@
 
 from __future__ import annotations
 
+import heapq
 import math
 import os
 import random
@@ -67,18 +68,24 @@ class _Entry:
 
 
 class _ObjectCacheState:
-    """The entries, their byte total, their in-flight refresh markers, and the lock guarding all three.
+    """The entries, their byte total, their expiry heap, their in-flight refresh markers, and the lock guarding them.
 
     One object so the at-fork hook can replace all of it with a single attribute store: a thread
     still inside a critical section finishes on the state it bound. The stats and the generation
     counter stay on the cache, so a swap keeps them.
     """
 
-    __slots__ = ("lock", "refreshing", "size_bytes", "store")
+    __slots__ = ("expiry", "lock", "refreshing", "size_bytes", "store")
 
     def __init__(self) -> None:
         self.store: OrderedDict[str, _Entry] = OrderedDict()
         self.size_bytes = 0
+        # Min-heap of (expires_at, key), one per stored or refreshed deadline, so the over-bound
+        # eviction finds expired entries without scanning the store. Deletion is lazy: a removal or a
+        # refresh leaves the old tuple behind. A tuple is live while its key holds an entry with that
+        # deadline, and every entry's current deadline has one, so a dead tuple is safe to drop.
+        # push() rebuilds the heap from the store once dead tuples outnumber live entries.
+        self.expiry: list[tuple[float, str]] = []
         # In-flight refresh ownership: key -> owning generation. A refresh captures the entry's
         # generation at read time and only lands — or clears/cancels its marker — for that exact
         # generation, so a stale refresh from a replaced entry can neither resurrect data nor
@@ -102,13 +109,46 @@ def remove(self, key: str) -> None:
         self.size_bytes -= entry.size_bytes
         self.refreshing.pop(key, None)
 
+    def push(self, key: str, entry: _Entry) -> None:
+        """Record a stored entry's deadline in the expiry heap; call with lock held, after the store write.
+
+        A rebuild from the store drops the dead tuples once they outnumber the live entries. It costs
+        O(n) in the entries left, and it at least halves the heap, so it amortises to O(1) per push.
+        """
+        if len(self.expiry) >= 2 * len(self.store):
+            self.expiry = [(e.expires_at, k) for k, e in self.store.items()]
+            heapq.heapify(self.expiry)
+        else:
+            heapq.heappush(self.expiry, (entry.expires_at, key))
+
+    def remove_expired(self, now: float) -> None:
+        """Remove every entry whose deadline has passed; call with lock held.
+
+        Pops heap tuples only while the earliest deadline is due, so when nothing has expired it
+        returns after one comparison. Each due tuple costs O(log n). A due tuple is dead when its key
+        holds no entry with that deadline, as a replaced or refreshed entry leaves behind; it is dropped.
+        """
+        heap = self.expiry
+        while heap and heap[0][0] <= now:
+            expires_at, key = heapq.heappop(heap)
+            entry = self.store.get(key)
+            if entry is not None and entry.expires_at == expires_at:
+                self.remove(key)
+
 
 class ObjectCache:
     """Thread-safe in-memory cache storing Python object references directly.
 
     Uses an OrderedDict for O(1) LRU ordering. On a put() when a bound is
-    exceeded, expired entries are swept first; if still over, the oldest fresh
-    entries are evicted (LRU).
+    exceeded, expired entries are removed first; if still over, the oldest fresh
+    entries are evicted (LRU). A min-heap of deadlines finds the expired entries,
+    so a put at capacity costs O(log n) per expired entry it removes and does not
+    scan for expired entries. The heap is memory the byte bound does not
+    count: about 64 to 72 bytes per entry (one 2-tuple and its list slot on 64-bit
+    CPython), plus the tuples of removed or refreshed entries. Once those
+    outnumber the live ones, the put that finds it rebuilds the heap: an O(n)
+    pause once per about n puts at capacity, against a full scan on every one
+    before the heap.
 
     Bounds (at least one required):
     - max_entries: entry-count bound (default 256, pass None to disable)
@@ -373,6 +413,7 @@ def complete_refresh(self, key: str, version: int, value: Any, ttl: float) -> bo
             entry.size_bytes = size
             entry.refresh_failed_at = None  # success ends any retry back-off
             s.store.move_to_end(key)
+            s.push(key, entry)  # the old deadline's tuple is now dead
             # New value may be larger — restore the byte bound by evicting LRU others
             self._evict(s, extra_bytes=0, need_slot=False)
             return True
@@ -418,7 +459,9 @@ def put(self, key: str, value: Any, ttl: int) -> None:
         """Store a value in the cache.
 
         When a bound is exceeded:
-        1. Expired entries are removed first.
+        1. Expired entries are removed first, wherever they sit in LRU order.
+           The expiry heap finds them, so no entry is scanned to look for them; the one
+           put in about n that rebuilds the heap makes one pass over the entries.
         2. If still over, the oldest (LRU) fresh entries are evicted.
 
         A value whose estimated size exceeds the entire byte budget is never
@@ -454,9 +497,10 @@ def put(self, key: str, value: Any, ttl: int) -> None:
 
             now = time.monotonic()
             self._generation += 1
-            s.store[key] = _Entry(value=value, expires_at=now + ttl, cached_at=now, size_bytes=size, generation=self._generation)
+            entry = _Entry(value=value, expires_at=now + ttl, cached_at=now, size_bytes=size, generation=self._generation)
+            s.store[key] = entry  # a new key appends at the MRU end, so no move_to_end
             s.size_bytes += size
-            # No move_to_end needed — OrderedDict.__setitem__ appends new keys to end
+            s.push(key, entry)
 
     def delete(self, key: str) -> bool:
         """Remove a single entry from the cache.
@@ -488,6 +532,7 @@ def clear(self) -> None:
         with s.lock:
             s.store.clear()
             s.size_bytes = 0
+            s.expiry.clear()
             s.refreshing.clear()
 
     # ------------------------------------------------------------------
@@ -540,7 +585,8 @@ def _evict(self, s: _ObjectCacheState, extra_bytes: int, need_slot: bool) -> Non
         Must be called with s.lock held.
 
         Strategy:
-        1. If any bound is exceeded, remove all expired entries first.
+        1. If any bound is exceeded, remove all expired entries first, popping them off the
+           expiry heap: one comparison when none has expired, never a scan of the store.
         2. While still over a bound, evict the oldest (LRU) fresh entry.
 
         Args:
@@ -558,11 +604,7 @@ def over_bounds() -> bool:
         if not over_bounds():
             return
 
-        # Sweep expired entries first
-        now = time.monotonic()
-        expired_keys = [k for k, e in s.store.items() if now >= e.expires_at]
-        for k in expired_keys:
-            s.remove(k)
+        s.remove_expired(time.monotonic())
 
         # Still over a bound — evict the least-recently-used fresh entries
         while s.store and over_bounds():

tests/performance/ir_baselines.json

@@ -3,44 +3,46 @@
     "cpython-3.12-x86_64": {
       "budgets": {
         "byte_storage_store_40b": 2656,
-        "byte_storage_store_64k": 720633,
-        "file_set": 75556,
-        "l1_hit": 11619,
-        "l2_hit": 189164,
-        "l2_hit_async_metrics": 178747,
-        "minimal_l1_hit": 10510,
-        "miss": 222591,
-        "secure_l1_hit": 158029,
-        "serializer_arrow": 1975014,
-        "serializer_arrow_usgs": 61849178,
-        "serializer_auto": 65821,
-        "serializer_auto_usgs": 61813019,
-        "serializer_default": 58618,
-        "serializer_default_records": 1423191,
-        "serializer_encrypted": 99816,
-        "serializer_orjson": 23679
+        "byte_storage_store_64k": 720634,
+        "file_set": 75881,
+        "l1_hit": 11632,
+        "l2_hit": 188634,
+        "l2_hit_async_metrics": 178846,
+        "minimal_l1_hit": 10505,
+        "miss": 222372,
+        "object_cache_put_full": 164507,
+        "secure_l1_hit": 157389,
+        "serializer_arrow": 1980225,
+        "serializer_arrow_usgs": 61768020,
+        "serializer_auto": 65943,
+        "serializer_auto_usgs": 61770277,
+        "serializer_default": 58743,
+        "serializer_default_records": 1421931,
+        "serializer_encrypted": 100048,
+        "serializer_orjson": 22068
       },
       "python": "3.12.12"
     },
     "cpython-3.14-x86_64": {
       "budgets": {
-        "byte_storage_store_40b": 2527,
-        "byte_storage_store_64k": 720645,
-        "file_set": 73669,
-        "l1_hit": 10911,
-        "l2_hit": 181850,
-        "l2_hit_async_metrics": 172693,
-        "minimal_l1_hit": 9796,
-        "miss": 212467,
-        "secure_l1_hit": 156881,
-        "serializer_arrow": 1970889,
-        "serializer_arrow_usgs": 61414623,
-        "serializer_auto": 66683,
-        "serializer_auto_usgs": 61434085,
-        "serializer_default": 59638,
-        "serializer_default_records": 1424545,
-        "serializer_encrypted": 101866,
-        "serializer_orjson": 23775
+        "byte_storage_store_40b": 2540,
+        "byte_storage_store_64k": 720648,
+        "file_set": 73987,
+        "l1_hit": 10896,
+        "l2_hit": 182083,
+        "l2_hit_async_metrics": 172851,
+        "minimal_l1_hit": 9790,
+        "miss": 212493,
+        "object_cache_put_full": 160678,
+        "secure_l1_hit": 155969,
+        "serializer_arrow": 1982998,
+        "serializer_arrow_usgs": 61261299,
+        "serializer_auto": 66651,
+        "serializer_auto_usgs": 61264653,
+        "serializer_default": 59601,
+        "serializer_default_records": 1422896,
+        "serializer_encrypted": 101948,
+        "serializer_orjson": 22688
       },
       "python": "3.14.3"
     }

tests/performance/ir_budget.py

@@ -112,6 +112,7 @@
     "byte_storage_store_40b",
     "byte_storage_store_64k",
     "file_set",
+    "object_cache_put_full",
 )
 
 # ── gate ────────────────────────────────────────────────────────────────────────────────────

tests/performance/ir_workload.py

@@ -18,6 +18,7 @@
 from typing import Any
 
 FILE_SET_ENTRIES = 1000
+OBJECT_CACHE_ENTRIES = 10_000
 # ByteStorage.store() inputs. 40 B is under cachekit-core's 64 B literals-only LZ4 threshold, which
 # every serializer path's input clears (the smallest, serializer_default's msgpack, is 79 B). The
 # 64 KiB input is address-like text, so the encoder finds many short matches rather than one run.
@@ -150,6 +151,19 @@ def call(*args: Any) -> dict:
             backend.set(f"entry:{i}", payload)
         return lambda: backend.set("entry:0", payload)
 
+    if path == "object_cache_put_full":
+        # ObjectCache.put() of a new key into a full byte budget, as an L1-only miss stores its result
+        # once the budget holds OBJECT_CACHE_ENTRIES fresh entries: each put evicts the LRU head.
+        import itertools
+
+        from cachekit.object_cache import ObjectCache, _estimate_object_size
+
+        oc = ObjectCache(max_entries=None, max_size_bytes=OBJECT_CACHE_ENTRIES * _estimate_object_size(value))
+        keys = itertools.count()
+        for _ in range(OBJECT_CACHE_ENTRIES):
+            oc.put(f"entry:{next(keys)}", value, ttl=3600)  # pinned clocks never reach the deadline
+        return lambda: oc.put(f"entry:{next(keys)}", value, ttl=3600)
+
     if path in BYTE_STORAGE_PAYLOADS:
         # ByteStorage.store() alone: LZ4, xxHash3-64 and the envelope, with no serializer around it.
         from cachekit._rust_serializer import ByteStorage