perf(metrics): record cache operations into per-thread cells (LAB-7117)

docs/api-reference.md

@@ -360,11 +360,12 @@ orchestrator = FeatureOrchestrator(
 These components work behind the scenes to provide reliability features:
 
 #### `AsyncMetricsCollector`
-Non-blocking metrics collection system that prevents performance degradation:
-- Queue-based collection with overflow protection
-- Background thread processing
+Metrics collection kept off the shared hot path:
+- Cache operations record into per-thread cells, which the Prometheus series sum when scraped: a record takes no
+  shared lock, and a scrape counts every operation recorded before it
+- Generic counters and histograms record synchronously, or queue for a background worker once a collector is busy,
+  with overflow protection
 - Self-healing worker thread management
-- Zero impact on critical path latency
 
 #### `get_cached_redis_client()`
 Thread-local Redis client caching (`cachekit.backends.redis.client`):

docs/features/prometheus-metrics.md

@@ -331,7 +331,9 @@ field list and an example stats endpoint.
 
 ## Performance Impact
 
-No wall-clock guard times metrics recording, so the Performance Guide publishes no timing for it. Its [Instruction Budgets](../performance.md#instruction-budgets) give the instruction cost of an L2 hit with synchronous and with batched metrics recording.
+No wall-clock guard times metrics recording, so the Performance Guide publishes no timing for it. Its [Instruction Budgets](../performance.md#instruction-budgets) give the instruction cost of an L2 hit with a synchronous and with a batched collector.
+
+Each thread records a cache operation into cells of its own, and the series add those cells up when Prometheus scrapes them. So a record takes no lock another thread waits on, which matters on free-threaded Python, and a scrape counts every operation recorded before it, in either collector mode. With `PROMETHEUS_MULTIPROC_DIR` set, a scrape reads each process's files instead of these values, so cache operations then record into the series directly.
 
 ---
 

docs/free-threading.md

@@ -75,8 +75,9 @@ covered, plus what the free-threaded CI lane surfaced and sites added since:
 | `reliability/metrics_collection.py` `get_async_metrics_collector` | Double-checked module-global singleton | Safe — single-assignment publication of a fully-constructed object; worst case a benign duplicate worker-restart check |
 | `reliability/metrics_collection.py` `MetricsCollector` + `get_all_metrics` + `_metrics_lock()` | Module dicts of counters and gauges (`cache_operations_total`, `circuit_breaker_state`, ...) behind one `RLock` per PID, created with `dict.setdefault` | **Fixed** — every read and write takes the lock. A child gets its own, so a lock a parent thread held at a fork from C, with no at-fork hook run, is never waited on; before it was one module `RLock`, so a hookless child's first health check or `StructuredLogger.cache_operation()` could hang. The child's counts start from the values it inherited. Tested in `tests/unit/test_structured_logging.py` (`TestEntriesAfterFork`) |
 | `reliability/async_metrics.py` `_metrics_cache` + `_metrics_cache_lock()` | Double-checked module dict of Prometheus metric objects; one `Lock` per PID, created with `dict.setdefault`; held across `os.fork` by at-fork hooks | Safe — the lock-free read sees `None` or a fully constructed metric (single-assignment publication). `setdefault` is atomic under per-object locking, so threads racing to create a new process's lock all get the same one; a child forked without the hooks never takes its parent's lock. Tests in `tests/unit/test_async_metrics_shared_series.py` |
+| `reliability/async_metrics.py` `AsyncMetricsCollector.record_cache_operation` + `_ThreadState` + `_Shards` + `_ShardView` | Every backed cache operation counts once per collector and updates three Prometheus series. It used to write a shared `_operation_count` (unlocked, so lossy with the GIL off) and either the series' value locks or the batching queue | **Fixed** — each thread writes only its own `_ThreadState`, held in a `threading.local`: one cell per series it records into, and one operation count per collector. The series' values are `_ShardView`s that sum the live cells plus an exact retired total, read under one per-PID lock that no record takes, so a scrape and `get_stats()` count every operation recorded before them. Freed at thread exit, a `_ThreadState` moves its counts into the retired totals and drops its cells in one locked step, so reads never decrease and cells stay bounded to the live threads; sums are retired as exact partials (`math.fsum`), so a retirement never re-rounds a total one ulp lower, which `rate()` would read as a counter reset (an infinite term collapses the sum to `inf`). The lock is reentrant, because a garbage collection under it can run a finalizer on the same thread. A count move such a finalizer starts there is deferred to the next section that is not nested. One example is a cyclic collection freeing a thread's state that a traceback kept alive past the exit, so no read sees a count half moved. The at-fork hooks hold the lock across `os.fork`; a forked child starts from the counts it inherited. In prometheus_client's multiprocess mode cache operations record into the series directly. Tests in `tests/unit/test_async_metrics_per_thread.py` |
 | `backends/memcached/backend.py` `MemcachedBackend` | pymemcache `HashClient(use_pooling=True)`: one `PooledClient` per server, whose `ObjectPool` `threading.Lock` guards only the free/used deque pop and append, never socket I/O | Safe across threads: each operation owns its checked-out connection, so no socket is shared. The pool raises at `max_pool_size` rather than waiting (`tests/unit/backends/test_memcached_pooling.py`). pymemcache's failover state (`HashClient._failed_clients`) is unlocked: while a failed server is retried, concurrent operations can raise a spurious `BackendError` wrapping `KeyError` (documented in `docs/backends/memcached.md#concurrency`). Not fork-safe, by design: a child that reused its parent's backend would share the parent's pooled sockets, and could deadlock on a pool lock a parent thread held at fork. A per-PID lock would cure only the deadlock, not the shared sockets, so the rule is that a forked child builds its own backend (documented in `docs/backends/memcached.md#concurrency`) |
-| `reliability/async_metrics.py` `AsyncMetricsCollector._maybe_switch_mode`, `shutdown`, `_take_over_if_forked` | Auto sync/batched switch; `_should_check_mode` is an unlocked check-then-set, so two producers can both run a switch. A forked child inherits the batching state (queue, stop event, worker, pool lock) and prometheus_client's metric locks | **Fixed** — one `Lock` held across the switch, so concurrent switches start one worker. The lock is per PID (a per-instance dict filled with `setdefault`, like `_metrics_locks`), so a child forked while a thread is mid-switch never waits on the copy that thread still holds. `shutdown()` takes the same lock, bars any later switch from starting a worker, and sets sync mode before stopping the worker, so every later record lands synchronously; a record another thread had already chosen to queue can miss the worker's final drain (at most one per such thread). Returning to batched mode starts a worker on the existing queue only once the previous worker has exited (its exit drain assumes a single consumer); while it is alive the collector stays synchronous and retries at the next mode check, never joining on the caller's thread. Within a process the queue is never replaced, so producers always find one. **Fork:** an owner-PID take-over gives an inherited collector fresh batching state and sync mode in a forked child. The at-fork hook (after a fork made from C, the first take-over or collector built in the child instead) gives cachekit's own metrics and the default registry fresh prometheus_client locks; an application's own metrics are not covered, and neither are values in prometheus_client's multiprocess mode (`PROMETHEUS_MULTIPROC_DIR`), which share one lock the reset cannot reach, so a child forked while a parent thread holds it can still hang. No collector, inherited or built there, starts a thread in the child of a fork made from C, where at-fork hooks are skipped and `is_alive()` stays stale, nor in a child that first imports cachekit after such a fork of a parent that ran threads (uWSGI `--lazy-apps`; the detection of the `l1_cache.py` row). On free-threaded builds such a fork without `--py-call-uwsgi-fork-hooks` is unsupported: any Python code in that child can hang on a per-object lock a parent thread held, before cachekit can act. The `_take_over_if_forked`, `_reset_metric_locks` and `_reset_metric_locks_once` docstrings give the mechanics and the open cases. There is no per-record `getpid()` on the sync path. Tests in `tests/unit/test_async_metrics_mode_switch.py` |
+| `reliability/async_metrics.py` `AsyncMetricsCollector._maybe_switch_mode`, `shutdown`, `_take_over_if_forked` | Auto sync/batched switch; `_should_check_mode` is an unlocked check-then-set, so two producers can both run a switch. A forked child inherits the batching state (queue, stop event, worker, pool lock) and prometheus_client's metric locks | **Fixed** — one `Lock` held across the switch, so concurrent switches start one worker. The lock is per PID (a per-instance dict filled with `setdefault`, like `_metrics_locks`), so a child forked while a thread is mid-switch never waits on the copy that thread still holds. `shutdown()` takes the same lock, bars any later switch from starting a worker, and sets sync mode before stopping the worker, so every later record lands synchronously; a record another thread had already chosen to queue can miss the worker's final drain (at most one per such thread). Returning to batched mode starts a worker on the existing queue only once the previous worker has exited (its exit drain assumes a single consumer); while it is alive the collector stays synchronous and retries at the next mode check, never joining on the caller's thread. Within a process the queue is never replaced, so producers always find one. **Fork:** an owner-PID take-over gives an inherited collector fresh batching state and sync mode in a forked child. The at-fork hook (after a fork made from C, the first take-over or collector built in the child instead) gives cachekit's own metrics and the default registry fresh prometheus_client locks; an application's own metrics are not covered, and neither are values in prometheus_client's multiprocess mode (`PROMETHEUS_MULTIPROC_DIR`), which share one lock the reset cannot reach, so a child forked while a parent thread holds it can still hang. No collector, inherited or built there, starts a thread in the child of a fork made from C, where at-fork hooks are skipped and `is_alive()` stays stale, nor in a child that first imports cachekit after such a fork of a parent that ran threads (uWSGI `--lazy-apps`; the detection of the `l1_cache.py` row). On free-threaded builds such a fork without `--py-call-uwsgi-fork-hooks` is unsupported: any Python code in that child can hang on a per-object lock a parent thread held, before cachekit can act. The `_take_over_if_forked`, `_reset_metric_locks` and `_reset_metric_locks_once` docstrings give the mechanics and the open cases. Cache operations never queue, so only generic metrics and mode checks touch the batching state. There is no per-record `getpid()` on the sync path. Tests in `tests/unit/test_async_metrics_mode_switch.py` |
 | Rust extension (`rust/src/`, cachekit-core 0.7.2) | `#[pymodule(gil_used = false)]`; every `#[pyclass]` exposes only `&self` methods; the nonce counters (the process-wide instance counter and each encryptor's own), core's only shared state in the features cachekit-py enables, are `AtomicU64`; PyO3 enforces `Send + Sync` on pyclasses at compile time | Safe — declared free-threading-ready. (The wasm32 `Cell` nonce variant is single-threaded by target.) |
 
 ## The CI safety net