Skip to content

feat(metrics): gauges for spawned's own retained structures - #297

Merged
unbalancedparentheses merged 4 commits into
mainfrom
feat/internal-structure-gauges
Nov 18, 2026
Merged

unbalancedparentheses merged 4 commits into
mainfrom
feat/internal-structure-gauges

Conversation

@unbalancedparentheses

@unbalancedparentheses unbalancedparentheses commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Why

An ethrex operator's memory growth needed a jemalloc heap-profiling build and a day-long run to rule spawned in or out. With these gauges one scrape answers it: if a spawned_* retained-structure gauge climbs with uptime while actors_active / requests_active stay flat, the growth is inside spawned (or in framework objects the application holds); if they are flat, look elsewhere.

Gauges (all under the optional metrics feature; compiled out otherwise)

Gauge Counts Where counted
spawned_reply_slots_alive request reply slots Live field in reply::slot::Slot (via sync::SlotAlive; () in the Loom model)
spawned_reply_tokens_alive ReplyTo tokens, answered or not Live field in ReplyTo
spawned_pending_replies_entries entries in every PendingReplies, unpurged terminal ones included Live in each table tuple
spawned_reply_registries_alive per-instance reply registries Live field in reply::registry::Registry
spawned_deadline_heap_entries per-instance deadline heap entries, stale/superseded included Live field in deadline::Entry
spawned_deadline_service_entries process-wide service queue, superseded and dead-owner entries included read at scrape under the service lock
spawned_deadline_service_actors owners the service will visit read at scrape under the service lock
spawned_jobs_alive job states (running work, held/queued result, handle) Live field in JobState
spawned_monitors_alive monitor-table entries (active + queued Down) insert/remove points in monitor/fence.rs, under the existing table lock (sync hooks; no-ops in the Loom model)
spawned_link_records_alive link records Live field in LinkEntry (() in the Loom model)
spawned_registered_names global registry names Live field in the registry entry
spawned_timers_alive send_after/send_interval timers still running Live moved into the timer task/thread

Documented in docs/GUIDE.md, "Retained internal structures". No new lock anywhere; nothing on send/dispatch/turn takes a lock. Live<G> is zero-sized; with metrics it is one relaxed fetch_add at construction and one fetch_sub at drop, at points that already allocate or free the structure.

TODO: #282 (disposal worker pool) is still open (checked after #298); its queue length and worker count gauges are left for that PR (noted in the GUIDE).

Depends on #298 (rebased on it)

timers_alive exposed a real leak: in threads mode every send_after/send_interval left a callback on the actor's stop token until the actor stopped (200 fired timers meant 200 retained registrations). #298 fixes it, and this branch is rebased on it, so merge #298 first. The first three commits here are this PR's. With the fix, the threads timer counts with a plain Live held by its thread. threads_timers_do_not_accumulate_on_a_running_actor checks that timers_alive returns to baseline while a long-lived threads actor keeps running (fired and cancelled timers).

Cost

benchmarks/hotpath.py gate on fede (callgrind, Ir/op, median of 3, sizes 2000/6000), head vs origin/main. hotpath.py gains --features so the metrics build can be gated too (CI still gates the default build).

benchmark default base default head Δ --features metrics base metrics head Δ
async_ct_burst 3,591 3,591 +0.0% 4,379 4,371 -0.2%
async_ct_request_seq 3,091 3,091 +0.0% 3,800 3,800 -0.0%
async_ct_request_conc16 2,304 2,304 -0.0% 3,042 3,036 -0.2%
async_ct_send_p1 634 634 +0.0% 1,097 1,099 +0.2%
async_ct_send_p2 630 630 +0.0% 1,103 1,097 -0.5%
threads_request_seq 2,113 2,112 -0.0% 2,874 2,843 -1.1%

Default build: identical instruction counts (the fields are zero-sized and have no Drop). Metrics build: within noise (±1%); the only per-request addition is the slot's add/sub. Instruction counts do not see cache-line traffic: the new counters live in the same COUNTERS static the request path already bumps under metrics; no new lock or shared line on the default path.

Tests

  • concurrency/tests/metrics_retained.rs (async + metrics, not sim): one churn per backend (24 actor pairs: requests, a raw send_request receiver awaited pending, 3 deferred replies in PendingReplies answered by a flush + 1 left for the death point, pipe/pipe_blocking jobs, monitor + link in both stop orders, send_after + send_interval, a registered name). Asserts every retained gauge rose during the round (the churn exercises it) and returns to the baseline (plus actors_active, requests_active, deferred_pending, jobs_running, mailbox_depth). long_churn_does_not_drift repeats both backends for a bounded time (SPAWNED_RETAINED_CHURN_SECS, default 6). The exported text declares each as a gauge.
  • Leak-detection proof (local mutations, not committed):
    • planted patch 22 (timer-interval-outlives-actor): tasks churn: gauges did not return to baseline: ["reply_registries_alive: baseline 48, now 96", "timers_alive: baseline 48, now 96"]
    • std::mem::forget(state.clone()) in JobSet::ended (a leaked job state): threads churn: … ["jobs_alive: baseline 24, now 48"], tasks churn: … ["jobs_alive: baseline 96, now 144"]
  • Soak harness: the twelve gauges are now soak series with trend, growth and quiesce (back to baseline after shutdown) checks.
  • fede: cargo test -p spawned-concurrency --features metrics, --no-default-features --features threads,metrics, the Loom harnesses that include the touched files (reply_slot_loom, link_fence_loom), and metrics_retained with SPAWNED_RETAINED_CHURN_SECS=120 (release) all pass.
  • 15-minute soak on fede (seed 1, 2.4M actors, 0 errors): every new gauge passes its trend check and settles to baseline at quiesce. Peaks: reply_slots_alive 12, reply_registries_alive 15, timers_alive 13, link_records_alive 2, monitors_alive 1. The run's verdict was FAIL, only on rss_mb trend (33.8 MiB/h against a 16 limit, measured on a 15-min run before the rebase with a 225 s warm-up). heap_mb was flat (0.44 MiB/h), so this is allocator warm-up on a run far shorter than the 4 h the thresholds are tuned for, not a leak. Gap: the soak workload never defers a reply or runs a job (reply_tokens_alive, pending_replies_entries, deadline_*, jobs_alive stayed 0), so those gauges are covered only by metrics_retained. Worth a follow-up scenario.
  • cargo test --workspace could not build on fede (openssl-sys for the updater examples, an environment problem); the spawned-concurrency suites above cover the touched code.
  • Baselines are taken once the gauges are quiet, so one serialized test's asynchronous tail cannot shift the next one's baseline (8 consecutive full-binary runs pass).
  • Local: fmt; clippy -D warnings for the workspace with and without spawned-concurrency/metrics, and --no-default-features with and without metrics; cargo doc with threads,metrics; all 24 planted patches git apply --check.

Model: no change because the gauges only count existing structures at their existing creation/drop points (zero-sized fields and relaxed counters); no transition, lock, ordering or outcome of any modelled protocol changes. The Loom harnesses get ()/no-op stand-ins for the new sync items, and planted patches 15/16 are regenerated only for the added counting lines (all 24 still git apply --check).

@unbalancedparentheses unbalancedparentheses added the no-model-change Modelled code changed without a model change; the PR body says why (Model: no change because ...) label Nov 17, 2026
@unbalancedparentheses
unbalancedparentheses force-pushed the feat/internal-structure-gauges branch 2 times, most recently from fdb6568 to 7d918e7 Compare November 18, 2026 00:47
Add process-wide gauges under the metrics feature for the structures the
framework keeps in memory, so growth can be attributed to spawned (or ruled
out) from one scrape instead of a heap-profiling build:

reply_slots_alive, reply_tokens_alive, pending_replies_entries,
reply_registries_alive, deadline_heap_entries, deadline_service_entries,
deadline_service_actors, jobs_alive, monitors_alive, link_records_alive,
registered_names, timers_alive.

Each is counted by a zero-sized Live<G> field at the structure's creation
and drop (one relaxed atomic per end, no lock); monitor tables count at
their insert/remove points under the existing table lock; the deadline
service's queue and owner counts are read at scrape under its own lock.
Without the feature the field is a zero-sized no-op.

tests/metrics_retained.rs churns actors in both backends (requests, raw
receivers, deferred replies in PendingReplies, jobs, monitors, links,
timers, names) and checks every gauge rises and returns to baseline; a
bounded long churn checks for drift. hotpath.py gains --features to gate
the metrics build. Planted patches 15 and 16 are regenerated for the
monitor-table counting lines.
Add the twelve gauges as soak series (trend, growth and quiesce checks), so a release soak fails when a framework structure grows or survives shutdown.
…ning actor

On top of the threads timer fix (#298): the timer thread holds a plain Live guard. Baselines are taken once the gauges are quiet, so one test's asynchronous tail cannot shift the next one's baseline.
The job-core Loom harness includes reply/slot.rs through its own sync shim, which needs the slot's gauge field type (unit, as in reply_slot_loom).
@unbalancedparentheses
unbalancedparentheses force-pushed the feat/internal-structure-gauges branch from 7d918e7 to acfc783 Compare November 18, 2026 01:03
@unbalancedparentheses
unbalancedparentheses marked this pull request as ready for review November 18, 2026 01:29
@unbalancedparentheses
unbalancedparentheses added this pull request to the merge queue Nov 18, 2026
@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

  • [P2] Churn failures can hang the test indefinitely — metrics_retained.rs:426. If round(1) panics, done.store(true, ...) is skipped. The watcher loops forever, and thread::scope waits for it during unwinding. Use a drop guard that always signals the watcher to stop.

  • [P2] Peak assertions depend on thread scheduling — metrics_retained.rs:415. Sampling every 200 μs can miss short-lived jobs or pending entries, causing assert_exercised to fail despite correct accounting. Hold representative objects behind a barrier until the watcher observes their gauges.

No production correctness issues found. Tests could not run: the installed toolchain’s offline dependency cache lacks ctrlc.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt · 275.0k in / 1.2k out · ~$0.13

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review of PR #297: retained-structure gauges

Overall this is well designed. The zero-sized Live<G> RAII field ties each count to construction and drop, which is much harder to get wrong than paired manual calls. It compiles away without metrics, and the churn test doubles as a useful leak test. My comments are mostly about the places that fall back to manual bulk counting, plus one hot-path concern.

Correctness

1. monitor/fence.rs: counting is manual and unconditional, so it can drift or underflow.

  • register (~L74, L81). monitors_added(1) assumes the insert never replaces an entry; the code comment says so. Tying the count to the result removes the assumption for free:
    if entries.insert(key, Entry::Active(value)).is_none() { monitors_added(1); }
  • remove (~L140). monitors_removed(1) runs before entries.remove(key), and it runs whatever remove returns. It is only correct if the guard above guarantees the key is present. If that invariant is ever broken, the fetch_sub on a u64 wraps, and Prometheus shows 1.8e19. Count from the result instead:
    let entry = entries.remove(key);
    if entry.is_some() { monitors_removed(1); }
  • Table dropped without close. Any table dropped while it still holds entries leaks the gauge permanently. So does any future path that uses retain/clear. The updated unit test in monitor.rs (L137–149) does exactly this: other is left in the table when it is dropped. That is harmless in the lib test binary, but it shows the hazard.
    • Consider putting a Live<Monitors> inside fence::Entry, as you already did for links, names and pending entries. Loom can alias it to () just as LinkRecordAlive does, which removes the need for bulk added/removed altogether.

2. metrics.rs snapshot() (L253–262): a reader writes global state.

Storing the service sizes into COUNTERS and then reading them back causes three problems:

  • Concurrent scrapes race each other.
  • Any path that calls read_snapshot() directly would see stale values.
  • The deadline service's lock is now taken inside a previously lock-free public function.

Patching the returned value is simpler and has none of these issues:

pub fn snapshot() -> Snapshot {
    let (entries, actors) = crate::reply::deadline::service_sizes();
    let mut s = read_snapshot();
    s.deadline_service_entries = entries as u64;
    s.deadline_service_actors = actors as u64;
    s
}
  • With this change the two fields are no longer real counters. They could be dropped from COUNTERS, provided render_counters can still emit them as gauges.
  • Please also document on snapshot() that it now takes the service lock. It must never be called while that lock is held, because std::sync::Mutex is not reentrant and the call would deadlock (for example from a future hook running inside the service).

3. instrumentation.rs disabled Live (L91–103) has no Debug impl, but the metrics version does (metrics.rs L446).

Any struct that holds _alive and uses #[derive(Debug)] (LinkEntry, JobState, Slot, ...) would then compile only with metrics. Add a matching Debug impl, and probably #[derive(Clone, Copy)]-free parity in general, so the two configurations can't diverge.

Performance

4. Every request now does extra global atomic RMWs.

Each request performs reply_slots_alive ±1, and deferred replies, registries and jobs add more. All of these are fetch_add/fetch_sub on fields of one COUNTERS static, so they share cache lines with each other and with the existing gauges.

  • Under multi-core request load, this contention/false sharing is not free, even with Relaxed ordering. "One relaxed atomic each end" understates the cost on a contended line.
  • Since you added hotpath.py --features metrics for this purpose, please quote both the gate result and a multi-core wall-clock ABBA in the PR.
  • If the cost is measurable, consider CachePadded for the hottest gauges (slots, tokens), or striped/per-thread counters summed at scrape.

Tasks timers (tasks/time.rs L44, L85)

5. let _held_until_here = &alive; at the end of the future works, but it is fragile:

  • In send_interval, it sits after the loop. If the loop ever becomes -> ! with only return exits, the statement is unreachable and only produces a warning.
  • Prefer let _alive = alive; as the first statement inside async move, mirroring the threads version.

Tests (tests/metrics_retained.rs)

6. threads_timers_do_not_accumulate_on_a_running_actor (L461–493).

  • quiet()[11] is a magic index. Look the gauge up by name instead, e.g. GAUGES.iter().position(|(n, _)| *n == "timers_alive").
  • The wait-then-assert pattern is racy.
    • If the worker's started (which arms its own two timers) has not yet run when the 150 extra timers drain, timers_alive can pass through before + 1 transiently.
    • The loop then exits, and the assert_eq! re-reads a different value.
  • Making the first call actor.request(Ask).unwrap() before arming the extra timers would guarantee started has completed.

7. Minor test notes.

  • RETAINED = 12 silently couples to the ordering of GAUGES. A debug_assert or a separate const slice would be safer.
  • usize::is_multiple_of (L247, L366) needs Rust 1.87. Check this against the crate's MSRV, or use i % 2 == 0.

Nits

  • verification/planted/patches/15-mon-down-after-demonitor.patch. The hunk header -164 doesn't match where let entry = … sits in the new fence.rs, which is ~L160 per this diff's @@ -153,6 +158,7 @@. git apply tolerates offsets, but please confirm the planted-bug job still applies the patch cleanly.
  • docs/GUIDE.md L1669. The reflowed line is now well over the wrap width used elsewhere.
  • pending.rs. The (K, ReplyTo<R>, Counted) tuple is fine. A small named struct would read better than three-way tuple patterns in every method.

The CHANGELOG, GUIDE table, soak series/thresholds (the 11 + 12 = 23 array) and the hotpath.py --features plumbing all look correct.


Automated review by Claude · claude-opus-5-5 · custom prompt · 28.3k in / 8.7k out · ~$0.29

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

Overall, this is a very clean, well-structured PR. The zero-sized Live<G> guard pattern keeps the instrumentation low-overhead (Relaxed atomic increment/decrement at allocation/drop boundaries, compiled out when metrics is disabled) and structurally tied to object lifetimes.

Here are a few issues worth addressing, primarily around test determinism and minor edge cases:

1. Potential test flakiness in concurrency/tests/metrics_retained.rs

  • Un-synchronized watcher thread and short-lived jobs_alive in churn_returns_to_baseline (concurrency/tests/metrics_retained.rs:408-432, 199-212, 328-337):
    In churn_returns_to_baseline, scope.spawn(...) starts a background polling thread without waiting for it to begin executing before running round(1) and setting done.store(true, Relaxed). On a loaded CI runner, if the OS scheduler does not schedule the watcher thread before round(1) completes, the while !done.load(...) loop can execute zero iterations, causing assert_exercised to panic with peak == 0.
    Additionally, in StartJob (lines 199-212 and 328-337), the returned JobHandle is dropped immediately and the job body (|_| 7u64) completes in microseconds. Unlike the other gauges (which stay elevated until the actor stops), jobs_alive is only above baseline for the brief moment the trivial closure and JobDone message are in flight, so a 200 µs polling loop can miss it if the watcher thread is briefly descheduled.

    • Suggestion: Store the JobHandle in a field on AsyncWorker / NativeWorker (so it stays alive until the actor stops and is dropped) and sample the peak directly at the end of async_pair / native_pair right before first.context().stop() (when all 11 gauges checked by assert_exercised are deterministically above baseline at the same time).
  • Startup race in threads_timers_do_not_accumulate_on_a_running_actor (concurrency/tests/metrics_retained.rs:483-500):
    NativeWorker::new(...).start() spawns the actor thread and returns before NativeWorker::started() has necessarily executed on that thread. If the 100 1 ms send_after timers and 50 cancelled send_interval timers finish (or leave 1 timer running) before the actor thread runs started(), metrics::snapshot().timers_alive can momentarily equal baseline (before + 1) and exit the wait loop before started() arms its own timers.

    • Suggestion: Call actor.request(Ask).unwrap(); right after let actor = NativeWorker::new("retained-timer-host".into()).start(); (at line 485) so started() is guaranteed to have run and armed its interval timer before polling timers_alive.

2. Minor issues

  • Concurrent metrics::snapshot() calls can overwrite deadline_service_* (concurrency/src/metrics.rs:254-263):

    pub fn snapshot() -> Snapshot {
        let (entries, actors) = crate::reply::deadline::service_sizes();
        COUNTERS
            .deadline_service_entries
            .store(entries as u64, Relaxed);
        COUNTERS
            .deadline_service_actors
            .store(actors as u64, Relaxed);
        read_snapshot()
    }

    snapshot() writes entries and actors to the global COUNTERS and then reads them back in read_snapshot(). Concurrent callers can interleave between store and load. Setting the fields directly on the returned Snapshot (or doing so in addition to storing in COUNTERS) avoids this race:

    pub fn snapshot() -> Snapshot {
        let (entries, actors) = crate::reply::deadline::service_sizes();
        let mut s = read_snapshot();
        s.deadline_service_entries = entries as u64;
        s.deadline_service_actors = actors as u64;
        s
    }
  • Unclosed table in unit test leaves monitors_alive incremented (concurrency/src/monitor.rs:134-150):
    In dropped_registration_removes_only_its_own_entry, fence::register is called for both mine and other (+2), mine is removed on drop (-1), and then table is dropped at the end of the test without calling fence::close(&table), leaving COUNTERS.monitors_alive at +1 in the unit-test process. Calling fence::close(&table); at the end of the test (or holding a Live<Monitors> inside fence::Entry) will keep the counter balanced.

  • Missing Debug impl on disabled Live<G> (concurrency/src/instrumentation.rs:90-98):
    metrics::live::Live<G> implements std::fmt::Debug (concurrency/src/metrics.rs:447-451), whereas disabled::live::Live<G> in concurrency/src/instrumentation.rs does not. Adding the same Debug impl to disabled::live::Live<G> ensures any future #[derive(Debug)] on a struct containing Live<G> won't compile under --features metrics while failing without metrics.


Automated review by Gemini · gemini-4-argon · custom prompt · 20.1k in / 14.0k out · ~$0.18

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

Overall the design is clean: zero-sized Live guards keep the no-metrics build free of the atomic traffic, and the gauge wiring is straightforward. A couple of issues worth addressing:

  1. Possible gauge underflow in monitor/fence.rs remove
    In remove, monitors_removed(1) is called unconditionally before the map removal result is inspected:

    removed();
    monitors_removed(1);
    match entries.remove(key) {
        Some(Entry::Active(value)) => Some(value),
        _ => None,
    }

    If the key is absent (for example, the entry was already consumed by take_queued, or remove is called twice), monitors_removed decrements the gauge anyway. Since the gauge is a u64 atomic, this underflows to a huge value instead of going negative. Even if current callers tend to guarantee presence, it’s fragile for a metric that is explicitly meant to aid leak diagnosis. Make the decrement conditional on actually removing something:

    let removed = entries.remove(key);
    if removed.is_some() {
        removed();
        monitors_removed(1);
    }
    match removed {
        Some(Entry::Active(value)) => Some(value),
        _ => None,
    }
  2. metrics::snapshot() mutates shared counters on read
    In concurrency/src/metrics.rs, the new public snapshot writes the deadline-service sizes into shared atomics before calling read_snapshot:

    COUNTERS.deadline_service_entries.store(entries as u64, Relaxed);
    COUNTERS.deadline_service_actors.store(actors as u64, Relaxed);
    read_snapshot()

    This makes a read path write shared state, and two concurrent snapshots can interleave the two stores, producing an internally inconsistent pair (entries from one scrape, actors from another). Since Snapshot fields are public, this is avoidable:

    pub fn snapshot() -> Snapshot {
        let (entries, actors) = crate::reply::deadline::service_sizes();
        let mut s = read_snapshot();
        s.deadline_service_entries = entries as u64;
        s.deadline_service_actors = actors as u64;
        s
    }

    This keeps snapshot read-only with respect to the atomics.

  3. Minor: brittle index in test
    concurrency/tests/metrics_retained.rs uses quiet()[11] to mean timers_alive in threads_timers_do_not_accumulate_on_a_running_actor. That magic index will silently change if the GAUGES array is reordered. A small named helper or const would make this maintainable.

The rest of the PR looks consistent; the metric list updates in the soak bench, thresholds, docs, and hotpath feature plumbing are all aligned.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt · 18.6k in / 33.0k out · ~$0.16

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

Findings

  • Low — concurrency/src/monitor.rs:137-151: the updated unit test leaks the process-wide monitor gauge. It now registers two real entries, but only mine is removed through MonitorRegistration; other remains in the table when the table is dropped. Because monitor accounting uses explicit monitors_removed calls rather than drop accounting, a --features metrics test run leaves spawned_monitors_alive elevated by one, potentially affecting later gauge assertions. After the assertions, drop the table guard and call fence::close(&table) (or explicitly remove other).

  • Low — concurrency/tests/metrics_retained.rs:424-456 (churn_returns_to_baseline): assert_exercised depends on nondeterministic peak sampling. The watcher may not be scheduled until after a round, or may miss short-lived states such as jobs and reply slots, causing a correct implementation to fail peak[i] > before[i]. Consider bounded retries until every gauge has been observed rising, or synchronize the workload so one instance of each structure remains alive until the watcher observes it.

Otherwise, the production instrumentation looks sound: Live is zero-sized without the feature, construction/drop accounting is balanced, relaxed atomics are appropriate for approximate gauges, timer guards live for the full task/thread lifetime, and the deadline-service scrape lock does not introduce an evident lock cycle.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt · 17.0k in / 9.2k out · ~$0.13

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

The retained-structure gauges are a sound design: RAII Live fields, relaxed atomics, metrics compiled out as a ZST, and a real churn/leak test. A few correctness and maintainability issues are worth fixing before this lands.

Monitor counts are the weak point

concurrency/src/monitor/fence.rs is the only retained structure counted by hand instead of Drop. That is easy to desync:

  • monitors_removed(1) in remove runs before entries.remove and still runs on the _ => None arm (~line 141). If that arm can be a missing key, the gauge underflows (wraps to u64::MAX). Decrement only on Some, including Queued.
  • register adds after insert (~lines 75–82). HashMap::insert drops a replaced entry with no matching monitors_removed. The “fresh MonitorRef” comment is now a load-bearing invariant; a later caller that reuses a key silently leaks the count.
  • active() sits between insert and monitors_added. A panic there leaves an entry that was never counted.

A zero-sized guard inside Entry (same pattern as LinkEntry / Counted) would make those paths structural. close’s bulk monitors_removed(entries.len()) (~line 173) is correct only if every insert was counted.

snapshot() is no longer a pure read

concurrency/src/metrics.rs (~250): snapshot() takes the deadline-service mutex and stores both sizes, then reloads them. Same-thread store/load is fine, but two scrapes can tear the pair (entries from one call, actors from the other). More important: if the deadline thread ever runs a callback while holding that same mutex, a scrape from that callback deadlocks (Mutex is not reentrant). Keep user code and snapshot() outside lock(service).

Also confirm render_prometheus calls this new snapshot() and not read_snapshot(). Otherwise deadline_service_entries / deadline_service_actors stay at the last scrape (or 0) in the export. The new test only checks # TYPE ... gauge, not that a live deadline shows up in the text.

Timer guard is correct but brittle

concurrency/src/tasks/time.rs (~44 and ~85): let _held_until_here = &alive; after the .await is required. Without a use across the await, the generator drops TimerAlive before the timer finishes and timers_alive undercounts. Please comment that; deleting the “unused” binding reintroduces the leak this gauge exists to catch. The threads backend (let _alive = alive in the worker closure) does not have this footgun.

TimerAlive::default() runs before spawn / spawn_framework. A failed spawn drops it, which is fine; the gauge is briefly high until the task is polled. Acceptable.

Default that mutates a global

Live::default (metrics.rs, fetch_add) is a side-effecting Default. That makes ..Default::default() and mem::take count an extra instance under metrics only. A Live::acquire() used at construction would be harder to call by accident. The disabled Live (instrumentation.rs ~90) also lacks the Debug impl the metrics build has; any #[derive(Debug)] on a container will pass one cfg and fail the other.

Tests

concurrency/tests/metrics_retained.rs (~478): quiet()[11] assumes timers_alive stays at index 11 (RETAINED is already a separate magic 12). A named accessor would fail at compile time instead of asserting the wrong series.

The peak watcher (Relaxed flag, 200µs sleep, peak initialized to 0) can miss a short-lived rise and fail assert_exercised, or theoretically miss done. deadline_service_actors is already excluded for that reason; deadline_heap_entries may be the next flake. Sample from the worker threads as they go, or hold a known structure across the sample.

SERIAL plus poison recovery is the right way to keep process-wide gauges honest. Good.

Minor

  • pending.rs take / retain / drain drop Counted on every removal path, including a closed token returned as None. That matches the documented “terminal until purge” rule.
  • Link, slot, job, registry, and reply-token fields are the right shape: one count per value, moved not cloned, so Drop cannot double-free the gauge. LinkRecordAlive = () in the loom shim matches _alive: Default::default().
  • Scrape-time lock and one relaxed RMW on alloc/free are the right cost model, and they stay behind metrics. --features on hotpath.py is the right way to keep that off the default gate.
  • Soak quiesce.* = 0 is slack against baseline, not an absolute zero, so a warmed deadline service does not false-fail. Trend of 10/hour will not catch a 1-per-hour leak; that is consistent with the other series, just not tight.

No change needed on the planted-patch context updates; they only keep the injections applying.


Automated review by Grok (xAI) · grok-4.7 · custom prompt · 19.7k in / 23.6k out · ~$0.18

Merged via the queue into main with commit 2f414aa Nov 18, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-model-change Modelled code changed without a model change; the PR body says why (Model: no change because ...)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant