Repository navigation
feat(metrics): gauges for spawned's own retained structures - #297
Conversation
fdb6568 to
7d918e7
Compare
Add process-wide gauges under the metrics feature for the structures the framework keeps in memory, so growth can be attributed to spawned (or ruled out) from one scrape instead of a heap-profiling build: reply_slots_alive, reply_tokens_alive, pending_replies_entries, reply_registries_alive, deadline_heap_entries, deadline_service_entries, deadline_service_actors, jobs_alive, monitors_alive, link_records_alive, registered_names, timers_alive. Each is counted by a zero-sized Live<G> field at the structure's creation and drop (one relaxed atomic per end, no lock); monitor tables count at their insert/remove points under the existing table lock; the deadline service's queue and owner counts are read at scrape under its own lock. Without the feature the field is a zero-sized no-op. tests/metrics_retained.rs churns actors in both backends (requests, raw receivers, deferred replies in PendingReplies, jobs, monitors, links, timers, names) and checks every gauge rises and returns to baseline; a bounded long churn checks for drift. hotpath.py gains --features to gate the metrics build. Planted patches 15 and 16 are regenerated for the monitor-table counting lines.
Add the twelve gauges as soak series (trend, growth and quiesce checks), so a release soak fails when a framework structure grows or survives shutdown.
…ning actor On top of the threads timer fix (#298): the timer thread holds a plain Live guard. Baselines are taken once the gauges are quiet, so one test's asynchronous tail cannot shift the next one's baseline.
The job-core Loom harness includes reply/slot.rs through its own sync shim, which needs the slot's gauge field type (unit, as in reply_slot_loom).
7d918e7 to
acfc783
Compare
🤖 Codex Code Review
No production correctness issues found. Tests could not run: the installed toolchain’s offline dependency cache lacks Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt · 275.0k in / 1.2k out · ~$0.13 |
🤖 Claude Code ReviewReview of PR #297: retained-structure gaugesOverall this is well designed. The zero-sized Correctness1.
2. Storing the service sizes into
Patching the returned value is simpler and has none of these issues: pub fn snapshot() -> Snapshot {
let (entries, actors) = crate::reply::deadline::service_sizes();
let mut s = read_snapshot();
s.deadline_service_entries = entries as u64;
s.deadline_service_actors = actors as u64;
s
}
3. Any struct that holds Performance4. Every request now does extra global atomic RMWs. Each request performs
Tasks timers (
|
🤖 Gemini Code ReviewOverall, this is a very clean, well-structured PR. The zero-sized Here are a few issues worth addressing, primarily around test determinism and minor edge cases: 1. Potential test flakiness in
|
🤖 DeepSeek Code ReviewOverall the design is clean: zero-sized
The rest of the PR looks consistent; the metric list updates in the soak bench, thresholds, docs, and hotpath feature plumbing are all aligned. Automated review by DeepSeek · deepseek-v4-pro · custom prompt · 18.6k in / 33.0k out · ~$0.16 |
🤖 Kimi Code ReviewFindings
Otherwise, the production instrumentation looks sound: Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt · 17.0k in / 9.2k out · ~$0.13 |
🤖 Grok Code ReviewThe retained-structure gauges are a sound design: RAII Monitor counts are the weak point
A zero-sized guard inside
|
Why
An ethrex operator's memory growth needed a jemalloc heap-profiling build and a day-long run to rule spawned in or out. With these gauges one scrape answers it: if a
spawned_*retained-structure gauge climbs with uptime whileactors_active/requests_activestay flat, the growth is inside spawned (or in framework objects the application holds); if they are flat, look elsewhere.Gauges (all under the optional
metricsfeature; compiled out otherwise)spawned_reply_slots_aliveLivefield inreply::slot::Slot(viasync::SlotAlive;()in the Loom model)spawned_reply_tokens_aliveReplyTotokens, answered or notLivefield inReplyTospawned_pending_replies_entriesPendingReplies, unpurged terminal ones includedLivein each table tuplespawned_reply_registries_aliveLivefield inreply::registry::Registryspawned_deadline_heap_entriesLivefield indeadline::Entryspawned_deadline_service_entriesspawned_deadline_service_actorsspawned_jobs_aliveLivefield inJobStatespawned_monitors_aliveDown)monitor/fence.rs, under the existing table lock (synchooks; no-ops in the Loom model)spawned_link_records_aliveLivefield inLinkEntry(()in the Loom model)spawned_registered_namesLivefield in the registry entryspawned_timers_alivesend_after/send_intervaltimers still runningLivemoved into the timer task/threadDocumented in docs/GUIDE.md, "Retained internal structures". No new lock anywhere; nothing on send/dispatch/turn takes a lock.
Live<G>is zero-sized; withmetricsit is one relaxedfetch_addat construction and onefetch_subat drop, at points that already allocate or free the structure.TODO: #282 (disposal worker pool) is still open (checked after #298); its queue length and worker count gauges are left for that PR (noted in the GUIDE).
Depends on #298 (rebased on it)
timers_aliveexposed a real leak: inthreadsmode everysend_after/send_intervalleft a callback on the actor's stop token until the actor stopped (200 fired timers meant 200 retained registrations). #298 fixes it, and this branch is rebased on it, so merge #298 first. The first three commits here are this PR's. With the fix, the threads timer counts with a plainLiveheld by its thread.threads_timers_do_not_accumulate_on_a_running_actorchecks thattimers_alivereturns to baseline while a long-lived threads actor keeps running (fired and cancelled timers).Cost
benchmarks/hotpath.py gateon fede (callgrind, Ir/op, median of 3, sizes 2000/6000), head vsorigin/main.hotpath.pygains--featuresso the metrics build can be gated too (CI still gates the default build).--features metricsbaseasync_ct_burstasync_ct_request_seqasync_ct_request_conc16async_ct_send_p1async_ct_send_p2threads_request_seqDefault build: identical instruction counts (the fields are zero-sized and have no
Drop). Metrics build: within noise (±1%); the only per-request addition is the slot's add/sub. Instruction counts do not see cache-line traffic: the new counters live in the sameCOUNTERSstatic the request path already bumps undermetrics; no new lock or shared line on the default path.Tests
concurrency/tests/metrics_retained.rs(async + metrics, not sim): one churn per backend (24 actor pairs: requests, a rawsend_requestreceiver awaited pending, 3 deferred replies inPendingRepliesanswered by a flush + 1 left for the death point,pipe/pipe_blockingjobs, monitor + link in both stop orders,send_after+send_interval, a registered name). Asserts every retained gauge rose during the round (the churn exercises it) and returns to the baseline (plusactors_active,requests_active,deferred_pending,jobs_running,mailbox_depth).long_churn_does_not_driftrepeats both backends for a bounded time (SPAWNED_RETAINED_CHURN_SECS, default 6). The exported text declares each as a gauge.timer-interval-outlives-actor):tasks churn: gauges did not return to baseline: ["reply_registries_alive: baseline 48, now 96", "timers_alive: baseline 48, now 96"]std::mem::forget(state.clone())inJobSet::ended(a leaked job state):threads churn: … ["jobs_alive: baseline 24, now 48"],tasks churn: … ["jobs_alive: baseline 96, now 144"]cargo test -p spawned-concurrency --features metrics,--no-default-features --features threads,metrics, the Loom harnesses that include the touched files (reply_slot_loom,link_fence_loom), andmetrics_retainedwithSPAWNED_RETAINED_CHURN_SECS=120(release) all pass.reply_slots_alive12,reply_registries_alive15,timers_alive13,link_records_alive2,monitors_alive1. The run's verdict was FAIL, only onrss_mbtrend (33.8 MiB/h against a 16 limit, measured on a 15-min run before the rebase with a 225 s warm-up).heap_mbwas flat (0.44 MiB/h), so this is allocator warm-up on a run far shorter than the 4 h the thresholds are tuned for, not a leak. Gap: the soak workload never defers a reply or runs a job (reply_tokens_alive,pending_replies_entries,deadline_*,jobs_alivestayed 0), so those gauges are covered only bymetrics_retained. Worth a follow-up scenario.cargo test --workspacecould not build on fede (openssl-sysfor theupdaterexamples, an environment problem); the spawned-concurrency suites above cover the touched code.-D warningsfor the workspace with and withoutspawned-concurrency/metrics, and--no-default-featureswith and without metrics;cargo docwiththreads,metrics; all 24 planted patchesgit apply --check.Model: no change because the gauges only count existing structures at their existing creation/drop points (zero-sized fields and relaxed counters); no transition, lock, ordering or outcome of any modelled protocol changes. The Loom harnesses get
()/no-op stand-ins for the newsyncitems, and planted patches 15/16 are regenerated only for the added counting lines (all 24 stillgit apply --check).