Repository navigation
feat(metrics): disposal worker pool gauges - #299
Conversation
Add spawned_disposal_queue_entries, disposal_workers_alive and disposal_workers_idle, read at scrape under the pool's lock (no change on the hand-off path), and spawned_disposal_workers_capped_total, counted once per cap episode beside the existing warning (a no-op without metrics). tests/metrics_retained.rs expires deadlines in both backends with non-blocking (waiting callers' wake-ups) and blocking (a discarded emit whose destructor sleeps) disposal and checks the queue drains and every worker retires after its keep-alive (disposal.keep_alive failpoint with failpoints, the real 10 s otherwise). The soak harness judges the three gauges. Removes the GUIDE note that deferred them to #282.
🤖 Codex Code Review
No production-code issues found. Tests could not run because rustup attempted to write to a read-only directory. Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt · 84.2k in / 544 out · ~$0.07 |
🤖 Claude Code ReviewReview: PR #299, disposal pool metricsThe design is sound overall:
The main problem is in the new test. Bug: the test hangs instead of failing
The sampler thread loops
The most likely trigger is a loaded host where Fix: set struct SetOnDrop<'a>(&'a AtomicBool);
impl Drop for SetOnDrop<'_> { fn drop(&mut self) { self.0.store(true, Relaxed) } }
// first line inside the scope body:
let _stop = SetOnDrop(&done);Possible flakiness in the test
Minor
DocsThe GUIDE and TESTING updates are accurate. Removing the stale "#282 will add…" note is good. Automated review by Claude · claude-opus-5-5 · custom prompt · 10.3k in / 4.7k out · ~$0.14 |
🤖 Gemini Code ReviewThe implementation, metrics wiring, soak harness updates, and documentation look solid. One minor note on test timing:
Automated review by Gemini · gemini-4-argon · custom prompt · 7.3k in / 6.4k out · ~$0.08 |
🤖 Kimi Code Review
The production metric wiring, feature-disabled hook, and mutex-protected pool snapshot otherwise look consistent; I found no new production-path synchronization issues in the diff. Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt · 6.3k in / 5.4k out · ~$0.07 |
🤖 Grok Code ReviewThe metrics wiring matches the existing scrape-under-lock pattern and is fine. The production change is small and consistent; the new test can fail for reasons that do not mean the gauges are wrong. Production path.
Soak. Quiesce slack of 0 is only valid if the harness waits out the 10 s keep-alive (plus a scrape interval) before judging. The comment says it does — worth one check against the actual quiesce timeout, since a single early sample will fail a healthy process. No change needed on the hand-off path itself, the Automated review by Grok (xAI) · grok-4.7 · custom prompt · 8.0k in / 15.8k out · ~$0.11 |
🤖 DeepSeek Code ReviewOverall the metrics plumbing looks clean: A few concerns worth addressing:
None of these are necessarily bugs in the visible diff, but they are the main correctness/flakiness risks for this PR. Automated review by DeepSeek · deepseek-v4-pro · custom prompt · 6.9k in / 30.7k out · ~$0.13 |
What and why
Follow-up to #297, which left the disposal worker pool from #282 as a TODO. Under the
metricsfeature:spawned_disposal_queue_entries,spawned_disposal_workers_alive,spawned_disposal_workers_idle(gauges): read at scrape under the pool's own lock (disposal::pool_sizes, viaPOOL.get()so a scrape never creates the pool). Nothing is added to the hand-off or worker paths.spawned_disposal_workers_capped_total(counter): incremented once per cap episode at the same point as the existingdisposal_workers_cappedwarning (already a cold path). Withoutmetricsit is an#[inline(always)]no-op.The GUIDE "Retained internal structures" table gains the three gauges plus a paragraph on the counter, and the "will add ... when it lands" note is gone. The soak harness judges the three gauges (trend/growth/quiesce; idle workers retire within the 60 s quiesce timeout). TESTING.md's soak table and CHANGELOG are updated.
How it was verified
metrics_retained::disposal_pool_gauges_return_to_baseline: 2 rounds × 24 actors in each backend. Each round expires deferred requests while callers wait (non-blocking wake-ups) and discards a job's scheduledemit_atafter its deadline, whose destructor sleeps 50 ms (blocking items). The test checks that every blocking item ran, that peakworkers_aliveis at least 2 and at least one worker went idle, and that the queue, workers and idle gauges reach 0 within a bounded wait. It also checks that the cap counter did not move. Withfailpoints,disposal.keep_aliveshortens the keep-alive (0.5 s; passed 5/5 runs). Without it, the test waits for the real 10 s (the CI--features metricsjob runs it that way).retained_gauges_are_exportedchecks the new gauge and counter types in the Prometheus output.cargo fmt --check.clippy -D warnings: workspace default, workspace withspawned-concurrency/metrics,--no-default-featureswith metrics, andmetrics,failpoints,lifecycle-trace.cargo test -p spawned-concurrency --features metrics, and--no-default-features --features threads,metrics.#[path]-includesreply/disposal.rs, so no shim change.Model: no change because the pool's behaviour is unchanged; this only reads its sizes under the existing lock at scrape and counts the already-reported cap episode.