Skip to content

feat(metrics): disposal worker pool gauges - #299

Merged
unbalancedparentheses merged 1 commit into
mainfrom
feat/metrics-disposal-gauges
Nov 18, 2026
Merged

unbalancedparentheses merged 1 commit into
mainfrom
feat/metrics-disposal-gauges

Conversation

@unbalancedparentheses

Copy link
Copy Markdown
Contributor

What and why

Follow-up to #297, which left the disposal worker pool from #282 as a TODO. Under the metrics feature:

  • spawned_disposal_queue_entries, spawned_disposal_workers_alive, spawned_disposal_workers_idle (gauges): read at scrape under the pool's own lock (disposal::pool_sizes, via POOL.get() so a scrape never creates the pool). Nothing is added to the hand-off or worker paths.
  • spawned_disposal_workers_capped_total (counter): incremented once per cap episode at the same point as the existing disposal_workers_capped warning (already a cold path). Without metrics it is an #[inline(always)] no-op.

The GUIDE "Retained internal structures" table gains the three gauges plus a paragraph on the counter, and the "will add ... when it lands" note is gone. The soak harness judges the three gauges (trend/growth/quiesce; idle workers retire within the 60 s quiesce timeout). TESTING.md's soak table and CHANGELOG are updated.

How it was verified

  • New metrics_retained::disposal_pool_gauges_return_to_baseline: 2 rounds × 24 actors in each backend. Each round expires deferred requests while callers wait (non-blocking wake-ups) and discards a job's scheduled emit_at after its deadline, whose destructor sleeps 50 ms (blocking items). The test checks that every blocking item ran, that peak workers_alive is at least 2 and at least one worker went idle, and that the queue, workers and idle gauges reach 0 within a bounded wait. It also checks that the cap counter did not move. With failpoints, disposal.keep_alive shortens the keep-alive (0.5 s; passed 5/5 runs). Without it, the test waits for the real 10 s (the CI --features metrics job runs it that way).
  • retained_gauges_are_exported checks the new gauge and counter types in the Prometheus output.
  • cargo fmt --check. clippy -D warnings: workspace default, workspace with spawned-concurrency/metrics, --no-default-features with metrics, and metrics,failpoints,lifecycle-trace.
  • cargo test -p spawned-concurrency --features metrics, and --no-default-features --features threads,metrics.
  • Loom: no harness #[path]-includes reply/disposal.rs, so no shim change.
  • perf-gate: no change expected (hot paths untouched; the only non-scrape addition is the no-op hook on the cap path). Left to CI (valgrind).

Model: no change because the pool's behaviour is unchanged; this only reads its sizes under the existing lock at scrape and counts the already-reported cap episode.

Add spawned_disposal_queue_entries, disposal_workers_alive and disposal_workers_idle, read at scrape under the pool's lock (no change on the hand-off path), and spawned_disposal_workers_capped_total, counted once per cap episode beside the existing warning (a no-op without metrics).

tests/metrics_retained.rs expires deadlines in both backends with non-blocking (waiting callers' wake-ups) and blocking (a discarded emit whose destructor sleeps) disposal and checks the queue drains and every worker retires after its keep-alive (disposal.keep_alive failpoint with failpoints, the real 10 s otherwise). The soak harness judges the three gauges. Removes the GUIDE note that deferred them to #282.
@unbalancedparentheses unbalancedparentheses added the no-model-change Modelled code changed without a model change; the PR body says why (Model: no change because ...) label Nov 18, 2026
@unbalancedparentheses
unbalancedparentheses marked this pull request as ready for review November 18, 2026 02:27
@unbalancedparentheses
unbalancedparentheses added this pull request to the merge queue Nov 18, 2026
@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

  • [P2] Test hangs indefinitely on failure — metrics_retained.rs:763. If this timeout assertion or an earlier operation panics, done.store(true, …) is skipped. The scoped sampler keeps looping, and thread::scope waits forever for it, turning a test failure into a CI hang. Add a drop guard inside the scope that always sets done during unwinding.

No production-code issues found. Tests could not run because rustup attempted to write to a read-only directory.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt · 84.2k in / 544 out · ~$0.07

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: PR #299, disposal pool metrics

The design is sound overall:

  • pool_sizes() uses POOL.get(), so a scrape doesn't create the pool.
  • The pool and the deadline service are locked one after the other, never nested, so a scrape can't deadlock against a hand-off.
  • The no-op disposal_capped() stub keeps the counter compiled out without metrics.

The main problem is in the new test.

Bug: the test hangs instead of failing

concurrency/tests/metrics_retained.rs, disposal_pool_gauges_return_to_baseline

The sampler thread loops while !done.load(..). done is only set at the end of the scope body. If anything in the body panics, std::thread::scope waits for the sampler before it re-raises the panic, and the sampler never exits. A failure therefore becomes a CI hang. Panics that trigger this:

  • the "a discarded emit was never dropped" assert;
  • an assert!(matches!(.., RequestTimeout)) inside async_expiry or native_expiry;
  • a panicking native thread, which re-raises when the inner scope joins.

The most likely trigger is a loaded host where Slow is delivered instead of discarded. The handler sets msg.0 = None, so the counter never reaches 4 * ACTORS.

Fix: set done from a drop guard created at the top of the scope.

struct SetOnDrop<'a>(&'a AtomicBool);
impl Drop for SetOnDrop<'_> { fn drop(&mut self) { self.0.store(true, Relaxed) } }
// first line inside the scope body:
let _stop = SetOnDrop(&done);

Possible flakiness in the test

  • peak[2] >= 1. With failpoints, disposal.keep_alive shortens the keep-alive. If that makes idle workers retire almost at once, the idle window may be shorter than the sampler's 200 µs period plus scrape cost, and the assert can fail intermittently. Two options:
    • drop this assertion under failpoints;
    • keep a non-zero shortened keep-alive (e.g. ≥ 50 ms) so idle workers are reliably seen.
  • disposal.keep_alive failpoint. This PR doesn't add it to disposal.rs. Please confirm it already exists. If it doesn't, the test silently waits out the real 10 s keep-alive twice, since disposal_pool_rests runs before and after the churn. Without failpoints that costs about 20 s+ anyway, so it may be worth putting the whole test behind failpoints, or marking it #[ignore]-heavy.
  • Strict [0, 0, 0] baseline. disposal_pool_rests requires an exact zero. Any test in this binary that doesn't take SERIAL and causes a deadline expiry will start a worker and stretch the wait. This is probably fine today, but it's fragile.

Minor

  • metrics.rs, snapshot(): torn snapshots. The three values are read consistently under the pool lock, but they are then written to separate atomics and re-read by read_snapshot(). Two concurrent scrapes can interleave and export, for example, idle > alive. The deadline-service pair already has the same weakness. It's harmless for Prometheus, but if the test or the soak harness compares fields, consider returning the values directly into the Snapshot instead of round-tripping through COUNTERS.
  • thresholds.toml, trend/growth on worker gauges. disposal_workers_alive and disposal_workers_idle are hard-capped at 64 and change with expiry bursts and the 10 s retirement, so trend detection on them will be noisy. disposal_queue_entries, together with capped_total, is the actual leak signal. Consider exempting the worker gauges or giving them larger growth floors, as the file's comment does for other bounded pools.
  • thresholds.toml, quiesce. The quiesce.disposal_workers_* = 0 entries assume the quiesce timeout comfortably exceeds 10 s plus scheduling slack. Please verify the harness timeout. The comment asserts this but nothing enforces it.
  • disposal.rs, pool_sizes doc. "idle ones one woken and not yet back under the lock" is hard to parse. Suggested wording: "idle may include a worker that has been notified but has not yet reacquired the lock."
  • disposal.rs, Start::Capped arm. The string continuation line is now under-indented relative to the re-nested tracing::warn!. Run cargo fmt. Output is unaffected because \ strips the leading whitespace.

Docs

The GUIDE and TESTING updates are accurate. Removing the stale "#282 will add…" note is good.


Automated review by Claude · claude-opus-5-5 · custom prompt · 10.3k in / 4.7k out · ~$0.14

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

The implementation, metrics wiring, soak harness updates, and documentation look solid. One minor note on test timing:

  • concurrency/tests/metrics_retained.rs:589-604 (JOB_DEADLINE = 20ms): In StallJob::handle, the 20 ms job deadline starts ticking inside ctx.pipe_blocking(...) before the blocking task/OS thread is scheduled and calls job.emit_at_blocking(...). Under heavy CI load (especially during the native_expiry loop where ACTORS actor threads and ACTORS pipe_blocking threads are spawned concurrently), a worker thread taking >20 ms to start will miss the deadline window before registering the Slow emit. Giving JOB_DEADLINE (and correspondingly EMIT_AFTER / JOB_RUNS) a bit more headroom would make this test more resilient on loaded runners.

Automated review by Gemini · gemini-4-argon · custom prompt · 7.3k in / 6.4k out · ~$0.08

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

  • [P2] Test failures can hang indefinitely instead of reporting — concurrency/tests/metrics_retained.rs:736-766. The sampling thread exits only after done.store(true, ...) at line 765. If an actor-setup unwrap, an expiry assertion, or the timeout assertion at line 762 panics first, std::thread::scope still joins its scoped threads during unwinding. The sampler then loops forever with done == false, hanging the test. Set the flag through an RAII drop guard, or capture the failure, always terminate/join the sampler, and assert afterward.

The production metric wiring, feature-disabled hook, and mutex-protected pool snapshot otherwise look consistent; I found no new production-path synchronization issues in the diff.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt · 6.3k in / 5.4k out · ~$0.07

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

The metrics wiring matches the existing scrape-under-lock pattern and is fine. The production change is small and consistent; the new test can fail for reasons that do not mean the gauges are wrong.

Production path. pool_sizes behind OnceLock::get (so pre-init is (0,0,0)), poison recovery via lock, and disposal_capped() beside the existing Start::Capped warning are all sound. Keeping that read to three fields under the pool mutex is the right cost, as long as workers still drop the mutex before running user Drops — a scrape now shares that lock.

disposal_pool_gauges_return_to_baseline is racy (metrics_retained.rs).

  • peak[2] >= 1 (idle) is sampled only until done is set, and done is set when the blocking Drop increments the counter — before that worker reacquires the pool lock and becomes idle. Idle is only visible if some earlier gap happened to be sampled. With failpoints + Action::Error on disposal.keep_alive, confirm the error path still waits in the idle state. If it skips the wait, idle never sticks and this assertion fails whenever that feature is on.
  • peak[1] >= 2 assumes the pool starts another worker while one is inside a 50 ms Drop. If one submit(Vec) only spawns a single worker and then drains the queue, every blocking item runs on one thread and the assertion fails even though disposal worked. Drive the scale-up explicitly, or assert queue drain / drop count only.
  • assert_eq!(capped_total, capped_before) plus 4 * ACTORS overlapping 50 ms drops will trip the cap if ACTORS is anywhere near 64, or if leftover idle workers from the async wave are still inside the 10 s keep-alive when the native wave starts. Bound the batch, or allow a capped bump.
  • done is Relaxed. The scope join saves the peak data, but a Release store / Acquire load is the correct handshake so the sampler cannot miss the stop flag longer than it should.
  • The pool, the capped counter, and the disposal.keep_alive failpoint are process-global. SERIAL only orders tests in this file; parallel tests that expire deadlines can keep workers alive past the 20 s bound or move capped_total. That will show up as a flake, not a gauge bug.

Soak. Quiesce slack of 0 is only valid if the harness waits out the 10 s keep-alive (plus a scrape interval) before judging. The comment says it does — worth one check against the actual quiesce timeout, since a single early sample will fail a healthy process.

No change needed on the hand-off path itself, the counters! / soak SERIES length update, or the GUIDE wording.


Automated review by Grok (xAI) · grok-4.7 · custom prompt · 8.0k in / 15.8k out · ~$0.11

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

Overall the metrics plumbing looks clean: pool_sizes() safely reads under the pool lock, the counter hook is compiled out without metrics, and the soak/docs/changelog updates are consistent.

A few concerns worth addressing:

  • Test flakiness due to the 20 ms job deadline
    In concurrency/tests/metrics_retained.rs (JOB_DEADLINE = 20 ms, EMIT_AFTER = 40 ms), the blocking closure must start running and call job.emit_at_blocking(...) before the job deadline passes. If thread startup is delayed on a loaded host, emit_at_blocking may return an error and the test panics at .unwrap(), or the Slow is never scheduled and the final dropped count waits until the 10 s timeout. Consider making JOB_DEADLINE larger than the expected thread-start latency, or making emit_at_blocking handle the already-expired case instead of unwrapping.

  • Peak assertion may race the 200 µs sampler under failpoints
    In disposal_pool_gauges_return_to_baseline, the sampler calls read_disposal() every 200 µs and records max values. With failpoints configured to Action::Error, idle disposal workers may retire almost immediately. If the idle/workers peak lasts less than one sample interval, the assertions peak[2] >= 1 and possibly peak[1] >= 2 can fail. This is especially relevant for disposal_workers_idle, whose non-zero lifetime may be extremely short with the error failpoint. Consider either sampling more frequently, ensuring the failpoint keeps workers alive for at least one full sample, or replacing the sampler with a deterministic check after synchronization.

  • Verify lock ordering introduced by snapshot()
    concurrency/src/metrics.rs:264-271 now takes the deadline-service lock first, then the disposal-pool lock. If any disposal-worker path ever holds the pool lock and then calls into the deadline service, this will deadlock with the scrape path. Based on the shown diff the workers release the pool lock before processing work, but please confirm there is no reverse-order acquisition path.

  • Start::Capped counter semantics
    concurrency/src/reply/disposal.rs increments disposal_capped() before logging the warning. This is fine only if Start::Capped is already guaranteed to be delivered once per episode, not per queued item while at the cap. The changelog claims once-per-episode; make sure the existing cap-detection logic indeed coalesces repeated queued submissions.

None of these are necessarily bugs in the visible diff, but they are the main correctness/flakiness risks for this PR.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt · 6.9k in / 30.7k out · ~$0.13

Merged via the queue into main with commit e6d83ef Nov 18, 2026
58 of 59 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-model-change Modelled code changed without a model change; the PR body says why (Model: no change because ...)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant