Skip to content

fix(threads): timers release their stop-token callbacks when they end - #298

Merged
unbalancedparentheses merged 1 commit into
mainfrom
fix/threads-timer-callbacks
Nov 18, 2026
Merged

unbalancedparentheses merged 1 commit into
mainfrom
fix/threads-timer-callbacks

Conversation

@unbalancedparentheses

@unbalancedparentheses unbalancedparentheses commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Bug

In threads mode every send_after / send_interval registered an on_cancel callback (a boxed closure holding the timer's mpsc Sender, which keeps the channel alive) on the actor's stop token. That list only drains when the token is cancelled, so the callbacks stayed until the actor stopped. A long-lived threads actor that keeps setting timers grew by one per timer, without bound. Found with the timers_alive gauge from #297: 200 fired send_afters left 200 registrations on a running actor.

Fix

  • spawned_rt::threads::CancellationToken stores callbacks in a slab (free list plus a per-entry sequence number). New register_on_cancel(cb) -> CancelRegistration: dropping the registration removes the callback in O(1), unless cancel() already took it. The sequence check means a stale handle never removes a later entry that reuses the slot.
  • New callbacks_registered() for diagnostics and tests.
  • on_cancel is unchanged (it keeps its callback until cancellation). cancel() still drains every callback exactly once, in registration order, outside the lock.
  • threads::send_after / send_interval: the timer thread holds both registrations (its own token's and the actor's) and drops them when it fires, is cancelled or the actor stops.
  • Tasks mode was checked and is not affected: the timer task awaits Tokio's cancelled() futures, whose waiters are removed when the futures are dropped at the end of the timer.

Unregister semantics (documented): once the drop returns the callback will not start; if cancel() took it first, it runs (or is running) on the cancelling thread.

Tests

  • Failing-first regression: threads::timer_tests::threads_timers_release_their_stop_token_callbacks_while_the_actor_runs. A running actor arms 50 short send_afters that fire, plus 50 long send_afters and 10 send_intervals that are cancelled. The actor token's callbacks_registered() must return to baseline while the actor is still running. With concurrency/src/threads/time.rs reverted to main it fails with the running actor's stop token kept 110 timer callbacks.
  • Loom (rt/tests/cancellation_loom.rs, the real token under --cfg spawned_loom):
    • unregister racing cancel: fires at most once, the captures are released, nothing stays registered;
    • one registration dropped while another registers racing the cancel: the kept one fires exactly once, the dropped one never, the late one exactly once.
  • Unit and stress (rt/src/threads/mod.rs):
    • slot reuse and removal;
    • firing order across reused slots;
    • 8 threads × 2000 register/unregister racing cancel(), 20 rounds: at most once, every kept callback exactly once, every callback dropped, 0 left registered. Repeated 10× on fede.

fede (toolchain 1.93): cargo test -p spawned-rt -p spawned-concurrency -p spawned-test and -p spawned-concurrency --no-default-features --features threads,metrics pass; the stress test passed 10 repeated runs; cancellation_loom (release) passes 4/4.

Perf gate (callgrind, Ir/op, head vs main): no change on the hot paths.

benchmark base head change
async_ct_burst 3,563 3,563 -0.0%
async_ct_request_seq 3,077 3,077 -0.0%
async_ct_request_conc16 2,294 2,294 +0.0%
async_ct_send_p1 640 640 +0.0%
async_ct_send_p2 636 636 -0.0%
threads_request_seq 2,095 2,080 -0.7%

Model-sync: no modelled file is touched (check-model-sync.py paths passes). All 24 planted patches still git apply --check. fmt and clippy -D warnings pass for the workspace with and without spawned-concurrency/metrics, and for --no-default-features.

Merge before #297, which is rebased on this.

unbalancedparentheses added a commit that referenced this pull request Nov 17, 2026
…ning actor

On top of the threads timer fix (#298): the timer thread holds a plain Live guard. Baselines are taken once the gauges are quiet, so one test's asynchronous tail cannot shift the next one's baseline.
@unbalancedparentheses
unbalancedparentheses marked this pull request as ready for review November 17, 2026 23:57
@unbalancedparentheses
unbalancedparentheses added this pull request to the merge queue Nov 17, 2026
@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

The changes look solid and well-tested.

Summary of Review

  • Memory & Resource Management (rt/src/threads/mod.rs): Replacing the append-only Vec<CancelCallback> with the slab-backed Callbacks struct and CancelRegistration cleanly resolves the unbounded callback and channel growth on long-lived actor stop tokens in threads mode.
  • Concurrency & Lock Safety (rt/src/threads/mod.rs:197-257):
    • insert releases the callbacks mutex before returning Err(callback), ensuring immediate execution on an already-cancelled token happens outside the lock.
    • CancelRegistration::drop extracts the callback under the lock and drops it after the MutexGuard is released (lines 250-255), preventing deadlocks if callback captures hold nested registrations or locks.
    • Monotonic sequence numbers (seq: u64) both guard against ABA slot-reuse hazards on remove and preserve FIFO execution order in drain.
  • Timer Lifecycle (concurrency/src/threads/time.rs:23-90): Moving registrations into the spawned timer threads in send_after and send_interval ensures both timer_token and actor_token callbacks (and their captured wake_tx senders) are cleaned up as soon as the timer fires, is cancelled, or fails to send.
  • Testing (rt/tests/cancellation_loom.rs, rt/src/threads/mod.rs, concurrency/src/threads/timer_tests.rs): Good coverage across unit, stress, integration, and Loom concurrency models for registration/unregistration/cancellation races.

No issues found; this is ready to merge.


Automated review by Gemini · gemini-4-argon · custom prompt · 7.0k in / 3.0k out · ~$0.04

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable bugs found. The registration removal is synchronized correctly, callback captures are dropped outside the mutex, and timers retain their registrations until their threads exit. The added tests cover slot reuse, cancellation races, and timer callback cleanup.

I couldn’t run the tests: the pinned toolchain is unavailable, and the installed toolchain lacks the cached ctrlc dependency.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt · 133.2k in / 762 out · ~$0.07

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: PR #298, unregisterable cancel callbacks for threads-mode timers

Summary

The PR fixes a real leak. Previously every threads-mode send_after / send_interval left a callback on the actor's stop token until the actor stopped. The fix is sound:

  • Slab with sequence numbers: a registration removes its own entry, and firing order is kept by sequence number.
  • Lock discipline: callbacks run outside the lock, so reentrancy is safe.
  • Timer threads: they now own their registrations and drop them when they finish.

The Loom models and the stress test are a good addition. I found no correctness bugs. The notes below are mostly about memory behaviour, test strength and API surface.

Correctness and concurrency (verified)

  • cancel() drops the guard before running callbacks (rt/src/threads/mod.rs, cancel). The guard is a temporary of the let callbacks = ….drain(); statement, so it is released before the callbacks run. A callback can safely register on, or unregister from, the same token.
  • CancelRegistration::drop frees captures outside the lock. removed is dropped after the guard temporary. This matters because a callback's captures could themselves hold a CancelRegistration for the same token; dropping them under the lock would deadlock.
  • Insert vs. cancel race is unchanged and still correct. The flag is checked under the lock in insert, and cancel() sets the flag before locking.
  • No inserts happen after a drain. After drain, slots is empty and insert refuses new callbacks because the token is cancelled. A stale registration's remove therefore hits get_mut(index) == None.
    • As a result, the seq check in remove is currently only defensive. Reuse-while-alive cannot happen unless a token-reset API is added later. It is still worth keeping, but a one-line comment saying so would help future readers.
  • Timer threads (concurrency/src/threads/time.rs).
    • The Disconnected arm stays unreachable while the thread holds _registrations, which is correct.
    • If the timer token is already cancelled, register_on_cancel runs the callback inline. The message is buffered, so recv_timeout returns Ok(()) immediately. Good.

Suggestions

  1. The slab never shrinks (Callbacks::remove, around lines 108–117). slots and free keep their peak length forever. For example, an actor that once had 10k timers armed at the same time keeps 10k slots for its lifetime. This is bounded, but it is the same class of issue the PR is fixing. A cheap fix:

    self.live -= 1;
    if self.live == 0 {
        self.slots.clear();
        self.free.clear();
    } else {
        self.free.push(index);
    }

    This is still O(1) amortized, and ordering is unaffected because seq keeps increasing.

  2. The second Loom model doesn't test what its doc claims (rt/tests/cancellation_loom.rs, unregister_leaves_other_registrations_to_fire).

    • drop(gone) runs on the main thread before the cancel thread is spawned. So dropped == 0 holds trivially and no unregister/cancel race is explored.
    • Fix: move drop(gone) into its own loom::thread::spawn, and change the assertion to <= 1 plus strong_count == 1. Alternatively, reword the doc comment.
    • Watch the state-space size if you add a third thread.
  3. The stress test probably exercises little of the race (cancel_registration_tests::unregister_racing_cancel_stress). With yield_now() followed by cancel() right away, most of the 16k registrations likely happen after cancellation and take the run-inline path. Consider having the workers signal through a Barrier that they have started before cancel() is called.

    • Separately, the in-worker assert!(count.load(..) <= 1) can never fail. Consider removing it.
  4. callbacks_registered() becomes public API (rt/src/threads/mod.rs, around line 210). It is useful for diagnostics, but it now becomes public API you have to keep supporting. If its main purpose is the test in timer_tests.rs, consider #[doc(hidden)]. Otherwise, document that the value is only a snapshot and is racy under concurrent use.

  5. drain now costs O(n log n) because of sort_unstable_by_key (around line 122); it was O(n) before. This only runs once per token, so it is acceptable. Since on_cancel-only tokens never free a slot, you could skip the sort when free was empty and no removals occurred, but that is optional.

Tests and docs

  • threads_timers_release_their_stop_token_callbacks_while_the_actor_runs is a good regression test.
    • The >= assertion before cancelling tolerates the in-flight 1 ms timers.
    • Waiting for the count to return to baseline is robust, provided wait_for is bounded. It appears to be, given test_wait::bounded.
    • The new Clone on Fired is needed for send_interval. Fine.
  • Docs:
    • The CHANGELOG and GUIDE wording is accurate.
    • The register_on_cancel doc correctly states the weaker guarantee: drop does not wait for a callback that is already running. That is worth keeping explicit.

Overall: LGTM, with item 1 (shrink the slab) and item 2 (make the Loom model actually race) recommended before merge.


Automated review by Claude · claude-opus-5-5 · custom prompt · 9.8k in / 5.6k out · ~$0.15

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

  • Low — clarify the drop guarantee (rt/src/threads/mod.rs:194-195). cancel() drains callbacks while holding the mutex but invokes them after releasing it. Consequently, cancel() can drain a callback, CancelRegistration::drop() can return, and the callback can start afterward. Only successful unregistration guarantees the callback will never run. The following qualification hints at this, but “Once the drop returns the callback will not start” is too absolute; explicitly state that it may run during or after drop when cancellation won the race. This does not affect timer correctness because wake-up send failures are ignored.

Otherwise, the change looks sound. The slot/sequence check prevents stale registrations from removing reused entries, callbacks still execute outside the mutex, and the timer closures retain their registrations until every exit path. Slot reuse, ordering, cancellation races, and callback release are well covered by the unit and Loom tests.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt · 5.9k in / 5.8k out · ~$0.07

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

The leak fix is correct. Sequence-numbered slab removal, the existing “flag then lock” cancel protocol, and dropping callbacks outside the mutex give exactly-once firing without retaining finished timers on the actor token. let _registrations (not let _) is the right keep-alive.

Concurrency. insert still decides under the callbacks mutex, so a registration is either drained by cancel or removed by Drop, never both. The generation check in remove (rt/src/threads/mod.rs, around the seq match) makes slot reuse safe against a stale CancelRegistration. drain empties the slab before callbacks run, and a cancelled token refuses new inserts, so a callback that drops another registration during fire cannot free a live slot. Wakeups stay on an unbounded mpsc, so cancel cannot block on the timer thread. Unregister only locks after the timer has left recv_timeout, so it does not invert with cancel’s drain.

Timers. wake_on_cancel plus holding both guards until the thread returns fixes both the one-shot and interval paths, including cancel-before-recv (immediate send, then Ok(())). Interval guards correctly live until the loop exits, not until each tick. Tasks mode is rightly untouched.

Nits (non-blocking):

  • Callbacks never shrinks below high-water mark (slots / free in insert). Callbacks themselves are freed; only None slots remain. Fine unless one actor can have a huge simultaneous timer peak.
  • drain is now O(n log n) to restore registration order. Right tradeoff; only matters for a large callback set at shutdown.
  • callbacks_registered is 0 once drain runs, even while those callbacks are still executing. The doc is close enough; don’t treat it as “callbacks not currently running.”
  • Loom comment on unregister_leaves_other_registrations_to_fire overclaims: drop(gone) happens before the cancel thread starts. The real drop/cancel race is the other loom test.

Automated review by Grok (xAI) · grok-4.7 · custom prompt · 7.6k in / 25.5k out · ~$0.17

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

The fix looks correct. The slab + sequence-number design is a good fit: unregistering is O(1), stale removal after slot reuse is safe, and drain() preserves registration order by sorting on the sequence.

A few minor observations:

  • rt/src/threads/mod.rs:120 — Callbacks::drain now sorts all callbacks on every cancel, making cancellation O(n log n). This is probably fine for stop tokens, but if an actor token can accumulate a very large number of live timer registrations, shutdown cost might become noticeable.

  • rt/src/threads/mod.rs:88 — self.next += 1 is unchecked. In practice wrapping after 2^64 registrations is unrealistic, but formally it would eventually break ordering/uniqueness if it ever wrapped.

  • CancelRegistration’s Debug impl uses entry.is_some() to report registered. After cancel() has already drained the callback but before the registration is dropped, this still reports registered: true even though the callback is no longer registered. If the field is intended to mean “currently registered,” that’s misleading; documenting it as “not yet dropped” or renaming it would clarify.

No correctness or concurrency issues found in the diff.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt · 6.4k in / 67.1k out · ~$0.27

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 18, 2026
In threads mode every send_after/send_interval registered an on_cancel callback (holding the timer's wake-up Sender) on the actor's stop token, which only drains on cancel, so a long-lived threads actor that kept arming timers grew one callback per timer until it stopped.

spawned_rt::threads::CancellationToken now stores callbacks in a slab and gains register_on_cancel, returning a CancelRegistration whose drop removes the callback in O(1) (sequence-checked, so a reused slot is never removed by a stale handle), plus callbacks_registered. cancel() still drains every callback once, in registration order. The timer thread holds both registrations (its own token's and the actor's) and drops them when it fires or is cancelled.

Tasks-mode timers are unaffected: Tokio's cancellation waiter is removed when its future drops.

Tests: a failing-first regression (110 callbacks kept on main for 50 fired + 60 cancelled timers on a running actor), Loom models of unregister racing cancel, and a multi-thread unregister/cancel stress test.
@unbalancedparentheses
unbalancedparentheses added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 1bb209a Nov 18, 2026
32 checks passed
unbalancedparentheses added a commit that referenced this pull request Nov 18, 2026
…ning actor

On top of the threads timer fix (#298): the timer thread holds a plain Live guard. Baselines are taken once the gauges are quiet, so one test's asynchronous tail cannot shift the next one's baseline.
unbalancedparentheses added a commit that referenced this pull request Nov 18, 2026
…ning actor

On top of the threads timer fix (#298): the timer thread holds a plain Live guard. Baselines are taken once the gauges are quiet, so one test's asynchronous tail cannot shift the next one's baseline.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant