Repository navigation
fix: keep user code off the shared deadline-service thread - #282
Conversation
95ef178 to
3f0a255
Compare
42c2f3f to
6999e95
Compare
🤖 Codex Code ReviewNo actionable correctness or concurrency issues found. The disposal pool keeps deadline transitions and credit returns independent of user callbacks, and payload disposal rechecks the owner’s shutdown hold. Validation was limited to code inspection: tests could not run because rustup attempted to write to a read-only directory. Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt · 325.4k in / 1.7k out · ~$0.17 |
🤖 Claude Code ReviewReview: PR #282 — deadline-service disposal poolThe design is sound. The service thread decides expiry and returns credits, and user code goes to an elastic pool. A payload handed off before the owner's exit is re-checked against the hold on the worker ( Issues1. super::disposal::defer_drop(self);This defers the drop of the
That case now costs:
all for an Suggested fix: super::disposal::defer_drop_arc(self);
2. Isolation granularity is coarser than the docs claim
Either:
3.
4. The doc says the flag is "reset once below it again". In practice it is reset only when an idle worker times out. Under sustained load that never lets a worker retire, the pool can go below the cap and hit it again with no further warning.
5. When
6.
Suggested fix: poll Minor / nits
Looks good
Automated review by Claude · claude-opus-5-5 · custom prompt · 64.3k in / 10.7k out · ~$0.47 |
🤖 DeepSeek Code ReviewThe PR is a substantial, well-documented change: dedicated disposal workers, TLA model, trace validation, and regression tests all look coherent. A few points worth checking before merge:
Overall, the architecture and tests are strong; the points above are primarily about tracing reentrancy, a couple of performance optimizations, and defensive robustness. Automated review by DeepSeek · deepseek-v4-pro · custom prompt · 43.0k in / 39.3k out · ~$0.21 |
🤖 Gemini Code ReviewThis PR looks great. The separation of expiry decisions and credit returns on the deadline-service thread from user-code execution (wakers and payload destructors) on the elastic disposal pool is clean, well-isolated, and thoroughly verified. A few highlights from the review:
No issues found — approved. Automated review by Gemini · gemini-4-argon · custom prompt · 45.6k in / 29.5k out · ~$0.39 |
🤖 Kimi Code ReviewFindings
The expiry/credit ordering and the two-phase payload Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt · 40.0k in / 13.6k out · ~$0.22 |
🤖 Grok Code ReviewThe disposal pool’s condvar protocol, the R6 hold recheck in Last drop can still run on the deadline thread
Emitter wakes are one queue item
A panicking worker permanently shrinks the poolIn Wake latency on the expiry pathTransition and credit return stay on the service thread, but the caller is not woken until a worker runs. After
|
The process-wide deadline thread now only decides expiry (terminal CAS, credit return, job notification enqueue). Wakers, payload destructors and last references it releases go to an elastic disposal pool, so a blocking user destructor or waker in one actor no longer delays other actors' deadlines.
…al control and Loom case
…st-reference-only hand-off Review follow-ups for the disposal pool: one blocking waker or destructor no longer holds back other items of the same visit (two callers of one actor); deferral after thread-local teardown runs inline instead of aborting; an unwinding visit submits what it collected; registry, job and run references are handed off only when they are the last; the 64-worker cap is logged; docs state where emit credits return.
…l hand-off (Disposal.tla)
…r the Disposal model
…he start rule and two-step drops
…sposal trace docs
6999e95 to
a6a2536
Compare
…ype code Fold the decision into Expirable::expire (it returns whether the last drop could run user code) and read only the slot word in one non-generic function, so the generic Slot code instantiated in callers' crates is unchanged. An extra per-type method had shifted LLVM's inlining of the tasks receive loop (+5.5% on async_ct_send).
Closes the 0.8 known limitation "Deadline-service follow-up" (ROADMAP). The process-wide deadline thread used to run user code inline: a caller's waker, a payload destructor, the last reference to a job's
done. One blocking callback could stall every other actor's deadlines and credit returns.Design
The deadline thread now only decides expiry. It pops due entries, makes the terminal transition (
OPEN -> TIMED_OUT, a job'sJobDeadlinenotification, a scheduled emit's fire), returns the deferral credit, unparks a parked blocking receiver, and enqueues job messages on the owner's mailbox. Whatever can run user code goes to disposal workers: wakers, payloads routed to the owner'sDropFailures, the last references to jobs, runs and the registry, and a due slot that may hold an unread reply.reply/disposal.rs(new). While the service visits a registry, it collects deferrable items in a thread-local batch and then queues them one entry per item in an elastic pool. When queued items outnumber idle workers, a worker starts (one start in flight at a time, up to 64; idle ones exit after 10 s), and each worker re-checks when it takes an item. A blocked callback holds only the item it runs. While fewer than 64 callbacks are blocked at once, no other expiry, credit return or wake-up waits for it, whether from another actor or another caller of the same actor.R6 (callers before payloads). A payload handed off is settled on the worker, under the hold lock: it is held if the owner's exit began, and dropped at its death point. A drop decided just before the exit is pre-exit work, like an inline discard. The exit never waits for it, because such a destructor could wait for this actor's callers (deferred-replies.md §5).
Cost. Requests that never expire take no new step. Empty payloads are not handed off. Due entries drop in place unless they may hold an unread reply.
What still runs on the service thread. These cases are documented in
disposal.rs, deferred-replies.md §2 and semantics.md:tracingsubscriber calls for a contained panic or the cap warning;rundrop.Beyond the cap, later items queue.
Why a pool. Deferring to the owning actor would tie a caller's timeout wake-up to a possibly wedged owner. Deferring to the caller leaves no thread for an unpolled token. A single disposal thread just moves the stall.
Tests
concurrency/tests/deadline_service.rs. Subprocess regression tests with bounded waits:All of them fail on main. The same-actor test also fails with whole-batch queueing: the second caller is woken only after 3.0 s.
Pool unit tests (
reply::disposal::tests):Loom.
reply_slot_loom.rsservice_expiry_with_handed_off_wake_vs_reply.Models
verification/tla/Disposal.tlais a companion of DeferredReplies. It models:Expire(transition and credit return on the service thread);ReleaseandOtherhand-offs;Submit, one entry per item;disposal.rsruns it: worker states, the idle count, and the start rule afterSubmitand after everyTake(one start in flight, the cap);Wait,Woke,Retire;Settledecides under the hold lock, then the destructor runs untilDone;Blockingitems.Invariants:
CreditWithTransition,WakeAfterTransition,NoDropWhileHeld(on the decision),WithinCap,TimelyExpiry,TimelyWake. Liveness:EveryWakeRuns,EveryDueExpires.Configurations:
MCDisposal, a blocking waker: 65,961 distinct states.MCDisposalDrop, a blocking destructor: 535,521 distinct states.MCDisposalCapped, one worker: 762 states. It checks a weakenedTimelyWakeBelowCap; the fullTimelyWakefails there.Negative controls (each violates its invariant):
InlineDisposalTimelyExpiryWakeBeforeTransitionWakeAfterTransitionDropWhileHeldNoDropWhileHeldBatchSerial(the shared-batch bug)TimelyWakeNoStartOnTake(stranding)TimelyWakeInlineDisposalis also kept as a control in DeferredReplies.Tied to the code. With
lifecycle-trace(compiled out otherwise), every action is recorded at its linearization point:Poolsnapshot, which becomes the model'sInitIdle;Expire,Defer(every other hand-off) andSubmitwith its start decision;Begin,Take(with its start decision),Wait,WokeandRetire, all under the pool lock;Run,SettleandDone, with worker and item ids;ExitandDeath.A worker running an item it never took, or a start the rule requires but the code skips, is rejected.
Trace scenarios (
concurrency/tests/disposal_trace.rs):disposal.itemfailpoint);disposal.keep_alive);disposal.batch_serial, written tocontrols/), which must be rejected.Five log-edit mutations are rejected as well.
Results on fede: 102/102 traces accepted, all controls pass, coverage 13/13 action keys (
coverage_floor = 13), stability 25/25.Model-sync passes with no waiver.
Performance (perf gate, callgrind Ir/op, fede, CI toolchain 1.93.0, 9 reps, on main with #284/#296)
async_ct_burstasync_ct_request_seqasync_ct_request_conc16async_ct_send_p1async_ct_send_p2threads_request_seqPASS. After the rebase onto #284/#296, the first head measured +5.5% on
async_ct_send. The cause was an extra generic method inExpirable for Slot<R>, instantiated per reply type in the caller's crate, which made LLVM stop inlining the tasks receive loop'spoll_fn. The fix folds the decision intoexpire(it returns whether the last drop could run user code) and reads only the slot word in one non-generic function. I found the cause by bisecting the commits and testing variants on fede.Review follow-ups
First review:
try_with.disposal_workers_capped.Second review:
discard_someis used at the call sites; entries drop in place;JobStateroutes what exit released throughdefer.Checks run
cargo fmt, clippy (default features, and withlifecycle-trace+failpoints),cargo test -p spawned-concurrency(595 passed), the trace and failpoint suites, the sim suites, Loom, the model-syncevents/models/pathschecks, and the planted patches still applying.