Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/verification.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ jobs:
- name: Record supervisor lifecycle traces
env:
LIFECYCLE_TRACE_DIR: ${{ github.workspace }}/lifecycle-traces
run: cargo test --locked -p spawned-concurrency --features lifecycle-trace,failpoints --test supervision_trace --test cohort_trace --test tree_trace --test nested_supervision --test coverage_trace --test job_trace --test reply_trace --test lifecycle_trace --test coordinator_trace
run: cargo test --locked -p spawned-concurrency --features lifecycle-trace,failpoints --test supervision_trace --test cohort_trace --test tree_trace --test nested_supervision --test coverage_trace --test job_trace --test reply_trace --test disposal_trace --test lifecycle_trace --test coordinator_trace
- name: Validate traces and negative controls
run: nix run .#check-tla -- --traces-only lifecycle-traces --negative-controls
# Which model actions, projected states and transitions the recorded
Expand Down
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,26 @@
(the selection trace missed it in 24 of 30 runs). New
`lifecycle_trace::recorded_with` test helper.

- Keep user code off the shared deadline-service thread (the known 0.8
limitation). The thread still decides expiry and returns credits, but the
wakers and payload destructors an expiry releases (a caller's waker, a
discarded scheduled emit, a displaced job result, a `done` closure and its
captures) now run on an elastic pool of disposal workers, one item at a
time, so a blocking destructor or waker no longer delays other actors'
deadlines or other callers of the same actor. A payload handed off still
waits for its owner's death point (R6). No cost
on requests that do not expire. New regression tests (tasks and threads),
a Loom case for the handed-off wake-up, an `InlineDisposal` negative
control in `DeferredReplies.tla`, and a new trace-validated model,
`Disposal.tla` (expiry and hand-off, per-item queueing, the pool's start
rule, idle count and cap, worker wake-ups after the transition, two-step
payload drops held until the death point), validated against traces that
record every hand-off and worker step (`disposal_trace`, with
`disposal.item`, `disposal.keep_alive` and `disposal.batch_serial`
failpoints; the last records a control trace from code with the
shared-batch bug, which must be rejected) and five negative controls.
Empty payloads are no longer handed off, and a due entry is dropped in
place unless it may hold an unread reply.
- Regenerate the 11 planted-bug patches (04, 10-17, 19, 20) that no longer
applied, each planting the same fault where the code now lives, and check in
the required `model-sync` job that every planted patch still applies.
Expand Down
14 changes: 7 additions & 7 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,13 +39,13 @@ handlers and the existing default behavior. New policies are explicit and opt-in
performance campaign against 0.7.3, paired four-hour soaks and the full
Lifecycle models (evidence attached to the GitHub release). Ethrex and
Ethlambda pin 0.8.0 in their draft PRs and run it on 24-hour canaries.
- **Deadline-service follow-up (known 0.8 limit):** separate user payload
disposal from shared deadline bookkeeping, with no default-path cost.
Acceptance: block a job result destructor in actor A while actor B's
unpolled token/raw deadline expires and returns its credit. Keep
foreign-waker progress assumptions explicit; never make exit wait for a
pre-exit destructor that can wait for the owner's callers. The current
scheduling limit is documented in the guide.
- **Deadline-service follow-up (done, unreleased):** the shared deadline
thread only decides expiry; wakers and payload destructors go to an elastic
disposal pool, with no default-path cost. A blocking destructor or waker in
actor A no longer delays actor B's token, raw or job deadlines
(`concurrency/tests/deadline_service.rs`, tasks and threads). The remaining
assumption (the owner's own executor wake runs on the service thread) is in
`docs/design/deferred-replies.md` §2.
- **Post-0.8 cleanups (in progress):** experiment features moved to
`benchmarks/`; the large actor and supervision files split; a lint and a
`model-sync` mapping that keep the Loom-checked cores tied to their Loom tests.
Expand Down
22 changes: 22 additions & 0 deletions concurrency/src/failpoints.rs
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,28 @@ pub const SITES: &[Site] = &[
refused, and its payload is destroyed only after every caller is \
answered (tasks and threads).",
},
Site {
name: "disposal.item",
has_error: false,
description: "A disposal worker is about to run an item the deadline service \
handed off (a wake-up or a payload drop). Delay lets the owner's \
exit begin first: a payload handed off before it is held until the \
death point, not dropped by the worker.",
},
Site {
name: "disposal.batch_serial",
has_error: true,
description: "Error: the deadline service's handed-off batch runs as one entry, \
serially on one worker (the shared-batch bug fixed in PR #282). A \
negative control: the disposal trace it records must be rejected by \
Disposal.tla.",
},
Site {
name: "disposal.keep_alive",
has_error: true,
description: "Error: an idle disposal worker waits 5 ms instead of the keep-alive \
before it retires, so a test sees workers retire and new ones start.",
},
Site {
name: "job.native_worker_unused",
has_error: false,
Expand Down
134 changes: 132 additions & 2 deletions concurrency/src/lifecycle_trace.rs
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,36 @@ impl ReplyEvent {
}
}

/// One action of `verification/tla/Disposal.tla`: the deadline service's
/// expiry and hand-off, the disposal pool's steps, and the owner's payload
/// hold. On the service thread: `Expire` (a slot's transition; `r` the
/// request, `id` its wake-up item, `x`: `wake` or `none`), `Defer` (an item
/// handed off; `x`: `drop` or `other`) and `Submit` (the visit's items
/// queued; `x`: `spawn` if a worker start was decided). In the pool (`w` the
/// worker): `Begin`, `Take` (`id`, `x` as for `Submit`), `Run` and `Done`
/// (an item starts and returns), `Settle` (a payload's hold check; `x`:
/// `held` or `dropped`), `Wait`, `Woke` and `Retire`. The owner: `Exit`
/// (its hold begins) and `Death` (the hold is released at the death point).
/// Numbers are fresh and normalized by the checker.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DisposalEvent {
pub action: &'static str,
pub w: u64,
pub id: u64,
pub r: u64,
pub x: &'static str,
}

impl DisposalEvent {
/// JSON object understood by `verification/tla/trace_check.py`.
pub fn to_json(&self) -> String {
format!(
r#"{{"a":"{}","w":{},"id":{},"r":{},"x":"{}"}}"#,
self.action, self.w, self.id, self.r, self.x
)
}
}

/// One base-lifecycle action of `verification/tla/Lifecycle.tla`, recorded
/// for the actors spawned while `record_lifecycle` is on. `t` is the actor
/// the action belongs to (the target of a signal or message, the caller of
Expand Down Expand Up @@ -486,6 +516,94 @@ mod imp {
})
}

// Deadline-service disposal (`Disposal.tla`): one process-wide recording
// at a time.
static DISPOSAL_ON: AtomicBool = AtomicBool::new(false);
static DISPOSAL: Mutex<Vec<super::DisposalEvent>> = Mutex::new(Vec::new());

/// Start recording disposal events until [`take_disposal`]. Tests that
/// record must not run concurrently with each other or with other
/// deadline expiries or actor exits in the process.
///
/// The first event, `Pool`, is the pool's state when recording starts
/// (`w`: idle workers, `r`: workers, `x`: `starting` if a start is in
/// flight); the checker starts the model from it.
pub fn record_disposal() {
crate::reply::disposal::with_pool_state(|idle, workers, starting| {
atomic(|| {
let mut events = lock(&DISPOSAL);
events.clear();
events.push(super::DisposalEvent {
action: "Pool",
w: idle as u64,
id: 0,
r: workers as u64,
x: if starting { "starting" } else { "" },
});
DISPOSAL_ON.store(true, Ordering::Release);
});
});
}

/// Stop recording and return the events in their recorded order.
pub fn take_disposal() -> Vec<super::DisposalEvent> {
atomic(|| {
DISPOSAL_ON.store(false, Ordering::Release);
std::mem::take(&mut *lock(&DISPOSAL))
})
}

fn disposal_on() -> bool {
DISPOSAL_ON.load(Ordering::Acquire)
}

/// Run `f` as one atomic step of the recorded linearization while
/// disposal events are recorded (and as is otherwise).
pub(crate) fn disposal_section<R>(f: impl FnOnce() -> R) -> R {
if disposal_on() {
atomic(f)
} else {
f()
}
}

/// A fresh slot or payload number for a disposal event (0 when off).
pub(crate) fn disposal_id() -> u64 {
static NEXT: AtomicU64 = AtomicU64::new(1);
if disposal_on() {
NEXT.fetch_add(1, Ordering::Relaxed)
} else {
0
}
}

/// Record a disposal action (inside the section of the step it names).
/// An item or request numbered 0 (taken while recording was off), or a
/// worker step off a worker, is not recorded.
pub(crate) fn disposal(action: &'static str, w: u64, id: u64, r: u64, x: &'static str) {
if !disposal_on() {
return;
}
let recorded = match action {
"Expire" => r != 0,
"Defer" => id != 0,
"Take" | "Run" | "Done" | "Settle" => id != 0 && w != 0,
"Begin" | "Wait" | "Woke" | "Retire" => w != 0,
_ => true,
};
if recorded {
atomic(|| {
lock(&DISPOSAL).push(super::DisposalEvent {
action,
w,
id,
r,
x,
});
});
}
}

/// Whether actor `id` is recorded.
pub(crate) fn life_traced(id: u64) -> bool {
LIFE_ON.load(Ordering::Acquire) && lock(&MEMBERS).contains(&id)
Expand Down Expand Up @@ -1082,10 +1200,22 @@ mod imp {
pub(crate) fn in_actor<T>(_: u64, turn: T) -> T {
turn
}
#[allow(dead_code)]
#[inline(always)]
pub(crate) fn disposal(_: &'static str, _: u64, _: u64, _: u64, _: &'static str) {}
#[inline(always)]
pub(crate) fn disposal_section<R>(f: impl FnOnce() -> R) -> R {
f()
}
#[allow(dead_code)]
#[inline(always)]
pub(crate) fn disposal_id() -> u64 {
0
}
}

#[cfg(feature = "lifecycle-trace")]
pub use imp::{
record_lifecycle, recorded, recorded_with, take, take_jobs, take_lifecycle, take_replies,
take_tree, watch_deferred,
record_disposal, record_lifecycle, recorded, recorded_with, take, take_disposal, take_jobs,
take_lifecycle, take_replies, take_tree, watch_deferred,
};
82 changes: 70 additions & 12 deletions concurrency/src/mailbox.rs
Original file line number Diff line number Diff line change
Expand Up @@ -31,27 +31,85 @@ impl DropFailures {
}
/// The owner's exit began: hold every later drop until `release`.
pub(crate) fn hold(&self) {
self.held
.lock()
.unwrap_or_else(|p| p.into_inner())
.get_or_insert_with(Vec::new);
// Disposal.tla `Exit`, linearized with the hold checks below.
crate::lifecycle_trace::disposal_section(|| {
let mut held = self.held.lock().unwrap_or_else(|p| p.into_inner());
if held.is_none() {
*held = Some(Vec::new());
crate::lifecycle_trace::disposal("Exit", 0, 0, 0, "");
}
});
}
/// Every caller was answered: drop what was held (contained), and drop
/// at once from now on. Idempotent.
pub(crate) fn release(&self) {
let held = self.held.lock().unwrap_or_else(|p| p.into_inner()).take();
// Disposal.tla `Death`: the hold ends.
let held = crate::lifecycle_trace::disposal_section(|| {
let held = self.held.lock().unwrap_or_else(|p| p.into_inner()).take();
if held.is_some() {
crate::lifecycle_trace::disposal("Death", 0, 0, 0, "");
}
held
});
for value in held.into_iter().flatten() {
self.drop_contained(value);
}
}
pub(crate) fn discard<T: Send + 'static>(&self, value: T) {
let mut held = self.held.lock().unwrap_or_else(|p| p.into_inner());
if let Some(held) = held.as_mut() {
// Moved, not dropped: no user code under the lock.
held.push(Box::new(value));
return;
/// On the deadline service thread the drop goes to a disposal worker,
/// which settles it there ([`settle`](Self::settle)): the hold is
/// re-checked, so a drop deferred before the owner's exit still waits for
/// its death point.
pub(crate) fn discard<T: Send + 'static>(self: &Arc<Self>, value: T) {
let mut value = Some(value);
// One step of Disposal.tla with `Exit` and `Death`: the hold check
// and, on the service thread, `Release` (the hand-off, recorded as
// `Defer` with `x: drop`).
let inline = crate::lifecycle_trace::disposal_section(|| {
let mut held = self.held.lock().unwrap_or_else(|p| p.into_inner());
if let Some(held) = held.as_mut() {
// Moved, not dropped: no user code under the lock.
held.push(Box::new(value.take()));
return false;
}
drop(held);
if crate::reply::disposal::collecting() {
let failures = self.clone();
let value = value.take();
crate::reply::disposal::defer_as("drop", move || failures.settle(value));
return false;
}
true
});
if inline {
self.drop_contained(value);
}
}
/// [`discard`](Self::discard) of an optional payload: nothing to dispose
/// of (and no hand-off) when it is `None`.
#[inline]
pub(crate) fn discard_some<T: Send + 'static>(self: &Arc<Self>, value: Option<T>) {
if let Some(value) = value {
self.discard(value);
}
drop(held);
}
/// A disposal worker's step for a payload the service handed off
/// (Disposal.tla `Settle`, then the destructor until `Done`): held if
/// the owner's exit began meanwhile (dropped at its death point), else
/// dropped now. A drop decided before the exit is pre-exit work and may
/// still run while the exit proceeds; the exit never waits for it
/// (deferred-replies.md §5).
fn settle<T: Send + 'static>(&self, value: T) {
let mut value = Some(value);
let (w, item) = crate::reply::disposal::current();
crate::lifecycle_trace::disposal_section(|| {
let mut held = self.held.lock().unwrap_or_else(|p| p.into_inner());
if let Some(held) = held.as_mut() {
held.push(Box::new(value.take()));
crate::lifecycle_trace::disposal("Settle", w, item, 0, "held");
} else {
crate::lifecycle_trace::disposal("Settle", w, item, 0, "dropped");
}
});
self.drop_contained(value);
}
/// Drop at once (contained): the mailbox's own items, destroyed after
Expand Down
35 changes: 25 additions & 10 deletions concurrency/src/reply/deadline.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,9 @@
//! due entries even when no caller polls, so credits return on time:
//!
//! - real time: one process-wide service thread with its own timer, so
//! per-test Tokio runtimes cannot break it;
//! per-test Tokio runtimes cannot break it. What an expiry releases
//! (wakers, payload destructors) goes to a disposal worker
//! (`super::disposal`, which lists what still runs inline);
//! - under `spawned_sim`: a task of the simulated runtime per actor heap, so
//! clock domains never mix and virtual time drives it.

Expand All @@ -25,8 +27,14 @@ use std::sync::{Arc, Weak};
pub(crate) trait Expirable: Send + Sync {
/// The current deadline while the item is pending, else `None`.
fn pending_deadline(&self) -> Option<u64>;
/// The deadline passed (`now >= deadline`): expire the item.
fn expire(&self, now: u64);
/// Service a popped entry: if `due` (its deadline passed, `now >=
/// deadline`), expire it. Returns whether dropping the last reference
/// could still run user code (an unread reply): the service hands such a
/// drop off ([`super::disposal`]) and drops the others in place, with no
/// hand-off. Job entries route their user payloads through the owner's
/// drop sink (deferred there), so they return `false`. One method, not
/// two: each is instantiated per reply type in the caller's crate.
fn expire(&self, now: u64, due: bool) -> bool;
}

struct Entry {
Expand Down Expand Up @@ -304,15 +312,22 @@ mod thread_service {
state.actors.remove(&key);
drop(state);
// Entries are contained one by one; this also covers the
// visit itself and dropping the upgraded registry, so the
// one process-wide service never stops.
super::super::registry::contain_entry(|| {
if let Some(registry) = weak.upgrade() {
if let Some(next) = registry.service_deadlines(now) {
schedule(&registry, next);
// visit itself, so the one process-wide service never
// stops. Whatever the visit releases that can run user
// code (wakers, payloads, the upgraded registry) goes to
// a disposal worker: this thread runs none of it.
let released = super::super::disposal::collect(|| {
super::super::registry::contain_entry(|| {
if let Some(registry) = weak.upgrade() {
if let Some(next) = registry.service_deadlines(now) {
schedule(&registry, next);
}
super::super::disposal::defer_drop_arc(registry);
}
}
});
});
// Disposal.tla `Submit` (recorded there).
super::super::disposal::submit(released);
state = lock(service);
}
Some(Reverse((at, _))) => {
Expand Down
Loading
Loading