Skip to content

feat!: unified start API (0.9): spawn/start/start_link/start_monitor, StartOptions, fallible started(), continuation - #291

Draft
unbalancedparentheses wants to merge 33 commits into
mainfrom
feat/unified-start
Draft

unbalancedparentheses wants to merge 33 commits into
mainfrom
feat/unified-start

Conversation

@unbalancedparentheses

@unbalancedparentheses unbalancedparentheses commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Implements the approved unified start design (rank 5h; gaps G1 fallible startup, G2 continuation, G8 start_monitor and link/monitor on every placement with one StartOptions) for 0.9, in both modes. Draft: the first PR of a stack (scope ownership S8 and supervisor ignore follow; see "Not in this PR").

API (breaking; every break in docs/MIGRATION.md, 0.8 to 0.9)

0.8 0.9
a.start() a.spawn() (no wait), or a.start().await? (waits for started())
a.start_with_backend(b) / a.start_on(&g)? / g.start(a)? a.with_options(b).spawn()? / a.with_options(&g).spawn()?
a.start_linked(&ctx) a.start_link(&ctx).await? (inherits the parent's placement, waits, refuses an exiting parent)
ctx.monitor(&a.start().child_handle()) a.start_monitor(&ctx).await? (atomic)
ready() -> Result<(), SystemError> (managed only) ready() -> Result<StartOutcome<()>, StartError> (every placement)
stopped() after ctx.stop() in started() ctx.stop() in started() is Init::stop: no stopped() (OTP); an administrative stop during started() waits for it and runs stopped()

started() returns (), Result<(), E>, Result<Init<Self>, E> or Init<Self> (ready, continue_with, error, stop, ignore), resolved by the design's 2.3 precedence table. StartOptions<A>: group, backend, placement_of, typed name(Name<A>), label, init_timeout, stack_size, max_outstanding, max_native_jobs. Errors answer after the child's publication (state dropped, name free) with the final reason, or with pending after the cleanup deadline.

Trace validation for Start.tla (no longer TLC-only)

  • StartEvent (28 kinds) recorded under lifecycle-trace at each linearization point (zero cost off: every hook is #[inline(always)] no-op or cfg'd).
  • verification/tla/TraceStart.tla maps each event to its Start.tla action; trace_check.py schema start-v1.
  • concurrency/tests/start_trace.rs: 16 deterministic scenarios (each waits for the events it needs), added to the "Record supervisor lifecycle traces" CI step. All accepted.
  • negative_checks.py::start_trace_controls: 15 recorded-trace mutations (design 16.2's list), all rejected.
  • Coverage: 28/28 arms, coverage_floor = 28; model-map.toml lists the code, tlc_only removed.
  • Code/model disagreements found and fixed: (1) the model released a group slot before publication; the code holds it until the scope completes (a thread group's slot is the native worker still running the job join), so the model was wrong and changed (ReleaseSlot after Publish/WorkDoneC); all 7 per-PR configurations and 40 controls rerun; (2) trace validation rejected the first threads-mode recording: the code delivered a no-op Normal failed-start notice to a waiting non-trapping parent where the model (and design table) drops it; the code was wrong and now drops it.

Further findings while validating (fixed in this PR):

  • Trace stability on fede first showed 1-2 of 16 traces rejected per run: SSend was recorded after the message became receivable, so the child's SHandle could precede it (a recording linearization gap, not a model or runtime bug). Admission and dispatch of start-recorded actors are now trace sections, as for Lifecycle-recorded ones.
  • A start never decided (the runtime dropped the actor, or its lifecycle unwound) left the cell pending and a waiter hanging; it now fails at the death point (InitExited { Killed }), with a test.
  • The fuzzer's L4 oracle encoded 0.8 (a panic in started() always wins); per the 2.3 table an administrative stop that arrived first outranks it. Oracle updated; Linux-only, simulator-only, fuzz and experiments call sites migrated.
  • Client finding: ethrex PeerConnectionServer::started (rlpx/connection/server.rs:553) calls ctx.stop() after a failed initialize_connection and relies on stopped() for peer-table/broadcaster cleanup; under 0.9 that cleanup must move into started(). The client mapping (now re-checked against the canary sources) said no site changes behaviour; corrected.

Deviations from the design (each in the design's new "Implementation status" section)

Slot release point (above); the commit takes the parent's table before its exit-request lock (the existing 0.8 order; the design's order would deadlock against notice delivery); a non-ready decision freezes the reason under the exit-request lock and the table closes right after (observably one step); refining_impl_trait lint on manual started() impls (#[started] adds the allow); a Backend::Thread thread is spawned after the commit.

Not in this PR

Scope ownership (4.1/S8), supervisor ignore/ChildStart/ChildStartError/ChildSpec::options (section 9, with Supervision/Cohorts model changes), the simulator races and fuzzer start generator (16.3-16.4), four of the five Loom tests. Mock::on_started/start_delay/drop_flag and the ten start planted bugs are in the stacked #293.

Revision after maintainer decisions and the independent review

OTP stop during started() (design revision 4).

  • ChildHandle::stop* arriving while started() runs is now held, as OTP's gen_server:stop system message. The start answers Started, then the actor exits with the stop's reason through stopped(). A start that fails for another reason discards the held stop.
  • Exit signals during started() keep failing the start without stopped(): a kill, or a held non-trapping parent exit or link notice. So do Init::stop, Init::error and Init::ignore.
  • ctx.stop() inside started() is Init::stop (OTP {stop, R}). It does not run stopped().
  • The ethrex PeerConnectionServer guidance (MIGRATION, client mapping) is to do the cleanup in started() and return Init::stop/Init::error.
  • Model: AdminStopApply, invariant ReadyReturnCommitsReady, and control AdminStopFailsInit (revision 3's row 3). There is also a recorded-trace control.
  • The panic_safety and failure_windows workarounds are reverted.

Review items (each code bug got a test that fails without the fix):

  1. Fallible dispatch after the commit.
    • A dispatch or thread-spawn failure now kills the committed child before started() (KilledEarly) and drives its lifecycle inline. The Starting answers InitExited { Killed }, and no waiter stays Waiting.
    • Failpoints: system.dispatch_actor, actor.thread_spawn.
    • Recorded traces: managed link and monitor, and Backend::Thread. Two trace controls.
    • The test fails on the old code with Err(Closed).
  2. Trace mapping.
    • SResolve binds started()'s recorded return (StartedEndOf).
    • SPublish deliver and drop are constrained.
    • A trace is accepted only if no notice owed to a live P is left undelivered.
    • SCommit/SRefuse are recorded at the decision point under the innermost lock (registry, exit request or group), so they are ordered with SReleaseName, SLookup and SReleaseSlot.
    • Three new controls, all rejected.
  3. Init timer.
    • Tasks mode uses a runtime sleep task (virtual time under the simulator). Threads mode uses one shared timer thread whose entries are removed at the commitment.
    • Sim test init_timer_uses_the_runtime_clock: the old per-start thread fails it as a determinism leak.
  4. Spawn cost.
    • The cell's watch channel is lazy.
    • available_parallelism was read on every spawn (about a third of a spawn's instructions, already in 0.8.0). It is now read once per process.
    • Tasks message loop: it is its own future with one copy of the turn body.
    • New gated benchmark async_ct_spawn, and threads_spawn (wall clock).
  5. Monitor commit order. fence::register checks the table is open before the observer callback, so no name is registered for a refused start. The debug_assert is gone. This is defensive: no reachable failing test.
  6. AlreadyStarted.existing. It is now the holder for names held through ctx.register_name. register_owned takes the handle. Test fails before the fix.
  7. Parent exit while waiting (OTP).
    • A kill or untrapped signal that begins a waiting parent's exit abandons its start_link/start_monitor wait (ParentExited). The parent's turn ends and it dies without waiting for the child's started(); the child gets the parent's exit through the link.
    • Both modes. Recorded and validated as a Start trace.
    • Model: ParentKillEnabled, set on in MCStartTimeout and in trace validation.
    • The test times out without the fix.
  8. Hardening.
    • runtime_drop respects a frozen decision. Unit test fails before the fix (Killed instead of Normal).
    • SAbandon records only a real hand-on: DownWatch clears down_pending when the monitor waiter ends without a Down.
  9. API docs.
    • StartOutcome::started on Ignored.
    • StartOptions::label is now also the Backend::Thread thread name and is documented as ignored for unmanaged Async/Blocking.
    • An abandoned start_monitor still delivers its Down (documented).

Also fixed:

  • The CI sim suite (run exactly as CI) timed out on real-Tokio start tests; they are now gated off the simulator.
  • An exit_signals_sim oracle now accepts the OTP outcome.
  • The thread-only build (--no-default-features) and its clippy job.
  • coordinator_trace (from main) is ported to the 0.9 start API.
  • Planted patches 13 and 16 are regenerated.

Validation (this head)

  • TLC: all seven per-PR MCStart* configs pass, and all 41 Start model controls are rejected.
  • MCStartDeep (weekly, model with the OTP stop): 268,520,014 distinct states, depth 31, 3,332 s, no violation.
  • CI verification steps run locally: 117/117 traces accepted, with every negative control rejected. Trace coverage: Start 28/28 (19 traces), Coordinator 9/9, nothing lost.
  • Tests:
    • workspace, all features, thread-only matrix and clippy 1.93 -D warnings (all features and thread-only) pass;
    • the CI sim suite, run with --cfg spawned_sim, passes;
    • the fede 25-run trace stability rerun is posted as a comment.
  • Perf gate (callgrind, fede, instructions per op):
benchmark vs main vs v0.8.0
async_ct_burst +0.8% +3.8% (main itself is +2.9% over v0.8.0)
async_ct_request_seq +1.1% +0.3%
async_ct_request_conc16 +0.8% +0.5%
async_ct_send_p1 / p2 +2.5% / +2.4% -2.1% / -2.0%
async_ct_spawn (new) -31.2% -31.3% (44,002 to 30,217)
threads_request_seq -0.5% +0.8% (2,066 to 2,082)

Model: no change because supervision/run.rs only reads the scope's readiness barrier directly instead of through ActorRef::ready (same barrier, same states), supervision/policy_tests.rs only migrates call sites, and tasks/jobs.rs/reply/jobs.rs only switch to the managed() accessor, add Bounds::or and cache the default native-job bound; the Supervision, Cohorts, Nested, DeferredJobs and DeferredReplies behaviour is unchanged. Start.tla and the LinkFence Loom harness change in this PR for the start code.

@unbalancedparentheses unbalancedparentheses added the no-model-change Modelled code changed without a model change; the PR body says why (Model: no change because ...) label Nov 17, 2026
… StartOptions, fallible started(), continuation

Implements docs/design/unified-start.md (G1, G2, G8) in both modes:

- verbs: spawn (no wait), start/start_link/start_monitor (wait for
  started(); answer Started, Ignored or the error after publication),
  with_options(StartOptions<A>) -> Starter; Starting::actor_ref();
  ActorRef::ready() for every placement
- started() returns (), Result<(), E>, Result<Init<Self>, E> or Init<Self>:
  ready, continue_with, error, stop, ignore; the 2.3 precedence table; no
  failed start runs stopped()
- stage 1: prepare privately, then one commit (group admission, the
  parent's edge, the name) under the lock order; refusals are invisible
- the startup cell: immutable commitment, waiter, notice disposition at
  publication, abandonment, cleanup-deadline pending answers, init timer
- placement inheritance for start_link/start_monitor; typed Name<A>
- removed: start_on, start_with_backend, start_linked,
  ExecutionGroup::start, SystemError::StartupFailed; test kit
  Mock::spawn/spawn_with, TestSystem::spawn/start/spawn_mock/options

Trace validation for Start.tla: StartEvent recording (lifecycle-trace),
TraceStart.tla, start-v1 in trace_check.py, 16 recorded scenarios
(tests/start_trace.rs), 15 recorded-trace negative controls, coverage 28/28
arms. Model fix: a group slot is released at scope completion (after
publication and the owned-work join), not before publication.
…ics, TESTING; contract tests

- MIGRATION.md: 0.8 to 0.9 old-call -> new-call table for every start break
- GUIDE.md: starting actors (verbs, results, fallible started(), Init,
  continuations, links and monitors at start, placement, names)
- semantics.md: L1, L2, X5, S6 changed; L8, L9, L10, X8, S7, S9, T5 new,
  each with its checks; TESTING.md coverage rows
- unified-start.md: implementation status, deviations and their reasons
- tests/start_contract.rs: the new guarantees in both modes
- Loom: start_edge_commit_races_parent_exit_begin (LinkFence harness)
…ary sources (one behaviour change in ethrex)
… gen_server:stop

The start commits ready (Started), then the actor exits with the stop's
reason through stopped(); a failed start discards the stop. Exit signals
during started() (kill, a held non-trapping parent exit or link notice) and
Init::stop/error/ignore keep failing the start without stopped();
ctx.stop() inside started() is Init::stop (OTP {stop, R}).

Design revision 4 (2.3 row 3), Start.tla (AdminStopApply,
ReadyReturnCommitsReady, control AdminStopFailsInit), TraceStart mapping,
a recorded-trace control, semantics L2/L9, MIGRATION, GUIDE, client
mapping. Reverts the panic_safety and failure_windows workarounds; the
early-stop tests expect both hooks on every placement again.
…ild (InitExited Killed), not a stage-1 refusal

Review #291 item 1. The edge, name and scope were already applied, so the
caller got Closed/System while its waiter stayed Waiting forever. The
child is now killed before started() (KilledEarly) and its lifecycle is
driven inline; the Starting answers InitExited { Killed }. Failpoints
system.dispatch_actor and actor.thread_spawn; two recorded traces (managed
link+monitor, Backend::Thread) and two trace controls.
…tion, monitor fence order, holder handle, frozen runtime drop)

- TraceStart: SResolve binds started()'s recorded return (StartedEndOf);
  SPublish deliver/drop constrained; acceptance also requires no notice
  owed to a live P undelivered. Three new trace controls.
- SCommit/SRefuse recorded at the decision point under the innermost lock
  (registry, exit request or group), ordered with SReleaseName, SLookup and
  SReleaseSlot (commit_begin/commit_point/commit_end).
- fence::register checks the table open before the observer callback: a
  start_monitor never registers a name for a refused start; the
  debug_assert that could fire in the parent's turn is gone.
- ctx.register_name stores the holder: AlreadyStarted { existing } is Some.
- ExitRequest::runtime_drop respects a frozen (non-ready) decision.
- SAbandon: a start_monitor waiter that ends without a Down clears
  down_pending (DownWatch), so f records only a real hand-on.
- Docs: StartOutcome::started on Ignored, StartOptions::label (now the
  Backend::Thread thread name), abandoned start_monitor's Down.
- failpoints test: stop the running actor (ctx.stop in started is Init::stop).
…ead (threads); spawn allocates no watch channel

Review #291 items 3 and 4. A tasks-mode init timer is a runtime sleep task
(virtual time under spawned_sim; sim test init_timer_uses_the_runtime_clock
fails with the old per-start thread as a determinism leak); threads mode
uses one process-wide timer thread whose entries are removed at the
commitment. The startup cell's watch channel is created by the first async
observer, so a spawn nobody awaits allocates none. New hotpath benchmarks
async_ct_spawn (gated) and threads_spawn (wall clock).

Also: start_contract and start_trace tokio tests are gated off the
simulator (the CI sim suite timed out on them); exit_signals_sim's late
signal oracle accepts the OTP outcome of an exit signal during started();
the thread-only build (StartError From<SystemError> cfg, test imports).
# Conflicts:
#	.github/workflows/verification.yml
#	verification/model-map.toml
#	verification/tla/README.md
#	verification/tla/coverage-baseline.json
#	verification/tla/trace_check.py
#	verification/tla/trace_coverage.py
…on is the loop's first turn); coordinator_trace on the 0.9 start API
…r of a decided start is kept)

The 2/25 fede flake of link_stop_in_started_ends_nontrapping_parent: the
failed child's Shutdown notice is delivered to its non-trapping parent at
publication, before the answer is published; that begins the parent's exit
by a signal, which (item 7) abandoned the wait, so start_link answered
ParentExited instead of InitExited { Shutdown }. Item 7 is about not
waiting for the child's started(): a decided start is answered at its
publication (bounded by the cleanup deadline). Unit tests fail first.
@unbalancedparentheses

Copy link
Copy Markdown
Contributor Author

Trace-test flake (2/25 fede runs): root cause and fix (f882cec9)

The failing scenario was link_stop_in_started_ends_nontrapping_parent. Its start_link sometimes answered ParentExited instead of InitExited { Shutdown }.

It is a real bug, introduced by review item 7, not a test race:

  • At publication, the failed child's Shutdown notice is delivered to its non-trapping parent before the answer is published.
  • That notice begins the parent's exit by a signal, and item 7 abandoned the parent's start wait on any such exit.
  • So the answer depended on whether the waiter observed the cell before or after published.

Fix: item 7's purpose is "don't wait for the child's started()", so only an undecided start (commit still pending) is abandoned. A decided start is answered at its publication, which is bounded by the cleanup deadline.

Failing-first unit tests:

  • start::parent_exit_tests::a_decided_start_is_answered_not_abandoned fails before the fix.
  • start::parent_exit_tests::an_undecided_start_is_abandoned is the companion case, which still abandons.

The scenario now prints the result on failure.

Stability on fede at f882cec9: 50/50 runs clean. Every run's test log and trace directory is kept, and each run had 19/19 traces accepted by TLC. The scenarios use explicit event waits only. The one sleep-based wait (the threads parent-kill test) now waits explicitly for the child's started().

CI fixes in 6c5d539a:

  • The native docs build linked crate::tasks::* from shared items.
  • The fuzz harness crate needed refining_impl_trait allowed.

Blocker: main just merged #287 (G7: ExitReason::Shutdown becomes Shutdown(Option<Arc<str>>)) and #284 (threads ActorShared refactor). #291 now conflicts in 13 files, including the threads lifecycle and context, so GitHub runs no PR CI on it. Integrating those changes is the next step.

Renaming the "wip: item 7" commit needs a force push to feat/unified-start, which the repository rules decline. A squash merge drops it.

# Conflicts:
#	concurrency/src/link.rs
#	concurrency/src/tasks/actor/lifecycle.rs
#	concurrency/src/threads/actor/context.rs
#	concurrency/src/threads/actor/idle_stop_regression.rs
#	concurrency/src/threads/actor/lifecycle.rs
#	concurrency/src/threads/actor/stop_wakeup_regression.rs
#	concurrency/tests/exit_signals_sim.rs
#	concurrency/tests/otp_exit_signals.rs
#	docs/MIGRATION.md
#	docs/TESTING.md
#	fuzz/src/driver.rs
#	testing/tests/threads_kit.rs
#	verification/planted/patches/14-exit-shutdown-reports-normal.patch
# Conflicts:
#	.github/workflows/verification.yml
#	benchmarks/hotpath.py
#	concurrency/src/lifecycle_trace.rs
#	concurrency/src/registry.rs
#	verification/planted/patches/16-mon-dead-target-no-down.patch
#	verification/tla/README.md
#	verification/tla/trace_check.py
…n; metrics churn stops actors administratively; fuzz L4 oracle accepts a held signal outranking an init panic (regression added); untrack examples/thread_only/target
….9 placement inheritance), so the system's shutdown is a stop cause for X3 (regression added)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-model-change Modelled code changed without a model change; the PR body says why (Model: no change because ...)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant