Repository navigation
Run synchronous activity handlers off the worker event loop by default #171
Description
Problem
Plain def activities are the documented default style, as in the README and examples. In the default Worker configuration they run directly on the asyncio event loop:
- Remote activity tasks call
_execute_activity_callablewithoutrun_sync_in_thread. That falls through toresult = fn(*ctx.args)on the loop thread. - The worker's
dw-remote-activitythread pool runs only whenremote=True, and no call site passes it. Local activities already run sync handlers in threads. - The thread path is reached only when the worker explicitly opts into the
cooperative_cancellationcapability and the Server supports it.
So one blocking sync activity stops activity and workflow polling, workflow task processing, worker heartbeats and every other activity slot until it returns. max_concurrent_activity_tasks gives no parallelism for sync handlers. A probe at 412a10f showed a def activity sleeping 1 s running on MainThread and stalling a concurrent asyncio task for the full second. Sync handlers also have no supported way to heartbeat, because ActivityContext.heartbeat is async.
Acceptance criteria
- In the default configuration, sync
defactivities run on a bounded thread pool. Polling, workflow tasks, worker heartbeats and other activity slots keep progressing while one blocks. - Up to
max_concurrent_activity_taskssync activities run in parallel, and the slot counts in worker heartbeats stay accurate. - Sync handlers can heartbeat and observe cancellation from their thread, and activity context is available there.
- Shutdown behavior for running sync threads is explicit and documented.
- Regression tests: a blocking sync activity alongside a workflow task and a heartbeat, and N concurrent sync activities finishing in about one handler duration.
- The SDK reference and the Python guide explain
defversusasync defexecution and when to use process isolation for CPU-bound work.
Activity
rmcdaniel commentedon Oct 20, 2026
Accepted as a P1 runtime defect. The default sync-handler path must use bounded remote-activity threads so one ordinary blocking function cannot stall polling, workflows or worker heartbeats. This is separate from cooperative process isolation and must work without opting into that capability.
The correction will preserve the original activity claim/attempt and context, add a supported synchronous heartbeat/cancellation bridge to the owner event loop, and retain capacity until a running thread actually returns. A cancelled await is not proof of a stopped callback; shutdown must retain registration and report the unconfirmed thread instead of claiming a graceful stop. Async handlers remain on the event loop; CPU-heavy work needs a separately isolated process.
I am first running bounded tests against the unchanged source and the normally installed published 2.6.1 producer, then the correction. Those controls cover parallel sync activities alongside loop/heartbeat progress, exact heartbeat ownership, cancellation, physical slot accounting and shutdown. Original failures will remain distinct. Normal source CI, immutable package publication and installed consumer acceptance follow separately. No Server/runtime wire, dependency or protocol floor change is proposed.
rmcdaniel commentedon Oct 20, 2026
Default sync activity correction and actual Server controls
Draft #172 is now
7f547a221a05689f743d1bd906e60696e3b78b71, whole80e55045077cc93d1fcf93bf9a789ae2de30eca6. The default remote callable uses the existing bounded context-preserving pool, andheartbeat_sync()schedules the original attempt's heartbeat on its owner's event loop. Async callbacks retain async execution. Cancelling an await neither releases a still-running thread's slot nor proves it stopped; incomplete shutdown reports the thread and retains registration. Cooperative process supervision is unchanged.Tests-first
d127bc3has five failures on unchanged main and on the normally installed published 2.6.1 producer. Corrected focused controls pass, including actual workflow-task completion while two sync callbacks remain blocked, loop heartbeat progress, exact cancellation heartbeat ownership, physical slot retention and truthful shutdown. The final worker selection is 252 pass / two existing skips; the separate cooperative/remote/prepared supervision selection is 198 pass. Ruff and strict Mypy for the changed source pass.Fresh actual HTTP controls use immutable published Server2.9.7 image
06ed2b6305a0and normally installed Python2.6.1 clients, with original 30-second run / 60-second execution budgets:MainThreadactivity blocks the loop. The second callback and an independent workflow do not progress, and no worker heartbeat is sent during the held interval. Releasing the fixture's external gate later lets all three runs return exact results; that does not turn the failed progress observation into a pass.dw-remote-activitycallbacks stay blocked concurrently while the independent workflow completes. Before release, six genuine activity-heartbeat receipts preserve task/attempt/owner identity and one worker-heartbeat receipt reports zero free activity slots. All three runs return the same exact signed int64 maximum/null/Unicode results. Both business workers exit 0.mode=ro/query_onlySQL independently verifies six qualifying completed runs, four completed one-attempt activities, fourteen completed first-attempt tasks, 37 events, zero repairs/registrations/failed jobs, exact microsecond start-plus-30/60 deadlines and unchanged database/WAL hashes.Unsuccessful observer originals stay separate: one assumed
describe()was a dict, another assumed a nonexistentrawproperty, and a third sampled an earlier in-flight heartbeat rather than waiting for the actual zero-slot acknowledgement. They account for nine separately retained run rows and are not product failures or passing originals. Each fresh corrected cohort has its own namespace-compatible queue and instance IDs. The API needed 137/no OOM at teardown; no graceful API claim.The first source CI's corpus gate failed because an unnecessary activity-capacity edit inside the sequential
run_untildispatcher also selected its unchanged history-page context. That edit is removed: sequential admission remains unchanged, and the bounded pool still owns physical capacity. An exact old-source counterfactual fails the official corpus command; the current source passes unchanged corpus policy with 42 codec / 61 replay fixtures and all 54 policy controls. No fixture was fabricated, and no policy or threshold was weakened. Sonar's valid S9073/S5778 assertion findings have focused test-source corrections; final analyzer resolution is still pending.Exact final normal CI/Sonar, source merge/fresh main, immutable package publication and normally installed fixed-package acceptance remain pending. No protocol floor, dependency, Server code or frozen actor change. This issue remains open until delivery.
rmcdaniel commentedon Oct 20, 2026
Published Python 2.6.2 has now passed normal-install HTTP acceptance against immutable Server 2.9.9: two blocked synchronous activities run concurrently on the bounded worker threads while an independent workflow and worker heartbeats progress. The original task/attempt/owner heartbeat receipts include zero free activity slots. All three exact typed results complete once and survive separate-process cold reads; stopped SQL proves first attempts, zero repairs/registrations and original deadlines. The actual package hash, failed earlier observers and shutdown limits are recorded in the installed receipt. Supported async queries/validators also pass separately. The live SDK Worker reference includes the thread, heartbeat and truthful-shutdown contract. Remaining follow-through before closing this issue is the main Python website guide in all eight locales; I will publish that independently of the broader entry-recovery documentation hold.
rmcdaniel commentedon Oct 21, 2026
Delivered in published Python 2.6.2, source
1b03ed2, with the normal-install HTTP acceptance and regression coverage already recorded above. Two synchronous callbacks run concurrently on separate bounded threads while workflow processing and heartbeats progress; context, synchronous heartbeat receipts, original identities and cold results are verified. Cancellation retains the physical slot until the thread exits, and an uncooperative thread produces an explicit shutdown error rather than a false graceful-stop result.The remaining documentation is now deployed through website #339, merged source
4ea594b6612d63f55cbdc1063a3c0d9e80891601, with the exact qualified tree. Both the source publisher and the separate GitHub Pages deployment pass. The live release manifest selects 2.6.2. Actual HTTP checks pass for English, German, Spanish, French, Brazilian Portuguese, Ukrainian, Japanese and Simplified Chinese: each route has its translated activity-execution section, matching API identifiers, the Worker reference link and the rendered 2.6.2 installation.The live Python guide explains synchronous versus asynchronous handlers, bounded concurrency,
heartbeat_sync, cancellation,shutdown_timeoutand process isolation for work that needs a killable process. This completes #171. Broader entry-recovery work in #177 remains separate and open.