Repository navigation
Validate workflow handler shapes and prevent leaked async results #177
Description
Confirmed problem and corrected scope
Coroutine signal and update handlers are accepted but their asynchronous bodies are not executed by the synchronous replay path. Generator and async-generator handlers likewise do not run as synchronous state handlers. This can leave a signalled workflow waiting forever. The original published 2.6.1 async-signal reproduction remains valid.
Source review corrects the broader initial premise: the released Worker does await async queries and async update validators. Existing query-authority and validator-concurrency controls exercise those paths. Preserve that behavior. Local synchronous query_state() and validate_update() must not return leaked coroutines. Maintainer decision and original failed-candidate limits.
Acceptance
- Reject coroutine signal/update handlers and generator/async-generator handlers of every kind with a named
InvalidWorkflowDefinitionduring definition, Worker/WorkflowEnvironment/Replayer registration and direct replay, including inherited and wrapped handlers. - Preserve asynchronous Worker query and update-validator execution, original deadline/claim authority and workflow slot accounting.
- Guard dynamic returned values. Local synchronous helpers refuse and close returned coroutines with
QueryFailed/UpdateRejected; no coroutine becomes an update result or a silently ignored signal. - Cover every unsupported shape and handler kind, inheritance, wrappers and the public entry paths. Add the smallest replay corpus reproducer.
- Explain the lifecycle in the SDK reference and all eight website locales: fresh instance before replay; initial signals after the first
run()yield; durable work inrun(); signal errors fail the run, update errors fail the update, validator errors reject it, query errors fail the query. Validators inspect disposable replayed state before acceptance. - Publish and verify the normally installed correction before closing this issue. Source implementation is #181; all-locale website follow-through is Website #333.
Activity
rmcdaniel commentedon Oct 20, 2026
Accepted as a P1 product defect. The supported handler model remains synchronous: signals and updates mutate replayed state, queries inspect it, and durable work belongs in
run(). Async and generator handler definitions must be refused before registration or replay rather than silently skipped.I am implementing the definition validation and runtime returned-value guard on
fix/177-synchronous-workflow-handlers, starting from exact main7fe39db. Coverage will include all four handler kinds, coroutine/async-generator/generator functions, inheritance, Worker, WorkflowEnvironment and offline replay. The SDK reference and all eight website locales need the lifecycle explanation. No client signal/update transport, receipt, Server, package-version or shared/native change is part of this window.The already-qualified entry-safety 2.6.2 source and its pending immutable publication remain separate. This issue is not complete until the correction is published and normally installed behavior and documentation are verified.
rmcdaniel commentedon Oct 20, 2026
Source review corrects part of the original premise: the released Worker explicitly awaits asynchronous query results inside its query-authority/deadline guard, and explicitly awaits asynchronous update validators. The normal async-query and validator-concurrency controls exercise those working paths; the installed HTTP query-authority fixture also uses an async query. Rejecting those definitions in a patch would remove existing supported behavior, so I am preserving it.
The correction therefore rejects coroutine signal/update handlers and all generator/async-generator handlers before execution, including inheritance and offline registration/replay. Synchronous wrappers cannot leak async/generator values into signal/update execution. The local synchronous
query_state()andvalidate_update()helpers refuse and close coroutines rather than returning them as results; the Worker still explicitly awaits query/validator results and preserves the original authority/capacity controls.The first unchanged-source definition controls fail as expected. An initial broader candidate correctly exposed the existing async-query fixtures during collection; it is retained as a failed candidate and has been corrected prospectively. No published async-query defect, false full-suite pass, or release acceptance is claimed. The pending unpublished Python 2.6.2 release will include this fix only after its new exact source passes normal qualification; previous source passes do not qualify it.
[-]Reject async and generator workflow handlers and document when handlers run[/-][+]Validate workflow handler shapes and prevent leaked async results[/+]rmcdaniel commentedon Oct 20, 2026
Source checkpoint: PR #181 merged normally at 1b03ed2; its whole tree 24da1f319561e72ab67504d2fbe7beed5c9d952f exactly matches candidate c9ed469. All candidate checks passed, including Python 3.10/3.11/3.12, integration, lint, packaging, corpus, docs, Sonar and the required target-branch qualification. Fresh-main CI 39486865995 is independent and still pending. This is not a PyPI 2.6.2 publication or normally installed acceptance claim; those follow the exact main gates, and the issue remains open through that acceptance.
rmcdaniel commentedon Oct 20, 2026
Published 2.6.2 registry and normal-install verification is complete. The PyPI wheel SHA-256 is
28379c16bcd598c9aa545071d074bee297e449158a550e580d1df0ef9fc27ab8. A clean CPython 3.13 installation imports that installed distribution, with no repository source override. All 116 handler-validation cases pass with RuntimeWarning treated as an error, preserving supported async queries/validators and rejecting unsupported signal/update coroutines and generator handlers.The separate three-run HTTP progress cohort on immutable Server 2.9.9 did not qualify: its 2.5-second observation ended with one callback started, no observed worker heartbeat and the side run not yet observed complete. The registration actually negotiated a 10-second heartbeat interval, overriding the fixture's requested one second, so the fixture's heartbeat expectation is invalid. That does not establish the cause of the other missed observations. Four real activity heartbeat receipts and one real activity completion were recorded; the side run subsequently completed. Worker shutdown exits zero. Stopped SQL has one completed/two waiting runs, four completed/two ready tasks, one completed/one pending activity, zero registrations and zero failed queue jobs. No failed original is called a pass.
Two pre-admission setup attempts were also unsuccessful: the first bind hid the published seeder; a later directory move left the API on an unbootstrapped path. The final setup preserves the image's seeder, completes actual bootstrap and gets /api/ready 200 before business admission. The first stopped-database fingerprint observer failed; a separate kernel-read-only snapshot succeeds and preserves main/WAL hashes. It does not rescue the earlier observer.
Both API and queue exit zero without OOM or restart. All phase containers, network, installed dependencies, databases, wheel and raw captures are verified absent at 22:46:34 UTC, before the 23:00 cleanup bound. The borrowed published Server and Python images are preserved. #177 and the dependent documentation follow-through stay open: next is a fresh actual HTTP cohort with the advertised heartbeat cadence explicitly prepared and observed, retaining the original business deadlines and the failed cohort above.
Separate installed functional acceptance
A fresh cohort uses the same immutable Server 2.9.9 and hash-verified normal Python 2.6.2 wheel, with Server explicitly configured to advertise a one-second worker heartbeat. The actual registration confirms that policy. The functional observer allows ten seconds within the unchanged 15-second callback, 30-second run and 60-second execution bounds. The earlier 2.5-second failure remains separate; no SDK change is made to accommodate the observer.
Both synchronous activity callbacks execute concurrently on
dw-remote-activitythreads while an independent workflow completes before release. Three actual worker heartbeats and four activity-heartbeat acknowledgements are observed before release, including one acknowledgement with zero free activity slots and exact task/attempt/owner matches. All three runs return exact signed-int64-maximum/null/Unicode results. A separate normally installed Python process reads all three completed results. Stopped SQL confirms three completed runs, two completed first-attempt activities, seven completed first-attempt tasks, 21 events and zero repairs, registrations or failed queue jobs; original microsecond 30/60 deadlines and database/WAL hashes are unchanged.A separate real handler workflow preserves supported asynchronous execution: two async queries return exact state before/after an accepted update; an async validator rejects -1 before the update body executes, then accepts 42. The workflow completes after its ordinary synchronous signal, and a separate cold process reads the exact typed result. Its original run is
01m5dttpeqpjcjszbdqmyv801e. Stopped SQL independently records one completed and one rejected update, the matching accepted/applied/completed history, three completed first-attempt workflow tasks and exact original deadlines. The combined database retains four completed runs and two waiting runs from unsuccessful observers. It contains no registrations.Those two handler observers incorrectly expected an unwrapped result and then a legacy
successfield. The published response contract and SDK controls establish the canonical result envelope; the corrected fresh original verifies workflow/run/name identity and both JSON and Avro results. No runtime changed. A pre-admission invocation also omitted its cohort argument. The first final SQL observer could not open the WAL-mode source on a read-only parent. A disposable stopped database copy allows SQLite's transient index; its first guard incorrectly counted the newly created zero-byte WAL as a data change. Final inspection requires every original database/WAL hash to remain exact, permits only a transient shared-memory index or empty new WAL on the copy, and keeps the source kernel read-only. These observer failures are retained as unsuccessful checks, not Server defects.Every SDK Worker exits zero. The first API/queue shutdown needed 137 without OOM; on final teardown the queue's documented restart/drain exits zero while the API still needs 137. No graceful API shutdown is claimed. Current task resources, dependencies, databases and raw captures are retired before the 00:05 UTC bound; borrowed images remain. The published package and actual HTTP behavior are qualified within these scopes. The all-language Website #333 lifecycle and entry-recovery follow-through remains separate; #177 is not closed ahead of its documentation acceptance.
rmcdaniel commentedon Oct 22, 2026
Closing after the remaining documentation acceptance is actually delivered.
The published correction remains Python 2.6.2, immutable source/tag
1b03ed2abc25a27dd6596498ed5e1e235ec90af0, wheel SHA-25628379c16bcd598c9aa545071d074bee297e449158a550e580d1df0ef9fc27ab8. Today's registry audit confirms the same non-yanked wheel. The recorded normal-install acceptance covers all 116 validation cases with RuntimeWarning treated as an error, and the genuine HTTP handler original preserves asynchronous queries/validators and exact accepted/rejected update results. Installed and actual HTTP evidence.Website #333 is now delivered at source
f37c05ab, Pages81b48f75, with publishing 39690486288 and actual Pages 39691510686 PASS. Its 64 changed public pages were verified in all eight languages. A fresh closure audit also gets HTTP 200 from the SDK's/reference/workflow/and all eight localized Python guides, confirming the named definition failure and synchronous-helper lifecycle documentation are live. Documentation delivery.Unsupported coroutine signal/update handlers and generator handlers are rejected explicitly; supported async Worker queries and update validators remain supported. This does not claim async signal execution. The earlier unsuccessful setup, timing and response-envelope observations remain unsuccessful and separate from the fresh passing originals. The one-shot closure verifier is removed; no application or dependency environment is held.