Skip to content

Outlive worker stalls in unfinished-handler tests - #1844

Closed
DABH wants to merge 1 commit into
mainfrom
flake/unfinished-handler-wft-failed
Closed

Outlive worker stalls in unfinished-handler tests#1844
DABH wants to merge 1 commit into
mainfrom
flake/unfinished-handler-wft-failed

Conversation

@DABH

@DABH DABH commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

What was changed

  • test_unfinished_update_handler, test_unfinished_signal_handler and test_unfinished_handler_on_workflow_termination start their workflows with a 60s workflow task timeout instead of the 10s default.
  • The exceptions sub-test of test_unfinished_*_handler now cancels and awaits the background execute_update/result task it starts instead of leaking it.

Why

test_unfinished_update_handler and the -update- variants of test_unfinished_handler_on_workflow_termination fail on the macOS runners with RPCError: (9, 'Unable to perform workflow execution update due to Workflow Task in failed state.'), or with an AssertionError in the -no-wait- variants (the same RPCError is caught but its status is FAILED_PRECONDITION, not the expected NOT_FOUND). #1824 reordered the cancel and update requests; that is harmless but did not address the cause and the failures continued (e.g. runs 34256407686, 34396086071).

The server fails UpdateWorkflowExecution fast once the workflow task attempt reaches 3 (failUpdateWorkflowTaskAttemptCount in service/history/api/updateworkflow/api.go). In every failing run the test took ~28s, the update long-poll expired after the server's 20s limit, and the worker's Python side did not process any activation until the very end, at which point the completion of the first workflow task was rejected as stale ("Evicting workflow ... Error reporting WFT to server") and a fresh task completed the workflow. That is the signature of the test process stalling for 20-30s after the worker received the first workflow task: the task hits its 10s start-to-close timeout, the transient retry times out too, the attempt counter reaches 3, and the re-issued update RPC is rejected. The junit artifacts for those runs show the same stalls across unrelated tests (~1s tests taking 13-29s; a -signal- variant took 24s and passed, since signals do not have this cliff).

A longer workflow task timeout removes the cliff without changing what the tests assert: the update, signal and cancel/continue-as-new are still delivered in the first workflow task and the same warning/outcome checks run. These tests are not about workflow task timeouts, so the timeout only needs to outlive a stall; 60s matches the suite's per-test timeout. The exceptions sub-test change removes the orphaned task whose eventual RPCError surfaced as "Task exception was never retrieved" in later tests' logs.

Testing

  • Reproduced deterministically with a harness that mirrors the test and injects a 25s synchronous stall into sandbox instance creation: before the change it fails exactly like CI (server logs "Fail update fast due to WorkflowTask in failed state", history shows attempt 1 timed out and attempt 3 completing the workflow; -no-wait- variant fails with the AssertionError); with the 60s task timeout the same stall passes. A 12s stall (one timeout) passes either way.
  • pytest tests/worker/test_workflow.py -k unfinished --flake-finder --flake-runs=30 -n 4 (50 tests, 1500 executions) under CPU load: 1500 passed before, 1500 passed after. The CI stall itself does not reproduce locally.
  • poe lint clean.

On the macOS CI runners the test process periodically stalls for 20-30s
(junit durations show ~1s tests taking 13-29s). When that happens after
the worker has received the first workflow task, the task hits its 10s
start-to-close timeout, the transient retry times out too, and the server
then rejects the pending UpdateWorkflowExecution because the workflow task
attempt reached 3: "Unable to perform workflow execution update due to
Workflow Task in failed state." The -no-wait- variants surface the same
error as an AssertionError because the RPCError is caught but its status
is FAILED_PRECONDITION rather than NOT_FOUND. Signal variants survive the
same stall, which is why only the update variants flake. #1824 reordered
the cancel and update requests, which is harmless but unrelated.

Give these workflows a 60s workflow task timeout so a stall cannot push
the task into the failed state, and cancel the background update task
left behind by the exceptions sub-test so its eventual RPC error no longer
shows up as an unretrieved task exception in later tests.

Reproduced deterministically by injecting a 25s synchronous stall into
sandbox instance creation: fails before, passes with the longer timeout.
@DABH

DABH commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #1845, which raises the dev server's default workflow task timeout so this test does not need its own. Reopen if the maintainers prefer per-test timeouts.

@DABH DABH closed this Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant