Outlive worker stalls in unfinished-handler tests - #1844
Closed
DABH wants to merge 1 commit into
Closed
Conversation
On the macOS CI runners the test process periodically stalls for 20-30s (junit durations show ~1s tests taking 13-29s). When that happens after the worker has received the first workflow task, the task hits its 10s start-to-close timeout, the transient retry times out too, and the server then rejects the pending UpdateWorkflowExecution because the workflow task attempt reached 3: "Unable to perform workflow execution update due to Workflow Task in failed state." The -no-wait- variants surface the same error as an AssertionError because the RPCError is caught but its status is FAILED_PRECONDITION rather than NOT_FOUND. Signal variants survive the same stall, which is why only the update variants flake. #1824 reordered the cancel and update requests, which is harmless but unrelated. Give these workflows a 60s workflow task timeout so a stall cannot push the task into the failed state, and cancel the background update task left behind by the exceptions sub-test so its eventual RPC error no longer shows up as an unretrieved task exception in later tests. Reproduced deterministically by injecting a 25s synchronous stall into sandbox instance creation: fails before, passes with the longer timeout.
Contributor
Author
|
Superseded by #1845, which raises the dev server's default workflow task timeout so this test does not need its own. Reopen if the maintainers prefer per-test timeouts. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was changed
test_unfinished_update_handler,test_unfinished_signal_handlerandtest_unfinished_handler_on_workflow_terminationstart their workflows with a 60s workflow task timeout instead of the 10s default.test_unfinished_*_handlernow cancels and awaits the backgroundexecute_update/resulttask it starts instead of leaking it.Why
test_unfinished_update_handlerand the-update-variants oftest_unfinished_handler_on_workflow_terminationfail on the macOS runners withRPCError: (9, 'Unable to perform workflow execution update due to Workflow Task in failed state.'), or with anAssertionErrorin the-no-wait-variants (the sameRPCErroris caught but its status isFAILED_PRECONDITION, not the expectedNOT_FOUND). #1824 reordered the cancel and update requests; that is harmless but did not address the cause and the failures continued (e.g. runs 34256407686, 34396086071).The server fails
UpdateWorkflowExecutionfast once the workflow task attempt reaches 3 (failUpdateWorkflowTaskAttemptCountinservice/history/api/updateworkflow/api.go). In every failing run the test took ~28s, the update long-poll expired after the server's 20s limit, and the worker's Python side did not process any activation until the very end, at which point the completion of the first workflow task was rejected as stale ("Evicting workflow ... Error reporting WFT to server") and a fresh task completed the workflow. That is the signature of the test process stalling for 20-30s after the worker received the first workflow task: the task hits its 10s start-to-close timeout, the transient retry times out too, the attempt counter reaches 3, and the re-issued update RPC is rejected. The junit artifacts for those runs show the same stalls across unrelated tests (~1s tests taking 13-29s; a-signal-variant took 24s and passed, since signals do not have this cliff).A longer workflow task timeout removes the cliff without changing what the tests assert: the update, signal and cancel/continue-as-new are still delivered in the first workflow task and the same warning/outcome checks run. These tests are not about workflow task timeouts, so the timeout only needs to outlive a stall; 60s matches the suite's per-test timeout. The exceptions sub-test change removes the orphaned task whose eventual
RPCErrorsurfaced as "Task exception was never retrieved" in later tests' logs.Testing
-no-wait-variant fails with theAssertionError); with the 60s task timeout the same stall passes. A 12s stall (one timeout) passes either way.pytest tests/worker/test_workflow.py -k unfinished --flake-finder --flake-runs=30 -n 4(50 tests, 1500 executions) under CPU load: 1500 passed before, 1500 passed after. The CI stall itself does not reproduce locally.poe lintclean.