Skip to content

Pin task timeout in tests that assume a single workflow task attempt - #1838

Closed
DABH wants to merge 1 commit into
mainfrom
flake/single-attempt-task-timeout
Closed

Pin task timeout in tests that assume a single workflow task attempt#1838
DABH wants to merge 1 commit into
mainfrom
flake/single-attempt-task-timeout

Conversation

@DABH

@DABH DABH commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

What was changed

test_workflow_patch_activation_callback, test_workflow_patch_activation_callback_can_decline, test_workflow_patch_activation_callback_not_recalled_on_replay and test_workflow_current_update now start their workflows with task_timeout=timedelta(hours=1), the same idiom test_workflow_hello_eager already uses.

Why

On loaded macOS CI runners the first workflow task of these tests is occasionally not completed within the default 10s workflow task timeout. Every failing job logs Evicting workflow ... message: Error reporting WFT to server, which Core emits when the task completion is rejected with NOT_FOUND because the server already timed the task out. The server then retries the task from scratch, and two correct behaviors break the tests:

  • The patch activation callback is consulted once per attempt, since nothing from the timed-out attempt was persisted, so assert len(calls) == 1 sees 2 (run 34253408987 attempt 2, 3.14/macos-arm).
  • After two timed-out attempts the server refuses new updates with FAILED_PRECONDITION: Unable to perform workflow execution update due to Workflow Task in failed state (failUpdateWorkflowTaskAttemptCount = 3 in the history service). execute_update re-sends its request when the server's 20s update long-poll expires, and that re-send is what fails test_workflow_current_update with the RPCError (run 33848127955 attempt 5, 3.14/macos-arm).

Both tests are only meaningful for a single task attempt, so the long task timeout makes that precondition explicit instead of depending on the runner being fast. The 60s pytest timeout still bounds a genuinely stuck task, and the deadlock detector is unaffected.

Testing

  • Reproduced the mechanism deterministically: a patch callback that blocks 1.5s with task_timeout=1s yields calls=2, two WorkflowTaskStarted events, one WorkflowTaskTimedOut event and the same eviction log as CI.
  • With the change, a worker whose first activation is blocked for 11s (patch tests) or 21s (update test, past two default task timeouts and the 20s update long-poll) passes with calls=1 / all five update IDs.
  • pytest --flake-finder --flake-runs=30 of the four tests alongside a concurrent -n 8 pytest session: all passes (30 runs each; the run covered the eight tests from the same investigation, 240/240 in total, load average 45-53). The unmodified tests also pass 240/240 on the same machine, so the CI failures only reproduce through the stall emulation above.
  • poe lint clean.

Under CI load the first workflow task of these tests occasionally takes longer than the default 10s workflow task timeout. The server then retries the task, which re-runs the workflow from scratch: the patch activation callback is consulted once per attempt (observed as 'assert 2 == 1' in test_workflow_patch_activation_callback), and after two timed-out attempts the server rejects new updates with 'Workflow Task in failed state' (observed as RPCError in test_workflow_current_update). Both are correct server/SDK behaviors, so make the single-attempt precondition explicit by setting a task timeout that cannot expire within the test's own timeout.
@DABH

DABH commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #1845, which raises the dev server's default workflow task timeout so this test does not need its own. Reopen if the maintainers prefer per-test timeouts.

@DABH DABH closed this Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant