Give manual activity tests time for the start handshake - #1832
Closed
DABH wants to merge 1 commit into
Closed
Conversation
test_manual_heartbeat and its siblings start a standalone activity with a
5s start-to-close timeout, then wait for the activity to report that it
started through an EventWorkflow before completing, cancelling, failing or
heartbeating it by ID. That handshake includes the first workflow task of a
cold worker, which on a loaded macOS runner took over 5s; the attempt timed
out and the heartbeat failed with RPCError NOT_FOUND ("activity not found
for ID"). Nothing in these tests depends on the attempt timing out, so give
the attempt a minute.
Contributor
Author
|
Folded into #1842, which now covers both the cancel-activity test and these manual activity tests under one rationale. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was changed
test_manual_completion,test_manual_cancellation,test_manual_failureandtest_manual_heartbeatstart their standalone activity withstart_to_close_timeout=timedelta(minutes=1)instead of 5 seconds.Why
tests/test_activity.py::test_manual_heartbeatfailed on macOS CI (run 33848127955 attempt 2) withRPCError: activity not found for ID: <uuid>(NOT_FOUND) from the heartbeat-by-ID call.These tests learn that the activity attempt is running through a handshake: the activity signals an
EventWorkflow, the test waits for that workflow, then completes/cancels/fails/heartbeats the attempt by ID. The handshake includes the first workflow task of a freshly started worker. In the failing run the captured log shows the attempt starting at 08:00:57 and the handshake workflow completing at 08:01:02, so by the time the test heartbeated, the 5 second start-to-close timeout had already closed the attempt. Nothing in these tests relies on the attempt timing out; the bound only decides how slow a runner may be before the test breaks, so it is raised well above anything the per-test pytest timeout allows.Testing
--flake-finder -n 6under CPU load (the handshake is too fast on a development machine to reproduce the CI timing).poe lintclean.