Fix activity pause test races - #1831
Draft
DABH wants to merge 2 commits into
Draft
Conversation
The pause tests raced their own activities: - unpause_and_assert required the activity to still be pending after the unpause RPC, but the retried attempt sees heartbeat details and completes immediately, so a slow describe observed no pending activity. - The pause workflows set heartbeat timeouts of 1-2s while relying on the default throttle (80% of the timeout), leaving a few hundred milliseconds before the server timed the attempt out and the pause RPC or the next describe found no pending activity. Pause is delivered through heartbeat responses whose cadence is the worker throttle interval, so drop the timeouts and set a 300ms throttle explicitly. - assert_pending_activity_exists_eventually waited only 5s for the first workflow task to schedule the activity, which a cold worker on a loaded runner exceeds; use the 10s the other helpers use. Also assert that the external heartbeat raises after the pause instead of silently passing when it does not.
There was a problem hiding this comment.
🟢 Approval recommended
The focused test changes consistently address the documented races without altering production behavior.
Pull request overview
Stabilizes activity pause tests by reducing timing races around heartbeats, retries, and cold starts.
Changes:
- Uses fixed 300 ms heartbeat throttling without short heartbeat timeouts.
- Accepts completed activities as successfully unpaused.
- Strengthens cancellation assertions and extends pending-activity polling to 10 seconds.
File summaries
| File | Description |
|---|---|
tests/worker/test_workflow.py |
Adjusts heartbeat timing and cancellation assertions. |
tests/helpers/__init__.py |
Makes pending and unpause assertions race-resistant. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was changed
unpause_and_assertaccepts an activity that is no longer pending as unpaused; the pause tests drop their 1-2s heartbeat timeouts and set a 300ms heartbeat throttle; the cold-start bound for the first activity uses the suite's 10s default; the external cancellation test asserts the heartbeat raises instead of passing silently.Why
After unpause the retried attempt can complete within a second, so a slow
describe()found nothing pending. With a 1-2s heartbeat timeout and the default 80% throttle the server had 200-400ms of slack before timing the attempt out, after which pause by ID fails withCan't find pending activity. A cold worker's first task exceeded the 5s bound on CI.Testing
Each mechanism reproduced by injecting describe or heartbeat latency: fails before, passes after. 120/120 flake-finder runs under load. Lint clean.