Skip to content

fix(testing): deliver due waits on checkpoint#588

Open
yaythomas wants to merge 1 commit into
mainfrom
fix/testing-arm-timer-inprocess-checkpoint
Open

fix(testing): deliver due waits on checkpoint#588
yaythomas wants to merge 1 commit into
mainfrom
fix/testing-arm-timer-inprocess-checkpoint

Conversation

@yaythomas

@yaythomas yaythomas commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Problem

In DurableFunctionTestRunner, a map/parallel branch that waits cannot resume while a sibling branch keeps working. The runner takes an extra invocation that the real service does not take: verified against Lambda in us-west-2, the service resumes the waiting branch inline in a single invocation while the sibling is still running.

Closes #587

Fix

Complete due waits and step retries at checkpoint time, inside CheckpointCore.apply, immediately before the response delta is computed. Each completion touches its operation, so the transition lands in the same checkpoint response and the suspended branch resumes within the running invocation. Between invocations, the existing earliest-pending timer remains the trigger, unchanged.

  • Execution.complete_due_operations holds the shared due predicate and transitions (wait STARTED with scheduled_end_timestamp <= now, step PENDING with next_attempt_timestamp <= now)
  • CheckpointCore.apply picks up due operations before pinning the paginator, covering both checkpoint entry points identically
  • Executor._fire_due_operations delegates to the shared method instead of duplicating the walk
  • No new persisted state: the in-memory, filesystem, and sqlite stores behave identically

Tests

  • e2e: a wait in one parallel branch completes while the sibling is still working, in a single invocation, on both the skip clock and the wall clock (wait_inline_completion_test.py)
  • e2e: a wait completed while suspended is delivered as new on the next invocation, in exactly two invocations, with on_operation_end firing once with is_replayed=False (wait_suspend_replay_test.py, guards the contract previously only covered by the examples plugin/otel tests)
  • unit: due-operation walk transitions and touch behavior (execution_wait_retry_test.py)
  • unit: checkpoint-time delivery of a due wait in the response delta (processor_test.py)

Verification

  • hatch run dev-testing:test 1362 passed
  • hatch run dev-testing:typecheck clean
  • hatch run dev-core:test 1457 passed
  • hatch run dev-examples:pytest --runner-mode=local 85 passed, 2 skipped (pre-existing cloud/X-Ray skips), including test_plugin and test_plugin_on_operation_end_called_for_wait_completed_during_suspend
  • hatch fmt --check clean
  • lintcommit PASS

@yaythomas
yaythomas marked this pull request as draft July 27, 2026 08:27
Complete due waits and step retries at checkpoint time so the
completion is returned in the same checkpoint response delta and a
suspended parallel branch resumes within the running invocation. The
real service resumes such branches inline in a single invocation; the
local runner previously waited for the invocation to return and took
an extra invocation.

- Add Execution.complete_due_operations holding the shared due
  predicate and transitions for waits and step retries
- Pick up due operations in CheckpointCore.apply before the response
  delta is computed, covering both checkpoint entry points
- Delegate Executor._fire_due_operations to the shared method
- Add e2e tests: a wait in a parallel branch completes inline in one
  invocation (skip and wall clock), and a wait completed while
  suspended is delivered as new on the next invocation
- Add unit tests for the due-operation walk and for checkpoint-time
  delivery of a due wait

Closes #587
@yaythomas
yaythomas force-pushed the fix/testing-arm-timer-inprocess-checkpoint branch from 4395fe0 to 2d930bb Compare July 27, 2026 09:02
@yaythomas yaythomas changed the title fix(testing): fire timers during invocation fix(testing): deliver due waits on checkpoint Jul 27, 2026
@yaythomas
yaythomas marked this pull request as ready for review July 27, 2026 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Local runner does not complete a due wait until the invocation returns

1 participant