Repository navigation
Stop activities when a heartbeat says the attempt can't continue #186
Description
Delivered — October 23, 2026
The heartbeat-refusal correction is merged and present in published Python SDK 2.9.0, peeled tag/source 04116871c83e6a502c713789cb452e90917a3438.
Fresh isolated normal PyPI installation on Python 3.12.3 passed all 46 heartbeat/context/lease-worker controls in 5.78 seconds. All 36 installed Python files match the exact tag and the wheel's SHA256 RECORD, without editable installation or source overlay. Wheel SHA256: 1e7547e895485dc8f670bd1b9cc34184b135f249db8da98b16b250bfc6002f73. Installed component receipt.
The all-eight-locale Python/timeouts follow-through is delivered in Website #375 / PR #376, source merge 6ebff5c4c91639173430b6ba844730f04ac380a8. Exact-head qualification, artifact deployment and actual Pages publication 39935867402 pass. The SDK owner independently confirms all 16 current HTTP routes, matching canonicals, Python section anchors and localized timeout-to-Python links. The wording preserves conditional execution and physical-slot boundaries.
The SDK controls use synthetic heartbeat/lease responses. They establish installed SDK handling, not deployed Server timeout admission. Workflow #878 remains separate for automatic-maintenance and six-minute/worker-loss acceptance. The original unsuccessful source/integration evidence remains unchanged; later delivery is independent evidence.
Problem
When a Python activity's attempt can no longer finish, the activity keeps running. This happens when its start-to-close deadline passes, its attempt is closed, or its run closes. Its slot stays taken until it returns, and then its result is rejected.
- In each of these cases, the heartbeat response says
can_continue: falsewithcancel_requested: false, with reasons such asactivity_deadline_elapsed,attempt_closedandrun_closed. - Python's default heartbeat path raises only on
cancel_requested. Remote activities use that path unless cooperative cancellation is enabled. - The PHP SDK stops on
can_continue: false, and so does Python's own cooperative path.
So a Python activity that heartbeats faithfully gets no signal that its work is now wasted. The timeouts guide says a start-to-close timeout closes the attempt, but it doesn't say the worker's code keeps running.
Source-verified, not reproduced.
Acceptance criteria
- On the default path,
heartbeat()andheartbeat_sync()raise when the response hascan_continue: false, matching PHP. The exception carries the reason, so an activity can tell a timeout or closed run apart from cancellation. - The worker frees the slot once the activity exits and submits no result for the closed attempt.
- Tests cover an elapsed start-to-close deadline, a closed attempt and a closed run, and confirm that
cancel_requestedstill raises. - The Python guide and the timeouts page state that the activity stops at its next heartbeat after its attempt closes, and that code which never heartbeats runs until it returns.
Activity
rmcdaniel commentedon Oct 21, 2026
Accepted product correction: stop a refused heartbeat
This is an activity-authority bug, not a documentation-only gap. The default context currently checks cancellation but ignores an explicit
can_continue=false. The correction must preserve the original refusal reason, stop async and synchronous activity code at its next heartbeat, release its physical slot after exit, and send no completion or failure for the closed attempt. Cancellation remains distinct.Tests-first
68f6ed1adds thirteen controls for elapsed start-to-close, closed attempt and closed run on both heartbeat APIs and default worker paths, plus publication fencing when user code catches the heartbeat exception. Production is byte-identical tod2939d4efor the counterfactual: all thirteen actually FAIL in 0.77 seconds. That parent contains the independently qualified #185 maintenance-stop fix.Functional correction
a8659b70888e7780e4d2aaa04ca7b1506d1f52b8/ tree0724221f81a5a2639fae0a09e1b611f5c9141143preserves the original refusal inActivityLeaseAuthorityLost.reason, separately from cancellation. The default worker returnsclaim_abortedand emits neither result nor failure, including after user code suppresses the refusal and returns or raises an ordinary/non-retryable error. The same original Client authority is invalidated by task/attempt/generation/owner binding; a different original is not invalidated. Two real Client/mock-HTTP controls prove transport publication is fenced before an outgoing completion request.Final normal-install Python 3.10.22 acceptance: 3,120 tests PASS / three skips / 105 integration deselections, 164.06 seconds; 46 focused heartbeat/context/lease-worker tests PASS, overlapping the full total; Ruff passes and mypy passes all 36 source files. The earlier source component also passed 3,118 tests; a test-only lint failure was corrected explicitly. The final result includes the additional original-client authority/isolation controls. No failed original is called a pass.
Current reviewed head is
0c215c9a8ec56946930a511af1b518bffc9710c4/ tree87eaf99724600168ad3190bf7ec8f45fa606fc95. It addresses the two actionable Sonar locations by separating ordinary exception disposition from callback dispatch and creating the selected coroutine outside the exception guard. Original storage-admission propagation, failure metadata/logging and heartbeat-authority order are preserved. Actual focused worker/storage/context/heartbeat qualification on this head passes 387 tests / two existing skips, plus Ruff and mypy on 36 source files. The 3,120-test full result belongs to functional parenta8659b70, not a whole-head claim for this cleanup. Fresh hosted supported-version/integration/Sonar gates qualify the current head independently; no threshold/profile change or analyzer-fixed claim before readback.Before advancement, the original 3.12, corpus, package, portal/boundary and advisory cells had passed; 3.10/3.11/lint remained queued with no admitted assertions. Their original partial results remain visible. No running supported-version test was cancelled to obtain a pass.
README, the SDK reference, changelog and method docs explain the refusal and its physical-stop limits. No dependency, package version, Server/native/shared/schema or wire change. The branch incorporates merged #185 via
fe40c647, not a borrowed old runtime result. Fresh normal CI/Sonar, compatible publication, actual installed refusal/slot acceptance and the eight-locale website timeout explanation remain required before closing #186. Workflow #878's larger automatic-maintenance acceptance stays open independently. No provider operation or paid resource.rmcdaniel commentedon Oct 21, 2026
Source correction delivered through Python #187, merged at
d6da755e80357bf2bebf460c925ff71f3433e683on October 21 at 10:28:34 UTC. Its whole tree exactly matches qualified candidate87eaf99724600168ad3190bf7ec8f45fa606fc95.All supported Python 3.10/3.11/3.12, actual source-Server integration, lint/package/corpus/portal/boundary and target qualification gates pass. Exact-head Sonar passes with both actionable reports individually CLOSED/FIXED and zero OPEN/CONFIRMED. No threshold change or suppression. The completed remote head is deleted and independently returns 404.
The default heartbeat now preserves the original authority-loss reason and fences both results and failures, including when user code catches the refusal. Cancellation remains distinct. The thirteen actual parent-source failures remain recorded; the focused and full local results retain their exact source attribution in the PR.
Fresh main CI
39584856529, portal39584856514, Sonar39584856467and boundary39584856460remain independent and pending. Compatible publication, actual installed refusal/physical-slot acceptance and the eight-locale website follow-through remain required before closing #186. Workflow #878 remains open for the larger automatic-maintenance and published six-minute/worker-loss acceptance.rmcdaniel commentedon Oct 21, 2026
Fresh-main SDK source
d6da755e80357bf2bebf460c925ff71f3433e683remains unqualified for publication. Its normal integration job119190338104in run39584856529exceeded the unchanged 25-minute job limit (11:39:27–12:04:43 UTC), during Start Server stack, before endpoint provenance and test execution. The job annotation confirms the timeout and missing scenario artifacts. Supported Python unit cells, lint, package and corpus pass independently.No failing integration assertion or runtime cause is established. The job-log download also timed out; that network retrieval failure is not a product diagnosis. The failed/cancelled original is retained. I will inspect the available startup evidence and qualify the actual SDK/Server tuple before publication; no timeout increase, hidden pass or automatic repeated retry is applied.
rmcdaniel commentedon Oct 23, 2026
Published-package acceptance is now independently verified on PyPI 2.9.0 / source
04116871c83e6a502c713789cb452e90917a3438. A fresh isolated normal install on Python 3.12.3 passed all 46 tests intest_activity_context.py,test_activity_heartbeat_authority.pyandtest_activity_lease_worker.pyin 5.78 seconds. Those test files are unchanged from the published tag. All 36 installed Python files match the exact tag and wheel RECORD; the wheel SHA256 is1e7547e895485dc8f670bd1b9cc34184b135f249db8da98b16b250bfc6002f73.These are installed SDK controls with synthetic heartbeat/lease responses, including actual callback/thread exit and slot retention. They verify the SDK's response handling and result/failure fencing; they do not assert a live Server timeout experiment or close Workflow #878. The earlier failed integration run and its original observations remain recorded as failures.
The issue remains open for its eight-locale Python/timeouts explanation. The Website owner has been asked to confirm or own this remaining criterion; source/publication and installed SDK acceptance are now recorded separately in the current-state section.
rmcdaniel commentedon Oct 23, 2026
Closing with the remaining documentation acceptance delivered. Website #375's publication receipt records all eight localized Python/timeouts guides, source
6ebff5c4, and successful actual Pages publication. Independent SDK-owner reads now confirm all 16 live routes return 200, match their canonicals, expose the refusal section and link from each localized timeout guide.Combined with the normal installed Python 2.9.0 acceptance, the issue's source, publication, installed handling and guide criteria are complete. Exact package source is
04116871c83e6a502c713789cb452e90917a3438; wheel SHA256 is1e7547e895485dc8f670bd1b9cc34184b135f249db8da98b16b250bfc6002f73.Earlier failed originals remain failed and retained. Synthetic SDK response controls and published guidance do not close Workflow #878 or assert a live Server timeout-admission experiment.