Skip to content

Add guarded activity lease renewal client transport - #183

Merged
rmcdaniel merged 9 commits into
mainfrom
fix/878-activity-lease-renewal
Oct 21, 2026
Merged

rmcdaniel merged 9 commits into
mainfrom
fix/878-activity-lease-renewal

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Scope

Part of durable-workflow/workflow#878 and Server draft durable-workflow/server#564. Candidate source 437a6aa7e6a46d0026bc8272602b3a33d0979443, whole tree caf6e199ac96f7295c921b63edf33ad8e65ccc8e, includes qualified Python main ab7ee6f8. It adds automatic activity maintenance around loading, callbacks and publication. The qualified source is now merged as recorded below. It remains unpublished, with actual released long-activity acceptance outstanding.

Affirmative capability activates exact original task/run/execution/attempt/owner validation. Bounded initial confirmation charges transport latency without charging an empty poll against a lease that did not exist yet. Runtime renewals grant no application progress, preserve original business deadlines and cannot revive an expired claim. Actual acknowledged application heartbeats move only their recorded interval; stale concurrent renewal replies cannot overwrite newer progress.

Each terminal HTTP attempt is fenced, including retries and callbacks that swallow cancellation. A cancelled synchronous thread retains its physical executor slot until it really exits. Separate read-only status and callback-stop receipt paths remain usable after execution authority is lost. Old runtimes retain their existing behavior. This does not claim cancellation stopped a running thread.

Client.renew_activity_task_lease requires affirmative activity_lease_renewal=true discovery, original task/attempt/owner identity and worker-plane credentials. Discovery, transport retries and the request share one caller-selected window of at most five seconds. It records no application progress. The caller must validate the response identity and actual grant before using it; a successful HTTP response alone does not extend local publication authority.

No new protocol floor, dependency, package version or published Server capability is introduced. Explicit false and absent/malformed discovery have distinct existing capability errors. Empty identities and unbounded/invalid request windows are refused before discovery.

Automatic worker verification

  • Five initial worker controls fail on preceding 6087870. A later application-heartbeat control exposes the initial stale progress window. The empty-long-poll control fails with the preceding 5e11e029 client overlaid onto the final source, then passes after fresh confirmation.
  • The earlier dfe10e7 component passes 65 focused tests and the overlapping combined selection of 557 tests / two skips on CPython 3.13.
  • Four further discovery controls expose a real omission: affirmative capability in the Client's canonical cache did not fence a task missing original authority when a separate Worker flag had not been set. The positive control fails on dfe10e7; false, absent and malformed capability retain their existing behavior. The correction uses the same Client discovery as actual renewal requests and removes the duplicated Worker registration state.
  • Current focused authority/transport/worker selection: 69 passing tests. Current combined client, worker, query-authority, renewal and telemetry-wire selection: 561 passing, two skipped, on CPython 3.10.22, pytest 8.4.2, httpx 0.28.1, fastavro 1.12.2 and optional OpenTelemetry API/SDK 1.45.1. These selections overlap. A normal installation is used, and all 36 installed production Python files match the current source before and after qualification.
  • Focused Ruff, strict mypy on all four affected source files and diff checks pass.

The first broad run incorrectly demanded exactly one completion attempt; the correct assertion verifies every retry precedes the original deadline. A later synthetic fixture anchored its claim clock before client bootstrap and correctly received an expired-lease refusal. The corrected fixture starts execution timing at its actual claim. Those failed observations remain separate; no product deadline or refusal was relaxed. The separate real empty-poll/lease-clock problem is fixed in source as recorded above.

Earlier client primitive verification

  • Test-first b3be4d8: all ten initial cases fail on the missing method.
  • Final source: 17 focused transport/discovery/identity/window cases pass, including real client header/namespace construction and cancellation of a stalled discovery or request.
  • The complete existing client test file and new selection pass together: 217 tests on CPython 3.13, pytest 8.4.2, pytest-asyncio 1.2.0, httpx 0.28.1 and fastavro 1.12.2.
  • Focused Ruff and git diff --check pass.

The first tools image lacked pytest and the initial temporary tool filesystem was not executable. A later fixture used the wrong namespace-header name, then a broad source-only invocation lacked installed distribution metadata. Those observations remain unsuccessful. The final run installs an exact copy of this source normally in its disposable environment, preserving the existing package-identity assertion. No expectation is weakened and no published-artifact acceptance is inferred.

Remaining

Run 39540797099 attempt 1 is terminal: Python 3.11/3.12, lint, package, regression corpus and Sonar pass, while Python 3.10 job 119030104611 reaches the unchanged 15-minute maximum and is cancelled. Integration is skipped and the required aggregate fails. Its exact annotation establishes the timeout, not a failing assertion or cause. One retry of that job and its dependents, on the same source and limits, passes in attempt 2: Python 3.10 job 119058315915, integration 119064237930 and required qualification 119068312162. The other jobs retain their original timestamps and were not rerun. Original outcomes remain preserved.

Actual full local Python 3.10.22 normal-install run passes 3,101 tests, three skips, 105 integration deselections, in 183.79 seconds, with pytest 9.1.1 / pytest-asyncio 1.4.0. All 36 installed production files match this exact source. The first local full run failed because its tool-only /tmp was mounted noexec: executable fake-Git fixtures fell through to real checkouts, filling its 128 MiB temporary filesystem. A fresh exact-source checkout with executable task temporary storage passes unchanged assertions and deadlines. That local tool failure does not establish the hosted timeout's cause. All local Python tools, dependencies, copied source and temporary data are now removed; idle tool PID 1 exit 137 / no OOM is not application-shutdown evidence.

Fresh exact-head Sonar workflow 39540797101 and its app check pass. Independent issue API review confirms all nine source/test findings CLOSED/FIXED, zero OPEN/CONFIRMED findings and zero new issues; new-code duplication is 0.0%. This source extracts original heartbeat validation, lease maintenance, activity argument decoding and failure reporting, and clarifies six test locations. The tenth asyncio.wait_for replacement suggestion is individually RESOLVED/FALSE-POSITIVE because this package supports Python 3.10, where the proposed asyncio.timeout context manager is unavailable. The current source is actually tested on Python 3.10.22. No rule, profile or threshold changed. All candidate gates now pass as recorded below.

The preceding source's hosted regression corpus job remains failed; a bounded log fetch timed out, so its exact remote assertion is not independently read. Running the unchanged official validator locally on that source reproduces the refusal: the extra Worker registration state was classified as a replay change without a newly added history. Canonical Client discovery removes that duplicated registration state while also correcting the actual missing-authority refusal above. The current official validator passes against ab7ee6f8, with codec 42 to 42 and replay 63 to 63, both classified unrelated. All 54 validator self-tests and the current candidate's hosted corpus pass. No history fixtures, policy, evidence or expectations changed.

This is source and constructed-transport qualification, not an actual published Server cohort. Complete Server's original run/execution deadline projection and current API documentation, real database/worker-session races, matching PHP/Rust/embedded maintenance, and genuine published six-minute/killed-worker acceptance. Fresh hosted checks qualify the exact combined tree independently. This source remains separate from the tracing release and its consumers, and claims no native/frozen adoption.

Source merge

All required candidate checks pass. Merged at dbace3da09f2d1f0801464100e1b468690c572bc on October 21 at 06:15:26 UTC. The merge tree is exactly the tested caf6e199ac96f7295c921b63edf33ad8e65ccc8e. The completed remote head is deleted and its absence verified. Fresh-main CI 39554306509, Sonar 39554306508, portal 39554306638 and boundary 39554306618 are separate and initially queued. No package version or tag is changed, and #878 remains open for the remaining product and published-artifact acceptance.

The source worktree is retained. The preceding task tool container stopped at October 21 02:57:36 UTC; subsequent inventory independently confirms it absent at 03:01:32. The separate Python 3.10 quality/corpus tool container stopped at 03:51:02 and is independently verified absent at 03:51:53, ahead of its 04:20 retirement boundary. Its normal installation, tools and caches lived on tmpfs and are gone. The idle tool PID 1 required exit 137 after the completed test commands; this is not application-shutdown evidence. No database, application stack, provider resource or paid host was started. Borrowed images and other owners' resources are preserved.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel marked this pull request as ready for review October 21, 2026 06:15
@rmcdaniel
rmcdaniel merged commit dbace3d into main Oct 21, 2026
23 of 25 checks passed
@rmcdaniel
rmcdaniel deleted the fix/878-activity-lease-renewal branch October 21, 2026 06:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants