Repository navigation
Keep run_until alive through typed retryable poll refusals #218
Description
Delivered
Delivered in normally published Python SDK 2.11.1, exact source 3511924, whole 4466d411b8f9b20eebd1d1c4c082b46f546d2b49.
All applicable candidate and independent fresh-main checks pass, including Python 3.10/3.11/3.12, integration, quality, corpus, CodeQL and boundaries. The protected PyPI OIDC publisher 40061568529 completed successfully. The normal PyPI 2.11.1 package is available; wheel SHA-256 is 1e7f0210ef30ae33e2224f6857386a43e7793b725a298a0974b2bd6e53c0318f. All 39 installed production files match the tagged Git source and all 51 RECORD hashes are verified, including five declared console launchers.
Actual installed acceptance: five new originals on unchanged published PHP Server 2.11.1, immutable index bf8bed13f5f809ee3425233180a581353eac4683adbee5c4ecbf63bb18bbd102, pass: healthy default plus workflow-poll and activity-poll pressure separately in default and named namespaces. Each pressure case receives three actual typed HTTP 503 backend_lock_pressure responses from a real SQLite writer lock through the normal Client, with null task and exact namespace/queue/kind/retry delay, then completes the same original. Responses are neither mocked nor rewritten.
Every original retains its 25-second helper budget, one-second poll and 30-second run/execution deadline, with separate ten-second shutdown policy. Completion plus shutdown takes 1.68 seconds healthy and 3.24–3.40 seconds under pressure. Stopped SQL and cold reads prove 5 completed runs, 5 real callbacks, 5 first activity attempts, 15 first-attempt completed tasks, 30 events, zero repairs and zero failed jobs. Exact int64, Unicode, false/null values and history identities survive cold delivery. All five APIs exit 0 without OOM.
The earlier normal-PyPI 2.11.0 counterfactual failures, broad local original and unchanged-baseline diagnostic failures remain recorded. The initial installed-file verifier refused pip --target console paths; the corrected verifier resolves only the five declared console entries and validates every hash, without skipping files or changing the package. No historical native/shared cohort failure is rescued or retupled by this new acceptance.
All owned applications, databases, networks, installed SDK/dependencies, tooling and raw database state are removed; empty task inventories and borrowed-image preservation were verified at 04:39:14 UTC, before the declared 04:50 bound. General's SDK acceptance is complete. Server #721 retains independent cohort scheduling/adoption and every original observation.
Accepted SDK correction
Server #721 reports that its separately started Python patch Worker exits after the published PHP reference returns a typed 503 backend_lock_pressure poll refusal. The cohort uses Worker.run_until(..., timeout=25.0). Its exact installed SDK identity still needs confirmation from the owning retained receipt; no frozen tuple is changed here.
Current published Python 2.11.0 source independently confirms the helper gap: _run_until_loop rethrows workflow-poll exceptions after releasing capacity, and its activity-poll branch has no recovery handler. By contrast, normal long-running Worker.run() poll loops already back off after errors; that separate #189 behavior is delivered. Client transport retry exhaustion and a bounded worker helper are different layers.
Product decision: run_until should recover from a complete, correctly bound, typed transient poll refusal within its ORIGINAL remaining caller deadline. It should not abandon a valid workflow wait early solely because that refusal reached the helper. Direct Client methods may still surface errors after their transport policy. Persistent failures must exhaust the original helper budget honestly, without renewing it.
Acceptance
- First reproduce the helper failure on normally installed published 2.11.0, using the published worker-poll refusal shape and retaining the failing control. Test workflow and activity polls separately. Do not execute or modify the native owner's frozen/retired observation recipes.
- Recognize supported transient/non-admission refusals precisely. Preserve namespace, queue, task kind, worker registration, poll identity/claim authority and existing capacity/storage handling. Never treat a malformed or mismatched task as an empty success or retry an unsafe admitted operation blindly.
- Release reserved slots before backoff, bound delay by the original remaining run_until deadline, reset after successful polls and retain useful bounded logs/metrics. Registration time remains inside the original caller budget; shutdown retains its separate existing drain policy.
- Stop/cancellation interrupts the wait and prevents new polling/dispatch. Preserve fatal authorization/protocol/identity errors and original business errors. Do not create new activity attempts or alter workflow deadlines merely to recover transport polling.
- Cover one refusal then completion of the same original, persistent refusal until the original deadline, stop during backoff, workflow/activity lanes, malformed/foreign refusals, and unchanged long-running worker behavior.
- Publish a compatible SDK patch and verify a normal registry installation plus a fresh real published-Server cohort. The native owner retains cohort scheduling and all original observations. No historical failure is called fixed, and no frozen SDK/runtime adoption occurs automatically.
General owns the Python implementation and release. Server #721 retains shared-cohort ownership. The normal PyPI 2.11.0 counterfactual reproduces both helper recovery failures and the separate cold-history refusal, while four fatal/foreign controls pass. The delivered source passes 163 focused controls, Ruff, strict mypy and corpus validation. Published-artifact acceptance is complete as recorded above; the failed broad local original and its separate unchanged-baseline diagnostic counterparts remain preserved in PR #219.
Activity
rmcdaniel commentedon Oct 24, 2026
Source delivery: PR #219 is merged at
351192443943d645f84b541b4e45f2b002e81193, whole4466d411b8f9b20eebd1d1c4c082b46f546d2b49, identical to candidate961144f6. All eighteen candidate checks pass, including Python 3.10/3.11/3.12, actual integration, final branch qualification, corpus and analyzers. The remote branch is deleted and absence verified.Fresh-main CI https://github.com/durable-workflow/sdk-python/actions/runs/40055126621 now qualifies that merge independently. Version 2.11.1 remains unpublished; normal protected publication, installed identity and a fresh real published-Server helper-recovery cohort with original budgets are next. This issue remains open. The earlier local/published failures and frozen native observations remain separate; no automatic source or tuple adoption is requested.
rmcdaniel commentedon Oct 24, 2026
Real HTTP counterfactual and exact-source prepublication component, October 24.
Normally installed Python 2.11.0 reproduces the helper failure against immutable published PHP Server 2.11.1 (
bf8bed13…, source5336a98d, Core 2.10.2). A separate 2.11.1 source preview from merged35119244/ whole4466d411completes the same four fresh controls. This preview is not a normally published PyPI artifact.Method: each control uses a fresh normal-bootstrap SQLite database and the unmodified published API. A real
BEGIN IMMEDIATEwriter is held during one unchanged public Client poll call; the wrapper changes neither its arguments nor responses. Each pressured original records three actual HTTP 503 responses after normal client transport retries:task=null,poll_status=reason=backend_lock_pressure, the original namespace/queue/task kind andretry_after_seconds=1. The writer is released afterward. The old helper propagatesServerError; the corrected helper recovers on its next poll. No response is mocked or rewritten.The prospective fixture explicitly uses SQLite busy timeout 100 ms, helper timeout 25 s including registration, poll timeout 1 s, and separate existing shutdown timeout 10 s. It does not change any historical cohort budget. Source-preview pressured originals finish helper plus shutdown in 3.09–3.26 s. Both normal-package and preview healthy controls also pass. All 39 installed production files match their respective declared source, with identical eight transitive dependency versions.
All six completed originals have exactly one real activity callback, one completed activity attempt, three first-attempt completed tasks, six events, zero repairs and zero failed jobs. Independent cold clients preserve exact int64
9223372036854775807, Unicode, bool and null values, the original history prefix and original deadlines. Read-only stopped-state SQL independently verifies those counts. The four published failure originals remain pending/waiting, with no activity callback or attempt; none is relabelled completed.One earlier healthy original completed its workflow/cold read but failed the stopped read-only audit under consumer UID33. That aggregate remains failed; fresh subsequent controls use the normal Python image default UID0. No Server or SDK defect is inferred from that reader refusal. Every API in the final ten-control cohort exits 0 without OOM, and all test projects/data/dependencies are retired.
Fresh-main qualification and normal 2.11.1 protected publication remain pending. Next: verify the normal PyPI artifact and rerun affected real-HTTP acceptance with that installation. These source-preview results do not qualify a published SDK or rescue an unrelated retained parity failure.
rmcdaniel commentedon Oct 24, 2026
Delivered in normally published Python SDK 2.11.1, exact source 3511924, whole 4466d411b8f9b20eebd1d1c4c082b46f546d2b49.
All applicable candidate and independent fresh-main checks pass, including Python 3.10/3.11/3.12, integration, quality, corpus, CodeQL and boundaries. The protected PyPI OIDC publisher 40061568529 completed successfully. The normal PyPI 2.11.1 package is available; wheel SHA-256 is
1e7f0210ef30ae33e2224f6857386a43e7793b725a298a0974b2bd6e53c0318f. All 39 installed production files match the tagged Git source and all 51 RECORD hashes are verified, including five declared console launchers.Actual installed acceptance: five new originals on unchanged published PHP Server 2.11.1, immutable index
bf8bed13f5f809ee3425233180a581353eac4683adbee5c4ecbf63bb18bbd102, pass: healthy default plus workflow-poll and activity-poll pressure separately in default and named namespaces. Each pressure case receives three actual typed HTTP 503backend_lock_pressureresponses from a real SQLite writer lock through the normal Client, with null task and exact namespace/queue/kind/retry delay, then completes the same original. Responses are neither mocked nor rewritten.Every original retains its 25-second helper budget, one-second poll and 30-second run/execution deadline, with separate ten-second shutdown policy. Completion plus shutdown takes 1.68 seconds healthy and 3.24–3.40 seconds under pressure. Stopped SQL and cold reads prove 5 completed runs, 5 real callbacks, 5 first activity attempts, 15 first-attempt completed tasks, 30 events, zero repairs and zero failed jobs. Exact int64, Unicode, false/null values and history identities survive cold delivery. All five APIs exit 0 without OOM.
The earlier normal-PyPI 2.11.0 counterfactual failures, broad local original and unchanged-baseline diagnostic failures remain recorded. The initial installed-file verifier refused pip --target console paths; the corrected verifier resolves only the five declared console entries and validates every hash, without skipping files or changing the package. No historical native/shared cohort failure is rescued or retupled by this new acceptance.
All owned applications, databases, networks, installed SDK/dependencies, tooling and raw database state are removed; empty task inventories and borrowed-image preservation were verified at 04:39:14 UTC, before the declared 04:50 bound. General's SDK acceptance is complete. Server #721 retains independent cohort scheduling/adoption and every original observation.