Skip to content

Bound worker shutdown and fence deregistration recovery - #214

Merged
rmcdaniel merged 13 commits into
mainfrom
fix/175-fenced-deregistration
Oct 23, 2026
Merged

rmcdaniel merged 13 commits into
mainfrom
fix/175-fenced-deregistration

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Worker deregistration now shares the original shutdown deadline and retains the exact namespace, worker ID and registration generation. A runtime advertising the existing fenced receipt contract allows safe retries for matching typed transients or an uncertain transport response. Legacy DELETE makes one attempt. A committed-but-lost response can be reconciled without removing a replacement registration. Unconfirmed removal stays explicit, and any original worker error remains primary.

Refs #175. The normal installed SDK 2.9.0 / published Server 2.11.1 SQLite control made three DELETE attempts in 0.534 seconds against a 0.05-second shutdown budget. The fixed deadline control returns unconfirmed removal in 0.051 seconds. The older callback original remains failed; its physical lock cause was not captured.

Candidate 2.10.1, exact source 352cbcf1a5ccad4a0e068aa2f17357740b1f378d, follows the independently published 2.10.0. Exact wheel and sdist pass metadata validation, Twine and both maintained installed-package smokes. A separate non-editable wheel installation matches all 37 Python source/RECORD files and passes 143 controls, including the official replay corpus. An initial missing optional OpenTelemetry test dependency is retained separately; installing the repository's declared pinned test dependency makes the unchanged artifact and assertions pass.

  • Wheel SHA256: 6edabe1a64c24ca4ae0dfe58ec71339f417011c71e5a7925dae9af16d685389f.
  • Sdist SHA256: 02ca3843adb1120f3a7e0a80ddf1b063c42d9b84b5aa926d93774d8276b7602b.

All 37 production Python files are identical to the earlier 7de44aa installed cohort, which passes 80 new controls and 11 actual published-image integration controls. Actual HTTP qualification separately covers contention recovery/refusal, lost committed responses, exact deadlines and definitive refusals. Fault injection is distinguished from actual Server responses. The genuine cold-recovery fixture records W1's recovered claim, a same-ID W2 replacement, an idempotent old receipt retry and cold replay after the recovered task executes.

Original CI 39967609788 remains failed: all six task-family failures counted successful DELETE only. A fresh physical probe confirms eight fenced POST 200 receipts with exact tokens, zero remaining worker rows, cleared Worker registration state and still-open parent clients. The observer now recognizes fenced POST and legacy DELETE as deregistration, keeps the successful count requirement and adds state/client checks. Handler milestones alone were not treated as proof of removal.

Source is merged at c376605af929e9466e7f04099fb5985df7483c8f, with all eighteen independent main checks passing. Immutable tag 2.10.1 is pushed; protected publication 40022175405 is queued. Actual normal registry acceptance remains required before delivery. Preserve physical thread/process stop, session and claim fences, original time budgets and the minimum protocol/runtime floors. No Server/Core/native/frozen source, Cloud adoption or shared #225 behavior is imported.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Local artifact and real HTTP qualification now passes on 636a0b28661d5d302caceace2640bb695139fae6, tree 24e8e84bc86bdf0473da8819db0b045e80952e65.

Separate #175 development artifact SHA-256
durable_workflow-2.10.0-py3-none-any.whl 9c236d92054cbeaa7d35c44d1d679e60d808b31957b0b0a6714b21dc14ab9dde
durable_workflow-2.10.0.tar.gz 8ff498add565bfcafb2f781d6c89eff6701762a74a53947f97c39e04ad989422

Both distributions pass exact-source metadata, Twine and the maintained isolated package smoke script. A normal non-editable wheel installation, without a source overlay, matches all 37 Python source files and RECORD hashes. 63 new receipt/recovery controls pass without skips; the wider normally installed Client/Worker/retry set has 547 passes and two pre-existing optional-Avro-reference skips. Python 3.12.3, pip 26.2.2, certifi 2026.10.23, httpx 0.28.1 and fastavro 1.13.1. Full source qualification passes 3,453 tests, with two existing skips, 109 integration deselections and one existing fork warning; Ruff and mypy pass.

The exact published Server 2.11.1 / Core 2.10.2 image in the linked baseline finding is unchanged. Four committed integration controls pass against it, including discarding a real committed HTTP response after a replacement registers. A separate actual TCP-drop control also passes: two byte-identical fenced POSTs replay the old receipt while the replacement remains active. Real contention controls distinguish transient recovery, persistent refusal and the unchanged 0.05-second deadline returning unconfirmed removal after 0.051 seconds. Actual 401, 404, 409 and controlled-expiry 410 refusals remain failures. Explicitly labeled discovery/response fault injection confirms capability-absent one-attempt behavior and refusal of mismatched HTTP-success identities; these are not claims that the producer returned malformed data.

Strict legacy receipt validation exposed incomplete inline storage-admission stubs. Those stubs now return the already-existing Server 2.8+ envelope; business execution counts, request identities, acknowledgement bytes, original budgets and physical-stop guards remain asserted. Earlier failing tests/runs stay failures. Server 2.8.0 already advertises this v1 fenced contract, so no floor change is required; capability-absent legacy cleanup retains its documented lack of a generation fence.

The local runtime, SQLite state, test-role credentials, completed build worktrees and normal-install environments are retired after preserving proof. Current-head hosted CI and final analyzer review are still required before merge. No tag, upload, publication, registry acceptance or automatic adoption is claimed. This distinct #175 development build does not automatically replace the existing #213 4f853f39 cohort; the already-requested Trusted Publisher setup and separate security-only 2.9.1 remain unchanged.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel marked this pull request as ready for review October 23, 2026 16:54
@rmcdaniel

Copy link
Copy Markdown
Member Author

General takeover review: independently reviewed the four production diffs, including single-attempt transport calls, exact typed receipt/transient matching, registration namespace/worker/generation snapshots and the original monotonic shutdown deadline. Legacy DELETE remains one attempt; fenced retries cannot consume a new deadline or remove a replacement registration. The original worker error remains primary.

Exact candidate 352cbcf has all applicable normal checks complete and direct Sonar review has zero open/confirmed findings. The prior owner's 143 normal installed controls / 37 production files and eleven actual published-image controls remain attributed to their recorded cohorts; this review does not call them a new run.

Proceeding through normal source merge, then independent main qualification. Version 2.10.1 publication and normal registry/artifact acceptance remain required before closing #175. No Server/Core/native/frozen, shared failure-report or Cloud adoption is imported.

@rmcdaniel
rmcdaniel merged commit c376605 into main Oct 23, 2026
18 checks passed
@rmcdaniel
rmcdaniel deleted the fix/175-fenced-deregistration branch October 23, 2026 19:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant