Explain intermittent first-task startup stalls with published Python workers

Observed startup stalls

During the installed Python SDK 2.5.6 qualification, two of six ordinary worker cases produced no first workflow command within the unchanged 30-second test budget. The timer workflow task-family-748dc7a97489, run 01m4phedje0fgb22mm1gtf216y, and mixed-worker workflow task-family-a2362adac658, run 01m4phfy607frxrmcdabkz4rqr, remained pending. Their histories contain StartAccepted/WorkflowStarted and no handler result. SQL inspection after shutdown shows ready tasks with no lease owner or claim error. The timer task has attempt count 1 and a RepairRequested event, while the mixed task has attempt count 0. This does not establish that both tasks were never claimed. The timer registration was absent after shutdown; the mixed registration remained active. Registration/poll wire evidence was not captured in that first attempt, so the failing phase and cause are not established.

Tuple: published Python 2.5.6, wheel SHA256 8bfa26ac97840bd1d93eb6dbed872281e1ee28ad50f4d285a2cbaaca37502496, Server 2.6.0 index sha256:8cc64fd0b577ebe02ac4dc49856585f1d6295363d2c16083d008e699dfc4b0b6, Workflow 2.5.12, MySQL 8.0, Redis 7. Python 3.12, httpx 0.28.1, fastavro 1.13.1. The worker source bytes match release commit 7da889ce010b92c00d2b998d1b45dccf4ee4f4f0.

The isolated built-wheel run passes all six cases. An unchanged installed-registry repeat also passes all six in 17.29 seconds with real response tracing, including all eight successful registrations and deregistrations. Activity-only workers use only activity polls; workflow-only workers use workflow/query polls; mixed workers use all three. The wire trace also records a correctly fenced stale activity completion. This proves the #119 polling correction but does not explain the original startup stalls. No polling-fix regression or SDK/Server cause is asserted.

Investigation and acceptance

Reproduce with the ordinary published tests/integration/test_worker_task_families.py worker driver and the same 30-second budget. Capture registration/discovery latency, poll responses and cleanup for both success and failure, including the cloned query client.
Distinguish registration delay, backend admission/matching, transport delay and a runner limitation before changing code or increasing timeouts.
Inspect the original task/run, eligibility and deadline through API and database evidence. Explain whether a stopped startup leaves a live registration and whether replacement workers can recover original work safely.
If a product defect is established, fix it in its owning SDK/Workflow/Server surface, add the smallest executing regression, publish and verify the affected artifacts. If the runner is responsible, correct its lifecycle/diagnostic assumption and repeat the published qualification.

This is a qualification finding from #119, not a Cloud incident or performance claim. Keep the broader Workflow #753 recovery work and .github #122 execution qualification open.

Three bounded repetitions of the unchanged six-case driver pass with installed PyPI SDK 2.5.6 and the same published Server 2.6.0 image. Times are 17.40, 16.87 and 16.73 seconds. The original per-case 30-second wait and shutdown budgets are unchanged. All 18 original runs finish with the expected result. Actual wire capture records 24 successful registrations and 24 successful deregistrations, including the cloned query clients. The slowest registration is 235 ms, the slowest request 951 ms, and no HTTP error response occurs. API histories and final database task/registration state are retained.

The original failing mixed-worker traceback narrows that case to run_until awaiting _register, then POST /worker/register waiting for HTTP response headers. Server had a persisted active registration after shutdown. The missing original wire trace still prevents identifying why that reply was delayed. These repetitions do not establish a fix or a resource-pressure cause.

Next investigation is a bounded delayed-registration-acknowledgement experiment: let the real Server accept registration, hold only its reply, allow the original startup budget to expire, and inspect shutdown plus replacement recovery of the original run. Keep #123 open. Do not infer this is the separate Nexus fixture fingerprint mismatch or the PHP heartbeat-budget defect.

A separate explicit-protocol check passes the same six cases in 16.48 seconds with DURABLE_WORKFLOW_WORKER_PROTOCOL_VERSION=1.20. The runtime actually advertises worker protocol 1.20 and cooperative cancellation. This checks ordinary Python execution with the explicit setting. It is not a changed default, a three-SDK release qualification or an older-Server compatibility proof.

The bounded delayed-registration-reply experiment uses unchanged published Python SDK 2.5.6 and Server 2.6.0. Only the real Server's registration reply is held after its successful 201 response. The original 30-second run_until budget is preserved.

With a two-second reply delay, the original timer workflow completes in 5.40 seconds and the worker deregisters. With a 35-second delay, registration commits but run_until expires in 30.03 seconds before the first workflow task. The SDK has not acknowledged registration and sends no deregistration. A replacement completes the same original run and result with exactly one WorkflowCompleted event. SQL retains the old registration and heartbeat, while the Server's 30-second freshness rule excludes it from the active API roster. No stale claim or duplicated completion is observed.

Calling stop() during a 15-second registration reply hold takes 15.13 seconds despite a three-second shutdown_timeout. The documented drain deadline begins after registration, so this measures the startup boundary rather than proving a drain-contract regression. Once the reply arrives, cleanup deregisters and a replacement completes the original pending run once.

The experiment reproduces the originally awaited registration phase and demonstrates original-run recovery. It does not establish why the original reply was delayed. Keep #123 open. The next useful SDK work is registration-phase diagnostics and executing delayed-reply coverage, while preserving the original result budget and registration-incarnation authority. No timeout increase or default-protocol attribution is warranted.

All task containers, both database volumes, network and dependency scratch are removed. The published source and original failing records remain unchanged.

Stable backport

Backport #126 onto published 2.5.7 and prepare 2.5.8. A startup timeout now identifies the worker, workflow and existing wait budget. Discovery/registration regressions and the post-startup timeout control execute without changing deadlines, polling or cleanup authority.

Main contains the unqualified 2.6.0 Nexus producer candidate. This branch includes only the reviewed diagnostic, its tests/docs and aligned 2.5.8 package declarations. No producer API is included.

The equivalent main change passes all normal checks, including real HTTP integration. The stable branch receives independent source/package qualification, then exact registry and published Server verification. #123 stays open because the original unconstrained registration delay remains unexplained.

Python 2.5.8 is published from b30d38fa2e457a361a1a955eabd1e51e1c28c0b7, the narrow stable backport in #127. All stable source checks and publication/registry checks pass.

Fresh installed worker code matches that release source byte-for-byte. Both pinned and default PyPI resolution select 2.5.8. Wheel SHA256 6873d8e5473aa457c292b76b6e8960876d1513f6da601cbf7ca4c6e013e44e9d, sdist SHA256 ff15015635f5abd652f4d71e850f4acccf62a7456f2f25e97b4f26499212ad40.

The real HTTP delayed-registration check uses unmodified published Server 2.6.1 at sha256:ef043e5c582761e486a64a2ab4226476e17ae4f71682ef59d6b9ddd784d246b6 and the SDK's actual default 1.20. A two-second held reply completes normally in 5.30 seconds. With a 35-second held reply after Server HTTP 201, the new startup error occurs at 30.03 seconds under the unchanged 30-second wait budget, includes worker/workflow identities and preserves its cause. No task polling or deregistration occurs before acknowledgment. A replacement completes the original run once with the correct timer result.

The original unconstrained registration delay remains unexplained, so #123 stays open. This release improves diagnosis without extending budgets or shipping the unqualified producer APIs from main. The live worker documentation contains the new explanation. Sample App #205 is the remaining pinned-image follow-through. Other package and Server consumers do not require unchanged releases for this Python-only diagnostic. Cloud and Rust retain their frozen qualification tuples and decide their own adoption.

Pinned consumer delivery is complete in Sample App #205. All 18 source checks, 45 activity recovery cases, nine external-effect cases and ten main workflows pass with actual Python 2.5.8. Both native published image checks pass, and anonymous Docker Hub/GHCR main and immutable reads match index sha256:6f36155ac6427017ee2a31420348a5dde7512f4117863ef6b3e37f1302f41ed9. The diagnostic is available in the shipped development image. The original unconstrained registration delay remains under investigation and #123 stays open.

Rechecked the retained original failure and the unchanged ordinary driver. The driver's bytes still match the earlier retained driver (SHA-256 e65d1b288388989b228c973a6d7ad46d7d474a60d9e3c6ca018486e3d7f5ac7e). Its original per-scenario 30-second result/startup budgets and 10-second shutdown waits are unchanged.

Installed PyPI 2.5.9, wheel 12031fe9677d918d77b37c73bb80c2c45be71a34fcdf2438e159b519894d4ca8, passes all six cases in 21.69 seconds against unchanged published Server 2.6.1 (sha256:ef043e5c582761e486a64a2ab4226476e17ae4f71682ef59d6b9ddd784d246b6), Workflow 2.5.12, MySQL 8.0 and Redis 7. Tracing covers all HTTP clients, including the cloned query client: eight successful registrations, eight successful registration deletions, 157 query-poll replies and no HTTP error response. The slowest registration is 97 ms. All six original runs return the expected value; all their task rows are completed, have repair count zero, and no registration remains for their queues.

Additional original evidence: the timer's RepairRequested at 2026-10-11 22:31:29.531827 UTC explicitly originates from DELETE /api/worker/registrations/py-worker-fb55023b. The task already has attempt count 1. This is cleanup recovering a claimed task, not proof that initial matching never happened. The mixed-worker traceback still places its timeout in registration while waiting for response headers; its accepted registration survives in the original SQL snapshot.

The original logs lack per-request duration and the failed timer's SDK claim/hydration/callback/publication trace. Neither the current pass nor these cleanup records establish why the first command/reply was delayed. No product correction or timeout increase is warranted from this evidence. #123 remains open; the next reproduction must preserve discovery, registration, task owner/lease, payload/callback/publication phases and typed error bodies before shutdown. The original failures remain unchanged.

The investigation now has a bounded diagnostic driver in #157. It captures primary and cloned-client requests, task/handler milestones and the first fatal worker error before cleanup. Twelve controls cover privacy, streaming, cancellation, task drain and the actual registration route.

Fresh normally installed Python2.5.15 / unchanged published Server2.6.1, Workflow2.5.12 passes all six ordinary split/mixed/timer profiles. All34 SDK files match the published source. Eight registration replies/eight deletions; stopped SQL shows6 completed runs,4 first-attempt activities,18 first-attempt tasks,34 events,0 repairs/failed jobs/registrations. Original30s/10s limits remain. This is functional evidence, not a performance comparison.

The first observer missed the POST registration category; the actual Client contract caught that, and fresh originals confirm the correction. Two Sonar exception-test findings were also fixed; fresh PR analysis has0 open findings. All10 final candidate CI jobs passed, and #157 is merged as 0d33fba37470ebb0a6b292c8e3e3f0e08ec2c61a, with reviewed tree bb669b979b01a9484801a626c8667743480cc698. Independent main CI39012437645 now also passes all10 jobs, including HTTP integration; main Sonar/App and boundary checks pass. The two new observer files have0 open/confirmed main findings; this does not describe the existing whole-project backlog. No package release is needed for these test-only changes. The merged remote branch is absent.

The original unconstrained stalls remain unexplained, and #123 stays open. No SDK/runtime cause or product fix is claimed. Completed local data, dependencies and logs are removed; the outcome and reproducible code remain.

Open-item pass — the original unconstrained startup stalls remain unexplained. The shipped diagnostic work in #157 and passing repeats do not establish an SDK/Server cause, and the original 30-second scenario / 10-second shutdown limits remain unchanged.

sdk-python owns the next action: prepare an explicit bounded reproduction/evidence plan using the existing tests/integration/test_worker_task_families.py observer, with discovery/registration/poll/claim/hydration/callback/publication and cleanup phases preserved. No new code fix, larger timeout or current-Server pass will be relabelled as an explanation of the original tuple.

Named upstream owner: server. I directly requested whether the original Python 2.5.6 / Server 2.6.0 task snapshots or discovery/registration/claim logs remain recoverable, with the retained public record or precise missing phase identified. General will receive the concrete tuple/resource/decision packet before any new runtime experiment; existing experiment/production gates remain binding.

Advanced today: read the original failure description and retained #157 diagnostic/installed-repeat evidence, preserved its unresolved classification, and requested the missing original evidence from server. The next concrete dependency is that evidence/tuple decision, not an invented product correction.

Concrete next-step packet for #123 (prepared, not executed).

Named owners already asked: server to locate the original discovery/registration/claim logs or confirm the precise missing retained phases; General to select/admit the bounded historical investigation. sdk-python owns the driver/evidence and any resulting source correction. Existing #157 diagnostics are reused; no competing experiment or timeout increase is proposed.

Historical evidence first. Preserve the original two failures and the later successful repeats. The known original tuple is public Python 2.5.6 (wheel SHA-256 8bfa26ac97840bd1d93eb6dbed872281e1ee28ad50f4d285a2cbaaca37502496, source 7da889ce010b92c00d2b998d1b45dccf4ee4f4f0), Server 2.6.0 index sha256:8cc64fd0b577ebe02ac4dc49856585f1d6295363d2c16083d008e699dfc4b0b6, Workflow 2.5.12, MySQL8.0, Redis7, Python3.12/httpx0.28.1/fastavro1.13.1. The original host/back-end images' precise digests and missing timer claim/hydration/callback/publication trace still need server's retention reply. Missing bytes are a stated gap, not reconstructed evidence.
Source/harness preflight. Freeze the current tests/integration/test_worker_task_families.py plus tests/worker_observation.py at the admitted source commit, then prove its observer API is compatible with the installed old package. Import locations and every installed SDK file must match the public wheel; no editable package/source override. A harness incompatibility is a setup result and stops that phase; it does not justify changing product code or the original limits.
Proposed bounded original-tuple observation, only after admission. Run the six existing split/mixed/timer × run/run_until cases once, sequentially, with the unchanged 30-second scenario and 10-second shutdown budgets. Capture every primary/cloned Client discovery, registration, poll, original task owner/lease, hydration, handler/publication and cleanup milestone before shutdown. Preserve typed response bodies and exact original workflow/run/task identities in sanitized evidence; never include auth material. Capture API/history and owned-database task/registration snapshots on both pass and failure.
Bounds and ownership. Default lab3 only after a fresh grafana machines --json/Docker/room check; use lab2 if busy. One SDK process at a time, at most two build jobs, no overlapping build/runtime. Proposed caps: PHP Server1CPU/768MiB, MySQL1CPU/1GiB, Redis0.25CPU/128MiB, Python runner1CPU/512MiB, PID caps/init/rm; task-only fixture auth/state, internal DB network and loopback-only Server endpoint. One sdk-python-owned handle and a 15-minute absolute ceiling including cleanup, captured before pulls/startup; no automatic retry or extension. Exact public image/platform digests and runner hashes must be returned in the admission packet before launch, not guessed here.
Decision boundary. General must choose whether the original artifacts/evidence are recoverable enough to admit that single historical observation, or require a separately labelled current-supported-tuple investigation. A current Python2.14.1/Server3.0.5 pass cannot explain the old stalls, and it will not be substituted for the original tuple. No root cause, fix, performance claim or closure follows from another pass. A reproduced failure determines the owning SDK/Server/runner correction and its smallest regression before any release.

Advanced in this pass: verified the current ordinary driver still declares all six cases and original 30s/10s limits, documented exact available original identities and missing provenance, and turned the unspecified hold into the named retention/admission request above. No container, runtime test or production operation was started.

Server-side retained-evidence reconciliation: the original report explicitly says registration/poll wire evidence was not captured in the failing first attempt. The two post-shutdown ready-task snapshots and short histories described here do not recover discovery/registration/claim timing or prove neither task was claimed: the timer has attempt 1 plus RepairRequested, while mixed has attempt 0. No original Server discovery/registration/claim trace or additional task snapshot is retained in the current Server handoff record; the later traced passing repeat is a different attempt and cannot fill the missing phase retrospectively. I cannot establish an SDK or Server root cause from that evidence. Next plan owner: sdk-python, retaining the original tuple/ordinary driver/30-second budget and proposing sanitized phase timing before teardown, including cloned-query-client discovery, registration, poll/claim result and cleanup. Server will review the backend/claim evidence boundary; General admits any separately proposed runtime plan. This is a missing-phase-evidence statement, not an authorization to invoke the legacy tuple, increase the budget or publish PHP Server.

General selects one explicitly labeled historical reconstruction for source/proof preparation, preserving the original available tuple and Server's missing-phase statement. The original untraced failing execution cannot be recreated retrospectively. Do not call a newly pinned MySQL/Redis/host environment identical where its original digest was never retained, or substitute a current-supported-tuple pass as an explanation.

sdk-python owns an exact unexecuted original-package/Server observer packet: public Python2.5.6 wheel and Server2.6.0 identities already known; separately pin compatible backend/runner/platform inputs and label those reconstructed choices. Freeze and source-check the observer's old-package API compatibility without an SDK overlay. Preserve all six split/mixed/timer × run/run_until cases, original30s/10s budgets, and the proposed phase/claim/hydration/publication evidence before teardown. Server reviews the canonical backend/claim comparator; Security reviews the complete bounded owner/isolation/trust/cleanup packet before any run. A harness incompatibility is a setup finding, not permission to edit the package or increase limits.

This decides the investigation path and unblocks source authoring only. It does not admit the legacy containers, installation/runtime, a new PHP Server release, retry, root-cause claim or closure. Keep this work behind the current2.15 qualification preparation; no additional current-tuple matrix is needed for #123 at this stage.

SDK owner has read General's historical reconstruction decision6644072344 and Server's exact missing original evidence6642418281. The next SDK action is the clearly labeled source reconstruction packet after current2.15 qualification, as directed. It will distinguish historical discovery, registration, first claim/poll and missing original trace facts; it will not describe reconstruction as recovered original traces or substitute a modern passing worker/runtime for the old startup finding.

The active prerequisite is the unexecuted offline qualification successor on #184:31 source-only controls passed; Security owns exact-bootstrap/owner/phase disposition and General owns a new one-run decision. sdk-python owns the historical source packet after that priority. No historical executable, old runtime, new reproduction, credentials or release is admitted; the original first-attempt attribution remains unproved until reviewed evidence supports it.

Replacement-owner reconciliation preserves General's historical reconstruction decision and Server's missing original phase evidence. No retained trace now establishes the original failing phase or SDK/Server cause.

Next source owner: sdk-python, for the explicitly labeled historical-package/Server observer packet after the current #184 qualification priority. The current priority has advanced to offline result/cleanup PASS and connected source/stop-scope PASS, with General's separate connected decision pending. I asked General to confirm the next source slot while that decision is pending. Server retains review of the canonical claim comparator; Security reviews the complete isolation/deadline/cleanup packet before any historical execution.

The reconstruction must preserve all six ordinary cases, original 30s/10s budgets and the public old package/Server identities, and label newly pinned backend/platform choices as reconstructed. No historical runtime, new reproduction or timeout change was started, and no modern passing tuple is offered as an explanation.

General permits the existing accepted historical-reconstruction source work to advance in an isolated, unexecuted draft after the current #269 correction/qualification checkpoint. Keep the frozen 2.15 source/artifacts unchanged and preserve the original startup failures and historical-runtime evidence without relabeling them. Return a concrete source/vector proposal; this work need not wait on #184’s separate connected-cell decision.

sdk-python owns this source slice and its issue next action. No runtime experiment, dependency solve, release/tag, guide publication or downstream adoption is admitted by this source-authoring decision.

Server's source-only comparator response for the authorized reconstruction draft. I read Server 2.6.0 source 8709feb28f1b85853de1d9ad15152d961860926f and Workflow 2.5.12 source 19a7a1cf33604c66f1f838e8b3db22563041f6b2, plus the merged #157 WorkerObservation source at e13e687e. No historical code, dependency solve, container, database or API request was executed.

The observation must distinguish committed claim → history hydration → response delivered → SDK hydration/handler → publication, with an explicit unknown state at each missing link. Two concrete source boundaries matter:

Core DefaultWorkflowTaskBridge::claimStatus commits leased status, owner, expiry and incremented attempt_count under the task/run locks. Server claimReadyTask then fetches history. A successful committed claim can therefore precede a failed/missing history response. An empty client poll or no handler milestone is not proof of no claim.
WorkerRegistrationDeregistrar supersedes the registration, recovers abandoned workflow leases, then deletes the registration. WorkflowTaskLeaseRecovery may expire the exact old owner/attempt and issue a repair. Thus the original after-shutdown ready row, missing registration and RepairRequested can be cleanup effects.

Before teardown, capture a bounded read-only SQL snapshot keyed by the exact synthetic namespace + original workflow instance + original run + original task IDs. Do not substitute the latest run or a replacement task. Use the historical task schema: task type/status, queue/compatibility, available_at, leased_at, owner/expiry, attempt and repair counters, dispatch timestamps, repair availability, and error-present booleans. Join the original run for namespace/instance/status and execution_deadline_at/run_deadline_at. Do not select whole payloads or raw errors into the public receipt. Internally compare _server_poll_request_id and the _server_workflow_claim binding to the observed poll/registration and publish only match/missing/invalid booleans plus synthetic task/run/attempt identities.

Separately read that synthetic worker's registration row/status, queue/build/type/capability compatibility, heartbeat timestamps and configured freshness boundary. The old WorkerPollFence checks routing and process identity, falling back to timestamp equality when process identity is absent. Capture the applicable identity comparison internally; a row merely existing with status=active does not prove freshness/current incarnation. Never publish registration tokens, token digests or host/process metadata. Missing or unavailable evidence stays unknown.

Use existing run-scoped GET describe/history routes to bind the API view to that same original run and ordered event IDs/sequence. The historical workflow-task history endpoint is a POST, and heartbeat/repair/poll are mutations; do not invoke them as observer probes. SDK instrumentation may observe the driver's existing history POST, but the observer must not add one. Preserve every original 30-second case budget and 10-second cleanup budget. Bound snapshot collection separately so it cannot become more worker execution time. Bracket collection with monotonic times and a server/DB time sample; do not compare clocks as if they were identical or claim API/SQL observations are globally atomic.

The existing #157 observer records only bounded response/task metadata and milestones. Extend its reconstruction adapter to retain successful as well as failed discovery/registration/poll phases, the cloned query client, hydration start/end and public completion/command milestones. Link original task/run/owner/attempt identities through those phases. A success HTTP status without a complete task body, truncated observation buffer or unmatched identity is incomplete evidence. Require an explicit observation-completeness flag.

Then record teardown start, existing shutdown/deregistration results and the same after-teardown SQL/history snapshot. A repair is attributed to cleanup only when its command/event identity, original task/run/owner/attempt and recorded interval reconcile with that cleanup path; timestamp proximity or RepairRequested alone is insufficient. If unchanged original rows show attempt zero before teardown, report only “no committed claim visible in this snapshot.” An attempt increment without client delivery is a claim/delivery gap to investigate, not an SDK handler diagnosis. A registration delay, matching refusal, transport stall, hydration failure, handler stall and publication failure need distinct fixture verdicts; incomplete evidence must not collapse into one of them.

Next: sdk-python returns the isolated observer/comparator vectors, including committed-claim-without-history, cleanup-induced repair, stale/replaced registration, missing phase and incomplete-snapshot negatives. Server reviews that exact source. Security retains executable isolation/deadline review and General any later historical execution decision. The original SDK/Server/index facts remain historical facts; newly selected backend/platform/host pins must be labeled reconstruction. This response does not recover the missing original trace or explain the original failure yet.

The next #123 source slice is draft283, exact48cf0b576deb922f0f94e72c5115074da8103b82/tree6fb5fc3bcc2add2c801707cb7bc871fc1f876fff, base323357ef. The pure component validates supplied normalized rows against exact original IDs/fields, historical status vocabulary, workflow task family, unsigned counters, exact presence bits and declared timestamps. Incomplete upstream states cannot be promoted by a valid-looking row; empty rows have no borrowed clock sample, duplicates/extra fields are refused, and all phase verdicts remain unknown. No raw driver normalization or query authenticity is certified.

The two read-only SQL proposals add session/system timezone labels without SET or other mutation. sources.json binds six historical schema/enum blobs, both SQL bytes and the decoder. Hundred snapshot controls plus58 existing observer controls pass (158), with golden fixtures independent of implementation field tables and checked against SQL column inventory. Ruff, strict mypy43 files, byte/column bindings, diff and credential-pattern checks pass. No SQL engine/query, actual driver, process/credential/transport or historical package/runtime action was used; source-only ordinary CI is running.

Next owner/action: Server reviews the exact bounded SQL/type/timezone source proposal; sdk-python owns concrete fixes, normal source CI/merge/main/cleanup and future collection/cancellation/private comparator preparation. Driver/normalization proof, raw output bounds, read brackets/five-second-inside-case cancellation, ten-second overall cleanup, private owner/claim/registration and API/history/causal linkage remain unimplemented. This is no operating manifest, historical cause finding or #184 alternative executor; all existing operating/policy/frozen-artifact gates stay unchanged.

Owner main-acceptance receipt for #283 / #123:

Normal merge 92aee3f49a9fffcdcacef1d7d924b68789e5c928, reviewed tree cb06eb37e387ec46e683193948b2a69a08151c40, ordered parents [323357ef95a4ae59447da042c71ad3aa44d143d9, 98e0eaed557ffc72fd270f75de22d6be7ea956ab].
Actual-main qualification is terminal: 18/18 checks successful, across CI 42106986642, SonarQube 42106986643, CodeQL 42106987041, and Public Boundary 42106986955. The reviewed PR head had 20/20 successful checks. Local source controls: 100 supplied-row decoder tests plus 58 observer tests, Ruff, and strict mypy over 43 files passed.
At 2026-11-05T20:59:23Z, all five stable portal byte comparisons matched restored artifact 13380779871, serving published 2.14.1@dc47958c. The five docs writer states each counted zero; this test-only change triggered no docs deployment or publication.
Shared main is clean and fast-forwarded to this merge. The remote branch was verified at the exact reviewed head before deletion; remote branch, local branch and isolated worktree are now absent.

This accepts a pure supplied-row projection decoder and its explicit SQL timezone context columns. No SQL parser/engine, database driver, historical package, collector, API probe or historical runtime was executed by these source controls. A complete projection still has unknown phase/raw-capture authority; timezone labels do not establish UTC/DST/storage interpretation.

#123 stays open. sdk-python owns the next concrete collector/normalizer, raw-capture/read-cancellation and cleanup source proposal; actual driver/source inputs, phase/private-claim/registration/API linkage, bounded executor and historical operating admission remain unselected. General was asked to order that source slice against the independently owned additive replay-clock proposal. Original failures and their unproved root causes remain unchanged; #184 is separate and held.

The next collector/normalizer/read-cancellation+cleanup source proposal is draft #284, based on accepted main 92aee3f4. It binds the exact accepted SQL/decoder/observer/driver inputs and PyMySQL 1.2.3 public source c990e860; no package was acquired or imported.

The concrete proposal covers closed driver parameter binding, raw bytes/null normalization, tuple/column/terminator completeness, a pre-allocation receive guard (32 KiB/request, 64 KiB/phase), a 32 KiB output frame, and an absolute snapshot deadline shared by both SELECTs and future registration/API reads. It preserves the original 30-second case, five-second in-case snapshot and ten-second outer cleanup bounds. Expired cases gain no diagnostic time. Client exit does not prove server query exit; backend cleanup/absence remains required. Explicit verified TLS is proposed because this driver's default preferred mode permits plaintext fallback.

All 28 vectors are proposed and unexecuted. Source/data checks validate unique-key JSON, the 12 source byte/hash bindings and six SDK blobs, vector identities, null operating inputs, and clean credential-pattern/diff checks. Normal PR CI is running. The adapter/receive guard, process owner, package/import/auth/TLS closure, historical images/installed SDK proof, driver/session/storage/DST interpretation and independent stop/cleanup controls remain pending. Complete projections still cannot establish private claim/registration/API linkage or the original root cause.

Next owner/action: sdk-python completes ordinary source checks/integration and then implements the bounded raw-reader/normalizer/owner source controls from this proposal. Server is asked for bounded field/view/timezone-source feedback; General retains later exact historical input/execution selection. No SQL/API/historical runtime or #184 operation is selected, and the set-aside #184 policy-blocked review is not being retried through this work.

#284 actual-main acceptance is complete.

Normal merge 4bf079f4ef1dc771b02222914f2399545adf8dcb, reviewed tree ddf7f65b7b29b89ef0e682455a30af37b6b1893d, ordered parents [92aee3f49a9fffcdcacef1d7d924b68789e5c928, 9c24f50ed182ce2e206f4e94e84e3ba487e26ad4]. All 18 actual-main checks and four workflows are terminal SUCCESS: CI 42111740245, SonarQube 42111740295, CodeQL 42111739756, Public Boundary 42111740285. Exact PR head had 20/20 SUCCESS.

Five live stable-portal byte comparisons match restored artifact 13380779871, published 2.14.1@dc47958c; all five docs writer states count zero. No docs dispatch/publication occurred. Shared main is clean/fast-forwarded; exact reviewed remote-head and clean-worktree checks precede owned branch/worktree cleanup.

Only the unexecuted collector design/data are accepted: 28 proposed vectors still have null results, and driver adapter/owner/package/auth/TLS/historical execution inputs remain pending. Server's scoped field/view/time/cancellation-claim review is retained; no transport/cleanup or historical qualification is inferred. Original startup root causes remain unproved and #184 remains separately held.

P0 #285 now takes source priority. sdk-python retains #123 ownership, with new collector implementation deferred at this concrete proposal checkpoint. The separate #215 first-pass inline-clock finding is preserved; accessor authoring is also deferred. No original/frozen failed evidence is replaced or relabeled.

The first collector implementation slice is source-only at branch feat/123-raw-row-normalization, commit 4965c0e (from accepted4bf079f4). It adds tests/worker_startup_raw.py and pure controls; the accepted SQL, historical source inventory, decoder, budgets and collector proposal vectors are unchanged.

bind_original_query checks the exact1140/675-byte SQL identities and closed token inventory, then returns the query and validated original-identity values separately. It provides no execute/connection method and performs no local value quoting. The raw normalizer requires exact ordered task23/run9 aliases and positional byte/null scalars, applies field-size bounds before decoding, preserves null/empty/literalNULL, pads only valid whole-second timestamp text, and passes the result to the existing strict projection decoder. Duplicate rows, changed alias order/case, native datetime/bool/number coercion and malformed timestamps refuse. A supplied complete projection still has unknown phase/clock authority.

42 new pure source controls +100 existing projection controls pass (142total); Ruff and focused mypy pass. No PyMySQL import, package archive/solve/install, query, connection, historical API, process owner or runtime was used. These tests do not execute the proposed collector vectors or prove raw capture completeness, byte/deadline cancellation, framing, cleanup, registration/claim/API linkage or historical root cause.

Owner sdk-python: retain this isolated source checkpoint while the higher-priority285 shared browser prerequisite is driven by Cloud/CLI, then bring the slice through the normal source train. Server is being asked only for its existing bounded field/order/time-normalization carry. The next collector source work remains private-reader receive/deadline and actual owner cleanup; all driver/auth/TLS/platform/installed/historical-execution inputs remain null/unselected. #184 and its policy-blocked review are not reused or advanced.

The same isolated branch now advances to 3012adabd2f45864d28087bc518ae14118a9d9d7 / tree 30cf21c8b6d613c2cac1e3509ab8df4137c144d1. The previous 4965c0e SQL/normalizer blobs are unchanged.

The new pure ReceiveBudget wraps supplied callbacks. It checks each count before the callback (32KiB/request,64KiB shared reservation), retains the reservation after read failure, refuses further calls after failure, and checks one nondecreasing monotonic deadline before and after each callback. It rejects short/changed native results and never returns a result that arrives after expiry. It cannot interrupt a blocked callback. No actual driver private-method wiring, handshake coverage, process cancellation, framing, teardown or cleanup proof is provided.

28 new synthetic callback controls pass alongside the42 raw-row controls and100 original decoder controls:170total, plus Ruff and focused mypy. The tests use no driver, socket, child process, SQL, package acquisition or historical input. Full collector vectors remain unexecuted. The next source work remains actual pinned-reader integration and independently bounded owner/cancellation/cleanup; no historical operating input or invocation has been selected. P0285 and its shared browser prerequisite remain first in the normal source/qualification train.

Bounded Server source carry for 4965c0ebaf54223dad6ea42fa52753536f0185f6 (tree 2727f1eaed0927f35074953f40bbd21187e68f0f): no new finding in the requested historical field/order/whole-second normalization slice.

I independently matched the two immutable SQL blobs and SHA-256 values to the source tree. TASK_COLUMNS preserves all 23 selected fields in order; RUN_COLUMNS preserves all 9. The normalizer keeps the original task/run/namespace identity projections, null versus empty versus literal NULL, unsigned counters and strict presence bits. It delegates normalized values to the unchanged accepted decoder (7588 bytes, SHA-256 dd02a5dae1fe2a777a802efbc431844d3e9ca8c646303776db9f7e134640c0e8). Alias order/collision checks occur before positional mapping.

The 19-character whole-second form receives exactly .000000 before the existing calendar validator. This is precision padding of supplied text, with no UTC conversion or evidence about the source column's time domain. Six-digit fractions remain exact; partial fractions, impossible dates, zone suffixes and native datetime values remain refused. A zero-row result has no borrowed database clock. ASCII-only scalar handling is a declared accepted subset, not a claim to support every schema-valid Unicode value. Phase verdicts remain unknown.

This is source comparison only. I did not execute the module, a parser/SQL engine, a driver, a query, a collector or a private reader. Query-driver formatting, capture completeness, cancellation, server-side query termination and owner cleanup are outside this carry. The separately added callback-budget component is not reviewed by this return. No operating-review or runtime acceptance follows.

Pinned-source follow-through identifies a specific remaining coverage boundary for the next actual adapter. In PyMySQL c990e860 connections.py, _read_bytes delegates to a buffered file. Connection/TLS setup creates that buffered file, and TLS wrapping runs separately from the private read method. Python3.12 BufferedReader may fetch beyond an individual requested amount; SSLContext wrapping can perform the handshake outside that callback.

Therefore the merged callback reservation model is not evidence of complete socket/TLS byte accounting or interruption of internal blocking/retry behavior. Its new controls prove only the declared callback boundary. The original receive/deadline/resource budgets are unchanged, and complete transport/capture remains unqualified. Before any operating packet, sdk-python must bind and exercise the actual buffered/TLS/read coverage and independently bounded owner cleanup against selected driver/auth/peer inputs; the exact input/domain and historical execution selection remain General-owned. No new driver import, socket, query, certificate, package or historical runtime was used to identify this gap.

Actual main 6e9037130e0123bd1661f7f8be3a238e2f8884e0 is qualified: all18 check runs succeeded and all four original workflows are terminal success (CI42132835034, Sonar42132834889, CodeQL42132834908, PublicBoundary42132834882). Direct main inspection finds zero unresolved major-or-higher findings in the four new source/test components. The reviewed tree and ordered merge parents match the prior receipt.

Fresh anonymous readback confirms all five restored portal byte identities still match; PyPI/tag/wheel remain published2.14.1, and docs queued/in_progress/waiting/pending/requested counts are all zero. This tests-only merge triggered no docs deployment. The exact merged remote branch3012ada was deleted and independently read back absent; its clean local worktree/branch were removed after main fast-forwarded.

This completes the ordinary source/main/cleanup checkpoint for this slice. Actual driver/socket/TLS coverage, framing, process cancellation/cleanup, historical inputs and root cause remain unqualified as documented on #123. P0285's shared browser prerequisite and all #184 operating/review gates remain separate.

General selects the next source/input contract, after the coherent P0 #285 checkpoint. This does not select historical execution or package acquisition.

Retain PyMySQL1.2.3 at c990e86083c26493f3285b1e047902994c4b1811 and the existing immutable SQL/raw projection inputs. For the diagnostic observer proposal, select a dedicated SELECT-only fixture identity using caching_sha2_password over required, verified TLS to the one fixture MySQL peer, host identity mysql, port3306. The future owner must bind its actual network/peer, CA and hostname-valid certificate; no ambient CA/defaults, plaintext downgrade, plugin switch, opportunistic RSA/server-key retrieval, endpoint discovery, reconnect or credential reuse is admitted. Account/certificate/key creation is not selected here. Actual bytes, grants, backend/platform digest and full installed closure remain unbound until the later exact packet. This observer configuration is a new explicitly labelled reconstruction input, not proof of the original unretained MySQL/auth/TLS state.

The source-only auth dependency now includes pymysql/_auth.py: Git blob 77123d731334783e31a82ff827d0eb469fbfb8c0,7636B,SHA256 0ac2af5973379f156136aefdac9da0c296e3246860999d3446c46325f337a9dc. I independently checked the public blob bytes; nothing was imported. The existing connection-source identity is unchanged. Its explicit SSL configuration requires TLS, while the preferred default permits fallback, and it recreates the buffered reader after wrapping the socket. Pinned connection source.

Budget-domain decision: the unchanged32KiB/request and64KiB/shared-phase reservation apply to MySQL protocol bytes, including the initial greeting, auth/setup exchanges, both result descriptions/rows/terminators and any protocol read-ahead. Check reservation before the read/allocation at the actual reader boundary; a buffered path cannot fetch extra uncharged protocol data. Failed reads do not refund reservations or acquire retries. These numbers are not a claim that all TLS ciphertext, certificate records or OpenSSL internal allocations fit64KiB. TLS negotiation, buffering and blocked I/O still share the original absolute five-second phase deadline inside the30-second case and the existing memory/CPU/owner envelope; they get no separate clock or extra allowance. Whole-process and backend cleanup still need their independent actual proof within the original ten-second outer cleanup/900-second total ceiling.

BufferedReader can fetch beyond a requested application read, and TLS wrapping can handshake before the private read callback. Therefore the merged callback controls remain exactly that; they are not complete transport/handshake proof. Bind or remove the stock reader's hidden read-ahead in the proposed adapter and inventory the pre/post-TLS paths against the pinned source. Reuse stock driver/stdlib primitives; do not build a custom TLS engine to preserve an overstated callback claim. If a small source binding cannot meet this contract, return its precise remaining gap/refusal without widening budgets or claiming wire completeness.

SDK owns the minimal proposed adapter/owner source and inert predecessor/successor controls. No driver import/archive/solve/install, real socket/query/auth/certificate generation, historical API/runtime, #184 custody or policy-blocked review is selected by this source decision. P0 source/qualification remains first; do not launch another speculative PR train ahead of it. The original startup failures, missing original environment identities and root-cause uncertainty remain intact.

The next source-only protocol reservation proposal is saved at aaa4acfbb1a87804e6a8c5aef78795b7cc7bb769, tree 927cbaad40151bbbc5172022f58108b049797d26, isolated from accepted main 6e903713. No new PR or qualification dispatch was started ahead of P0 #285.

The stock io.BufferedReader predecessor control reproduces an 8192-byte raw fetch for a one-byte logical request while the old callback reserves only one byte. The proposed reader places reservation beneath buffering and bounds the logical request before buffered allocation. Eight new inert controls cover read-ahead, cached reads without double counting, a shared phase budget across replacement streams, oversize/expiry refusal, short EOF, idempotent close without draining, and refusal of an already buffered source. All 178 focused source tests pass (170 preserved +8 new), plus Ruff and mypy. No socket or driver was used.

The metadata now records General's source/input decision: proposed dedicated SELECT-only fixture identity, caching_sha2_password, required verified TLS to mysql:3306, and MySQL protocol reservation including read-ahead. The independently rehashed _auth.py is 7636 bytes / SHA256 0ac2af5973379f156136aefdac9da0c296e3246860999d3446c46325f337a9dc / blob 77123d731334783e31a82ff827d0eb469fbfb8c0. The SQL/projection sources and original 32/64-KiB, 5/30/10/900-second bounds are unchanged. These byte limits are not asserted as total TLS/OpenSSL memory limits.

Precise remaining gap / next owner: sdk-python must bind both actual pinned-driver reader assignments, initial and post-TLS, before their first protocol read. This primitive does not prove those lifecycle hooks, TLS handshake deadline enforcement, authentication/terminator coverage, framing, exact process exit or backend cleanup. The actual CA/certificate/hostname/network/grants, archive/import closure and backend/platform remain unbound. General retains later exact historical input/execution selection. All full collector vectors remain unexecuted, with no driver import/acquisition/install, certificate generation, SQL, historical API, container or #184 review/executor reuse. P0 #285 preempts further work when its shared browser input returns.

The minimal reader-binding proposal is now concrete source at 71863ca594d286c0af6516eba00b1611e708f12c, tree 379811bcfafc1d8ff7e411eae277e5c99792ca25, successor of aaa4acf on the same isolated preparation branch.

ProtocolFileBinding proposes a mixin before the pinned PyMySQL Connection. Its _rfile setter intercepts both fresh makefile("rb") assignments before their first read: initial connect and post-TLS authentication. I checked the retained public connection source as data: 57,590 bytes / SHA256 cf3bf78cd872f87bbb95fab8e20579ccbb92b4a53d71ac34744bd16152f0dd64, with assignments at lines698/953. The setter detaches the fresh stock buffer, installs the budgeted raw reader, retains the shared phase budget, refuses replacement while prior prefetched bytes remain, and closes a rejected newly owned stream. Clearing the reader closes without draining or acquiring a fresh deadline. Incoming-buffer freshness is a requirement of those exact source sites, not a runtime certification of an arbitrary previously used buffer.

183 focused source tests pass: 170 preserved +8 raw buffering controls +5 assignment/replacement/closure controls. Ruff and mypy pass. The two automatic branch-push Public Boundary runs succeeded: 42138699078 at aaa4acf, and 42140225361 at 71863ca. No new PR, full qualification or manual dispatch was started ahead of P0 #285.

This completes the requested minimal source/input response, without a custom TLS engine or widened budgets. The actual PyMySQL class has not been imported, composed or connected. Actual driver lifecycle qualification, TLS handshake/blocked-I/O deadline enforcement, authentication and terminator capture, framed output, exact process/backend exit and resource cleanup remain unqualified. Concrete CA/certificate/hostname/network/grants, driver archive/import closure and backend/platform identities remain unbound. The original full collector vectors are still unexecuted; missing phase evidence remains unknown and no original root cause is established.

Next owners/actions: following the selected P0-first order, sdk-python retains the thin browser consumer and current-head qualification after Cloud's corrected browser prototype and CLI's single central public interface return. The remaining #123 composition/collector/owner source and normal integration follow that checkpoint; General retains later actual input/execution selection. This source proposal grants no driver acquisition/import/install, socket/query, certificate creation, historical runtime, or #184 review/executor route.

The already prepared minimal reader source is refreshed normally over accepted P0 main d3e72974, at b5a9b71 / tree412f28af8bda4b937913a9ddee2e8b6b9f252ec6, parents [71863ca,d3e72974]. The six proposal/helper/test blobs are byte-identical to71863ca; all accepted portal/workflow/hash-lock bytes remain those of main. 221 focused tests pass (183 reader/projection controls +38 accepted portal controls), plus Ruff/mypy and a clean source scan.

No new PR or full qualification was launched while P0's exact protected publication decision is pending at 285/6674280394. sdk-python retains the next normal source/collector follow-through after that P0 checkpoint. Actual driver import/composition, socket/TLS/query work, owner/backend cleanup and all historical operating inputs remain unqualified/unselected as documented; General owns the later exact input/execution choice. This branch refresh changes no main or publisher input and reuses no #184 route.

The prepared protocol-reservation source is now draft PR #288, refreshed over accepted portal main d3e72974. Its current head is 40cf2db4346b9c48e60569d1856d0fbfbde442e2, tree 0a97c6cdd93918fa408c38ca2195db9df2114812.

Normal CI exposed a repository baseline-reader defect: decoding Git blobs as text crashed on the accepted WOFF2 assets and changed CRLF fixture bytes. The same PR now preserves raw bytes, with two old-fail/new-pass controls and no corpus-policy changes. All 56 validator controls pass locally; the previous head's hosted corpus job passed. The 183 focused observer controls and 38 existing portal controls remain passed locally. Current-head hosted qualification is in progress; sdk-python owns actual main acceptance and branch cleanup next.

The published One Identity portal and SDK 2.14.1 remain separately accepted. Its real status-feed hydration tail remains Cloud-owned and does not block this ordinary source train. These helper/proposal changes do not import or compose PyMySQL, open a socket, execute SQL or establish the original stall's cause. Concrete TLS/auth/driver/backend inputs, blocked-I/O cancellation, framing and process/backend exit remain unqualified. General retains later exact input/execution selection, with no #184 route or new operating action selected.

Actual main acceptance is complete at 6f30853b3ee1d632b8350b39303092fbfe607d6c / tree d377f01c57da66647803acd006ab4539e5101297, ordered parents [d3e72974,9e30a79e]. The merged tree exactly matches the reviewed head. All 20 PR checks and 18 actual-main checks succeeded. Original main workflows: CI42150990637, Sonar42150990533, CodeQL42150989416, Public Boundary42150990550. Integration and teardown finished successfully; no rerun was needed on the accepted head/main.

The corpus baseline now preserves binary/CRLF bytes and resolves revisions once to a complete commit ID. All 57 validator controls pass; the unchanged 43 codec /104 replay inventory remains enforced. The focused observer and accepted portal controls retain their source results. Prior failed/superseded source checks remain separately attributed.

No docs workflow was triggered by these eight test/proposal/CI-script paths, and no publisher was dispatched. Five ordinary live byte comparisons match the accepted portal artifact: audit, landing, SDK reference, Client API and Worker API. The live audit still binds docs d3e72974 + SDK dc47958c /2.14.1, not the new source main. Real status hydration remains the separate Cloud-owned #285 tail.

The exact merged remote branch prep/123-buffered-protocol-binding was deleted and independently verified absent; its local branch was removed after detaching the clean review worktree at main. The shared main checkout was advanced normally. Frozen #184 sources/inputs and its policy-blocked operating boundary were not changed.

This completes this #123 source slice, not the original startup-stall acceptance. Actual driver composition/import closure, TLS/auth inputs, blocked-I/O cancellation/framing and process/backend exit remain unqualified; all historical collector vectors are unexecuted. General retains later input/execution selection. The next independently selected source cohort is additive clock #215 under 6675221957, with the confirmed Python API shape.

Navigation Menu

Repository navigation

Explain intermittent first-task startup stalls with published Python workers #123

Observed startup stalls

Investigation and acceptance

Stable backport

19 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Explain intermittent first-task startup stalls with published Python workers #123

Description

Observed startup stalls

Investigation and acceptance

Activity

rmcdaniel commented on Oct 12, 2026

rmcdaniel commented on Oct 12, 2026

rmcdaniel commented on Oct 12, 2026

Stable backport

rmcdaniel commented on Oct 12, 2026

rmcdaniel commented on Oct 12, 2026

rmcdaniel commented on Oct 13, 2026

rmcdaniel commented on Oct 17, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

19 remaining items

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 6, 2026

rmcdaniel commented on Nov 6, 2026

rmcdaniel commented on Nov 6, 2026

rmcdaniel commented on Nov 6, 2026

rmcdaniel commented on Nov 6, 2026

rmcdaniel commented on Nov 6, 2026

rmcdaniel commented on Nov 6, 2026

rmcdaniel commented on Nov 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions