Repository navigation
Preserve Worker registration during slow query payload hydration #146
Description
Observed failure
Normally installed PHP SDK 2.3.5 (c08b18a1855d0ba66d5a504d9238511213b90767) against unchanged published Server 2.6.1 accepts a query task and downloads three external history payloads. Each encoded body is about 11.2 MB and completes in about 22.4–22.7 seconds under a continuously limited 4 Mbps connection.
The retained HTTP trace shows no worker-presence heartbeat during those downloads. The interval between the last successful heartbeat and the next rejected heartbeat is 79.478 seconds. Server advertises heartbeat interval 10 seconds and stale threshold 30 seconds. The next heartbeat returns 409, Worker registration was fenced after its leases were recovered, and the PHP Worker exits. The caller's original 180-second query ends with No active worker is registered. The independently selected query lease is 185 seconds. PHP memory stays within its selected limits, with no container OOM.
This is a concrete published-consumer failure and trace. An independent normally installed 2.3.5 reproduction now confirms the query-presence gap, and the corrected main and compatible stable source pass the affected HTTP checks. See the qualification. Published successor installation and affected consumer follow-through remain pending. This finding does not establish the cause of earlier experiments or other SDK failures.
Source finding
- Worker::workflowPayloadProgress creates transfer progress only for workflow-task polls and workflow history pages. Query polls return no observer.
- Client::request resolves the poll response's external payloads synchronously before returning the task.
- Worker ticks heartbeat after that query poll returns, before dispatching the callback. The missing presence servicing during successful query hydration is distinct from retry waits, which already service presence.
Related prior workflow hydration correction: #137. This ticket does not reopen its completed workflow/result behavior.
Required outcome
- Reproduce with normally installed 2.3.5 and the unchanged published Server. Retain the original query/run IDs, accepted deadlines, unresolved query poll, payload bytes/timing, worker registration identity and successful/refused heartbeat receipts. Freeze caller/query/transfer budgets before admission.
- Keep the same registered Worker alive during successful query payload loading. Preserve Server registration fencing, the query's original deadline and callback/result identity. Stop loading on shutdown or unconfirmed/fenced registration before application dispatch. Do not re-register to conceal lost authority.
- Add focused streamed-response regressions plus real HTTP slow-transfer, registration revocation, shutdown and integrity checks. Review other non-workflow loading paths for the same presence gap without manufacturing activity progress heartbeats or extending task execution budgets.
- Implement and qualify the current main correction and its compatible stable backport. Publish a qualified successor, verify its actual installed query consumer and update affected Sample artifacts. Coordinate Cloud consumer follow-through separately.
General owns the PHP SDK diagnosis and delivery. Server native/shared fixtures remain with the Rust Server owner. Rust SDK activity input investigation #90 remains separate. No default protocol or major-version change is requested.
Activity
rmcdaniel commentedon Oct 12, 2026
The source correction is in draft #147 at
60b719685fcb3b694e0884b37fa4978405b5be32. All 33 ordinary hosted checks pass. The optional connected cooperative suite was not selected. Local full source tests pass 2,331 cases / 11,732 assertions with 51 skips, and static analysis passes. The new focused payload suite passes 29 cases / 489 assertions. The six initial regressions fail on unchanged code.The first independent HTTP controls complete with exact decoded results on normally installed PHP 2.3.5 and the exact source candidate. Their observed downloads were unshaped, so they establish control behavior rather than slow-transfer qualification. Both Workers stop normally and both original runs are terminated through the API. The controls and their original receipts remain separate.
A fresh slow-history workload uses an 8 MiB workflow input, remote echo activity and pending timer. Its query reads the original input and activity result and returns independently checked byte counts and SHA-256 hashes. The corrected Nginx limiter is measured before admission: one complete 11,184,836-byte referenced body takes 22.346 seconds and retains its exact hash. Each payload response is limited to 512,000 bytes/second while heartbeat/API responses remain unshaped. This is a per-response limit, not an aggregate bandwidth or managed-plan performance claim. Fresh original identities are required. Ordinary Workers use the published transport with read-only wire observation, and the candidate remains an identified source overlay.
Authenticated operator and worker discovery reports storage available before admission. Query timeout 180 seconds, effective query lease 185 seconds, controller timeout 210 seconds, timer 900 seconds and run/execution deadlines 1,800 seconds remain selected before the new seed. Poll/task identities, references, complete body timing and worker heartbeat receipts are retained.
Results and compatible stable publication remain pending. No original query deadline or Server fencing threshold is extended.
rmcdaniel commentedon Oct 13, 2026
Normally Composer-installed PHP 2.3.7 now passes the affected ordinary three-reference query against unchanged immutable published Server 2.6.1. Installed source/dist are
275b3bf5d82fa334e6ff1f5ba21e2b4dd55ddf78, product train/registration are 2.3.7, and default protocol is 1.20. The ordinary original run and query preserve one registration, initial attempt/lease, original 180s caller budget and one activity effect. Three complete slow references return independently checked exact 8 MiB input/echo hashes; ordinary query wall time is 83.419s including queue/parent readiness. Explicit SIGTERM, credential revocation and HTTP-reader integrity controls are also recorded with their precise limits.See the complete installed qualification and limits, published smoke using SDK-declared Server 2.5.0, and merged main metadata PR #153. Fresh merged-main CI passes. All qualification resources are retired; namespace read is authenticated 404.
General still owns Sample and cross-repository follow-through. Sample's activity-recovery receipt is pending, so the parent remains open for that required handoff. These checks do not establish Cloud consumer execution.
rmcdaniel commentedon Oct 13, 2026
Closing the delivered PHP 2.3.7 correction after normal Composer installation, affected ordinary/cooperative published-HTTP consumers, main metadata alignment and the required published Sample follow-through.
The installed qualification records exact source/dist
275b3bf5d82fa334e6ff1f5ba21e2b4dd55ddf78, product train/registration 2.3.7 and unchanged default 1.20/minimum 1.19. SDK-declared Server 2.5.0 passes the normal published smoke; unchanged immutable Server 2.6.1 passes the separate affected slow input/activity/replay/three-reference query cells. Metadata #153 and fresh merged-main CI pass.General's final Sample #211 receipt confirms 45/45 original activity-recovery cases, the expanded matrix, all ten fresh-main workflows and actual installed PHP SDK/Laravel 2.3.7 in both native image architectures. Both public registries publish index
sha256:8721e00e1bc5e841df444fcad063929ee6030ddfd768d0c28c4b74c405b8bb3ffor merged9d298b6161def330a2405288ee0164097ddec058.The original limits remain: the inherited SIGWINCH ordinary control ended 137 and is excluded from the later explicit SIGTERM exit-0 pass; the cooperative stopped control lacks an exact final Docker exit/OOM state. Its cold replacement enters a callback after the original caller timeout and completion is fenced 409. Earlier source early-cancellation terminal convergence remains unqualified. Credential revocation is an authority refusal, not clean shutdown. No Cloud, Nexus/shared cohort or frozen rewrite tuple is qualified by this closure. All task resources and credentials are retired.
#112 cache/Octane and #116 Nexus replay/capability-cohort follow-through remain open and separately owned.