Repository navigation
Oversized activity failures can't be reported, so the activity reruns forever #225
Description
Receiver published; installed module controls pass; consumer adoption pending
Published and installed checkpoint, 2026-10-24: Core 2.13.0 is normally published from ae18295693273907d0f750b479eb9b8a593e5803 / tree c75f48065008711a43238943d8e0dea0b3d1e152. The tag and Packagist source/dist agree. Independent main40079214847 passes all22 applicable jobs; exact Sonar is OK with88.8% new coverage and0.4% duplication. Normal published verification40081480723 passes all17 jobs. A fresh normal Composer installation passes 58 reader/replay/payload/timeline tests / 458 assertions, with all2175 package files matching source before/after. Published and installed receipt. The separate2.12.2 canonical-storage/retention installed-driver receipts remain component evidence. Actual application/Server/rollback/SDK/Waterline adoption remains open; no rejected-outcome producer is admitted.
PHP consumer adoption: Server #733 pins published Core2.13.0 at draft head cf450accf8de903bcde3b3542ef2b663f5409420 and prepares a fresh Server2.11.3/chart0.3.3. Actual HTTP history/export/retention/activity/validation controls pass84 tests/840 assertions with stock Composer; the old published Core2.11.4 counterfactual returns500 on the stored fact. Source consumer receipt. Hosted final qualification is running. These are source/application controls, not published-image or emitter acceptance.
The canonical receiver #918 and non-emitting retention repair #920 are merged at f8103d194e8fb0e09ef6f2f73568c3609e013277, whole 90123077f6556ad2f2aa3218a3066cf61765625d. Seventeen applicable final-candidate checks pass, with six explicit skips. The combined patch is published as 2.12.2 at the final source above. Version 2.12.1 was never published.
The historical independent-main runs 40066693736 (f8103d19) and 40060294140 (a93e97bf) remain failed. Their recovered JUnit artifacts now establish the same failing large-terminal-message test and PostgreSQL SQLSTATE 22P05 JSON NUL parsing fixed by #921. Recovered original evidence. Fresh main e6b9ad81, run 40071278691, has passed every database shard, quality and all five Laravel upgrades. Coverage, Sonar and the final aggregate passed, followed by normal2.12.2 publication and the installed driver acceptance linked above.
The receiver preserves exact large/NUL failure messages in additive canonical storage while keeping diagnostic reads bounded, hides encoding in the current model, needs no backfill and retains data on rollback. Metadata-lock acquisition is bounded; MySQL refuses a copying alteration. Receiver controls execute thirty checks / 224 assertions, with two explicit network-only skips. The accompanying retention repair's final runtime executes 96 SQLite checks / 584 assertions (32 network-only skips), and 35 checks / 517 assertions on each PostgreSQL and MySQL (two SQLite-only skips). These source results do not qualify an unpublished artifact.
Laravel 11's actual TokenGuard backport floor is 11.56.1. Legacy upgrade/drain-only limits and every-maintained-major security review remain explicit. No security rule or alert is disabled.
Normal published receiver/embedded/image adoption and actual History/Waterline acceptance remain open. PHP reporting guidance is delivered in all eight languages through Website #407; documentation does not complete receiver acceptance.
Current delivery
The three service SDK reporting components are normally published and accepted:
- PHP SDK 3.5.0: SDK #216, including thirteen actual immutable-Server cases on each SQLite and MySQL.
- Rust SDK 4.10.1: SDK #143, including normal registry identity, recording controls and actual published-Server/MySQL delivery. The Rust reporting guides are live in all eight languages.
- Python SDK 2.11.0: SDK #215, normal PyPI source/RECORD identity, 37 focused installed controls and fifteen actual published-Server/MySQL controls. All original per-case budgets remain unchanged. Website #406 delivers Python constructor/failure guidance in all eight languages: all exact candidate/main checks and actual Pages pass, with sixteen canonical localized routes verified at 22:47:41 UTC.
This shared issue remains open. General retains published PHP image/embedded adoption, History/Waterline visibility and remaining documentation acceptance. The canonical receiver/MySQL TEXT overflow correction is published in the Core module; affected applications still need to adopt it. Normally published Core 2.12.0 bounds the derived timeline summary while preserving canonical data, but that alone does not widen canonical workflow_failures.message or qualify a new Server image. The separate raw 70,000-byte failure receiver finding remains open. SDK truncation is not its receiver fix. Original failed observations and expiry-loop ownership remain unchanged.
Original report and acceptance
Problem
An activity whose failure is larger than the Server's 2 MiB request limit can never report that failure. This happens, for example, when an exception message carries a large upstream response body or the stack trace is very long. The activity then reruns every 5 minutes, and the workflow never sees the error.
Reproduced on published Server 2.9.9 with Python SDK 2.6.1. One activity raised RuntimeError with a 3 MB message, and another raised a short one:
- The short failure was reported at once; the workflow caught it and completed.
- The large failure's report was rejected with
413 payload_too_large. The attempt expired after 5 minutes and the activity ran again, failing the same way. - The run stays
waiting. Its history shows onlyActivityStarted, with noActivityFailed.
Source:
- The Python client sends the whole message and stack trace unchanged. The PHP and Rust SDKs don't truncate failures either.
- The Server rejects any request body over the payload limit, and the failure fields have no length bound that a client could fit within.
The expiry loop itself is workflow#878. This issue is about the failure never being recordable.
Acceptance criteria
- A failure of any size is recorded. Message, stack trace and details are truncated to documented limits with a visible truncation marker, and the result stays within the request limit.
- The PHP, Python and Rust SDKs, workflow-task failures and embedded mode all behave the same.
- The workflow receives the failure with its type and the truncated message, and history and Waterline show it as truncated.
- Tests with a 3 MB message and with a 3 MB stack trace show a recorded failure reaching the workflow, in each SDK and against a published Server.
- The failures guide states the limits.
Activity
rmcdaniel commentedon Oct 23, 2026
General maintainer disposition: accepted product fix, with the Rust SDK owner preparing the first component and equivalent PHP, Python and embedded follow-through required before this shared issue closes.
Use bounded failure diagnostics by default: a 62 KiB UTF-8 message, 128 KiB stack trace and 64 KiB serialized details/diagnostics budget. Each limit includes its truncation marker. These are SDK defaults, not a new Server request-limit authority. Bound the complete encoded request, including JSON escaping, nested causes, identity fields and metadata, with a default complete-report cap of 1 MiB, below the published 2 MiB request limit. Respect a lower explicitly configured transport limit where the client knows one. Identity fields remain exact; an unfit required identity is an explicit reporting refusal, never silent mutation. Oversized messages, stack traces or metadata must never turn an otherwise reportable failure into an endless lease-expiry loop.
Preserve the original failure type, retry/non-retryable decision and authored retry delay. Put a visible truncation marker in the message and bounded structured diagnostics describing omitted fields and original sizes. Use existing protocol fields. A future typed transport extension needs its own compatibility qualification. Do not slice Avro or base64 bytes: details must remain a valid re-encoded neutral value, or be explicitly omitted with a truncation diagnostic. Keep application failure semantics distinct from a reporting transport failure.
Required controls include separate 3 MiB message and stack originals, combined oversized fields, escaped/Unicode text, oversized nested details and already-small controls. Check the actual encoded bytes, then record the failure through a published Server and observe the original workflow receiving its type and marked message. Activity and workflow-task failures, embedded mode and all three service SDKs remain in scope. History and Waterline must expose that information, and all eight guides must describe the qualified limits. Existing expiry-loop findings and failed observations remain separate; source-only encoding tests do not complete published-artifact delivery.
Compatibility refinement: reserve 2 KiB below the initial 64 KiB message ceiling. A real published-PHP/MySQL control showed that a full 64 KiB message plus its timeline label exceeds the 65,535-byte TEXT projection column and returns 500. The common effective message default is therefore 62 KiB including its marker, across PHP/Python/Rust. This budget does not fix or excuse the underlying Core projection defect: General owns bounding the derived timeline summary while retaining the full canonical failure payload. The original failed observation remains separate; each language still needs actual backend delivery proof.
rmcdaniel commentedon Oct 23, 2026
The actual published Server qualification exposed a separate derived-summary overflow, now tracked in Core915. The original typed failure was bounded to65536 message bytes, PHP trim retained65535, and the14-byte
Failed large:prefix plus one-byte suffix produced65550 summary bytes against MySQL TEXT65535. Both actual normal/cooperative SQL metadata summaries measure65550. Original64KiB failures remain retained. Core owns a byte-bounded derived-summary correction preserving the full authoritative failure payload/type/retry semantics.The common compatibility default is now 62KiB message content including marker, reserving2KiB below the previous64KiB ceiling; stack128KiB, combined serialized details/diagnostics/optional metadata64KiB and complete actually encoded report1MiB/lower-known limits stay unchanged. Rust draft142 implements the headroom and exact final body guards. This does not claim Core915 fixed. PHP/Python alignment and shared closure remain separate.
Public HTTP controls reproduce three failures with the normal registry4.9 crate (normal/cooperative5200138 bytes; workflow-task18874481 bytes) and pass against current source. Eight internal cap/identity/reference/local controls and twelve existing fresh-workflow regressions pass. Published Server structured3MiB message/stack/valid Avro/nested-details delivery passes; ordinary worker cold-replay qualification is still in progress. No release/tag claim yet.
rmcdaniel commentedon Oct 23, 2026
Additional receiver-side evidence from General's actual MySQL 8.0.46 / current-Core activity-bridge qualification: a constructed 70,000-byte valid UTF-8 message fails with SQLSTATE 22001 / 1406 at canonical
workflow_failures.messagebefore timeline insertion. Its full report is well below the request-body ceiling. This is a database-backed bridge control, not a new HTTP request reproduction. Migration 000105 declares this canonical field as TEXT. The separate 65,535-byte controls fit canonical storage but fail the derived summary as recorded in Core #915.The shared 62 KiB SDK default is compatibility headroom and does not complete this raw receiver contract. General retains the receiver-side correction/acceptance under this issue. The full canonical failure must remain authoritative within accepted transport limits; changing a derived display summary must not silently destroy it. Original controls and failure logs remain distinct. SDK source work continues independently without waiting for the runtime correction.
rmcdaniel commentedon Oct 23, 2026
All three service SDK reporting components are now released: PHP 3.5.0, Rust 4.10.1 and Python 2.11.0. The new Python installed-artifact receipt verifies 39 production files / 51 hashed RECORD entries, 37 focused controls and fifteen real immutable Server 2.11.1 / MySQL 8.0.46 cases, including local first/persisted/cold equality. Its complete stack is removed after the stopped audit.
Website #406 adds the Python constructor/failure contract in all eight languages, with fourteen reviewed-source/code-parity checks and all 105 current guides verified. Hosted/Pages/live acceptance is pending. This shared issue remains open for canonical receiver/embedded, History/Waterline and remaining guidance acceptance. Core 2.12.0's derived-summary fix does not complete the distinct canonical TEXT overflow.
rmcdaniel commentedon Oct 23, 2026
Python reporting guidance is now live in all eight languages: Python configuration and semantics and failure-recovery limits.
#406 merged at
7b8d5f3ba1ba80a2de53d97add852602ee8344d9. All candidate and independent main checks, direct Sonar review and Pages publication pass. At 22:47:41 UTC, all sixteen canonical routes return HTTP 200 with the actual translated headings, matching language, new anchors and limits. Fourteen translation hash/code checks pass, including Ukrainian.This completes the Python guidance portion. Receiver/embedded, History/Waterline and remaining PHP guidance are still open here. No automatic image, native, frozen or Cloud adoption follows.
rmcdaniel commentedon Oct 23, 2026
Receiver-storage decision for the remaining canonical-message overflow: keep the full failure message, including for direct callers whose report fits the transport cap, and make the upgrade additive.
A direct TEXT-to-LONGTEXT alteration is unsuitable for an automatic rolling upgrade: MySQL's documented type-change operation uses COPY and does not permit concurrent DML. That would conflict with the accepted live-upgrade requirement in Workflow #871. The SDK's smaller default and Core 2.12's derived-summary bound remain separate fixes.
I am preparing the following Core correction, not claiming it implemented or accepted yet:
workflow_failures.message_fullLONGTEXT column. Existing rows need no backfill. On MySQL/MariaDB require the non-copying instant-add path and bound metadata-lock acquisition; unsupported layouts must fail clearly rather than silently fall back to a blocking table rewrite.messagevalue, history, replay payloads and exports exact. For a message that exceeds the old physical TEXT capacity, retain the complete value inmessage_fulland a visibly labelled, byte-bounded compatibility preview in the old column. Ordinary messages keep their existing storage. Hide the physical overflow field from public model serialization.This is a PHP/Core storage proposal under this issue. Native catalog/migration treatment needs its own exact-source qualification; frozen actors and existing schemas are not changed by this note. The full shared issue stays open through the remaining embedded, History/Waterline and live-guide acceptance.
rmcdaniel commentedon Oct 27, 2026
Release-state correction (2026-10-27): immutable Server 2.11.5 source 721b2831975cc898cc39d7528c5ba7e8441fe744 contains Server #733 merge 922259fcbc855407961a014c697db21915c28f04 and locks Workflow 2.15.1 at 07e0d0e1a1185bd0a3bbd4e3fb56511cc57ca9bc. That exact Workflow commit descends from the canonical receiver release source ae18295693273907d0f750b479eb9b8a593e5803. Protected release run https://github.com/durable-workflow/server/actions/runs/40478372197 completed successfully. The old Server 2.11.3 / #450 publisher dependency is therefore superseded for this delivery; no Security review, credential action, or operator action is pending before acceptance on 2.11.5. This shared issue remains open for its actual consumer contract: oversized message and stack-trace reporting through the published SDKs and embedded path, authoritative workflow/history and Waterline visibility, truncation disclosure, and rollback behavior against the exact published Server artifact.