Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
1fc8f3d
Bound failure reports while preserving original claim semantics
rmcdaniel Oct 23, 2026
09f4691
Qualify published failure delivery and refuse unencodable identities
rmcdaniel Oct 23, 2026
ed0b2fe
Bound cooperative failure IPC independently of payload transfers
rmcdaniel Oct 23, 2026
dc6e83e
Extract failure budget stages and cover isolated callback delivery
rmcdaniel Oct 23, 2026
f8faf6e
Declare the qualification activity context explicitly
rmcdaniel Oct 23, 2026
907a466
Reserve summary headroom and freeze local failure observations before…
rmcdaniel Oct 23, 2026
afbd271
Declare local activity capability in failure-delivery fixture
rmcdaniel Oct 23, 2026
f1904b6
Drive legitimate worker turns throughout failure-delivery qualification
rmcdaniel Oct 23, 2026
b859421
Distinguish transmitted callback traces from published history visibi…
rmcdaniel Oct 23, 2026
36e459e
Maintain original fixture registrations during extended failure controls
rmcdaniel Oct 23, 2026
b080bd8
Keep normal payload budgets and plan uploads for mixed failure reports
rmcdaniel Oct 23, 2026
54dfff6
Carry local size provenance separately from application error text
rmcdaniel Oct 23, 2026
95f5304
Bound failure metadata before planning business payload uploads
rmcdaniel Oct 23, 2026
97fc0d7
Qualify lower-budget diagnostic omission through published backends
rmcdaniel Oct 23, 2026
3ad8dbd
Preserve omission provenance and add an immutable local-failure corpus
rmcdaniel Oct 23, 2026
f35457f
Remove the redundant nullable planner guard
rmcdaniel Oct 23, 2026
88dcd07
Prepare compatible SDK 3.5.0 failure-report release
rmcdaniel Oct 23, 2026
4efb8d6
Extract bounded upload stages and simplify failure classifiers
rmcdaniel Oct 23, 2026
895d292
Align quickstart package identities with SDK 3.5.0
rmcdaniel Oct 23, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,29 @@ project follows [Semantic Versioning](https://semver.org/).

## [Unreleased]

## [3.5.0] - 2026-10-23

### Added

- `Client::withFailureReportLimit()` narrows the complete encoded failure-report
budget. Transports may expose a lower known limit. Activity and workflow-task
reporting accept optional stack traces, and activity reporting accepts
diagnostics, after all existing arguments.

### Fixed

- Bound failure messages to 62 KiB, stacks to 128 KiB, combined details and
diagnostics to 64 KiB, and complete encoded reports to 1 MiB by default.
Markers and JSON escaping count toward these limits. Oversized details remain
whole valid Avro values or are omitted with original-size diagnostics.
- Preserve original failure types, retry decisions, authored delays and claim
identities. Local durable failures normalize before workflow code observes
them, so initial execution, persistence and cold replay agree. Cooperative
failure IPC has its own diagnostic budget, separate from business payloads.
- Keep ordinary successful payload budgets unchanged. Mixed failure/result
completions use the existing upload planner without changing or slicing the
business payload. Unfit required report content refuses explicitly.

## [3.4.1] - 2026-10-23

### Fixed
Expand Down
2 changes: 1 addition & 1 deletion composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@
}
},
"durable-workflow": {
"product-train": "3.4.1",
"product-train": "3.5.0",
"supported-server-versions": "2.5.0",
"worker-protocol-version": "1.20",
"control-plane-version": "2",
Expand Down
56 changes: 56 additions & 0 deletions docs/portal/build/failures-retries.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,3 +107,59 @@ A client-side wait timeout is not proof that the workflow failed; describe the h
Do not paper over `NonDeterministicWorkflow` with process retries. Compare the deployed build with the history's worker routing, restore the old command sequence, and roll forward with a replay-compatible change. Keep a replay corpus for long-lived workflow histories in [testing](/build/testing/).

<div class="outcome"><strong>Retry question</strong><p>Before adding a retry, identify the durable request ID, the side effect's idempotency boundary, and what evidence proves the prior attempt did not already succeed.</p></div>

## Bound failure diagnostics

Failure reporting keeps diagnostic content within finite defaults:

| Content | Default budget |
| --- | --- |
| UTF-8 message, including its marker | 62 KiB |
| Stack trace, including diagnostic markers | 128 KiB |
| Serialized details and diagnostics together | 64 KiB |
| Complete encoded failure request, including JSON escaping and identity | 1 MiB |

The message budget reserves 2 KiB below the original 64 KiB ceiling for
published runtime summaries. It does not repair the separate runtime summary
byte-bound defect tracked in shared issue 225.

Oversized content produces a visible `[truncated failure report; ...]` message
marker and bounded diagnostics with original sizes and omitted fields. The SDK
retains a complete valid Avro details envelope or explicitly omits it. It never
slices Avro or base64 bytes. Small reports keep their message and details.

The original failure type, retry decision, authored retry delays and claimed
attempt identity remain unchanged. Diagnostic limits are separate from business
payload and worker request timeout budgets. Cooperative callbacks also bound
failure diagnostics before sending them to their owning worker.

For a transport with a smaller request budget, narrow the client limit:

```php
$client = $client->withFailureReportLimit(128 * 1024);
```

An injected transport can expose a lower known body budget through
`RequestSizeLimitTransport`. The lower limit applies after markers and JSON
escaping are included. If required identity or fixed request content cannot fit,
`FailureReportRefused` reports `failure_report_refused` locally. Treat this as a
reporting refusal requiring configuration or identity repair, rather than a new
application failure or permission to publish under a different attempt.

Successful local records keep their normal business-payload budget. When a
completion includes both failure diagnostics and a large business result, the
existing runtime upload protocol preserves the complete Avro payload while
keeping the final failure request bounded. If fixed content cannot fit and the
namespace cannot store that payload, reporting refuses explicitly. No partial
business payload is published.

Local callback failures normalize before workflow code branches on them. The
first execution, persisted local record and fresh replay use the same bounded
message, type and retry decision.

Activity failures use existing diagnostic fields. Workflow-task failures use
the existing stack string for structured truncation information. The published
Server qualification records task failures and grants a legitimate retry turn,
but that runtime does not append a workflow-task failure event to workflow
history or persist its stack diagnostics. That visibility work remains tracked
in [shared issue 225](https://github.com/durable-workflow/.github/issues/225).
6 changes: 3 additions & 3 deletions docs/quickstart-contract.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
"schema_version": 2,
"package": {
"name": "durable-workflow/sdk",
"published_version": "3.4.1",
"composer_requirement": "3.4.1",
"onboarding_requirement": "^3.4"
"published_version": "3.5.0",
"composer_requirement": "3.5.0",
"onboarding_requirement": "^3.5"
},
"runtime_targets": {
"server": {
Expand Down
68 changes: 47 additions & 21 deletions scripts/ci/run-replay-regression-fixture.php
Original file line number Diff line number Diff line change
Expand Up @@ -377,6 +377,7 @@ public function run(WorkflowContext $context): array
final class ReplayRegressionConsumer
{
private const FIXTURE_SCHEMA = 'durable-workflow.replay-regression/v1';
private const BOUNDED_LOCAL_FAILURE = 'golden.bounded-local-failure';

public static function executeFile(string $path): void
{
Expand Down Expand Up @@ -554,34 +555,15 @@ private static function executeLayout(
);
}
$localActivityInvocations = 0;
if (in_array($workflowType, ['golden.local-activity-terminal-failure', 'golden.local-activity-recovered'], true)) {
$worker->registerActivity(
$workflowType === 'golden.local-activity-recovered' ? 'golden.local-receipt' : 'golden.local-failure',
static function (ActivityContext $_context) use (&$localActivityInvocations): never {
++$localActivityInvocations;

throw new RuntimeException('Cold replay repeated the recorded local activity side effect.');
},
);
}
if ($workflowType === 'golden.activity-completion-backend-loss') {
$worker->registerActivity(
'golden.backend-recovery',
static function (ActivityContext $_context) use (&$localActivityInvocations): string {
++$localActivityInvocations;

return 'activity-after-backend-recovery';
},
);
}
self::registerLocalActivities($worker, $workflowType, $localActivityInvocations);
foreach ($tasks as $_task) {
$worker->tick(0);
gc_collect_cycles();
}
if ($workflowType === 'golden.workflow-lease-recovery') {
$worker->tick(0);
}
if ($localActivityInvocations !== ($workflowType === 'golden.activity-completion-backend-loss' ? 1 : 0)) {
if ($localActivityInvocations !== (in_array($workflowType, ['golden.activity-completion-backend-loss', self::BOUNDED_LOCAL_FAILURE], true) ? 1 : 0)) {
throw new RuntimeException("{$identity} repeated a recorded local activity during cold replay.");
}

Expand Down Expand Up @@ -640,6 +622,40 @@ private static function assertExpected(string $identity, array $fixture, array $
self::assertMatches($expected, $observed, "{$identity}.expected");
}

private static function registerLocalActivities(Worker $worker, string $workflowType, int &$localActivityInvocations): void
{
if (in_array($workflowType, ['golden.local-activity-terminal-failure', 'golden.local-activity-recovered'], true)) {
$worker->registerActivity(
$workflowType === 'golden.local-activity-recovered' ? 'golden.local-receipt' : 'golden.local-failure',
static function (ActivityContext $_context) use (&$localActivityInvocations): never {
++$localActivityInvocations;

throw new RuntimeException('Cold replay repeated the recorded local activity side effect.');
},
);
}
if ($workflowType === self::BOUNDED_LOCAL_FAILURE) {
$worker->registerActivity(self::BOUNDED_LOCAL_FAILURE,
static function (ActivityContext $context) use (&$localActivityInvocations): never {
if ($context->activityType !== self::BOUNDED_LOCAL_FAILURE) {
throw new UnexpectedValueException('Unexpected bounded-local activity context.');
}
++$localActivityInvocations;
throw new RuntimeException(str_repeat('雪', 1024 * 1024));
});
}
if ($workflowType === 'golden.activity-completion-backend-loss') {
$worker->registerActivity(
'golden.backend-recovery',
static function (ActivityContext $_context) use (&$localActivityInvocations): string {
++$localActivityInvocations;

return 'activity-after-backend-recovery';
},
);
}
}

/** @return callable(WorkflowContext, mixed ...$input): mixed */
private static function workflow(string $workflowType): callable
{
Expand Down Expand Up @@ -872,6 +888,16 @@ private static function workflow(string $workflowType): callable

return ['key' => $selected->key];
},
self::BOUNDED_LOCAL_FAILURE => static function (WorkflowContext $context): array {
try {
$context->localActivity('golden.bounded-local-failure', [], ['retry_policy' => ['max_attempts' => 1]]);
return ['unexpected_success' => true];
} catch (ActivityFailed $failure) {
return ['marked' => str_contains($failure->getMessage(), '[truncated failure report;'),
'message_bytes' => strlen($failure->getMessage()), 'failure_type' => $failure->failureType,
'non_retryable' => $failure->nonRetryable];
}
},
'golden.local-activity-recovered' => static fn (WorkflowContext $context): mixed =>
$context->localActivity('golden.local-receipt'),
'golden.local-activity-terminal-failure' => static function (WorkflowContext $context): never {
Expand Down
Loading
Loading