Skip to content

fix: make direct workflow retry atomic - #3086

Merged
chubes4 merged 1 commit into
mainfrom
fix-3085-atomic-direct-retry
Aug 9, 2026
Merged

fix: make direct workflow retry atomic#3086
chubes4 merged 1 commit into
mainfrom
fix-3085-atomic-direct-retry

Conversation

@chubes4

@chubes4 chubes4 commented Aug 9, 2026

Copy link
Copy Markdown
Member

Summary

  • route missing-action direct retries through the existing transactional recovery primitive so the original job row is locked, fenced, and advanced exactly once
  • require a durable Action Scheduler receipt before normal enqueue or transactional recovery can report success
  • reject unsafe replay after operation effects begin and add coverage for rollback, duplicate prevention, terminal accounting, and truthful results

Fixes #3085.

Root cause

RetryJobAbility handled a processing direct job with no live action by terminalizing and reopening it before calling the general enqueuer. That bypassed the newer atomic missing-action recovery path. Separately, DirectJobEnqueuer and the transactional recovery primitive treated any positive as_schedule_single_action() return as success without confirming that the action row was durably queryable.

The retry path itself reuses the original job ID. The two fresh production IDs were downstream direct submissions emitted after the retried workflow resumed, not replacement rows created by jobs retry; each could be left pathless because positive scheduler IDs were accepted without durable receipt confirmation. The command then reported success based on the original enqueue result before scheduler ownership was proven end to end.

Atomicity and exactly-once behavior

For a processing direct operation whose recorded action is missing and whose effects have not begun, manual retry now calls commit_missing_direct_operation_requeue(). That method locks the original row, schedules a unique next-generation action inside the same database transaction, verifies the action receipt by primary key, updates the fenced operation owner, and commits. Scheduler failure, an unusable receipt, ownership drift, or commit failure rolls back without creating or reopening another processing job.

The generation/token fence prevents concurrent retries from scheduling duplicate owner generations. If operation_effects_begun_at is present, retry fails instead of replaying potentially completed side effects. Existing failed-job retry behavior remains unchanged.

Authorization and compatibility

Authorization and ownership checks remain in RetryJobAbility before any mutation. Anonymous CLI access remains denied. The Action Scheduler contract is tightened only for success reporting: callers now receive a retryable failure when a returned action ID has no durable receipt. The optional receipt callback added to DirectJobEnqueuer preserves existing constructor call compatibility.

Verification

  • php tests/direct-job-generation-smoke.php passed: 15 assertions
  • PHP syntax checks passed for all five modified files
  • focused vendor/bin/phpcs passed
  • homeboy review lint data-machine --changed-only --summary --placement=local passed with zero findings, run ccbc845a-b5e8-42cd-9781-63f1204dc161
  • clean PR-style homeboy review --summary --placement=local --changed-since=origin/main data-machine: audit passed with no introduced findings; lint passed with zero findings; test provisioning failed before executing tests because the managed wordpress-database MySQL 8.4 Docker service could not become ready, run be54de2f-6e0e-4f22-8eb7-4e8070babb99
  • focused Homeboy PHPUnit attempt failed for the same infrastructure reason with zero tests executed; captured error was RuntimeServiceProvisionError: Managed runtime service failed: wordpress-database
  • git diff --check origin/main...HEAD passed

Existing historical jobs

After this ships, reconcile 649593, 649599, and 649600 one at a time. First rerun liveness and confirm the recorded action is still missing, no current-generation scheduler path exists, and operation_effects_begun=false. Then run the authorized retry for only that job and immediately verify that the same job ID is pending with a new generation and a durable scheduler action before proceeding to the next. If effects have begun, do not retry; use stuck-job recovery to terminalize and account for it instead. A failed retry is now safe to inspect because it cannot leave a fresh processing orphan.

@chubes4
chubes4 merged commit 33187e6 into main Aug 9, 2026
12 of 14 checks passed
@chubes4
chubes4 deleted the fix-3085-atomic-direct-retry branch August 9, 2026 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: direct workflow retry creates fresh jobs without scheduler paths

1 participant