Skip to content

fix: retry the packagist update on transient upstream failures - #148

Open
markshust wants to merge 1 commit into
developfrom
feature/packagist-notify-retry
Open

fix: retry the packagist update on transient upstream failures#148
markshust wants to merge 1 commit into
developfrom
feature/packagist-notify-retry

Conversation

@markshust

Copy link
Copy Markdown
Collaborator

Problem

Every package in the split matrix reaches the Notify Packagist step within the same second, so a release fires ~92 update-package calls at Packagist at once and reliably trips its limits.

On 0.8.5 eight of them came back HTTP 500 at the same instant (run 30208431432). The single un-retried curl turned that into eight failed jobs — core, health, lsp, hashing, http, notification, page-cache, mail-smtp.

Because the failure landed after Split and push, the tags were already correct in every split repo. The only real damage was a stale Packagist index — including marko/core, which every other package resolves against. Re-running the failed jobs was enough to fix it, which is the definition of a retry we should not be doing by hand.

Change

Wrap the update call in five attempts with jittered exponential backoff:

  • Retried: 5xx, 429, and curl's 000 transport failure.
  • Not retried: 404 still falls through to the existing create-package self-heal; 401/403 fail immediately, since no retry fixes a bad token.

The jitter matters as much as the backoff — all 92 jobs fail on the same second, so a fixed delay would just line their retries up on the same second too.

Retrying means the curl can no longer be allowed to abort the step, so the final status is now matched against the 2xx class rather than tested with -ge 400. A transport failure arrives as 000, which is numerically zero and would otherwise be waved through as success, leaving the package stale behind a green job.

Verification

Tested behaviourally against a stub Packagist rather than only by reading the diff:

Scenario Expected Result
succeeds first try exit 0 update:200
two 500s then 200 (the 0.8.5 case) exit 0 500 500 200
five 500s, retries exhausted exit 1 500 ×5
rate-limit 429 exit 0 429 200
unregistered package exit 0 404 → create → 200
bad token exit 1, no retry 401
503 then 401 exit 1 503 401
connection refused exit 1 000 after 5 attempts

That harness also turned up a smaller wart, fixed here: curl leaves the response file untouched when it cannot connect, so a failed attempt printed the previous attempt's body as its own.

tests/SplitWorkflowTest.php gains six cases covering the retry loop, the retryable classes, the jittered backoff, the preserved 404 path, the non-2xx failure check, and the response-file reset. Full composer ci is green.

Note

The workflow tests assert on the YAML's structure, matching the existing convention in that file — they pin the shape of the fix but do not execute it. The behavioural table above is what actually verifies the logic, and it was run locally rather than in CI.

🤖 Generated with Claude Code

Every package in the split matrix reaches the notify step within the same
second, so a release fires ~92 update-package calls at Packagist at once and
reliably trips its limits. On 0.8.5 eight of them came back HTTP 500 at the
same instant. The single un-retried curl turned that into eight failed jobs,
and because the failure landed after "Split and push", the tags were already
correct — the only real damage was a stale Packagist index, marko/core among
it, which every other package resolves against. Re-running the failed jobs
was enough to fix it, which is the definition of a retry we should not be
doing by hand.

Wrap the update call in five attempts with jittered exponential backoff.
Only the transient classes retry (5xx, 429, and curl's 000 transport
failure); 404 still falls through to the create-package self-heal, and
401/403 fail immediately since no retry fixes a bad token. The jitter
matters as much as the backoff: all 92 jobs fail on the same second, so a
fixed delay would just line their retries up on the same second too.

Retrying means the curl can no longer be allowed to abort the step, so the
final status is now matched against the 2xx class rather than tested with
`-ge 400` — a transport failure arrives as 000, which is numerically zero
and would otherwise be waved through as success, leaving the package stale
behind a green job.

Verified behaviourally against a stub Packagist rather than only by reading:
recovery after two 500s (the 0.8.5 case), failure once five attempts are
exhausted, 429 retry, the 404 registration path, fail-fast on 401, and
connection-refused exiting non-zero. That harness also turned up a smaller
wart now fixed here — curl leaves the response file untouched when it cannot
connect, so a failed attempt printed the previous attempt's body as its own.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@markshust markshust added bug Something isn't working ci CI/CD pipeline and automation changes labels Jul 26, 2026
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working ci CI/CD pipeline and automation changes labels Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant