Payments 3/7: Failure Taxonomy, Retries, Partial Outcomes & Payment Reconciliation Engine

[michaelsimeon001]

Summary

• Adds a scheduled reconciliation job (ReconciliationService, @Cron every 5 minutes) that treats provider truth as authoritative: re-verifies AWAITING_CONFIRMATION payments past a due threshold directly against the provider, reusing Payments 2/7: Real-Time Confirmation Pipeline & the App/Provider State Boundary #1571's verifyByReference call and PaymentConfirmationService.apply's idempotent path (a payment resolved here can never be double-applied, and re-running the job is always safe).
• Per-payment exponential backoff (PAYMENT_RECONCILE_BACKOFF_*) so a still-processing payment (e.g. a bank-side hold) gets polled less and less often over time instead of on every cron tick.
• A new MANUAL_REVIEW status: payments unresolved past PAYMENT_MANUAL_REVIEW_AFTER_HOURS escalate to it — but only on a reconciliation pass where the provider was actually reachable. Payment#providerErrorStreak tracks consecutive provider-unreachable attempts separately from the age-based threshold, specifically so a single provider outage across a batch never mass-flags every in-flight payment (tested).
• A failure taxonomy as enum data (PaymentFailureReason: DECLINED, EXPIRED, PROVIDER_ERROR, ABANDONED) recorded on Payment#failureReason alongside a terminal status, plus new MANUAL_REVIEW/DISPUTED/VOIDED values wired into Payments 1/7: Payment Domain Model, Initiation Flow & Idempotent Transaction Lifecycle #1570's state-machine transition guard.
• An expiry sweep as part of the same job: INITIATED past its TTL → EXPIRED/ABANDONED (never even reached the provider), AWAITING_CONFIRMATION past its TTL → EXPIRED/EXPIRED (did start, but nothing confirmed it in time).
• Admin recovery endpoints (PaymentsAdminController, role-restricted to ADMIN): manual-review queue list, metrics (queue depth + alert status, logged as a WARN when the threshold is exceeded), force-reconcile-now, resolve-manually (reason required, audited via Payment#manualReviewReason), void.
• A refund sub-ledger (Refund entity / payment_refunds table) supporting multiple partial refunds per payment — "amount refunded" is always SUM(refunds.amount), never a single boolean. RefundsService.requestRefund locks the payment row (pessimistic_write) for the duration of the check-and-insert, so two refund requests that would together exceed the captured amount can never both succeed — the loser gets a 409, not a corrupted ledger (tested).
• A shared, independently-tested retryWithBackoff utility (exponential backoff + jitter, capped attempts, optional isRetryable predicate to fail fast on terminal errors) used by the refund flow's best-effort provider-side execution call.
• Failure taxonomy and the reconciliation/escalation model documented in backend/src/payments/README.md, per the issue's definition of done.

Test plan

• npm run build
• npm run lint (scoped to changed files — 0 errors; the full-repo baseline has pre-existing CRLF noise on untouched files, confirmed unrelated to this change)
• npm run test — all 9 suites / 189 tests pass, including new specs covering all 4 acceptance criteria:
  • reconciliation.service.spec.ts — resolves a simulated "webhook never arrives" scenario within its due window; excludes payments still in their initial grace window or backoff window; does not escalate to MANUAL_REVIEW on a single simulated provider outage but does escalate once the provider is reachable again and the payment is still stuck; resets the provider-error streak on successful contact; safe to run twice with no duplicate apply() calls (idempotency); expiry sweep with correct failure reasons; all admin recovery actions.
  • refunds.service.spec.ts — partial/full refund booking, multi-partial accumulation into REFUNDED, and the double-refund race test (second concurrent-equivalent request atomically rejected via the row-lock re-read).
  • retry-with-backoff.spec.ts — retry/success, exhaustion, fail-fast on non-retryable errors, exponential growth + jitter bounds, delay capping.
  • payment-state-machine.spec.ts updated for the new transitions.

Depends on and builds directly on #1570 and #1571 (already merged).

Closes #1572

[vercel]

@mftee is attempting to deploy a commit to the naijabuz's projects Team on Vercel.

A member of the Team first needs to authorize it.

[yusuftomilola]

Reviewed the failure taxonomy, retries, partial outcomes & reconciliation engine PR (closes #1572, building on already-merged #1570/#1571).

• Reusing #1571's verifyByReference and PaymentConfirmationService.apply's idempotent path for reconciliation, rather than inventing a parallel resolution mechanism, means a payment resolved by the cron job goes through the exact same terminal-status/duplicate/conflict guarantees as the webhook and verify-on-return paths already do. Genuinely safe to re-run, by construction, not just by testing.
• The MANUAL_REVIEW escalation logic is the standout piece here: separating providerErrorStreak (consecutive provider-unreachable attempts) from the age-based due threshold, and only escalating on a pass where the provider was actually reachable, is exactly the right fix for the failure mode most reconciliation systems get wrong — a single provider outage mass-flagging every in-flight payment for human review. Good that this specific scenario is explicitly tested rather than just asserted in the PR description.
• Per-payment exponential backoff so a still-processing payment gets polled less often over time (instead of every 5-minute tick hitting the provider) is a reasonable, considerate default against a real payment provider.
• The refund sub-ledger design — "amount refunded" always computed as SUM(refunds.amount) rather than a boolean flag — is the correct model for supporting multiple partial refunds, and locking the payment row (pessimistic_write) for the duration of the check-and-insert closes the actual race (two concurrent refund requests that would together overshoot the captured amount) at the DB layer rather than relying on an app-level check that a second request could still slip past. The 409-on-loser-request behavior is verified with a real concurrent-equivalent test, not just asserted.
• The expiry sweep correctly distinguishes "never even reached the provider" (INITIATED → ABANDONED) from "did start but nothing confirmed it in time" (AWAITING_CONFIRMATION → EXPIRED) — that distinction is genuinely useful for whoever triages the manual-review queue later.
• retryWithBackoff as a shared, independently-tested utility (jitter, capped attempts, fail-fast on non-retryable errors) rather than one-off retry logic embedded in the refund flow is good factoring, especially since it's already positioned to be reused by future payment-adjacent work.
• Admin recovery surface (manual-review queue, metrics with WARN-on-threshold, force-reconcile, audited resolve-manually requiring a reason, void) gives operators real tools instead of leaving stuck payments as a black box only fixable via direct DB access.
• Test coverage matches the risk surface precisely across all three subsystems — reconciliation (including the outage-vs-escalation distinction and idempotent double-run safety), refunds (the actual concurrency race), and retry/backoff (growth, jitter bounds, capping, fail-fast).

CI is green across Backend, Frontend, and Frontend E2E. Vercel's FAILURE is the usual unauthorized deployment integration link, unrelated to the code.

Approving — thorough, carefully-reasoned closure of the payment reconciliation gap.