Repository navigation
fix(purchase): guard the remaining post-CAS full-row saves on status - #1111
Conversation
Six saves that follow a status CAS (or a fresh read) still wrote the whole row with SavePurchaseExecution, so a status another writer set in between was put back. The reachable cases: a revoke landing between the delayed approve's CAS and its scheduled_execution_at stamp was undone and the scheduler then bought the revoked purchase; a session revoke landing after the terminal save was erased by the revocation-token rotation or the ramp-refusal note. Convert scheduleApprovedExecution, safeFail, failExistingPlanStep, rotateApprovalToken, recordRampAdvanceRefusal and clearApprovalToken to SaveExecutionIfStatus(Tx), each expecting the status its own CAS or read established. The executors' post-claim audit stamp now fails closed when its running guard is lost (the claim is provably gone): approve/run-now/direct execute return a 409-mapped error and the scheduler counts a race loss, with no provider call. Other store errors stay best-effort as before. Closes LeanerCloud#835 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: APPROVE 1faae48 Reviewed head 1faae48 (base 8caed39). Findings (none blocking):
Site audit: every remaining SavePurchaseExecution(Tx) caller on the merged tree was checked: execution.go:264/334/364 (per-account rows, first written at completion), plan_step_attach.go:306/335, handler_plans.go:746, handler_purchases.go:2578/3249 (inserts), store_postgres_ramp_target_met.go:60 (insert) and handler_purchases.go:3101 (above). None follows a status CAS, which matches the body. The six converted sites each guard on the status their own CAS or read established. rotateApprovalToken runs only after a successful terminal save, so exec.Status equals the stored status. stampClaimedAudit fails closed only on ErrExecutionNotInExpectedStatus or ErrNotFound (SaveExecutionIfStatusTx returns exactly those). executeFailureError maps nil row plus either error to 409. fireOneDue reports a race loss. A lost guard in failExistingPlanStep rolls back the suppression delete in the same tx. SaveExecutionIfStatusTx locks the row FOR UPDATE, so the check and the write are atomic. Attacked (mutations in a scratch copy, each restored, the tree diffed clean):
Tests (merged tree, AWS config /dev/null): go build ok. go vet -tags integration on purchase, api and config: ok. Unit tests for purchase, api, mocks and config: 5621 passed. |
|
Independent review 2 (claude-fleet, cold): APPROVE 1faae48 Findings (none blocking):
Against #835:
Attacked:
Verified locally, on
Reviewed by an independent agent with no author context (second reviewer). Review only: nothing was pushed, and the GitHub review state is unchanged. |
Resolve conflicts with LeanerCloud#1095 and LeanerCloud#1176: - internal/purchase/manager.go (safeFail): keep main's strandedRecoveryError(exec), which has the same text, and this branch's stamp guarded on "failed". - internal/purchase/notifications.go (failExistingPlanStep): keep main's single-transaction failPlanStepTx (pinned CAS, stamp and suppression release commit together) and guard its save on "failed" with SaveExecutionIfStatusTx. The CAS's row lock already keeps other writers out, so the guard is defense in depth. - Drop the failExistingPlanStep subtest of TestPostCASSave_SimulatedWriterSurvives. Its replayed writer can no longer land between the CAS and the save, and LeanerCloud#1176's plan_step_fail_atomic_integration_test.go covers this site's concurrent writers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merged origin/main ( Conflicts Before resolving, I read the main commits that touched each file: #1095
Review findings (both reviews APPROVE on
Verification on the merged tree (
No frontend, migration or Terraform changes. The PR stays a draft. |
Brings in main's CI fixes (LeanerCloud#1236, LeanerCloud#1252) and CI changes since, so this PR gets a fresh CI run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: APPROVE e8a9854 Delta re-review after the refresh onto main (previously approved at 70e6d09). What changed: e8a9854 is a plain merge of 503eb2e (origin/main) into the old head. Interactions with main since the old base (a14dc8f..origin/main):
Results at e8a9854 (git archive export):
Findings: none. |
|
Independent review 2 (claude-fleet, cold), re-review: APPROVE e8a9854 Delta: 1faae48..e8a9854 (first-parent). It contains merge 70e6d09 (main a14dc8f, a non-empty remerge-diff) and merge e8a9854 (main 503eb2e, an empty remerge-diff). The head merges cleanly with current main 2ded648. Hand resolution in 70e6d09, attacked hunk by hunk:
Net production diff of (main + PR) vs main, compared with 8caed39..1faae48: identical except the relocated failPlanStepTx line and its comment. All six guarded post-CAS saves survive:
stampClaimedAudit (claim.go:81) is still the only stamp at approvals.go:443 and scheduled_fire.go:98. No new unguarded write from main: the remaining SavePurchaseExecution(Tx) sites are the same 9 in the same 9 functions at 1faae48 and in the merged tree. They are creates, retry successors, the executor's own terminal/best-effort paths, and plan-step attach. None of main's new files (stranded_redrive.go, failure_notify.go, ri_exchange_direct*) calls a full-row execution save. Findings: none. Attacked: the remerge hunks above; a call-site census of guarded and unguarded saves across 1faae48, e8a9854 and the merged tree; probes M1 to M9 re-run on the merged export, each restored and cmp-checked against the merged tree blob. All KILLED:
Verified locally (go1.26.10, merged export, heavy lock):
Reviewed by an independent agent with no author context (second reviewer). Review only: nothing was pushed, and the GitHub review state is unchanged. |
|
Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE Blocking:
Non-blocking:
Attacked:
Body-stale:
Reviewer: rA1. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths. |
|
Local verification at
|
|
Marked ready and merged by the owner's agent after 2 independent reviews and local verification (see the review and verification comments above, all at |
|
@IAmJSD heads-up: main is red at 6c84280. The combination of #1245 and #1111 broke |
… race test (#1418) main is red since #1111 merged (6c84280): Integration Tests (purchase) panics in TestPostCASSave_ConcurrentStatusChangeSurvives/safeFail stamp after a late recovery. #1245, merged an hour earlier, made safeFail send SendPurchaseFailedNotification as a terminal-failure writer, and #1111's new test was written before that, so its email mock had no expectation for the call. The panic also killed the rest of the purchase integration package. Each PR was green on its own. Register the existing failedEmailRecorder in that subtest only and assert the race never alerts twice for one row. A fixture-wide .Maybe() stub would shadow the recorder in #1245's own TestRecoverStrandedApprovals_ClaimedRowTurnedUnsafeIsFailedNotBought, because testify matches the first registered expectation. The test's row assertions are unchanged. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
Update: #1418 fixed the red test on main (26004a6), thanks @IAmJSD. #1417 stays open as the follow-up for the underlying false failure email: in that race, |
Pick up LeanerCloud#1111 and LeanerCloud#1418 so CI runs against the fixed main (LeanerCloud#1418 fixes main's red purchase integration test). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pick up LeanerCloud#1111 and LeanerCloud#1418 so CI runs against the fixed main (LeanerCloud#1418 fixes main's red purchase integration test). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pick up LeanerCloud#1111 and LeanerCloud#1418, which register the failure-email recorder in the post-CAS safeFail race test; it must keep passing with the outcome-unknown email. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Root cause
After #829 (#833), the executors claim
runningby CAS and guard their own audit stamp and terminal save onrunning. Six other saves that run after a status CAS (or right after a fresh read) still wrote the whole row withSavePurchaseExecution(Tx), an upsert that setsstatusfrom an in-memory copy. If another writer changed the status between the CAS and the save, the save put the old status back.Confirmed on current main (
8caed39c). Nothing merged covers it:gh pr list --state merged --search 835is empty, andgit log --grep '#835'only matches1c632e6e, an unrelated frontend PR whose own number was #835.Two of these races are reachable today:
scheduleApprovedExecution,internal/api/handler_purchases.go). The approve CAS moves the rowpending|notified -> scheduled, then saves thescheduled_execution_at/approved_bystamp. If a revoke (CancelScheduledExecutionAtomic) lands between the two, the stamp writesscheduledback overcanceled. The row now has ascheduled_execution_at, so the scheduler buys a purchase the user revoked for free. The real-Postgres replay below shows exactly this before the fix.rotateApprovalToken,recordRampAdvanceRefusal). Both write the executor's in-memorycompletedrow after the terminal save. ArevokeViaSessionCAS (completed -> revocation_requested) landing first was erased.The other two are not reachable today. The guard is defense in depth for them:
safeFail(manager.go, stranded-approval sweep). Its stamp could turn a row that the late original run had already recovered tocompleted(viaRecoverReapedExecution) back intofailed, which invites a Retry that buys a second commitment. This only applies to legacyapprovedrows.failExistingPlanStep(notifications.go) andclearApprovalToken(approvals.go, after cancel). No production writer moves afailedplan-step row or acanceledrow today. Since fix(purchase): fail a plan step's root row atomically with its reason and suppression release #1176 (now on main),failExistingPlanStepruns its CAS, save and suppression release in one transaction, so the CAS's row lock already keeps other writers out until commit.Fix
Each site now uses
SaveExecutionIfStatus(Tx), which locks the rowFOR UPDATE, checks the status and writes in one transaction. Each one expects the status its own CAS or read established:scheduleApprovedExecutionscheduledapproveWithDelayreturns 409 (same as a lost CAS)safeFailstampfailedfailExistingPlanStepfailedrotateApprovalTokenrecordRampAdvanceRefusalclearApprovalTokencanceledThe issue's "Also" item: a failed audit stamp after the claim. I chose fail-closed on a lost claim and best-effort otherwise. The new helper is
stampClaimedAuditinclaim.go, used bytransitionApproveAndExecuteandfireOneDue:running-guarded stamp returnsErrExecutionNotInExpectedStatusorErrNotFound, the claim is provably gone. The run stops before any provider call and returnserrClaimLostBeforePurchase, which wraps the store error.executeFailureErrormaps it to 409.fireOneDuereports a race loss.running, and missing attribution should not block a purchase on a transient DB error.Merge with main
Main has been merged into this branch, with no rebase. Two files conflicted:
internal/purchase/manager.go(safeFail): fix(purchase): re-drive the claimed row, not the pre-claim snapshot #1095 moved the recovery message intostrandedRecoveryError(exec), which produces the same text. The merge keeps that helper and this PR's save guarded onfailed.internal/purchase/notifications.go(failExistingPlanStep): fix(purchase): fail a plan step's root row atomically with its reason and suppression release #1176 moved the pinned CAS, the stamp and the suppression release into one transaction (failPlanStepTx). The merge keeps that structure and changes only its save toSaveExecutionIfStatusTx(..., failed). The CAS's row lock already makes the guard redundant there, so it is defense in depth. ThefailExistingPlanStepsubtest ofTestPostCASSave_SimulatedWriterSurviveswas removed, because its replayed writer can no longer land between the CAS and the save. fix(purchase): fail a plan step's root row atomically with its reason and suppression release #1176'splan_step_fail_atomic_integration_test.gocovers this site's concurrent writers.Verification
All commands were run with
AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null, because this machine's~/.aws/configdefault profile is invalid. Nothing here needs AWS config, and nothing reaches a cloud API.Unit tests (mock store and mock provider). These pin the guarded store methods themselves, because the mock falls back to
SavePurchaseExecutionwhen no guarded expectation is set:internal/purchase/post_cas_guarded_save_test.go:safeFail,failExistingPlanStep(guard, plus "a lost guard releases nothing"),recordRampAdvanceRefusal, approve andfireOneDuewith a lost stamp (no provider call), andstampClaimedAudit(store error stays best-effort; a missing row counts as a lost claim).TestApprove_ClaimsRunningAndStampsGuarded(three guarded writes, no unguarded one),TestClearApprovalToken_PersistsEmpty,TestMintRevocationToken_PersistsFreshToken,TestHandler_scheduleApprovedExecution_HappyPath.TestApproveWithDelay_StampLostToRevokeMaps409.Integration tests (
integrationtag, real PostgreSQL via testcontainers, real store, Manager and handler, counting fake provider). Note that the repo's container helper runspostgres:16-alpine, not 17. In each test the concurrent write is replayed deterministically between the CAS (or read) and the save, using a wrapper store that runs it once before the first execution save of any kind, guarded or not. The replay is therefore identical before and after the fix.internal/api/handler_purchases_schedule_revoke_race_integration_test.go,TestApproveWithDelay_Postgres_RevokeBeforeStampStaysCanceled: the revoke is the exact store workrevokeScheduledExecutiondoes (CancelScheduledExecutionAtomicplus the suppression delete, in one transaction).internal/purchase/post_cas_guarded_save_integration_test.go:TestPostCASSave_ConcurrentStatusChangeSurvives, with production writers:safeFailvs. a lateRecoverReapedExecutionrevokeViaSession's CASrevokeViaSession's CASTestPostCASSave_ClaimLostBeforeStampBuysNothing: approve and scheduled fire, with the reaper's CAS landing right after the claim. This ordering is not reachable in production today, because the reaper waits 10 minutes.TestPostCASSave_SimulatedWriterSurvives:clearApprovalTokenwith a simulated writer (a status CAS toexpired), labeled as such in the test. ThefailExistingPlanStepsubtest was removed in the main merge (see above).Fail before / pass after (run at
1faae48d, before the main merge). I reverted the six production files toorigin/main, left the tests unchanged, and re-ran them. One temporary shim, not committed, provided the pre-fixstampClaimedAudit/errClaimLostBeforePurchaseso the helper's direct tests compile:Commands
Out of scope (noted, not changed)
finalizePurchaseStatus(handler_purchases.go) writes a just-insertedpendingrow tofailedwhen the approval email fails. It follows an insert, not a status CAS, so it is not in this issue's class. Its narrow window (an approve landing while the email is being sent) could be closed the same way in a follow-up, by guarding onpending.SavePurchaseExecution(Tx)callers are inserts of brand-new rows: plan step creation, retry successors, submit, plan-step attach/fail, the target-met row, and the per-account fan-out rows, which only their own goroutine writes. None of them follows a CAS.scheduledwith noscheduled_execution_at, which the scheduler never fires. This is pre-existing and unchanged here.Closes #835
Maintainers: please mirror the issue's triage labels on this PR:
priority/p2,severity/medium,triaged,urgency/this-quarter,impact/few,effort/m,type/bug.🤖 Generated with Claude Code