Skip to content

pool: a settled archive intent keeps a replicated claim's row - #426

Merged
CMGS merged 6 commits into
mainfrom
fix/archive-intent-replica-row
Nov 3, 2026
Merged

CMGS merged 6 commits into
mainfrom
fix/archive-intent-replica-row

Conversation

@CMGS

@CMGS CMGS commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

One registry fault can destroy a hibernated replicated claim that nobody adopted or released. This is on main today.

settleIntent took any row this node owns for the row its archive wrote, and deleted it. For a replicated claim that row is the replica's. Two sequences follow:

  • No crash. The Replace of an archive fails, or its reply is lost, so ArchivePending stays. The next archive pass, 5 s later, settles the intent and deletes the replica row. writeArchiveRow then does Replace at the held epoch, finds no row, and refusedReplace reads that as an adopter finishing: dropAdopted removes the claim and its VM. A PostgreSQL outage that starts during the upload is enough.
  • One crash. A restart mid-archive deletes the row and keeps the journal's epoch. The first unfence reads "row absent, epoch confirmed" as an adopter finishing, and drops the claim.

Fix. The row stays. A row naming the replica is left alone, and one the archive moved goes back to the replica. The row is never absent while the claim is held, so absent keeps its one meaning, and the claim stays adoptable from its replica while the archive retries. Deleting the row and zeroing the epoch also fixed both sequences, but left a window of absent row that a crash or failed journal write could still misread; it closed 7 of the 12 failures below, this closes 10.

Hot path: none. This runs only when an archive intent settles.

Evidence:

A fault matrix ran 1358 combinations of 11 scenarios, each durable step, 5 fault kinds and 6 continuations (heal, lease lapse with adoption, wake, release, a failing boot write, a sweep then a crash):

claim lost new violations
before 12
after 2 0

The other invariants (two serving owners, a released claim adoptable or served, a live row naming a deleted checkpoint) had no failures before or after. Checkpoint residue went from 97 lines to 95. The matrix is a temporary harness, not in this PR, with an in-memory registry whose compare-and-set arms were read against the PostgreSQL ones.

The two that remain both need a disk that keeps failing claim writes across a crash, plus a failing boot write or an hourly sweep. Both are left as they are: the disk failure is already loud, and no mechanism is added for it.

Tests in this PR, each failing without the fix:

  • TestReplicaArchiveRetriedAfterARegistryFaultKeepsTheClaim
  • TestReplicaRestartMidArchiveKeepsTheClaim

go test -race ./pool, the new tests -race -count=20, go vet and asl on both GOOS and the e2e module's tests pass.

Not run: hardware on this PR alone. The full hardware round runs on the tree with every open PR merged, and the result lands here.

Stacked on #420.

Update

  • Archive rollback keeps a replicated claim's row. A failed journal write during an archive deleted the row, and the first unfence after a restart then removed the claim, its VM and snapshot. The fault matrix's two archive-replicated claim-lost lines are gone. A hardware run on tmpfs did not reach the journal failure, so this rests on the regression test and the matrix.
  • No adoption while fenced or within 8 s of a lift, so after a registry outage every healthy owner renews before a peer can take its rows. Two nodes, PostgreSQL stopped 30 s: with this branch nothing is adopted and every claim keeps its VM; on main both healthy claims were adopted within 0.5 s of PostgreSQL returning.
  • Unfence takes its own rows in one statement and one journal write (TakeOwned). Two nodes, 1000 replicated claims, restart to lift:
PostgreSQL round trip this branch main
local 0.02-0.07 s 0.35-0.41 s
10 ms 0.07-0.12 s 10.8-10.9 s
20 ms 0.18 s never lifts: the pass outlasts the lease window every time, and all replicated claims answer 503

Conflict to expect with #424 in adoptArchived: keep the held check first, then the fence.

CMGS added 6 commits November 3, 2026 16:22
settleIntent took any own row for the archive's and deleted it. For a hibernated replicated claim that row is the replica's: after one failed or lost Replace, the next archive pass deleted it, refusedReplace read the absent row as an adopter finishing, and a claim nobody adopted or released was destroyed. A restart mid-archive did the same at the first unfence. The row now stays, and one the archive moved goes back to the replica, so it is never absent while the claim is held. Off the claim path.
The rollback deleted the row it had just moved from the replica to the archive checkpoint and zeroed the epoch, while the journal on disk still named the replica at the old epoch. After a restart the first unfence read the missing row as a finished adoption and removed the claim, its VM and its snapshot. A replicated claim now keeps the row's epoch and its archive intent, so the next settle moves the row back to the replica.
After a registry outage longer than the lease every node is fenced and every lease expired. A client polling a node made it adopt while it was fenced, or right after its own lift, before the healthy owner had renewed; the owner then confirmed, saw a foreign row and removed its running VM. Adoption now waits for the fence and for one renewal interval plus one registry query after the lift. A claim this node already holds keeps its path.
The unfence pass confirms each replicated claim with its own Take, so its length grows with the claim count times the registry round trip. TakeOwned moves the owner's live rows at their expected epochs in one statement, with Take's compare-and-set on the epoch and no lease arm, since the owner is the caller.
… write

The confirmation ran one Take and one whole-file claims commit per replicated claim, serially. Past the 12 s window the lift re-tripped and the next renewal started over, so about 1200 claims at a 10 ms registry round trip never lifted. The rows this node holds now move in one TakeOwned and their epochs land in one commit; absent, foreign, consumed or moved rows keep the per-row path.
@CMGS
CMGS force-pushed the fix/archive-intent-replica-row branch from 38f9e24 to 243a0ea Compare November 3, 2026 08:23
@CMGS
CMGS changed the base branch from review/post-417 to main November 3, 2026 08:23
@CMGS
CMGS merged commit 8a5e9cc into main Nov 3, 2026
1 check passed
@CMGS
CMGS deleted the fix/archive-intent-replica-row branch November 3, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant