Repository navigation
pool: a settled archive intent keeps a replicated claim's row - #426
Merged
Merged
Conversation
settleIntent took any own row for the archive's and deleted it. For a hibernated replicated claim that row is the replica's: after one failed or lost Replace, the next archive pass deleted it, refusedReplace read the absent row as an adopter finishing, and a claim nobody adopted or released was destroyed. A restart mid-archive did the same at the first unfence. The row now stays, and one the archive moved goes back to the replica, so it is never absent while the claim is held. Off the claim path.
The rollback deleted the row it had just moved from the replica to the archive checkpoint and zeroed the epoch, while the journal on disk still named the replica at the old epoch. After a restart the first unfence read the missing row as a finished adoption and removed the claim, its VM and its snapshot. A replicated claim now keeps the row's epoch and its archive intent, so the next settle moves the row back to the replica.
After a registry outage longer than the lease every node is fenced and every lease expired. A client polling a node made it adopt while it was fenced, or right after its own lift, before the healthy owner had renewed; the owner then confirmed, saw a foreign row and removed its running VM. Adoption now waits for the fence and for one renewal interval plus one registry query after the lift. A claim this node already holds keeps its path.
The unfence pass confirms each replicated claim with its own Take, so its length grows with the claim count times the registry round trip. TakeOwned moves the owner's live rows at their expected epochs in one statement, with Take's compare-and-set on the epoch and no lease arm, since the owner is the caller.
… write The confirmation ran one Take and one whole-file claims commit per replicated claim, serially. Past the 12 s window the lift re-tripped and the next renewal started over, so about 1200 claims at a 10 ms registry round trip never lifted. The rows this node holds now move in one TakeOwned and their epochs land in one commit; absent, foreign, consumed or moved rows keep the per-row path.
CMGS
force-pushed
the
fix/archive-intent-replica-row
branch
from
November 3, 2026 08:23
38f9e24 to
243a0ea
Compare
This was referenced Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One registry fault can destroy a hibernated replicated claim that nobody adopted or released. This is on
maintoday.settleIntenttook any row this node owns for the row its archive wrote, and deleted it. For a replicated claim that row is the replica's. Two sequences follow:Replaceof an archive fails, or its reply is lost, soArchivePendingstays. The next archive pass, 5 s later, settles the intent and deletes the replica row.writeArchiveRowthen doesReplaceat the held epoch, finds no row, andrefusedReplacereads that as an adopter finishing:dropAdoptedremoves the claim and its VM. A PostgreSQL outage that starts during the upload is enough.Fix. The row stays. A row naming the replica is left alone, and one the archive moved goes back to the replica. The row is never absent while the claim is held, so absent keeps its one meaning, and the claim stays adoptable from its replica while the archive retries. Deleting the row and zeroing the epoch also fixed both sequences, but left a window of absent row that a crash or failed journal write could still misread; it closed 7 of the 12 failures below, this closes 10.
Hot path: none. This runs only when an archive intent settles.
Evidence:
A fault matrix ran 1358 combinations of 11 scenarios, each durable step, 5 fault kinds and 6 continuations (heal, lease lapse with adoption, wake, release, a failing boot write, a sweep then a crash):
The other invariants (two serving owners, a released claim adoptable or served, a live row naming a deleted checkpoint) had no failures before or after. Checkpoint residue went from 97 lines to 95. The matrix is a temporary harness, not in this PR, with an in-memory registry whose compare-and-set arms were read against the PostgreSQL ones.
The two that remain both need a disk that keeps failing claim writes across a crash, plus a failing boot write or an hourly sweep. Both are left as they are: the disk failure is already loud, and no mechanism is added for it.
Tests in this PR, each failing without the fix:
TestReplicaArchiveRetriedAfterARegistryFaultKeepsTheClaimTestReplicaRestartMidArchiveKeepsTheClaimgo test -race ./pool, the new tests-race -count=20,go vetandaslon both GOOS and the e2e module's tests pass.Not run: hardware on this PR alone. The full hardware round runs on the tree with every open PR merged, and the result lands here.
Stacked on #420.
Update
archive-replicatedclaim-lost lines are gone. A hardware run on tmpfs did not reach the journal failure, so this rests on the regression test and the matrix.TakeOwned). Two nodes, 1000 replicated claims, restart to lift:Conflict to expect with #424 in
adoptArchived: keep the held check first, then the fence.