Skip to content

pool: an env write on an archived claim replaces its failover row - #422

Merged
CMGS merged 3 commits into
mainfrom
fix/archive-env-row
Nov 3, 2026
Merged

CMGS merged 3 commits into
mainfrom
fix/archive-env-row

Conversation

@CMGS

@CMGS CMGS commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

A PATCH or PUT of /env on an archived claim stayed in the local journal. With archive_failover, the row an adopter starts from kept the value from the archive, so after node loss the claim went back to the old credential, and the old value stayed in the database.

The write now replaces the row, with these outcomes:

Case Result
Registry down 503. The claim's own record has already changed. A resend replaces the row, the way a resend repairs a failed guest write
Stale epoch from a lost reply The row is read, the epoch refreshed, and the write retried once
Another node took the row 404, and the claim drops here
Row consumed by a wake Nothing to do, since a consumed row cannot be adopted
Another transition holds the claim The write waits for it, then checks whether a row is still there

The row still holds the env in plaintext. Encrypting it is a separate question, in #423.

Hot path: none. Only a claim with a row writes one. Running and hibernated claims make no registry call, and a test asserts that.

Known residue: a write landing in the milliseconds between an archive's row write and the claim taking its checkpoint id reaches the journal but not the row. It needs a rotation inside that window and then a node loss before the next write. No mechanism is added for it.

Evidence: tests cover each case above and the adopter holding the rotated value. Mutants, each from a saved copy: without the row write, 5 tests fail (adopter holds "old", want the rotated value), and without the stale-epoch retry, 2 fail (row env "new", want newer). go test ./pool ./server ./types, -race ./pool, the changed tests -race -count=20, go vet on both GOOS and the e2e module's tests pass.

Hardware: two bare-metal nodes, a real PostgreSQL, S3-compatible storage, cocoon v0.7.1, at this PR's head, n=1 per step:

after archive  epoch 3  old    owner a
PATCH new      epoch 4  new    owner a   204 in 22 ms, no wake
PG down        503 {"error":"archive registry down; retry"}, row unchanged
resend newer   epoch 5  newer  owner a   204 in 25 ms, still archived
adoption by b  epoch 6  newer  owner b
  • After SIGKILL of node a, the row moved 0.8 s after a's lease ended. An injected request through node b carried newer, and the guest env held no GH_TOKEN.
  • Three PATCHes on a running claim wrote no sandboxd_archives statement (log_statement='mod'), and the last value still reached an injected request.

Two details from that run, both as designed:

  • The first rewrite moves the row's deadline by about 1 ms, to the claim's own Deadline, because an archive's first row computes it separately. It then holds across later rewrites and the adoption. A claim kept forever stays at null.
  • After a 503 the node's record holds the new value and the row the old one, until the resend. A client that never resends leaves them apart: a wake on this node uses the new value, an adoption after node loss the old one. The API docs state this.

Not run: a rotation inside an in-flight wake on hardware (a test covers it), and any cocoon other than v0.7.1.

Stacked on #420.

Update

  • An env write or a renew on a hibernated replicated claim also replaces its failover row, so an adopter gets the rotated value, the renewed lease and on_expire. Replication rounds skip hibernated claims, so nothing else would. With the registry down the call answers 503 and a resend repairs the row. A running replicated claim stays on the round.
  • Two-node hardware: a rotation and a renew on a hibernated claim reached the adopter.

CMGS added 3 commits November 3, 2026 16:23
A rotation stayed in the local journal, so after node loss an adopter started from the row's old credential and the old value stayed in the database. The write now replaces the row. With the registry down it answers 503 with the local record changed, and a resend repairs the row, as it does a failed guest write. A row another node took drops the claim. Hot path: none, only a claim with a row pays.
…over row

Replication rounds do not run while a claim is hibernated, so a rotation answered there stayed in the local journal and an adopter started from the replica row's old value, with no time bound. The write now replaces the row at its replica checkpoint, as it does for an archived claim. A running replicated claim keeps getting its env through the next round.
Rounds skip hibernated claims, so a renew there left the row's lease and on_expire as they were for as long as the claim slept, and an adopter after node loss restored the old values: a renewed on_expire archive came back as destroy. The renew now replaces the row at its replica checkpoint, as an env write does; with the registry down it answers 503 with the local lease already granted, and a resend repairs the row. A running replicated claim keeps getting its lease through the next round.
@CMGS
CMGS force-pushed the fix/archive-env-row branch from 829354f to 2bbecd6 Compare November 3, 2026 08:24
@CMGS
CMGS changed the base branch from review/post-417 to main November 3, 2026 08:24
@CMGS
CMGS merged commit e58b9ed into main Nov 3, 2026
1 check passed
@CMGS
CMGS deleted the fix/archive-env-row branch November 3, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant