pool: an env write on an archived claim replaces its failover row

docs/deploy.md

@@ -157,7 +157,7 @@ the intersection of the two.
 | `idle_hibernate_seconds` | 0 (off) | node-wide idle policy for unpooled claims (template/checkpoint claims): a none-lane claim is hibernated once it has had no open data-plane connection (relay, buffered exec, preview) and no egress request in flight for this long; the clock restarts when the last connection closes, so a long command is never cut short. The next call that reaches the guest wakes it transparently. Per-pool `idle_hibernate_seconds` does the same for that pool's claims; pooled keys ignore the node-wide value, and egress pools reject it because they cannot resume safely. Opt in deliberately: a wake costs latency and the snapshot, so callers with their own idle logic must not pay twice |
 | `archive_after_seconds` | 0 (off) | tier below hibernation: a hibernated claim idle this long is checkpointed to the store and its local VM dropped, freeing the node entirely; the next call that reaches the guest restores it transparently (a checkpoint restore's latency) with a fresh lease of the length the claim was granted (the server default of 5m when it asked for none); a lease moves only when a client calls [`/renew`](sandboxd-api.md#post-v1sandboxesidrenew). Keep `idle_hibernate_seconds` under the leases your claims use, or a woken sandbox that idles again is destroyed by the reap before it can hibernate, its archive already consumed by the wake; one still in use when its lease ends is destroyed the same way, unless it claimed `on_expire: archive`, which archives at the lease end whatever this setting. Requires `idle_hibernate_seconds > 0` and must exceed it. Node-wide for unpooled keys; per-pool overrides for that pool |
 | `archive_delete_after_seconds` | 0 (keep) | purge an archived claim's store checkpoint this long after it was archived; the claim is then gone for good. On S3, chunks and manifests remain until [offline reclamation](#reclaiming-s3-storage). On archive this retention window replaces the live claim deadline; 0 clears the deadline so the archive is kept forever. Same node-wide/per-pool split |
-| `archive_failover` | false | let another node adopt this node's archived claims once its node lease expires, and the running claims of pools with `replicate_seconds`: each archive and replication round also writes a row (token SHA-256, owner, epoch, checkpoint id, retention deadline, and the claim record with its env, `guest: false` values included, in plaintext) to the shared PostgreSQL, and a `Lookup` that reaches any node takes the claim over there (see [archived claims across node loss](cluster.md#archived-claims-across-node-loss)). Requires `meta_store` with a `cell` (rows are scoped by it, so meshes sharing one database never adopt from each other), a `checkpoint_store` of kind `s3` and a `mesh`, all checked at config load. The node also renews a lease row every 5 s (see [node lease](cluster.md#node-lease)). With the meta store down, archives, wakes and releases of archived claims fail and retry, and replicated claims stop being served once the lease is overdue (see [running claims](cluster.md#running-claims-across-node-loss)); other running claims never touch it. Restart only |
+| `archive_failover` | false | let another node adopt this node's archived claims once its node lease expires, and the running claims of pools with `replicate_seconds`: each archive and replication round also writes a row (token SHA-256, owner, epoch, checkpoint id, retention deadline, and the claim record with its env, `guest: false` values included, in plaintext) to the shared PostgreSQL, and a `Lookup` that reaches any node takes the claim over there (see [archived claims across node loss](cluster.md#archived-claims-across-node-loss)). Requires `meta_store` with a `cell` (rows are scoped by it, so meshes sharing one database never adopt from each other), a `checkpoint_store` of kind `s3` and a `mesh`, all checked at config load. The node also renews a lease row every 5 s (see [node lease](cluster.md#node-lease)). With the meta store down, archives, wakes, releases and env writes of archived claims, and env writes and renews of hibernated replicated claims, fail and retry, and replicated claims stop being served once the lease is overdue (see [running claims](cluster.md#running-claims-across-node-loss)); other running claims never touch it. Restart only |
 | `mesh` | unset | join a cluster ([Clusters](cluster.md)); unset = single node |
 | `pools[]` | — | warm pools, keyed by `(template, net, size)`. `warm` defaults to 4 only when both `warm` and `warm_max` are omitted; an explicit `warm: 0` is kept, so `warm: 0` under a `warm_max` starts the pool empty and grows it on demand, and `warm: 0` alone registers the key with no warm VMs (`PUT /v1/pools` never defaults `warm`); `net` is `none` or `egress`; `size` is a tier, below. Retune online without a restart via [`PUT /v1/pools`](sandboxd-api.md#put-v1pools) — omitted pools drain, and their goldens are deleted once they are idle (`warm: 0` keeps the golden). This is the **first-boot seed**: once a node takes a `PUT /v1/pools`, the applied set persists to `<data_dir>/pools.json` and overrides this section on every later boot (a startup log notes it); delete `pools.json` to return to config-owned pools. With a `meta_store` `cell`, the cell's set overrides it instead (see [shared pool set](#shared-pool-set)). Egress stays config-owned either way. A template re-pulled or re-imported under the same ref is picked up within a minute: the pool rebuilds its golden and replaces its warm VMs. See [state ownership](cluster.md#state-ownership) |
 

docs/egress.md

@@ -362,9 +362,12 @@ guest's next connection; a tunnel the guest keeps open stays as it was.
 The placeholder is not a secret: it stays in the guest and in `GET /env`.
 The entry rides the claim request or any `/env` write, and rotation is the
 same `PATCH`, accepted on a hibernated or archived claim without waking it.
-The write does not reach an archived claim's `archive_failover` row: after that
-claim's node is lost, an adopter starts from the row, which holds the value
-from the archive.
+With [`archive_failover`](deploy.md#configuration) the write also replaces the
+row of an archived claim, or of a hibernated claim with a replica, so an adopter
+starts from the rotated value; a running replicated claim's row takes it at the
+next replication round. While the
+meta store is down the write answers 503 with the claim's own record already
+changed: resend it, and the resend replaces the row.
 
 - **Envelope.** The operator still decides which hosts can carry a
   credential: every host must be covered by an `intercept` rule (`true` or

docs/sandboxd-api.md

@@ -538,7 +538,9 @@ checkpoint for deletion. Wake it first. A claim whose [tenant was
 removed](#tenants-v1tenants) runs to its deadline but cannot extend it: its own
 token gets 403, the root token still renews. 400 a malformed body; 401 missing
 bearer token; 403 the claim's tenant was removed; 404 unknown id or wrong
-token; 409 archived, or being archived right now.
+token; 409 archived, or being archived right now; 503 the archive registry is
+down, for a hibernated claim with a replica under `archive_failover`: the node
+already holds the new lease, and a resend replaces the row.
 
 ## POST /v1/sandboxes/{id}/fork
 
@@ -985,7 +987,9 @@ are already live, but the guest write failed; resend the same request to
 deliver it. The guest side is described in [silkd](silkd.md#claim-env).
 `PUT`/`PATCH` answer 204, `GET` 200; 400 a bad entry, an unknown body
 field or an `inject` outside the claim's intercepted hosts; 404 unknown id or another tenant's claim; 409 a guest change on a
-paused or failed sandbox.
+paused or failed sandbox; 503 the archive registry is down, for an archived
+claim or a hibernated claim with a replica under `archive_failover`: the node
+already holds the new env, and a resend replaces the row.
 
 ## PUT /v1/sandboxes/{id}/instance-metadata
 

sandboxd/pool/claim.go

@@ -183,7 +183,7 @@ func (m *Manager) Renew(ctx context.Context, id string, cred Cred, ttl time.Dura
 	}
 	prev, prevLease, prevExpire := sb.Deadline, sb.LeaseSeconds, sb.OnExpire
 	sb.Deadline, sb.LeaseSeconds, sb.OnExpire = time.Now().Add(lease), int(lease/time.Second), onExpire.Or(sb.OnExpire)
-	deadline, js := sb.Deadline, m.store.set(sb)
+	deadline, js, rowed := sb.Deadline, m.store.set(sb), m.rowFollows(sb)
 	m.mu.Unlock()
 	if err := m.store.commit(js); err != nil {
 		m.mu.Lock()
@@ -199,6 +199,11 @@ func (m *Manager) Renew(ctx context.Context, id string, cred Cred, ttl time.Dura
 	}
 	sb.Touch()
 	m.recordUsage(ctx, usage.Event{Event: "renew", ID: id, Tenant: sb.Tenant, Deadline: deadline})
+	if rowed {
+		if err := m.rewriteArchiveRow(ctx, sb, false); err != nil {
+			return time.Time{}, fmt.Errorf("renew %s: %w", id, err)
+		}
+	}
 	return deadline, nil
 }
 

sandboxd/pool/env.go

@@ -2,12 +2,14 @@ package pool
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"slices"
 	"strings"
 
 	"golang.org/x/sync/errgroup"
 
+	"github.com/cocoonstack/sandbox/sandboxd/archives"
 	"github.com/cocoonstack/sandbox/sandboxd/types"
 )
 
@@ -78,25 +80,65 @@ func (m *Manager) writeEnv(ctx context.Context, id, tenant, verb string, next fu
 		}
 		return ErrPaused
 	}
+	rowed, deliver := m.rowFollows(sb), !paused
 	if env.Equal(prev) {
 		m.mu.Unlock()
-		return m.deliverGuestEnv(ctx, verb, id, sock, env, !paused)
+	} else {
+		deliver = changed
+		sb.SetEnv(env)
+		js := m.store.set(sb)
+		m.mu.Unlock()
+		if err := m.store.commit(js); err != nil {
+			m.mu.Lock()
+			var rb claimSnapshot
+			if m.claimed[id] == sb && sb.Env.Equal(env) {
+				sb.SetEnv(prev)
+				rb = m.store.set(sb)
+			}
+			m.mu.Unlock()
+			m.store.recommit(ctx, rb)
+			return fmt.Errorf("%s env %s: persist claims: %w", verb, id, err)
+		}
 	}
-	sb.SetEnv(env)
-	js := m.store.set(sb)
-	m.mu.Unlock()
-	if err := m.store.commit(js); err != nil {
-		m.mu.Lock()
-		var rb claimSnapshot
-		if m.claimed[id] == sb && sb.Env.Equal(env) {
-			sb.SetEnv(prev)
-			rb = m.store.set(sb)
+	if rowed {
+		if err := m.rewriteArchiveRow(ctx, sb, locked); err != nil {
+			return fmt.Errorf("%s env %s: %w", verb, id, err)
 		}
+	}
+	return m.deliverGuestEnv(ctx, verb, id, sock, env, deliver)
+}
+
+func (m *Manager) rewriteArchiveRow(ctx context.Context, sb *types.Sandbox, locked bool) error {
+	if !locked {
+		sb.Transition.Lock()
+		defer sb.Transition.Unlock()
+	}
+	for range 2 {
+		m.mu.Lock()
+		ck, replica, rowed := sb.ArchiveCk, sb.ReplicaCk, m.claimed[sb.ID] == sb && m.rowFollows(sb)
 		m.mu.Unlock()
-		m.store.recommit(ctx, rb)
-		return fmt.Errorf("%s env %s: persist claims: %w", verb, id, err)
+		if !rowed {
+			return nil
+		}
+		var epoch int64
+		var err error
+		if ck != "" {
+			epoch, err = m.replaceArchive(ctx, sb, ck)
+		} else {
+			epoch, err = m.replaceReplica(ctx, sb, replica)
+		}
+		if err == nil {
+			_, _ = m.commitIfLive(ctx, sb, func() { sb.ArchiveEpoch = epoch })
+			return nil
+		}
+		if !errors.Is(err, archives.ErrLost) {
+			return err
+		}
+		if err = m.refusedReplace(ctx, sb); err != nil {
+			return err
+		}
 	}
-	return m.deliverGuestEnv(ctx, verb, id, sock, env, changed)
+	return nil
 }
 
 func (m *Manager) deliverGuestEnv(ctx context.Context, verb, id, sock string, env types.Env, deliver bool) error {

sandboxd/pool/failover.go

@@ -688,6 +688,10 @@ func (m *Manager) archiveRowed(sb *types.Sandbox) bool {
 	return m.archives != nil && sb.ArchiveCk != "" && sb.ArchiveEpoch != 0
 }
 
+func (m *Manager) rowFollows(sb *types.Sandbox) bool {
+	return m.archiveRowed(sb) || m.replicaRowed(sb) && sb.HibernateSnap != ""
+}
+
 func (m *Manager) expiryDropsRow(sb *types.Sandbox) bool {
 	return m.archiveRowed(sb) || m.archives != nil && sb.HasReplica() && !m.archivesAtDeadline(sb)
 }

sandboxd/pool/failover_test.go

@@ -72,6 +72,153 @@ func TestFailoverRowWriteFailureKeepsTheClaimHibernated(t *testing.T) {
 	}
 }
 
+func TestFailoverEnvRotationReachesTheAdopter(t *testing.T) {
+	c := newFailoverCluster(t)
+	a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour)
+	sb := mustClaim(t, a, testKey)
+	patchHostOnly(t, a, sb, "old")
+	mustArchive(t, a, sb)
+	if rowEnv(t, c, sb) != "old" {
+		t.Fatalf("archive row env %q, want old", rowEnv(t, c, sb))
+	}
+
+	patchHostOnly(t, a, sb, "new")
+	if got := rowEnv(t, c, sb); got != "new" {
+		t.Errorf("row env %q after a rotation, want new", got)
+	}
+	rec, _, err := c.reg.Get(t.Context(), sb.ID)
+	if err != nil {
+		t.Fatalf("row: %v", err)
+	}
+	if rec.Epoch != sb.ArchiveEpoch || rec.Checkpoint != sb.ArchiveCk || rec.Deadline.IsZero() {
+		t.Errorf("row %+v claim epoch %d ck %s, want the same epoch and checkpoint and a kept deadline", rec, sb.ArchiveEpoch, sb.ArchiveCk)
+	}
+
+	c.expire("a")
+	adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token)
+	if err != nil || !adopted {
+		t.Fatalf("adopt: %v %v", adopted, err)
+	}
+	held, ok := b.byID(sb.ID)
+	if !ok {
+		t.Fatal("adopter has no claim")
+	}
+	if got, _ := held.HiddenEnv("GH_TOKEN"); got != "new" {
+		t.Errorf("adopter holds %q, want the rotated value", got)
+	}
+}
+
+func TestFailoverEnvRotationWithTheRegistryDownIsLoudAndARetryRepairs(t *testing.T) {
+	c := newFailoverCluster(t)
+	a := c.node(t, "a", t.TempDir(), keepAnHour)
+	sb := mustClaim(t, a, testKey)
+	patchHostOnly(t, a, sb, "old")
+	mustArchive(t, a, sb)
+
+	reg := &replaceFailRegistry{Registry: c.reg, fail: true}
+	a.archives = reg
+	patch := types.EnvPatch{"GH_TOKEN": {Value: "new", Guest: hostOnly}}
+	if err := a.PatchEnv(t.Context(), sb.ID, patch, ""); !errors.Is(err, archives.ErrUnavailable) {
+		t.Fatalf("rotation with the registry down: %v, want ErrUnavailable", err)
+	}
+	if got := rowEnv(t, c, sb); got != "old" {
+		t.Errorf("row env %q, want old until the write lands", got)
+	}
+	reg.fail = false
+	if err := a.PatchEnv(t.Context(), sb.ID, patch, ""); err != nil {
+		t.Fatalf("resend: %v", err)
+	}
+	if got := rowEnv(t, c, sb); got != "new" {
+		t.Errorf("row env %q after the resend, want new", got)
+	}
+}
+
+func TestFailoverEnvRotationSurvivesALostReply(t *testing.T) {
+	c := newFailoverCluster(t)
+	a := c.node(t, "a", t.TempDir(), keepAnHour)
+	sb := mustClaim(t, a, testKey)
+	patchHostOnly(t, a, sb, "old")
+	mustArchive(t, a, sb)
+
+	reg := &replaceFailRegistry{Registry: c.reg, loseReply: true}
+	a.archives = reg
+	patch := types.EnvPatch{"GH_TOKEN": {Value: "new", Guest: hostOnly}}
+	if err := a.PatchEnv(t.Context(), sb.ID, patch, ""); !errors.Is(err, archives.ErrUnavailable) {
+		t.Fatalf("rotation with a lost reply: %v, want ErrUnavailable", err)
+	}
+	patch["GH_TOKEN"] = &types.EnvVar{Value: "newer", Guest: hostOnly}
+	if err := a.PatchEnv(t.Context(), sb.ID, patch, ""); err != nil {
+		t.Fatalf("resend at a stale epoch: %v", err)
+	}
+	if got := rowEnv(t, c, sb); got != "newer" {
+		t.Errorf("row env %q, want newer", got)
+	}
+	rec, _, _ := c.reg.Get(t.Context(), sb.ID)
+	if rec.Epoch != sb.ArchiveEpoch {
+		t.Errorf("row epoch %d, claim holds %d", rec.Epoch, sb.ArchiveEpoch)
+	}
+}
+
+func TestFailoverEnvRotationOfAClaimAdoptedElsewhereDropsIt(t *testing.T) {
+	c := newFailoverCluster(t)
+	a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour)
+	sb := mustClaim(t, a, testKey)
+	patchHostOnly(t, a, sb, "old")
+	mustArchive(t, a, sb)
+	c.expire("a")
+	if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted {
+		t.Fatalf("adopt: %v %v", adopted, err)
+	}
+
+	patch := types.EnvPatch{"GH_TOKEN": {Value: "new", Guest: hostOnly}}
+	if err := a.PatchEnv(t.Context(), sb.ID, patch, ""); !errors.Is(err, ErrUnknownSandbox) {
+		t.Errorf("rotation on the old owner: %v, want ErrUnknownSandbox", err)
+	}
+	if _, ok := a.byID(sb.ID); ok {
+		t.Error("the old owner kept a claim another node adopted")
+	}
+	if got := rowEnv(t, c, sb); got != "old" {
+		t.Errorf("row env %q, the old owner must not overwrite the adopter's row", got)
+	}
+}
+
+func TestFailoverEnvRotationWaitsForATransitionInFlight(t *testing.T) {
+	c := newFailoverCluster(t)
+	a := c.node(t, "a", t.TempDir(), keepAnHour)
+	sb := mustClaim(t, a, testKey)
+	patchHostOnly(t, a, sb, "old")
+	mustArchive(t, a, sb)
+
+	sb.Transition.Lock()
+	done := make(chan error, 1)
+	go func() {
+		done <- a.PatchEnv(t.Context(), sb.ID, types.EnvPatch{"GH_TOKEN": {Value: "new", Guest: hostOnly}}, "")
+	}()
+	waitFor(t, func() bool { got, _ := sb.HiddenEnv("GH_TOKEN"); return got == "new" })
+	if got := rowEnv(t, c, sb); got != "old" {
+		t.Errorf("row env %q while another transition holds the claim, want old", got)
+	}
+	sb.Transition.Unlock()
+	if err := <-done; err != nil {
+		t.Fatalf("rotation: %v", err)
+	}
+	if got := rowEnv(t, c, sb); got != "new" {
+		t.Errorf("row env %q after the transition, want new", got)
+	}
+}
+
+func TestFailoverEnvWriteOfARunningClaimLeavesTheRegistryAlone(t *testing.T) {
+	c := newFailoverCluster(t)
+	a := c.node(t, "a", t.TempDir(), keepAnHour)
+	sb := mustClaim(t, a, testKey)
+	reg := &replaceFailRegistry{Registry: c.reg, fail: true}
+	a.archives = reg
+	patchHostOnly(t, a, sb, "new")
+	if reg.calls != 0 {
+		t.Errorf("%d registry replaces for a claim with no row, want 0", reg.calls)
+	}
+}
+
 func TestFailoverOwnerWakeTakesThenDeletesTheRow(t *testing.T) {
 	c := newFailoverCluster(t)
 	a := c.node(t, "a", t.TempDir(), keepAnHour)
@@ -1975,6 +2122,50 @@ func seedArchiveCkBy(t *testing.T, m *Manager, node, id, sandboxID string) {
 	}
 }
 
+func patchHostOnly(t *testing.T, m *Manager, sb *types.Sandbox, value string) {
+	t.Helper()
+	if err := m.PatchEnv(t.Context(), sb.ID, types.EnvPatch{"GH_TOKEN": {Value: value, Guest: hostOnly}}, ""); err != nil {
+		t.Fatalf("patch env: %v", err)
+	}
+}
+
+func rowEnv(t *testing.T, c *failoverCluster, sb *types.Sandbox) string {
+	t.Helper()
+	return rowClaim(t, c, sb).Env["GH_TOKEN"].Value
+}
+
+func rowClaim(t *testing.T, c *failoverCluster, sb *types.Sandbox) *types.Sandbox {
+	t.Helper()
+	rec, ok, err := c.reg.Get(t.Context(), sb.ID)
+	if err != nil || !ok {
+		t.Fatalf("row: %v %v", ok, err)
+	}
+	claim := &types.Sandbox{}
+	if err = json.Unmarshal(rec.Claim, claim); err != nil {
+		t.Fatalf("decode row claim: %v", err)
+	}
+	return claim
+}
+
+type replaceFailRegistry struct {
+	archives.Registry
+	fail, loseReply bool
+	calls           int
+}
+
+func (r *replaceFailRegistry) Replace(ctx context.Context, rec archives.Record, expect int64) (int64, error) {
+	r.calls++
+	if r.fail {
+		return 0, archives.ErrUnavailable
+	}
+	epoch, err := r.Registry.Replace(ctx, rec, expect)
+	if r.loseReply && err == nil {
+		r.loseReply = false
+		return 0, archives.ErrUnavailable
+	}
+	return epoch, err
+}
+
 type hookRegistry struct {
 	archives.Registry
 	beforePut        func()