diff --git a/docs/deploy.md b/docs/deploy.md index 11ad42158..8266a7db3 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -157,7 +157,7 @@ the intersection of the two. | `idle_hibernate_seconds` | 0 (off) | node-wide idle policy for unpooled claims (template/checkpoint claims): a none-lane claim is hibernated once it has had no open data-plane connection (relay, buffered exec, preview) and no egress request in flight for this long; the clock restarts when the last connection closes, so a long command is never cut short. The next call that reaches the guest wakes it transparently. Per-pool `idle_hibernate_seconds` does the same for that pool's claims; pooled keys ignore the node-wide value, and egress pools reject it because they cannot resume safely. Opt in deliberately: a wake costs latency and the snapshot, so callers with their own idle logic must not pay twice | | `archive_after_seconds` | 0 (off) | tier below hibernation: a hibernated claim idle this long is checkpointed to the store and its local VM dropped, freeing the node entirely; the next call that reaches the guest restores it transparently (a checkpoint restore's latency) with a fresh lease of the length the claim was granted (the server default of 5m when it asked for none); a lease moves only when a client calls [`/renew`](sandboxd-api.md#post-v1sandboxesidrenew). Keep `idle_hibernate_seconds` under the leases your claims use, or a woken sandbox that idles again is destroyed by the reap before it can hibernate, its archive already consumed by the wake; one still in use when its lease ends is destroyed the same way, unless it claimed `on_expire: archive`, which archives at the lease end whatever this setting. Requires `idle_hibernate_seconds > 0` and must exceed it. Node-wide for unpooled keys; per-pool overrides for that pool | | `archive_delete_after_seconds` | 0 (keep) | purge an archived claim's store checkpoint this long after it was archived; the claim is then gone for good. On S3, chunks and manifests remain until [offline reclamation](#reclaiming-s3-storage). On archive this retention window replaces the live claim deadline; 0 clears the deadline so the archive is kept forever. Same node-wide/per-pool split | -| `archive_failover` | false | let another node adopt this node's archived claims once its node lease expires, and the running claims of pools with `replicate_seconds`: each archive and replication round also writes a row (token SHA-256, owner, epoch, checkpoint id, retention deadline, and the claim record with its env, `guest: false` values included, in plaintext) to the shared PostgreSQL, and a `Lookup` that reaches any node takes the claim over there (see [archived claims across node loss](cluster.md#archived-claims-across-node-loss)). Requires `meta_store` with a `cell` (rows are scoped by it, so meshes sharing one database never adopt from each other), a `checkpoint_store` of kind `s3` and a `mesh`, all checked at config load. The node also renews a lease row every 5 s (see [node lease](cluster.md#node-lease)). With the meta store down, archives, wakes and releases of archived claims fail and retry, and replicated claims stop being served once the lease is overdue (see [running claims](cluster.md#running-claims-across-node-loss)); other running claims never touch it. Restart only | +| `archive_failover` | false | let another node adopt this node's archived claims once its node lease expires, and the running claims of pools with `replicate_seconds`: each archive and replication round also writes a row (token SHA-256, owner, epoch, checkpoint id, retention deadline, and the claim record with its env, `guest: false` values included, in plaintext) to the shared PostgreSQL, and a `Lookup` that reaches any node takes the claim over there (see [archived claims across node loss](cluster.md#archived-claims-across-node-loss)). Requires `meta_store` with a `cell` (rows are scoped by it, so meshes sharing one database never adopt from each other), a `checkpoint_store` of kind `s3` and a `mesh`, all checked at config load. The node also renews a lease row every 5 s (see [node lease](cluster.md#node-lease)). With the meta store down, archives, wakes, releases and env writes of archived claims, and env writes and renews of hibernated replicated claims, fail and retry, and replicated claims stop being served once the lease is overdue (see [running claims](cluster.md#running-claims-across-node-loss)); other running claims never touch it. Restart only | | `mesh` | unset | join a cluster ([Clusters](cluster.md)); unset = single node | | `pools[]` | — | warm pools, keyed by `(template, net, size)`. `warm` defaults to 4 only when both `warm` and `warm_max` are omitted; an explicit `warm: 0` is kept, so `warm: 0` under a `warm_max` starts the pool empty and grows it on demand, and `warm: 0` alone registers the key with no warm VMs (`PUT /v1/pools` never defaults `warm`); `net` is `none` or `egress`; `size` is a tier, below. Retune online without a restart via [`PUT /v1/pools`](sandboxd-api.md#put-v1pools) — omitted pools drain, and their goldens are deleted once they are idle (`warm: 0` keeps the golden). This is the **first-boot seed**: once a node takes a `PUT /v1/pools`, the applied set persists to `/pools.json` and overrides this section on every later boot (a startup log notes it); delete `pools.json` to return to config-owned pools. With a `meta_store` `cell`, the cell's set overrides it instead (see [shared pool set](#shared-pool-set)). Egress stays config-owned either way. A template re-pulled or re-imported under the same ref is picked up within a minute: the pool rebuilds its golden and replaces its warm VMs. See [state ownership](cluster.md#state-ownership) | diff --git a/docs/egress.md b/docs/egress.md index 0844eb56a..172a2647e 100644 --- a/docs/egress.md +++ b/docs/egress.md @@ -362,9 +362,12 @@ guest's next connection; a tunnel the guest keeps open stays as it was. The placeholder is not a secret: it stays in the guest and in `GET /env`. The entry rides the claim request or any `/env` write, and rotation is the same `PATCH`, accepted on a hibernated or archived claim without waking it. -The write does not reach an archived claim's `archive_failover` row: after that -claim's node is lost, an adopter starts from the row, which holds the value -from the archive. +With [`archive_failover`](deploy.md#configuration) the write also replaces the +row of an archived claim, or of a hibernated claim with a replica, so an adopter +starts from the rotated value; a running replicated claim's row takes it at the +next replication round. While the +meta store is down the write answers 503 with the claim's own record already +changed: resend it, and the resend replaces the row. - **Envelope.** The operator still decides which hosts can carry a credential: every host must be covered by an `intercept` rule (`true` or diff --git a/docs/sandboxd-api.md b/docs/sandboxd-api.md index 962e14ff0..183642096 100644 --- a/docs/sandboxd-api.md +++ b/docs/sandboxd-api.md @@ -538,7 +538,9 @@ checkpoint for deletion. Wake it first. A claim whose [tenant was removed](#tenants-v1tenants) runs to its deadline but cannot extend it: its own token gets 403, the root token still renews. 400 a malformed body; 401 missing bearer token; 403 the claim's tenant was removed; 404 unknown id or wrong -token; 409 archived, or being archived right now. +token; 409 archived, or being archived right now; 503 the archive registry is +down, for a hibernated claim with a replica under `archive_failover`: the node +already holds the new lease, and a resend replaces the row. ## POST /v1/sandboxes/{id}/fork @@ -985,7 +987,9 @@ are already live, but the guest write failed; resend the same request to deliver it. The guest side is described in [silkd](silkd.md#claim-env). `PUT`/`PATCH` answer 204, `GET` 200; 400 a bad entry, an unknown body field or an `inject` outside the claim's intercepted hosts; 404 unknown id or another tenant's claim; 409 a guest change on a -paused or failed sandbox. +paused or failed sandbox; 503 the archive registry is down, for an archived +claim or a hibernated claim with a replica under `archive_failover`: the node +already holds the new env, and a resend replaces the row. ## PUT /v1/sandboxes/{id}/instance-metadata diff --git a/sandboxd/pool/claim.go b/sandboxd/pool/claim.go index 87432c1cb..86d83871c 100644 --- a/sandboxd/pool/claim.go +++ b/sandboxd/pool/claim.go @@ -183,7 +183,7 @@ func (m *Manager) Renew(ctx context.Context, id string, cred Cred, ttl time.Dura } prev, prevLease, prevExpire := sb.Deadline, sb.LeaseSeconds, sb.OnExpire sb.Deadline, sb.LeaseSeconds, sb.OnExpire = time.Now().Add(lease), int(lease/time.Second), onExpire.Or(sb.OnExpire) - deadline, js := sb.Deadline, m.store.set(sb) + deadline, js, rowed := sb.Deadline, m.store.set(sb), m.rowFollows(sb) m.mu.Unlock() if err := m.store.commit(js); err != nil { m.mu.Lock() @@ -199,6 +199,11 @@ func (m *Manager) Renew(ctx context.Context, id string, cred Cred, ttl time.Dura } sb.Touch() m.recordUsage(ctx, usage.Event{Event: "renew", ID: id, Tenant: sb.Tenant, Deadline: deadline}) + if rowed { + if err := m.rewriteArchiveRow(ctx, sb, false); err != nil { + return time.Time{}, fmt.Errorf("renew %s: %w", id, err) + } + } return deadline, nil } diff --git a/sandboxd/pool/env.go b/sandboxd/pool/env.go index b31ba8f94..5af30ac65 100644 --- a/sandboxd/pool/env.go +++ b/sandboxd/pool/env.go @@ -2,12 +2,14 @@ package pool import ( "context" + "errors" "fmt" "slices" "strings" "golang.org/x/sync/errgroup" + "github.com/cocoonstack/sandbox/sandboxd/archives" "github.com/cocoonstack/sandbox/sandboxd/types" ) @@ -78,25 +80,65 @@ func (m *Manager) writeEnv(ctx context.Context, id, tenant, verb string, next fu } return ErrPaused } + rowed, deliver := m.rowFollows(sb), !paused if env.Equal(prev) { m.mu.Unlock() - return m.deliverGuestEnv(ctx, verb, id, sock, env, !paused) + } else { + deliver = changed + sb.SetEnv(env) + js := m.store.set(sb) + m.mu.Unlock() + if err := m.store.commit(js); err != nil { + m.mu.Lock() + var rb claimSnapshot + if m.claimed[id] == sb && sb.Env.Equal(env) { + sb.SetEnv(prev) + rb = m.store.set(sb) + } + m.mu.Unlock() + m.store.recommit(ctx, rb) + return fmt.Errorf("%s env %s: persist claims: %w", verb, id, err) + } } - sb.SetEnv(env) - js := m.store.set(sb) - m.mu.Unlock() - if err := m.store.commit(js); err != nil { - m.mu.Lock() - var rb claimSnapshot - if m.claimed[id] == sb && sb.Env.Equal(env) { - sb.SetEnv(prev) - rb = m.store.set(sb) + if rowed { + if err := m.rewriteArchiveRow(ctx, sb, locked); err != nil { + return fmt.Errorf("%s env %s: %w", verb, id, err) } + } + return m.deliverGuestEnv(ctx, verb, id, sock, env, deliver) +} + +func (m *Manager) rewriteArchiveRow(ctx context.Context, sb *types.Sandbox, locked bool) error { + if !locked { + sb.Transition.Lock() + defer sb.Transition.Unlock() + } + for range 2 { + m.mu.Lock() + ck, replica, rowed := sb.ArchiveCk, sb.ReplicaCk, m.claimed[sb.ID] == sb && m.rowFollows(sb) m.mu.Unlock() - m.store.recommit(ctx, rb) - return fmt.Errorf("%s env %s: persist claims: %w", verb, id, err) + if !rowed { + return nil + } + var epoch int64 + var err error + if ck != "" { + epoch, err = m.replaceArchive(ctx, sb, ck) + } else { + epoch, err = m.replaceReplica(ctx, sb, replica) + } + if err == nil { + _, _ = m.commitIfLive(ctx, sb, func() { sb.ArchiveEpoch = epoch }) + return nil + } + if !errors.Is(err, archives.ErrLost) { + return err + } + if err = m.refusedReplace(ctx, sb); err != nil { + return err + } } - return m.deliverGuestEnv(ctx, verb, id, sock, env, changed) + return nil } func (m *Manager) deliverGuestEnv(ctx context.Context, verb, id, sock string, env types.Env, deliver bool) error { diff --git a/sandboxd/pool/failover.go b/sandboxd/pool/failover.go index 63a1f348c..e5b71761e 100644 --- a/sandboxd/pool/failover.go +++ b/sandboxd/pool/failover.go @@ -688,6 +688,10 @@ func (m *Manager) archiveRowed(sb *types.Sandbox) bool { return m.archives != nil && sb.ArchiveCk != "" && sb.ArchiveEpoch != 0 } +func (m *Manager) rowFollows(sb *types.Sandbox) bool { + return m.archiveRowed(sb) || m.replicaRowed(sb) && sb.HibernateSnap != "" +} + func (m *Manager) expiryDropsRow(sb *types.Sandbox) bool { return m.archiveRowed(sb) || m.archives != nil && sb.HasReplica() && !m.archivesAtDeadline(sb) } diff --git a/sandboxd/pool/failover_test.go b/sandboxd/pool/failover_test.go index 099d01388..9bb3e593e 100644 --- a/sandboxd/pool/failover_test.go +++ b/sandboxd/pool/failover_test.go @@ -72,6 +72,153 @@ func TestFailoverRowWriteFailureKeepsTheClaimHibernated(t *testing.T) { } } +func TestFailoverEnvRotationReachesTheAdopter(t *testing.T) { + c := newFailoverCluster(t) + a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) + sb := mustClaim(t, a, testKey) + patchHostOnly(t, a, sb, "old") + mustArchive(t, a, sb) + if rowEnv(t, c, sb) != "old" { + t.Fatalf("archive row env %q, want old", rowEnv(t, c, sb)) + } + + patchHostOnly(t, a, sb, "new") + if got := rowEnv(t, c, sb); got != "new" { + t.Errorf("row env %q after a rotation, want new", got) + } + rec, _, err := c.reg.Get(t.Context(), sb.ID) + if err != nil { + t.Fatalf("row: %v", err) + } + if rec.Epoch != sb.ArchiveEpoch || rec.Checkpoint != sb.ArchiveCk || rec.Deadline.IsZero() { + t.Errorf("row %+v claim epoch %d ck %s, want the same epoch and checkpoint and a kept deadline", rec, sb.ArchiveEpoch, sb.ArchiveCk) + } + + c.expire("a") + adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token) + if err != nil || !adopted { + t.Fatalf("adopt: %v %v", adopted, err) + } + held, ok := b.byID(sb.ID) + if !ok { + t.Fatal("adopter has no claim") + } + if got, _ := held.HiddenEnv("GH_TOKEN"); got != "new" { + t.Errorf("adopter holds %q, want the rotated value", got) + } +} + +func TestFailoverEnvRotationWithTheRegistryDownIsLoudAndARetryRepairs(t *testing.T) { + c := newFailoverCluster(t) + a := c.node(t, "a", t.TempDir(), keepAnHour) + sb := mustClaim(t, a, testKey) + patchHostOnly(t, a, sb, "old") + mustArchive(t, a, sb) + + reg := &replaceFailRegistry{Registry: c.reg, fail: true} + a.archives = reg + patch := types.EnvPatch{"GH_TOKEN": {Value: "new", Guest: hostOnly}} + if err := a.PatchEnv(t.Context(), sb.ID, patch, ""); !errors.Is(err, archives.ErrUnavailable) { + t.Fatalf("rotation with the registry down: %v, want ErrUnavailable", err) + } + if got := rowEnv(t, c, sb); got != "old" { + t.Errorf("row env %q, want old until the write lands", got) + } + reg.fail = false + if err := a.PatchEnv(t.Context(), sb.ID, patch, ""); err != nil { + t.Fatalf("resend: %v", err) + } + if got := rowEnv(t, c, sb); got != "new" { + t.Errorf("row env %q after the resend, want new", got) + } +} + +func TestFailoverEnvRotationSurvivesALostReply(t *testing.T) { + c := newFailoverCluster(t) + a := c.node(t, "a", t.TempDir(), keepAnHour) + sb := mustClaim(t, a, testKey) + patchHostOnly(t, a, sb, "old") + mustArchive(t, a, sb) + + reg := &replaceFailRegistry{Registry: c.reg, loseReply: true} + a.archives = reg + patch := types.EnvPatch{"GH_TOKEN": {Value: "new", Guest: hostOnly}} + if err := a.PatchEnv(t.Context(), sb.ID, patch, ""); !errors.Is(err, archives.ErrUnavailable) { + t.Fatalf("rotation with a lost reply: %v, want ErrUnavailable", err) + } + patch["GH_TOKEN"] = &types.EnvVar{Value: "newer", Guest: hostOnly} + if err := a.PatchEnv(t.Context(), sb.ID, patch, ""); err != nil { + t.Fatalf("resend at a stale epoch: %v", err) + } + if got := rowEnv(t, c, sb); got != "newer" { + t.Errorf("row env %q, want newer", got) + } + rec, _, _ := c.reg.Get(t.Context(), sb.ID) + if rec.Epoch != sb.ArchiveEpoch { + t.Errorf("row epoch %d, claim holds %d", rec.Epoch, sb.ArchiveEpoch) + } +} + +func TestFailoverEnvRotationOfAClaimAdoptedElsewhereDropsIt(t *testing.T) { + c := newFailoverCluster(t) + a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) + sb := mustClaim(t, a, testKey) + patchHostOnly(t, a, sb, "old") + mustArchive(t, a, sb) + c.expire("a") + if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { + t.Fatalf("adopt: %v %v", adopted, err) + } + + patch := types.EnvPatch{"GH_TOKEN": {Value: "new", Guest: hostOnly}} + if err := a.PatchEnv(t.Context(), sb.ID, patch, ""); !errors.Is(err, ErrUnknownSandbox) { + t.Errorf("rotation on the old owner: %v, want ErrUnknownSandbox", err) + } + if _, ok := a.byID(sb.ID); ok { + t.Error("the old owner kept a claim another node adopted") + } + if got := rowEnv(t, c, sb); got != "old" { + t.Errorf("row env %q, the old owner must not overwrite the adopter's row", got) + } +} + +func TestFailoverEnvRotationWaitsForATransitionInFlight(t *testing.T) { + c := newFailoverCluster(t) + a := c.node(t, "a", t.TempDir(), keepAnHour) + sb := mustClaim(t, a, testKey) + patchHostOnly(t, a, sb, "old") + mustArchive(t, a, sb) + + sb.Transition.Lock() + done := make(chan error, 1) + go func() { + done <- a.PatchEnv(t.Context(), sb.ID, types.EnvPatch{"GH_TOKEN": {Value: "new", Guest: hostOnly}}, "") + }() + waitFor(t, func() bool { got, _ := sb.HiddenEnv("GH_TOKEN"); return got == "new" }) + if got := rowEnv(t, c, sb); got != "old" { + t.Errorf("row env %q while another transition holds the claim, want old", got) + } + sb.Transition.Unlock() + if err := <-done; err != nil { + t.Fatalf("rotation: %v", err) + } + if got := rowEnv(t, c, sb); got != "new" { + t.Errorf("row env %q after the transition, want new", got) + } +} + +func TestFailoverEnvWriteOfARunningClaimLeavesTheRegistryAlone(t *testing.T) { + c := newFailoverCluster(t) + a := c.node(t, "a", t.TempDir(), keepAnHour) + sb := mustClaim(t, a, testKey) + reg := &replaceFailRegistry{Registry: c.reg, fail: true} + a.archives = reg + patchHostOnly(t, a, sb, "new") + if reg.calls != 0 { + t.Errorf("%d registry replaces for a claim with no row, want 0", reg.calls) + } +} + func TestFailoverOwnerWakeTakesThenDeletesTheRow(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) @@ -1975,6 +2122,50 @@ func seedArchiveCkBy(t *testing.T, m *Manager, node, id, sandboxID string) { } } +func patchHostOnly(t *testing.T, m *Manager, sb *types.Sandbox, value string) { + t.Helper() + if err := m.PatchEnv(t.Context(), sb.ID, types.EnvPatch{"GH_TOKEN": {Value: value, Guest: hostOnly}}, ""); err != nil { + t.Fatalf("patch env: %v", err) + } +} + +func rowEnv(t *testing.T, c *failoverCluster, sb *types.Sandbox) string { + t.Helper() + return rowClaim(t, c, sb).Env["GH_TOKEN"].Value +} + +func rowClaim(t *testing.T, c *failoverCluster, sb *types.Sandbox) *types.Sandbox { + t.Helper() + rec, ok, err := c.reg.Get(t.Context(), sb.ID) + if err != nil || !ok { + t.Fatalf("row: %v %v", ok, err) + } + claim := &types.Sandbox{} + if err = json.Unmarshal(rec.Claim, claim); err != nil { + t.Fatalf("decode row claim: %v", err) + } + return claim +} + +type replaceFailRegistry struct { + archives.Registry + fail, loseReply bool + calls int +} + +func (r *replaceFailRegistry) Replace(ctx context.Context, rec archives.Record, expect int64) (int64, error) { + r.calls++ + if r.fail { + return 0, archives.ErrUnavailable + } + epoch, err := r.Registry.Replace(ctx, rec, expect) + if r.loseReply && err == nil { + r.loseReply = false + return 0, archives.ErrUnavailable + } + return epoch, err +} + type hookRegistry struct { archives.Registry beforePut func() diff --git a/sandboxd/pool/replica_test.go b/sandboxd/pool/replica_test.go index 91e5773e6..931576b8e 100644 --- a/sandboxd/pool/replica_test.go +++ b/sandboxd/pool/replica_test.go @@ -1261,6 +1261,125 @@ func TestReplicaArchiveWhoseJournalWriteFailsKeepsTheClaimAcrossARestart(t *test } } +func TestReplicaEnvRotationOfAHibernatedClaimReachesTheAdopter(t *testing.T) { + c := newReplicaCluster(t) + a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) + sb := mustClaim(t, a, testKey) + patchHostOnly(t, a, sb, "old") + replicateNow(t, a, sb) + if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatalf("hibernate: %v", err) + } + replica := sb.ReplicaCk + + patchHostOnly(t, a, sb, "new") + if got := rowEnv(t, c, sb); got != "new" { + t.Errorf("row env %q after a rotation, want new", got) + } + rec, _, err := c.reg.Get(t.Context(), sb.ID) + if err != nil { + t.Fatalf("row: %v", err) + } + if rec.Epoch != a.epochOf(sb) || rec.Checkpoint != replica || !rec.Deadline.IsZero() { + t.Errorf("row %+v claim epoch %d, want it on the replica %s at the claim's epoch with no deadline", rec, a.epochOf(sb), replica) + } + + c.expire("a") + if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { + t.Fatalf("adopt: %v %v", adopted, err) + } + held, ok := b.byID(sb.ID) + if !ok { + t.Fatal("adopter has no claim") + } + if got, _ := held.HiddenEnv("GH_TOKEN"); got != "new" { + t.Errorf("adopter holds %q, want the rotated value", got) + } +} + +func TestReplicaEnvWriteOfARunningClaimLeavesTheRowToTheRound(t *testing.T) { + c := newReplicaCluster(t) + a := c.node(t, "a", t.TempDir(), keepAnHour) + sb := mustClaim(t, a, testKey) + replicateNow(t, a, sb) + reg := &replaceFailRegistry{Registry: c.reg, fail: true} + a.archives = reg + patchHostOnly(t, a, sb, "new") + if reg.calls != 0 { + t.Errorf("%d registry replaces for a running replicated claim, want 0", reg.calls) + } +} + +func TestReplicaRenewOfAHibernatedClaimReachesTheAdopter(t *testing.T) { + c := newReplicaCluster(t) + a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) + sb := mustClaim(t, a, testKey) + replicateNow(t, a, sb) + if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatalf("hibernate: %v", err) + } + + if _, err := a.Renew(t.Context(), sb.ID, Cred{Token: sb.Token}, 2*time.Hour, types.ExpireArchive); err != nil { + t.Fatalf("renew: %v", err) + } + if got := rowClaim(t, c, sb); got.OnExpire != types.ExpireArchive || got.LeaseSeconds != 7200 { + t.Errorf("row claim on_expire %q lease %d s after a renew, want archive and 7200", got.OnExpire, got.LeaseSeconds) + } + + c.expire("a") + if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { + t.Fatalf("adopt: %v %v", adopted, err) + } + held, ok := b.byID(sb.ID) + if !ok { + t.Fatal("adopter has no claim") + } + if held.OnExpire != types.ExpireArchive || held.LeaseSeconds != 7200 { + t.Errorf("adopter holds on_expire %q lease %d s, want the renewed archive and 7200", held.OnExpire, held.LeaseSeconds) + } +} + +func TestReplicaRenewOfAHibernatedClaimWithTheRegistryDownIsLoudAndARetryRepairs(t *testing.T) { + c := newReplicaCluster(t) + a := c.node(t, "a", t.TempDir(), keepAnHour) + sb := mustClaim(t, a, testKey) + replicateNow(t, a, sb) + if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatalf("hibernate: %v", err) + } + reg := &replaceFailRegistry{Registry: c.reg, fail: true} + a.archives = reg + + if _, err := a.Renew(t.Context(), sb.ID, Cred{Token: sb.Token}, 2*time.Hour, types.ExpireArchive); !errors.Is(err, archives.ErrUnavailable) { + t.Fatalf("renew with the registry down: %v, want ErrUnavailable", err) + } + if got := rowClaim(t, c, sb); got.OnExpire == types.ExpireArchive { + t.Error("row took the renew while the registry was down") + } + reg.fail = false + if _, err := a.Renew(t.Context(), sb.ID, Cred{Token: sb.Token}, 2*time.Hour, types.ExpireArchive); err != nil { + t.Fatalf("resend: %v", err) + } + if got := rowClaim(t, c, sb); got.OnExpire != types.ExpireArchive || got.LeaseSeconds != 7200 { + t.Errorf("row claim on_expire %q lease %d s after the resend, want archive and 7200", got.OnExpire, got.LeaseSeconds) + } +} + +func TestReplicaRenewOfARunningClaimLeavesTheRowToTheRound(t *testing.T) { + c := newReplicaCluster(t) + a := c.node(t, "a", t.TempDir(), keepAnHour) + sb := mustClaim(t, a, testKey) + replicateNow(t, a, sb) + reg := &replaceFailRegistry{Registry: c.reg, fail: true} + a.archives = reg + if _, err := a.Renew(t.Context(), sb.ID, Cred{Token: sb.Token}, 2*time.Hour, types.ExpireArchive); err != nil { + t.Fatalf("renew: %v", err) + } + if reg.calls != 0 { + t.Errorf("%d registry replaces for a running replicated claim, want 0", reg.calls) + } +} + func restartWithoutVM(t *testing.T, c *failoverCluster, a *Manager, sb *types.Sandbox, deleteAfter int) *Manager { t.Helper() if err := a.FlushClaims(); err != nil {