sandboxd: evacuate a drained node so peers adopt its claims at once

docs/cluster.md

@@ -186,7 +186,8 @@ it while it sleeps write it, so the claim path is unchanged.
   not hold the claim reads the row; when the token hash matches, it takes the
   row with one compare-and-set on the epoch. The same statement requires the
   owner's [node lease](#node-lease) to be expired, unless the owner is this
-  node (with its record gone). The node records the claim as archived and
+  node (with its record gone) or the row has no owner, since its owner
+  released it in an [evacuation](#evacuating-a-node). The node records the claim as archived and
   answers as the owner. Two nodes adopting at once leave one owner; the other
   answers 404. A tombstone is never adopted. The next call that reaches the
   guest wakes the claim from the store as usual. An owner with a live lease answers for
@@ -262,6 +263,49 @@ never returns leaves behind only the tombstones and checkpoints of the claims it
 was waking or releasing when it died, plus what a replicated claim leaves (see
 [running claims](#running-claims-across-node-loss)).
 
+### Evacuating a node
+
+[`POST /v1/drain?evacuate=1`](sandboxd-api.md#post-v1drain) empties a node for
+maintenance or retirement without losing its claims. The node cordons as a
+drain does, then moves each claim with bounded concurrency, on the maintenance
+budget, archived claims first and running ones last: it hibernates a running claim, archives a hibernated one through the
+archive path above, releases the row, and asks a peer to adopt it. A release is
+one compare-and-set: the row this node owns at the epoch it holds moves to no
+owner at a new epoch. A row with no owner has no lease to wait out, so a peer
+takes it with the same epoch compare-and-set as any adoption, and two peers
+adopting at once still leave one owner. This node keeps the claim listed until
+a peer answers `GET /v1/sandboxes/{id}/owner`, so a root listing finds the claim
+on at least one node at every moment, then forgets it under the claim's transition lock
+and answers 404 for it; the checkpoint stays, since it belongs to the adopter.
+With no adopter (no peer, every peer draining or unreachable) the node takes the
+row back and the claim stays here. Until the hand-off, a wake of a claim the
+evacuation still has to move answers 503 with `Retry-After`, so a client call
+cannot wake it back onto the node. A running claim keeps serving until its turn.
+The run takes the budget one batch at a time, so crash recovery and reaps of the
+claims that stay keep their turn while it runs.
+
+The release keeps the fencing intact. Every later write of the old owner is
+refused: a put is owner-guarded, and a replace, consume or second release
+needs the epoch the release moved on. A release that finds its row moved
+reads it: a row with no owner is one an outage cut short or a later owner
+released, so the node asks a peer to adopt it and never takes it back; a row another node took means
+the claim moved already, and the node drops it; a row this node still owns at
+another epoch keeps the claim, which is skipped. A crash between the release and a peer's answer leaves a row no node
+lists: a restart drops the claim, and the next `Lookup` with its token adopts it,
+so a holder that recovers the token from a root listing loses that claim. A
+released row past its retention deadline is purged by any
+node's hourly sweep, like the row of a departed owner.
+
+An archive the sweeps already run for a claim is waited out and the claim
+retried every 5 seconds. Claims that cannot move are skipped and keep serving
+here; once `evacuating` is gone `claimed` counts them, and the log names each
+one with its reason. A client that holds a handle gets 404 from this node after
+the move, the same answer as for a released claim, and only a `Lookup` tells
+the two apart: the SDKs' `Close` does this itself. A reader-served connection
+gets the retry as an error frame, not a 503, and a preview URL minted before
+the move stops working until it is minted again. `DELETE /v1/drain` stops the evacuation: claims not yet started
+stay as they are, and one mid-archive finishes its archive but keeps its row.
+
 ### Node lease
 
 With `archive_failover` every node renews one row of `sandboxd_nodes (cell,

docs/deploy.md

@@ -157,7 +157,7 @@ the intersection of the two.
 | `disk_reserve` | unset | `{"free": "100G", "dirs": ["/srv/sbx/run"]}`: while the filesystem of any listed directory (default `data_dir`) has less than `free` available, checked every 2 s, the node refuses new claims, forks, checkpoint claims, wakes of hibernated or archived sandboxes, hibernates, checkpoint captures and promotes with 429, and parks refill; the idle sweep and a lease-end archive leave their sandboxes running and retry on the next tick; `/v1/info` reports it as at capacity and the mesh sees no warm VMs on it. Running sandboxes keep running. On an XFS project quota the free space is the project's. List cocoon's `run_dir` when it sits on another filesystem, since the copy-on-write disks grow there. Size `free` to at least two sandboxes' worst-case growth (copy-on-write disk plus memory file each) and keep the refusal above your disk alert's warning level, so the warning comes first: on a 640G project with 40G disks and 8G memory files, `100G` refuses at 84% used |
 | `idle_hibernate_seconds` | 0 (off) | node-wide idle policy for unpooled claims (template/checkpoint claims): a none-lane claim is hibernated once it has had no open data-plane connection (relay, buffered exec, preview) and no egress request in flight for this long; the clock restarts when the last connection closes, so a long command is never cut short. The next call that reaches the guest wakes it transparently. Per-pool `idle_hibernate_seconds` does the same for that pool's claims; pooled keys ignore the node-wide value, and egress pools reject it because they cannot resume safely. Opt in deliberately: a wake costs latency and the snapshot, so callers with their own idle logic must not pay twice |
 | `archive_after_seconds` | 0 (off) | tier below hibernation: a hibernated claim idle this long is checkpointed to the store and its local VM dropped, freeing the node entirely; the next call that reaches the guest restores it transparently (a checkpoint restore's latency) with a fresh lease of the length the claim was granted (the server default of 5m when it asked for none); a lease moves only when a client calls [`/renew`](sandboxd-api.md#post-v1sandboxesidrenew). Keep `idle_hibernate_seconds` under the leases your claims use, or a woken sandbox that idles again is destroyed by the reap before it can hibernate, its archive already consumed by the wake; one still in use when its lease ends is destroyed the same way, unless it claimed `on_expire: archive`, which archives at the lease end whatever this setting. Requires `idle_hibernate_seconds > 0` and must exceed it. Node-wide for unpooled keys; per-pool overrides for that pool |
-| `archive_delete_after_seconds` | 0 (keep) | purge an archived claim's store checkpoint this long after it was archived; the claim is then gone for good. On S3, chunks and manifests remain until [offline reclamation](#reclaiming-s3-storage). On archive this retention window replaces the live claim deadline; 0 clears the deadline so the archive is kept forever. Same node-wide/per-pool split |
+| `archive_delete_after_seconds` | 0 (keep) | purge an archived claim's store checkpoint this long after it was archived; the claim is then gone for good. On S3, chunks and manifests remain until [offline reclamation](#reclaiming-s3-storage). On archive this retention window replaces the live claim deadline; 0 clears the deadline so the archive is kept forever. An [evacuation](cluster.md#evacuating-a-node) keeps a longer live deadline, and a claim of a pool without an archive tier, whose `on_expire` is not `archive`, keeps its deadline. Same node-wide/per-pool split |
 | `archive_failover` | false | let another node adopt this node's archived claims once its node lease expires, and the running claims of pools with `replicate_seconds`: each archive and replication round also writes a row (token SHA-256, owner, epoch, checkpoint id, retention deadline, and the claim record with its env, `guest: false` values included, in plaintext) to the shared PostgreSQL, and a `Lookup` that reaches any node takes the claim over there (see [archived claims across node loss](cluster.md#archived-claims-across-node-loss)). Requires `meta_store` with a `cell` (rows are scoped by it, so meshes sharing one database never adopt from each other), a `checkpoint_store` of kind `s3` and a `mesh`, all checked at config load. The node also renews a lease row every 5 s (see [node lease](cluster.md#node-lease)). With the meta store down, archives, wakes, releases and env writes of archived claims, and env writes and renews of hibernated replicated claims, fail and retry, and replicated claims stop being served once the lease is overdue (see [running claims](cluster.md#running-claims-across-node-loss)); other running claims never touch it. Restart only |
 | `mesh` | unset | join a cluster ([Clusters](cluster.md)); unset = single node |
 | `pools[]` | — | warm pools, keyed by `(template, net, size)`. `warm` defaults to 4 only when both `warm` and `warm_max` are omitted; an explicit `warm: 0` is kept, so `warm: 0` under a `warm_max` starts the pool empty and grows it on demand, and `warm: 0` alone registers the key with no warm VMs (`PUT /v1/pools` never defaults `warm`); `net` is `none` or `egress`; `size` is a tier, below. Retune online without a restart via [`PUT /v1/pools`](sandboxd-api.md#put-v1pools) — omitted pools drain, and their goldens are deleted once they are idle (`warm: 0` keeps the golden). This is the **first-boot seed**: once a node takes a `PUT /v1/pools`, the applied set persists to `<data_dir>/pools.json` and overrides this section on every later boot (a startup log notes it); delete `pools.json` to return to config-owned pools. With a `meta_store` `cell`, the cell's set overrides it instead (see [shared pool set](#shared-pool-set)). Egress stays config-owned either way. A template re-pulled or re-imported under the same ref is picked up within a minute: the pool rebuilds its golden and replaces its warm VMs. See [state ownership](cluster.md#state-ownership) |
@@ -713,7 +713,11 @@ To empty a node for maintenance, cordon it first:
 [`POST /v1/drain`](sandboxd-api.md#post-v1drain) (root) stops new claims and
 drains the warm pools; poll `GET /v1/info` until `claimed` reaches zero (or
 let the leases expire), then stop sandboxd. `DELETE /v1/drain` uncordons.
-The drain is not persisted — a restarted node serves again.
+The drain is not persisted — a restarted node serves again. With
+`archive_failover`, `POST /v1/drain?evacuate=1` also moves the claims to the
+peers with their state, and a peer holds each one before this node drops it; poll `GET /v1/info` until `evacuating` is gone, and
+`claimed` then counts the claims it skipped (see
+[evacuating a node](cluster.md#evacuating-a-node)).
 
 ### Disabling archive failover
 

docs/sandboxd-api.md

@@ -518,6 +518,7 @@ success, 404 unknown id or wrong token, 409 a failed egress-lane or volume
 sandbox, or one whose VM record is gone, 403 a removed tenant's archive,
 429 while the disk reserve is low or the node is at [`max_running`](deploy.md#configuration), 500 when the cold boot or a restore fails,
 503 `lease lost; retry` for a replicated claim while its node's lease is overdue,
+503 `sandbox moving to another node; retry` while an [evacuation](#post-v1drain) has yet to move it,
 503 `archive registry down; retry` for an archived claim under `archive_failover` while the meta store is down.
 
 ## POST /v1/sandboxes/{id}/renew
@@ -784,10 +785,35 @@ or TTL. Pool ownership is untouched — no pools.json write, no config change.
 Answers the fresh `GET /v1/info` payload; poll `claimed` to zero to know the
 node is empty. Deliberately not persisted: a restarted node serves again.
 
+`POST /v1/drain?evacuate=1` cordons the same way, then moves the claims to the
+peers with their state, so the node empties without waiting for the leases to
+run out (see [evacuating a node](cluster.md#evacuating-a-node)). It needs
+[`archive_failover`](cluster.md#archived-claims-across-node-loss). In the
+background, with bounded concurrency, each claim is hibernated if it runs,
+archived to the checkpoint store if it is not archived, its registry row is
+released, and a peer is asked to adopt it through `GET /v1/sandboxes/{id}/owner`,
+so the root listing on that peer shows it with its token. Open streams to a
+claim end when it hibernates. Until a peer holds it, a call that would wake the
+claim answers 503 `sandbox moving to another node; retry` with `Retry-After`;
+after that this node answers 404 and the client looks the claim up again. With
+no peer to adopt it, the node takes the row back and the claim stays. A claim
+that cannot move stays here
+and keeps serving: an egress-lane claim, one with volumes, a failed one, one
+whose tenant was removed, one archived before `archive_failover` was on, one
+whose step failed, and every claim of a node without `archive_failover`. The
+answer and `GET /v1/info` count the claims left in `evacuating`; once it is
+gone, `claimed` counts the ones that stayed, and the log names each and why. A
+claim of a pool without an archive tier, whose `on_expire` is not `archive`,
+keeps its lease deadline across the move; any other keeps the later of its
+lease and `archive_delete_after_seconds`, or no deadline when that is 0. A call
+while an evacuation runs changes nothing; a call after one finished starts a
+new one over the claims still here.
+
 ## DELETE /v1/drain
 
-Auth: root only (tenant tokens get 403). Lifts the drain and kicks an immediate refill. Answers the
-fresh info payload.
+Auth: root only (tenant tokens get 403). Lifts the drain, stops an evacuation, and kicks an immediate refill. Answers the
+fresh info payload. A claim an evacuation is archiving finishes its archive and
+stays here archived; the next call wakes it.
 
 ## POST /v1/sandboxes/{id}/preview
 
@@ -1010,7 +1036,7 @@ verb never wakes one) or its image serves no instance metadata.
 
 Auth: root only (tenant tokens get 403). Prometheus text format,
 hand-rendered: pool warm/target gauges, claimed/running/max_running/hibernated/archived/failed/draining
-gauges, a per-tenant live-claim gauge (`sandboxd_tenant_claims{tenant="…"}`,
+gauges, `sandboxd_evacuating` (the claims an evacuation has yet to move), a per-tenant live-claim gauge (`sandboxd_tenant_claims{tenant="…"}`,
 tenants holding live claims only, so the label set is bounded by the node's claims), `sandboxd_config_digest_mismatch` on a mesh, claims
 by tier (warm/clone/cold),
 wake/hibernate/fork/checkpoint/promote/release/reap counters plus
@@ -1224,8 +1250,9 @@ Auth: the sandbox's own token. Answers `{"owner_addr": "host:port"}` when
 this node owns the sandbox, 404 otherwise. With
 [`archive_failover`](cluster.md#archived-claims-across-node-loss), a node that
 does not hold the sandbox adopts it when its row names an owner whose node
-lease expired, and answers 200; 503 with `Retry-After` when the meta store is
-down. A row whose owner's lease is live is never adopted: 404. A draining or fenced
+lease expired, or whose owner released it in an
+[evacuation](#post-v1drain), and answers 200; 503 with `Retry-After` when the meta store is
+down. A row whose owner's lease is live is never adopted unless the owner released it: 404. A draining or fenced
 node, or one that lifted its fence less than 8 seconds ago, does not adopt: 404. A node whose
 lease is overdue answers 404 for the replicated claims it holds, so the
 `Lookup` scatter moves on. With `client_advertise` configured,
@@ -1267,7 +1294,8 @@ and was not restarted (omitted at zero); all are included in `claimed`.
 `running` counts claims that hold a VM, with a wake in flight counted, and
 `max_running` is the [cap](deploy.md#configuration) on it (omitted when off).
 A node cordoned via [`POST /v1/drain`](#post-v1drain) additionally reports
-`"draining": true`. When refill is parked because the node cannot start
+`"draining": true`, and during an evacuation `evacuating` (claims left to
+move, omitted at zero). When refill is parked because the node cannot start
 another VM, `"at_capacity": true` distinguishes a full node from one still
 filling its warm target, and `at_capacity_reason` carries the engine's capacity
 reason. Both capacity fields are omitted while refill is not capacity-blocked. Each VM holds a console pty, so a node stops at `kernel.pty.max` (4096 by default) with `no space left on device`; raise it past the VM count you plan to run.

docs/sdk.md

@@ -227,8 +227,11 @@ are exported.
 `Sandbox.TemplateDigest` is the exact content identity when the claim cloned a
 promoted template; it is empty for other sources. `Owner()` names the owning
 node, and `Token()` returns the per-sandbox bearer to persist with `ID` for a
-later `Lookup`; `Close()` releases the sandbox (releasing one already gone is
-not an error, and `Close` is bounded internally so it stays defer-friendly).
+later `Lookup`; `Close()` releases the sandbox (it follows a sandbox that
+moved to another node with a `Lookup`, releasing one already gone is not an
+error, and `Close` is bounded internally so it stays defer-friendly). Any other
+call on a handle answers 404 once its sandbox has moved; run `Lookup` to tell
+that from a released sandbox.
 Volume sandboxes cannot hibernate, fork, checkpoint, or promote. Egress-lane
 sandboxes cannot either, nor take `WithArchiveOnExpire`; each answers 409. Passing
 `WithVolumes` or `WithClaimRef` to `Checkpoint.New` returns a local error:

sandboxd/archives/archives.go

@@ -1,4 +1,4 @@
-// Package archives keeps one cluster-wide row per archived or replicated claim, so another node can adopt the claim once its owner's lease expires.
+// Package archives keeps one cluster-wide row per archived or replicated claim, so another node can adopt the claim once its owner's lease expires or its owner releases the row.
 package archives
 
 import (
@@ -37,6 +37,8 @@ type Registry interface {
 	Replace(ctx context.Context, rec Record, expect int64) (int64, error)
 	// Take moves the live row at epoch expect to owner at a new epoch and returns it; another node's row moves only once its owner's lease expired.
 	Take(ctx context.Context, id string, expect int64, owner string) (int64, error)
+	// Release moves the live row owner holds at epoch expect to no owner at a new epoch and returns it, so any node can take the row at once.
+	Release(ctx context.Context, id string, expect int64, owner string) (int64, error)
 	// TakeOwned moves each live row owner holds at its epoch in expect to a new epoch in one call and returns the new epochs of the rows it moved.
 	TakeOwned(ctx context.Context, owner string, expect map[string]int64) (map[string]int64, error)
 	// Consume turns the live row at epoch expect into a tombstone at a new epoch and returns it.