review: follow-ups

docs/cluster.md

@@ -511,7 +511,7 @@ fields and deletes the row, so no third node can adopt a stale replica while
 the target runs the claim. It keeps the replica until the move ends. The target's
 first replica round puts a new row when the claim's carried interval comes due,
 from its `replica_at`, at once when the last round is older than the interval,
-and an aborted move lets the source's next round put it again. A source cut off from the
+and an aborted move makes the claim due at the source's next replica sweep, which puts it again. A source cut off from the
 registry changes nothing and its move fails: it keeps the row's epoch, so the
 fence still finds a peer that adopted the claim during the outage, and the source
 drops it. A consume whose reply was lost leaves the source's own tombstone, which
@@ -564,10 +564,13 @@ checkpoint store and keeps each claim's registry row on the latest snapshot:
   write that never landed otherwise. A refused write reads the row: one this node still owns only had a
   stale epoch and is refreshed, and an absent or foreign row drops the claim and
   its VM. A claim's interval runs from its last round, which `claims.json` keeps
-  as `replica_at` and a restart, a move or an adoption carries, so a restart does
-  not reset it and a claim whose last round is older than the interval is due at
-  the first sweep. A claim that has never replicated gets its first round within
-  one interval of the sweep first seeing it, at an offset taken from its id, so
+  as `replica_at` and a restart or a move carries, so a restart does not reset
+  it and a claim whose last round is older than the interval is due at the first
+  sweep. An adoption carries the round before the last, since a round writes the
+  row before it stamps its own time. A claim that has never replicated gets its
+  first round one interval after it was claimed; one claimed more than an
+  interval ago (a restart onto claims that never replicated) gets it within one
+  interval of the sweep first seeing it, at an offset taken from its id, so those
   first rounds spread instead of landing together. Until a claim woken from an
   archive or an adoption commits its first replica it has no row, and it is lost
   with its node. A hibernated, archived or failed claim,

docs/deploy.md

@@ -38,6 +38,21 @@ Roll the guest images with the daemon: a guest image from another release can
 read a different lane file. Each note names the release it starts from; a note
 for a release you already run does not apply.
 
+From v0.1.31, the `balloon` config block is gone: a config that still has it
+fails to load, so the node does not start and a reload fails. Remove the block
+on every node before the roll. Claims that v0.1.30 inflated stay inflated,
+both running idle claims and idle-hibernated ones, whose snapshots carry the
+target. To deflate the running ones, remove the block on v0.1.30 with a config
+[reload](#reloading-the-config), not a restart: the next sweep then deflates
+them. Then roll. After the roll, restore every claim whose balloon target is
+still above its default; `cocoon vm inspect` reports both as `balloon.target`
+and `balloon.default`. That covers claims that were hibernated when the block
+came off, which keep the inflated target when they wake, and running claims
+the reload did not deflate, such as one moved onto a v0.1.30 node without the
+block. Send each [`POST /v1/sandboxes/{id}/balloon`](sandboxd-api.md#post-v1sandboxesidballoon)
+with its `default` as `target`, waking a hibernated one first. That call now needs a
+`target` body: an empty `POST`, which v0.1.30 answered 202, answers 400.
+
 An S3 `checkpoint_store` written before v0.1.18 used another object layout
 (whole export files, now chunk manifests). Those records are not readable and
 there is no fallback reader: before starting the new binary, wake or release
@@ -175,12 +190,12 @@ the intersection of the two.
 | `audit_log` | false | append every relayed frame that opens an RPC — its op + addressing fields, never payloads — to `<data_dir>/audit.jsonl`; continuation `data`, `data_end`, `stdin`, and `stdin_close` frames pass through unrecorded. The file is size-rotated with one `.1` backup. Records are `{t, id, op}` plus whichever addressing fields the op carries (`argv`, `path`, `dest`, `from`, `to`, `url`, `session`, `port`), plus `method` (`GET`, `CONNECT`, `SOCKS5`, …), `decision` (`allow`, `deny`, `rate_limited`), `secret` (the ref name, never its value), `upstream` (the [upstream proxy](egress.md#upstream-proxies)'s `host:port`, when one carried it), `tenant`, and on an allowed connection `src_addr` and `dst_addr` (the node-side and destination `ip:port` of the upstream TCP connection, the tunnel target through an upstream proxy) on `egress` records, which a relayed frame cannot set; preview accesses record as op `preview`, one per request; a tenant change records as op `tenants` with the `added`, `changed` and `removed` names, never a token; a config reload records as op `config_reload` with `changed`, a runtime volume change as op `volumes` with `added` or `removed`, and each ended flow of the [advanced network mode](advanced-network.md) as op `tun_flow` with `path` (`tun`), `tenant`, `proto` (`tcp`, `udp`, `icmp`), `dst_addr`, `decision` (`allow`, `deny`), `sent` and `received` bytes, `start`, `end` and the exit `profile`; a guest DNS-relay query records as op `dns` ([fields](egress.md#name-resolution-dns)), a cordon and its lift as ops `cordon` and `uncordon`, and a guest-port relay as op `port` with `port`. An opening frame whose first line exceeds 4 KiB records as op `oversized` with no addressing fields. A record a root, operator or sandbox-token request caused carries `caller` (`root`, `operator` or `claim`), which a relayed frame cannot set; an operator's read of a claim token by id records as op `token` |
 | `disk_reserve` | unset | `{"free": "100G", "dirs": ["/srv/sbx/run"]}`: while the filesystem of any listed directory (default `data_dir`) has less than `free` available, checked every 2 s, the node refuses new claims, forks, checkpoint claims, wakes of hibernated or archived sandboxes, hibernates, checkpoint captures and promotes with 429, and parks refill; the idle sweep and a lease-end archive leave their sandboxes running and retry on the next tick; `/v1/info` reports it as at capacity and the mesh sees no pools on it. Running sandboxes keep running. On an XFS project quota the free space is the project's. List cocoon's `run_dir` when it sits on another filesystem, since the copy-on-write disks grow there. Size `free` to at least two sandboxes' worst-case growth (copy-on-write disk plus memory file each) and keep the refusal above your disk alert's warning level, so the warning comes first: on a 640G project with 40G disks and 8G memory files, `100G` refuses at 84% used |
 | `memory_limit` | unset | `{"anon": "300G", "psi_some_avg10": 40}`, either key alone: while anon, `AnonPages` plus used swap (`SwapTotal` minus `SwapFree`) less `SwapCached`, the swapped pages already back in memory, in `/proc/meminfo`, is above `anon`, or the `some avg10` of `/proc/pressure/memory` is above `psi_some_avg10`, checked every 2 s, the node refuses new claims, forks and checkpoint claims with 429 and parks refill; `/v1/info` reports it as at capacity and the mesh sees no pools on it, so a warm or pool redirect skips it; a template or volume held only there still redirects to it and answers 429. Hibernates, checkpoints and archives go on, and running sandboxes keep running. Wakes of its own claims go on too, except an archived claim's on a node with `archive_failover`: that wake is [handed to a peer with room](cluster.md#waking-an-archived-claim-on-a-full-node), and one no peer adopts wakes here anyway; a squeezed peer does not adopt it. Guest memory is a private file mapping, so page cache and clean guest pages are reclaimable and only anon is what a hibernate has to free. Both signals are node-wide; anon leaves out explicit hugepages and counts swapped shmem as anon. A signal the node cannot read, such as PSI on a kernel without it, logs an error and is ignored. PSI also counts a task stalled by its own cgroup limit and weighs by busy CPUs, so on a mostly idle node a few throttled tasks in one small cgroup can cross it: 16 tasks under a 64M `MemoryHigh` reached 75 on an idle 384-CPU host with memory to spare. Treat `anon` as the main limit, and set `psi_some_avg10` only where no workload runs under a tight memory cgroup. Size `anon` so the running guests' further growth still fits under physical memory |
-| `idle_hibernate_seconds` | 0 (off) | node-wide idle policy for unpooled claims (template/checkpoint claims): a none-lane claim is hibernated once it has had no open data-plane connection (relay, buffered exec, preview), no egress request in flight and no [advanced-network](advanced-network.md) packet carried for this long; the clock restarts when the last connection closes, so a long command is never cut short. The next call that reaches the guest wakes it transparently. Per-pool `idle_hibernate_seconds` does the same for that pool's claims; pooled keys ignore the node-wide value, and egress pools reject it because they cannot resume safely. Opt in deliberately: a wake costs latency and the snapshot, so callers with their own idle logic must not pay twice |
+| `idle_hibernate_seconds` | 0 (off) | node-wide idle policy for unpooled claims (template/checkpoint claims): a none-lane claim is hibernated once it has had no open data-plane connection (relay, buffered exec, preview), no egress request in flight and no [advanced-network](advanced-network.md) packet carried for this long; the clock restarts when the last connection closes, so a long command is never cut short. The next call that reaches the guest wakes it transparently. Per-pool `idle_hibernate_seconds` does the same for that pool's claims; pooled keys ignore the node-wide value, and egress pools reject it because they cannot resume safely. Opt in deliberately: a wake costs latency and the snapshot, so callers with their own idle logic must not pay twice. A clean restart keeps each claim's idle clock from the shutdown flush, so claims already idle past the limit hibernate at the first sweep; after a crash the clock starts again at the restart |
 | `archive_after_seconds` | 0 (off) | tier below hibernation: a hibernated claim idle this long is checkpointed to the store and its local VM dropped, freeing the node entirely; the next call that reaches the guest restores it transparently (a checkpoint restore's latency) with a fresh lease of the length the claim was granted (the server default of 5m when it asked for none); a lease moves only when a client calls [`/renew`](sandboxd-api.md#post-v1sandboxesidrenew). Keep `idle_hibernate_seconds` under the leases your claims use, or a woken sandbox that idles again is destroyed by the reap before it can hibernate, its archive already consumed by the wake; one still in use when its lease ends is destroyed the same way, unless it claimed `on_expire: archive`, which archives at the lease end whatever this setting. A call that wakes a hibernated claim past its lease, before the export, starts that fresh lease too. Requires `idle_hibernate_seconds > 0` and must exceed it. Node-wide for unpooled keys; per-pool overrides for that pool |
 | `archive_delete_after_seconds` | 0 (keep) | purge an archived claim's store checkpoint this long after it was archived; the claim is then gone for good. On S3, chunks and manifests remain until [offline reclamation](#reclaiming-s3-storage). On archive this retention window replaces the live claim deadline; 0 clears the deadline so the archive is kept forever. An [evacuation](cluster.md#evacuating-a-node) keeps a longer live deadline, and a claim of a pool without an archive tier, whose `on_expire` is not `archive`, keeps its deadline until its first wake. Same node-wide/per-pool split |
 | `archive_failover` | false | let another node adopt this node's archived claims once its node lease expires, and the running claims of pools with `replicate_seconds`: each archive and replication round also writes a row (token SHA-256, owner, epoch, checkpoint id, retention deadline, and the claim record with its env, `guest: false` values included, in plaintext, unless `host_env_memory_only` is on) to the shared PostgreSQL, and a `Lookup` that reaches any node takes the claim over there (see [archived claims across node loss](cluster.md#archived-claims-across-node-loss)). Requires `meta_store` with a `cell` (rows are scoped by it, so meshes sharing one database never adopt from each other), a `checkpoint_store` of kind `s3` and a `mesh` with a `cluster_key` (a hand-off presents the claim's token, or the root `api_token` for a claim whose token this node has not seen, to an address learned from gossip), all checked at config load. The node also renews a lease row every 5 s (see [node lease](cluster.md#node-lease)). With the meta store down, archives, wakes, releases and env writes of archived claims, and env writes and renews of hibernated replicated claims, fail and retry, and replicated claims stop being served once the lease is overdue (see [running claims](cluster.md#running-claims-across-node-loss)); other running claims never touch it. Restart only |
 | `mesh` | unset | join a cluster ([Clusters](cluster.md)); unset = single node |
-| `pools[]` | — | warm pools, keyed by `(template, net, size)`. `warm` defaults to 4 only when both `warm` and `warm_max` are omitted; an explicit `warm: 0` is kept, so `warm: 0` under a `warm_max` starts the pool empty and grows it on demand, and `warm: 0` alone registers the key with no warm VMs (`PUT /v1/pools` never defaults `warm`); `net` is `none` (the default) or `egress`; `size` is a tier, below (default `small`). Retune online without a restart via [`PUT /v1/pools`](sandboxd-api.md#put-v1pools) — omitted pools drain, and their goldens are deleted once they are idle (`warm: 0` keeps the golden). This is the **first-boot seed**: once a node takes a `PUT /v1/pools`, the applied set persists to `<data_dir>/pools.json` and overrides this section on every later boot (a startup log notes it); delete `pools.json` to return to config-owned pools. With a `meta_store` `cell`, the cell's set overrides it instead (see [shared pool set](#shared-pool-set)). Egress stays config-owned either way. A template re-pulled or re-imported under the same ref is picked up within a minute: the pool rebuilds its golden while claims keep getting the old image's warm VMs, and once the new golden lands each new warm VM retires one old VM. See [state ownership](cluster.md#state-ownership) |
+| `pools[]` | — | warm pools, keyed by `(template, net, size)`. `warm` defaults to 4 only when both `warm` and `warm_max` are omitted; an explicit `warm: 0` is kept, so `warm: 0` under a `warm_max` starts the pool empty and grows it on demand, and `warm: 0` alone registers the key with no warm VMs (`PUT /v1/pools` never defaults `warm`); `net` is `none` (the default) or `egress`; `size` is a tier, below (default `small`). Retune online without a restart via [`PUT /v1/pools`](sandboxd-api.md#put-v1pools) — omitted pools drain, and their goldens are deleted once they are idle (`warm: 0` keeps the golden). This is the **first-boot seed**: once a node takes a `PUT /v1/pools`, the applied set persists to `<data_dir>/pools.json` and overrides this section on every later boot (a startup log notes it); delete `pools.json` to return to config-owned pools. With a `meta_store` `cell`, the cell's set overrides it instead (see [shared pool set](#shared-pool-set)). Egress stays config-owned either way. A template re-pulled or re-imported under the same ref is picked up within a minute: the pool rebuilds its golden while claims keep getting the old image's warm VMs, and once the new golden lands and the warm set is back at target, each new warm VM retires one old VM. See [state ownership](cluster.md#state-ownership) |
 
 Size tiers (free-form CPU/memory is deliberately not accepted — it would
 fragment the warm pools):

sandboxd/pool/balloon.go

@@ -12,7 +12,7 @@ import (
 
 // SetBalloon sets a running claim's balloon target through cocoon and returns cocoon's report; the caller restores the reply's default.
 func (m *Manager) SetBalloon(ctx context.Context, id, target string) (types.VMBalloon, error) {
-	if !m.balloonReady() {
+	if !m.balloonReady(ctx) {
 		return types.VMBalloon{}, ErrBalloonOff
 	}
 	m.mu.Lock()
@@ -45,11 +45,14 @@ func (m *Manager) SetBalloon(ctx context.Context, id, target string) (types.VMBa
 	return b, err
 }
 
-func (m *Manager) checkBalloon() bool {
-	ctx := context.Background()
+func (m *Manager) balloonReady(ctx context.Context) bool {
+	if m.balloonOK.Load() {
+		return true
+	}
 	if err := m.eng.CheckBalloonSupport(ctx); err != nil {
-		log.WithFunc("pool.checkBalloon").Error(ctx, err, "the balloon verb is off")
+		log.WithFunc("pool.balloonReady").Error(ctx, err, "the balloon verb is off")
 		return false
 	}
+	m.balloonOK.Store(true)
 	return true
 }

sandboxd/pool/balloon_test.go

@@ -33,9 +33,11 @@ func TestSetBalloonRefusesWhatItCannotReach(t *testing.T) {
 	eng := newFakeEngine()
 	m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 6})
 	asleep, archived, failed, waking, capturing, arriving := mustClaim(t, m, testKey), mustClaim(t, m, testKey), mustClaim(t, m, testKey), mustClaim(t, m, testKey), mustClaim(t, m, testKey), mustClaim(t, m, testKey)
+	leaving := mustClaim(t, m, testKey)
 	m.mu.Lock()
 	asleep.HibernateSnap, archived.ArchiveCk, failed.Failed = "snap", "ck_00000000000000aa", "vmm exited"
 	waking.Waking, capturing.PendingSnap, arriving.MoveIn = true, "pending", true
+	m.moving[leaving.ID] = &outMove{}
 	m.mu.Unlock()
 	for _, tt := range []struct {
 		id   string
@@ -48,6 +50,7 @@ func TestSetBalloonRefusesWhatItCannotReach(t *testing.T) {
 		{waking.ID, ErrPaused},
 		{capturing.ID, ErrPaused},
 		{arriving.ID, ErrEvacuating},
+		{leaving.ID, ErrEvacuating},
 	} {
 		if _, err := m.SetBalloon(t.Context(), tt.id, "50%"); !errors.Is(err, tt.want) {
 			t.Errorf("set balloon of %s: %v, want %v", tt.id, err, tt.want)
@@ -56,6 +59,7 @@ func TestSetBalloonRefusesWhatItCannotReach(t *testing.T) {
 	m.mu.Lock()
 	asleep.HibernateSnap, archived.ArchiveCk, failed.Failed = "", "", ""
 	waking.Waking, capturing.PendingSnap, arriving.MoveIn = false, "", false
+	delete(m.moving, leaving.ID)
 	m.mu.Unlock()
 	if calls := balloonCalls(eng); len(calls) != 0 {
 		t.Errorf("balloon calls %q for refused claims, want none", calls)
@@ -90,6 +94,22 @@ func TestSetBalloonRefusesOnACocoonWithoutTheVerb(t *testing.T) {
 	}
 }
 
+func TestSetBalloonChecksTheVerbAgainAfterAFailedCheck(t *testing.T) {
+	eng := newFakeEngine()
+	eng.balloonSupportErr = errors.New("cocoon vm balloon --help: context deadline exceeded")
+	m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey})
+	sb := mustClaim(t, m, testKey)
+	if _, err := m.SetBalloon(t.Context(), sb.ID, "50%"); !errors.Is(err, ErrBalloonOff) {
+		t.Fatalf("set balloon while the check fails: %v, want ErrBalloonOff", err)
+	}
+	eng.mu.Lock()
+	eng.balloonSupportErr = nil
+	eng.mu.Unlock()
+	if _, err := m.SetBalloon(t.Context(), sb.ID, "50%"); err != nil {
+		t.Errorf("set balloon once the check passes: %v, want it set", err)
+	}
+}
+
 func balloonCalls(eng *fakeEngine) []string {
 	eng.mu.Lock()
 	defer eng.mu.Unlock()

sandboxd/pool/failover.go

@@ -669,6 +669,9 @@ func (m *Manager) fenceArchived(ctx context.Context, claims map[string]*types.Sa
 				drop = true
 			default:
 				if ok && !rec.Consumed {
+					if rec.Checkpoint == sb.ReplicaPending {
+						sb.ReplicaAt = time.Now()
+					}
 					sb.ReplicaCk, sb.ArchiveEpoch = rec.Checkpoint, rec.Epoch
 				}
 				sb.ReplicaPending = ""

sandboxd/pool/move_out.go

@@ -349,6 +349,9 @@ func (m *Manager) abortMoveOut(ctx context.Context, sb *types.Sandbox, mv *outMo
 	logger := log.WithFunc("pool.abortMoveOut")
 	m.mu.Lock()
 	vm, live := sb.VMName, m.claimed[sb.ID] == sb
+	if live && (mv.final || sb.ReplicaCk == "" && !sb.ReplicaAt.IsZero()) {
+		sb.ReplicaDue = true
+	}
 	m.mu.Unlock()
 	if live && mv.Series != "" {
 		if err := m.eng.MigrateAbort(ctx, vm, mv.Series); err != nil {