sandboxd: move a running claim to a named peer with cocoon live migration

cocoon now moves a running Cloud Hypervisor VM between nodes with its memory and disks (cocoonstack/cocoon#303: vm migrate capture|receive|abort). Cocoon captures and restores. The caller moves the files, picks the round count and decides which node owns the VM. This issue is that caller: a root-only move of one running claim to a named peer. It is a design for discussion; no code yet.

Cold and hot moves stay separate

	Cold: `POST /v1/drain?evacuate=1` (today)	Hot: `POST /v1/sandboxes/{id}/move` (this issue)
What moves	every claim of a draining node	one running claim
How	hibernate, archive to the checkpoint store, release the row, a peer adopts	memory and disk rounds straight to the target, the target restores and starts
Guest after the move	stopped until the next call wakes it	running, with its processes and memory
Target	any peer that answers	the one the caller names
When it cannot move	the claim is skipped and keeps serving	the call fails

evacuate does not change and never uses move. move never hibernates. A caller that wants hot moves calls move per running claim, then evacuate for what is left.

Decided

Requires archive_failover. The hand-off is a registry compare-and-set, under the existing node lease and fence.
Transfer. Peer to peer over the control port with the bearer token, like checkpoint_peer_heal, and it needs mesh.cluster_key in the same way. Files are opaque: sandboxd sends the files and the order each round's --output json lists, and reads none of them.
Fail fast. A move that cannot go ahead or fails before the hand-off aborts the series, so the source guest runs again. The target deletes what it received, and the call answers an error with the claim where it was. There is no hibernate fallback and no retry on another peer. The caller decides.

Flow

Admit on the target. The target admits the move like a new claim and holds a reservation through the transfer (see Counting). A refusal answers the call at once.
Rounds. Baseline, then a few deltas while each round shrinks. If the final round would exceed a downtime budget, the move aborts.
Final round. The source guest is paused. Calls that reach the guest answer 503 sandbox moving to another node; retry, as during an evacuation. Fork, checkpoint, promote, hibernate and the replica round are held off for the whole series, because a snapshot save ends it.
Receive. The target runs vm migrate receive, so its guest is paused, and journals the claim.
Hand-off. The target moves the row from the source to itself in one compare-and-set. This is the point of no return.
Start. The target runs vm start and records the claim's start.
Drop. The source drops the claim and runs vm rm --force. The client meets a 404 there and runs Lookup, as after an evacuation.

Connections do not survive: exec and pty streams, port forwards and egress connections through the doors end. Only the guest's processes and memory carry over.

The row, and why a new registry call

Checked against main 7fd17a2:

A running claim has a row only when it is replicated (replicate_seconds). An unreplicated running claim has none: archiveRowed needs an archive checkpoint, and replica rows need HasReplica.
The target cannot take the source's row while the source's lease is live. Take moves another node's row only after that lease expired, and Put is owner-guarded.
evacuate gets around this with Release and then Take. In between the row has no owner, and any node's Lookup may adopt it from rec.Checkpoint. That is safe for an archive, whose checkpoint is the claim's latest state. It is not safe for a live move: the source guest holds state newer than any replica, and a third node adopting the replica would start a stale copy.

Proposed: Registry.Hand(ctx, id, expect, from, to). It moves the live row from holds at epoch expect to to at a new epoch, in one compare-and-set, with no ownerless step. The target calls it with the epoch the source sent. Every later write of the source then fails, so the source's next replica Replace gets ErrLost and drops the claim even if the target's notice never arrives. It needs the PostgreSQL implementation, the in-memory archivestest implementation and the shared contract test.

Rows with no checkpoint. An unreplicated claim would need a row that names no checkpoint for the length of the move. Every row path assumes a checkpoint today. For example, reconcileOwnRow treats a claim as referencing a row when sb.ArchiveCk == rec.Checkpoint, which is trivially true for two empty strings. Adopt, purge, the intent settle, the replica round, unfence and the checkpoint sweep would each need a rule. Proposed for the first version: replicated claims only. Their row already exists, and every path understands it.

Counting

Each limit is per node today: node max_claims (len(m.claimed)), tenant max_claims (tenantLive, also per node), max_running (claims holding a VM, with a wake in flight counted) and the memory_limit gauge. Adoption skips the caps because a refusal could strand a claim. A move cannot strand one, because the source still holds it, so the target checks them all.

Claimed, running and tenant counts on source S and target T:

Step	S	T
Before	1 / 1 / 1	0 / 0 / 0
T admits	1 / 1 / 1	reserved 1 / 1 / 1
Rounds, final, receive	1 / 1 / 1	reserved 1 / 1 / 1
Hand-off	1 / 1 / 1	claim 1 / 1 / 1, swapped under one `m.mu`
S drops	0 / 0 / 0	1 / 1 / 1

The reservation holds until the hand-off. Without it, other claims or another move could take T's slot during a transfer of seconds, and the move would fail after the guest paused. The reservation is a counter that quotaErr and runningErr add in. It is not an entry in m.claimed, so listings, reaps and sweeps never see it.
S does not let go before the hand-off. An abort resumes the guest on S, so S must not give its slot to a new claim first.
The cluster counts the claim twice during the move, which is true: S runs the guest while T receives it. It can make nodes refuse earlier, never over-admit.
After the hand-off, S over-counts until it drops the claim, which takes at most until its next refused write if T's notice is lost.
memory_limit cannot be reserved, because it is a host gauge. T checks it at admission and again before the receive, and fails the move if it is over. A claim admitted between the two checks is the window left.
Usage. S records an end when it drops the claim and T records a start at vm start, so the paused window is not billed twice. Event names follow the usage folding rules and land with the docs.

Failures

Where	Outcome
Admission refused, or a round fails	`vm migrate abort` on S, the guest runs, T drops its reservation and files; the call fails
Final round or receive fails	the same
S and T lose each other before the hand-off	both time out: S aborts, T removes what it received; a restart of T removes a received VM no claim owns, as reconcile does for any stale VM
T crashes after the hand-off	the claim is T's; T's restart reconciles and starts it, S stays paused until it sees the row moved, then removes its VM
S crashes during the rounds	T's receive or hand-off fails and T drops what it has. S journals the move before the baseline, and its restart runs `vm migrate abort` for a journaled move, which resumes a paused guest and deletes the series; without that, the guest would stay paused while reading running, the window #433 records for hibernate

First version scope

Claim	First version	Why
`none` lane, replicated pool, CH, run dir on XFS	yes
Run dir on tmpfs (high-density pools)	refused	cocoon needs reflink
With volumes	refused	cocoon refuses hot-attached disks
Egress lane	later	see below
Firecracker	refused	cocoon is CH only
Hibernated or archived	refused, 409	that is the cold path

Egress lane, later. It cannot hibernate today because cocoon resumes the guest before the new tap can be locked. A receive leaves the guest paused, so the tap can be locked before vm start. The received VM gets new NICs with new MACs, though, and the guest's network recovery after that is not checked yet.

Prerequisites

A cocoon release that contains review: solo whole-repo round on d5d7887 (config seed fix, cuts, owner calls) #303. The CH fork at 35b098c65260 or newer; a VM keeps the VMM it started with, so older VMs need a restart first.
Same-model nodes. cocoon did not test TSC across hosts of different frequency.
sandboxd is the first real caller. cocoon's runs used the same root_dir and run_dir on both nodes, so different paths must be tested.

Out of scope

Automatic placement or rebalancing, and moves triggered by memory pressure: the caller decides where a claim goes.
Keeping open connections across a move, or SDK retries beyond today's Lookup.

Open questions

First version for replicated claims only, as proposed above, or the row with no checkpoint now?
The downtime budget for the final round: fixed, configurable, or a request field?
A cap on concurrent moves per node: reuse the maintenance budget, or a separate one?

Acceptance

Two bare-metal nodes: move and move back with memory stamps and disk checksums intact, and the guest's downtime measured directly.
Fault injection: kill the source, the target and the transfer at each step above, and show one owner, the counts on both nodes back to their expected values, and no leftover VM or files.
Each refusal in the scope table answers before any pause.
Warm claim and release on both nodes unchanged in an interleaved A/B, because the reservation counter sits on the admission path.

Decisions on the open questions:

Downtime budget: a request field on the move. A move whose final round would exceed it aborts and fails, with the guest running on the source. When the field is omitted, a node default applies, and that value is set after the downtime is measured on hardware.
Concurrency: a budget of its own, not the maintenance budget, as a node-wide config field that reloads without a restart. It counts the node's moves in both directions. A move past it answers 429 at once. Lowering it refuses new moves until the count falls under it and does not stop moves in flight, the same as max_running.

Still open: whether the first version covers only replicated claims. A claim is replicated when its pool entry sets replicate_seconds. Each interval its guest pauses for a memory and disk capture that goes to the checkpoint store, and its registry row names that replica. Such a claim already has a row, so Hand just moves it. Any other running claim has no row, so the alternative is a row that lives only for the length of the move and names no checkpoint. That row needs a rule on each path that assumes a checkpoint:

adoption refuses it, because there is nothing to restore;
reconcileOwnRow must not count an empty checkpoint as a reference;
purge skips the checkpoint delete;
a fenced source aborts the move;
the target deletes the row after the hand-off.

A source lost mid-move then loses an unreplicated claim, as it does today.

Design change: a move does not use the archive registry. A checkpoint row records where a stored checkpoint lives and who may restore it. A move hands a live guest from one node to another. These are two concepts, so the move gets its own record. It reuses the registry's compare-and-set and epoch code and the node lease, but none of the rows. This replaces Registry.Hand and the row with no checkpoint, and covers replicated and unreplicated claims the same way.

The move record

It lives in its own table beside the archive rows, with a PostgreSQL implementation, an in-memory test implementation and one contract suite. One record per claim: id, from, to, state, epoch, updated_at. States, each change a compare-and-set:

From	To	Who	When
(none)	`pending`	source	before the baseline; one live record per claim
`pending`	`final`	source	the final round paused the guest
`final`	`committed`	target	after the receive; the point of no return
`pending` or `final`	`aborted`	source	before it resumes the guest

committed and aborted exclude each other. That replaces the row hand-off:

The target commits only from final, so it never starts a guest restored from a round the source kept running after.
The source resumes only after its aborted write lands. A write that finds committed means the claim moved: the source drops it and removes its VM. While the database is down, a guest paused in the final round stays paused. The caller sees the move fail, and the guest resumes once the abort lands.
Crash recovery reads the record. A target restart with committed starts the received VM, and with anything else removes it. A source restart with committed drops the claim, and with pending or final writes aborted, then resumes. A source journals the move locally before pending, so its restart knows to look.
Cleanup. The target deletes a committed record once the source has dropped the claim. An aborted record is deleted by the source. Any node's hourly sweep removes a record left past a bound, as for departed owners' rows.

Requires archive_failover, as decided, for the meta store and the node lease. It does not use the archive rows.

Replicated claims

A replicated claim's archive row names its latest replica. Left in place, that row lets a third node adopt the stale replica once the source's lease expires, while the target runs the claim. So the source deletes the claim's row and replica before the final round. The target's first replica round puts a new row, one interval after the start, like any new claim. An abort before final lets the source's next round put the row again. In both cases the claim goes one interval without a replica, the same as a claim just made.

Unchanged

The counting table, the failure matrix apart from the rows above, the fail-fast rule, the downtime budget as a request field and the separate reloadable move budget all stand as written above.

Open question 1 is closed: the first version moves replicated and unreplicated claims alike.

PR #445 implements this. The design moved after these comments: the record gained a started state, a source resumes only on proof that its move stayed, and only the source deletes a record. docs/cluster.md in the PR is the current statement.

The first version no longer requires archive_failover: a move needs meta_store with a cell for its record and mesh.cluster_key. S3 and the node lease stay requirements of archived and replicated claims only.

Follow-up proposal after #445: recover exec and terminal sessions across a live move. The connection may close; the SDK finds the owner and attaches to the same guest task. The expected result is a pause in output, with the same process, shell state and terminal after it.

The move's ownership protocol stays as implemented. Its current contract is docs/cluster.md at the merge.

What needs to change. Guest memory carries silkd's task state, but it does not carry the source sandboxd's relay sockets. Today a foreground exec is killed when writing to a disconnected client fails; a silent command can still finish. A pty disconnect kills its shell. Detached processes and persistent shell sessions already outlive connections, but their existing interfaces do not provide complete stream recovery.

Object	Proposed recovery
Exec	Keep the task in silkd; reconnect to its ID and recover output and exit status
Pty	Keep the shell and pty master; reconnect to the same terminal, including its size
Persistent shell session	Keep the session ID and shell state; give each command its own operation ID and output cursor
Port forward, preview WebSocket, LSP, file watch	Keep their current disconnect behavior in this stage; each needs its own recovery contract

Task lifetime. silkd owns the task independently of the relay. A transport loss detaches the client. Explicit cancel or close ends the task; an abandoned attachment has a bounded grace period before cleanup. Completed tasks retain their output and exit status for a bounded period. Existing detached jobs keep their existing lifetime policy. The first increment should make this behavior explicit in the API, then wire it into SDK recovery.

Task IDs must distinguish task lifetimes; a reused PID must never attach a caller to a different process. Keep the task table, replay buffers and attachment state in the guest, where the move already carries them. No new session registry or per-frame database write is needed.

Reconnect flow.

On a broken transport, discard the affected connection and resolve the claim's owner with Lookup. A cached connection to the old owner is no longer usable.
Retry discovery and attachment through the final round's temporary unavailability, with backoff and one caller-controlled recovery deadline. Stop on authentication failure, an expired task or cancellation.
Attach by task ID and the last output cursor delivered to the caller. On an aborted move this can be the source again; on a committed move it is the target.
Resume the existing operation. Do not resubmit the command as a new exec. A recovery timeout reports the task ID and leaves its outcome explicit; it must not claim the command failed or was canceled.

The lost started reply needs the same rule. Allocate an operation ID before sending a start request and register it before spawning the task. Duplicate starts with that ID return the existing task; a conflicting request is rejected. Recovery looks up the ID without creating it. An absent or expired record reports an unknown outcome and never starts a replacement command.

Stream recovery.

Output: use a monotonic sequence over stdout, stderr and the terminal exit event. Attach supplies the last delivered sequence. silkd replays the retained suffix and switches to live output without a gap between replay and subscription. Repeated attachments at the same cursor are safe.
Bounds: cap retained output per task and across the guest. If the requested cursor was evicted, return an explicit gap with the earliest available cursor. Never report a seamless resume after silently dropping bytes.
Input: number input frames and acknowledge acceptance into a bounded guest-owned queue, with backpressure when full. The accepted position and pending bytes move with the guest. On reconnect, resend only unacknowledged frames; duplicates do not reach stdin twice. Serialize acceptance and delivery, including partial writes. Apply the same ordering to stdin-close; replaying the latest terminal size is idempotent.
Attachment: authenticate attachments with the existing claim credentials and allow one input writer per task. Reattachment replaces the previous writer, and late input or cleanup from the old attachment cannot affect the new one. Read-only observers can remain separate.
Cancellation: retry an explicit cancel against the resolved owner and the same task ID. Transport recovery alone must not send a kill. If cancellation cannot be confirmed, report that outcome.

These guarantees cover reconnection while the same guest survives, including a live move. They do not promise exactly-once effects after a guest loss, restore from an older checkpoint, or SDK process crash.

Delivery and acceptance.

Step	Completion criterion
Guest task lifetime	Break a relay during exec and pty use; reconnect manually to the same task. Verify shell state, explicit close, grace expiry and bounded cleanup
Replay protocol	Cut after start acceptance but before its reply, during output replay, after input acceptance but before its ack, and after exit but before delivery. Verify one task, ordered output, input applied once and recoverable exit status; buffer overflow reports a gap
Go and Python SDKs	Move to a peer and back while streaming output and typing into a pty. The same public handle resumes; an aborted move reconnects to the source. Cancellation, deadlines and stale connections follow the rules above

Measure the time until the first resumed output separately from VM downtime. Exercise the writer replacement race and repeated moves, and check that tasks, relays and buffers are reclaimed after close or expiry.

Keeping the client's TCP/WebSocket open would require a stable proxy outside the moving nodes, plus this same backend recovery protocol. That can follow if needed. Egress TCP/NAT state and arbitrary application streams stay outside this stage.

Update after #455 (merged 14b4692): the move's contract changed in six places. docs/cluster.md at that commit is current.

Release during a move answers 503 without archive_failover too. Before, a node without archives released the claim (204) while the move still committed, so the released sandbox ran on the target (8d0d678).
The target refuses at admission when its cocoon cannot migrate. Only the source checked before, so a move to an older cocoon streamed every round and paused the guest first (be5e432).
A target that starts draining during the rounds refuses the commit, so evacuation never misses a claim that lands late (2b6dd05).
A failed vm migrate abort answers with the move's own error, not 503 "move registry down", and the abort keeps retrying (b1b7fd8).
A target restarted after a host crash between commit and start hands the claim to liveness instead of treating cocoon's cold boot as the moved guest: the restart is recorded and the guest env is rewritten. An inspect error at that restart retries; it no longer starts the guest (7db8477, c4b161b).
The source journals the target, so a source restarted mid-move drops the target's reservation at once instead of holding a move_concurrency slot until the 2-minute idle drop (58204dc). The move transport is installed before reconcile (8b580a9).

Each change ran on two hardware nodes against the pre-#455 binary as a negative control. The move's crash windows (source or target killed at a round, the final capture, the commit, the start) also ran: each one ends with exactly one owner.

Still outstanding before live move is used in production: a boundary-scale run. Everything measured so far moved an idle 512 MiB guest between two nodes, at 180-330 ms downtime with a baseline round of about 11.5 s. Three things are untested: an 8-16 GiB guest under a memory-writing load (whether the deltas converge inside max_downtime_ms, and whether a round hits its per-round budget of 1 minute plus 10 MB/s of transfer time), two moves in flight at move_concurrency 2, and a move off a loaded node. #463 tracks the hashing cost that run should break out of the final round. #457 and #459-#462 did not change the move contract described above.

Implemented by #445: POST /v1/sandboxes/{id}/move moves one running claim to a named peer with cocoon's live migration. It differs from this design in one point: a move keeps its own record in the meta store (a cell and mesh.cluster_key are needed) instead of a registry compare-and-set, so it does not require archive_failover or an S3 checkpoint_store. docs/cluster.md describes it; #463 tracks the hashing cost on a large guest, and the boundary-scale move test is still to run.

Navigation Menu

Repository navigation

sandboxd: move a running claim to a named peer with cocoon live migration #443

Cold and hot moves stay separate

Decided

Flow

The row, and why a new registry call

Counting

Failures

First version scope

Prerequisites

Out of scope

Open questions

Acceptance

The move record

Replicated claims

Unchanged

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

sandboxd: move a running claim to a named peer with cocoon live migration #443

Description

Cold and hot moves stay separate

Decided

Flow

The row, and why a new registry call

Counting

Failures

First version scope

Prerequisites

Out of scope

Open questions

Acceptance

Activity

CMGS commented on Nov 4, 2026

CMGS commented on Nov 4, 2026

The move record

Replicated claims

Unchanged

CMGS commented on Nov 4, 2026

CMGS commented on Nov 4, 2026

CMGS commented on Nov 5, 2026

CMGS commented on Nov 5, 2026

CMGS commented on Nov 5, 2026

CMGS commented on Nov 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions