pool: answer a release once the claim is durably dropped #411

POST /v1/sandboxes/{id}/release holds the caller until the VM is gone. Nothing the caller can observe depends on that for a claim with no volume, so this asks whether a release should answer once the claim is durably dropped.

Today

releaseResolved (sandboxd/pool/claim.go) drops the claim from memory and commits claims.json. From that point the claim is gone, and the code says so: cleanup must survive the caller hanging up; the claim is already dropped. The caller still waits for:

• quiesceVolumes: the guest unmount and flush of each mounted volume
• removeOrRetry: one cocoon vm rm --force
• Disarm, dropSnap and the release usage event

Measured release p50: 162-201 ms across the last rounds on two bare-metal nodes, with single runs up to 870 ms.

release_delay_seconds already defers the removal, but as a different feature: the VM waits for the delay and then for the next reap tick (5 s), so it holds its memory that long, on the refill_concurrency budget.

Proposal

For a claim with no writable volume, answer after the claims.json commit and run the rest in the background, dispatched at once rather than on the reap tick. Expected caller latency is the commit, about a millisecond, but this is not measured yet.

A claim with a writable volume keeps waiting. Its caller may attach that volume elsewhere right after the reply, so the clean unmount has to be done by then.

Hot path: the claim path is untouched. A release gains one goroutine or one queue push.

Questions to settle

1. Failure reporting. Today vm %s survived removal reaches the caller as an error, although the claim is already dropped and the removal is queued for retry. With an early answer it becomes a log line only. Does any caller act on that error?
2. Capacity. The VM holds its memory after the reply, without counting as a claim. A client looping claim then release could outrun teardown, and disk_reserve and the refill_concurrency budget only see part of it. Does this need a bound on in-flight removals, falling back to inline removal when it is full?
3. Read-only volumes. Their hold is released in finishVolumeTeardown, after removal. Is a short delay before a retired volume's hold drops acceptable, or do these claims stay inline too?
4. Usage event. release closes compute at its t. Should t stay after the teardown, which bills the teardown, or move to the commit, which is the instant the caller was told it ended?
5. Egress. Disarm closes the proxy doors and unlocks the tap. Should the doors close before the reply, so that a released guest cannot use the proxy during teardown?
6. Shutdown. Claims are gone from claims.json, so a restart must not lose the removals in flight. The reconcile sweep of VMs no claim owns should already cover this, but it needs a check.

Not in scope: release_delay_seconds keeps its meaning.