pool: allow live writable and attach-only volumes

README.md

@@ -35,7 +35,7 @@ performance) — source in
 - `sandboxd/` — per-node control plane (Go): warm pools refilled from golden
   snapshot exports (online-retunable), claim/release/renew/hibernate/fork/
   promote/checkpoint HTTP API, operator-catalog dataset volumes (read-only or
-  writable; read-only ones change on a running claim), signed preview URLs,
+  writable, mounted or attach-only, with live add and remove), signed preview URLs,
   the HTTP-upgrade byte relays to silkd and to any guest port, read-only file
   access to hibernated or archived claims from a warm VM of their pool
   (opt-in per pool `reader`), claim env with

docs/deploy.md

@@ -145,7 +145,7 @@ gets the intersection of the two.
 | `reader` (pool entry) | `false` | Serve read-only file calls on this pool's hibernated or archived claims from a warm VM of the same pool instead of waking them; needs `warm` of at least 1 and a cocoon whose `vm disk attach` takes `--from-snapshot` and `--from-dir`. See [Reading sleeping claims](#reading-sleeping-claims-without-waking-them). Config-owned like `capture_trim`: `PUT /v1/pools` rejects it |
 | `egress_classes` | unset | named tenant egress layers: `[{"name": "desk", "egress": {…}, "egress_upstream_env": "DESK_UPSTREAM"}]`. A tenant names one with `egress_class` through the [tenant API](sandboxd-api.md#tenants-v1tenants) and its claims take that policy intersected with the pool's. `egress` is required; `intercept` stays a pool-rule flag. Reloadable: a class's policy reaches live claims on their next request. A class removed while tenants still name it leaves them with no egress and is logged at load and at reload |
 | `egress_upstream_env` (pool or egress class entry) | unset | names a node environment variable holding the default upstream proxy URL for the pool's or class's claims (class before pool, a claim's own entry before both); needs `egress_upstream`, and the URL must be on its `allow`. Read and checked at startup. Config-owned like `egress`: `PUT /v1/pools` rejects it |
-| `replicate_seconds` (pool entry) | 0 (off) | snapshot every running claim of this pool to the checkpoint store this often and point the claim's registry row at the snapshot, so another node restores the claim, memory and processes included, when this node is lost (see [running claims across node loss](cluster.md#running-claims-across-node-loss)). Each round pauses the guest for the length of a memory and disk capture. A restore resumes from the last committed replica and loses everything after it: up to the longer of the interval and one replication pass, plus the 5-second tick, plus one more pass, since a claim can run first in one pass and last in the next. A pass covers all the node's due claims of the replicating pools with bounded concurrency, so it grows with claim count and guest memory; the first round of a claim after a wake or adoption uploads the whole claim, so that pass is the long pass. The claim's row carries no retention deadline while it runs. Requires `archive_failover` and a `none`-lane pool, both checked at config load; a claim with volumes is skipped. Config-owned like `egress`: `PUT /v1/pools` rejects it. Restart only: a claim is replicated by its pool's setting from the moment it is claimed, so a reload that changes it is refused (a pool restarted with 0 drops its claims' rows and replicas on the first pass) |
+| `replicate_seconds` (pool entry) | 0 (off) | snapshot every running claim of this pool to the checkpoint store this often and point the claim's registry row at the snapshot, so another node restores the claim, memory and processes included, when this node is lost (see [running claims across node loss](cluster.md#running-claims-across-node-loss)). Each round pauses the guest for the length of a memory and disk capture. A restore resumes from the last committed replica and loses everything after it: up to the longer of the interval and one replication pass, plus the 5-second tick, plus one more pass, since a claim can run first in one pass and last in the next. A pass covers all the node's due claims of the replicating pools with bounded concurrency, so it grows with claim count and guest memory; the first round of a claim after a wake or adoption uploads the whole claim, so that pass is the long pass. The claim's row carries no retention deadline while it runs. Requires `archive_failover` and a `none`-lane pool, both checked at config load; a claim with volumes is skipped. Config-owned like `egress`: `PUT /v1/pools` rejects it. Restart only: a claim is replicated by its pool's setting from the moment it is claimed, so a reload that changes it is refused (a pool restarted with 0 starts retiring its replicas while the registry remains enabled; see [Disabling archive failover](#disabling-archive-failover)) |
 | `storage` (pool entry) | unset (cocoon's default, 10G) | size of the copy-on-write disk the pool's golden is cold-booted with, in cocoon's own `--storage` spelling (`"40G"`, `"40GiB"`, `"40Gi"`: Docker/Kubernetes units, all binary) and passed to it verbatim; every clone, fork, checkpoint branch and promoted template of the pool inherits it. At least cocoon's default of 10G. Equal sizes spelled differently build the same golden; a changed size rebuilds the golden on the next build, and claims already out keep their disk. Config-owned like `egress`: `PUT /v1/pools` rejects it |
 | `max_claims` | 0 (unlimited) | node-wide cap on live claims; claim/fork/branch requests beyond it answer 429 with the pool state unharmed (on a cluster a non-volume claim tries a warm-peer redirect first; a volume claim answers the 429 with no redirect) |
 | `audit_log` | false | append every relayed frame that opens an RPC — its op + addressing fields, never payloads — to `<data_dir>/audit.jsonl`; continuation `data`, `data_end`, `stdin`, and `stdin_close` frames pass through unrecorded. The file is size-rotated with one `.1` backup. Records are `{t, id, op}` plus whichever addressing fields the op carries (`argv`, `path`, `dest`, `from`, `to`, `url`, `session`, `port`), plus `method` (`GET`, `CONNECT`, `SOCKS5`, …), `decision` (`allow`, `deny`, `rate_limited`), `secret` (the ref name, never its value), `upstream` (the [upstream proxy](egress.md#upstream-proxies)'s `host:port`, when one carried it), `tenant`, and on an allowed connection `src_addr` and `dst_addr` (the node-side and destination `ip:port` of the upstream TCP connection, the tunnel target through an upstream proxy) on `egress` records, which a relayed frame cannot set; preview accesses record as op `preview`, one per request; a tenant change records as op `tenants` with the `added`, `changed` and `removed` names, never a token. An opening frame whose first line exceeds 4 KiB records as op `oversized` with no addressing fields |
@@ -248,9 +248,13 @@ the pool had `reader`, and its reads wake the claim as without the feature.
 
 ### Dataset volumes
 
-Running claims can change mounted read-only catalog entries through the
+Running claims can add or remove mounted and attach-only catalog entries in
+read-only or writable mode through the
 [GET, PUT, and PATCH volume API](sandboxd-api.md#get-v1sandboxesidvolumes)
-and both SDKs. The image must already exist on the owner node. Live changes
+and both SDKs. Changing a held entry's descriptor requires successful removal
+first. The [live volume contract](sandboxd-api.md#put-v1sandboxesidvolumes)
+defines admission policy and dirty-marker handling.
+The image must already exist on the owner node. Live changes
 require Cocoon v0.7.2 or newer for idempotent disk attach and detach. PUT and
 PATCH reject an older release, or a runtime whose version cannot be queried,
 with 503 before any intent, hold, or device change; a dev build is assumed
@@ -337,7 +341,8 @@ below. With `directio=off`, readers share the host page cache; use
 `directio=on` when cache interference matters.
 A dataset mounted into an egress-lane sandbox can be uploaded wherever that
 tenant's egress policy permits, so treat the ACL and egress policy as one access
-decision. Image replication, detach, and refcounting remain out of scope.
+decision. Operators remain responsible for image distribution and cross-node
+ownership. sandboxd tracks local holds and handles live detach through the volume API.
 
 #### Writable dataset volumes
 
@@ -705,6 +710,36 @@ drains the warm pools; poll `GET /v1/info` until `claimed` reaches zero (or
 let the leases expire), then stop sandboxd. `DELETE /v1/drain` uncordons.
 The drain is not persisted — a restarted node serves again.
 
+### Disabling archive failover
+
+Stop admitting claims that use `on_expire: archive`. Set the node and affected
+pools' `archive_after_seconds` to `0`, and use
+[`renew`](sandboxd-api.md#post-v1sandboxesidrenew) to switch existing active claims
+to `on_expire: destroy` while keeping their leases valid. These changes stop
+idle-time and expiry-triggered archives. Before setting `archive_failover` to `false`:
+
+1. Set every affected pool's `replicate_seconds` to `0` and restart its nodes
+   with failover and the registry still enabled. Keep the registry reachable
+   while replica retirement retries.
+2. Check each node's `<data_dir>/claims.json` for affected claims. Wait until
+   `replica_ck` and `replica_pending` are absent after successful journal writes.
+   Use `sync_claims: true` when completion must survive power loss.
+3. Wake or release existing registered archives while failover remains enabled.
+   Wait for `archive_pending` to clear in the journal. Verify that the affected
+   IDs have no rows in PostgreSQL's
+   `sandboxd_archives` table for your `meta_store.cell`. A consumed row is a
+   non-adoptable tombstone; wait for its cleanup retries to finish too.
+4. Set `archive_failover` to `false` and restart the nodes.
+
+The public API omits replica journal fields. A fixed delay or an absence of
+warnings does not prove retirement. Disabling failover first stops replica
+retirement and leaves residual references that reject new live RW attachments.
+A local archive can still wake and clear `archive_ck`, but that does not clear a
+retained `replica_pending` or retire an old shared row. Release can remove local
+state without reconciling that row. Re-enabling failover against stale rows can
+fence or drop a local claim if another node has adopted it. Resolve registry
+ownership before serving that claim again.
+
 ### Shared tenant database
 
 With `meta_store` set, every node reads tenants from one PostgreSQL table

docs/live-volumes.md

@@ -10,7 +10,8 @@ v0.7.2 or newer and set `SANDBOX_ADDR` and `SANDBOX_API_TOKEN` for the client.
 A sandbox token alone cannot manage volumes.
 
 Entries [registered at runtime](sandboxd-api.md#put-v1volumesname) work the
-same way.
+same way. The API also supports
+[writable and attach-only entries](sandboxd-api.md#put-v1sandboxesidvolumes).
 
 Pause application reads before replacement. Close files, mmap mappings, cwd,
 and bind mount references on the old disk. Replacement unmounts the old disk
@@ -61,9 +62,10 @@ with client.new("rt:24.04", ttl_seconds=900) as sb:
 
 A successful run reads `v1`, then `v2`, and preserves every identity assertion.
 The repeated full target verifies that a settled mount can be retained.
-This example starts without other volumes. If your sandbox has writable or
-attach-only volumes, retain them in a full PUT or use PATCH to replace only
-the dataset names. `None`, a nil Go slice, and a missing `volumes` field do not
+This example starts without other volumes. Retain any volumes you intend to
+keep in a full PUT, or use PATCH to replace only the dataset names. All mounted
+and attach-only entries support live removal in either mode.
+`None`, a nil Go slice, and a missing `volumes` field do not
 mean an empty target.
 
 Go uses the same owning handle and client API token.

docs/sandboxd-api.md

@@ -87,13 +87,12 @@ Auth: `Authorization: Bearer <api_token>` (when configured).
   defaults to `/volumes/<name>`; a custom value must be absolute and clean,
   outside the guest OS tree, unique, and non-nesting within the request.
   `mode` is `"ro"` (default, omitted) or `"rw"`; `"rw"` requires the catalog
-  entry's `writable: true` (see [deploy](deploy.md#dataset-volumes))
-- `volumes_attach_only` (default `false`) attaches every requested volume
-  without mounting it, handing the whole mount contract to the workload. It
-  requires at least one volume, and rejects any entry carrying a `mount` —
-  the path is meaningless when the caller mounts the device. Everything below
-  describes the default, eager behaviour, which is completely unchanged; the
-  attach-only contract is in [its own section](#attach-only-volumes)
+  entry's `writable: true` (see [deploy](deploy.md#dataset-volumes)).
+  `attach_only: true` attaches that entry without mounting it and requires an
+  empty or omitted `mount`. Mounted and attach-only entries can share a list
+- `volumes_attach_only` (default `false`) is the legacy claim-wide form of
+  `attach_only`. The server expands it onto every entry before validation.
+  It requires at least one volume and rejects a nonempty `mount`.
 
 Success:
 
@@ -135,12 +134,14 @@ an existing populated guest directory for the claim's life.
 
 ### Attach-only volumes
 
-`"volumes_attach_only": true` stops after the attach. The device appears in
-the guest, nothing is mounted, and the echoed entries carry no `mount` key:
+`"attach_only": true` stops after the attach for that entry. The device appears
+in the guest without a mount. Responses preserve `attach_only: true` and omit
+`mount`:
 
 ```json
 {"id": "sb_…", "token": "…",
- "volumes": [{"name": "imagenet"}, {"name": "scratch-db", "mode": "rw"}]}
+ "volumes": [{"name": "imagenet", "attach_only": true},
+             {"name": "scratch-db", "mode": "rw", "attach_only": true}]}
 ```
 
 Find each device by its virtio serial, which is the catalog name — poll this,
@@ -221,7 +222,7 @@ an unknown and a forbidden volume return the same error text.
 
 Errors: 400 unknown template axis, metadata over a bound (the message names
 it), an unknown `on_expire`, invalid/duplicate volumes,
-`volumes_attach_only` with no volumes or with an entry carrying a `mount`,
+`volumes_attach_only` with no volumes, or an attach-only entry with a nonempty `mount`,
 `mode: "rw"`
 against a non-writable entry, or a volume that is unknown or forbidden (the
 latter two are deliberately indistinguishable), or
@@ -354,22 +355,32 @@ Replaces the full desired set and converges synchronously. Auth matches GET.
 The owner tenant's catalog access list applies to newly admitted names, including
 for root. Retaining, repairing, or removing held names does not reauthorize
 them. A new image must exist on the owning node. The request must include a non-null `volumes`
-array. An explicit empty array removes all mutable entries.
+array. An explicit empty array removes all volumes.
 
 ```json
 {"volumes": [{"name": "imagenet", "mount": "/datasets/training"}]}
 ```
 
-Only mounted read-only catalog entries can change live (`mode` omitted or
-`ro`). A PUT must retain every existing writable or attach-only entry unchanged.
-The claim-time name, mount, and eight-volume limits apply to the desired set.
-An omitted mount defaults to `/volumes/<name>`, except that a held attach-only
-entry is retained by sending it as the claim returned it, without `mount`. To
-change an existing name's
-mount or mode, first remove that name and wait for successful convergence.
-An identical target can be retried. Claims with `on_expire: archive` must change
-that policy before adding a volume.
-
+Mounted read-only, mounted writable, attach-only read-only, and attach-only
+writable entries can change live, including mixed lists. The claim-time name,
+mount, and eight-volume limits apply. An omitted mount defaults to
+`/volumes/<name>` unless that entry has `attach_only: true`. Responses carry
+that flag directly, so a GET target can be sent back with PUT. Omitting the flag
+or sending `false` does not inherit an existing attachment's shape.
+
+To change a held name's mount, mode, or `attach_only`, first remove the name and
+wait for successful convergence. An identical target can be retried. Claims
+with `on_expire: archive` must change that policy before adding a volume.
+
+New live writable attachments require a non-replicating pool, including a
+promoted template's source pool. A replicating pool returns 409 before changing
+intent, holds, or devices. Initial claim-time writable volumes remain supported.
+An existing or pending replica, a pending archive, or this claim's unacknowledged checkpoint
+retirement, returns 503 with `Retry-After`. A replica or archive intent left
+with `archive_failover` disabled returns 409 because it cannot retire through
+the registry. A retained local checkpoint reference alone still returns 503.
+Volume mutation does not evict remote replicas. Read-only additions, exact
+retries of held writers, and removals remain available under these restrictions.
 The sandbox must be running. Paused, archived, and failed sandboxes return 409
 without waking. A failed sandbox holding volumes is release-only after its VMM
 exits; PUT with an empty target cannot restore its processes. Missing, invalid,
@@ -390,13 +401,30 @@ unchanged. An empty patch retries the current desired set.
 ```
 
 Adding an identical entry or removing an absent valid name is harmless.
-Duplicate names, a name in both arrays, or changes to writable or attach-only
-entries return 400. Removal refuses busy or foreign mounts and uses neither
+Duplicate names, a name in both arrays, or changes to a held entry's mount,
+mode, or `attach_only` return 400. Removal refuses busy or foreign mounts and uses neither
 lazy nor forced unmount. A foreign mount, another bind mount, or a completed
 ordinary unmount with nonzero exit returns 409 with the guest mount path and
 a safe per-volume reason. Guest transport failures remain server errors;
 timeouts return 504 and cancellation returns 408.
 
+A mounted writable attachment marks the image dirty before the guest can write.
+Live removal clears the marker only when this call observes and normally
+unmounts the exact writable whole-device filesystem, finds no residual device
+mounts, and confirms host detach. Both mount and superblock options must be
+writable. A missing device or pre-unmounted filesystem does not prove clean
+unmount. The marker remains on uncertainty and blocks subsequent readers.
+Attach-only writers have no automatic dirty-marker protection. The caller owns
+mounting, flushing, unmounting, recovery, and fencing before reuse.
+
+The per-entry field requires coordinated daemon and SDK releases. Older daemons
+ignore `attach_only` in claims and reject it in live PUT and PATCH with 400.
+Older Python SDKs reject the field when
+returned descriptors are reused in requests. Responses without the field do
+not safely roundtrip attach-only intent through new clients. The server still
+accepts the legacy claim-wide request flag and normalizes old persisted empty
+mounts when loading claims.
+
 ### Partial failure and recovery
 
 PUT and PATCH return the GET response shape on success. A convergence error