diff --git a/docs/cluster.md b/docs/cluster.md index e92411c3c..9fa5da1c9 100644 --- a/docs/cluster.md +++ b/docs/cluster.md @@ -172,6 +172,21 @@ client, _ := sandbox.Connect("10.0.0.6:7777", sandbox.WithAPIToken(token)) sb, err := client.Lookup(ctx, savedID, savedToken) ``` +## Node admission + +A node is in one of these states. `POST /v1/cordon` and `POST /v1/drain` set +them, and each `DELETE` lifts only its own; both are root-only (see the +[API](sandboxd-api.md#post-v1cordon)). A cap of `0` or an omitted cap means no +cap; to take no claims, cordon the node. + +| State | New claim, fork, branch | Existing claims | Root live move in | Adoption of archived claims | Refill and pool gossip | Survives restart and reboot | +|---|---|---|---|---|---|---| +| serving | yes, under `max_claims`, `max_running`, the tenant's `max_claims`, `disk_reserve` and `memory_limit`; a 429 for a claim without volumes redirects to a peer | yes; a wake is bound by `max_running` and `disk_reserve` | yes, under the same caps and `move_concurrency` | yes; no claim cap binds, `max_running` binds at the wake | yes | — | +| cordoned | no: 429 `node cordoned`; peers stop redirecting here | yes | yes, under the same caps and `move_concurrency` | no; its own claims stay | no; warm VMs are destroyed | yes, as `/cordoned` | +| draining | no: 429 `node draining`; peers stop redirecting here | yes | no: admission and commit refuse | no | no; warm VMs are destroyed | no | +| draining with `evacuate=1` | no, as draining | yes until the claim's turn, then 503 until a peer holds it and 404 after; a claim that cannot move stays and serves | no | no | no | no: a restart stops the evacuation | +| cordoned and draining | no: 429 `node cordoned` | yes | no | no | no | the cordon only | + ## Archived claims across node loss A claim lives on one node, so a lost node takes its running and hibernated @@ -195,7 +210,7 @@ it while it sleeps write it, so the claim path is unchanged. answers 404. A tombstone is never adopted. The next call that reaches the guest wakes the claim from the store as usual. An owner with a live lease answers for itself, and a node that tries to take its row is refused with a 404. - A draining node never adopts, nor does a fenced node or one that lifted its + A draining or [cordoned](sandboxd-api.md#post-v1cordon) node never adopts, nor does a fenced node or one that lifted its fence less than 8 seconds ago (one renewal interval plus one registry query), so after a registry outage every healthy owner renews before a peer can take its rows. Adoption checks no tenant `max_claims` or node @@ -269,8 +284,8 @@ was waking or releasing when it died, plus what a replicated claim leaves (see ### Evacuating a node [`POST /v1/drain?evacuate=1`](sandboxd-api.md#post-v1drain) empties a node for -maintenance or retirement without losing its claims. The node cordons as a -drain does, then moves each claim with bounded concurrency, on the maintenance +maintenance or retirement without losing its claims. The node stops taking +claims as a drain does, then moves each claim with bounded concurrency, on the maintenance budget, archived claims first and running ones last: it hibernates a running claim, archives a hibernated one through the archive path above, releases the row, and asks a peer to adopt it. A release is one compare-and-set: the row this node owns at the epoch it holds moves to no @@ -280,7 +295,7 @@ adopting at once still leave one owner. This node keeps the claim listed until a peer answers `GET /v1/sandboxes/{id}/owner`, so a root listing finds the claim on at least one node at every moment, then forgets it under the claim's transition lock and answers 404 for it; the checkpoint stays, since it belongs to the adopter. -With no adopter (no peer, every peer draining or unreachable) the node takes the +With no adopter (no peer, every peer draining, cordoned or unreachable) the node takes the row back and the claim stays here. Until the hand-off, a wake of a claim the evacuation still has to move answers 503 with `Retry-After`, so a client call cannot wake it back onto the node. A running claim keeps serving until its turn. @@ -390,7 +405,8 @@ guest resumes on the source unless the target committed. **Counting.** The target admits the move like a new claim: drain, `disk_reserve`, `memory_limit`, node `max_claims` and `max_running`, the -tenant's `max_claims` and `move_concurrency`, and holds a reservation from admission to its commit. The +tenant's `max_claims` and `move_concurrency`, and holds a reservation from admission to its commit. A +[cordon](sandboxd-api.md#post-v1cordon) refuses only new claims, so a cordoned target admits the move. The reservation counts in every cap but is no claim: listings, reaps and gossip never see it. At the commit the reservation becomes the claim under one lock, so the target never counts it twice. The source counts the claim until it drops diff --git a/docs/deploy.md b/docs/deploy.md index 25baddb7d..69ed5963f 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -158,7 +158,7 @@ the intersection of the two. | `egress_upstream_env` (pool or egress class entry) | unset | names a node environment variable holding the default upstream proxy URL for the pool's or class's claims (class before pool, a claim's own entry before both); needs `egress_upstream`, and the URL must be on its `allow`. Read and checked at startup. Config-owned like `egress`: `PUT /v1/pools` rejects it | | `replicate_seconds` (pool entry) | 0 (off) | snapshot every running claim of this pool to the checkpoint store this often and point the claim's registry row at the snapshot, so another node restores the claim, memory and processes included, when this node is lost (see [running claims across node loss](cluster.md#running-claims-across-node-loss)). Each round pauses the guest for the length of a memory and disk capture. A restore loses what ran after the last committed replica: [the bound](cluster.md#running-claims-across-node-loss). The claim's row carries no retention deadline while it runs. Requires `archive_failover` and a `none`-lane pool, both checked at config load; a claim with volumes is skipped. Config-owned like `egress`: `PUT /v1/pools` rejects it. Restart only: a claim is replicated by its pool's setting from the moment it is claimed, so a reload that changes it is refused (a pool restarted with 0 starts retiring its replicas while the registry remains enabled; see [Disabling archive failover](#disabling-archive-failover)) | | `storage` (pool entry) | unset (cocoon's default, 10G) | size of the copy-on-write disk the pool's golden is cold-booted with, in cocoon's own `--storage` spelling (`"40G"`, `"40GiB"`, `"40Gi"`: Docker/Kubernetes units, all binary) and passed to it verbatim; every clone, fork, checkpoint branch and promoted template of the pool inherits it. At least cocoon's default of 10G. Equal sizes spelled differently build the same golden; a changed size rebuilds the golden on the next build, and claims already out keep their disk. Config-owned like `egress`: `PUT /v1/pools` rejects it | -| `max_claims` | 0 (unlimited) | node-wide cap on live claims; claim/fork/branch requests beyond it answer 429 with the pool state unharmed (on a cluster a non-volume claim first tries a redirect to a warm peer, else to a peer that serves the pool; a volume claim answers the 429 with no redirect) | +| `max_claims` | 0 (unlimited) | node-wide cap on live claims (to take none, [cordon](sandboxd-api.md#post-v1cordon) the node); claim/fork/branch requests beyond it answer 429 with the pool state unharmed (on a cluster a non-volume claim first tries a redirect to a warm peer, else to a peer that serves the pool; a volume claim answers the 429 with no redirect) | | `max_running` | 0 (off) | node-wide cap on claims that hold a VM: not hibernated, not archived, not failed, with a wake in flight counted; claim, fork, branch and wake beyond it answer 429 and leave the claim as it was; a claim takes the same redirect as `max_claims`. Hibernated and archived claims hold no slot, so `max_claims` can follow disk while this follows RAM and CPU. Warm and reader VMs are not counted: keep `max_running` × the largest size plus every pool's warm target × its size within the node's RAM. A wake cannot move to another node, so its 429 stays until a running claim sleeps or goes. Lowering the cap stops nothing that runs. A [reload](#reloading-the-config) applies a change | | `move_concurrency` | 2 | node-wide cap on [live moves](cluster.md#moving-a-running-claim) in flight, counting moves this node sends and reservations it holds for moves to it; a move past it answers 429. Moves need `meta_store` with a `cell`, for the move record, and `mesh.cluster_key`, but not `archive_failover` or an S3 `checkpoint_store`: a node that sets this without them refuses to load, and one that leaves it unset without them cannot move (409). A move also needs a cocoon with live migration (v0.7.7 or newer; a master build is judged by the newest tag it reports), the Cloud Hypervisor fork at `35b098c65260` or newer (a VM started before the upgrade cannot move), a run dir that supports reflink such as XFS, nodes of the same CPU model, and the same path strings in cocoon's `root_dir` and `run_dir` on both nodes ([same paths](cluster.md#moving-a-running-claim)). Lowering the cap stops no move in flight. A [reload](#reloading-the-config) applies a change | | `audit_log` | false | append every relayed frame that opens an RPC — its op + addressing fields, never payloads — to `/audit.jsonl`; continuation `data`, `data_end`, `stdin`, and `stdin_close` frames pass through unrecorded. The file is size-rotated with one `.1` backup. Records are `{t, id, op}` plus whichever addressing fields the op carries (`argv`, `path`, `dest`, `from`, `to`, `url`, `session`, `port`), plus `method` (`GET`, `CONNECT`, `SOCKS5`, …), `decision` (`allow`, `deny`, `rate_limited`), `secret` (the ref name, never its value), `upstream` (the [upstream proxy](egress.md#upstream-proxies)'s `host:port`, when one carried it), `tenant`, and on an allowed connection `src_addr` and `dst_addr` (the node-side and destination `ip:port` of the upstream TCP connection, the tunnel target through an upstream proxy) on `egress` records, which a relayed frame cannot set; preview accesses record as op `preview`, one per request; a tenant change records as op `tenants` with the `added`, `changed` and `removed` names, never a token; a config reload records as op `config_reload` with `changed`, and a runtime volume change as op `volumes` with `added` or `removed`. An opening frame whose first line exceeds 4 KiB records as op `oversized` with no addressing fields | @@ -620,7 +620,7 @@ can delete only its own checkpoints and templates (root sees everything). Operator surfaces stay root-only — per-id sandbox reads and stats, `PUT/DELETE /v1/volumes/{name}`, `PUT /v1/sandboxes/{id}/instance-metadata`, `GET /v1/checkpoints/{id}/blob`, `GET /v1/info`, `PUT /v1/pools`, -`/v1/tenants`, `/v1/config/reload`, `POST/DELETE /v1/drain`, `POST /v1/sandboxes/{id}/move` and the peer `/v1/moves/{id}` calls, `/metrics`, `GET /v1/usage`; a +`/v1/tenants`, `/v1/config/reload`, `POST/DELETE /v1/drain`, `POST/DELETE /v1/cordon`, `POST /v1/sandboxes/{id}/move` and the peer `/v1/moves/{id}` calls, `/metrics`, `GET /v1/usage`; a tenant token there is authenticated but not authorized, so it answers 403 (a wrong token stays 401). Per-sandbox tokens are unchanged: whoever holds a sandbox's token drives that sandbox. @@ -766,16 +766,23 @@ PID (cocoonstack/cocoon#274) to never read a recycled PID as a live VMM. ### Maintenance -To empty a node for maintenance, cordon it first: +To empty a node for maintenance, drain it first: [`POST /v1/drain`](sandboxd-api.md#post-v1drain) (root) stops new claims and drains the warm pools; poll `GET /v1/info` until `claimed` reaches zero (or -let the leases expire), then stop sandboxd. `DELETE /v1/drain` uncordons. +let the leases expire), then stop sandboxd. `DELETE /v1/drain` lifts the drain. The drain is not persisted — a restarted node serves again. With `archive_failover`, `POST /v1/drain?evacuate=1` also moves the claims to the peers with their state, and a peer holds each one before this node drops it; poll `GET /v1/info` until `evacuating` is gone, and `claimed` then counts the claims it skipped (see [evacuating a node](cluster.md#evacuating-a-node)). +To keep a node out of service across restarts, cordon it: +[`POST /v1/cordon`](sandboxd-api.md#post-v1cordon) (root) refuses new claims +and destroys the warm pools as a drain does, persists as +`/cordoned` so a restarted node starts no warm VM, and still admits a +root [live move](cluster.md#moving-a-running-claim) to the node. Its live claims +keep serving. `DELETE /v1/cordon` lifts it; a cordon and a drain are independent. + ### Disabling archive failover Replicas and archive rows retire only through the registry. Before setting diff --git a/docs/index.md b/docs/index.md index 774af3e04..2a1c6861a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -27,7 +27,7 @@ vsock-only I/O (hardened default); `net=egress` attaches a bridge/CNI NIC. - [sandboxd HTTP API](sandboxd-api.md) — every endpoint: claim/release, renew, hibernate, fork, promote, checkpoints, preview, the guest-port relay, claim env, runtime volume registration, live volume changes, - runtime tenants, config reload, online pool reconfigure, drain and + runtime tenants, config reload, online pool reconfigure, drain, cordon and evacuation, live move, metrics, the usage journal and its shared table - [Replace a dataset in a running sandbox](live-volumes.md) — swapping a read-only catalog volume on a running claim with the volume API diff --git a/docs/sandboxd-api.md b/docs/sandboxd-api.md index 5e0afcece..b70b9afdd 100644 --- a/docs/sandboxd-api.md +++ b/docs/sandboxd-api.md @@ -10,7 +10,7 @@ All bodies are JSON. Three token kinds: tenant creates is stamped with its name. Operator surfaces (the per-id sandbox reads, `PUT /v1/sandboxes/{id}/instance-metadata`, `GET /v1/info`, `PUT /v1/pools`, `POST /v1/config/reload`, the [tenant API](#tenants-v1tenants), - `POST /v1/sandboxes/{id}/move`, `PUT/DELETE /v1/volumes/{name}`, `POST/DELETE /v1/drain`, + `POST /v1/sandboxes/{id}/move`, `PUT/DELETE /v1/volumes/{name}`, `POST/DELETE /v1/drain`, `POST/DELETE /v1/cordon`, `GET /metrics`, `GET /v1/usage`, `GET /v1/checkpoints/{id}/blob`) answer a tenant token `403` — authenticated but not authorized; an unknown token stays `401`. On a node with a shared @@ -224,7 +224,7 @@ Redirects (mutually exclusive with the fields above) name peers to retry at — sent on a warm miss with warm peers, when the node lacks a golden for the key but gossip names a template owner, and when a claim without `volumes`, `no_redirect` or `require_promoted` meets a 429 (the node at `max_claims` or -`max_running`, the calling tenant at its cap, the node draining, the disk under +`max_running`, the calling tenant at its cap, the node draining or cordoned, the disk under `disk_reserve`, or memory over `memory_limit`) and a peer reports warm capacity or, failing that, serves the pool: ```json @@ -259,7 +259,7 @@ exclude a writer), or a `ro` claim against a writable image left dirty by an unclean `rw` release (needs recovery — one `rw` claim must replay and cleanly release before `ro` claims resume; see [deploy](deploy.md#writable-dataset-volumes)); 429 node at `max_claims` or `max_running`, the -calling tenant at its own `max_claims`, the node draining, the disk under +calling tenant at its own `max_claims`, the node draining or cordoned, the disk under `disk_reserve`, or memory over `memory_limit`; 500 provisioning failed. ## GET /v1/volumes @@ -608,7 +608,7 @@ an unknown `on_expire`; 401 bad api token; 403 the parent's tenant was removed (sandbox token); 404 unknown id or wrong sandbox token; 409 egress-lane or volume parent (neither forks, checkpoints, or promotes; see [egress](egress.md)), a failed parent, or an archived parent (`POST /wake` or an exec wakes it first); -429 node or the parent's tenant at `max_claims`, the node at `max_running`, the node draining, the disk +429 node or the parent's tenant at `max_claims`, the node at `max_running`, the node draining or cordoned, the disk under `disk_reserve`, or memory over `memory_limit`; 503 `lease lost; retry` for a replicated parent while its node's lease is overdue, `tenant store down; retry` for a sandbox token whose tenant the node has not cached while the meta store is down. ## POST /v1/sandboxes/{id}/promote @@ -803,15 +803,18 @@ A refused reload changes nothing. ## POST /v1/drain -Auth: root only (tenant tokens get 403). Cordons the node for maintenance: claim/fork/branch answer +Auth: root only (tenant tokens get 403). Empties the node for maintenance: claim/fork/branch answer 429 with an error ending `node draining` (on a cluster a non-volume claim tries a peer redirect first, and peers stop redirecting here within a tick, because this node gossips no pools), +a live [move](#post-v1sandboxesidmove) to this node is refused, unclaimed warm VMs are destroyed, and live claims keep serving until release or TTL. Pool ownership is untouched — no pools.json write, no config change. Answers the fresh `GET /v1/info` payload; poll `claimed` to zero to know the -node is empty. Deliberately not persisted: a restarted node serves again. +node is empty. Deliberately not persisted: a restarted node serves again. To +keep a node out of service across restarts, [cordon](#post-v1cordon) it; the +[admission table](cluster.md#node-admission) compares the states. -`POST /v1/drain?evacuate=1` cordons the same way, then moves the claims to the +`POST /v1/drain?evacuate=1` drains the same way, then moves the claims to the peers with their state, so the node empties without waiting for the leases to run out (see [evacuating a node](cluster.md#evacuating-a-node)). It needs [`archive_failover`](cluster.md#archived-claims-across-node-loss). In the @@ -840,7 +843,32 @@ new one over the claims still here. Auth: root only (tenant tokens get 403). Lifts the drain, stops an evacuation, and kicks an immediate refill. Answers the fresh info payload. A claim an evacuation is archiving finishes its archive and -stays here archived; the next call wakes it. +stays here archived; the next call wakes it. A cordon stays in place. + +## POST /v1/cordon + +Auth: root only (tenant tokens get 403). Takes the node out of service until +`DELETE /v1/cordon`, across restarts: claim/fork/branch answer 429 with an +error ending `node cordoned`, whatever `max_claims` and `max_running` allow (on +a cluster a non-volume claim tries a peer redirect first, and peers stop +redirecting here within a tick, because this node gossips no pools), refill +stops and unclaimed warm VMs are destroyed. Live claims keep serving, and unlike +a drain, a root [move](#post-v1sandboxesidmove) to this node is admitted. The +[admission table](cluster.md#node-admission) lists what each state allows; a +cordon and a drain are independent, and each `DELETE` lifts only its own. + +The cordon is the file `/cordoned`; it survives a restart, a reboot +and a config reload, and a node that starts cordoned starts no warm VM. With +`audit_log` on, `audit.jsonl` records op `cordon`. Answers the fresh +`GET /v1/info` payload with `"cordoned": true`; 500 when the file cannot be +written, and the node stays as it was. + +## DELETE /v1/cordon + +Auth: root only (tenant tokens get 403). Removes `/cordoned`, lifts +the cordon and kicks an immediate refill; `audit.jsonl` records op `uncordon`. +A drain stays in place. Answers the fresh info payload; 500 when the file cannot +be removed. ## POST /v1/sandboxes/{id}/move @@ -934,7 +962,7 @@ the claim. See the [placement lifecycle](cluster.md#checkpoints-on-a-cluster). 404 for an unknown checkpoint (locally, and after redirect and heal both miss), 429 node -or calling tenant at `max_claims`, the node at `max_running`, the node draining, the disk under +or calling tenant at `max_claims`, the node at `max_running`, the node draining or cordoned, the disk under `disk_reserve`, or memory over `memory_limit`, 503 when the node's concurrent-heal cap is already full — retryable, and the response carries a `Retry-After` hint. @@ -1106,7 +1134,7 @@ verb never wakes one) or its image serves no instance metadata. ## GET /metrics Auth: root only (tenant tokens get 403). Prometheus text format, -hand-rendered: pool warm/target gauges, claimed/running/max_running/hibernated/archived/failed/draining +hand-rendered: pool warm/target gauges, claimed/running/max_running/hibernated/archived/failed/draining/cordoned gauges, `sandboxd_evacuating` (the claims an evacuation has yet to move), `sandboxd_memory_low` and `sandboxd_anon_bytes` (see [`memory_limit`](deploy.md#configuration)), a per-tenant live-claim gauge (`sandboxd_tenant_claims{tenant="…"}`, tenants holding live claims only, so the label set is bounded by the node's claims), `sandboxd_config_digest_mismatch` on a mesh, claims by tier (warm/clone/cold), @@ -1321,7 +1349,7 @@ this node owns the sandbox, 404 otherwise. With does not hold the sandbox adopts it when its row names an owner whose node lease expired, or whose owner released it in an [evacuation](#post-v1drain), and answers 200; 503 with `Retry-After` when the meta store is -down. A row whose owner's lease is live is never adopted unless the owner released it: 404. A draining or fenced +down. A row whose owner's lease is live is never adopted unless the owner released it: 404. A draining, cordoned or fenced node, or one that lifted its fence less than 8 seconds ago, does not adopt: 404. A node whose lease is overdue answers 404 for the replicated claims it holds, so the `Lookup` scatter moves on. With `client_advertise` configured, @@ -1362,8 +1390,9 @@ checkpointed to the store with the local VM dropped (see and was not restarted (omitted at zero); all are included in `claimed`. `running` counts claims that hold a VM, with a wake in flight counted, and `max_running` is the [cap](deploy.md#configuration) on it (omitted when off). -A node cordoned via [`POST /v1/drain`](#post-v1drain) additionally reports -`"draining": true`, and during an evacuation `evacuating` (claims left to +A node drained via [`POST /v1/drain`](#post-v1drain) additionally reports +`"draining": true`, a node cordoned via [`POST /v1/cordon`](#post-v1cordon) +`"cordoned": true`, and during an evacuation `evacuating` (claims left to move, omitted at zero). When refill is parked because the node cannot start another VM, `"at_capacity": true` distinguishes a full node from one still filling its warm target, and `at_capacity_reason` carries the engine's capacity diff --git a/docs/sdk-python.md b/docs/sdk-python.md index 48e51da29..038bef7a8 100644 --- a/docs/sdk-python.md +++ b/docs/sdk-python.md @@ -525,7 +525,9 @@ pty.close() ```python sb = client.new("rt:24.04", claim_ref="ns/workload") client.sandboxes() # one summary dict per live claim — never tokens -client.drain() # cordon: refuse new claims, run leases out +client.drain() # refuse new claims, run leases out +client.undrain() +client.cordon() # refuse new claims across restarts; moves in still land client.uncordon() client.attach(owner_addr, id, token) # bind a known handle, no lookup round-trip ``` @@ -533,7 +535,11 @@ client.attach(owner_addr, id, token) # bind a known handle, no lookup round-tr `sandboxes()` is scoped to the calling token, so a tenant sees only its own claims; the fields are those of [`GET /v1/sandboxes`](sandboxd-api.md#get-v1sandboxes). `drain()` leaves live claims alone — poll `info()` until `claimed` is -zero. +zero. `undrain()` lifts only a drain and `uncordon()` only a +[cordon](sandboxd-api.md#post-v1cordon); the +[admission table](cluster.md#node-admission) compares the two. Through v0.1.23 +`uncordon()` sent `DELETE /v1/drain`; a caller that still pairs it with +`drain()` leaves the node drained, and `info()` reports `draining`. ## Errors diff --git a/docs/sdk.md b/docs/sdk.md index 5e178e290..83238a0cc 100644 --- a/docs/sdk.md +++ b/docs/sdk.md @@ -700,14 +700,16 @@ ctx) tears the shell down. ## Node operations -Verbs for operating a node — `Drain`, `Uncordon`, `SetPools`, +Verbs for operating a node — `Drain`, `Undrain`, `Cordon`, `Uncordon`, `SetPools`, `SetPoolsCluster` and the tenant verbs need the root token — plus the reference the aggregated apiserver claims under: ```go sb, _ := client.New(ctx, "rt:24.04", sandbox.WithClaimRef("ns/workload")) list, _ := client.Sandboxes(ctx) // one SandboxSummary per live claim — never tokens -info, _ := client.Drain(ctx) // cordon: refuse new claims, run leases out +info, _ := client.Drain(ctx) // refuse new claims, run leases out +info, _ = client.Undrain(ctx) +info, _ = client.Cordon(ctx) // refuse new claims across restarts; moves in still land info, _ = client.Uncordon(ctx) info, _ = client.SetPools(ctx, pools) // retune warm targets without a restart res, _ := client.SetPoolsCluster(ctx, pools) // per-node results; retry the failures @@ -726,6 +728,10 @@ res, _ = client.SetTenantsCluster(ctx, specs) claims; the fields are those of [`GET /v1/sandboxes`](sandboxd-api.md#get-v1sandboxes), including each claim's `Metadata`, `CPUCount`, and `MemTotalBytes`. `Drain` leaves live claims alone — poll `Info` until `Claimed` is zero. +`Undrain` lifts only a drain and `Uncordon` only a [cordon](sandboxd-api.md#post-v1cordon); +the [admission table](cluster.md#node-admission) compares the two. Through v0.1.23 +`Uncordon` sent `DELETE /v1/drain`; a caller that still pairs it with `Drain` +leaves the node drained, and `Info` reports `Draining`. The tenant verbs follow [`/v1/tenants`](sandboxd-api.md#tenants-v1tenants): a `TenantSpec` without `Token` keeps the stored token, and the `*Cluster` forms return one `NodeResult` per node to retry (see diff --git a/sandboxd/pool/claim.go b/sandboxd/pool/claim.go index 6d53ef9d7..1f2499525 100644 --- a/sandboxd/pool/claim.go +++ b/sandboxd/pool/claim.go @@ -436,6 +436,14 @@ func (m *Manager) admitClaim(tenant string, volumes []resolvedVolume, applied [] // Callers hold m.mu. func (m *Manager) quotaErr(extra int, tenant string) error { + if m.cordoned { + return fmt.Errorf("%w: node cordoned", ErrQuota) + } + return m.capacityErr(extra, tenant) +} + +// capacityErr is quotaErr less the cordon, which admits moves in; callers hold m.mu. +func (m *Manager) capacityErr(extra int, tenant string) error { if m.draining { return fmt.Errorf("%w: node draining", ErrQuota) } diff --git a/sandboxd/pool/cordon.go b/sandboxd/pool/cordon.go new file mode 100644 index 000000000..3a70ee598 --- /dev/null +++ b/sandboxd/pool/cordon.go @@ -0,0 +1,62 @@ +package pool + +import ( + "context" + "errors" + "fmt" + "os" + "path/filepath" + + "github.com/cocoonstack/sandbox/sandboxd/utils" +) + +const cordonFile = "cordoned" + +// Cordon refuses new claims until Uncordon, across restarts, and destroys the warm VMs; live moves in stay admitted. +func (m *Manager) Cordon(ctx context.Context) error { + m.cordonMu.Lock() + defer m.cordonMu.Unlock() + if err := utils.WriteFileSync(m.cordonPath(), nil, 0o600); err != nil { + return fmt.Errorf("persist cordon: %w", err) + } + m.mu.Lock() + m.cordoned = true + trim := m.trimAllWarmLocked() + m.mu.Unlock() + m.recordAudit(ctx, "", auditFrame{Op: "cordon"}) + m.destroyAll(context.WithoutCancel(ctx), trim).Wait() + return nil +} + +// Uncordon lifts a cordon and kicks an immediate refill. +func (m *Manager) Uncordon(ctx context.Context) error { + m.cordonMu.Lock() + defer m.cordonMu.Unlock() + if err := os.Remove(m.cordonPath()); err != nil && !errors.Is(err, os.ErrNotExist) { + return fmt.Errorf("remove cordon: %w", err) + } + if err := utils.SyncDir(m.dataDir); err != nil { + return fmt.Errorf("sync cordon removal: %w", err) + } + m.mu.Lock() + m.cordoned = false + m.mu.Unlock() + m.recordAudit(ctx, "", auditFrame{Op: "uncordon"}) + m.refillOnce(context.WithoutCancel(ctx)) + return nil +} + +func (m *Manager) loadCordon() error { + _, err := os.Stat(m.cordonPath()) + switch { + case err == nil: + m.cordoned = true + case !errors.Is(err, os.ErrNotExist): + return fmt.Errorf("read cordon: %w", err) + } + return nil +} + +func (m *Manager) cordonPath() string { + return filepath.Join(m.dataDir, cordonFile) +} diff --git a/sandboxd/pool/cordon_test.go b/sandboxd/pool/cordon_test.go new file mode 100644 index 000000000..3a33bd546 --- /dev/null +++ b/sandboxd/pool/cordon_test.go @@ -0,0 +1,281 @@ +package pool + +import ( + "errors" + "os" + "path/filepath" + "strings" + "testing" + "testing/synctest" + "time" + + "github.com/cocoonstack/sandbox/sandboxd/config" +) + +func TestCordonRefusesClaimsTrimsWarmAndSurvivesARestart(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + eng := newFakeEngine() + m := newTestManager(t, eng) + seedGolden(t, m, testKey, "") + spec := config.PoolSpec{PoolKey: testKey, Warm: 2} + if err := m.SetPools(t.Context(), []config.PoolSpec{spec}); err != nil { + t.Fatalf("SetPools: %v", err) + } + waitFor(t, func() bool { + infos, _ := m.Info() + return len(infos) == 1 && infos[0].Warm == 2 + }) + + if err := m.Cordon(t.Context()); err != nil { + t.Fatalf("cordon: %v", err) + } + assertCordonRefuses(t, m) + infos, g := m.Info() + if !g.Cordoned || g.Draining || infos[0].Warm != 0 { + t.Fatalf("gauges %+v warm=%d, want cordoned, not draining, no warm", g, infos[0].Warm) + } + if removed := eng.removedNames(); len(removed) != 2 { + t.Fatalf("removed=%v, want both warm VMs destroyed", removed) + } + if counts := m.WarmCounts(); len(counts) != 0 { + t.Fatalf("warm counts %v, want none gossiped", counts) + } + + reborn := newFakeEngine() + restarted := newTestManagerAt(t, reborn, m.dataDir, spec) + restarted.refillOnce(t.Context()) + restarted.mu.Lock() + p := restarted.pools[testKey] + started := p.building || p.refilling > 0 || len(p.warm) > 0 + restarted.mu.Unlock() + if _, g = restarted.Info(); !g.Cordoned || started || len(reborn.clones) != 0 { + t.Fatalf("after restart gauges %+v started %v clones %v, want cordoned with no build or refill", g, started, reborn.clones) + } + assertCordonRefuses(t, restarted) + + if err := restarted.Uncordon(t.Context()); err != nil { + t.Fatalf("uncordon: %v", err) + } + waitFor(t, func() bool { + now, gauges := restarted.Info() + return !gauges.Cordoned && now[0].Warm == 2 + }) + if _, err := os.Stat(m.cordonPath()); !errors.Is(err, os.ErrNotExist) { + t.Errorf("cordon file after uncordon: %v, want gone", err) + } + if _, err := claimAny(t.Context(), restarted, testKey, ClaimOptions{}); err != nil { + t.Fatalf("claim after uncordon: %v", err) + } + }) +} + +func TestACloneInFlightAtCordonIsNotKept(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + eng := newFakeEngine() + eng.cloneStall = make(chan struct{}) + m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1}) + m.pools[testKey].goldenDir = "/goldens/x" + m.refillOnce(t.Context()) + synctest.Wait() + if err := m.Cordon(t.Context()); err != nil { + t.Fatalf("cordon: %v", err) + } + close(eng.cloneStall) + waitFor(t, func() bool { + infos, _ := m.Info() + return infos[0].Refilling == 0 + }) + if infos, _ := m.Info(); infos[0].Warm != 0 || len(eng.removedNames()) != 1 { + t.Fatalf("warm %d removed %v, want the late clone destroyed", infos[0].Warm, eng.removedNames()) + } + }) +} + +func TestACordonedNodeStillDropsAPoolRemovedMidClone(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + eng := newFakeEngine() + eng.cloneStall = make(chan struct{}) + m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1}) + m.pools[testKey].goldenDir = "/goldens/x" + m.refillOnce(t.Context()) + synctest.Wait() + if err := m.Cordon(t.Context()); err != nil { + t.Fatalf("cordon: %v", err) + } + if err := m.SetPools(t.Context(), nil); err != nil { + t.Fatalf("SetPools: %v", err) + } + close(eng.cloneStall) + waitFor(t, func() bool { + m.mu.Lock() + defer m.mu.Unlock() + p := m.pools[testKey] + return p == nil || p.settled() + }) + m.refillOnce(t.Context()) + m.mu.Lock() + _, kept := m.pools[testKey] + m.mu.Unlock() + if kept { + t.Fatal("the cordoned node kept a removed pool after its clone settled") + } + }) +} + +func TestCordonRefusesForksAndCheckpointClaims(t *testing.T) { + m := newTestManager(t, newFakeEngine()) + src := mustClaim(t, m, testKey) + ckpt, err := m.Checkpoint(t.Context(), src.ID, Cred{Token: src.Token}, "step-1", "") + if err != nil { + t.Fatalf("checkpoint: %v", err) + } + if err = m.Cordon(t.Context()); err != nil { + t.Fatalf("cordon: %v", err) + } + if _, err = m.Fork(t.Context(), src.ID, Cred{Token: src.Token}, 1, 0, "", ""); !isCordonErr(err) { + t.Errorf("fork: %v, want node cordoned", err) + } + if _, err = m.ClaimCheckpoint(t.Context(), ckpt.ID, ClaimOptions{}); !isCordonErr(err) { + t.Errorf("checkpoint claim: %v, want node cordoned", err) + } +} + +func TestACordonedNodeKeepsServingItsOwnClaims(t *testing.T) { + m := newTestManager(t, newFakeEngine(), archivePool(3600)) + sb := mustClaim(t, m, testKey) + gone := mustClaim(t, m, testKey) + cred := Cred{Token: sb.Token} + if err := m.Cordon(t.Context()); err != nil { + t.Fatalf("cordon: %v", err) + } + if _, err := m.Renew(t.Context(), sb.ID, cred, time.Hour, ""); err != nil { + t.Fatalf("renew: %v", err) + } + hibernateClaim(t, m, sb) + if _, _, _, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { + t.Fatalf("wake hibernated: %v", err) + } + mustArchive(t, m, sb) + if _, _, _, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { + t.Fatalf("wake archived: %v", err) + } + if err := m.Release(t.Context(), gone.ID, Cred{Token: gone.Token}); err != nil { + t.Fatalf("release: %v", err) + } + if _, g := m.Info(); !g.Cordoned || g.Claimed != 1 { + t.Errorf("gauges %+v, want cordoned with one claim", g) + } +} + +func TestACordonedNodeNeverAdoptsButKeepsItsOwnRowAcrossARestart(t *testing.T) { + c := newFailoverCluster(t) + a := c.node(t, "a", t.TempDir(), keepAnHour) + b := c.node(t, "b", t.TempDir(), keepAnHour) + lost := mustClaim(t, a, testKey) + mustArchive(t, a, lost) + own := mustClaim(t, b, testKey) + mustArchive(t, b, own) + if err := b.FlushClaims(); err != nil { + t.Fatalf("flush: %v", err) + } + if err := b.Cordon(t.Context()); err != nil { + t.Fatalf("cordon: %v", err) + } + c.expire("a") + mustNotAdopt(t, b, lost, "adoption by a cordoned node") + + restarted := c.restart(t, b) + restarted.liftFence() + if err := restarted.Reconcile(t.Context()); err != nil { + t.Fatalf("reconcile: %v", err) + } + if _, g := restarted.Info(); !g.Cordoned { + t.Fatal("the restart lifted the cordon") + } + mustNotAdopt(t, restarted, lost, "adoption by a restarted cordoned node") + mustAdopt(t, restarted, own, "its own archived claim") + if _, _, _, err := restarted.WakeAgentSocket(t.Context(), own.ID, own.Token); err != nil { + t.Fatalf("wake own archived claim: %v", err) + } + + if err := restarted.Uncordon(t.Context()); err != nil { + t.Fatalf("uncordon: %v", err) + } + mustAdopt(t, restarted, lost, "adoption after uncordon") +} + +func TestAMoveLandsOnACordonedNode(t *testing.T) { + mc := newMoveCluster(t) + sb := mustClaim(t, mc.s, testKey) + if err := mc.t.Cordon(t.Context()); err != nil { + t.Fatalf("cordon: %v", err) + } + if _, err := mc.s.Move(t.Context(), sb.ID, "b", time.Minute); err != nil { + t.Fatalf("move to a cordoned node: %v", err) + } + if _, held := mc.t.byID(sb.ID); !held { + t.Error("the cordoned node did not take the claim") + } + if _, held := mc.s.byID(sb.ID); held { + t.Error("the source kept the moved claim") + } + assertCordonRefuses(t, mc.t) +} + +func TestCordonOutlivesAReloadAndAnUndrain(t *testing.T) { + m := newTestManager(t, newFakeEngine()) + if err := m.Cordon(t.Context()); err != nil { + t.Fatalf("cordon: %v", err) + } + next := *m.cfg + if _, err := m.ReloadConfig(t.Context(), &next); err != nil { + t.Fatalf("reload: %v", err) + } + m.Drain(t.Context()) + m.Undrain(t.Context()) + if _, g := m.Info(); !g.Cordoned || g.Draining { + t.Fatalf("gauges %+v, want cordoned and not draining", g) + } + assertCordonRefuses(t, m) + + m.Drain(t.Context()) + if err := m.Uncordon(t.Context()); err != nil { + t.Fatalf("uncordon: %v", err) + } + if _, err := claimAny(t.Context(), m, testKey, ClaimOptions{}); !errors.Is(err, ErrQuota) || !strings.HasSuffix(err.Error(), "node draining") { + t.Fatalf("claim after uncordon of a drained node: %v, want node draining", err) + } +} + +func TestCordonIsAudited(t *testing.T) { + dir := t.TempDir() + cfg := &config.Config{DataDir: dir, AuditLog: true} + m, err := NewManager(t.Context(), cfg, newFakeEngine(), testSecrets(t), testBackends(t, cfg)) + if err != nil { + t.Fatalf("setup manager: %v", err) + } + if err = m.Cordon(t.Context()); err != nil { + t.Fatalf("cordon: %v", err) + } + if err = m.Uncordon(t.Context()); err != nil { + t.Fatalf("uncordon: %v", err) + } + raw, err := os.ReadFile(filepath.Join(dir, "audit.jsonl")) + if err != nil || !strings.Contains(string(raw), `"op":"cordon"`) || !strings.Contains(string(raw), `"op":"uncordon"`) { + t.Errorf("audit %s, %v; want cordon and uncordon records", raw, err) + } +} + +func assertCordonRefuses(t *testing.T, m *Manager) { + t.Helper() + if _, err := m.ClaimWarm(t.Context(), testKey, ClaimOptions{}); !isCordonErr(err) { + t.Fatalf("warm claim: %v, want node cordoned", err) + } + if _, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{}); !isCordonErr(err) { + t.Fatalf("provision claim: %v, want node cordoned", err) + } +} + +func isCordonErr(err error) bool { + return errors.Is(err, ErrQuota) && strings.HasSuffix(err.Error(), "node cordoned") +} diff --git a/sandboxd/pool/drain.go b/sandboxd/pool/drain.go index 82b9f9224..c64bbec3a 100644 --- a/sandboxd/pool/drain.go +++ b/sandboxd/pool/drain.go @@ -20,14 +20,11 @@ type evacuation struct { stop context.CancelFunc } -// Drain cordons the node: claims answer 429 and warm VMs are destroyed. +// Drain closes the node until Undrain or a restart: claims and moves in answer 429 and warm VMs are destroyed. func (m *Manager) Drain(ctx context.Context) { - var trim []string m.mu.Lock() m.draining = true - for _, p := range m.pools { - trim = append(trim, p.trimWarm(0)...) - } + trim := m.trimAllWarmLocked() m.mu.Unlock() m.destroyAll(context.WithoutCancel(ctx), trim).Wait() } @@ -45,8 +42,8 @@ func (m *Manager) Evacuate(ctx context.Context) { go m.runEvacuation(ctx, m.evac) } -// Uncordon lifts a drain, stops an evacuation, and kicks an immediate refill. -func (m *Manager) Uncordon(ctx context.Context) { +// Undrain lifts a drain, stops an evacuation, and kicks an immediate refill. +func (m *Manager) Undrain(ctx context.Context) { m.mu.Lock() m.draining = false e := m.evac @@ -187,6 +184,18 @@ func (m *Manager) takeBack(ctx context.Context, sb *types.Sandbox, released int6 return errNoAdopter } +func (m *Manager) closedLocked() bool { + return m.draining || m.cordoned +} + +func (m *Manager) trimAllWarmLocked() []string { + var trim []string + for _, p := range m.pools { + trim = append(trim, p.trimWarm(0)...) + } + return trim +} + func (m *Manager) evacuatingLocked(id string) bool { return m.evac != nil && m.evac.pending[id] != nil } diff --git a/sandboxd/pool/drain_test.go b/sandboxd/pool/drain_test.go index 1c95c61c1..c91060a28 100644 --- a/sandboxd/pool/drain_test.go +++ b/sandboxd/pool/drain_test.go @@ -14,7 +14,7 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/types" ) -func TestDrainRefusesClaimsTrimsWarmAndUncordonRefills(t *testing.T) { +func TestDrainRefusesClaimsTrimsWarmAndUndrainRefills(t *testing.T) { synctest.Test(t, func(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) @@ -47,13 +47,13 @@ func TestDrainRefusesClaimsTrimsWarmAndUncordonRefills(t *testing.T) { t.Fatalf("refill ran during drain: %+v", infos) } - m.Uncordon(t.Context()) + m.Undrain(t.Context()) waitFor(t, func() bool { now, gauges := m.Info() return !gauges.Draining && len(now) == 1 && now[0].Warm == 2 }) if _, err := claimAny(t.Context(), m, testKey, ClaimOptions{}); err != nil { - t.Fatalf("claim after uncordon: %v", err) + t.Fatalf("claim after undrain: %v", err) } }) } @@ -216,7 +216,7 @@ func TestEvacuateWaitsOutAnArchiveInFlight(t *testing.T) { }) } -func TestUncordonStopsAnEvacuationBeforeItsHandOff(t *testing.T) { +func TestUndrainStopsAnEvacuationBeforeItsHandOff(t *testing.T) { synctest.Test(t, func(t *testing.T) { c := newFailoverCluster(t) a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) @@ -227,29 +227,29 @@ func TestUncordonStopsAnEvacuationBeforeItsHandOff(t *testing.T) { synctest.Wait() assertEvacuation(t, a, 1, 0) - a.Uncordon(t.Context()) + a.Undrain(t.Context()) close(stall) synctest.Wait() assertEvacuation(t, a, 0, 0) if _, g := a.Info(); g.Draining { - t.Error("still draining after uncordon") + t.Error("still draining after undrain") } held, ok := a.byID(sb.ID) if !ok || held.ArchiveCk == "" { t.Fatalf("held %v, want the claim kept archived on a", ok) } if rec, _, _ := c.reg.Get(t.Context(), sb.ID); rec.Owner != "a" { - t.Errorf("row owner %q after uncordon, want a", rec.Owner) + t.Errorf("row owner %q after undrain, want a", rec.Owner) } assertNotAdopted(t, b, sb, "adopt a claim whose evacuation stopped") if _, _, _, err := a.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { - t.Errorf("wake after uncordon: %v", err) + t.Errorf("wake after undrain: %v", err) } }) } -func TestAnUncordonDuringAnEvacuationsDrainStopsIt(t *testing.T) { +func TestAnUndrainDuringAnEvacuationsDrainStopsIt(t *testing.T) { synctest.Test(t, func(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) @@ -270,7 +270,7 @@ func TestAnUncordonDuringAnEvacuationsDrainStopsIt(t *testing.T) { }() synctest.Wait() - a.Uncordon(t.Context()) + a.Undrain(t.Context()) release() <-done synctest.Wait() @@ -278,12 +278,12 @@ func TestAnUncordonDuringAnEvacuationsDrainStopsIt(t *testing.T) { assertEvacuation(t, a, 0, 0) held, ok := a.byID(sb.ID) if !ok || held.HibernateSnap != "" || held.ArchiveCk != "" { - t.Errorf("held %v, want the claim still running after the uncordon", ok) + t.Errorf("held %v, want the claim still running after the undrain", ok) } }) } -func TestEvacuateHoldsTheWakeOfAQueuedClaimUntilUncordon(t *testing.T) { +func TestEvacuateHoldsTheWakeOfAQueuedClaimUntilUndrain(t *testing.T) { tests := []struct { name string park func(*testing.T, *Manager, *types.Sandbox) @@ -308,7 +308,7 @@ func TestEvacuateHoldsTheWakeOfAQueuedClaimUntilUncordon(t *testing.T) { } assertEvacuation(t, a, 1, 0) - a.Uncordon(t.Context()) + a.Undrain(t.Context()) release() synctest.Wait() @@ -316,7 +316,7 @@ func TestEvacuateHoldsTheWakeOfAQueuedClaimUntilUncordon(t *testing.T) { t.Fatalf("held %v, want the claim left as it was parked", ok) } if _, _, _, err := a.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { - t.Errorf("wake after uncordon: %v", err) + t.Errorf("wake after undrain: %v", err) } }) }) @@ -339,7 +339,7 @@ func TestAQueuedFailedClaimStillRestarts(t *testing.T) { t.Errorf("wake of a queued failed claim: %v", err) } - a.Uncordon(t.Context()) + a.Undrain(t.Context()) release() synctest.Wait() }) @@ -361,7 +361,7 @@ func TestEvacuateRequeuesAClaimAWakeUnarchived(t *testing.T) { t.Errorf("hand-off of a running claim: %v, want errArchiveBusy", err) } - a.Uncordon(t.Context()) + a.Undrain(t.Context()) release() synctest.Wait() }) @@ -451,11 +451,11 @@ func TestASecondEvacuateWhileOneRunsChangesNothing(t *testing.T) { t.Error("a second evacuate replaced the run") } - a.Uncordon(t.Context()) + a.Undrain(t.Context()) release() synctest.Wait() if held, ok := a.byID(sb.ID); !ok || held.HibernateSnap != "" || held.ArchiveCk != "" { - t.Errorf("held %v, want the claim left running after uncordon", ok) + t.Errorf("held %v, want the claim left running after undrain", ok) } }) } @@ -469,9 +469,9 @@ func TestAStoppedEvacuationDoesNotPauseAClaimItHadQueued(t *testing.T) { a.Evacuate(t.Context()) synctest.Wait() - a.Uncordon(t.Context()) + a.Undrain(t.Context()) if err := a.evacuateClaim(t.Context(), sb); err != nil { - t.Errorf("evacuateClaim after uncordon: %v, want nil", err) + t.Errorf("evacuateClaim after undrain: %v, want nil", err) } if held, ok := a.byID(sb.ID); !ok || held.HibernateSnap != "" || held.ArchiveCk != "" { t.Errorf("held %v, want the claim left running", ok) diff --git a/sandboxd/pool/failover.go b/sandboxd/pool/failover.go index e1b7056d2..e4e2e0f91 100644 --- a/sandboxd/pool/failover.go +++ b/sandboxd/pool/failover.go @@ -31,7 +31,7 @@ func (m *Manager) AdoptArchived(ctx context.Context, id, token string) (bool, er func (m *Manager) adoptArchived(ctx context.Context, id, token string) (bool, error) { m.mu.Lock() - draining := m.draining + draining, cordoned := m.draining, m.cordoned _, held := m.authed(id, token) fenced := m.adoptFenced() m.mu.Unlock() @@ -40,7 +40,7 @@ func (m *Manager) adoptArchived(ctx context.Context, id, token string) (bool, er return false, nil case held: return true, nil - case fenced: + case fenced, cordoned: return false, nil } ctx = context.WithoutCancel(ctx) diff --git a/sandboxd/pool/move_in.go b/sandboxd/pool/move_in.go index 0d447118e..32098a111 100644 --- a/sandboxd/pool/move_in.go +++ b/sandboxd/pool/move_in.go @@ -320,7 +320,7 @@ func (m *Manager) admitMoveLocked(id string, a MoveAdmission) error { } else if err := m.moveBudgetErr(); err != nil { return err } - if err := m.quotaErr(extra, a.Tenant); err != nil { + if err := m.capacityErr(extra, a.Tenant); err != nil { return err } m.incoming[id] = &inMove{tenant: a.Tenant, from: a.From, touched: time.Now()} diff --git a/sandboxd/pool/pool.go b/sandboxd/pool/pool.go index fa8ac4594..332feab1a 100644 --- a/sandboxd/pool/pool.go +++ b/sandboxd/pool/pool.go @@ -229,6 +229,7 @@ type Gauges struct { Archived int Failed int Draining bool + Cordoned bool // Evacuating counts the claims an evacuation has yet to move. Evacuating int // AtCapacity marks refill parked by a refused VM or a low disk or memory reserve. @@ -308,6 +309,9 @@ type Manager struct { draining bool // guarded by m.mu; deliberately not persisted evac *evacuation + // cordoned, under m.mu, mirrors the cordon file; cordonMu orders its writers. + cordoned bool + cordonMu sync.Mutex tenants tenants.Source rootSum string @@ -416,6 +420,9 @@ func NewManager(ctx context.Context, cfg *config.Config, eng Engine, secrets *eg healSem: make(chan struct{}, maxConcurrentHeals), } m.seedVolumes(cfg.Volumes) + if err := m.loadCordon(); err != nil { + return nil, err + } if err := m.loadRuntimeVolumes(); err != nil { return nil, err } @@ -565,7 +572,7 @@ func (m *Manager) Info() ([]PoolInfo, Gauges) { }) } g := Gauges{ - Claimed: len(m.claimed), Running: m.store.runningCount(), MaxRunning: m.view.Load().maxRunning, Draining: m.draining, + Claimed: len(m.claimed), Running: m.store.runningCount(), MaxRunning: m.view.Load().maxRunning, Draining: m.draining, Cordoned: m.cordoned, MemoryLow: m.memoryLow.Load(), AnonBytes: int(m.anonBytes.Load()), } if m.evac != nil { diff --git a/sandboxd/pool/refill.go b/sandboxd/pool/refill.go index 7cd4d43a9..14d4bbd5f 100644 --- a/sandboxd/pool/refill.go +++ b/sandboxd/pool/refill.go @@ -22,12 +22,9 @@ func (m *Manager) kickRefill() { func (m *Manager) refillOnce(ctx context.Context) { m.mu.Lock() defer m.mu.Unlock() - if m.draining { - return - } now := time.Now() // Spawning VMs into a full node still takes the global rtnl lock even when doomed to fail - parked := m.capacityReasonLocked(now) != "" + parked := m.closedLocked() || m.capacityReasonLocked(now) != "" inFlight := 0 for _, p := range m.pools { inFlight += p.refilling @@ -108,7 +105,7 @@ func (m *Manager) refillOne(ctx context.Context, p *pool, golden string, gen uin m.mu.Lock() now := time.Now() p.refilling-- - if err == nil && !m.draining && p.goldenDir != "" && p.goldenGen == gen && len(p.warm) < p.effectiveTarget(now) { + if err == nil && !m.closedLocked() && p.goldenDir != "" && p.goldenGen == gen && len(p.warm) < p.effectiveTarget(now) { p.warm = append(p.warm, sb) p.noteLead(time.Since(start)) keep = true diff --git a/sandboxd/server/metrics.go b/sandboxd/server/metrics.go index 1897b0369..4b60d1433 100644 --- a/sandboxd/server/metrics.go +++ b/sandboxd/server/metrics.go @@ -22,9 +22,12 @@ func (s *Server) handleMetrics(w http.ResponseWriter, _ *http.Request) { metric(name, "gauge", help) _, _ = fmt.Fprintf(w, "sandboxd_%s %d\n", name, value) } - draining := 0 - if g.Draining { - draining = 1 + flag := func(name, help string, on bool) { + value := 0 + if on { + value = 1 + } + scalar(name, help, value) } scalar("claimed", "live claims on this node", g.Claimed) scalar("running", "claims holding a VM, including wakes in flight", g.Running) @@ -32,13 +35,10 @@ func (s *Server) handleMetrics(w http.ResponseWriter, _ *http.Request) { scalar("hibernated", "claims currently hibernated", g.Hibernated) scalar("archived", "claims archived to the checkpoint store", g.Archived) scalar("failed", "claims whose VMM is down and not restarted", g.Failed) - scalar("draining", "1 while the node is cordoned for maintenance", draining) + flag("draining", "1 while the node is draining", g.Draining) + flag("cordoned", "1 while the node is cordoned", g.Cordoned) scalar("evacuating", "claims an evacuation has yet to move to a peer", g.Evacuating) - memoryLow := 0 - if g.MemoryLow { - memoryLow = 1 - } - scalar("memory_low", "1 while memory_limit refuses claims and refills", memoryLow) + flag("memory_low", "1 while memory_limit refuses claims and refills", g.MemoryLow) scalar("anon_bytes", "node anon memory, 0 when memory_limit has no anon limit", g.AnonBytes) if tenants := s.mgr.TenantClaims(); len(tenants) > 0 { diff --git a/sandboxd/server/server.go b/sandboxd/server/server.go index fb1bff06c..ca0ae9bbb 100644 --- a/sandboxd/server/server.go +++ b/sandboxd/server/server.go @@ -143,7 +143,9 @@ type Manager interface { DeleteTenant(ctx context.Context, name string) error Drain(ctx context.Context) Evacuate(ctx context.Context) - Uncordon(ctx context.Context) + Undrain(ctx context.Context) + Cordon(ctx context.Context) error + Uncordon(ctx context.Context) error Info() ([]pool.PoolInfo, pool.Gauges) Templates() []pool.TemplateInfo Move(ctx context.Context, id, addr string, maxDowntime time.Duration) (pool.MoveResult, error) @@ -197,6 +199,7 @@ type InfoResponse struct { Archived int `json:"archived"` Failed int `json:"failed,omitzero"` Draining bool `json:"draining,omitzero"` + Cordoned bool `json:"cordoned,omitzero"` Evacuating int `json:"evacuating,omitzero"` AdvertiseAddr string `json:"advertise_addr,omitempty"` Peers []string `json:"peers,omitempty"` @@ -347,7 +350,9 @@ func (s *Server) Handler() http.Handler { mux.HandleFunc("POST /v1/moves/{id}/commit", s.requireRoot(s.handleCommitMove)) mux.HandleFunc("DELETE /v1/moves/{id}", s.requireRoot(s.handleDropMove)) mux.HandleFunc("POST /v1/drain", s.requireRoot(s.handleDrain)) - mux.HandleFunc("DELETE /v1/drain", s.requireRoot(s.handleUncordon)) + mux.HandleFunc("DELETE /v1/drain", s.requireRoot(s.handleUndrain)) + mux.HandleFunc("POST /v1/cordon", s.requireRoot(s.handleCordon)) + mux.HandleFunc("DELETE /v1/cordon", s.requireRoot(s.handleUncordon)) mux.HandleFunc("GET /v1/usage", s.requireRoot(s.handleUsage)) mux.HandleFunc("GET /v1/sandboxes/{id}/agent", s.handleAgent) mux.HandleFunc("GET /v1/sandboxes/{id}/ports/{port}", s.handlePort) @@ -840,11 +845,19 @@ func (s *Server) handleDrain(w http.ResponseWriter, r *http.Request) { s.handleInfo(w, r) } -func (s *Server) handleUncordon(w http.ResponseWriter, r *http.Request) { - s.mgr.Uncordon(r.Context()) +func (s *Server) handleUndrain(w http.ResponseWriter, r *http.Request) { + s.mgr.Undrain(r.Context()) s.handleInfo(w, r) } +func (s *Server) handleCordon(w http.ResponseWriter, r *http.Request) { + writeResult(w, r, "cordon", "", "cordon failed", s.mgr.Cordon(r.Context()), func() { s.handleInfo(w, r) }) +} + +func (s *Server) handleUncordon(w http.ResponseWriter, r *http.Request) { + writeResult(w, r, "uncordon", "", "uncordon failed", s.mgr.Uncordon(r.Context()), func() { s.handleInfo(w, r) }) +} + func (s *Server) handlePeers(w http.ResponseWriter, _ *http.Request) { var peers []string if s.placer != nil { @@ -857,7 +870,7 @@ func (s *Server) handleInfo(w http.ResponseWriter, _ *http.Request) { pools, g := s.mgr.Info() resp := InfoResponse{ Pools: pools, Templates: s.mgr.Templates(), Claimed: g.Claimed, Running: g.Running, MaxRunning: g.MaxRunning, Hibernated: g.Hibernated, Archived: g.Archived, Failed: g.Failed, - Draining: g.Draining, Evacuating: g.Evacuating, AtCapacity: g.AtCapacity, AtCapacityReason: g.AtCapacityReason, + Draining: g.Draining, Cordoned: g.Cordoned, Evacuating: g.Evacuating, AtCapacity: g.AtCapacity, AtCapacityReason: g.AtCapacityReason, AdvertiseAddr: s.advertise, } if s.placer != nil { diff --git a/sandboxd/server/server_test.go b/sandboxd/server/server_test.go index 25ad20406..fbcd18e83 100644 --- a/sandboxd/server/server_test.go +++ b/sandboxd/server/server_test.go @@ -543,6 +543,8 @@ func TestTenantAuthMatrix(t *testing.T) { {"tenant forbidden on metrics", http.MethodGet, "/metrics", "acme-tok", http.StatusForbidden, ""}, {"tenant forbidden on pools", http.MethodPut, "/v1/pools", "acme-tok", http.StatusForbidden, ""}, {"tenant forbidden on checkpoint blob", http.MethodGet, "/v1/checkpoints/ck_00000000000000aa/blob", "acme-tok", http.StatusForbidden, ""}, + {"tenant forbidden on cordon", http.MethodPost, "/v1/cordon", "acme-tok", http.StatusForbidden, ""}, + {"tenant forbidden on uncordon", http.MethodDelete, "/v1/cordon", "acme-tok", http.StatusForbidden, ""}, {"wrong token on info stays 401", http.MethodGet, "/v1/info", "nope", http.StatusUnauthorized, ""}, {"root reads info", http.MethodGet, "/v1/info", "sekret", http.StatusOK, ""}, } { @@ -717,6 +719,45 @@ func TestDrainEndpoints(t *testing.T) { } } +func TestCordonEndpoints(t *testing.T) { + mgr := &fakeManager{} + ts := newTestServer(t, "sekret", mgr, nil) + for _, tt := range []struct { + method string + cordoned bool + metric string + }{ + {http.MethodPost, true, "sandboxd_cordoned 1\n"}, + {http.MethodDelete, false, "sandboxd_cordoned 0\n"}, + } { + resp := doReq(t, tt.method, ts.URL+"/v1/cordon", "sekret", "") + info := decodeJSON[InfoResponse](t, resp.Body) + resp.Body.Close() + if resp.StatusCode != http.StatusOK || info.Cordoned != tt.cordoned || mgr.cordoned != tt.cordoned || info.Draining { + t.Fatalf("%s: status=%d info=%+v mgr=%v, want 200/%v and no drain", tt.method, resp.StatusCode, info, mgr.cordoned, tt.cordoned) + } + if _, body := bodyOf(t, http.MethodGet, ts.URL+"/metrics", "sekret", ""); !strings.Contains(string(body), tt.metric) { + t.Errorf("%s: metrics miss %q", tt.method, tt.metric) + } + } +} + +func TestCordonPersistFailureAnswers500AndKeepsState(t *testing.T) { + for _, tt := range []struct { + method string + cordoned bool + }{ + {http.MethodPost, false}, + {http.MethodDelete, true}, + } { + mgr := &fakeManager{cordoned: tt.cordoned, cordonErr: errors.New("disk gone")} + ts := newTestServer(t, "sekret", mgr, nil) + if code := statusOf(t, tt.method, ts.URL+"/v1/cordon", "sekret", ""); code != http.StatusInternalServerError || mgr.cordoned != tt.cordoned { + t.Errorf("%s: status %d cordoned=%v, want 500 and cordoned=%v", tt.method, code, mgr.cordoned, tt.cordoned) + } + } +} + func TestDrainEvacuateStartsAnEvacuationAndReportsIt(t *testing.T) { mgr := &fakeManager{evacuateLeft: 2} ts := newTestServer(t, "", mgr, nil) @@ -2260,6 +2301,8 @@ type fakeManager struct { sandboxIndex func(tenant, claimRef string) []pool.SandboxSummary sandboxByID func(id string) (pool.SandboxSummary, bool) draining bool + cordoned bool + cordonErr error evacuated bool evacuateLeft int memoryLow bool @@ -2587,7 +2630,7 @@ func (f *fakeManager) SetPools(_ context.Context, pools []config.PoolSpec) error } func (f *fakeManager) Info() ([]pool.PoolInfo, pool.Gauges) { - return f.infoPools, pool.Gauges{Draining: f.draining, Evacuating: f.evacuateLeft, MemoryLow: f.memoryLow, AnonBytes: f.anonBytes} + return f.infoPools, pool.Gauges{Draining: f.draining, Cordoned: f.cordoned, Evacuating: f.evacuateLeft, MemoryLow: f.memoryLow, AnonBytes: f.anonBytes} } func (f *fakeManager) Templates() []pool.TemplateInfo { return f.infoTemplates } @@ -2633,7 +2676,21 @@ func (f *fakeManager) Drain(context.Context) { f.draining = true } func (f *fakeManager) Evacuate(context.Context) { f.draining, f.evacuated = true, true } -func (f *fakeManager) Uncordon(context.Context) { f.draining = false } +func (f *fakeManager) Undrain(context.Context) { f.draining = false } + +func (f *fakeManager) Cordon(context.Context) error { + if f.cordonErr == nil { + f.cordoned = true + } + return f.cordonErr +} + +func (f *fakeManager) Uncordon(context.Context) error { + if f.cordonErr == nil { + f.cordoned = false + } + return f.cordonErr +} func (f *fakeManager) FetchCheckpoint(_ context.Context, _ string) (string, []byte, func(), error) { if f.ckptDir == "" { diff --git a/sdk/go/drain.go b/sdk/go/drain.go index fb7fb5fd2..20d382696 100644 --- a/sdk/go/drain.go +++ b/sdk/go/drain.go @@ -5,13 +5,23 @@ import ( "net/http" ) -// Drain cordons the entry node (root token): new claims/forks/branches are +// Drain closes the entry node (root token): new claims/forks/branches are // refused, live claims run to their leases; poll Info until Claimed is zero. func (c *Client) Drain(ctx context.Context) (*NodeInfo, error) { return c.doJSONPtr[NodeInfo](ctx, http.MethodPost, c.addr, "/v1/drain", nil, c.apiToken, "drain") } -// Uncordon lifts a drain on the entry node (root token). +// Undrain lifts a drain on the entry node (root token). +func (c *Client) Undrain(ctx context.Context) (*NodeInfo, error) { + return c.doJSONPtr[NodeInfo](ctx, http.MethodDelete, c.addr, "/v1/drain", nil, c.apiToken, "undrain") +} + +// Cordon refuses new claims on the entry node across restarts (root token); live claims and moves in stay admitted. +func (c *Client) Cordon(ctx context.Context) (*NodeInfo, error) { + return c.doJSONPtr[NodeInfo](ctx, http.MethodPost, c.addr, "/v1/cordon", nil, c.apiToken, "cordon") +} + +// Uncordon lifts a cordon on the entry node (root token); a drain stays. func (c *Client) Uncordon(ctx context.Context) (*NodeInfo, error) { - return c.doJSONPtr[NodeInfo](ctx, http.MethodDelete, c.addr, "/v1/drain", nil, c.apiToken, "uncordon") + return c.doJSONPtr[NodeInfo](ctx, http.MethodDelete, c.addr, "/v1/cordon", nil, c.apiToken, "uncordon") } diff --git a/sdk/go/drain_test.go b/sdk/go/drain_test.go index c3bc1f3c9..1b0959c3f 100644 --- a/sdk/go/drain_test.go +++ b/sdk/go/drain_test.go @@ -7,36 +7,44 @@ import ( "testing" ) -func TestDrainAndUncordon(t *testing.T) { - var methods []string +func TestDrainCordonVerbs(t *testing.T) { + var calls []string ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/v1/drain" { - t.Errorf("path = %s, want /v1/drain", r.URL.Path) - } if auth := r.Header.Get("Authorization"); auth != "Bearer root" { t.Errorf("auth = %q, want Bearer root", auth) } - methods = append(methods, r.Method) - _ = json.MarshalWrite(w, NodeInfo{Claimed: 3, Draining: r.Method == http.MethodPost}) + calls = append(calls, r.Method+" "+r.URL.Path) + post := r.Method == http.MethodPost + _ = json.MarshalWrite(w, NodeInfo{Claimed: 3, Draining: post && r.URL.Path == "/v1/drain", Cordoned: post && r.URL.Path == "/v1/cordon"}) })) defer ts.Close() c := testClient(t, ts, WithAPIToken("root")) - info, err := c.Drain(t.Context()) - if err != nil { - t.Fatalf("Drain: %v", err) - } - if !info.Draining || info.Claimed != 3 { - t.Errorf("info = %+v, want Draining true, Claimed 3", info) - } - info, err = c.Uncordon(t.Context()) - if err != nil { - t.Fatalf("Uncordon: %v", err) + for _, tt := range []struct { + name string + verb func() (*NodeInfo, error) + draining, cordoned bool + }{ + {"Drain", func() (*NodeInfo, error) { return c.Drain(t.Context()) }, true, false}, + {"Undrain", func() (*NodeInfo, error) { return c.Undrain(t.Context()) }, false, false}, + {"Cordon", func() (*NodeInfo, error) { return c.Cordon(t.Context()) }, false, true}, + {"Uncordon", func() (*NodeInfo, error) { return c.Uncordon(t.Context()) }, false, false}, + } { + info, err := tt.verb() + if err != nil { + t.Fatalf("%s: %v", tt.name, err) + } + if info.Draining != tt.draining || info.Cordoned != tt.cordoned || info.Claimed != 3 { + t.Errorf("%s: info = %+v, want draining %v cordoned %v", tt.name, info, tt.draining, tt.cordoned) + } } - if info.Draining { - t.Errorf("info = %+v, want Draining false", info) + want := []string{"POST /v1/drain", "DELETE /v1/drain", "POST /v1/cordon", "DELETE /v1/cordon"} + if len(calls) != len(want) { + t.Fatalf("calls = %v, want %v", calls, want) } - if len(methods) != 2 || methods[0] != http.MethodPost || methods[1] != http.MethodDelete { - t.Errorf("methods = %v, want [POST DELETE]", methods) + for i := range want { + if calls[i] != want[i] { + t.Errorf("call %d = %s, want %s", i, calls[i], want[i]) + } } } diff --git a/sdk/go/info.go b/sdk/go/info.go index d0f17ebfc..f0553466b 100644 --- a/sdk/go/info.go +++ b/sdk/go/info.go @@ -20,6 +20,7 @@ type NodeInfo struct { Archived int `json:"archived"` Failed int `json:"failed,omitzero"` Draining bool `json:"draining,omitzero"` + Cordoned bool `json:"cordoned,omitzero"` Evacuating int `json:"evacuating,omitzero"` AdvertiseAddr string `json:"advertise_addr,omitempty"` Peers []string `json:"peers,omitempty"` diff --git a/sdk/python/cocoonsandbox/client.py b/sdk/python/cocoonsandbox/client.py index e65fa0373..f3f97ba05 100644 --- a/sdk/python/cocoonsandbox/client.py +++ b/sdk/python/cocoonsandbox/client.py @@ -112,12 +112,20 @@ def sandboxes(self) -> list[dict[str, Any]]: return [dict(sb) for sb in reply.get("sandboxes") or []] def drain(self) -> dict[str, Any]: - """Cordons the node (root token): new claims are refused, live ones run to their leases.""" + """Drains the node (root token): new claims are refused, live ones run to their leases.""" return self._request(self.addr, "POST", "/v1/drain", None, "drain") - def uncordon(self) -> dict[str, Any]: + def undrain(self) -> dict[str, Any]: """Lifts a drain on the node (root token).""" - return self._request(self.addr, "DELETE", "/v1/drain", None, "uncordon") + return self._request(self.addr, "DELETE", "/v1/drain", None, "undrain") + + def cordon(self) -> dict[str, Any]: + """Refuses new claims across restarts (root token); live claims and moves in stay admitted.""" + return self._request(self.addr, "POST", "/v1/cordon", None, "cordon") + + def uncordon(self) -> dict[str, Any]: + """Lifts a cordon on the node (root token); a drain stays.""" + return self._request(self.addr, "DELETE", "/v1/cordon", None, "uncordon") def checkpoint(self, id: str) -> Checkpoint: """Binds a checkpoint handle to the entry node without a lookup.""" diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index 3d55fa485..1572ace17 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -262,6 +262,20 @@ def claim(body, path): ) +@pytest.mark.parametrize( + ("verb", "method", "path"), + [ + ("drain", "POST", "/v1/drain"), + ("undrain", "DELETE", "/v1/drain"), + ("cordon", "POST", "/v1/cordon"), + ("uncordon", "DELETE", "/v1/cordon"), + ], +) +def test_node_hold_verbs_hit_their_routes(node, verb, method, path): + FakeNode.routes[(method, path)] = lambda body, path: (200, {"claimed": 3}) + assert getattr(Client(node), verb)() == {"claimed": 3} + + def test_api_error_carries_server_message(node): FakeNode.routes[("POST", "/v1/claim")] = lambda body, path: (409, {"error": "no egress"}) with pytest.raises(APIError) as exc: