diff --git a/docs/cluster.md b/docs/cluster.md index f40a47523..a8e6248fd 100644 --- a/docs/cluster.md +++ b/docs/cluster.md @@ -299,8 +299,8 @@ checkpoint store and keeps each claim's registry row on the latest snapshot: starts none, and a round whose registry is down fails before it captures. - **Restore.** A lost node's replica is adopted like an archive: the first `Lookup` that reaches another node after the lease expires takes the row, - records the claim as archived on the replica with the adopter pool's - retention window, and the next call wakes it + records the claim as archived on the replica with the row's retention deadline, + or the adopter pool's window when it has none, and the next call wakes it through the archive wake, which restores the guest with its memory and processes. The claim resumes from its last committed replica: everything after it, and the connections open at the time, are lost: up to the longer of @@ -330,15 +330,18 @@ checkpoint store and keeps each claim's registry row on the latest snapshot: checkpoint behind, the same residue as a node that never returns. A restart keeps the node's own live row for a running claim and takes the checkpoint a crashed round left on it; a claim whose row another node holds, or whose row a finished adopter - consumed, is dropped with its VM. A replica row has no retention deadline: the - hourly sweep never purges it, because no round refreshes it while the claim is - hibernated, and retention starts when the claim is archived or adopted. The + consumed, is dropped with its VM. A live claim's replica row has no retention + deadline: the hourly sweep never purges it, because no round refreshes it while + the claim is hibernated, and retention starts when the claim is archived or + adopted. A row the node holds with no claim, as after reconcile dropped it + because the VM record was gone, gains the retention deadline on that sweep: the claim stays adoptable + until it passes, and `archive_delete_after_seconds` 0 keeps the row. The rows of a node that never returns therefore stay until a client adopts them, a bounded residue of one row and up to two checkpoints per claim: the row's, and a committed pending replica or a previous replica the node died before deleting. Their metas stay live, so offline reclamation keeps them. They are archive checkpoints (`"archive": true` in their meta) that no - registry row names, which an operator can delete. Apart from that and the + registry row names, which an operator can delete. Apart from these and the lease rule the sweep is unchanged. - **Self-fencing.** A node whose renewal is overdue by 12 seconds (the lease length minus a 3 second margin, counted from the start of the last renewal @@ -373,7 +376,9 @@ checkpoint store and keeps each claim's registry row on the latest snapshot: claim's first replica included: the agent relay (including parked keep-alive relays, whose later RPCs never pass the resolution gate), the exec call, port relays and preview requests. A client - retries and meets the 503 until the lease is back or the claim moved. + retries and meets the 503 until the lease is back or the claim moved. Neither + an API call nor the idle sweep starts a hibernate or a wake of such a claim + while the fence is closed, so an idle claim stays billed until the fence lifts. Streams of claims that are not replicated stay open. - **Availability.** The design chooses consistency over availability for replicated claims. A registry outage longer than the fence threshold (12 diff --git a/docs/deploy.md b/docs/deploy.md index c6c44af5a..ae585a846 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -841,8 +841,10 @@ only; computing durations or prices stays with the consumer. vm text NOT NULL DEFAULT '', key text NOT NULL DEFAULT '', template text NOT NULL DEFAULT '', + net text NOT NULL DEFAULT '', size text NOT NULL DEFAULT '', tenant text NOT NULL DEFAULT '', + deadline timestamptz, volumes text[], volumes_rw text[], children text[], diff --git a/docs/sandboxd-api.md b/docs/sandboxd-api.md index 63f1795d3..1962935a8 100644 --- a/docs/sandboxd-api.md +++ b/docs/sandboxd-api.md @@ -491,7 +491,8 @@ archives. When to hibernate is the caller's policy — the node only provides the transition. 204 on success, 404 unknown id or wrong token, 409 on the egress lane or when volumes are attached (neither kind of sandbox hibernates; see [egress](egress.md)), 429 while the disk is under -[`disk_reserve`](deploy.md). +[`disk_reserve`](deploy.md), 503 `lease lost; retry` for a replicated claim that +is not archived while its node's lease is overdue, even one already hibernated. ## POST /v1/sandboxes/{id}/wake @@ -505,7 +506,8 @@ while stored, and waking it starts a fresh lease of the length the claim was gra [failed](deploy.md#dead-vmms-and-host-restarts) sandbox is cold-booted from its own disk (memory is lost) and its restart budget resets. 204 on success, 404 unknown id or wrong token, 409 a failed egress-lane or volume -sandbox, or one whose VM record is gone, 500 when the cold boot itself fails. +sandbox, or one whose VM record is gone, 500 when the cold boot itself fails, +503 `lease lost; retry` for a replicated claim while its node's lease is overdue. ## POST /v1/sandboxes/{id}/renew @@ -1013,13 +1015,18 @@ truth is the usage journal below. Always on: every lifecycle transition appends one JSONL event to `/usage.jsonl` — `{"t": , "ev": -"claim|hibernate|wake|fork|checkpoint|promote|release|reap|archive|unarchive|archive_delete|egress|egress_bytes|vmm_exit|vmm_restart|vmm_failed", -"id": "sb_…", "vm": "sbx-…"}` plus `key` (the pool key's stable hash, claim -events), `template` and `size` (the claim's template and size tier, claim -events, so a consumer can price a sandbox by its shape without the claim -record, which is gone after release), `tenant` (the owning tenant, on every -event of a tenant's claim), `children` (fork) and `ref` (the promoted -template / checkpoint id, the egress host, or a `vmm_failed` reason). An egress event that injected credentials names +"claim|renew|hibernate|wake|fork|checkpoint|adopt|promote|release|reap|archive|unarchive|archive_delete|egress|egress_bytes|vmm_exit|vmm_restart|vmm_failed", +"id": "sb_…"}` plus `vm` (the VM's name; absent on `renew`, `adopt`, `archive`, +`unarchive`, `archive_delete`, `egress`, `egress_bytes`, and the `reap` or +`release` of an archived claim), `key` (the pool key's stable hash, claim +events), `template`, `net` and `size` (the claim's template, lane and size +tier, claim events, so a consumer can price a sandbox by its shape without the +claim record, which is gone after release; `size` sets the CPU and memory, +and `template` is a label, since promote lets the caller name it), `deadline` (the deadline in force after this event: `claim`, +`renew`, `wake` and `unarchive` carry the lease, `archive` and `adopt` the retention +end, or nothing for an archive kept forever, and `reap` the deadline it ended under), `tenant` (the owning tenant, on every event of a +tenant's claim), `children` (fork) and `ref` (the promoted template / checkpoint +id, the egress host, a `vmm_failed` reason, or why reconcile ended a claim). An egress event that injected credentials names them in `secret` (`gh,claim:`, never a value). An egress event routed through an [upstream proxy](egress.md#upstream-proxies) carries `upstream` (its `host:port`), and with `egress_usage_bytes` on, `egress_bytes` events add the connection's payload `tx`/`rx` bytes. A volume claim also carries @@ -1028,16 +1035,30 @@ the subset of those names claimed `rw`, so billing can discriminate write access (mounts and host paths are not billing dimensions). The file rotates at 64 MiB keeping one `.1` backup, so a tailing collector never loses a window silently. A last line a crash cut short is ended with a newline when the node -reopens the journal, so it never swallows the next event. Folding rules: billable compute seconds per sandbox = -Σ(claim→release/reap) − Σ(hibernate→wake) − Σ(vmm_exit→vmm_restart/release/reap); -`vmm_exit` is stamped when the node first finds the VMM gone, so an outage -before sandboxd runs again (a host that was down) stays billed up to that -point, and `vmm_failed` is informational; hibernated storage seconds = -Σ(hibernate→wake) minus the archived span; archived storage seconds = -Σ(archive→unarchive/archive_delete), a cheaper store tier than a hibernated -VM's RAM. An interval left open by a crash clamps to the claim's -deadline; reconcile records nothing for the claims it drops (those whose VM -record is gone), so a restart leaves those intervals without a terminal event. The `vm` name joins +reopens the journal, so it never swallows the next event. Folding rules: fold a sandbox's events of all nodes in `t` order. +An event ends the current span at its `t` and enters a state: `claim`, `wake`, +`unarchive` and `vmm_restart` enter `running`, `hibernate` enters `hibernated`, +`archive` enters `archived`, and `vmm_exit`, `archive_delete`, `release` and +`reap` enter `ended`. Other events, `vmm_failed` included, change no state, and +`reap`, `adopt` and `archive_delete` have the exceptions below. +Billable compute, hibernated storage and archived storage seconds are the time +in `running`, `hibernated` and `archived`, a cheaper store tier than a hibernated +VM's RAM. `vmm_exit` is stamped when the node first finds the VMM gone, so an +outage before sandboxd runs again (a host that was down) stays billed up to that +point. A `reap` instead ends the current span at min(`t`, its `deadline`), or `t` +with none, because `t` can follow the true end by the reaper's tick, the VM's +teardown or the node's downtime. Outside `archived`, an `archive_delete` ends the +current span at min(`t`, the last `deadline`), because a purge can follow a claim +that has no end event by a whole retention window. An `adopt` names a claim +a node took over, from a lost node or from its own lost record, with `ref` the +checkpoint it holds. In `archived` it changes nothing; otherwise it ends the +current span at min(`t`, the last `deadline` before it) and enters `archived` at +its `t`. A span is never shorter than zero. A span still open ends at min(now, +the last `deadline`), and one entered by an `archive` or `adopt` without a +`deadline` stays open. A claim reconcile drops because its VM record is gone +ends with a `reap` whose `ref` is `vm_gone`, and one that cannot re-lock its +egress NIC with `quarantined`; a claim another node holds is dropped with no +event, and neither has a replicated claim dropped while the registry is down. The `vm` name joins cocoon's machine-level metering ledger for audit cross-checks. With `meta_store.usage`, the node also lands every line in the cell's [shared usage table](deploy.md#shared-usage-table), read through @@ -1049,12 +1070,14 @@ Auth: root only (tenant tokens get 403). Reads the raw usage events of every node of the cell from the [shared usage table](deploy.md#shared-usage-table). - `GET /v1/usage?tenant=&since=