diff --git a/.github/workflows/sandboxd.yml b/.github/workflows/sandboxd.yml index 8a1c4ddc0..ab627b58b 100644 --- a/.github/workflows/sandboxd.yml +++ b/.github/workflows/sandboxd.yml @@ -73,7 +73,7 @@ jobs: env: SANDBOXD_TEST_PG_URL: postgres://postgres:sbxci@localhost:5432/postgres?sslmode=disable run: | - GOWORK=off go test -race -count=1 -v ./tenants/... ./poolset/... | tee "$RUNNER_TEMP/pg-test.log" + GOWORK=off go test -race -count=1 -v ./tenants/... ./poolset/... ./archives/... ./usage/... | tee "$RUNNER_TEMP/pg-test.log" GOWORK=off go test -race -count=1 -v -run 'TestTwoNodesOnOneMetaStore|TestNodesOfACell|TestAMetaStoreNodeWithoutACell' ./pool/ | tee -a "$RUNNER_TEMP/pg-test.log" ! grep -- '--- SKIP' "$RUNNER_TEMP/pg-test.log" diff --git a/README.md b/README.md index 10eccfd47..e260e7e28 100644 --- a/README.md +++ b/README.md @@ -45,8 +45,8 @@ performance) — source in restart reconcile with VMM cold restart, memberlist mesh with redirect placement, a checkpoint store on a shared directory or S3 (sparse, content-addressed chunks), and an optional shared PostgreSQL meta store - for tenants, pools, node leases and the cross-node adoption of archived - claims and of running claims a pool replicates (`replicate_seconds`) + for tenants, pools, node leases, usage events and the cross-node adoption of + archived claims and of running claims a pool replicates (`replicate_seconds`) - `sdk/go/` — Go SDK: `Connect/New/Lookup`, `Exec/Run`, files, `Push/Pull`, sessions, `Find/Replace`, `Watch`, git verbs, `OpenPty`, `Renew/Fork/Hibernate/Promote/Checkpoint`, claim env, `DialPort/ProxyPort/PreviewURL`, diff --git a/docs/cluster.md b/docs/cluster.md index a94f4ded7..f40a47523 100644 --- a/docs/cluster.md +++ b/docs/cluster.md @@ -238,8 +238,11 @@ replication rounds write it, so the claim path is unchanged. replica retirement keeps its tombstone until the hourly sweep after that. - **Usage journal.** The journal stays per node: an adopted claim's `archive` is in the old owner's `usage.jsonl`, and its `unarchive`, `release` or - `archive_delete` is in the adopter's or the purging node's. Fold the journals - of all nodes by sandbox id. + `archive_delete` is in the adopter's or the purging node's. Fold the events + of all nodes by sandbox id. With `meta_store.usage` every node lands its + journal in the cell's [shared usage table](deploy.md#shared-usage-table), and + [`GET /v1/usage`](sandboxd-api.md#get-v1usage) on any node returns the + events of all of them. The node lease decides adoption, so a partitioned owner can lose an archived claim it still lists once its lease runs out; the epoch fences it out. A claim diff --git a/docs/deploy.md b/docs/deploy.md index aba0df01e..95dd46a8a 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -129,7 +129,7 @@ gets the intersection of the two. | `egress_usage_bytes` | `false` | append an `egress_bytes` usage event with the payload bytes of every allowed egress tunnel or request when it ends; one extra usage-journal write per connection | | `egress_ca` | unset | [HTTPS-interception](egress.md#https-interception) PKI: `root_cert` (the cluster root baked into intercepted guests; may bundle old+new roots during rotation) plus this node's `intermediate_cert`/`intermediate_key` from `sandboxd ca issue-intermediate`. Required when any pool rule sets `intercept` | | `api_token` | unset | the operator (root) credential: when set, guards the node-level endpoints (Bearer) with full access, including release-by-id cleanup. Per-sandbox tokens guard ordinary sandbox-scoped calls | -| `meta_store` | unset | `{"kind": "pg", "dsn_env": "SANDBOXD_PG_DSN"}`: keep the tenant set in one shared PostgreSQL instead of each node's `tenants.json` (see [shared tenant database](#shared-tenant-database)). `dsn_env` names the node env holding the connection string. An optional `cell` also keeps the API-applied pool set there, one per cell, instead of in each node's `pools.json` (see [shared pool set](#shared-pool-set)); without it pool targets stay per node. Needs `api_token`. Restart only | +| `meta_store` | unset | `{"kind": "pg", "dsn_env": "SANDBOXD_PG_DSN"}`: keep the tenant set in one shared PostgreSQL instead of each node's `tenants.json` (see [shared tenant database](#shared-tenant-database)). `dsn_env` names the node env holding the connection string. An optional `cell` also keeps the API-applied pool set there, one per cell, instead of in each node's `pools.json` (see [shared pool set](#shared-pool-set)); without it pool targets stay per node. An optional `usage: true` also lands this node's usage journal in one shared table and serves [`GET /v1/usage`](sandboxd-api.md#get-v1usage) from it (see [shared usage table](#shared-usage-table)). Needs `api_token`. Restart only | | `max_fork_count` | 16 | children a single `fork` may create; each is a full-RAM VM, so this bounds one request's memory blast radius to the node's capacity | | `refill_concurrency` | 0 (auto) | concurrent VM provisioning budget, shared by warm-pool refills, fork clones, and the reap/hibernate/reconcile engine batches; the batches never take the last quarter of it (at least one slot), so a mass expiry or idle sweep cannot stop refills. 0 sizes it from the node: `NumCPU*2/3` clamped to [4, 256] — a 384-core node gets 256; small nodes keep a floor of 4 | | `release_delay_seconds` | 0 | seconds a released VM waits in the removal queue before `cocoon vm rm`. The claim is dropped and journaled and its egress listener closed at release; the VM removal, the volume hold release and the egress tap unlock run on the first reap tick (5 s) after the delay, on the `refill_concurrency` budget like every other batch teardown, so a burst of releases does not compete with the claims still running. Until then the VM holds its memory and its volume reservations without counting as a claim. 0 removes inline | @@ -600,14 +600,14 @@ sandbox and checkpoint listings filter to the caller's tenant, and a tenant can delete only its own checkpoints and templates (root sees everything). Operator surfaces stay root-only — per-id sandbox reads and stats, `GET /v1/checkpoints/{id}/blob`, `GET /v1/info`, `PUT /v1/pools`, -`/v1/tenants`, `/v1/config/reload`, `POST/DELETE /v1/drain`, `/metrics`; a +`/v1/tenants`, `/v1/config/reload`, `POST/DELETE /v1/drain`, `/metrics`, `GET /v1/usage`; a tenant token there is authenticated but not authorized, so it answers 403 (a wrong token stays 401). Per-sandbox tokens are unchanged: whoever holds a sandbox's token drives that sandbox. Fork children inherit the parent's tenant and count against its `max_claims` (0 = unlimited, a per-node cap, so a tenant's cluster limit is `max_claims` × nodes); a tenant at its cap gets 429 exactly like a node at -`max_claims`, and the usage journal's claim events carry the tenant for +`max_claims`, and every usage event of a tenant's claim carries the tenant for per-tenant billing. `config.json` has no `tenants` field: a node starts with no tenants until the API adds them. @@ -748,6 +748,9 @@ through `/v1/tenants` on any node. - The pool defaults to `max(4, CPUs)` connections; set `pool_max_conns` in the DSN (4 suffices) so a fleet stays within the database's `max_connections`. + - Every other shared table a node opens adds a query pool of the same size: + the pool set with a `cell`, the archive registry with `archive_failover`, + and the usage table with `usage`. Count each in `max_connections`. - LISTEN needs a session-mode connection, so a connection pooler in front of the database must not run in transaction mode for that DSN. - A write runs in one transaction with a `pg_notify`. @@ -809,6 +812,98 @@ the tenant set. - **Reverting** to config-owned pools is a `PUT` of the config pools; changing a node's `cell` takes a restart. +### Shared usage table + +With `meta_store.usage: true`, each node also lands its usage journal in one +table, and root reads the events of every node of the cell through +[`GET /v1/usage`](sandboxd-api.md#get-v1usage) on any of them. It is off by +default, so an upgrade adds no table and no writes. The table holds raw events +only; computing durations or prices stays with the consumer. + +- **Landing:** a background loop reads `usage.jsonl` and its `.1` backup every + 5 s and lands the lines it has not landed yet, up to 1000 per transaction, + until it catches up. The journal append is unchanged, so a claim, release, + hibernate or wake never waits on the database. +- **Node:** rows carry the node's mesh `node_id` (its `bind` when unset), or + its host name without a mesh. The name must be unique within the cell. +- **Schema:** sandboxd creates the table at boot when it is missing; create it + ahead with the same DDL when the node's role has no `CREATE` right: + + ```sql + CREATE TABLE IF NOT EXISTS sandboxd_usage ( + cell text NOT NULL, + cursor bigint GENERATED ALWAYS AS IDENTITY, + node text NOT NULL, + line_sha256 bytea NOT NULL, + t timestamptz NOT NULL, + ev text NOT NULL, + id text NOT NULL, + vm text NOT NULL DEFAULT '', + key text NOT NULL DEFAULT '', + tenant text NOT NULL DEFAULT '', + volumes text[], + volumes_rw text[], + children text[], + ref text NOT NULL DEFAULT '', + upstream text NOT NULL DEFAULT '', + secret text NOT NULL DEFAULT '', + tx bigint NOT NULL DEFAULT 0, + rx bigint NOT NULL DEFAULT 0, + PRIMARY KEY (cell, cursor), + UNIQUE (cell, node, line_sha256) + ); + CREATE INDEX IF NOT EXISTS sandboxd_usage_node ON sandboxd_usage (cell, node, cursor); + CREATE INDEX IF NOT EXISTS sandboxd_usage_tenant ON sandboxd_usage (cell, tenant, cursor); + CREATE INDEX IF NOT EXISTS sandboxd_usage_t ON sandboxd_usage (cell, t); + ``` + +- **Columns:** the [journal line](sandboxd-api.md#usage-journal-usagejsonl)'s + keys, with an absent key as `''`, `0` or `NULL`, plus `cell` (`''` without + one), `node` and `line_sha256`, the SHA-256 of the line. `t` keeps + microseconds, and a NUL in `ref` lands as U+FFFD. +- **At most once per line:** `(cell, node, line_sha256)` is unique, and a + land skips a line whose digest already landed for its node, so no journal + line lands twice. The node's newest row (by `cursor`) is its resume point: + at boot and after a failed land, the node reads that row's `line_sha256` + and resumes after that line in + `usage.jsonl.1` or `usage.jsonl`. When neither file holds the line any + more (a crash lost a journal tail the node had already landed, or the + journal was reset), the node reads both files from the start and lands only + the lines not landed yet. A restart, a retry or a commit whose reply was + lost therefore lands no line twice. Two byte-identical lines of one node + (same nanosecond `t` and fields) land once. A line that does not decode is skipped with + a warning, and a last line without its newline waits for it. A line a + crash cut short stays a line of its own: the node ends it with a newline + when it opens the journal, so the first event after the restart is whole. +- **Order:** a cell's lands commit one at a time under a transaction-scoped + advisory lock, so `cursor` grows in commit order and a reader that pages by + cursor never skips a row that commits later. A node lands its lines in + journal order, so its rows follow that order by `cursor`. A backfilled row + gets a newer cursor than rows with a later `t`. +- **Connection:** the table has its own query pool, sized like the others (see + [shared tenant database](#shared-tenant-database)), and no LISTEN session. + The node lands on one connection at a time; `GET /v1/usage` reads take + more. +- **Outage:** claims and the journal carry on; the node logs one warning, + retries every 5 s, and backfills when the database returns. Nothing is lost + while the unlanded lines are still in `usage.jsonl` or `usage.jsonl.1`, so + an outage loses lines only after the journal rotates twice past the last + landed one (64 MiB each); the node no longer holds those either. A node + needs the database at boot, like the other shared tables. +- **Retention:** sandboxd never deletes a row. Prune only rows every consumer + has paged past, and keep each node's newest row: it is the node's resume + point, and a node that finds none lands its whole journal again. A crash + that loses that line from the journal also lands again the pruned lines the + journal still holds. + + ```sql + DELETE FROM sandboxd_usage u + WHERE cell = $1 AND cursor <= $2 + AND cursor < (SELECT max(cursor) FROM sandboxd_usage WHERE cell = u.cell AND node = u.node); + ``` +- **Renaming** a node, or moving it to another cell, starts it with no rows: + the node lands what its journal still holds again under the new name. + ### Reloading the config `kill -HUP ` or a root `POST /v1/config/reload` diff --git a/docs/index.md b/docs/index.md index 4005a401c..d3aaec111 100644 --- a/docs/index.md +++ b/docs/index.md @@ -27,7 +27,7 @@ vsock-only I/O (hardened default); `net=egress` attaches a bridge/CNI NIC. renew, hibernate, fork, promote, checkpoints, preview, the guest-port relay, claim env, runtime volume registration, live volume changes, runtime tenants, config reload, online pool reconfigure, metrics, the usage - journal + journal and its shared table - [Replace a dataset in a running sandbox](live-volumes.md) — swapping a read-only catalog volume on a running claim with the volume API - [Hibernate with an external volume controller](volume-controller.md) — detach, diff --git a/docs/sandboxd-api.md b/docs/sandboxd-api.md index 98b166394..8760c3084 100644 --- a/docs/sandboxd-api.md +++ b/docs/sandboxd-api.md @@ -9,7 +9,7 @@ All bodies are JSON. Three token kinds: tenant creates is stamped with its name. Operator surfaces (the per-id sandbox reads, `PUT /v1/sandboxes/{id}/instance-metadata`, `GET /v1/info`, `PUT /v1/pools`, `PUT/DELETE /v1/volumes/{name}`, `POST/DELETE /v1/drain`, - `GET /metrics`, `GET /v1/checkpoints/{id}/blob`) + `GET /metrics`, `GET /v1/usage`, `GET /v1/checkpoints/{id}/blob`) answer a tenant token `403` — authenticated but not authorized; an unknown token stays `401`. On a node with a shared [`meta_store`](deploy.md#shared-tenant-database), a tenant token the node @@ -1015,8 +1015,8 @@ Always on: every lifecycle transition appends one JSONL event to `/usage.jsonl` — `{"t": , "ev": "claim|hibernate|wake|fork|checkpoint|promote|release|reap|archive|unarchive|archive_delete|egress|egress_bytes|vmm_exit|vmm_restart|vmm_failed", "id": "sb_…", "vm": "sbx-…"}` plus `key` (the pool key's stable hash, claim -events), `tenant` (the owning tenant, on claim, egress, archive, unarchive and -archive_delete), `children` (fork) and `ref` (the promoted +events), `tenant` (the owning tenant, on every event of a tenant's claim), +`children` (fork) and `ref` (the promoted template / checkpoint id, the egress host, or a `vmm_failed` reason). An egress event that injected credentials names them in `secret` (`gh,claim:`, never a value). An egress event routed through an [upstream proxy](egress.md#upstream-proxies) carries `upstream` (its `host:port`), and with @@ -1025,7 +1025,8 @@ them in `secret` (`gh,claim:`, never a value). An egress event routed thr the subset of those names claimed `rw`, so billing can discriminate write access (mounts and host paths are not billing dimensions). The file rotates at 64 MiB keeping one `.1` backup, so a tailing collector never loses a window -silently. Folding rules: billable compute seconds per sandbox = +silently. A last line a crash cut short is ended with a newline when the node +reopens the journal, so it never swallows the next event. Folding rules: billable compute seconds per sandbox = Σ(claim→release/reap) − Σ(hibernate→wake) − Σ(vmm_exit→vmm_restart/release/reap); `vmm_exit` is stamped when the node first finds the VMM gone, so an outage before sandboxd runs again (a host that was down) stays billed up to that @@ -1035,7 +1036,39 @@ point, and `vmm_failed` is informational; hibernated storage seconds = VM's RAM. An interval left open by a crash clamps to the claim's deadline; reconcile records nothing for the claims it drops (those whose VM record is gone), so a restart leaves those intervals without a terminal event. The `vm` name joins -cocoon's machine-level metering ledger for audit cross-checks. +cocoon's machine-level metering ledger for audit cross-checks. With +`meta_store.usage`, the node also lands every line in the cell's +[shared usage table](deploy.md#shared-usage-table), read through +[`GET /v1/usage`](#get-v1usage). + +## GET /v1/usage + +Auth: root only (tenant tokens get 403). Reads the raw usage events of every +node of the cell from the [shared usage table](deploy.md#shared-usage-table). + +- `GET /v1/usage?tenant=&since=