usage: land usage events in the shared meta store and serve GET /v1/usage

.github/workflows/sandboxd.yml

@@ -73,7 +73,7 @@ jobs:
         env:
           SANDBOXD_TEST_PG_URL: postgres://postgres:sbxci@localhost:5432/postgres?sslmode=disable
         run: |
-          GOWORK=off go test -race -count=1 -v ./tenants/... ./poolset/... | tee "$RUNNER_TEMP/pg-test.log"
+          GOWORK=off go test -race -count=1 -v ./tenants/... ./poolset/... ./archives/... ./usage/... | tee "$RUNNER_TEMP/pg-test.log"
           GOWORK=off go test -race -count=1 -v -run 'TestTwoNodesOnOneMetaStore|TestNodesOfACell|TestAMetaStoreNodeWithoutACell' ./pool/ | tee -a "$RUNNER_TEMP/pg-test.log"
           ! grep -- '--- SKIP' "$RUNNER_TEMP/pg-test.log"
 

README.md

@@ -45,8 +45,8 @@ performance) — source in
   restart reconcile with VMM cold restart, memberlist mesh with redirect
   placement, a checkpoint store on a shared directory or S3 (sparse,
   content-addressed chunks), and an optional shared PostgreSQL meta store
-  for tenants, pools, node leases and the cross-node adoption of archived
-  claims and of running claims a pool replicates (`replicate_seconds`)
+  for tenants, pools, node leases, usage events and the cross-node adoption of
+  archived claims and of running claims a pool replicates (`replicate_seconds`)
 - `sdk/go/` — Go SDK: `Connect/New/Lookup`, `Exec/Run`, files,
   `Push/Pull`, sessions, `Find/Replace`, `Watch`, git verbs, `OpenPty`,
   `Renew/Fork/Hibernate/Promote/Checkpoint`, claim env, `DialPort/ProxyPort/PreviewURL`,

docs/cluster.md

@@ -238,8 +238,11 @@ replication rounds write it, so the claim path is unchanged.
   replica retirement keeps its tombstone until the hourly sweep after that.
 - **Usage journal.** The journal stays per node: an adopted claim's `archive`
   is in the old owner's `usage.jsonl`, and its `unarchive`, `release` or
-  `archive_delete` is in the adopter's or the purging node's. Fold the journals
-  of all nodes by sandbox id.
+  `archive_delete` is in the adopter's or the purging node's. Fold the events
+  of all nodes by sandbox id. With `meta_store.usage` every node lands its
+  journal in the cell's [shared usage table](deploy.md#shared-usage-table), and
+  [`GET /v1/usage`](sandboxd-api.md#get-v1usage) on any node returns the
+  events of all of them.
 
 The node lease decides adoption, so a partitioned owner can lose an archived
 claim it still lists once its lease runs out; the epoch fences it out. A claim

docs/deploy.md

@@ -129,7 +129,7 @@ gets the intersection of the two.
 | `egress_usage_bytes` | `false` | append an `egress_bytes` usage event with the payload bytes of every allowed egress tunnel or request when it ends; one extra usage-journal write per connection |
 | `egress_ca` | unset | [HTTPS-interception](egress.md#https-interception) PKI: `root_cert` (the cluster root baked into intercepted guests; may bundle old+new roots during rotation) plus this node's `intermediate_cert`/`intermediate_key` from `sandboxd ca issue-intermediate`. Required when any pool rule sets `intercept` |
 | `api_token` | unset | the operator (root) credential: when set, guards the node-level endpoints (Bearer) with full access, including release-by-id cleanup. Per-sandbox tokens guard ordinary sandbox-scoped calls |
-| `meta_store` | unset | `{"kind": "pg", "dsn_env": "SANDBOXD_PG_DSN"}`: keep the tenant set in one shared PostgreSQL instead of each node's `tenants.json` (see [shared tenant database](#shared-tenant-database)). `dsn_env` names the node env holding the connection string. An optional `cell` also keeps the API-applied pool set there, one per cell, instead of in each node's `pools.json` (see [shared pool set](#shared-pool-set)); without it pool targets stay per node. Needs `api_token`. Restart only |
+| `meta_store` | unset | `{"kind": "pg", "dsn_env": "SANDBOXD_PG_DSN"}`: keep the tenant set in one shared PostgreSQL instead of each node's `tenants.json` (see [shared tenant database](#shared-tenant-database)). `dsn_env` names the node env holding the connection string. An optional `cell` also keeps the API-applied pool set there, one per cell, instead of in each node's `pools.json` (see [shared pool set](#shared-pool-set)); without it pool targets stay per node. An optional `usage: true` also lands this node's usage journal in one shared table and serves [`GET /v1/usage`](sandboxd-api.md#get-v1usage) from it (see [shared usage table](#shared-usage-table)). Needs `api_token`. Restart only |
 | `max_fork_count` | 16 | children a single `fork` may create; each is a full-RAM VM, so this bounds one request's memory blast radius to the node's capacity |
 | `refill_concurrency` | 0 (auto) | concurrent VM provisioning budget, shared by warm-pool refills, fork clones, and the reap/hibernate/reconcile engine batches; the batches never take the last quarter of it (at least one slot), so a mass expiry or idle sweep cannot stop refills. 0 sizes it from the node: `NumCPU*2/3` clamped to [4, 256] — a 384-core node gets 256; small nodes keep a floor of 4 |
 | `release_delay_seconds` | 0 | seconds a released VM waits in the removal queue before `cocoon vm rm`. The claim is dropped and journaled and its egress listener closed at release; the VM removal, the volume hold release and the egress tap unlock run on the first reap tick (5 s) after the delay, on the `refill_concurrency` budget like every other batch teardown, so a burst of releases does not compete with the claims still running. Until then the VM holds its memory and its volume reservations without counting as a claim. 0 removes inline |
@@ -600,14 +600,14 @@ sandbox and checkpoint listings filter to the caller's tenant, and a tenant
 can delete only its own checkpoints and templates (root sees everything).
 Operator surfaces stay root-only — per-id sandbox reads and stats,
 `GET /v1/checkpoints/{id}/blob`, `GET /v1/info`, `PUT /v1/pools`,
-`/v1/tenants`, `/v1/config/reload`, `POST/DELETE /v1/drain`, `/metrics`; a
+`/v1/tenants`, `/v1/config/reload`, `POST/DELETE /v1/drain`, `/metrics`, `GET /v1/usage`; a
 tenant token there is authenticated but not authorized, so it answers 403 (a
 wrong token stays 401). Per-sandbox
 tokens are unchanged: whoever holds a sandbox's token drives that sandbox.
 Fork children inherit the parent's tenant and count against its
 `max_claims` (0 = unlimited, a per-node cap, so a tenant's cluster limit is
 `max_claims` × nodes); a tenant at its cap gets 429 exactly like a node at
-`max_claims`, and the usage journal's claim events carry the tenant for
+`max_claims`, and every usage event of a tenant's claim carries the tenant for
 per-tenant billing. `config.json` has no `tenants` field: a node starts with
 no tenants until the API adds them.
 
@@ -748,6 +748,9 @@ through `/v1/tenants` on any node.
   - The pool defaults to `max(4, CPUs)` connections; set `pool_max_conns` in
     the DSN (4 suffices) so a fleet stays within the database's
     `max_connections`.
+  - Every other shared table a node opens adds a query pool of the same size:
+    the pool set with a `cell`, the archive registry with `archive_failover`,
+    and the usage table with `usage`. Count each in `max_connections`.
   - LISTEN needs a session-mode connection, so a connection pooler in front
     of the database must not run in transaction mode for that DSN.
   - A write runs in one transaction with a `pg_notify`.
@@ -809,6 +812,98 @@ the tenant set.
 - **Reverting** to config-owned pools is a `PUT` of the config pools; changing
   a node's `cell` takes a restart.
 
+### Shared usage table
+
+With `meta_store.usage: true`, each node also lands its usage journal in one
+table, and root reads the events of every node of the cell through
+[`GET /v1/usage`](sandboxd-api.md#get-v1usage) on any of them. It is off by
+default, so an upgrade adds no table and no writes. The table holds raw events
+only; computing durations or prices stays with the consumer.
+
+- **Landing:** a background loop reads `usage.jsonl` and its `.1` backup every
+  5 s and lands the lines it has not landed yet, up to 1000 per transaction,
+  until it catches up. The journal append is unchanged, so a claim, release,
+  hibernate or wake never waits on the database.
+- **Node:** rows carry the node's mesh `node_id` (its `bind` when unset), or
+  its host name without a mesh. The name must be unique within the cell.
+- **Schema:** sandboxd creates the table at boot when it is missing; create it
+  ahead with the same DDL when the node's role has no `CREATE` right:
+
+  ```sql
+  CREATE TABLE IF NOT EXISTS sandboxd_usage (
+      cell        text        NOT NULL,
+      cursor      bigint      GENERATED ALWAYS AS IDENTITY,
+      node        text        NOT NULL,
+      line_sha256 bytea       NOT NULL,
+      t           timestamptz NOT NULL,
+      ev          text        NOT NULL,
+      id          text        NOT NULL,
+      vm          text        NOT NULL DEFAULT '',
+      key         text        NOT NULL DEFAULT '',
+      tenant      text        NOT NULL DEFAULT '',
+      volumes     text[],
+      volumes_rw  text[],
+      children    text[],
+      ref         text        NOT NULL DEFAULT '',
+      upstream    text        NOT NULL DEFAULT '',
+      secret      text        NOT NULL DEFAULT '',
+      tx          bigint      NOT NULL DEFAULT 0,
+      rx          bigint      NOT NULL DEFAULT 0,
+      PRIMARY KEY (cell, cursor),
+      UNIQUE (cell, node, line_sha256)
+  );
+  CREATE INDEX IF NOT EXISTS sandboxd_usage_node ON sandboxd_usage (cell, node, cursor);
+  CREATE INDEX IF NOT EXISTS sandboxd_usage_tenant ON sandboxd_usage (cell, tenant, cursor);
+  CREATE INDEX IF NOT EXISTS sandboxd_usage_t ON sandboxd_usage (cell, t);
+  ```
+
+- **Columns:** the [journal line](sandboxd-api.md#usage-journal-usagejsonl)'s
+  keys, with an absent key as `''`, `0` or `NULL`, plus `cell` (`''` without
+  one), `node` and `line_sha256`, the SHA-256 of the line. `t` keeps
+  microseconds, and a NUL in `ref` lands as U+FFFD.
+- **At most once per line:** `(cell, node, line_sha256)` is unique, and a
+  land skips a line whose digest already landed for its node, so no journal
+  line lands twice. The node's newest row (by `cursor`) is its resume point:
+  at boot and after a failed land, the node reads that row's `line_sha256`
+  and resumes after that line in
+  `usage.jsonl.1` or `usage.jsonl`. When neither file holds the line any
+  more (a crash lost a journal tail the node had already landed, or the
+  journal was reset), the node reads both files from the start and lands only
+  the lines not landed yet. A restart, a retry or a commit whose reply was
+  lost therefore lands no line twice. Two byte-identical lines of one node
+  (same nanosecond `t` and fields) land once. A line that does not decode is skipped with
+  a warning, and a last line without its newline waits for it. A line a
+  crash cut short stays a line of its own: the node ends it with a newline
+  when it opens the journal, so the first event after the restart is whole.
+- **Order:** a cell's lands commit one at a time under a transaction-scoped
+  advisory lock, so `cursor` grows in commit order and a reader that pages by
+  cursor never skips a row that commits later. A node lands its lines in
+  journal order, so its rows follow that order by `cursor`. A backfilled row
+  gets a newer cursor than rows with a later `t`.
+- **Connection:** the table has its own query pool, sized like the others (see
+  [shared tenant database](#shared-tenant-database)), and no LISTEN session.
+  The node lands on one connection at a time; `GET /v1/usage` reads take
+  more.
+- **Outage:** claims and the journal carry on; the node logs one warning,
+  retries every 5 s, and backfills when the database returns. Nothing is lost
+  while the unlanded lines are still in `usage.jsonl` or `usage.jsonl.1`, so
+  an outage loses lines only after the journal rotates twice past the last
+  landed one (64 MiB each); the node no longer holds those either. A node
+  needs the database at boot, like the other shared tables.
+- **Retention:** sandboxd never deletes a row. Prune only rows every consumer
+  has paged past, and keep each node's newest row: it is the node's resume
+  point, and a node that finds none lands its whole journal again. A crash
+  that loses that line from the journal also lands again the pruned lines the
+  journal still holds.
+
+  ```sql
+  DELETE FROM sandboxd_usage u
+  WHERE cell = $1 AND cursor <= $2
+    AND cursor < (SELECT max(cursor) FROM sandboxd_usage WHERE cell = u.cell AND node = u.node);
+  ```
+- **Renaming** a node, or moving it to another cell, starts it with no rows:
+  the node lands what its journal still holds again under the new name.
+
 ### Reloading the config
 
 `kill -HUP <sandboxd pid>` or a root `POST /v1/config/reload`

docs/index.md

@@ -27,7 +27,7 @@ vsock-only I/O (hardened default); `net=egress` attaches a bridge/CNI NIC.
   renew, hibernate, fork, promote, checkpoints, preview, the guest-port
   relay, claim env, runtime volume registration, live volume changes,
   runtime tenants, config reload, online pool reconfigure, metrics, the usage
-  journal
+  journal and its shared table
 - [Replace a dataset in a running sandbox](live-volumes.md) — swapping a
   read-only catalog volume on a running claim with the volume API
 - [Hibernate with an external volume controller](volume-controller.md) — detach,

docs/sandboxd-api.md

@@ -9,7 +9,7 @@ All bodies are JSON. Three token kinds:
   tenant creates is stamped with its name. Operator surfaces
   (the per-id sandbox reads, `PUT /v1/sandboxes/{id}/instance-metadata`, `GET /v1/info`,
   `PUT /v1/pools`, `PUT/DELETE /v1/volumes/{name}`, `POST/DELETE /v1/drain`,
-  `GET /metrics`, `GET /v1/checkpoints/{id}/blob`)
+  `GET /metrics`, `GET /v1/usage`, `GET /v1/checkpoints/{id}/blob`)
   answer a tenant token `403` — authenticated but not authorized; an unknown
   token stays `401`. On a node with a shared
   [`meta_store`](deploy.md#shared-tenant-database), a tenant token the node
@@ -1015,8 +1015,8 @@ Always on: every lifecycle transition appends one JSONL event to
 `<data_dir>/usage.jsonl` — `{"t": <RFC3339>, "ev":
 "claim|hibernate|wake|fork|checkpoint|promote|release|reap|archive|unarchive|archive_delete|egress|egress_bytes|vmm_exit|vmm_restart|vmm_failed",
 "id": "sb_…", "vm": "sbx-…"}` plus `key` (the pool key's stable hash, claim
-events), `tenant` (the owning tenant, on claim, egress, archive, unarchive and
-archive_delete), `children` (fork) and `ref` (the promoted
+events), `tenant` (the owning tenant, on every event of a tenant's claim),
+`children` (fork) and `ref` (the promoted
 template / checkpoint id, the egress host, or a `vmm_failed` reason). An egress event that injected credentials names
 them in `secret` (`gh,claim:<entry>`, never a value). An egress event routed through an
 [upstream proxy](egress.md#upstream-proxies) carries `upstream` (its `host:port`), and with
@@ -1025,7 +1025,8 @@ them in `secret` (`gh,claim:<entry>`, never a value). An egress event routed thr
 the subset of those names claimed `rw`, so billing can discriminate write
 access (mounts and host paths are not billing dimensions). The file rotates at
 64 MiB keeping one `.1` backup, so a tailing collector never loses a window
-silently. Folding rules: billable compute seconds per sandbox =
+silently. A last line a crash cut short is ended with a newline when the node
+reopens the journal, so it never swallows the next event. Folding rules: billable compute seconds per sandbox =
 Σ(claim→release/reap) − Σ(hibernate→wake) − Σ(vmm_exit→vmm_restart/release/reap);
 `vmm_exit` is stamped when the node first finds the VMM gone, so an outage
 before sandboxd runs again (a host that was down) stays billed up to that
@@ -1035,7 +1036,39 @@ point, and `vmm_failed` is informational; hibernated storage seconds =
 VM's RAM. An interval left open by a crash clamps to the claim's
 deadline; reconcile records nothing for the claims it drops (those whose VM
 record is gone), so a restart leaves those intervals without a terminal event. The `vm` name joins
-cocoon's machine-level metering ledger for audit cross-checks.
+cocoon's machine-level metering ledger for audit cross-checks. With
+`meta_store.usage`, the node also lands every line in the cell's
+[shared usage table](deploy.md#shared-usage-table), read through
+[`GET /v1/usage`](#get-v1usage).
+
+## GET /v1/usage
+
+Auth: root only (tenant tokens get 403). Reads the raw usage events of every
+node of the cell from the [shared usage table](deploy.md#shared-usage-table).
+
+- `GET /v1/usage?tenant=<name>&since=<time>&until=<time>&after=<cursor>&limit=<n>` →
+  `{"events": [{"cursor": 41, "node": "n1", "t": "2026-10-30T08:00:00.123456Z", "ev": "claim", "id": "sb_…", "vm": "sbx-…", "key": "…", "tenant": "acme"}], "next": 41}`.
+- **Events:** each is one journal line with its keys as in
+  [usage.jsonl](#usage-journal-usagejsonl), `t` in UTC to the microsecond, plus
+  `cursor` (its place in the table; one node's events follow its journal
+  order) and `node` (the node that recorded it).
+- **Filters:** `tenant` keeps one tenant's events; `since` (inclusive) and
+  `until` (exclusive) are RFC 3339 times compared with `t`. Each is optional.
+  A journal written by v0.1.20 or earlier stamps `tenant` only on `claim`,
+  `archive`, `unarchive`, `archive_delete`, `egress`, `egress_bytes` and
+  `vmm_*`; join its other events on `id` to the `claim`.
+- **Order and paging:**
+  - events come in `cursor` order, the order they landed, which is not `t`
+    order: a node that was cut off from the database lands its backlog later;
+  - `after` returns only events with a larger cursor (default 0);
+  - `limit` defaults to 1000, at most 10000;
+  - `next` is the last event's cursor when the page is full, absent otherwise;
+    pass it as the following page's `after`;
+  - the cell's lands commit in cursor order, so polling with the last cursor seen
+    as `after` returns every event exactly once, backfill included.
+- 400 on a malformed `after`, `limit`, `since` or `until`; 501
+  `no shared usage store configured` on a node without `meta_store.usage`; 503
+  `usage store down; retry` while the database is unreachable.
 
 ## GET /v1/sandboxes/{id}/agent
 

sandboxd/backends.go

@@ -17,6 +17,7 @@ import (
 	"github.com/cocoonstack/sandbox/sandboxd/tenants"
 	"github.com/cocoonstack/sandbox/sandboxd/tenants/file"
 	tenantspg "github.com/cocoonstack/sandbox/sandboxd/tenants/pg"
+	usagepg "github.com/cocoonstack/sandbox/sandboxd/usage/pg"
 )
 
 func openBackends(ctx context.Context, cfg *config.Config) (pool.Backends, error) {
@@ -47,6 +48,21 @@ func openBackends(ctx context.Context, cfg *config.Config) (pool.Backends, error
 	return b, nil
 }
 
+func openUsage(ctx context.Context, cfg *config.Config) (*usagepg.Store, error) {
+	ms := cfg.MetaStore
+	if ms == nil || !ms.Usage {
+		return nil, nil
+	}
+	return usagepg.Open(ctx, os.Getenv(ms.DSNEnv), ms.Cell)
+}
+
+func usageNode(cfg *config.Config) (string, error) {
+	if cfg.Mesh != nil {
+		return cfg.Mesh.ID(), nil
+	}
+	return os.Hostname()
+}
+
 func newStoreView(ctx context.Context, cfg *config.Config, staging string, idRe *regexp.Regexp) (store.Store, error) {
 	if cs := cfg.CheckpointStore; cs != nil && cs.Kind == "s3" {
 		return s3.New(ctx, *cs.S3, filepath.Join(cfg.DataDir, staging), idRe)

sandboxd/config/config.go

@@ -158,6 +158,8 @@ type MetaStoreConfig struct {
 	DSNEnv string `json:"dsn_env"`
 	// Cell opts the node into one pool set for every node of the cell; empty keeps the pool set per node.
 	Cell string `json:"cell,omitempty"`
+	// Usage lands the node's usage journal in the shared usage table and serves GET /v1/usage from it.
+	Usage bool `json:"usage,omitzero"`
 }
 
 // VolumeSpec declares one operator-managed dataset disk, held by exactly one node.