sandboxd: a missing state for host-only env, kept off disk behind host_env_memory_only

docs/deploy.md

@@ -45,6 +45,19 @@ every archived claim and delete the checkpoints, then re-promote templates, or
 point `prefix` at an empty location. Objects of the old layout under a reused
 prefix stay in the bucket until you delete them.
 
+From v0.1.24, a claim env write refuses a `guest: false` entry with an empty
+value, so sending back a `GET` env reply answers 400; send each entry with its
+value. An empty value no longer blocks the node's secret for one claim. A node
+that ran with `host_env_memory_only` stores entries marked `missing`, which
+v0.1.24 cannot read. Before going back to it, turn the flag off and restart,
+then send every missing entry again or release its claim, until no claim's
+`env_missing` is true. Each write stores the value in `claims.json` and in the
+row of an archived or hibernated claim. A running replicated claim's row takes
+it only at a later replica, with no fixed bound, so hibernate each such claim
+and send its entries again; a write that succeeds has rewritten its row. A
+`claims.json` that still holds a missing entry stops v0.1.24 from starting, and
+a row that holds one cannot be adopted by it.
+
 ## Configuration
 
 sandboxd reads one JSON file (`-config`, default
@@ -101,7 +114,8 @@ the intersection of the two.
 | `bridges` / `networks` | unset | egress-lane attachment: a list of host bridge devices, or a list of CNI conflist names. Mutually exclusive; with neither set the node serves only the no-network lane. A Linux bridge holds at most 1024 ports (kernel `BR_MAX_PORTS`), so an N-entry list raises the node's egress ceiling to N×1024 — VMs spread over the list by a stable hash of the VM name, so size it with headroom (the spread is statistical, not exact). `bridges` keeps the raw TAP-on-bridge attachment (taps in the root netns, no per-VM network namespace or CNI plugin execution); `networks` runs the CNI chain per VM. [Guarded egress](egress.md) on the egress lane (an egress-lane pool policy or any `egress_classes` entry) needs `bridges` and rejects a CNI network at load; none-lane pool policies ride the proxy on either |
 | `volumes` | unset | node-local catalog of operator-managed dataset images: `[ {"name":"imagenet","path":"/srv/datasets/imagenet.img","directio":"off","tenants":["acme"]}, {"name":"scratch-db","path":"/srv/datasets/scratch.img","writable":true} ]`. Names match `^[a-z][a-z0-9_-]{0,19}$` and cannot start with `cocoon-`; paths are absolute; `directio` is `on`, `off`, or `auto` and defaults to `off` for both read-only and writable entries. `tenants` is an optional [access list](sandboxd-api.md#volume-access-lists); a listed name the tenant set does not hold grants nothing and logs a warning at boot (the [tenant API](sandboxd-api.md#tenants-v1tenants) may add it later). `writable` (default `false`) lets a claim request `mode: "rw"` on that entry — see [Dataset volumes](#dataset-volumes). The catalog is intentionally not part of the cluster digest |
 | `volume_roots` | unset | absolute directories that bound [runtime registration](sandboxd-api.md#put-v1volumesname): the registered path, once cleaned, must equal a root or sit under one, or the PUT returns 400. The check is lexical: `..` is resolved but symlinks are not followed, so a symlink under a root can point outside it. Static `volumes` entries are not checked. A saved registration outside the roots prevents startup. Unregister it before narrowing the roots, or widen the roots to include it. Keep `data_dir` outside every root. Unset or an empty list accepts any absolute path |
-| `secrets` | unset | credentials the egress proxy injects by name: `[{"name": "gh", "header": "Authorization", "value_env": "GH_TOKEN"}]`. A pool or egress-class rule references the name (on a claim where both apply, only the pool rule's secret is injected); the value comes from the claim's `guest: false` env, then the node's environment, never this file. Without `value_env` the secret's name is the env to read, and only claims supply it. See [egress](egress.md#claim-env-and-secrets) |
+| `secrets` | unset | credentials the egress proxy injects by name: `[{"name": "gh", "header": "Authorization", "value_env": "GH_TOKEN"}]`. A pool or egress-class rule references the name (on a claim where both apply, only the pool rule's secret is injected); the value comes from the claim's `guest: false` env, then the node's environment, never this file. Without `value_env` the secret's name is the env to read, and only claims supply it. See [egress](egress.md#claim-env-and-secrets). Prefer a claim's own `guest: false` entry over `value_env`: a node value is shared by every claim of the cell, long-lived, and stored in the node's environment file |
+| `host_env_memory_only` | `false` | keep claims' `guest: false` env values in memory only: `claims.json` and the [archive and replica rows](#shared-archive-registry) store such an entry without its value and mark it `missing`, and a hibernating claim drops its values. A missing entry injects nothing and blocks the node's value for its secret until the control plane sends it again with `PATCH` ([claim env](sandboxd-api.md#getputpatch-v1sandboxesidenv)). A live move carries the values to its target. Turn it on only when every node of the cell runs v0.1.25 or later with it, and the control plane sends missing entries again. The first start with it on rewrites `claims.json` without the values; an archive row written before keeps its values until the row is next written. Restart only |
 | `egress_internal_allow` | unset | CIDR prefixes re-admitted through the egress proxy's SSRF guard, node-wide (every pool and tenant); `prefix:port,port` scopes an entry to those ports, a bare prefix admits every port. Prefer the port form. Prefixes, not a permit-private switch — the guest bridges are themselves ULA/RFC1918. See [egress](egress.md) |
 | `egress_upstream` | unset | per-claim upstream proxies: `{"claim_env": "EGRESS_UPSTREAM", "allow": ["res.example.com", "10.1.0.0/16"]}`. A claim's `guest: false` entry under `claim_env` (`http://` or `socks5://`, optional `user:pass@`, or `direct`) routes its egress through that upstream, which must be on `allow` (host names, IPs, CIDR prefixes). See [upstream proxies](egress.md#upstream-proxies) |
 | `egress_conn_rate` | unset | caps the new upstream connections each claim's proxy opens: `{"per_second": 30, "burst": 600}`, a token bucket per claim; a reused keep-alive connection and a connection to an internal IP address (the SSRF guard decides those) cost nothing, while a host name costs a token even when it resolves to one `egress_internal_allow` re-admits. A connection past it gets 429 (SOCKS5: not allowed) and an audit `decision` of `rate_limited`. A claim's bucket starts full at its first connection, and again when a reload changes `per_second` or `burst`. Unset leaves the rate unlimited |
@@ -138,7 +152,7 @@ the intersection of the two.
 | `idle_hibernate_seconds` | 0 (off) | node-wide idle policy for unpooled claims (template/checkpoint claims): a none-lane claim is hibernated once it has had no open data-plane connection (relay, buffered exec, preview), no egress request in flight and no [advanced-network](advanced-network.md) packet carried for this long; the clock restarts when the last connection closes, so a long command is never cut short. The next call that reaches the guest wakes it transparently. Per-pool `idle_hibernate_seconds` does the same for that pool's claims; pooled keys ignore the node-wide value, and egress pools reject it because they cannot resume safely. Opt in deliberately: a wake costs latency and the snapshot, so callers with their own idle logic must not pay twice |
 | `archive_after_seconds` | 0 (off) | tier below hibernation: a hibernated claim idle this long is checkpointed to the store and its local VM dropped, freeing the node entirely; the next call that reaches the guest restores it transparently (a checkpoint restore's latency) with a fresh lease of the length the claim was granted (the server default of 5m when it asked for none); a lease moves only when a client calls [`/renew`](sandboxd-api.md#post-v1sandboxesidrenew). Keep `idle_hibernate_seconds` under the leases your claims use, or a woken sandbox that idles again is destroyed by the reap before it can hibernate, its archive already consumed by the wake; one still in use when its lease ends is destroyed the same way, unless it claimed `on_expire: archive`, which archives at the lease end whatever this setting. A call that wakes a hibernated claim past its lease, before the export, starts that fresh lease too. Requires `idle_hibernate_seconds > 0` and must exceed it. Node-wide for unpooled keys; per-pool overrides for that pool |
 | `archive_delete_after_seconds` | 0 (keep) | purge an archived claim's store checkpoint this long after it was archived; the claim is then gone for good. On S3, chunks and manifests remain until [offline reclamation](#reclaiming-s3-storage). On archive this retention window replaces the live claim deadline; 0 clears the deadline so the archive is kept forever. An [evacuation](cluster.md#evacuating-a-node) keeps a longer live deadline, and a claim of a pool without an archive tier, whose `on_expire` is not `archive`, keeps its deadline until its first wake. Same node-wide/per-pool split |
-| `archive_failover` | false | let another node adopt this node's archived claims once its node lease expires, and the running claims of pools with `replicate_seconds`: each archive and replication round also writes a row (token SHA-256, owner, epoch, checkpoint id, retention deadline, and the claim record with its env, `guest: false` values included, in plaintext) to the shared PostgreSQL, and a `Lookup` that reaches any node takes the claim over there (see [archived claims across node loss](cluster.md#archived-claims-across-node-loss)). Requires `meta_store` with a `cell` (rows are scoped by it, so meshes sharing one database never adopt from each other), a `checkpoint_store` of kind `s3` and a `mesh` with a `cluster_key` (a hand-off presents the claim's token to an address learned from gossip), all checked at config load. The node also renews a lease row every 5 s (see [node lease](cluster.md#node-lease)). With the meta store down, archives, wakes, releases and env writes of archived claims, and env writes and renews of hibernated replicated claims, fail and retry, and replicated claims stop being served once the lease is overdue (see [running claims](cluster.md#running-claims-across-node-loss)); other running claims never touch it. Restart only |
+| `archive_failover` | false | let another node adopt this node's archived claims once its node lease expires, and the running claims of pools with `replicate_seconds`: each archive and replication round also writes a row (token SHA-256, owner, epoch, checkpoint id, retention deadline, and the claim record with its env, `guest: false` values included, in plaintext, unless `host_env_memory_only` is on) to the shared PostgreSQL, and a `Lookup` that reaches any node takes the claim over there (see [archived claims across node loss](cluster.md#archived-claims-across-node-loss)). Requires `meta_store` with a `cell` (rows are scoped by it, so meshes sharing one database never adopt from each other), a `checkpoint_store` of kind `s3` and a `mesh` with a `cluster_key` (a hand-off presents the claim's token to an address learned from gossip), all checked at config load. The node also renews a lease row every 5 s (see [node lease](cluster.md#node-lease)). With the meta store down, archives, wakes, releases and env writes of archived claims, and env writes and renews of hibernated replicated claims, fail and retry, and replicated claims stop being served once the lease is overdue (see [running claims](cluster.md#running-claims-across-node-loss)); other running claims never touch it. Restart only |
 | `mesh` | unset | join a cluster ([Clusters](cluster.md)); unset = single node |
 | `pools[]` | — | warm pools, keyed by `(template, net, size)`. `warm` defaults to 4 only when both `warm` and `warm_max` are omitted; an explicit `warm: 0` is kept, so `warm: 0` under a `warm_max` starts the pool empty and grows it on demand, and `warm: 0` alone registers the key with no warm VMs (`PUT /v1/pools` never defaults `warm`); `net` is `none` (the default) or `egress`; `size` is a tier, below (default `small`). Retune online without a restart via [`PUT /v1/pools`](sandboxd-api.md#put-v1pools) — omitted pools drain, and their goldens are deleted once they are idle (`warm: 0` keeps the golden). This is the **first-boot seed**: once a node takes a `PUT /v1/pools`, the applied set persists to `<data_dir>/pools.json` and overrides this section on every later boot (a startup log notes it); delete `pools.json` to return to config-owned pools. With a `meta_store` `cell`, the cell's set overrides it instead (see [shared pool set](#shared-pool-set)). Egress stays config-owned either way. A template re-pulled or re-imported under the same ref is picked up within a minute: the pool rebuilds its golden and replaces its warm VMs. See [state ownership](cluster.md#state-ownership) |
 
@@ -622,6 +636,9 @@ golden fail the same way, it is retired again at most once every 30 s.
 The systemd unit is
 [`packaging/sandboxd.service`](https://github.com/cocoonstack/sandbox/blob/main/packaging/sandboxd.service);
 install it as shipped. Its `KillMode` and `TimeoutStopSec` are explained below.
+`LimitCORE=0` keeps a crash from writing sandboxd's memory, which holds every
+running claim's host-only env values, to a core file. For the same reason,
+run sandboxd hosts with swap off or encrypted.
 
 Each record carries its syslog level, so alerting keys on systemd priority:
 `journalctl -u sandboxd -p err` lists the errors and `-p warning` the

docs/egress.md

@@ -372,20 +372,25 @@ gateway key per user, while the guest holds a placeholder.
   which hosts receive a secret, and the claim only supplies its value. A
   `guest: true` entry never feeds a secret.
 - **Precedence.** The claim's entry wins over the node's value for that claim;
-  an entry set to `""` suppresses the node's value. A secret with `value_env`
+  a `missing` entry ([claim env](sandboxd-api.md#getputpatch-v1sandboxesidenv))
+  blocks the node's value until it is sent again. Prefer claim entries for per-user credentials: a node value is
+  shared by every claim of the cell and long-lived. A secret with `value_env`
   needs that variable set on the node: unset or empty, it stops the daemon at
   start and a reload answers 400. A secret only claims supply omits `value_env`;
   on a claim with no entry for it, the matched request passes as the guest sent
   it.
 - **Lifetime.** The env lives in the claim record in `claims.json` (0600
   under `data_dir`), survives a restart, hibernate, wake, archive and
-  unarchive, and goes with the release or reap. A fork child, a checkpoint
+  unarchive, and goes with the release or reap. With
+  [`host_env_memory_only`](deploy.md#configuration) its `guest: false` values
+  live in memory only and come back `missing` after each of those but a
+  release, until the control plane sends them again. A fork child, a checkpoint
   branch and a promoted-template clone start with only the env their own claim
   sets. Processes captured live in a snapshot (sessions, detached execs,
   units) keep the source's guest values in their own environment; a
   `guest: false` value never enters the guest, so no snapshot, checkpoint or
-  store object holds one; the `archive_failover` row is the exception
-  ([security](security.md#deployment-assumptions)).
+  store object holds one; the `archive_failover` row is the exception unless
+  `host_env_memory_only` is on ([security](security.md#deployment-assumptions)).
 - **Audit.** An injection is recorded as the secret's name; the value is
   never logged, listed or returned.
 
@@ -470,7 +475,7 @@ changed: resend it, and the resend replaces the row.
 - **Precedence.** The pool rule's own `secret` is set first and wins: a
   credential naming that header for a host the rule covers is a 400. Among
   credentials for the same header, the first by entry name whose placeholder
-  (if any) the request carries wins. An empty value injects nothing.
+  (if any) the request carries wins. A missing entry injects nothing.
 - **Echo.** Origins reflect the request URL: a redirect's `Location`, a
   paging link, an echoed query. On a request that carried a `query`
   credential, its value, as sent, query- or path-escaped, or with `/` written