diff --git a/docs/deploy.md b/docs/deploy.md index ba916d6e4..0dc8da21e 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -45,6 +45,19 @@ every archived claim and delete the checkpoints, then re-promote templates, or point `prefix` at an empty location. Objects of the old layout under a reused prefix stay in the bucket until you delete them. +From v0.1.24, a claim env write refuses a `guest: false` entry with an empty +value, so sending back a `GET` env reply answers 400; send each entry with its +value. An empty value no longer blocks the node's secret for one claim. A node +that ran with `host_env_memory_only` stores entries marked `missing`, which +v0.1.24 cannot read. Before going back to it, turn the flag off and restart, +then send every missing entry again or release its claim, until no claim's +`env_missing` is true. Each write stores the value in `claims.json` and in the +row of an archived or hibernated claim. A running replicated claim's row takes +it only at a later replica, with no fixed bound, so hibernate each such claim +and send its entries again; a write that succeeds has rewritten its row. A +`claims.json` that still holds a missing entry stops v0.1.24 from starting, and +a row that holds one cannot be adopted by it. + ## Configuration sandboxd reads one JSON file (`-config`, default @@ -101,7 +114,8 @@ the intersection of the two. | `bridges` / `networks` | unset | egress-lane attachment: a list of host bridge devices, or a list of CNI conflist names. Mutually exclusive; with neither set the node serves only the no-network lane. A Linux bridge holds at most 1024 ports (kernel `BR_MAX_PORTS`), so an N-entry list raises the node's egress ceiling to N×1024 — VMs spread over the list by a stable hash of the VM name, so size it with headroom (the spread is statistical, not exact). `bridges` keeps the raw TAP-on-bridge attachment (taps in the root netns, no per-VM network namespace or CNI plugin execution); `networks` runs the CNI chain per VM. [Guarded egress](egress.md) on the egress lane (an egress-lane pool policy or any `egress_classes` entry) needs `bridges` and rejects a CNI network at load; none-lane pool policies ride the proxy on either | | `volumes` | unset | node-local catalog of operator-managed dataset images: `[ {"name":"imagenet","path":"/srv/datasets/imagenet.img","directio":"off","tenants":["acme"]}, {"name":"scratch-db","path":"/srv/datasets/scratch.img","writable":true} ]`. Names match `^[a-z][a-z0-9_-]{0,19}$` and cannot start with `cocoon-`; paths are absolute; `directio` is `on`, `off`, or `auto` and defaults to `off` for both read-only and writable entries. `tenants` is an optional [access list](sandboxd-api.md#volume-access-lists); a listed name the tenant set does not hold grants nothing and logs a warning at boot (the [tenant API](sandboxd-api.md#tenants-v1tenants) may add it later). `writable` (default `false`) lets a claim request `mode: "rw"` on that entry — see [Dataset volumes](#dataset-volumes). The catalog is intentionally not part of the cluster digest | | `volume_roots` | unset | absolute directories that bound [runtime registration](sandboxd-api.md#put-v1volumesname): the registered path, once cleaned, must equal a root or sit under one, or the PUT returns 400. The check is lexical: `..` is resolved but symlinks are not followed, so a symlink under a root can point outside it. Static `volumes` entries are not checked. A saved registration outside the roots prevents startup. Unregister it before narrowing the roots, or widen the roots to include it. Keep `data_dir` outside every root. Unset or an empty list accepts any absolute path | -| `secrets` | unset | credentials the egress proxy injects by name: `[{"name": "gh", "header": "Authorization", "value_env": "GH_TOKEN"}]`. A pool or egress-class rule references the name (on a claim where both apply, only the pool rule's secret is injected); the value comes from the claim's `guest: false` env, then the node's environment, never this file. Without `value_env` the secret's name is the env to read, and only claims supply it. See [egress](egress.md#claim-env-and-secrets) | +| `secrets` | unset | credentials the egress proxy injects by name: `[{"name": "gh", "header": "Authorization", "value_env": "GH_TOKEN"}]`. A pool or egress-class rule references the name (on a claim where both apply, only the pool rule's secret is injected); the value comes from the claim's `guest: false` env, then the node's environment, never this file. Without `value_env` the secret's name is the env to read, and only claims supply it. See [egress](egress.md#claim-env-and-secrets). Prefer a claim's own `guest: false` entry over `value_env`: a node value is shared by every claim of the cell, long-lived, and stored in the node's environment file | +| `host_env_memory_only` | `false` | keep claims' `guest: false` env values in memory only: `claims.json` and the [archive and replica rows](#shared-archive-registry) store such an entry without its value and mark it `missing`, and a hibernating claim drops its values. A missing entry injects nothing and blocks the node's value for its secret until the control plane sends it again with `PATCH` ([claim env](sandboxd-api.md#getputpatch-v1sandboxesidenv)). A live move carries the values to its target. Turn it on only when every node of the cell runs v0.1.25 or later with it, and the control plane sends missing entries again. The first start with it on rewrites `claims.json` without the values; an archive row written before keeps its values until the row is next written. Restart only | | `egress_internal_allow` | unset | CIDR prefixes re-admitted through the egress proxy's SSRF guard, node-wide (every pool and tenant); `prefix:port,port` scopes an entry to those ports, a bare prefix admits every port. Prefer the port form. Prefixes, not a permit-private switch — the guest bridges are themselves ULA/RFC1918. See [egress](egress.md) | | `egress_upstream` | unset | per-claim upstream proxies: `{"claim_env": "EGRESS_UPSTREAM", "allow": ["res.example.com", "10.1.0.0/16"]}`. A claim's `guest: false` entry under `claim_env` (`http://` or `socks5://`, optional `user:pass@`, or `direct`) routes its egress through that upstream, which must be on `allow` (host names, IPs, CIDR prefixes). See [upstream proxies](egress.md#upstream-proxies) | | `egress_conn_rate` | unset | caps the new upstream connections each claim's proxy opens: `{"per_second": 30, "burst": 600}`, a token bucket per claim; a reused keep-alive connection and a connection to an internal IP address (the SSRF guard decides those) cost nothing, while a host name costs a token even when it resolves to one `egress_internal_allow` re-admits. A connection past it gets 429 (SOCKS5: not allowed) and an audit `decision` of `rate_limited`. A claim's bucket starts full at its first connection, and again when a reload changes `per_second` or `burst`. Unset leaves the rate unlimited | @@ -138,7 +152,7 @@ the intersection of the two. | `idle_hibernate_seconds` | 0 (off) | node-wide idle policy for unpooled claims (template/checkpoint claims): a none-lane claim is hibernated once it has had no open data-plane connection (relay, buffered exec, preview), no egress request in flight and no [advanced-network](advanced-network.md) packet carried for this long; the clock restarts when the last connection closes, so a long command is never cut short. The next call that reaches the guest wakes it transparently. Per-pool `idle_hibernate_seconds` does the same for that pool's claims; pooled keys ignore the node-wide value, and egress pools reject it because they cannot resume safely. Opt in deliberately: a wake costs latency and the snapshot, so callers with their own idle logic must not pay twice | | `archive_after_seconds` | 0 (off) | tier below hibernation: a hibernated claim idle this long is checkpointed to the store and its local VM dropped, freeing the node entirely; the next call that reaches the guest restores it transparently (a checkpoint restore's latency) with a fresh lease of the length the claim was granted (the server default of 5m when it asked for none); a lease moves only when a client calls [`/renew`](sandboxd-api.md#post-v1sandboxesidrenew). Keep `idle_hibernate_seconds` under the leases your claims use, or a woken sandbox that idles again is destroyed by the reap before it can hibernate, its archive already consumed by the wake; one still in use when its lease ends is destroyed the same way, unless it claimed `on_expire: archive`, which archives at the lease end whatever this setting. A call that wakes a hibernated claim past its lease, before the export, starts that fresh lease too. Requires `idle_hibernate_seconds > 0` and must exceed it. Node-wide for unpooled keys; per-pool overrides for that pool | | `archive_delete_after_seconds` | 0 (keep) | purge an archived claim's store checkpoint this long after it was archived; the claim is then gone for good. On S3, chunks and manifests remain until [offline reclamation](#reclaiming-s3-storage). On archive this retention window replaces the live claim deadline; 0 clears the deadline so the archive is kept forever. An [evacuation](cluster.md#evacuating-a-node) keeps a longer live deadline, and a claim of a pool without an archive tier, whose `on_expire` is not `archive`, keeps its deadline until its first wake. Same node-wide/per-pool split | -| `archive_failover` | false | let another node adopt this node's archived claims once its node lease expires, and the running claims of pools with `replicate_seconds`: each archive and replication round also writes a row (token SHA-256, owner, epoch, checkpoint id, retention deadline, and the claim record with its env, `guest: false` values included, in plaintext) to the shared PostgreSQL, and a `Lookup` that reaches any node takes the claim over there (see [archived claims across node loss](cluster.md#archived-claims-across-node-loss)). Requires `meta_store` with a `cell` (rows are scoped by it, so meshes sharing one database never adopt from each other), a `checkpoint_store` of kind `s3` and a `mesh` with a `cluster_key` (a hand-off presents the claim's token to an address learned from gossip), all checked at config load. The node also renews a lease row every 5 s (see [node lease](cluster.md#node-lease)). With the meta store down, archives, wakes, releases and env writes of archived claims, and env writes and renews of hibernated replicated claims, fail and retry, and replicated claims stop being served once the lease is overdue (see [running claims](cluster.md#running-claims-across-node-loss)); other running claims never touch it. Restart only | +| `archive_failover` | false | let another node adopt this node's archived claims once its node lease expires, and the running claims of pools with `replicate_seconds`: each archive and replication round also writes a row (token SHA-256, owner, epoch, checkpoint id, retention deadline, and the claim record with its env, `guest: false` values included, in plaintext, unless `host_env_memory_only` is on) to the shared PostgreSQL, and a `Lookup` that reaches any node takes the claim over there (see [archived claims across node loss](cluster.md#archived-claims-across-node-loss)). Requires `meta_store` with a `cell` (rows are scoped by it, so meshes sharing one database never adopt from each other), a `checkpoint_store` of kind `s3` and a `mesh` with a `cluster_key` (a hand-off presents the claim's token to an address learned from gossip), all checked at config load. The node also renews a lease row every 5 s (see [node lease](cluster.md#node-lease)). With the meta store down, archives, wakes, releases and env writes of archived claims, and env writes and renews of hibernated replicated claims, fail and retry, and replicated claims stop being served once the lease is overdue (see [running claims](cluster.md#running-claims-across-node-loss)); other running claims never touch it. Restart only | | `mesh` | unset | join a cluster ([Clusters](cluster.md)); unset = single node | | `pools[]` | — | warm pools, keyed by `(template, net, size)`. `warm` defaults to 4 only when both `warm` and `warm_max` are omitted; an explicit `warm: 0` is kept, so `warm: 0` under a `warm_max` starts the pool empty and grows it on demand, and `warm: 0` alone registers the key with no warm VMs (`PUT /v1/pools` never defaults `warm`); `net` is `none` (the default) or `egress`; `size` is a tier, below (default `small`). Retune online without a restart via [`PUT /v1/pools`](sandboxd-api.md#put-v1pools) — omitted pools drain, and their goldens are deleted once they are idle (`warm: 0` keeps the golden). This is the **first-boot seed**: once a node takes a `PUT /v1/pools`, the applied set persists to `/pools.json` and overrides this section on every later boot (a startup log notes it); delete `pools.json` to return to config-owned pools. With a `meta_store` `cell`, the cell's set overrides it instead (see [shared pool set](#shared-pool-set)). Egress stays config-owned either way. A template re-pulled or re-imported under the same ref is picked up within a minute: the pool rebuilds its golden and replaces its warm VMs. See [state ownership](cluster.md#state-ownership) | @@ -622,6 +636,9 @@ golden fail the same way, it is retired again at most once every 30 s. The systemd unit is [`packaging/sandboxd.service`](https://github.com/cocoonstack/sandbox/blob/main/packaging/sandboxd.service); install it as shipped. Its `KillMode` and `TimeoutStopSec` are explained below. +`LimitCORE=0` keeps a crash from writing sandboxd's memory, which holds every +running claim's host-only env values, to a core file. For the same reason, +run sandboxd hosts with swap off or encrypted. Each record carries its syslog level, so alerting keys on systemd priority: `journalctl -u sandboxd -p err` lists the errors and `-p warning` the diff --git a/docs/egress.md b/docs/egress.md index 744757059..481c01ca6 100644 --- a/docs/egress.md +++ b/docs/egress.md @@ -372,20 +372,25 @@ gateway key per user, while the guest holds a placeholder. which hosts receive a secret, and the claim only supplies its value. A `guest: true` entry never feeds a secret. - **Precedence.** The claim's entry wins over the node's value for that claim; - an entry set to `""` suppresses the node's value. A secret with `value_env` + a `missing` entry ([claim env](sandboxd-api.md#getputpatch-v1sandboxesidenv)) + blocks the node's value until it is sent again. Prefer claim entries for per-user credentials: a node value is + shared by every claim of the cell and long-lived. A secret with `value_env` needs that variable set on the node: unset or empty, it stops the daemon at start and a reload answers 400. A secret only claims supply omits `value_env`; on a claim with no entry for it, the matched request passes as the guest sent it. - **Lifetime.** The env lives in the claim record in `claims.json` (0600 under `data_dir`), survives a restart, hibernate, wake, archive and - unarchive, and goes with the release or reap. A fork child, a checkpoint + unarchive, and goes with the release or reap. With + [`host_env_memory_only`](deploy.md#configuration) its `guest: false` values + live in memory only and come back `missing` after each of those but a + release, until the control plane sends them again. A fork child, a checkpoint branch and a promoted-template clone start with only the env their own claim sets. Processes captured live in a snapshot (sessions, detached execs, units) keep the source's guest values in their own environment; a `guest: false` value never enters the guest, so no snapshot, checkpoint or - store object holds one; the `archive_failover` row is the exception - ([security](security.md#deployment-assumptions)). + store object holds one; the `archive_failover` row is the exception unless + `host_env_memory_only` is on ([security](security.md#deployment-assumptions)). - **Audit.** An injection is recorded as the secret's name; the value is never logged, listed or returned. @@ -470,7 +475,7 @@ changed: resend it, and the resend replaces the row. - **Precedence.** The pool rule's own `secret` is set first and wins: a credential naming that header for a host the rule covers is a 400. Among credentials for the same header, the first by entry name whose placeholder - (if any) the request carries wins. An empty value injects nothing. + (if any) the request carries wins. A missing entry injects nothing. - **Echo.** Origins reflect the request URL: a redirect's `Location`, a paging link, an echoed query. On a request that carried a `query` credential, its value, as sent, query- or path-escaped, or with `/` written diff --git a/docs/sandboxd-api.md b/docs/sandboxd-api.md index 47bcaae80..810ce5c35 100644 --- a/docs/sandboxd-api.md +++ b/docs/sandboxd-api.md @@ -110,8 +110,9 @@ Auth: `Authorization: Bearer ` (when configured). - `env` is the claim's own environment, at most 64 entries: each name a shell identifier of at most 128 bytes, each value at most 8 KiB, a value and an inject `query` or `placeholder` with no control characters other than tab, and names, values and inject fields at most 64 KiB in total. An entry holds only `value`, `guest` and `inject`; any other - member, or `"guest": null`, answers 400, so a mistyped flag never lands a - host-only value in the guest. `guest` (default `true`) delivers the entry into the guest; a + member, `"guest": null`, a `guest: false` entry with an empty value, or + `missing` (the node's own marker) answers 400, so a mistyped flag never lands + a host-only value in the guest and a read sent back never blanks one. `guest` (default `true`) delivers the entry into the guest; a `guest: false` entry never enters it and only feeds the node's egress [secrets](egress.md#claim-env-and-secrets), or, with `inject` (`{"hosts": [...], "header": "...", "placeholder": "..."}` with `placeholder` @@ -1029,12 +1030,14 @@ an SDK caller should set. Auth: node API token. Root sees every live claim; a tenant sees only its own. The index is `{"sandboxes": [{id, key, tenant?, deadline, claimed_at?, hibernated, archived?, from_checkpoint?, claim_ref?, metadata?, on_expire?, cpu_count, mem_total_bytes, -volumes?: [{name, mount?, mode?, attach_only?}], volumes_want?: [{name, mount?, mode?, attach_only?}], restarts?, restarted_at?, failed?}]}` — `deadline` is `0001-01-01T00:00:00Z` on an archive kept forever; `cpu_count` and `mem_total_bytes` are the +volumes?: [{name, mount?, mode?, attach_only?}], volumes_want?: [{name, mount?, mode?, attach_only?}], restarts?, restarted_at?, failed?, env_missing?}]}` — `deadline` is `0001-01-01T00:00:00Z` on an archive kept forever; `cpu_count` and `mem_total_bytes` are the size tier's allocation; `restarts` counts the cold boots that replaced an exited VMM and `restarted_at` stamps the last one (each loses guest memory and the instance-metadata document, so a caller re-establishes guest state when `restarts` moves; the claim's env is written to the guest again); `failed` names why the VMM is down and was not restarted (see [dead VMMs](deploy.md#dead-vmms-and-host-restarts)); +`env_missing` is true while a `guest: false` env entry has no value on the node +([claim env](#getputpatch-v1sandboxesidenv)); `mode` is omitted for `ro`, matching the claim echo; never sandbox tokens, volume host paths, or catalog access lists. `claimed_at` is the first grant (renew and wake move `deadline` only). `?claim_ref=` keeps only the claims recorded under @@ -1085,13 +1088,13 @@ it reaches; on a cluster, send it to the claim's `owner_addr`. rules; `{"env": {}}` clears it. `PATCH` merges: each entry in the body is set, an entry of `null` removes that name, and every other entry is kept as stored, host-only values included, so a control plane that cannot read those values -back never has to send them again: +back sends them again only when the node reports them missing (below): ```json {"env": {"CB_META_wake": {"value": "w2"}, "OLD": null}} ``` -The claim rules apply to the merged env, so a patch that takes it past 64 +The size rules apply to the merged env, so a patch that takes it past 64 entries or 64 KiB answers 400 and changes nothing. `GET` answers the same shape with every `guest: false` value blanked, so a control plane sees the names without the values: @@ -1105,7 +1108,23 @@ An `inject` host the claim's pool does not intercept, or its tenant class does not allow, and a header the pool rule's own secret already sets on that host, answer 400 and store nothing. -The env is persisted with the claim. A `guest: false` change takes effect on +An entry the node holds no value for comes back with `"missing": true`, and +the claim's index row carries `env_missing`. Under +[`host_env_memory_only`](deploy.md#configuration) that happens when the claim +hibernates, after a restart and after an adoption. A missing entry injects +nothing and blocks the node's value for its secret; a missing upstream entry +falls back to the class or pool upstream. Send it again once the claim runs: +a `PATCH` replaces the entry whole, so repeat `guest` and `inject`: + +```json +{"env": {"GW_KEY": {"value": "Bearer …", "guest": false, "inject": {"hosts": ["api.example.com"], "header": "Authorization"}}}} +``` + +A value sent to a hibernated claim stays in that node's memory until the +claim hibernates again. + +The env is persisted with the claim, its `guest: false` values only without +`host_env_memory_only`. A `guest: false` change takes effect on the egress proxy's next request at any time, a hibernated or archived claim included, and never touches the guest. A change to the guest entries is written to the guest at once, so it needs a running guest: on a hibernated, diff --git a/docs/sdk.md b/docs/sdk.md index d42cab45c..a03027bac 100644 --- a/docs/sdk.md +++ b/docs/sdk.md @@ -315,6 +315,10 @@ the claim's owner with the client's API token, not the sandbox token. so a host-only value is never resent. A host-only change applies to the next egress request at any time; a guest change needs a running guest and answers a 409 `*APIError` on a hibernated or archived sandbox. +`Env` reports `Missing` on a host-only entry the node holds no value for, and +`SandboxSummary.EnvMissing` flags the claim; send the whole entry again with +`PatchEnv`. A host-only entry with an empty value, or one with `Missing` set, +answers 400. ## Renewing diff --git a/docs/security.md b/docs/security.md index 298c01f3b..9841f08c8 100644 --- a/docs/security.md +++ b/docs/security.md @@ -50,7 +50,11 @@ exposing any part of a deployment beyond a single trusted host. tokens travel as in-memory headers, never guest disk. Secrets come from the host environment or a claim's `guest: false` env, never the config file. That env stays in the node-local claim record, except in the - `archive_failover` row ([assumption 4](#deployment-assumptions)). + `archive_failover` row ([assumption 4](#deployment-assumptions)). With + [`host_env_memory_only`](deploy.md#configuration), `guest: false` values + stay in sandboxd's memory: `claims.json` and the rows written with it on hold + none, and a hibernating claim drops them. The control plane sends them again + after a hibernate, a restart or an adoption. - **Sandbox identity.** Every clone is reseeded (entropy, machine-id), so branches and forks never share an identity with their source. @@ -80,7 +84,7 @@ all of them. held at capture. Restrict store access; use S3 server-side encryption or an encrypted mount where that matters. With `archive_failover`, treat the shared PostgreSQL the same way: its rows hold claim env in plaintext, - `guest: false` values included. + `guest: false` values included unless `host_env_memory_only` is on. 5. **No custom NAT64 prefix routed on a sandboxd host** — the SSRF guard cannot see through an operator-specific translator prefix ([egress](egress.md#deployment-constraints)). diff --git a/packaging/sandboxd.service b/packaging/sandboxd.service index 8b6d3077c..f0f80516a 100644 --- a/packaging/sandboxd.service +++ b/packaging/sandboxd.service @@ -9,6 +9,7 @@ Restart=on-failure RestartSec=5s KillMode=process TimeoutStopSec=27min +LimitCORE=0 Environment=SANDBOXD_LOG_LEVEL=info [Install] diff --git a/sandboxd/config/config.go b/sandboxd/config/config.go index dcbd9e895..0a718b744 100644 --- a/sandboxd/config/config.go +++ b/sandboxd/config/config.go @@ -415,6 +415,8 @@ type Config struct { APIToken string `json:"api_token,omitempty"` MetaStore *MetaStoreConfig `json:"meta_store,omitempty"` Secrets []egress.SecretSpec `json:"secrets,omitempty"` + // HostEnvMemoryOnly keeps host-only env values out of claims.json and the archive rows, and drops them when a claim hibernates. + HostEnvMemoryOnly bool `json:"host_env_memory_only,omitzero"` IdleHibernateSeconds int `json:"idle_hibernate_seconds,omitzero"` ArchiveAfterSeconds int `json:"archive_after_seconds,omitzero"` diff --git a/sandboxd/outbound/claim.go b/sandboxd/outbound/claim.go index c4e23474a..ab67cd799 100644 --- a/sandboxd/outbound/claim.go +++ b/sandboxd/outbound/claim.go @@ -84,7 +84,7 @@ func (c *claim) Route() string { if view.upstreamEnv == "" { return "" } - if v, set := c.sb.HiddenEnv(view.upstreamEnv); set { + if v, set := c.sb.HiddenEnv(view.upstreamEnv); set && v != "" { if v == upstreamDirect { return "" } diff --git a/sandboxd/outbound/claim_test.go b/sandboxd/outbound/claim_test.go index db5406fd9..41d728fe7 100644 --- a/sandboxd/outbound/claim_test.go +++ b/sandboxd/outbound/claim_test.go @@ -38,6 +38,7 @@ func TestRouteTakesTheClaimThenClassThenPoolUpstream(t *testing.T) { {"class default", "desk", nil, "http://127.0.0.1:3002"}, {"claim value", "desk", types.Env{"EGRESS_UPSTREAM": {Value: "socks5://127.0.0.1:3003", Guest: hostOnly}}, "socks5://127.0.0.1:3003"}, {"claim direct", "desk", types.Env{"EGRESS_UPSTREAM": {Value: "direct", Guest: hostOnly}}, ""}, + {"claim value missing", "desk", types.Env{"EGRESS_UPSTREAM": {Guest: hostOnly, Missing: true}}, "http://127.0.0.1:3002"}, } { sb := &types.Sandbox{ID: "sb_route", Key: testKey, Tenant: "acme", EgressClass: tt.class} sb.SetEnv(tt.env) @@ -189,6 +190,22 @@ func TestAnInjectShapeRidesIntoTheCredential(t *testing.T) { } } +func TestAMissingEntryInjectsNothingAndBlocksTheNodeSecret(t *testing.T) { + h := newHost(t, &fakeEngine{}, &config.Config{}, nil) + sb := &types.Sandbox{} + sb.SetEnv(types.Env{ + "KEY": {Guest: hostOnly, Inject: &types.EnvInject{Hosts: []string{"api.github.com"}, Header: "X-Api-Key"}, Missing: true}, + "GH_TOKEN": {Guest: hostOnly, Missing: true}, + }) + c := newClaim(h, sb, nil, nil) + if creds := c.Credentials("api.github.com"); len(creds) != 0 { + t.Errorf("credentials %+v, want none for a missing entry", creds) + } + if _, value, ok := c.Header("gh"); !ok || value != "" { + t.Errorf("header %q %v, want the node value blocked", value, ok) + } +} + func TestNewConnectionsPastTheClaimRateAreRefused(t *testing.T) { var dials atomic.Int32 cfg := &config.Config{ diff --git a/sandboxd/outbound/view.go b/sandboxd/outbound/view.go index fc4c6ede9..89650384c 100644 --- a/sandboxd/outbound/view.go +++ b/sandboxd/outbound/view.go @@ -159,7 +159,7 @@ func (v *View) CheckEnv(env types.Env) error { } e, ok := env[name] switch { - case !ok: + case !ok || e.Missing: return nil case e.InGuest(): return fmt.Errorf("%s must be guest: false, it carries the upstream's credentials", name) diff --git a/sandboxd/pool/claims.go b/sandboxd/pool/claims.go index 4792ea04d..a3948463e 100644 --- a/sandboxd/pool/claims.go +++ b/sandboxd/pool/claims.go @@ -83,8 +83,9 @@ type claimRow struct { // claimStore persists claimed sandboxes across daemon restarts; warm VMs are not persisted. type claimStore struct { - path string - sync bool + path string + sync bool + dropHostEnv bool // writeMu orders commits and guards every row's raw and buf. writeMu sync.Mutex @@ -168,7 +169,11 @@ func (s *claimStore) put(id string, sb *types.Sandbox) { delete(s.rows, id) } else { refs = checkpointRefs(sb) - row := &claimRow{dto: dtoOf(sb), running: holdsVM(sb)} + dto := dtoOf(sb) + if s.dropHostEnv { + dto.Env = dto.Env.DropHostValues() + } + row := &claimRow{dto: dto, running: holdsVM(sb)} if row.running { s.running++ } diff --git a/sandboxd/pool/env.go b/sandboxd/pool/env.go index dd9646d25..f5d965de0 100644 --- a/sandboxd/pool/env.go +++ b/sandboxd/pool/env.go @@ -58,7 +58,7 @@ func (m *Manager) writeEnv(ctx context.Context, id, tenant, verb string, next fu if len(env) == 0 { env = nil } - if err := env.Validate(); err != nil { + if err := env.ValidateStored(); err != nil { m.mu.Unlock() return fmt.Errorf("%w: %w", ErrBadEnv, err) } diff --git a/sandboxd/pool/env_test.go b/sandboxd/pool/env_test.go index 4ed0ad221..957576239 100644 --- a/sandboxd/pool/env_test.go +++ b/sandboxd/pool/env_test.go @@ -632,6 +632,113 @@ func TestALostInterceptRefusesOnlyAChangedInject(t *testing.T) { } } +func TestHostEnvMemoryOnlyKeepsValuesOffDiskAndARestartReportsThemMissing(t *testing.T) { + for _, memoryOnly := range []bool{false, true} { + t.Run(fmt.Sprintf("memory_only=%v", memoryOnly), func(t *testing.T) { + eng := newFakeEngine() + dataDir := t.TempDir() + cfg := &config.Config{DataDir: dataDir, APIToken: "root", HostEnvMemoryOnly: memoryOnly} + m := envManagerOf(t, eng, cfg) + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: types.Env{"gw": {Value: "Bearer a", Guest: hostOnly}, "SEEN": {Value: "yes"}}}) + if err != nil { + t.Fatalf("claim: %v", err) + } + if v, _ := sb.HiddenEnv("gw"); v != "Bearer a" { + t.Errorf("running claim holds %q, want the value in memory", v) + } + claims, err := newClaimStore(dataDir, false).load() + if err != nil { + t.Fatalf("load: %v", err) + } + if gw := claims[sb.ID].Env["gw"]; (gw.Value == "") != memoryOnly || gw.Missing != memoryOnly || claims[sb.ID].Env["SEEN"].Value != "yes" { + t.Errorf("journal gw %+v, want the value kept only without memory_only", gw) + } + + m2 := envManagerOf(t, eng, cfg) + if err = m2.Reconcile(t.Context()); err != nil { + t.Fatalf("Reconcile: %v", err) + } + summary, _ := m2.Sandbox(sb.ID) + listed := m2.Sandboxes("", "", nil) + read, err := m2.Env(sb.ID, "") + if err != nil || summary.EnvMissing != memoryOnly || len(listed) != 1 || listed[0].EnvMissing != memoryOnly || read["gw"].Missing != memoryOnly { + t.Errorf("after a restart: summary %v, list %+v, read %+v %v; want missing reported only with memory_only", summary.EnvMissing, listed, read, err) + } + if err = m2.PatchEnv(t.Context(), sb.ID, types.EnvPatch{"gw": {Value: "Bearer b", Guest: hostOnly}}, ""); err != nil { + t.Fatalf("re-push: %v", err) + } + m2.mu.Lock() + restored := m2.claimed[sb.ID] + m2.mu.Unlock() + summary, _ = m2.Sandbox(sb.ID) + if v, _ := restored.HiddenEnv("gw"); v != "Bearer b" || summary.EnvMissing { + t.Errorf("after a re-push: %q, env_missing %v; want the value back and the flag clear", v, summary.EnvMissing) + } + }) + } +} + +func TestTurningHostEnvMemoryOnlyOnDropsTheValuesAJournalStillHolds(t *testing.T) { + eng := newFakeEngine() + dataDir := t.TempDir() + m := envManagerOf(t, eng, &config.Config{DataDir: dataDir, APIToken: "root"}) + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: types.Env{"gw": {Value: "Bearer a", Guest: hostOnly}}}) + if err != nil { + t.Fatalf("claim: %v", err) + } + m2 := envManagerOf(t, eng, &config.Config{DataDir: dataDir, APIToken: "root", HostEnvMemoryOnly: true}) + if err = m2.Reconcile(t.Context()); err != nil { + t.Fatalf("Reconcile: %v", err) + } + m2.mu.Lock() + restored := m2.claimed[sb.ID] + m2.mu.Unlock() + claims, err := newClaimStore(dataDir, false).load() + if v, _ := restored.HiddenEnv("gw"); v != "" || err != nil || claims[sb.ID].Env["gw"].Value != "" { + t.Errorf("after the first memory-only boot: held %q, journal %+v %v; want the value gone from both", v, claims[sb.ID].Env, err) + } +} + +func TestHostEnvMemoryOnlyDropsValuesWhenAClaimHibernates(t *testing.T) { + for _, memoryOnly := range []bool{false, true} { + t.Run(fmt.Sprintf("memory_only=%v", memoryOnly), func(t *testing.T) { + cfg := &config.Config{DataDir: t.TempDir(), APIToken: "root", HostEnvMemoryOnly: memoryOnly} + m := envManagerOf(t, newFakeEngine(), cfg) + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: types.Env{"gw": {Value: "Bearer a", Guest: hostOnly}}}) + if err != nil { + t.Fatalf("claim: %v", err) + } + hibernateClaim(t, m, sb) + v, _ := sb.HiddenEnv("gw") + held := v != "" + summary, _ := m.Sandbox(sb.ID) + if held == memoryOnly || summary.EnvMissing != memoryOnly { + t.Errorf("hibernated claim holds the value %v, env_missing %v; want it dropped only with memory_only", held, summary.EnvMissing) + } + }) + } +} + +func TestHostEnvMemoryOnlyDropsValuesWhenAnUnconfirmedHibernateIsAdopted(t *testing.T) { + eng := newFakeEngine() + m := envManagerOf(t, eng, &config.Config{DataDir: t.TempDir(), APIToken: "root", HostEnvMemoryOnly: true}) + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: types.Env{"gw": {Value: "Bearer a", Guest: hostOnly}}}) + if err != nil { + t.Fatalf("claim: %v", err) + } + leaveDanglingIntent(t, m, eng, sb) + eng.mu.Lock() + eng.snapListErr = nil + eng.mu.Unlock() + hibernateClaim(t, m, sb) + m.mu.Lock() + snap := sb.HibernateSnap + m.mu.Unlock() + if v, _ := sb.HiddenEnv("gw"); v != "" || snap == "" { + t.Errorf("adopted hibernate %q still holds %q, want it dropped", snap, v) + } +} + func injectEnv(value string, hosts ...string) types.Env { return types.Env{"KEY": {Value: value, Guest: hostOnly, Inject: &types.EnvInject{Hosts: hosts, Header: "X-Api-Key"}}} } @@ -649,13 +756,17 @@ func credentialsFor(sb *types.Sandbox, host string) string { } func envManager(t *testing.T, eng *fakeEngine, dataDir string, pools ...config.PoolSpec) *Manager { + t.Helper() + return envManagerOf(t, eng, &config.Config{DataDir: dataDir, APIToken: "root", Pools: pools}) +} + +func envManagerOf(t *testing.T, eng *fakeEngine, cfg *config.Config) *Manager { t.Helper() t.Setenv("GH_TOKEN", "node-gh") secrets := testSecrets(t, egress.SecretSpec{Name: "gh", Header: "Authorization", ValueEnv: "GH_TOKEN"}, egress.SecretSpec{Name: "gw", Header: "Authorization"}, ) - cfg := &config.Config{DataDir: dataDir, APIToken: "root", Pools: pools} m, err := NewManager(t.Context(), cfg, eng, secrets, testBackends(t, cfg)) if err != nil { t.Fatalf("manager: %v", err) diff --git a/sandboxd/pool/failover.go b/sandboxd/pool/failover.go index efbe621a5..6715c9e8b 100644 --- a/sandboxd/pool/failover.go +++ b/sandboxd/pool/failover.go @@ -139,6 +139,9 @@ func (m *Manager) putRow(ctx context.Context, sb *types.Sandbox, ck string, repl m.mu.Unlock() dto.Token, dto.VMName, dto.VsockSocket, dto.HibernateSnap, dto.PendingSnap, dto.ArchivePending, dto.ReplicaCk, dto.ReplicaPending = "", "", "", "", "", "", "", "" dto.ArchiveCk, dto.Deadline = ck, deadline + if m.view.Load().hostEnvMemory { + dto.Env = dto.Env.DropHostValues() + } dto.ReaderSafe = dto.ReaderSafe && !replica claim, err := json.Marshal(&dto) if err != nil { diff --git a/sandboxd/pool/failover_test.go b/sandboxd/pool/failover_test.go index 106a50932..f7113736d 100644 --- a/sandboxd/pool/failover_test.go +++ b/sandboxd/pool/failover_test.go @@ -4,6 +4,7 @@ import ( "context" "encoding/json/v2" "errors" + "fmt" "os" "path/filepath" "slices" @@ -1868,6 +1869,31 @@ func TestALookupRacingAWakeAdoptionTakesTheRowOnce(t *testing.T) { }) } +func TestFailoverRowsHoldNoHostOnlyValueWhenMemoryOnly(t *testing.T) { + for _, memoryOnly := range []bool{false, true} { + t.Run(fmt.Sprintf("memory_only=%v", memoryOnly), func(t *testing.T) { + c := newFailoverCluster(t) + c.hostEnvMemory = memoryOnly + a, b := c.pair(t) + sb := archivedWithEnv(t, a) + patchHostOnly(t, a, sb, "new") + if row := rowClaim(t, c, sb).Env["GH_TOKEN"]; (row.Value == "") != memoryOnly || row.Missing != memoryOnly { + t.Errorf("row entry %+v, want the value kept only without memory_only", row) + } + c.expire("a") + mustAdopt(t, b, sb, "adopt") + held, ok := b.byID(sb.ID) + if !ok { + t.Fatal("adopter has no claim") + } + summary, _ := b.Sandbox(sb.ID) + if v, _ := held.HiddenEnv("GH_TOKEN"); (v == "new") == memoryOnly || summary.EnvMissing != memoryOnly { + t.Errorf("adopter holds %q, env_missing %v; want the row's value only without memory_only", v, summary.EnvMissing) + } + }) + } +} + type failoverCluster struct { reg *archivestest.Mem moves moves.Registry @@ -1877,6 +1903,8 @@ type failoverCluster struct { nodes []*Manager egressPol *egress.Policy noArchives bool + + hostEnvMemory bool } func newFailoverCluster(t *testing.T) *failoverCluster { @@ -1951,7 +1979,7 @@ func (c *failoverCluster) nodeOn(t *testing.T, eng *fakeEngine, name, dataDir st spec.Egress = c.egressPol cfg := &config.Config{ APIToken: "root", DataDir: dataDir, CheckpointDir: c.ckDir, Pools: []config.PoolSpec{spec, {PoolKey: plainKey, Warm: 1}}, - Mesh: &config.MeshConfig{NodeID: name}, Volumes: c.volumes, + Mesh: &config.MeshConfig{NodeID: name}, Volumes: c.volumes, HostEnvMemoryOnly: c.hostEnvMemory, } if c.egressPol != nil { cfg.EgressCA = writeTestEgressCA(t) diff --git a/sandboxd/pool/hibernate.go b/sandboxd/pool/hibernate.go index e01fd42db..112ed7a2d 100644 --- a/sandboxd/pool/hibernate.go +++ b/sandboxd/pool/hibernate.go @@ -313,6 +313,7 @@ func (m *Manager) commitTransition(ctx context.Context, sb *types.Sandbox, snap, } // Under m.mu, so a racing release's event lands after this one and a shutdown cannot keep the record without it if snap != "" { + m.forgetHostEnvLocked(sb) m.recordHibernate(ctx, sb) } else { m.recordUsage(ctx, usage.Event{Event: "wake", ID: sb.ID, VMName: sb.VMName, Tenant: sb.Tenant, Deadline: sb.Deadline}) @@ -355,6 +356,7 @@ func (m *Manager) settlePendingSnap(ctx context.Context, sb *types.Sandbox) (ado if live { if exists { sb.HibernateSnap = pending + m.forgetHostEnvLocked(sb) m.recordHibernate(ctx, sb) } sb.PendingSnap = "" @@ -375,6 +377,12 @@ func (m *Manager) settlePendingSnap(ctx context.Context, sb *types.Sandbox) (ado return exists, nil } +func (m *Manager) forgetHostEnvLocked(sb *types.Sandbox) { + if m.view.Load().hostEnvMemory { + sb.SetEnv(sb.Env.DropHostValues()) + } +} + func (m *Manager) recordHibernate(ctx context.Context, sb *types.Sandbox) { m.counters.hibernates.Add(1) m.recordUsage(ctx, usage.Event{Event: "hibernate", ID: sb.ID, VMName: sb.VMName, Tenant: sb.Tenant}) diff --git a/sandboxd/pool/pool.go b/sandboxd/pool/pool.go index 431448479..5c946ec72 100644 --- a/sandboxd/pool/pool.go +++ b/sandboxd/pool/pool.go @@ -217,6 +217,8 @@ type SandboxSummary struct { Failed string `json:"failed,omitempty"` ClaimedAt time.Time `json:"claimed_at,omitzero"` Deadline time.Time `json:"deadline"` + // EnvMissing reports a host-only env entry whose value the node no longer holds. + EnvMissing bool `json:"env_missing,omitzero"` } // PoolInfo is the ops view of one pool. @@ -438,6 +440,7 @@ func NewManager(ctx context.Context, cfg *config.Config, eng Engine, secrets *eg refillKick: make(chan struct{}, 1), healSem: make(chan struct{}, maxConcurrentHeals), } + m.store.dropHostEnv = cfg.HostEnvMemoryOnly m.seedVolumes(cfg.Volumes) if err := m.loadCordon(); err != nil { return nil, err @@ -733,7 +736,7 @@ func summarize(sb *types.Sandbox) SandboxSummary { ID: sb.ID, Key: sb.Key, Tenant: sb.Tenant, Deadline: sb.Deadline, ClaimedAt: sb.ClaimedAt, Hibernated: sb.HibernateSnap != "", Archived: sb.ArchiveCk != "", FromCheckpoint: sb.FromCheckpoint, ClaimRef: sb.ClaimRef, Metadata: sb.Metadata, OnExpire: sb.OnExpire, - Restarts: sb.Restarts, RestartedAt: sb.RestartedAt, Failed: sb.Failed, + Restarts: sb.Restarts, RestartedAt: sb.RestartedAt, Failed: sb.Failed, EnvMissing: sb.Env.HasMissing(), CPUCount: spec.CPU, MemTotalBytes: spec.MemoryBytes, Volumes: slices.Clone(sb.Volumes), VolumesWant: slices.Clone(sb.VolumesWant), } diff --git a/sandboxd/pool/reconcile.go b/sandboxd/pool/reconcile.go index d65cb5dcf..c0a74a48b 100644 --- a/sandboxd/pool/reconcile.go +++ b/sandboxd/pool/reconcile.go @@ -42,6 +42,7 @@ func (m *Manager) Reconcile(ctx context.Context) error { m.mu.Lock() for _, sb := range claims { sb.SetEnv(sb.Env) + m.forgetHostEnvLocked(sb) rec, ok := live[sb.VMName] if !ok || rec.State != vmStateRunning { sb.SaveSnap = "" diff --git a/sandboxd/pool/upstream_test.go b/sandboxd/pool/upstream_test.go index 38c2d401b..dc5440b3d 100644 --- a/sandboxd/pool/upstream_test.go +++ b/sandboxd/pool/upstream_test.go @@ -21,6 +21,7 @@ func TestUpstreamEnvIsAdmittedHostOnlyAndAllowed(t *testing.T) { "guest entry": {types.EnvVar{Value: "http://127.0.0.1:3128"}, false}, "unlisted": {types.EnvVar{Value: "http://10.0.0.1:3128", Guest: hostOnly}, false}, "malformed": {types.EnvVar{Value: "ftp://127.0.0.1:21", Guest: hostOnly}, false}, + "missing": {types.EnvVar{Guest: hostOnly, Missing: true}, true}, } { err := m.checkEnv(types.Env{"EGRESS_UPSTREAM": tt.v}) if tt.ok != (err == nil) || (err != nil && !errors.Is(err, ErrBadEnv)) { diff --git a/sandboxd/pool/view.go b/sandboxd/pool/view.go index 0d37e1a17..b1e401640 100644 --- a/sandboxd/pool/view.go +++ b/sandboxd/pool/view.go @@ -23,6 +23,7 @@ type configView struct { memoryAnon int64 memoryPSI float64 noBalloon bool + hostEnvMemory bool maxClaims int maxRunning int @@ -45,6 +46,7 @@ func newConfigView(cfg *config.Config, secrets *egress.SecretStore) *configView poolReplicate: map[types.PoolKey]time.Duration{}, poolReaders: map[types.PoolKey]bool{}, noBalloon: cfg.NoBalloon, + hostEnvMemory: cfg.HostEnvMemoryOnly, maxClaims: cfg.MaxClaims, maxRunning: cfg.MaxRunning, diff --git a/sandboxd/server/server_test.go b/sandboxd/server/server_test.go index 238b09eb3..76b2ea132 100644 --- a/sandboxd/server/server_test.go +++ b/sandboxd/server/server_test.go @@ -287,10 +287,12 @@ func TestEnvVerbsForRootOrTheTenant(t *testing.T) { t.Errorf("read: %d %s as %q", code, body, mgr.gotTenant) } for body, want := range map[string]int{ - `{"vars":{}}`: http.StatusBadRequest, - `{"env":{"A-B":{"value":"x"}}}`: http.StatusBadRequest, - `{"env":{"A":{"value":"a\nb"}}}`: http.StatusBadRequest, - `{"env":{"A":{"value":"a","x":1}}}`: http.StatusBadRequest, + `{"vars":{}}`: http.StatusBadRequest, + `{"env":{"A-B":{"value":"x"}}}`: http.StatusBadRequest, + `{"env":{"A":{"value":"a\nb"}}}`: http.StatusBadRequest, + `{"env":{"A":{"value":"a","x":1}}}`: http.StatusBadRequest, + `{"env":{"H":{"value":"","guest":false}}}`: http.StatusBadRequest, + `{"env":{"H":{"value":"h","guest":false,"missing":true}}}`: http.StatusBadRequest, } { if code, _ := call(t, http.MethodPut, "root", body); code != want { t.Errorf("%s: %d, want %d", body, code, want) @@ -328,11 +330,13 @@ func TestEnvPatchVerb(t *testing.T) { t.Errorf("PatchEnv calls %q, want %q", mgr.envPatched, want) } for body, want := range map[string]int{ - `{"vars":{}}`: http.StatusBadRequest, - `{"env":{"A-B":null}}`: http.StatusBadRequest, - `{"env":{"A":{"value":"a\nb"}}}`: http.StatusBadRequest, - `{"env":{"A":{"value":"a","x":1}}}`: http.StatusBadRequest, - `{"env":{"A":{"value":"s","guest":null}}}`: http.StatusBadRequest, + `{"vars":{}}`: http.StatusBadRequest, + `{"env":{"A-B":null}}`: http.StatusBadRequest, + `{"env":{"A":{"value":"a\nb"}}}`: http.StatusBadRequest, + `{"env":{"A":{"value":"a","x":1}}}`: http.StatusBadRequest, + `{"env":{"A":{"value":"s","guest":null}}}`: http.StatusBadRequest, + `{"env":{"H":{"value":"","guest":false}}}`: http.StatusBadRequest, + `{"env":{"H":{"value":"h","guest":false,"missing":true}}}`: http.StatusBadRequest, } { if code := call(t, "root", body); code != want { t.Errorf("%s: %d, want %d", body, code, want) diff --git a/sandboxd/types/env.go b/sandboxd/types/env.go index 115da278d..837caebce 100644 --- a/sandboxd/types/env.go +++ b/sandboxd/types/env.go @@ -98,6 +98,8 @@ type EnvVar struct { Guest *bool `json:"guest,omitempty"` // Inject, on a host-only entry, makes the egress proxy send Value to the named hosts. Inject *EnvInject `json:"inject,omitempty"` + // Missing marks a host-only entry whose value the node no longer holds; only the node sets it. + Missing bool `json:"missing,omitzero"` } // InGuest reports whether the entry is delivered into the guest. @@ -106,14 +108,15 @@ func (v EnvVar) InGuest() bool { return v.Guest == nil || *v.Guest } // UnmarshalJSON rejects unknown members and a null guest, so a mistyped host-only flag never falls back to the guest. func (v *EnvVar) UnmarshalJSON(b []byte) error { var raw struct { - Value string `json:"value"` - Guest jsontext.Value `json:"guest,omitzero"` - Inject *EnvInject `json:"inject,omitempty"` + Value string `json:"value"` + Guest jsontext.Value `json:"guest,omitzero"` + Inject *EnvInject `json:"inject,omitempty"` + Missing bool `json:"missing,omitzero"` } if err := json.Unmarshal(b, &raw, json.RejectUnknownMembers(true)); err != nil { return fmt.Errorf("env entry: %w", err) } - *v = EnvVar{Value: raw.Value, Inject: raw.Inject} + *v = EnvVar{Value: raw.Value, Inject: raw.Inject, Missing: raw.Missing} if raw.Guest == nil { return nil } @@ -126,14 +129,27 @@ func (v *EnvVar) UnmarshalJSON(b []byte) error { } func (v EnvVar) same(other EnvVar) bool { - return v.Value == other.Value && v.InGuest() == other.InGuest() && v.Inject.equal(other.Inject) + return v.Value == other.Value && v.InGuest() == other.InGuest() && v.Missing == other.Missing && v.Inject.equal(other.Inject) } // Env is a claim's own environment: guest entries reach the guest, the others stay host-side for egress secrets. type Env map[string]EnvVar -// Validate enforces the entry count, the name grammar, a bounded value a header and an env file both carry, and a bounded total. +// Validate checks a caller's env: ValidateStored's rules, a value on every host-only entry, and no missing entry. func (e Env) Validate() error { + for name, v := range e { + switch { + case v.Missing: + return fmt.Errorf("env %s: missing is set by the node", name) + case !v.InGuest() && v.Value == "": + return fmt.Errorf("env %s: a host-only entry needs a value", name) + } + } + return e.ValidateStored() +} + +// ValidateStored enforces the entry count, the name grammar, a bounded value a header and an env file both carry, and a bounded total. +func (e Env) ValidateStored() error { if len(e) > maxEnvVars { return fmt.Errorf("env must contain at most %d entries, got %d", maxEnvVars, len(e)) } @@ -210,16 +226,24 @@ func (e Env) SameGuest(other Env) bool { return maps.EqualFunc(e.guest(), other.guest(), EnvVar.same) } -// Redacted copies e with every host-only value dropped, so a read names them without serving them. -func (e Env) Redacted() Env { - out := maps.Clone(e) - for name, v := range out { - if !v.InGuest() { - v.Value = "" - out[name] = v +// HasMissing reports whether any host-only entry lacks its value. +func (e Env) HasMissing() bool { + for _, v := range e { + if v.Missing { + return true } } - return out + return false +} + +// Redacted copies e with every host-only value dropped, so a read names them without serving them. +func (e Env) Redacted() Env { + return e.dropHostValues(false) +} + +// DropHostValues copies e with every host-only value dropped and marked missing. +func (e Env) DropHostValues() Env { + return e.dropHostValues(true) } // GuestFile renders the guest entries as a systemd EnvironmentFile, sorted by name. @@ -231,6 +255,17 @@ func (e Env) GuestFile() []byte { return b } +func (e Env) dropHostValues(missing bool) Env { + out := maps.Clone(e) + for name, v := range out { + if !v.InGuest() { + v.Value, v.Missing = "", v.Missing || missing + out[name] = v + } + } + return out +} + func (e Env) guest() Env { out := Env{} for name, v := range e { diff --git a/sandboxd/types/env_test.go b/sandboxd/types/env_test.go index 6a85dd423..1194812f4 100644 --- a/sandboxd/types/env_test.go +++ b/sandboxd/types/env_test.go @@ -33,6 +33,8 @@ func TestEnvValidate(t *testing.T) { {"over the bound", Env{"A": {Value: strings.Repeat("a", maxEnvValueBytes+1)}}, "at most"}, {"total at the bound", full, ""}, {"total over the bound", over, "total at most"}, + {"host-only without a value", Env{"TOKEN": {Guest: new(false)}}, "needs a value"}, + {"missing from a caller", Env{"TOKEN": {Value: "t", Guest: new(false), Missing: true}}, "set by the node"}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -219,6 +221,36 @@ func TestEnvSplitsGuestAndHostOnlyEntries(t *testing.T) { } } +func TestDroppedHostValuesAreMissingAndUnpublished(t *testing.T) { + inj := &EnvInject{Hosts: []string{"a.test"}, Header: "X-Key"} + env := Env{"G": {Value: "g"}, "H": {Value: "h", Guest: new(false), Inject: inj}} + dropped := env.DropHostValues() + if h := dropped["H"]; h.Value != "" || !h.Missing || h.Inject != inj || dropped["G"].Value != "g" || env["H"].Value != "h" || env.HasMissing() { + t.Errorf("DropHostValues = %v (source %v), want H missing with its inject kept, G and the source intact", dropped, env) + } + if !dropped.HasMissing() || !dropped.Redacted()["H"].Missing || dropped.Equal(env) { + t.Error("a dropped entry must stay missing through Redacted and differ from the held one") + } + if err := dropped.ValidateStored(); err != nil { + t.Errorf("ValidateStored = %v, want the node's missing entry admitted", err) + } + if err := dropped.Validate(); err == nil { + t.Error("Validate admitted a missing entry from a caller") + } + sb := &Sandbox{} + sb.SetEnv(dropped) + if v, ok := sb.HiddenEnv("H"); !ok || v != "" { + t.Errorf("HiddenEnv(H) = %q %v, want set and empty, so the entry blocks the node's value", v, ok) + } + for name := range sb.Injections("a.test") { + t.Errorf("a missing entry %s injects", name) + } + var decoded EnvVar + if err := json.Unmarshal([]byte(`{"value":"","guest":false,"missing":true}`), &decoded); err != nil || !decoded.Missing { + t.Errorf("decode = %+v %v, want missing", decoded, err) + } +} + func errMatches(err error, want string) bool { if want == "" { return err == nil diff --git a/sandboxd/types/types.go b/sandboxd/types/types.go index eecc59469..e52ba7c0c 100644 --- a/sandboxd/types/types.go +++ b/sandboxd/types/types.go @@ -383,7 +383,7 @@ func (s *Sandbox) SetEnv(e Env) { s.Env = e view := &envView{env: e} for name, v := range e { - if v.Inject != nil && !v.InGuest() { + if v.Inject != nil && !v.InGuest() && !v.Missing { view.injects = append(view.injects, namedVar{name, v}) } } diff --git a/sdk/go/env_test.go b/sdk/go/env_test.go index 191f402fe..b7a5884fd 100644 --- a/sdk/go/env_test.go +++ b/sdk/go/env_test.go @@ -54,7 +54,7 @@ func TestEnvVerbsUseTheAPITokenOnTheOwner(t *testing.T) { body, _ := io.ReadAll(r.Body) got <- request{r.Method, r.Header.Get("Authorization"), string(body)} if r.Method == http.MethodGet { - _, _ = io.WriteString(w, `{"env":{"MODE":{"value":"on"},"GW_KEY":{"value":"","guest":false}}}`) + _, _ = io.WriteString(w, `{"env":{"MODE":{"value":"on"},"GW_KEY":{"value":"","guest":false},"LOST":{"value":"","guest":false,"missing":true}}}`) return } w.WriteHeader(http.StatusNoContent) @@ -67,7 +67,7 @@ func TestEnvVerbsUseTheAPITokenOnTheOwner(t *testing.T) { if err != nil { t.Fatalf("Env: %v", err) } - if want := (map[string]EnvVar{"MODE": {Value: "on"}, "GW_KEY": {Guest: new(false)}}); !reflect.DeepEqual(env, want) { + if want := (map[string]EnvVar{"MODE": {Value: "on"}, "GW_KEY": {Guest: new(false)}, "LOST": {Guest: new(false), Missing: true}}); !reflect.DeepEqual(env, want) { t.Errorf("Env %v, want %v", env, want) } if r := <-got; r != (request{http.MethodGet, "Bearer api", ""}) { diff --git a/sdk/go/info.go b/sdk/go/info.go index f04b4a903..5fed90618 100644 --- a/sdk/go/info.go +++ b/sdk/go/info.go @@ -81,6 +81,8 @@ type SandboxSummary struct { RestartedAt time.Time `json:"restarted_at,omitzero"` // Failed names why the VMM is down and was not restarted; empty means serviceable. Failed string `json:"failed,omitempty"` + // EnvMissing reports a host-only env entry whose value the node no longer holds; re-push it with PatchEnv. + EnvMissing bool `json:"env_missing,omitzero"` } type sandboxListResponse struct { diff --git a/sdk/go/info_test.go b/sdk/go/info_test.go index d0ea754b4..522c2e5cb 100644 --- a/sdk/go/info_test.go +++ b/sdk/go/info_test.go @@ -42,7 +42,7 @@ func TestInfoReportsCapacityState(t *testing.T) { func TestSandboxesDecodeMetadataAndResources(t *testing.T) { ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _ = json.MarshalWrite(w, map[string]any{"sandboxes": []any{map[string]any{ - "id": "sb_1", "metadata": map[string]string{"team": "a"}, "cpu_count": 2, "mem_total_bytes": 1 << 30, "on_expire": "archive", + "id": "sb_1", "metadata": map[string]string{"team": "a"}, "cpu_count": 2, "mem_total_bytes": 1 << 30, "on_expire": "archive", "env_missing": true, }}}) })) t.Cleanup(ts.Close) @@ -51,7 +51,7 @@ func TestSandboxesDecodeMetadataAndResources(t *testing.T) { if err != nil { t.Fatalf("Sandboxes: %v", err) } - if len(list) != 1 || list[0].Metadata["team"] != "a" || list[0].CPUCount != 2 || list[0].MemTotalBytes != 1<<30 || list[0].OnExpire != "archive" { - t.Errorf("summaries %+v, want metadata team=a, 2 CPUs, 1 GiB", list) + if len(list) != 1 || list[0].Metadata["team"] != "a" || list[0].CPUCount != 2 || list[0].MemTotalBytes != 1<<30 || list[0].OnExpire != "archive" || !list[0].EnvMissing { + t.Errorf("summaries %+v, want metadata team=a, 2 CPUs, 1 GiB and env_missing", list) } } diff --git a/sdk/go/options.go b/sdk/go/options.go index aac6db590..f65b6ebbb 100644 --- a/sdk/go/options.go +++ b/sdk/go/options.go @@ -60,6 +60,8 @@ type EnvVar struct { Value string `json:"value"` Guest *bool `json:"guest,omitempty"` Inject *EnvInject `json:"inject,omitempty"` + // Missing, on a read, marks a host-only entry whose value the node no longer holds; a write that sets it is refused. + Missing bool `json:"missing,omitzero"` } // Option configures a New claim.